GNU bug report logs - #73418
ls (GNU coreutils) 9.4 is extremely slower than ls (GNU coreutils) 8.32 listing files on a cifs mounted share

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 73418 in the body.
You can then email your comments to 73418 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
To: bug-coreutils <at> gnu.org
Subject: ls (GNU coreutils) 9.4 is extremely slower than ls (GNU coreutils)
 8.32 listing files on a cifs mounted share
Date: Sun, 22 Sep 2024 08:21:30 +0200

[Message part 1 (text/plain, inline)]

Hi all,

During an upgrade from Ubuntu 22.04 to Ubuntu 24.04 we discovered that "ls
(GNU coreutils) 9.4" is extremely slower than "ls (GNU coreutils) 8.32"
listing files on a cifs mounted share.

Facts to help you in troubleshooting the problem.

In the same Ubuntu 24.04 machine I have the vanilla executable in
/usr/bin/ls and a copy of Ubuntu 22.04 in /tmp/ls

root <at> kube-node04:~# /usr/bin/ls --version
ls (GNU coreutils) 9.4

root <at> kube-node04:~# /tmp/ls --version
ls (GNU coreutils) 8.32

I have also a cifs share mounted by:

//172.16.0.100/export /cifs/colom/export cifs
vers=2.0,uid=50000,username=teledba,password=***** 0 1

This is what happens:

root <at> kube-node04:~# time /tmp/ls -l  /cifs/colom/export >/dev/null
real 0m0.232s
user 0m0.027s
sys 0m0.027s


root <at> kube-node04:~# time /usr/bin/ls -l  /cifs/colom/export >/dev/null
real 0m5.744s
user 0m0.068s
sys 0m0.349s

So "ls -l" in 9.4 is 50 time slower then "ls -l" in 8.32

If I launch the ls command without the -l parameter the computing times are
instead quite similar:

root <at> kube-node04:~# time /tmp/ls  /cifs/colom/export >/dev/null
real 0m0.198s
user 0m0.015s
sys 0m0.021s

root <at> kube-node04:~# time /usr/bin/ls  /cifs/colom/export >/dev/null
real 0m0.174s
user 0m0.017s
sys 0m0.016s

looking at the strace output I saw that the new version is doing for each
file processed the adjunctive call listxattr (in my case the directory
contains 14500 files):

...
statx(AT_FDCWD, "/cifs/colom/export/ARTP000.CSV",
AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT,
STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE,
{stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID,
stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=840, ...}) = 0
listxattr("/cifs/colom/export/ARTP000.CSV", "", 152) = 0
...

maybe it is the root cause of the elapsed time when the ls command had the
"-l" option fired ?

Regards

[Message part 2 (text/html, inline)]

Message #8 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>, 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Sun, 22 Sep 2024 10:42:10 +0100

On 22/09/2024 07:21, Gian Domenico Bonazzoli wrote:
> Hi all,
> 
> During an upgrade from Ubuntu 22.04 to Ubuntu 24.04 we discovered that "ls
> (GNU coreutils) 9.4" is extremely slower than "ls (GNU coreutils) 8.32"
> listing files on a cifs mounted share.
> 
> Facts to help you in troubleshooting the problem.
> 
> In the same Ubuntu 24.04 machine I have the vanilla executable in
> /usr/bin/ls and a copy of Ubuntu 22.04 in /tmp/ls
> 
> root <at> kube-node04:~# /usr/bin/ls --version
> ls (GNU coreutils) 9.4
> 
> root <at> kube-node04:~# /tmp/ls --version
> ls (GNU coreutils) 8.32
> 
> I have also a cifs share mounted by:
> 
> //172.16.0.100/export /cifs/colom/export cifs
> vers=2.0,uid=50000,username=teledba,password=***** 0 1
> 
> This is what happens:
> 
> root <at> kube-node04:~# time /tmp/ls -l  /cifs/colom/export >/dev/null
> real 0m0.232s
> user 0m0.027s
> sys 0m0.027s
> 
> 
> root <at> kube-node04:~# time /usr/bin/ls -l  /cifs/colom/export >/dev/null
> real 0m5.744s
> user 0m0.068s
> sys 0m0.349s
> 
> So "ls -l" in 9.4 is 50 time slower then "ls -l" in 8.32
> 
> If I launch the ls command without the -l parameter the computing times are
> instead quite similar:
> 
> root <at> kube-node04:~# time /tmp/ls  /cifs/colom/export >/dev/null
> real 0m0.198s
> user 0m0.015s
> sys 0m0.021s
> 
> root <at> kube-node04:~# time /usr/bin/ls  /cifs/colom/export >/dev/null
> real 0m0.174s
> user 0m0.017s
> sys 0m0.016s
> 
> looking at the strace output I saw that the new version is doing for each
> file processed the adjunctive call listxattr (in my case the directory
> contains 14500 files):
> 
> ...
> statx(AT_FDCWD, "/cifs/colom/export/ARTP000.CSV",
> AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT,
> STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE,
> {stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID,
> stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=840, ...}) = 0
> listxattr("/cifs/colom/export/ARTP000.CSV", "", 152) = 0
> ...
> 
> maybe it is the root cause of the elapsed time when the ls command had the
> "-l" option fired ?
> 
> Regards

You listxattr() above doesn't seem to be returning an error,
but perhaps this is another manifestation of
https://bugs.gnu.org/68283 ?

Message #11 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Sun, 29 Sep 2024 22:24:45 -0700

[Message part 1 (text/plain, inline)]

On 2024-09-21 23:21, Gian Domenico Bonazzoli wrote:
> looking at the strace output I saw that the new version is doing for each
> file processed the adjunctive call listxattr (in my case the directory
> contains 14500 files):
> 
> ...
> statx(AT_FDCWD, "/cifs/colom/export/ARTP000.CSV",
> AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT,
> STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE,
> {stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID,
> stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=840, ...}) = 0
> listxattr("/cifs/colom/export/ARTP000.CSV", "", 152) = 0
> ...
> 
> maybe it is the root cause of the elapsed time when the ls command had the
> "-l" option fired ?

What does "strace -r" report? That will tell us about elapsed time.

What options are you using with your CIFS mount? The "mount" command can 
tell you that.

What happens if you mount the CIFS file system with the nouser_xattr option?

Can you disable extended attributes from the server side? E.g., if it's 
a Samba server configure with "ea support = no". If so, does that help? 
See, for example, 
<https://www.truenas.com/community/threads/cifs-directory-browsing-slow-try-this.27751/>.

Looking at the current coreutils source, I noticed that 'ls' called 
getxattr when it didn't need to. I installed the attached patch to fix 
some of the issue; more could be done and perhaps I'll find the time. 
Among other things this patch should cause GNU ls to use llistxattr 
instead of listxattr which may make a difference.

That being said, it does appear that CIFS is quite slow about getting 
extended attributes.

(I wish Linux attributes weren't such a pain to deal with. Among other 
things, why aren't there any *at functions?)

[0001-ls-use-fewer-xattr-related-syscalls.patch (text/x-patch, attachment)]

Message #14 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Mon, 30 Sep 2024 09:19:34 +0200

[Message part 1 (text/plain, inline)]

Hi Paul:

These are the mount options on Ubuntu 24.04:

//172.16.0.100/export on /cifs/colom/export type cifs
(rw,relatime,vers=2.0,cache=strict,username=teledba,uid=50000,forceuid,gid=0,noforcegid,addr=172.16.0.100,file_mode=0755,dir_mode=0755,soft,nounix,mapposix,rsize=65536,wsize=65536,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)

instead on Ubuntu 22.04:

//172.20.150.103/External on /cifs/colom/export type cifs
(rw,relatime,vers=1.0,cache=strict,username=teledba,domain=WORKGROUP,uid=50000,forceuid,gid=0,noforcegid,addr=172.20.150.103,file_mode=0755,dir_mode=0755,soft,nounix,mapposix,rsize=61440,wsize=4096,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)

mentre lo strace -r shows:

     0.000105 listxattr("/cifs/colom/export/M606219.CSV", "", 152) = 0
     0.000630 statx(AT_FDCWD, "/cifs/colom/export/M606220.CSV",
AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT,
STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE,
{stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID,
stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=4992, ...}) = 0
     0.000077 listxattr("/cifs/colom/export/M606220.CSV", "", 152) = 0
     0.000464 statx(AT_FDCWD, "/cifs/colom/export/M606221.CSV",
AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT,
STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE,
{stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID,
stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=0, ...}) = 0
     0.000076 listxattr("/cifs/colom/export/M606221.CSV", "", 152) = 0
     0.000542 statx(AT_FDCWD, "/cifs/colom/export/M606222.CSV",
AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT,
STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE,
{stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID,
stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=0, ...}) = 0


Sorry but I can't change anything on the cifs server side, it is a closed
system (IBM AS/400)


On Mon, Sep 30, 2024 at 7:24 AM Paul Eggert <eggert <at> cs.ucla.edu> wrote:

> On 2024-09-21 23:21, Gian Domenico Bonazzoli wrote:
> > looking at the strace output I saw that the new version is doing for each
> > file processed the adjunctive call listxattr (in my case the directory
> > contains 14500 files):
> >
> > ...
> > statx(AT_FDCWD, "/cifs/colom/export/ARTP000.CSV",
> > AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT,
> > STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE,
> >
> {stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID,
> > stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=840, ...}) = 0
> > listxattr("/cifs/colom/export/ARTP000.CSV", "", 152) = 0
> > ...
> >
> > maybe it is the root cause of the elapsed time when the ls command had
> the
> > "-l" option fired ?
>
> What does "strace -r" report? That will tell us about elapsed time.
>
> What options are you using with your CIFS mount? The "mount" command can
> tell you that.
>
> What happens if you mount the CIFS file system with the nouser_xattr
> option?
>
> Can you disable extended attributes from the server side? E.g., if it's
> a Samba server configure with "ea support = no". If so, does that help?
> See, for example,
> <
> https://www.truenas.com/community/threads/cifs-directory-browsing-slow-try-this.27751/
> >.
>
> Looking at the current coreutils source, I noticed that 'ls' called
> getxattr when it didn't need to. I installed the attached patch to fix
> some of the issue; more could be done and perhaps I'll find the time.
> Among other things this patch should cause GNU ls to use llistxattr
> instead of listxattr which may make a difference.
>
> That being said, it does appear that CIFS is quite slow about getting
> extended attributes.
>
> (I wish Linux attributes weren't such a pain to deal with. Among other
> things, why aren't there any *at functions?)

[Message part 2 (text/html, inline)]

Message #17 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Mon, 30 Sep 2024 10:26:22 +0200

On 9/30/24 7:24 AM, Paul Eggert wrote:
> I installed the attached patch to fix some of the issue

There's a new syntax-check failure for a require_root_ test here:

FAIL: tests/ls/capability
=========================

++ initial_cwd_=/home/berny/git/coreutils
+++ testdir_prefix_
+++ printf gt
++ pfx_=gt
+++ mktempd_ /home/berny/git/coreutils gt-capability.sh.XXXX
+++ case $# in
+++ destdir_=/home/berny/git/coreutils
+++ template_=gt-capability.sh.XXXX
+++ MAX_TRIES_=4
+++ case $destdir_ in
+++ destdir_slash_=/home/berny/git/coreutils/
+++ case $template_ in
++++ unset TMPDIR
+++ d=/home/berny/git/coreutils/gt-capability.sh.h8yJ
+++ case $d in
+++ :
+++ test -d /home/berny/git/coreutils/gt-capability.sh.h8yJ
++++ ls -dgo /home/berny/git/coreutils/gt-capability.sh.h8yJ
+++ perms='drwx------ 2 4096 Sep 30 10:14 /home/berny/git/coreutils/gt-capability.sh.h8yJ'
+++ case $perms in
+++ :
+++ echo /home/berny/git/coreutils/gt-capability.sh.h8yJ
+++ return
++ test_dir_=/home/berny/git/coreutils/gt-capability.sh.h8yJ
++ cd /home/berny/git/coreutils/gt-capability.sh.h8yJ
++ case $srcdir in
++ srcdir=../.
++ builddir=..
++ export srcdir builddir
++ gl_init_sh_nl_='
'
++ IFS='
'
++ for sig_ in 1 2 3 13 15
+++ expr 1 + 128
++ eval 'trap '\''Exit 129'\'' 1'
+++ trap 'Exit 129' 1
++ for sig_ in 1 2 3 13 15
+++ expr 2 + 128
++ eval 'trap '\''Exit 130'\'' 2'
+++ trap 'Exit 130' 2
++ for sig_ in 1 2 3 13 15
+++ expr 3 + 128
++ eval 'trap '\''Exit 131'\'' 3'
+++ trap 'Exit 131' 3
++ for sig_ in 1 2 3 13 15
+++ expr 13 + 128
++ eval 'trap '\''Exit 141'\'' 13'
+++ trap 'Exit 141' 13
++ for sig_ in 1 2 3 13 15
+++ expr 15 + 128
++ eval 'trap '\''Exit 143'\'' 15'
+++ trap 'Exit 143' 15
++ saved_IFS='
'
++ IFS=:
++ new_PATH=
++ for dir in $PATH
++ IFS='
'
++ case "$dir" in
++ test -d /home/berny/coreutils/src/.
++ new_PATH=/home/berny/coreutils/src
++ for dir in $PATH
++ IFS='
'
++ case "$dir" in
++ test -d /home/berny/bin/.
++ new_PATH=/home/berny/coreutils/src:/home/berny/bin
++ for dir in $PATH
++ IFS='
'
++ case "$dir" in
++ test -d /usr/local/bin/.
++ new_PATH=/home/berny/coreutils/src:/home/berny/bin:/usr/local/bin
++ for dir in $PATH
++ IFS='
'
++ case "$dir" in
++ test -d /usr/bin/.
++ new_PATH=/home/berny/coreutils/src:/home/berny/bin:/usr/local/bin:/usr/bin
++ for dir in $PATH
++ IFS='
'
++ case "$dir" in
++ test -d /bin/.
++ new_PATH=/home/berny/coreutils/src:/home/berny/bin:/usr/local/bin:/usr/bin:/bin
++ for dir in $PATH
++ IFS='
'
++ case "$dir" in
++ test -d /sbin/.
++ new_PATH=/home/berny/coreutils/src:/home/berny/bin:/usr/local/bin:/usr/bin:/bin:/sbin
++ for dir in $PATH
++ IFS='
'
++ case "$dir" in
++ test -d /usr/sbin/.
++ new_PATH=/home/berny/coreutils/src:/home/berny/bin:/usr/local/bin:/usr/bin:/bin:/sbin:/usr/sbin
++ IFS='
'
++ PATH=/home/berny/coreutils/src:/home/berny/bin:/usr/local/bin:/usr/bin:/bin:/sbin:/usr/sbin
++ export PATH
++ trap remove_tmp_ EXIT
+ path_prepend_ ./src
+ test 1 '!=' 0
+ path_dir_=./src
+ case $path_dir_ in
+ abs_path_dir_=/home/berny/git/coreutils/./src
+ case $abs_path_dir_ in
+ PATH=/home/berny/git/coreutils/./src:/home/berny/coreutils/src:/home/berny/bin:/usr/local/bin:/usr/bin:/bin:/sbin:/usr/sbin
+ create_exe_shims_ /home/berny/git/coreutils/./src
+ case $EXEEXT in
+ return 0
+ shift
+ test 0 '!=' 0
+ export PATH
+ print_ver_ ls printf
+ require_built_ ls printf
+ skip_=no
+ for i in "$@"
+ case " $built_programs " in
+ for i in "$@"
+ case " $built_programs " in
+ test no = yes
+ test yes = yes
+ local i
+ for i in $*
+ env ls --version
ls (GNU coreutils) 9.5.121-e71b24
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Richard M. Stallman and David MacKenzie.
+ for i in $*
+ env printf --version
printf (GNU coreutils) 9.5.121-e71b24
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by David MacKenzie.
+ require_root_
+ uid_is_privileged_
++ id -u
+ my_uid=0
+ case $my_uid in
+ NON_ROOT_USERNAME=berny
++ id -g berny
+ NON_ROOT_GID=1000
+ grep '^[ ]*chroot' .././tests/ls/capability.sh
+ grep '^#define HAVE_CAP 1' /home/berny/coreutils/lib/config.h
+ grep 'usage: setcap'
+ umask 22
+ mkdir test test/dir
+ cd test
+ touch cap_pos dir/cap_pos dir/cap_neg
+ for file in cap_pos dir/cap_neg
+ setcap cap_net_bind_service=ep cap_pos
+ for file in cap_pos dir/cap_neg
+ setcap cap_net_bind_service=ep dir/cap_neg
+ code='30;41'
+ for ex in '' ex=:
+ LS_COLORS='di=:ca=30;41'
+ ls --color=always cap_pos dir
+ env printf '\e[0m\e[30;41mcap_pos\e[0m

dir:
\e[30;41mcap_neg\e[0m
cap_pos
'
+ compare out out_ok
+ compare_dev_null_ out out_ok
+ test 2 = 2
+ test xout = x/dev/null
+ test xout_ok = x/dev/null
+ return 2
+ case $? in
+ compare_ out out_ok
+ LC_ALL=C
+ diff -u out out_ok
--- out	2024-09-30 10:14:06.161475391 +0200
+++ out_ok	2024-09-30 10:14:06.164810391 +0200
@@ -1,5 +1,5 @@
-cap_pos
+cap_pos

 dir:
-cap_neg
+cap_neg
 cap_pos
+ fail=1
+ for ex in '' ex=:
+ LS_COLORS='di=:ex=:ca=30;41'
+ ls --color=always cap_pos dir
+ env printf '\e[0m\e[30;41mcap_pos\e[0m

dir:
\e[30;41mcap_neg\e[0m
cap_pos
'
+ compare out out_ok
+ compare_dev_null_ out out_ok
+ test 2 = 2
+ test xout = x/dev/null
+ test xout_ok = x/dev/null
+ return 2
+ case $? in
+ compare_ out out_ok
+ LC_ALL=C
+ diff -u out out_ok
--- out	2024-09-30 10:14:06.164810391 +0200
+++ out_ok	2024-09-30 10:14:06.168145391 +0200
@@ -1,5 +1,5 @@
-cap_pos
+cap_pos

 dir:
-cap_neg
+cap_neg
 cap_pos
+ fail=1
+ Exit 1
+ set +e
+ exit 1
+ exit 1
+ remove_tmp_
+ __st=1
+ cleanup_
+ :
+ test '' = yes
+ cd /home/berny/git/coreutils
+ chmod -R u+rwx /home/berny/git/coreutils/gt-capability.sh.h8yJ
+ rm -rf /home/berny/git/coreutils/gt-capability.sh.h8yJ
+ exit 1
FAIL tests/ls/capability.sh (exit status: 1)


Have a nice day,
Berny

Message #20 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>,
 Paul Eggert <eggert <at> cs.ucla.edu>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Wed, 2 Oct 2024 23:11:08 +0100

[Message part 1 (text/plain, inline)]

On 30/09/2024 09:26, Bernhard Voelker wrote:
> On 9/30/24 7:24 AM, Paul Eggert wrote:
>> I installed the attached patch to fix some of the issue
> 
> There's a new test failure for a require_root_ test here:
> 
> FAIL: tests/ls/capability
> =========================

That should be fixed by the attached.

thanks,
Pádraig

[ls-capability-fix.patch (text/x-patch, attachment)]

Message #23 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Wed, 2 Oct 2024 23:14:27 +0100

On 30/09/2024 06:24, Paul Eggert wrote:
> Looking at the current coreutils source, I noticed that 'ls' called
> getxattr when it didn't need to. I installed the attached patch to fix
> some of the issue; more could be done and perhaps I'll find the time.
> Among other things this patch should cause GNU ls to use llistxattr
> instead of listxattr which may make a difference.

I notice that tests/ls/getxattr-speedup.sh is failing now,
which I've not looked into yet.

cheers,
Pádraig

Message #26 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>,
 Bernhard Voelker <mail <at> bernhard-voelker.de>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Wed, 2 Oct 2024 15:24:41 -0700

On 2024-10-02 15:11, Pádraig Brady wrote:
> That should be fixed by the attached.

A quick look suggests that would be an improvement, yes.

I'm also working on a more comprehensive fix. This will require Gnulib 
changes.

Message #29 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>,
 Bernhard Voelker <mail <at> bernhard-voelker.de>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Wed, 2 Oct 2024 23:04:29 -0700

[Message part 1 (text/plain, inline)]

On 2024-10-02 15:24, Paul Eggert wrote:
> I'm also working on a more comprehensive fix. This will require Gnulib 
> changes.

I installed the attached (plus some other refactorings) to do that.

[0001-ls-tune-usage-of-getxattr-stat-syscalls.patch (text/x-patch, attachment)]

Message #32 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Wed, 2 Oct 2024 23:39:56 -0700

On 2024-10-02 15:14, Pádraig Brady wrote:
> 
> I notice that tests/ls/getxattr-speedup.sh is failing now,
> which I've not looked into yet.

I'm not seeing that after the changes I committed this evening. If you 
can still reproduce it please let us know the details.

Message #35 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Thu, 3 Oct 2024 15:28:59 +0100

On 03/10/2024 07:39, Paul Eggert wrote:
> On 2024-10-02 15:14, Pádraig Brady wrote:
>>
>> I notice that tests/ls/getxattr-speedup.sh is failing now,
>> which I've not looked into yet.
> 
> I'm not seeing that after the changes I committed this evening. If you
> can still reproduce it please let us know the details.

I can still repro.
I'm on BTRFS though I don't think that matters for this test.
I suspect the caching is being bypassed as we don't return ENOTSUP
in all cases where the underlying lgetxattr() returns ENOTSUP.
Actually in cases where the caching in file_has_aclinfo_cache()
is effective, and file_has_aclinfo() is bypassed, we get UMR
due to the uninitialized aclinfo struct.
So the whole interaction with file_has_aclinfo_cache() needs looking at.

If I remove the caching I also see issues with symlink handling,
where security contexts aren't read for symlinks.
Comparing system ls, with latest:
  $ ls -lZ INSTALL
  lrwxrwxrwx. 1 padraig padraig unconfined_u:object_r:user_home_t:s0 ...

  $ src/ls -lZ INSTALL
  lrwxrwxrwx 1 padraig padraig ? ...

I pushed two other fixes:

1. Since tests/ls/no-cap.sh is now always skipped.
I adjusted the test to avoid that.

2. ls -Z always output an error, so fixed up the
condition around the error() call in that case.

cheers,
Pádraig

Message #38 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Pádraig Brady
 <P <at> draigBrady.com>, Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Sun, 6 Oct 2024 19:53:09 +0200

[Message part 1 (text/plain, inline)]

On 10/3/24 08:04, Paul Eggert wrote:
> I installed the attached (plus some other refactorings) to do that.

I don't see test-suite failures anymore, thanks.

> [PATCH] ls: tune indicator_name

This change makes 'check-ls-dircolors' and therefore 'distcheck' fail.
The attached fixes the check.

Have a nice day,
Berny

[0001-maint-adjust-check-ls-dircolors-to-recent-change-in-.patch (text/x-patch, attachment)]

Message #41 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>,
 Paul Eggert <eggert <at> cs.ucla.edu>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Sun, 6 Oct 2024 21:23:54 +0100

On 06/10/2024 18:53, Bernhard Voelker wrote:
> On 10/3/24 08:04, Paul Eggert wrote:
>> I installed the attached (plus some other refactorings) to do that.
> 
> I don't see test-suite failures anymore, thanks.
> 
>   > [PATCH] ls: tune indicator_name
> 
> This change makes 'check-ls-dircolors' and therefore 'distcheck' fail.
> The attached fixes the check.

+1

thanks!

Pádraig

Message #44 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>,
 Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>
Cc: 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Sat, 9 Nov 2024 17:16:13 +0000

On 03/10/2024 15:28, Pádraig Brady wrote:
> On 03/10/2024 07:39, Paul Eggert wrote:
>> On 2024-10-02 15:14, Pádraig Brady wrote:
>>>
>>> I notice that tests/ls/getxattr-speedup.sh is failing now,
>>> which I've not looked into yet.
>>
>> I'm not seeing that after the changes I committed this evening. If you
>> can still reproduce it please let us know the details.
> 
> I can still repro.
> I'm on BTRFS though I don't think that matters for this test.
> I suspect the caching is being bypassed as we don't return ENOTSUP
> in all cases where the underlying lgetxattr() returns ENOTSUP.
> Actually in cases where the caching in file_has_aclinfo_cache()
> is effective, and file_has_aclinfo() is bypassed, we get UMR
> due to the uninitialized aclinfo struct.
> So the whole interaction with file_has_aclinfo_cache() needs looking at.

I can confirm that the following commit fixes both issues above:
https://github.com/coreutils/coreutils/commit/b857d66b5

> I also see issues with symlink handling,
> where security contexts aren't read for symlinks.
> Comparing system ls, with latest:
>     $ ls -lZ INSTALL
>     lrwxrwxrwx. 1 padraig padraig unconfined_u:object_r:user_home_t:s0 ...
> 
>     $ src/ls -lZ INSTALL
>     lrwxrwxrwx 1 padraig padraig ? ...

This is still an issue here.

thanks,
Pádraig

Message #49 received at 73418-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>, 73418-done <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Sat, 9 Nov 2024 16:57:45 -0800

[Message part 1 (text/plain, inline)]

On 2024-11-09 09:16, Pádraig Brady wrote:
>>     $ src/ls -lZ INSTALL
>>     lrwxrwxrwx 1 padraig padraig ? ...
> 
> This is still an issue here.

Oh, thanks for reminding me. That's a Gnulib bug, fixed here:

https://lists.gnu.org/r/bug-gnulib/2024-11/msg00075.html

with the fix propagated into Coreutils by installing the attached.

I thinks this closes out all the problems mentioned in this bug report, 
so I'm boldly closing it. We can reopen it if I'm wrong.

[0001-build-update-gnulib-submodule-to-latest.patch (text/x-patch, attachment)]

[0002-ls-add-test-case-for-ls-Z-bug.patch (text/x-patch, attachment)]

Message #52 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>, 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Sun, 10 Nov 2024 13:48:09 +0000

On 10/11/2024 00:57, Paul Eggert wrote:
> On 2024-11-09 09:16, Pádraig Brady wrote:
>>>      $ src/ls -lZ INSTALL
>>>      lrwxrwxrwx 1 padraig padraig ? ...
>>
>> This is still an issue here.
> 
> Oh, thanks for reminding me. That's a Gnulib bug, fixed here:
> 
> https://lists.gnu.org/r/bug-gnulib/2024-11/msg00075.html
> 
> with the fix propagated into Coreutils by installing the attached.
> 
> I thinks this closes out all the problems mentioned in this bug report,
> so I'm boldly closing it. We can reopen it if I'm wrong.

Thank you for the fixes.

There are some issues though with `./configure --disable-acl`.
In that case we no longer output security context with ls -lZ,
when previously we did. We should document that change at least,
but we may want to uncouple these also?

Note consequently the new test fails with --disable-acl.
As an aside, the new test would be best in its own test file
that then used require_selinux_, rather than messing with sestatus etc.

BTW I've pushed a tweak to gnulib to avoid a -Werror=unused-variable
issue with --disable-acl

thanks!
Pádraig.

Message #55 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>,
 Gnulib bugs <bug-gnulib <at> gnu.org>, 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Mon, 11 Nov 2024 08:47:09 -0800

[Message part 1 (text/plain, inline)]

On 2024-11-10 05:48, Pádraig Brady wrote:
> BTW I've pushed a tweak to gnulib to avoid a -Werror=unused-variable
> issue with --disable-acl

Thanks, I installed the attached further patch, since the res5t of the 
file uses MAYBE_UNUSED.

[0001-file-has-acl-_GL_UNUSED-MAYBE_UNUSED.patch (text/x-patch, attachment)]

Message #58 received at 73418 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Gian Domenico Bonazzoli <gbonazzoli <at> bonaz.it>,
 Gnulib bugs <bug-gnulib <at> gnu.org>, 73418 <at> debbugs.gnu.org
Subject: Re: bug#73418: ls (GNU coreutils) 9.4 is extremely slower than ls
 (GNU coreutils) 8.32 listing files on a cifs mounted share
Date: Mon, 11 Nov 2024 20:12:06 +0000

[Message part 1 (text/plain, inline)]

On 11/11/2024 16:47, Paul Eggert wrote:
> On 2024-11-10 05:48, Pádraig Brady wrote:
>> BTW I've pushed a tweak to gnulib to avoid a -Werror=unused-variable
>> issue with --disable-acl
> 
> Thanks, I installed the attached further patch, since the res5t of the
> file uses MAYBE_UNUSED.

Thanks for all the fixes.

With the attached two small changes, it all looks good here now.
I'll wait until tomorrow to apply these, in case it overlaps
with any changes you're making.

cheers,
Pádraig

[0001-ls-fix-security-context-indication-in-long-mode.patch (text/x-patch, attachment)]

[0002-tests-avoid-false-failure-with-disable-selinux.patch (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.