GNU bug report logs - #27127
Failure in "ls -t" when parameter list is long

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 27127 in the body.
You can then email your comments to 27127 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Teppo Mäenpää <teppo <at> runbox.com>
To: bug-coreutils <at> gnu.org
Subject: Failure in "ls -t" when parameter list is long
Date: Sun, 28 May 2017 20:21:37 +0300

[Message part 1 (text/plain, inline)]

Dear all,

The command "ls -t" fails to sort files as promised, when the total length of 
the names of the files to be listed (and sorted) exceeds 128 KB. All files 
are still listed, just the sorting part stops working.

Man page does not mention that the sorting would work only with short file 
lists. Is this a bug? If not, what have I understood wrong?

The attached script demonstrates this, in three slightly different scenarios. 
The outcome of the script is that the total lenght of filenames, given to ls 
as command line parameter, appears significant.

Regards,


	Teppo Mäenpää

[bug_ls_-t.tar.gz (application/gzip, attachment)]

Message #8 received at 27127 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Teppo Mäenpää <tm <at> iki.fi>, 27127 <at> debbugs.gnu.org
Subject: Re: bug#27127: Failure in "ls -t" when parameter list is long
Date: Mon, 29 May 2017 01:46:04 +0100

On 28/05/17 18:21, Teppo Mäenpää wrote:
> Dear all,
> 
> The command "ls -t" fails to sort files as promised, when the total length of 
> the names of the files to be listed (and sorted) exceeds 128 KB. All files 
> are still listed, just the sorting part stops working.
> 
> Man page does not mention that the sorting would work only with short file 
> lists. Is this a bug? If not, what have I understood wrong?
> 
> The attached script demonstrates this, in three slightly different scenarios. 
> The outcome of the script is that the total lenght of filenames, given to ls 
> as command line parameter, appears significant.


128K is the defafult max arg size for xargs,
so I presume you're using xargs with ls and
thus you're getting separate invocations of ls.
When commands need to operate on all files at once
(like wc and du to print totals for example)
they have the --files0-from option to take input
on stdin from xargs rather than through command arguments.
Now this option might be added to ls, though while
sorting is an operation that needs to operate on all data,
it has scalability issues best dealt with in the external sort command
(handling tmp files etc. for arbitrarily large inputs).

For example I've done this sort of ordering in the past
with the `find | sort | xargs ls` pattern, which you can see in:
http://www.pixelbeat.org/scripts/newest

cheers,
Pádraig

Message #11 received at 27127 <at> debbugs.gnu.org (full text, mbox):

From: Teppo Mäenpää <teppo <at> runbox.com>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 27127 <at> debbugs.gnu.org
Subject: Re: bug#27127: Failure in "ls -t" when parameter list is long
Date: Mon, 29 May 2017 07:20:02 +0300

On Mon, May 29, 2017 at 01:46:04AM +0100, Pádraig Brady wrote:

>> The command "ls -t" fails to sort files as promised, when the total length of
>> the names of the files to be listed (and sorted) exceeds 128 KB. All files
>> are still listed, just the sorting part stops working.
>
>128K is the defafult max arg size for xargs,

Thank you for your quick reply.

You are right, that was the cause of the problem.


>they have the --files0-from option to take input
...
>Now this option might be added to ls,

That would be wonderful.

>For example I've done this sort of ordering in the past
>with the `find | sort | xargs ls` pattern, which you can see in:
>http://www.pixelbeat.org/scripts/newest

Having the find-command print the timestamp is a great idea, which did not 
cross my mind. Thanks!

======

Now, When I know it, I am able to find the 128KiB limitation from the man 
page of xargs. However, in my opinion, that radically changes the behavior, 
and the fact that xargs can cause multiple invocations should be mentioned in 
the "DESCRIPTION" part of xargs manpage. xargs could also emit something to 
standard error when it hits the boundary and splits the command line into 
fractions.

Ways to proceed:
0) Close this ticket as stupid user error.
1) Convert this ticket into a feature request for "ls --files0-from"
2) Convert this to documentation enhancement / stderr verbosity increase request 
for xargs

I am unsure which one of these would make most sense.

Thanks once more for the informative response!

Regards,


	Teppo

Message #14 received at 27127 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Teppo Mäenpää <tm <at> iki.fi>,
 Pádraig Brady <P <at> draigBrady.com>
Cc: 27127 <at> debbugs.gnu.org
Subject: Re: bug#27127: Failure in "ls -t" when parameter list is long
Date: Mon, 29 May 2017 08:19:17 +0200

On 05/29/2017 06:20 AM, Teppo Mäenpää wrote:
> Now, When I know it, I am able to find the 128KiB limitation from the man 
> page of xargs. However, in my opinion, that radically changes the behavior, 
> and the fact that xargs can cause multiple invocations should be mentioned in 
> the "DESCRIPTION" part of xargs manpage.

This is already mentioned in the DESCRIPTION section of xarg's man page [1]:

  The  command  line for command is built up until it reaches a
  system-defined limit (unless the -n and -L options are used).
  The specified command will be invoked as many times as
  necessary to use up the list of input items.
  [...]

[1]
http://git.savannah.gnu.org/cgit/findutils.git/tree/xargs/xargs.1#n53

> [...] xargs could also emit something to 
> standard error when it hits the boundary and splits the command line into 
> fractions.

To get an idea what xargs is executing, please use the -t option:

  -t, --verbose
    Print the command line on the standard error output before executing it.

If you want to discuss more about xargs, then please post to the
findutils mailing list, as that tool is maintained there.

Have a nice day,
Berny

Message #17 received at 27127 <at> debbugs.gnu.org (full text, mbox):

From: Teppo Mäenpää <teppo <at> runbox.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>
Cc: 27127 <at> debbugs.gnu.org, Pádraig Brady <P <at> draigBrady.com>
Subject: Re: bug#27127: Failure in "ls -t" when parameter list is long
Date: Mon, 29 May 2017 17:54:09 +0300

On Mon, May 29, 2017 at 08:19:17AM +0200, Bernhard Voelker wrote:

>On 05/29/2017 06:20 AM, Teppo Mäenpää wrote:
>> Now, When I know it, I am able to find the 128KiB limitation from the man

>This is already mentioned in the DESCRIPTION section of xarg's man page [1]:

Very true. I was looking at the manpage on Debian stable, which does not have 
it. Newer ones do. My bad.

>If you want to discuss more about xargs, then please post to the 
>findutils mailing list, as that tool is maintained there.

No need anymore. Thanks for the suggestion, though.

Regards,


	Teppo Mäenpää

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.