GNU bug report logs - #28082
bash: /bin/rm: Argument list too long

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 28082 in the body.
You can then email your comments to 28082 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jonny Grant <jg <at> jguk.org>
To: bug-coreutils <at> gnu.org
Subject: bash: /bin/rm: Argument list too long
Date: Sun, 13 Aug 2017 22:35:54 +0100

Hello

Can this be resolved please? Seems a shame in 2017 on a machine with 
16GB of ram I suffer this. Each filename is around 40 bytes.

Roughly 60,000 files

So total strings around 2.4 MBytes. Quite a small limit.

$rm jonny*.*
bash: /bin/rm: Argument list too long


rm (GNU coreutils) 8.25

I'm not on this list, could you keep my email address in replies

Regards, Jonny

Message #10 received at 28082-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Jonny Grant <jg <at> jguk.org>, 28082-done <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Sun, 13 Aug 2017 15:26:53 -0700

Jonny Grant wrote:
> Can this be resolved please?

It cannot be resolved by sending email to bug-gnu-utils, because the problem is 
due to a limit enforced by your operating system kernel, not by bash or by 'rm' 
or by any other GNU utility. Please file a bug report with whoever maintains 
your kernel.

Message #13 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Jonny Grant <jg <at> jguk.org>, 28082 <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Sun, 13 Aug 2017 21:27:03 -0700

tag 28082 notabug
close 28082
stop

On 13/08/17 14:35, Jonny Grant wrote:
> Hello
> 
> Can this be resolved please? Seems a shame in 2017 on a machine with 
> 16GB of ram I suffer this. Each filename is around 40 bytes.
> 
> Roughly 60,000 files
> 
> So total strings around 2.4 MBytes. Quite a small limit.
> 
> $rm jonny*.*
> bash: /bin/rm: Argument list too long

The standard mechanism to scale this is to use xargs,
something like:

  find . -maxdepth 1 -name 'jonny*.*' -print0 | xargs -r0 rm

In any case this is not a limitation imposed by rm.

cheers,
Pádraig.

Message #16 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Pádraig Brady <P <at> draigBrady.com>,
 Jonny Grant <jg <at> jguk.org>, 28082 <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Mon, 14 Aug 2017 10:59:23 -0500

[Message part 1 (text/plain, inline)]

On 08/13/2017 11:27 PM, Pádraig Brady wrote:
>> $rm jonny*.*
>> bash: /bin/rm: Argument list too long
> 
> The standard mechanism to scale this is to use xargs,
> something like:
> 
>   find . -maxdepth 1 -name 'jonny*.*' -print0 | xargs -r0 rm
> 
> In any case this is not a limitation imposed by rm.

In fact, we have a FAQ about it:

https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Argument-list-too-long

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #19 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Jonny Grant <jg <at> jguk.org>
Cc: 28082 <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Mon, 14 Aug 2017 16:50:07 -0700

Jonny Grant wrote:
> do you know which kernel API has this limitation?

All kernels have a limitation there to some extent, except perhaps the Hurd. 
Sorry, I don't know what the limits are.

Message #22 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: Jonny Grant <jg <at> jguk.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 28082 <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Tue, 15 Aug 2017 08:19:13 +0100

On 15/08/17 00:50, Paul Eggert wrote:
> Jonny Grant wrote:
>> do you know which kernel API has this limitation?
> 
> All kernels have a limitation there to some extent, except perhaps the 
> Hurd. Sorry, I don't know what the limits are.

Ok thank you.

I imagine kernels just need a dynamic API, so it doesn't need to be a 
fixed buffer.

Jonny

Message #25 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: "Dmitry V. Levin" <ldv <at> altlinux.org>
To: Jonny Grant <jg <at> jguk.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 28082 <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Tue, 15 Aug 2017 12:45:49 +0300

On Tue, Aug 15, 2017 at 08:19:13AM +0100, Jonny Grant wrote:
> On 15/08/17 00:50, Paul Eggert wrote:
> > Jonny Grant wrote:
> >> do you know which kernel API has this limitation?
> > 
> > All kernels have a limitation there to some extent, except perhaps the 
> > Hurd. Sorry, I don't know what the limits are.
> 
> Ok thank you.
> 
> I imagine kernels just need a dynamic API, so it doesn't need to be a 
> fixed buffer.

It's a security limit rather than a fixed buffer, see e.g.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da029c11e6b12f321f36dac8771e833b65cec962


-- 
ldv

Message #28 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: "Dmitry V. Levin" <ldv <at> altlinux.org>, Jonny Grant <jg <at> jguk.org>
Cc: 28082 <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Fri, 25 Aug 2017 22:45:51 -0700

Dmitry V. Levin wrote:
> It's a security limit rather than a fixed buffer, see e.g.
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da029c11e6b12f321f36dac8771e833b65cec962
>
>
>   
Hmm,  original poster had about 2.3MB of data.
The patch referenced above says to limit arg stack usage to at
most 75% of _STK_LIM (6MB), or about 4.5MB.

Ah... also, 25% of your stack limit.  Mine's 8M, so that wouldn't allow
for 2.4MB of args.  You might get that specific case to pass
if you increased your stack limit(?), but it still would be
a problem in the general case.

Message #31 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: Jonny Grant <jg <at> jguk.org>
To: L A Walsh <coreutils <at> tlinx.org>, "Dmitry V. Levin" <ldv <at> altlinux.org>
Cc: 28082 <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Sat, 26 Aug 2017 10:39:00 +0300

On 26/08/17 08:45, L A Walsh wrote:
> Dmitry V. Levin wrote:
>> It's a security limit rather than a fixed buffer, see e.g.
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da029c11e6b12f321f36dac8771e833b65cec962 
>>
>>
>>
> Hmm,  original poster had about 2.3MB of data.
> The patch referenced above says to limit arg stack usage to at
> most 75% of _STK_LIM (6MB), or about 4.5MB.
> 
> Ah... also, 25% of your stack limit.  Mine's 8M, so that wouldn't allow
> for 2.4MB of args.  You might get that specific case to pass
> if you increased your stack limit(?), but it still would be
> a problem in the general case.


I don't know if there is a way that Bash, could split up the long 
argument lists to workaround a known limit. eg using xargs

Regards, Jonny

Message #34 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: Jonny Grant <jg <at> jguk.org>
To: "Dmitry V. Levin" <ldv <at> altlinux.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 28082 <at> debbugs.gnu.org
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Wed, 6 Sep 2017 15:54:10 +0300

On 15/08/17 12:45, Dmitry V. Levin wrote:
> On Tue, Aug 15, 2017 at 08:19:13AM +0100, Jonny Grant wrote:
>> On 15/08/17 00:50, Paul Eggert wrote:
>>> Jonny Grant wrote:
>>>> do you know which kernel API has this limitation?
>>>
>>> All kernels have a limitation there to some extent, except perhaps the
>>> Hurd. Sorry, I don't know what the limits are.
>>
>> Ok thank you.
>>
>> I imagine kernels just need a dynamic API, so it doesn't need to be a
>> fixed buffer.
> 
> It's a security limit rather than a fixed buffer, see e.g.
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da029c11e6b12f321f36dac8771e833b65cec962

Thank you for your reply.

My Ubuntu 16.04 limit is 2MB it seems:
$ getconf ARG_MAX
2097152


This laptop has 16GB RAM, so it is a shame it isn't much bigger, or 
dynamic so can be expanded when needed somehow. Those mapped pages of 
RAM wouldn't be wasted, as just VM right?

I imagine a lot of people may have 60,000 files in a directory like me 
these days. Latest Linux kernel just added support for billions of files 
per directory I read.

Regards, Jonny

Message #37 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: R0b0t1 <r030t1 <at> gmail.com>
To: Jonny Grant <jg <at> jguk.org>
Cc: 28082 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>,
 "Dmitry V. Levin" <ldv <at> altlinux.org>
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Thu, 7 Sep 2017 22:12:59 -0500

On Wed, Sep 6, 2017 at 7:54 AM, Jonny Grant <jg <at> jguk.org> wrote:
>
>
> On 15/08/17 12:45, Dmitry V. Levin wrote:
>>
>> On Tue, Aug 15, 2017 at 08:19:13AM +0100, Jonny Grant wrote:
>>>
>>> On 15/08/17 00:50, Paul Eggert wrote:
>>>>
>>>> Jonny Grant wrote:
>>>>>
>>>>> do you know which kernel API has this limitation?
>>>>
>>>>
>>>> All kernels have a limitation there to some extent, except perhaps the
>>>> Hurd. Sorry, I don't know what the limits are.
>>>
>>>
>>> Ok thank you.
>>>
>>> I imagine kernels just need a dynamic API, so it doesn't need to be a
>>> fixed buffer.
>>
>>
>> It's a security limit rather than a fixed buffer, see e.g.
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da029c11e6b12f321f36dac8771e833b65cec962
>
>
> Thank you for your reply.
>
> My Ubuntu 16.04 limit is 2MB it seems:
> $ getconf ARG_MAX
> 2097152
>
>
> This laptop has 16GB RAM, so it is a shame it isn't much bigger, or dynamic
> so can be expanded when needed somehow. Those mapped pages of RAM wouldn't
> be wasted, as just VM right?
>
> I imagine a lot of people may have 60,000 files in a directory like me these
> days. Latest Linux kernel just added support for billions of files per
> directory I read.
>

If this is a strict requirement, you could switch to Hurd. I checked
with a Hurd developer(?) some time ago and one of their design
philosophies is no artificial limits.

I wasn't able to find an actual citation for this behavior, sadly.

R0b0t1.

Message #40 received at 28082 <at> debbugs.gnu.org (full text, mbox):

From: Jonny Grant <jg <at> jguk.org>
To: R0b0t1 <r030t1 <at> gmail.com>
Cc: 28082 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>,
 "Dmitry V. Levin" <ldv <at> altlinux.org>
Subject: Re: bug#28082: bash: /bin/rm: Argument list too long
Date: Fri, 8 Sep 2017 13:04:13 +0300

On 08/09/17 06:12, R0b0t1 wrote:
> On Wed, Sep 6, 2017 at 7:54 AM, Jonny Grant <jg <at> jguk.org> wrote:
>>
>>
>> On 15/08/17 12:45, Dmitry V. Levin wrote:
>>>
>>> On Tue, Aug 15, 2017 at 08:19:13AM +0100, Jonny Grant wrote:
>>>>
>>>> On 15/08/17 00:50, Paul Eggert wrote:
>>>>>
>>>>> Jonny Grant wrote:
>>>>>>
>>>>>> do you know which kernel API has this limitation?
>>>>>
>>>>>
>>>>> All kernels have a limitation there to some extent, except perhaps the
>>>>> Hurd. Sorry, I don't know what the limits are.
>>>>
>>>>
>>>> Ok thank you.
>>>>
>>>> I imagine kernels just need a dynamic API, so it doesn't need to be a
>>>> fixed buffer.
>>>
>>>
>>> It's a security limit rather than a fixed buffer, see e.g.
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da029c11e6b12f321f36dac8771e833b65cec962
>>
>>
>> Thank you for your reply.
>>
>> My Ubuntu 16.04 limit is 2MB it seems:
>> $ getconf ARG_MAX
>> 2097152
>>
>>
>> This laptop has 16GB RAM, so it is a shame it isn't much bigger, or dynamic
>> so can be expanded when needed somehow. Those mapped pages of RAM wouldn't
>> be wasted, as just VM right?
>>
>> I imagine a lot of people may have 60,000 files in a directory like me these
>> days. Latest Linux kernel just added support for billions of files per
>> directory I read.
>>
> 
> If this is a strict requirement, you could switch to Hurd. I checked
> with a Hurd developer(?) some time ago and one of their design
> philosophies is no artificial limits.
> 
> I wasn't able to find an actual citation for this behavior, sadly.
> 
> R0b0t1.

Thank you for your reply.
Yes, probably Linux kernel / Glibc should include this improvement. I 
don't know who to ask there though.

Saves us each from needing to modularise/batch our commands in bash scripts.

Jonny

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.