GNU bug report logs - #14023
dirname/basename unexpected results when run by xargs -I

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 14023 in the body.
You can then email your comments to 14023 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Juho-Pekka Kuitunen <artanicus <at> mythmardyl.org>
To: bug-coreutils <at> gnu.org
Subject: dirname/basename unexpected results when run by xargs -I
Date: Thu, 21 Mar 2013 18:36:21 +0200

Reproduce example;
$ echo "testdir/testfile" | xargs -I '{}' echo '{}', dir: $(echo
dirname '{}') = $(dirname '{}')

Expected output;
testdir/testfile, dir: dirname testdir/testfile = testdir

Actual output;
testdir/testfile, dir: dirname testdir/testfile = .

This behavior seems to be limited to the xargs -I replace-str usage
pattern, any other way I can think of running dirname works fine.
Basename has similarly wonky behavior, it prints the full string
instead of doing any stripping.

Not sure if there's a bug involved (or if it's on the findutils or
coreutils side). Could also just be something silly I'm overlooking.
I've tested this with findutils 4.4.2 and coreutils 8.5 & 8.13 with
identical results.

--
Juho-Pekka Kuitunen

Message #8 received at 14023 <at> debbugs.gnu.org (full text, mbox):

From: Bob Proulx <bob <at> proulx.com>
To: Juho-Pekka Kuitunen <artanicus <at> mythmardyl.org>
Cc: 14023 <at> debbugs.gnu.org
Subject: Re: bug#14023: dirname/basename unexpected results when run by xargs
	-I
Date: Thu, 21 Mar 2013 19:05:49 -0600

Juho-Pekka Kuitunen wrote:
> Reproduce example;
> $ echo "testdir/testfile" | xargs -I '{}' echo '{}', dir: $(echo dirname '{}') = $(dirname '{}')

Thank you for the report and the very nice test case.  It made
debugging this problem so very much simpler.

> Expected output;
> testdir/testfile, dir: dirname testdir/testfile = testdir
>
> Actual output;
> testdir/testfile, dir: dirname testdir/testfile = .

Ah...  But you have missed a critical point!  :-)  The $(...) is
expanded by the current shell and not by the xargs.  Use echo to see
what you are asking.

  $ echo xargs -I '{}' echo '{}', dir: $(echo dirname '{}') = $(dirname '{}')
  xargs -I {} echo {}, dir: dirname {} = .

> This behavior seems to be limited to the xargs -I replace-str usage
> pattern, any other way I can think of running dirname works fine.

The problem is the $(...) which is running the dirname during the
earlier shell command line parsing pass and passing the result off to
the xargs command.

> Basename has similarly wonky behavior, it prints the full string
> instead of doing any stripping.

Also using $(...)?  :-)

> Not sure if there's a bug involved (or if it's on the findutils or
> coreutils side). Could also just be something silly I'm overlooking.
> I've tested this with findutils 4.4.2 and coreutils 8.5 & 8.13 with
> identical results.

So...  Not a bug.  Agreed?  (I am having a good chuckle.  I hope you
will too.)  We will close the bug then.

Bob

Message #13 received at 14023 <at> debbugs.gnu.org (full text, mbox):

From: Juho-Pekka Kuitunen <artanicus <at> mythmardyl.org>
To: Bob Proulx <bob <at> proulx.com>
Cc: 14023 <at> debbugs.gnu.org
Subject: Re: bug#14023: dirname/basename unexpected results when run by xargs
	-I
Date: Fri, 22 Mar 2013 04:52:25 +0200

On Fri, Mar 22, 2013 at 3:05 AM, Bob Proulx <bob <at> proulx.com> wrote:
> Juho-Pekka Kuitunen wrote:
>> Reproduce example;
>> $ echo "testdir/testfile" | xargs -I '{}' echo '{}', dir: $(echo dirname '{}') = $(dirname '{}')
>
> Thank you for the report and the very nice test case.  It made
> debugging this problem so very much simpler.
>
>> Expected output;
>> testdir/testfile, dir: dirname testdir/testfile = testdir
>>
>> Actual output;
>> testdir/testfile, dir: dirname testdir/testfile = .
>
> Ah...  But you have missed a critical point!  :-)  The $(...) is
> expanded by the current shell and not by the xargs.  Use echo to see
> what you are asking.
>
>   $ echo xargs -I '{}' echo '{}', dir: $(echo dirname '{}') = $(dirname '{}')
>   xargs -I {} echo {}, dir: dirname {} = .
>
>> This behavior seems to be limited to the xargs -I replace-str usage
>> pattern, any other way I can think of running dirname works fine.
>
> The problem is the $(...) which is running the dirname during the
> earlier shell command line parsing pass and passing the result off to
> the xargs command.

So if I understand correctly, subshell replacements gets executed
before xargs fills in the replacements? I tried to rule out this
wonkyness with the "$(echo '{}')" bit and it seemed to work in the
correct order. The echo should produce an empty string if the subshell
was evaluated too soon?

>
>> Basename has similarly wonky behavior, it prints the full string
>> instead of doing any stripping.
>
> Also using $(...)?  :-)
>
>> Not sure if there's a bug involved (or if it's on the findutils or
>> coreutils side). Could also just be something silly I'm overlooking.
>> I've tested this with findutils 4.4.2 and coreutils 8.5 & 8.13 with
>> identical results.
>
> So...  Not a bug.  Agreed?  (I am having a good chuckle.  I hope you
> will too.)  We will close the bug then.
>

Very much possibly not a bug but I'm not 100% convinced yet. I've been
chuckling at this all evening, love a good brain teaser even if it
turns out to be something I overlooked instead of a bug. :-)

> Bob



--
Juho-Pekka Kuitunen

Message #16 received at 14023 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Juho-Pekka Kuitunen <artanicus <at> mythmardyl.org>
Cc: 14023 <at> debbugs.gnu.org, Bob Proulx <bob <at> proulx.com>
Subject: Re: bug#14023: dirname/basename unexpected results when run by xargs
	-I
Date: Thu, 21 Mar 2013 21:11:09 -0600

[Message part 1 (text/plain, inline)]

On 03/21/2013 08:52 PM, Juho-Pekka Kuitunen wrote:
> On Fri, Mar 22, 2013 at 3:05 AM, Bob Proulx <bob <at> proulx.com> wrote:
>> Juho-Pekka Kuitunen wrote:
>>> Reproduce example;
>>> $ echo "testdir/testfile" | xargs -I '{}' echo '{}', dir: $(echo dirname '{}') = $(dirname '{}')

>> The problem is the $(...) which is running the dirname during the
>> earlier shell command line parsing pass and passing the result off to
>> the xargs command.
> 
> So if I understand correctly, subshell replacements gets executed
> before xargs fills in the replacements? I tried to rule out this
> wonkyness with the "$(echo '{}')" bit and it seemed to work in the
> correct order. The echo should produce an empty string if the subshell
> was evaluated too soon?

If you want to see when the shell is evaluating $(), use 'set -vx':

$ set -vx
$ echo testdir/testfile | xargs -I '{}' echo '{}', dir: $(echo dirname
'{}') = $(dirname '{}')
echo testdir/testfile | xargs -I '{}' echo '{}', dir: $(echo dirname
'{}') = $(dirname '{}')
+ echo testdir/testfile
echo dirname '{}')
dirname '{}')
echo dirname '{}')
echo dirname '{}'
++ echo dirname '{}'
dirname '{}')
dirname '{}'
++ dirname '{}'
+ xargs -I '{}' echo '{},' dir: dirname '{}' = .
testdir/testfile, dir: dirname testdir/testfile = .
$ set -
set -
+ set -

What you seem to want to do is invoke a shell instance on each file
name; that would be done as follows:

$ echo testdir/testfile | xargs -I {} sh -c \
    'echo "$1", dir: $(echo dirname "$1") = $(dirname "$1")' sh {}
testdir/testfile, dir: dirname testdir/testfile = testdir

Or again with shell tracing (note there are two levels of shells - the
shell that spawns xargs, and the shell that xargs spawns, so I'm posting
two different traces):

$ set -vx
$ echo testdir/testfile | xargs -I {} sh -c 'echo "$1", dir: $(echo
dirname "$1") = $(dirname "$1")' sh {}
echo testdir/testfile | xargs -I {} sh -c 'echo "$1", dir: $(echo
dirname "$1") = $(dirname "$1")' sh {}
+ xargs -I '{}' sh -c 'echo "$1", dir: $(echo dirname "$1") = $(dirname
"$1")' sh '{}'
+ echo testdir/testfile
testdir/testfile, dir: dirname testdir/testfile = testdir
$ set -
set -
+ set -
$ echo testdir/testfile | xargs -I {} sh -cvx 'echo "$1", dir: $(echo
dirname "$1") = $(dirname "$1")' sh {}
echo "$1", dir: $(echo dirname "$1") = $(dirname "$1")
echo dirname "$1")
echo dirname "$1"
++ echo dirname testdir/testfile
dirname "$1")
dirname "$1"
++ dirname testdir/testfile
+ echo testdir/testfile, dir: dirname testdir/testfile = testdir
testdir/testfile, dir: dirname testdir/testfile = testdir

By the way, your question is mostly related to shell, and a bit with
xargs, and practically nothing to do with dirname.  Your confusion on
WHEN $() is expanded would apply no matter what executable you plug in
instead of dirname.  But since neither sh nor xargs belongs to
coreutils, you might get better answers by asking on a general forum on
shell programming subtleties.

> 
> Very much possibly not a bug but I'm not 100% convinced yet. I've been
> chuckling at this all evening, love a good brain teaser even if it
> turns out to be something I overlooked instead of a bug. :-)

Hopefully shell tracing has managed to convince you.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #19 received at 14023 <at> debbugs.gnu.org (full text, mbox):

From: Juho-Pekka Kuitunen <artanicus <at> mythmardyl.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: 14023 <at> debbugs.gnu.org, Bob Proulx <bob <at> proulx.com>
Subject: Re: bug#14023: dirname/basename unexpected results when run by xargs
	-I
Date: Sat, 23 Mar 2013 13:03:11 +0200

On Fri, Mar 22, 2013 at 5:11 AM, Eric Blake <eblake <at> redhat.com> wrote:
> On 03/21/2013 08:52 PM, Juho-Pekka Kuitunen wrote:
>> On Fri, Mar 22, 2013 at 3:05 AM, Bob Proulx <bob <at> proulx.com> wrote:
>>> Juho-Pekka Kuitunen wrote:
>>>> Reproduce example;
>>>> $ echo "testdir/testfile" | xargs -I '{}' echo '{}', dir: $(echo dirname '{}') = $(dirname '{}')
>
>>> The problem is the $(...) which is running the dirname during the
>>> earlier shell command line parsing pass and passing the result off to
>>> the xargs command.
>>
>> So if I understand correctly, subshell replacements gets executed
>> before xargs fills in the replacements? I tried to rule out this
>> wonkyness with the "$(echo '{}')" bit and it seemed to work in the
>> correct order. The echo should produce an empty string if the subshell
>> was evaluated too soon?
>
> If you want to see when the shell is evaluating $(), use 'set -vx':
>
> $ set -vx
> $ echo testdir/testfile | xargs -I '{}' echo '{}', dir: $(echo dirname
> '{}') = $(dirname '{}')
> echo testdir/testfile | xargs -I '{}' echo '{}', dir: $(echo dirname
> '{}') = $(dirname '{}')
> + echo testdir/testfile
> echo dirname '{}')
> dirname '{}')
> echo dirname '{}')
> echo dirname '{}'
> ++ echo dirname '{}'
> dirname '{}')
> dirname '{}'
> ++ dirname '{}'
> + xargs -I '{}' echo '{},' dir: dirname '{}' = .
> testdir/testfile, dir: dirname testdir/testfile = .
> $ set -
> set -
> + set -
>
> What you seem to want to do is invoke a shell instance on each file
> name; that would be done as follows:
>
> $ echo testdir/testfile | xargs -I {} sh -c \
>     'echo "$1", dir: $(echo dirname "$1") = $(dirname "$1")' sh {}
> testdir/testfile, dir: dirname testdir/testfile = testdir
>
> Or again with shell tracing (note there are two levels of shells - the
> shell that spawns xargs, and the shell that xargs spawns, so I'm posting
> two different traces):
>
> $ set -vx
> $ echo testdir/testfile | xargs -I {} sh -c 'echo "$1", dir: $(echo
> dirname "$1") = $(dirname "$1")' sh {}
> echo testdir/testfile | xargs -I {} sh -c 'echo "$1", dir: $(echo
> dirname "$1") = $(dirname "$1")' sh {}
> + xargs -I '{}' sh -c 'echo "$1", dir: $(echo dirname "$1") = $(dirname
> "$1")' sh '{}'
> + echo testdir/testfile
> testdir/testfile, dir: dirname testdir/testfile = testdir
> $ set -
> set -
> + set -
> $ echo testdir/testfile | xargs -I {} sh -cvx 'echo "$1", dir: $(echo
> dirname "$1") = $(dirname "$1")' sh {}
> echo "$1", dir: $(echo dirname "$1") = $(dirname "$1")
> echo dirname "$1")
> echo dirname "$1"
> ++ echo dirname testdir/testfile
> dirname "$1")
> dirname "$1"
> ++ dirname testdir/testfile
> + echo testdir/testfile, dir: dirname testdir/testfile = testdir
> testdir/testfile, dir: dirname testdir/testfile = testdir
>
> By the way, your question is mostly related to shell, and a bit with
> xargs, and practically nothing to do with dirname.  Your confusion on
> WHEN $() is expanded would apply no matter what executable you plug in
> instead of dirname.  But since neither sh nor xargs belongs to
> coreutils, you might get better answers by asking on a general forum on
> shell programming subtleties.
>
>>
>> Very much possibly not a bug but I'm not 100% convinced yet. I've been
>> chuckling at this all evening, love a good brain teaser even if it
>> turns out to be something I overlooked instead of a bug. :-)
>
> Hopefully shell tracing has managed to convince you.
>

Yup, that did the trick. Looks like I didn't understand the way xargs
works nearly well enough. Thanks for the lesson, my bad!

> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>



--
Juho-Pekka Kuitunen

Message #24 received at 14023 <at> debbugs.gnu.org (full text, mbox):

From: Bob Proulx <bob <at> proulx.com>
To: Juho-Pekka Kuitunen <artanicus <at> mythmardyl.org>
Cc: 14023 <at> debbugs.gnu.org
Subject: Re: bug#14023: dirname/basename unexpected results when run by xargs
	-I
Date: Sat, 23 Mar 2013 09:11:06 -0600

Juho-Pekka Kuitunen wrote:
> Eric Blake wrote:
> > By the way, your question is mostly related to shell, and a bit with
> > xargs, and practically nothing to do with dirname.  Your confusion on
> > WHEN $() is expanded would apply no matter what executable you plug in
> > instead of dirname.  But since neither sh nor xargs belongs to
> > coreutils, you might get better answers by asking on a general forum on
> > shell programming subtleties.
> >
> >> Very much possibly not a bug but I'm not 100% convinced yet. I've been
> >> chuckling at this all evening, love a good brain teaser even if it
> >> turns out to be something I overlooked instead of a bug. :-)
> >
> > Hopefully shell tracing has managed to convince you.
> 
> Yup, that did the trick. Looks like I didn't understand the way xargs
> works nearly well enough. Thanks for the lesson, my bad!

The place to look for the explanation is in the shell documentation.
In the bash manual this is in the "EXPANSION" section.  In that
section it walks through a number of expansions.  One of those is
"Command Substitution".  This is done by the shell uniformly for all
commands before the command is invoked so it and the other expansions
are a good thing to understand.  The shell effectively modifies the
command line you typed into a dynamically created set of arguments for
the command being invoked.

I have closed the bug ticket to finish our accounting.  But feel free
to continue adding information to this bug log.  It continues to be
active and available.

Bob

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.