GNU bug report logs - #31038
mv copies in ls -r order

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 31038 in the body.
You can then email your comments to 31038 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: 積丹尼 Dan Jacobson <jidanni <at> jidanni.org>
To: bug-coreutils <at> gnu.org
Subject: mv copies in ls -r order
Date: Tue, 03 Apr 2018 15:52:15 +0800

$ mv -v dir1/* dir2
reveals that mv works backwards,
copying in ls -r order.
Well OK, but why is that order better than the order of the arguments it
was given?

Message #12 received at 31038-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: 積丹尼 Dan Jacobson <jidanni <at> jidanni.org>,
 31038-done <at> debbugs.gnu.org
Subject: Re: bug#31038: mv copies in ls -r order
Date: Tue, 3 Apr 2018 09:18:46 -0500

[Message part 1 (text/plain, inline)]

tag 31038 notabug
thanks

On 04/03/2018 02:52 AM, 積丹尼 Dan Jacobson wrote:
> $ mv -v dir1/* dir2
> reveals that mv works backwards,
> copying in ls -r order.

Not quite true.  It copies the arguments in the order given on the
command line (the * glob expands to a sorted list according to the
current locale's sorting rules) into the directory specified as the
final argument; this is NOT the same as 'ls -r' which lists ALL
arguments in a reverse-sorted lists.

For example:

$ mkdir dir1 dir2
$ touch dir1/a dir1/b
$ echo ls -r dira1/* dir2
ls -r dir1/a dir1/b dir2
$ ls -r dira1/* dir2
dir1/b  dir1/a

dir2:

(which listed both entries under dir1 first, rather than listing dir2
first - that is, the glob expanded things in sorted order, then ls -r
reversed files within the same directory to list b before a but did NOT
reverse directories themselves).

$ mv -v dir1/* dir2
'dir1/a' -> 'dir2/a'
'dir1/b' -> 'dir2/b'

Here, mv processed all arguments in the order they were given (a before
b), which is different from what you claim as the 'ls -r' behavior (b
before a, even when the command line listed a before b).

> Well OK, but why is that order better than the order of the arguments it
> was given?

Because treating the final argument as the destination directory is how
it's been done for 40+ years, and so it was standardized that way.
Changing it now would break users.  If you don't like it, use:

mv -v --target-directory dir2 dir1/*

This is not a bug, so I'm marking it as such in the database.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #15 received at 31038-done <at> debbugs.gnu.org (full text, mbox):

From: 積丹尼 Dan Jacobson <jidanni <at> jidanni.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: 31038-done <at> debbugs.gnu.org
Subject: Re: bug#31038: mv copies in ls -r order
Date: Tue, 03 Apr 2018 22:37:13 +0800

OK maybe I was just looking at the latter half of

$ sh O
/tmp
created directory '/var/tmp/y'
copied 'x/1' -> '/var/tmp/y/1'
copied 'x/2' -> '/var/tmp/y/2'
copied 'x/3' -> '/var/tmp/y/3'
copied 'x/4' -> '/var/tmp/y/4'
copied 'x/5' -> '/var/tmp/y/5'
copied 'x/6' -> '/var/tmp/y/6'
copied 'x/7' -> '/var/tmp/y/7'
copied 'x/8' -> '/var/tmp/y/8'
copied 'x/9' -> '/var/tmp/y/9'
removed 'x/9'
removed 'x/8'
removed 'x/7'
removed 'x/6'
removed 'x/5'
removed 'x/4'
removed 'x/3'
removed 'x/2'
removed 'x/1'
removed directory 'x'

$ cat O
set -eu
cd /tmp
mkdir x
cd x
seq 9|xargs touch
cd -
mv -v x /var/tmp/y

Message #18 received at 31038-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: 積丹尼 Dan Jacobson <jidanni <at> jidanni.org>
Cc: 31038-done <at> debbugs.gnu.org
Subject: Re: bug#31038: mv copies in ls -r order
Date: Tue, 3 Apr 2018 09:52:36 -0500

[Message part 1 (text/plain, inline)]

On 04/03/2018 09:37 AM, 積丹尼 Dan Jacobson wrote:
> OK maybe I was just looking at the latter half of
> 
> $ sh O
> /tmp
> created directory '/var/tmp/y'
> copied 'x/1' -> '/var/tmp/y/1'
> copied 'x/2' -> '/var/tmp/y/2'
> copied 'x/3' -> '/var/tmp/y/3'
> copied 'x/4' -> '/var/tmp/y/4'
> copied 'x/5' -> '/var/tmp/y/5'
> copied 'x/6' -> '/var/tmp/y/6'
> copied 'x/7' -> '/var/tmp/y/7'
> copied 'x/8' -> '/var/tmp/y/8'
> copied 'x/9' -> '/var/tmp/y/9'
> removed 'x/9'
> removed 'x/8'
> removed 'x/7'
> removed 'x/6'
> removed 'x/5'
> removed 'x/4'
> removed 'x/3'
> removed 'x/2'
> removed 'x/1'

Ah, so you're questioning the behavior of cross-volume moves, rather
than same-volume (where mv has to do separate non-atomic steps instead
of letting rename(2) do all the work).  Please, when you report
something, GIVE ALL THE DETAILS up front, rather than making us play
guessing games at how to reproduce things.

> removed directory 'x'
> 
> $ cat O
> set -eu
> cd /tmp
> mkdir x
> cd x
> seq 9|xargs touch
> cd -
> mv -v x /var/tmp/y

That is NOT the same command that you documented in your original report
(where you used a glob); rather, it is a single directory name, where mv
has to recurse itself.

Note that you have entirely changed the question from what I answered in
the previous mail.  My claim that POSIX requires mv to process things in
command line order is still true, but here, the command line order is
only a single directory, and there is no requirement in POSIX at the
order of the contents within the directory are handled, only that all
files within the directory eventually get reached.

In fact, there is no reason at all that file names in a cross-volume
move have to be visited in any particular order, so our choice is to
favor any order that is demonstratably faster, followed by any order
that is easier to code.  It's easier to process directory entries in
readdir() order (which is NOT necessarily sorted by filename); the only
reason to perform the copies in an order different from readdir() order
is if sorting things gives better performance (in fact, we've found that
sorting files, not by name order, but by inode number, tends to give
speedups for rotating disks, where visiting files in incrementing inode
order tends to be faster than visiting files in readdir() order).  But
when it comes to removing files, we have to perform bookkeeping in order
to carefully track which files need to be removed; and in that case, a
LIFO list is the simplest coding technique.

So, in your particular example, you touched nine files in order (which
probably had the side effect that readdir() order, name order, and inode
order all matched); once all nine files have been copied (using
whichever order was most efficient, although for your example all three
orders likely happened to be the same), then the removal is done by
processing the internal list of what still needs work, and that list
happens to be easiest in reverse order.

Still, I stand by my claim that this is not a bug, and that there is
nothing to change unless you can provide a benchmark that shows that
doing things in a different order offers a noticeable speedup.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #21 received at 31038-done <at> debbugs.gnu.org (full text, mbox):

From: 積丹尼 Dan Jacobson <jidanni <at> jidanni.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: 31038-done <at> debbugs.gnu.org
Subject: Re: bug#31038: mv copies in ls -r order
Date: Tue, 03 Apr 2018 23:13:33 +0800

OK. Please on (info "(coreutils) cp invocation") at
‘-v’
‘--verbose’
     Print the name of each file before copying it.
add:

"(Note this reveals internal order of operations, often in ls -U order,
thus occasionally appearing to be in ls, ls -t, ls -r, ls -rt etc. order.)"

(I mean on the same page you do go into such details:

‘-a’
‘--archive’
     Preserve as much as possible of the structure and attributes of the
     original files in the copy (but do not attempt to preserve internal
     directory structure; i.e., ‘ls -U’ may list the entries in a copied
     directory in a different order).  Try to preserve SELinux security
     context and extended attributes (xattr), but ignore any failure to
     do that and print no corresponding diagnostic.  Equivalent to ‘-dR

)

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.