GNU bug report logs -
#15727
doc: cp: expand dirs-vs-files with -f/--remove-dest
Previous Next
To reply to this bug, email your comments to 15727 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Sun, 27 Oct 2013 03:49:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Linda Walsh <coreutils <at> tlinx.org>:
New bug report received and forwarded. Copy sent to
bug-coreutils <at> gnu.org.
(Sun, 27 Oct 2013 03:49:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
I was trying to update a copy of /var (=>/root2/var).
It issues non-fatal, failure messages when copying from or to
a non-dir entry to a dir entry.
I tried -a to force the removal of the target dir, or target file.
It didn't work.
I tried --remove-destination, which is clearly not "file" specific
(i.e. should remove dir), it didn't work either.
The problem doesn't happen with pipes or sockets overwriting
files or any combo... seems just dirs are a problem.
mkdir a
touch a/file2dir
mkdir a/dir2file
cp -a a b
mv a/file2dir a/x
mv a/dir2file a/file2dir
mv a/x a/dir2file
---
state=
> ll a b
a:
total 0
-rw-rw-r-- 1 0 Oct 26 20:37 dir2file
drwxrwxr-x 2 6 Oct 26 20:37 file2dir/
b:
total 0
drwxrwxr-x 2 6 Oct 26 20:37 dir2file/
-rw-rw-r-- 1 0 Oct 26 20:37 file2dir
now try copying new changes from 'a' to 'b' -- can't do it.
> cp -a a/. b/.
cp: cannot overwrite non-directory ‘b/././file2dir’ with directory ‘a/./file2dir’
cp: cannot overwrite directory ‘b/././dir2file’ with non-directory
-a/--archive is suppose to duplicate *types* on the target -- it should
be able to make either change without --force or pre-removal,
HOWEVER,
I'd suggest
1) allowing directories to overwrite files as that seems reasonable...t
and
2) with either -f OR --remove-destination, a file would overwrite a directory
(dir removed).
2b) One might argue that with -f, where dest is a file the failure comes first,
aborting a try
for removal; however, certainly, with --remove-destination, the destination
(whatever it is)
should be removed 1st and no error should be occurring.
Version: (on suse linux)...
cp (GNU coreutils) 8.21 - Copyright (C) 2013
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Mon, 28 Oct 2013 00:39:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 15727 <at> debbugs.gnu.org (full text, mbox):
On 10/27/2013 03:48 AM, Linda Walsh wrote:
>
> I was trying to update a copy of /var (=>/root2/var).
> It issues non-fatal, failure messages when copying from or to
> a non-dir entry to a dir entry.
>
> I tried -a to force the removal of the target dir, or target file.
>
> It didn't work.
>
> I tried --remove-destination, which is clearly not "file" specific
> (i.e. should remove dir), it didn't work either.
>
> The problem doesn't happen with pipes or sockets overwriting
> files or any combo... seems just dirs are a problem.
>
> mkdir a
> touch a/file2dir
> mkdir a/dir2file
> cp -a a b
> mv a/file2dir a/x
> mv a/dir2file a/file2dir
> mv a/x a/dir2file
> ---
> state=
>
>> ll a b
> a:
> total 0
> -rw-rw-r-- 1 0 Oct 26 20:37 dir2file
> drwxrwxr-x 2 6 Oct 26 20:37 file2dir/
> b:
> total 0
> drwxrwxr-x 2 6 Oct 26 20:37 dir2file/
> -rw-rw-r-- 1 0 Oct 26 20:37 file2dir
>
> now try copying new changes from 'a' to 'b' -- can't do it.
>
>> cp -a a/. b/.
> cp: cannot overwrite non-directory ‘b/././file2dir’ with directory ‘a/./file2dir’
> cp: cannot overwrite directory ‘b/././dir2file’ with non-directory
>
>
> -a/--archive is suppose to duplicate *types* on the target -- it should
> be able to make either change without --force or pre-removal,
> HOWEVER,
>
> I'd suggest
> 1) allowing directories to overwrite files as that seems reasonable...t
> and
> 2) with either -f OR --remove-destination, a file would overwrite a directory (dir removed).
> 2b) One might argue that with -f, where dest is a file the failure comes first, aborting a try
> for removal; however, certainly, with --remove-destination, the destination (whatever it is)
> should be removed 1st and no error should be occurring.
>
> Version: (on suse linux)...
>
> cp (GNU coreutils) 8.21 - Copyright (C) 2013
So overwriting files with dirs and vice versa
is prohibited by POSIX. The existing cp options
do not adjust this aspect. If you don't care what's
in the destination at all, why not just rm it before the copy?
thanks,
Pádraig.
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Mon, 28 Oct 2013 07:10:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 15727 <at> debbugs.gnu.org (full text, mbox):
On 10/27/2013 5:38 PM, Pádraig Brady wrote:
> So overwriting files with dirs and vice versa
> is prohibited by POSIX. The existing cp options
> do not adjust this aspect. If you don't care what's
> in the destination at all, why not just rm it before the copy?
Um --- isn't that what the "--remove-destination" option is supposed to
do?
Also note, I tried to use it with the "update" option, since in my real
case, in "/var/log", some programs had changed from using a single
log file to using their own dir under /var/log and multiple log files.
It's not like I'm making that up.
As for the posix comment. Since when is FSF/Gnu governed by a
corporate controlled POSIX? I.e. POSIX is supposed to supply
interop standards, not limitations on functionality.
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Mon, 28 Oct 2013 20:57:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 15727 <at> debbugs.gnu.org (full text, mbox):
Hi Linda,
On 10/28/2013 08:09 AM, Linda Walsh wrote:
> On 10/27/2013 5:38 PM, Pádraig Brady wrote:
>> So overwriting files with dirs and vice versa
>> is prohibited by POSIX. The existing cp options
>> do not adjust this aspect. If you don't care what's
>> in the destination at all, why not just rm it before the copy?
>
>
> Um --- isn't that what the "--remove-destination" option is supposed to
> do?
"info coreutils 'cp invocation'" says:
`--remove-destination'
Remove each existing destination file before attempting to open it
(contrast with `-f' above).
In this case, "file" really means a regular file (or socket, fifo, ...)
but no directory. The documentation could be clearer about that ...
> Also note, I tried to use it with the "update" option, [...]
... and it may be a question to discuss whether --remove-destination
should be able to rmdir() emtpy directories, but that GNU extension
should never help you out in the case of non-empty directories.
I got the overall impression that you try to sync a source to a
destination. Tools like rsync may be better for such a scenario
than cp(1) which is made primarily to
cp: copy files and directories
Have a nice day,
Berny
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Mon, 28 Oct 2013 22:06:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 15727 <at> debbugs.gnu.org (full text, mbox):
On 10/28/2013 1:56 PM, Bernhard Voelker wrote:
>> If you don't care what's in the destination at all,
>> why not just rm it before the copy?
>>
>> "--remove-destination" option is supposed to do?
> "info coreutils 'cp invocation'" says:
----
But if I specify "-a" or "-r" those imply recursive. Why has it been
assumed that recursive doesn't mean to remove files (and then empty directories)
on the target?
>> Also note, I tried to use it with the "update" option, [...]
>
> ... and it may be a question to discuss whether --remove-destination
> should be able to rmdir() emtpy directories, but that GNU extension
> should never help you out in the case of non-empty directories.
---
But if combined with "recursive"? There's also the -T
option which says to treat the destination as a "file" and not a "dir" --
which could be construed as a deliberate choice on the part of the user
to circumvent restrictions on directory removal -- i.e. specifically
since its intent is to ignore the fact that the target is a directory and
to treat it as a file.
> I got the overall impression that you try to sync a source to a
> destination. Tools like rsync may be better for such a scenario
> than cp(1) which is made primarily to
>
> cp: copy files and directories
----
That was true until various options (and restrictions) were added.
"-f" used to be more forceful about removing obstacles (though I can't say
it ever used to override directory restrictions). But adding to it's copy
'mission', --update, and --remove-destination, but gave a basis for expanding
the definition of cp. As for 'rsync', it's speed is *comparatively* abysmal
for local-to-local copies. For that matter, 'cp' could _relearn_ a thing
or two from 'dd' when it comes to speed. Used to be that cp was nearly as
fast as 'dd', but now that often doesn't seem to be the case.
I'm fine with the idea of requiring a '__GNU_ENHANCEMENTS__=[list]'
option in the ENV, or consulting an "rc" file in
something like (~/.rc/Gnu/<package|tool>|/etc/Gnu/coreutils) that
could list areas to override restrictions & feature removals like:
(cp_dir_restrictions,rm_dot_override[etc] to restore or allow
enhancements beyond the growing, posix-restrictions. Even a
posix=pre2003 would be useful, since it was about the 2003 updates
that started changing posix from descriptive to proscriptive.
Please remember the idea behind things like 'ksh' and 'bash' (bourne-again)
were to allow features not in the original posix spec --- many of which posix
eventually accepted. If there is nothing that provides an outlet for these,
posix will only devolve instead of evolve. FSF/Gnu used to offer evolutionary
options over posix -- but now... it seems fsf/gnu has decided to submit to
the posix restrictions as a ceiling. I still assert that any posix rules
that go beyond the original POSIX mission statement to be Descriptive
(or to provide for minimum requirements/features) that move into *proscriptive*,
or disallowal of features is, by definition outside the bounds of the
original POSIX and shouldn't really be abusing the name (I *liked* POSIX up
until it started removing features!)....
> Have a nice day,
---
Cheers!
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Tue, 29 Oct 2013 11:26:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 15727 <at> debbugs.gnu.org (full text, mbox):
> On October 28, 2013 at 11:05 PM Linda Walsh <coreutils <at> tlinx.org> wrote:
> On 10/28/2013 1:56 PM, Bernhard Voelker wrote:
>> [...] Tools like rsync may be better for such a scenario
>> than cp(1) which is made primarily to
> [...] As for 'rsync', it's speed is *comparatively* abysmal
> for local-to-local copies.
So why not enhance rsync regarding performance?
> For that matter, 'cp' could _relearn_ a thing
> or two from 'dd' when it comes to speed.
IMO no: this would add further bloat to the sources
which in turn is harder to maintain. Ever looked at
copy.c, e.g. at copy_internal()?
http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/copy.c#n1598
> Used to be that cp was nearly as
> fast as 'dd', but now that often doesn't seem to be the case.
hmm, well, could it be that this is because many features have
been added over the years? ...
I think rm is simply not the right tool for such kind
of synchronization.
Have a nice day,
Berny
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Tue, 29 Oct 2013 16:08:01 GMT)
Full text and
rfc822 format available.
Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):
* Bernhard Voelker (mail <at> bernhard-voelker.de) [20131029 12:26]:
> hmm, well, could it be that this is because many features have
> been added over the years? ...
Leave alone checks that cp needs to do but dd doesn't.
Philipp
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Tue, 29 Oct 2013 18:21:01 GMT)
Full text and
rfc822 format available.
Message #26 received at 15727 <at> debbugs.gnu.org (full text, mbox):
> So why not enhance rsync regarding performance?
----
It's designed for slow, remote copies. It special-cases local copies, but
uses much of the of the old code. It also follows it's
manpage. It also does id translation and storage of ACL's and
Xattrs on file systems that don't support it. It has a complete
different set of requirements.
cp was designed for locally mounted file systems. Having
cp really be able to do "updates" with -u and really treat
destinations as files (-T) and really remove destinations
before attempting a copy, seems much more reasonable in line
with what cp should do.
>
>> For that matter, 'cp' could _relearn_ a thing
>> or two from 'dd' when it comes to speed.
>
> IMO no: this would add further bloat to the sources
> which in turn is harder to maintain. Ever looked at
> copy.c, e.g. at copy_internal()?
---
Yes. It's grown considerably in the past few years. But the main
thing it could relearn from 'dd' is using larger buffer sizes.
It's only half the speed of DD, so it's not like it's *that* slow.
(vs. rsync, being ~32x slower for a LAN mounted file to a local copy).
Where as cp from ~5 years ago used to be about 80% the speed of 'dd'.
So cp's speed has dropped by ~33%...
That's what I meant by "relearn" a thing to two from dd.
(likely in regards to buffer size, where on dd I'll specify 16M for
over LAN copies.
>> fast as 'dd', but now that often doesn't seem to be the case.
>
> hmm, well, could it be that this is because many features have
> been added over the years? ...
----
possibly -- but if it used larger buffer sizes, that problem would
mostly go away. Like I said -- it's not that much worse that 'dd'.
> I think rm is simply not the right tool for such kind
> of synchronization.
----
synchronization implies deleting files on target that don't exist
on source. That's not updating or "treating remote as file" for
purposes of update (-T)
You need to make the docs much more clear about "cp"s limitations.
update isn't eally update, and -T is certainly wrong at the very
least. If you feel you'd rather document cp's limitations,
that's fine...
cp is a great tool, don't get me wrong! But when it added update
and -T, --remove-destination, it started inferring or promising
more than you were willing to deliver. That should be documented.
To be honest, I missed the 'file' bit in --remove-destination and
focused on how it was different from -f in removing the destination
before the copy (manpage says to contrast w/-f)...
That still doesn't let it off the hook w/regards to "-T" where it
doesn't treat the destination as a file and remove it first.
Changed bug title to 'cp: "cp -af" breaks if one of files is a dir and other not' from 'Bug: cp <-a|-archive> (w/<-f|--remove-destination>) breaks if one of files is a dir and other not'
Request was from
Assaf Gordon <assafgordon <at> gmail.com>
to
control <at> debbugs.gnu.org.
(Fri, 11 Jan 2019 09:09:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#15727; Package
coreutils.
(Fri, 11 Jan 2019 09:18:01 GMT)
Full text and
rfc822 format available.
Message #31 received at 15727 <at> debbugs.gnu.org (full text, mbox):
severity 15727 wishlist
retitle 15727 doc: cp: expand dirs-vs-files with -f/--remove-dest
stop
Hello,
On 2013-10-29 12:20 p.m., Linda Walsh wrote:
[...]
> You need to make the docs much more clear about "cp"s limitations.
>
> update isn't eally update, and -T is certainly wrong at the very
> least. If you feel you'd rather document cp's limitations,
> that's fine...
>
> cp is a great tool, don't get me wrong! But when it added update
> and -T, --remove-destination, it started inferring or promising
> more than you were willing to deliver. That should be documented.
Based on the above (as the result of the long discussion),
I'm marking this as a documentation wish-list item:
clarify that "-f" and "--remove-destination" won't replace a file
with a directory (as explained by Pádraig and Bernhard in the thread).
Similarly for related limitations of "-T".
regards,
- assaf
Severity set to 'wishlist' from 'normal'
Request was from
Assaf Gordon <assafgordon <at> gmail.com>
to
control <at> debbugs.gnu.org.
(Fri, 11 Jan 2019 09:18:02 GMT)
Full text and
rfc822 format available.
Changed bug title to 'doc: cp: expand dirs-vs-files with -f/--remove-dest' from 'cp: "cp -af" breaks if one of files is a dir and other not'
Request was from
Assaf Gordon <assafgordon <at> gmail.com>
to
control <at> debbugs.gnu.org.
(Fri, 11 Jan 2019 09:18:02 GMT)
Full text and
rfc822 format available.
This bug report was last modified 6 years and 157 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.