GNU bug report logs - #10471
Severe or critical - deletes existing files and leaves nothing. (cp)

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 10471 in the body.
You can then email your comments to 10471 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
Subject: Severe or critical - deletes existing files and leaves nothing. (cp)
Date: Mon, 09 Jan 2012 16:11:49 -0800

Please do not close this bug... EB doesn't know what he is talking
about -- he didn't read the bug.

-------- Original Message --------
Subject: BUG: Severe or critical - deletes existing files and leaves nothing. (cp)
Date: Mon, 09 Jan 2012 13:19:43 -0800
From: Linda Walsh , <cygwin <at> tlinx.org> <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
CC: cygwin <at> cygwin.com <cygwin <at> cygwin.com>



I was trying to copy a font dir from a unix to a windows machine.

Problem is there are duplicates -- where only the case differs...

On unix, I've run a 'dedup' in my font dir, that joined any files that were the
same so they are hard links to each other. (made sense at the time... ;-) ).

windows treats different cased file names as the same.

Along comes cp, on cygwin, .. but I don't think this is a cygwin bug, as
it's doing what it is supposed to do.. it is cp  that is problematic.

Note, in the below, I an alias for cp is in effect:

alias cp='cp --preserve=mode,timestamps'

i.e. by default, I try to preserve 'simple'-permissions, and timestamps.


First I tried:

/usr/share/fonts> cp -avu //bliss/usr_share/fonts/. .


Got:

removed `././Adobe (afm) fonts/Symbol.afm'
cp: cannot create hard link `././Adobe (afm) fonts/Symbol.afm' to `././Adobe
(afm) fonts/symbol.afm': No such file or directory
removed `././Adobe (afm) fonts/cmr10.afm'
cp: cannot create hard link `././Adobe (afm) fonts/cmr10.afm' to `././Adobe
(afm) fonts/CMR10.afm': No such file or directory
removed `././Adobe (afm) fonts/cmex10.afm'
(and many more... )

It removes an existing file 'Symbol', because because it sees that 'Symbol' is not
a hard link to 'symbol'; I specified "-a" to make an exact copy, so it's trying
to copy links.

But by removing 'Symbol', it removes 'symbol' and THEN can't link to it...
thus 'symbol' is gone from the local dir.
---
ok, at this point, I would consider it 'my bad, for using cp on windows (it used
to work without these issues, not sure what has changed)...


Then I tried the less strict:

> cp -rvu /usr/share/fonts> cp -rvu //bliss/usr_share/fonts/. .

figuring without the "-a" it wouldn't try to force linking, thus no prob... ...
well...
removed `././OTF/AJensonPro-Bold.otf'
cp: cannot create hard link `././OTF/AJensonPro-Bold.otf' to
`././OTF/ajensonpro-bold.otf': No such file or directory
removed `././OTF/AJensonPro-BoldCapt.otf'
cp: cannot create hard link `././OTF/AJensonPro-BoldCapt.otf' to
`././OTF/ajensonpro-boldcapt.otf': No such file or directory
removed `././OTF/AJensonPro-BoldDisp.otf'
------
?!!?!   Why is it trying to create hard links again?  I didn't specify preserve
links!?!
(i.e. this is a cp BUG...)


Well if I could just get it to copy them and not try to link them... so it
copies them twice..
at least they get here...

so I try the only thing that might not remove files "n":

 cp -nrvu //bliss/usr_share/fonts/. .

and...

removed `././Adobe (afm) fonts/cmbx5.afm'
cp: cannot create hard link `././Adobe (afm) fonts/cmbx5.afm' to `././Adobe
(afm) fonts/CMBX5.afm': No such file or directory
removed `././Adobe (afm) fonts/cmbx7.afm'
cp: cannot create hard link `././Adobe (afm) fonts/cmbx7.afm' to `././Adobe
(afm) fonts/CMBX7.afm': No such file or directory
removed `././Adobe (afm) fonts/cmmi10.afm'

... *sigh*...
my poor cumberland fonts... so sad!
um.... hey, if it shouldn't have removed them on -rvu, it doubly shouldn't have
done so on
when I told it "-n" (--no-clobber) don't clobber existing files...



DOUBLE BUG!....

 cygcheck -f /bin/cp
coreutils-8.14-1

Message #8 received at 10471 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 10471 <at> debbugs.gnu.org
Subject: Re: bug#10471: Severe or critical - deletes existing files and leaves
	nothing. (cp)
Date: Mon, 09 Jan 2012 17:19:58 -0700

[Message part 1 (text/plain, inline)]

forcemerge 10468 10471
thanks

On 01/09/2012 05:11 PM, Linda Walsh wrote:
> Please do not close this bug... EB doesn't know what he is talking
> about -- he didn't read the bug.

I did too read your report, but I stand by my position that upstream cp
is doing the right things for a case-sensitive system, and that the
problems you are encountering is not due to a bug in the algorithm used
in cp, but due to a bug in your system configurations in not enabling
case-sensitivity, so that your system violated the assumptions made by
cp.  I further stand by my position that upstream coreutils should not
have to change anything; and that if changes are needed in cygwin (and
they very well may be needed), that the cygwin list is a more
appropriate place to discuss them, since only cygwin is affected by the
burden of exposing the POSIX notion of a case-sensitive system when
built on top of a case-insensitive setting in windows.

We can reopen things if you can demonstrate the same problem without
cygwin in the mix.  But for now, as long as your recipe for reproducing
the problem involves cygwin, then the cygwin list is the better place to
discuss it.  And no need to create a new bug number - if you will reply
to the existing threads instead of starting a new one, you can add to
the conversation of a closed bug, including adding new evidence on why
the bug might need to be reopened.

-- 
Eric Blake   eblake <at> redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #19 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "L. A. Walsh" <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#10471: Severe or critical - deletes existing files and leaves
 nothing. (cp)
Date: Thu, 02 Apr 2015 00:35:13 -0700

reopen 10468
thanks
=============

 Not to stir up problems, but those who do not learn from history are
doomed to repeat it.  Nothing in this re-opening should be
taken personally, but am trying to follow-up on a bug that, I
felt, was improperly closed, and that now looks like it may be
'touched' or changed by a new fix to 'mv'.  The description in
the git-coreutils (not yet released) matches symptoms almost
exactly as I described over 3 years ago in 'cp'. 

 Since 'cp', 'ln' and 'mv' are closely related, and since 10468/10471
were still unfixed in coreutils 8.21-7.1.3 (from suse 13.1) as that
bug 10468 was still unfixed in 8.21-7.1.3 (from OpenSuse 13.1),
on linux w/XFS (as was described in this bug's original follow-up),
I was wondering if the changes to 'mv' might also have fixed this
(10468/10471) bug.  As the comment in the NEWS file made no mention of
behavior changes to 'cp' or either of these bug #'s being fixed,
it wasn't clear if the new work on 'mv' also fixed the similar (if not 
identical) problems reported against 'cp'.


 From the NEWS file in the *current* coreutils git:


  mv no longer supports moving a file to a hardlink, instead issuing
  an error.  The implementation was susceptible to races in the
  presence of multiple mv instances, which could result in both
  hardlinks being deleted.  Also on case insensitive file systems
  like HFS, mv would just remove a hardlinked 'file' if called like
  `mv file File`.  The feature was added in coreutils-5.0.1.


Please note, as reported below that this problem was reported in
behavior of "cp -au" over 3 years ago and was shut out as being
a cygwin-only bug. 

I told the maintainer (at that time) how to reproduce it on linux & XFS.
Last I checked, ~6 months ago, it was still present on linux w/XFS as I'd
described). 

I was told to "trust the coreutils maintainers"(1) in the closing of
what I thought was a serious bug" -- even though also said it could be
reproduced on linux-only w/XFS (as it was among the 1st with an ignore
case option).  Nevertheless, the bug was dropped in a crack as I said
would happen if the bug was 'closed out' as 'not-a-bug'.  I tried to
emphasize the increasing impact of this bug as more file systems
implemented case-preservation+insensitivity (ZFS and MacOS's FS, can also
be added to the group having such an option).
 
(1) -------- Original Message --------
  Subject: Re: bug#10471: closed (Severe or critical -
           deletes existing files and leaves nothing. (cp))
  Date: Mon, 09 Jan 2012 20:59:36 -0500
  From: Glenn

  I'm saying you should trust the coreutils maintainers to reach the
  right decision.

  Anyway, this mailing list is for technical issues with
  debbugs.gnu.org.  It's not the place to discuss how the various
  projects that use it choose to do so.


-------------------

Given the comment in the NEWS file  -- that the behavior change was in
'mv', I'm wondering if this was also fixes the same problem in 'cp' or if
that is still in the "closed-in-coreutils-as-a-cygwin-bug" category.
Below is the original bug that shows examples of the upper and lower case
hard-linked filenames causing mutually assured destruction (with 'cp' both
w+w/o preserve-links & update) -- which I thought (and still do think) was
worse than the 'mv' case because of the "-u" flag -- which I thought (at
that time) meant only update those files that were newer or didn't exist
on the target.  But it didn't:

I had 'Aaa' (v2) on source, and something like 'Aaa' (v1) <=> 'aaa' (v1)
on the destination (the ' (v1)'/' (v2)' not being part of the name, but to
indicate relative time stamps).  It appeared (or one explanation was),
that:

1) src(Aaa) removed the linked 'aaa' and 'Aaa' files at dst & copied (Aaa)
2) src then tried to link 'aaa'->'Aaa' on dst, but at link time, both
copies at the dst were deleted so we saw msg 'cannot create hard link: 
ENOENT.
Result) -- both copies of the older file deleted on the dst.  i

There may have been some variations on this, I'm not familiar with the
exact algorithm used at the time.  I noted, however that in the NEWS file
under bug fixes, in the new coreutils that had recently made it to cygwin
(release 8.13 (2011-09-08) (...cont...)

  cp -u -p would fail to preserve one hard link for each up-to-date
  copy of a src-hard-linked name in the destination tree.  I.e., if
  s/a and s/b are hard-linked and dst/s/a is up to date, "cp -up
  s dst" would copy s/b to dst/s/b rather than simply linking
  dst/s/b to dst/s/a.

...and I suggested that this new bug might be related to that.

Regardless. It was closed out and fell into the cracks.  Now a related
bug in 'mv' (which shares much code with cp, I believe), is reported as
being 'fixed'...   So I'm wondering if the bug I reported was fixed as
a result of this, or if it is still there, or what?

Original bug response and a response about categorizing this bug as
a windows-only bug, being 'impractical' to look at in coreutils and
having near zero chance of being fixed via coreutils, attached
below.  I included it for reference, since I was overruled, at the time
by those who thought the bug was *only in cygwin* (even though I
did claim in followup discussion that it could also be reproduced
on linux w/XFS using its '-n version=ci' mkfs option.  It can probably
also be reproduced, now, on ZFS and some MacOS FS's that have similar
case-treatment options.

Cheers -- and still wondering if the 8.13 change was related to this
bug and if the new 'mv' behavior change affects this 'cp' bug?

-- Linda Walsh



-------- Original Message --------
Subject: Re: bug#10468: BUG: Severe or critical -
--------     deletes existing files and leaves nothing. (cp)

Date: Mon, 09 Jan 2012 14:41:22 -0700
From: Eric
Organization: Red Hat
To: Linda
CC: 10468-debbugs-gnu-org, "cygwin"
References: <4F0B59EF.7080204 <at> tlinx.org>

tag 10468 notabug
thanks

On 01/09/2012 02:19 PM, Linda,  wrote:
>
>
>  I was trying to copy a font dir from a unix to a windows machine.
>
>  Problem is there are duplicates -- where only the case differs...

The problem is not in coreutils, but in your operating system's
limitations, and in your configuration.  Windows (and thus Cygwin) can
be put in a mode where it preserves case (and I highly recommend doing
so if you want your example to work):
http://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-casesensitive

Without that setting, there is nothing that coreutils can do on your
behalf: the default case-insensitive behavior of Windows violates POSIX,
and forcing coreutils to add bloat to work around this violation is
rather difficult and unpalatable compared to all other operating systems
that already follow POSIX.  So I am closing the upstream coreutils bug
report.

There may be a way to make the cygwin port of coreutils smarter about
case-sensitive issues when the Windows setting is case-insensitive, but
it would be specific to the cygwin port.  Meanwhile, I'm not convinced
it is worth slowing down the common case by checking for case clash on
all operations, since most users that either avoid case clash or have
the registry setting on, just to help out the rare case of a user that
has case clash but the registry setting off.  You are more than welcome
to discuss this further on the cygwin list, and preferably provide
patches if you want behavior changed, but again, that would most likely
be cygwin-specific and does not need to involve the upstream coreutils
list (the cygwin port of coreutils already carries several other patches
for dealing with non-POSIX issues that don't need to be ported back
upstream, such as how to handle .exe suffixes).

> > cp -rvu /usr/share/fonts> cp -rvu //bliss/usr_share/fonts/. .
>
>  figuring without the "-a" it wouldn't try to force linking, thus no
>  prob... ... well...
>  removed `././OTF/AJensonPro-Bold.otf'
>  cp: cannot create hard link `././OTF/AJensonPro-Bold.otf' to
>  `././OTF/ajensonpro-bold.otf': No such file or directory
>  removed `././OTF/AJensonPro-BoldCapt.otf'
>  cp: cannot create hard link `././OTF/AJensonPro-BoldCapt.otf' to
>  `././OTF/ajensonpro-boldcapt.otf': No such file or directory
>  removed `././OTF/AJensonPro-BoldDisp.otf'
>  ------
>  ?!!?!   Why is it trying to create hard links again?  I didn't specify
>  preserve links!?!
>  (i.e. this is a cp BUG...)

If I understand correctly, your problem stems from the fact that your
source location has multiple hard links to the same file but with
different case, while your destination can't support two hard links to
the same file differing only in case, so there is no sane way to say
which of the two spellings should be copied.  It's not cp's fault that
you are copying from a permissive system to a restrictive one; and while
the errors may not be the most friendly, at least they are accurate.


-------- Original Message --------
Subject: Re: bug#10468: BUG: Severe or critical -
--------     deletes existing files and leaves nothing. (cp)
Date: Mon, 09 Jan 2012 17:15:26 -0800
From: Paul
Organization: UCLA Computer Science Department
To: debbugs-gnu-org, tlinx-org

On 01/09/12 17:00, Linda Walsh wrote:
>  cp is a gnu util, not part of linux, -- you should
>  demonstrate that it is not a bug in cygwin

This is not a practical division of labor.  As a general rule,
we don't have the resources to deal with Windows ports.

I suggest raising this issue with the people who are actually
doing the Windows port, whoever they are.  If it's a
problem with the port, they can fix it themselves without putting
further load on us; if not, they'll have a better chance to diagnose
and fix the problem than we will (our chance is essentially zero).

Message #22 received at 10471 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: "L. A. Walsh" <coreutils <at> tlinx.org>, 10471 <at> debbugs.gnu.org
Subject: Re: bug#10471: Severe or critical - deletes existing files and leaves
 nothing. (cp)
Date: Thu, 02 Apr 2015 21:07:44 -0700

L. A. Walsh wrote:
> I was wondering if the changes to 'mv' might also have fixed this
> (10468/10471) bug.

Haven't a clue.  Why don't you try the bleeding-edge version of coreutils and 
see?  Presumably you have the oddball file system in question available, and can 
easily try it out.  (I don't have it, and I don't expect other maintainers to 
have it either.)

Message #25 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "L. A. Walsh" <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: bug-coreutils <at> gnu.org, 10471 <at> debbugs.gnu.org
Subject: Re: bug#10471: Severe or critical - deletes existing files and leaves
 nothing. (cp)
Date: Thu, 09 Apr 2015 15:17:24 -0700

Paul Eggert wrote:
> L. A. Walsh wrote:
>> I was wondering if the changes to 'mv' might also have fixed this
>> (10468/10471) bug.
> 
> Haven't a clue.  Why don't you try the bleeding-edge version of 
> coreutils and see?  Presumably you have the oddball file system in 
> question available, and can easily try it out.  (I don't have it, and I 
> don't expect other maintainers to have it either.)
----

How was the bug with 'mv' fixed? -- it mentions duplicating it on 
on HFS (MacOS) that also has a case insensitive option.  
Specifically, a "mv file File" (where both were hard linked) the
mv operation would result in 'file' being removed.

As far as developers not having access to HFS(I seem to remember reading
that case insensitivity was the default now on MacOS), Solaris or BSD(ZFS-- don't
know its default status), Windows(NTFS - is default), Linux(xfs -- not
a default config) -- what OS's do developers work with?

Seems like it is reproducible on MacOS, linux, Solaris and Windows --
but was closed out because the claim was such file systems were "oddball"
but HFS, NTFS are default FS's, and ZFS and XFS don't seem that
rare on Solaris, BSD or linux... 

In my case, instead of 'mv', it was a 'cp' w/ -a or --preserve-hardlinks
that triggered the problem, _including_ in the presence of "-u" (which
I thought should have been non-destructive (i.e. not deleting linked
files if only 1 of the names on the source was 'newer' (and not hardlinked)).

Seems like "race" conditions w/hardlinked files (especially on
case-ignoring filesystems) and/or w/update look like they haven't
really been strongly thought out.

My issue here is that I reported the problem 3 years ago and it
got ignored because some thought it was windows+cygwin+NTFS only,
(even though at the time I pointed out it could be dupped on linux
w/xfs).  But the bug still got ignored -- where as now it is
being seen on HFS (and can likely be seen on ZFS from what (I've
been told).  

Given the number of file systems and OS's the behavior can be
reproduced on -- are you still claiming all of those OS's+FS's
are the "oddball case" -- they seem to be growing (i.e. case
ins.  was added to HFS and ZFS) since I first found the problem
on NTFS and claimed it also applied to XFS.

I will easily accept that resources are such that it is a 
low priority (as I said 3 years ago), but it is a bit galling
to have removal of needed features from 'rm' taking precedence
over data-loss cases.

Message #31 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: "L. A. Walsh" <coreutils <at> tlinx.org>
Cc: bug-coreutils <at> gnu.org, 10471 <at> debbugs.gnu.org
Subject: Re: bug#10471: Severe or critical - deletes existing files and leaves
 nothing. (cp)
Date: Thu, 09 Apr 2015 22:56:05 -0700

L. A. Walsh wrote:
> How was the bug with 'mv' fixed?

You should be able to find that out by looking at the Git history.

I generally work with ext4 and zfs myself, and I don't observe the bug with any 
of the file systems I use.  I don't know about the other coreutils developers, 
but I expect that case-insensitive file systems are just as much oddballs for 
them as they are for me.  If the bug bothers you, and if it still exists in the 
coreutils master version (something that's not at all clear), I urge you to send 
us a fix for it, as that's most likely going to be the fastest and most reliable 
way to get the bug fixed.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.