GNU bug report logs -
#13347
stat/fstat inappropriately used in cp, install, rm, sort, chown for Windows 7 platforms
Previous Next
To reply to this bug, email your comments to 13347 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-coreutils <at> gnu.org:
bug#13347; Package
coreutils.
(Thu, 03 Jan 2013 18:06:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
"Burkhardt, Glenn UTAS" <Glenn.Burkhardt <at> utas.utc.com>:
New bug report received and forwarded. Copy sent to
bug-coreutils <at> gnu.org.
(Thu, 03 Jan 2013 18:06:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
MINGW32_NT-6.1 BOS0DT-1QLS1R1 1.0.17(0.48/3/2) 2011-04-24 23:39 i686 Msys
gcc version 4.7.0 (GCC)
Windows 7 Enterprise SP1
coreutils-5.97
I noticed this when trying to use the "cp" command in a Clearcase locally hosted dynamic view. When trying to copy a view private file, the 'cp' command would fail with a message like
skipping file `jj', as it was replaced while being copied
The problem is that the 'cp' command uses 'stat' on the source file initially, and then, when it's ready to copy, uses 'fstat' on it again, after the file has been opened for reading (about line 237 in copy.c from coreutils-5.97). This works on a file that's on a local hard drive, but not on a Clearcase locally hosted dynamic view. The 'install' command has the same problem.
This test really shouldn't be done on a Windows platform. Microsoft's documentation describes different values for stat.st_dev returned by the 'stat()' and 'fstat()' functions. There's also a note that the stat.st_ino value is meaningless for NTFS filesystems. Although MinGW releases a very old version of 'coreutils', the current 8.9 version will have the same problem. The definition of the macro SAME_INODE is unchanged.
But then it gets interesting. The 'cp' command is built with a different version of 'stat' and 'fstat' than one normally gets with a MinGW build environment. The coreutils need the Msys build environment, and the 'stat/fstat' functions are pulled from libmsys-1.0.dll.a. A build in regular MinGW environment with the current compiler gets the functions from libmoldname.a. So were the coreutils built with gcc-4.7.0, and no libmsys-1.0.dll.a, the copy command would fail for files on the local hard drive, since stat.st_dev is different for stat() and fstat().
I've attached a small test program to show the stat()/fstat() output, and a patch for system.h in coreutils that allows a working version of 'cp.exe' and 'install.exe' to be built.
Please also see the bug report to the MinGW bug tracker, ID 3590842:
http://sourceforge.net/tracker/?func=detail&aid=3590842&group_id=2435&atid=102435
[system.h.patch (application/octet-stream, attachment)]
[test-stat.c (text/plain, attachment)]
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#13347; Package
coreutils.
(Thu, 03 Jan 2013 18:39:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 13347 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
[adding gnulib]
On 01/03/2013 10:10 AM, Burkhardt, Glenn UTAS wrote:
> MINGW32_NT-6.1 BOS0DT-1QLS1R1 1.0.17(0.48/3/2) 2011-04-24 23:39 i686 Msys
> gcc version 4.7.0 (GCC)
> Windows 7 Enterprise SP1
> coreutils-5.97
>
> I noticed this when trying to use the "cp" command in a Clearcase locally hosted dynamic view.
Clearcase is notoriously buggy when it comes to POSIX compliance, and
I'm not surprised you are running into problems because of those bugs.
Since you paid for Clearcase (and coreutils is free), you might want to
consider reporting this issue upstream to the clearcase folks; then
again, since clearcase is closed source, I wouldn't hold out any hope
for them to fix their bugs.
> But then it gets interesting. The 'cp' command is built with a different version of 'stat' and 'fstat' than one normally gets with a MinGW build environment. The coreutils need the Msys build environment, and the 'stat/fstat' functions are pulled from libmsys-1.0.dll.a. A build in regular MinGW environment with the current compiler gets the functions from libmoldname.a. So were the coreutils built with gcc-4.7.0, and no libmsys-1.0.dll.a, the copy command would fail for files on the local hard drive, since stat.st_dev is different for stat() and fstat().
If that's the case, then perhaps the right thing to do is to modify
gnulib to replace fstat() on mingw so that it is always massaged to
provide the same st_dev as stat() (on mingw, gnulib is already doing
other things to track the original file name alongside every open fd, to
work around other bugs in POSIX compliance, so it wouldn't actually cost
that much extra).
>
> I've attached a small test program to show the stat()/fstat() output, and a patch for system.h in coreutils that allows a working version of 'cp.exe
Unfortunately, your patch cannot be accepted upstream as-is, as it would
violate POSIX semantics on systems that actually obey POSIX. But it
might be something that could be made conditional on whether things are
being targeted to mingw.
> http://sourceforge.net/tracker/?func=detail&aid=3590842&group_id=2435&atid=102435
>
> *** system.h.orig Mon Nov 26 16:54:58 2012
> --- system.h Wed Nov 28 18:15:04 2012
> ***************
> *** 535,542 ****
> #include "unlocked-io.h"
>
> #define SAME_INODE(Stat_buf_1, Stat_buf_2) \
> ! ((Stat_buf_1).st_ino == (Stat_buf_2).st_ino \
> ! && (Stat_buf_1).st_dev == (Stat_buf_2).st_dev)
>
> #define DOT_OR_DOTDOT(Basename) \
> (Basename[0] == '.' && (Basename[1] == '\0' \
> --- 535,541 ----
> #include "unlocked-io.h"
>
> #define SAME_INODE(Stat_buf_1, Stat_buf_2) \
> ! ((Stat_buf_1).st_ino == (Stat_buf_2).st_ino)
>
> #define DOT_OR_DOTDOT(Basename) \
> (Basename[0] == '.' && (Basename[1] == '\0' \
>
Also, while this patch is unlikely to go in as-is to upstream coreutils,
you may want to pursue getting it included into mingw and msys builds as
downstream distros affected by the need for the workaround.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[signature.asc (application/pgp-signature, attachment)]
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#13347; Package
coreutils.
(Sun, 06 Jan 2013 16:55:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 13347 <at> debbugs.gnu.org (full text, mbox):
> Date: Thu, 03 Jan 2013 11:38:47 -0700
> From: Eric Blake <eblake <at> redhat.com>
> Cc: Bug-gnulib <bug-gnulib <at> gnu.org>, 13347 <at> debbugs.gnu.org
>
> > But then it gets interesting. The 'cp' command is built with a different version of 'stat' and 'fstat' than one normally gets with a MinGW build environment. The coreutils need the Msys build environment, and the 'stat/fstat' functions are pulled from libmsys-1.0.dll.a. A build in regular MinGW environment with the current compiler gets the functions from libmoldname.a. So were the coreutils built with gcc-4.7.0, and no libmsys-1.0.dll.a, the copy command would fail for files on the local hard drive, since stat.st_dev is different for stat() and fstat().
Sorry for chiming in, but are you saying that the 'cp' binary
sometimes dynamically links to the MSYS runtime and sometimes to the
native Windows runtime? That is a sure way to disaster, as these two
runtimes are incompatible. (MSYS is a fork of an old version of
Cygwin.) A given binary should only ever use one of these, depending
on whether it was built as a MinGW program or an MSYS program.
If that's not the issue here, then why MSYS is relevant to the
discussion at all? What am I missing?
> If that's the case, then perhaps the right thing to do is to modify
> gnulib to replace fstat() on mingw so that it is always massaged to
> provide the same st_dev as stat() (on mingw, gnulib is already doing
> other things to track the original file name alongside every open fd, to
> work around other bugs in POSIX compliance, so it wouldn't actually cost
> that much extra).
Actually, TRT would be to have a replacement for 'stat' and 'fstat'
that return the Windows equivalent of the inode, which is accessible
by a file handle or a descriptor. The problem is that this also
requires to replace the definition of 'struct stat', since the
resulting inodes are 64-bit numbers, while a typical 'struct stat'
declares the inode as 'short' (why waste storage on something that is
always zero?)
Information forwarded
to
bug-coreutils <at> gnu.org:
bug#13347; Package
coreutils.
(Mon, 07 Jan 2013 14:10:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 13347 <at> debbugs.gnu.org (full text, mbox):
> -----Original Message-----
> From: Eli Zaretskii [mailto:eliz <at> gnu.org]
> Sent: Sunday, January 06, 2013 11:54 AM
> To: Eric Blake
> Cc: Burkhardt, Glenn UTAS; bug-gnulib <at> gnu.org; 13347 <at> debbugs.gnu.org
> Subject: Re: bug#13347: stat/fstat inappropriately used in cp, install, rm, sort,
> chown for Windows 7 platforms
>
> > Date: Thu, 03 Jan 2013 11:38:47 -0700
> > From: Eric Blake <eblake <at> redhat.com>
> > Cc: Bug-gnulib <bug-gnulib <at> gnu.org>, 13347 <at> debbugs.gnu.org
> >
> > > But then it gets interesting. The 'cp' command is built with a different
> version of 'stat' and 'fstat' than one normally gets with a MinGW build
> environment. The coreutils need the Msys build environment, and the
> 'stat/fstat' functions are pulled from libmsys-1.0.dll.a. A build in regular
> MinGW environment with the current compiler gets the functions from
> libmoldname.a. So were the coreutils built with gcc-4.7.0, and no libmsys-
> 1.0.dll.a, the copy command would fail for files on the local hard drive, since
> stat.st_dev is different for stat() and fstat().
>
> Sorry for chiming in, but are you saying that the 'cp' binary sometimes
> dynamically links to the MSYS runtime and sometimes to the native Windows
> runtime? That is a sure way to disaster, as these two runtimes are
> incompatible. (MSYS is a fork of an old version of
> Cygwin.) A given binary should only ever use one of these, depending on
> whether it was built as a MinGW program or an MSYS program.
>
> If that's not the issue here, then why MSYS is relevant to the discussion at
> all? What am I missing?
The MinGW folks have their own problems, of course. I made that note only because if one tries using the test code I supplied, the result from stat/fstat will be different than what MinGW's 'cp' command gets, since the 'cp' command uses a different stat/fstat implementation than one gets using creating a new program in a standard way.
I started exploring this problem by building 'cp.exe' using MinGW's build script. When I discovered the stat/fstat problem, I tried duplicating it with a simple test program. But the st_ino/st_dev values for my program were different that what 'cp.exe' got on the same files.
The comment is only relevant for testing in the MinGW environment.
This bug report was last modified 12 years and 165 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.