GNU bug report logs - #6657
mkstemp on cygwin creates binary files

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 6657 in the body.
You can then email your comments to 6657 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paolo Bonzini <bonzini <at> gnu.org>
To: bug-coreutils <at> gnu.org
Subject: mkstemp on cygwin creates binary files
Date: Fri, 16 Jul 2010 22:27:18 +0200

I recently got by private email a report that "sed -i" changed the line 
endings of the file to bare linefeeds on cygwin.  The reason for this is 
that mkstemp on cygwin hardcodes the flags to O_EXCL|O_BINARY:

http://www.cygwin.com/ml/cygwin-patches/2006-q2/msg00013.html

I fixed it by using instead mkostemp(template, 0).  From a quick "git 
grep", it seems like sort and tac are affected by the bug in coreutils.

Thanks!

Paolo

Message #8 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> CS.UCLA.EDU>
To: Paolo Bonzini <bonzini <at> gnu.org>
Cc: 6657 <at> debbugs.gnu.org
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Fri, 16 Jul 2010 14:47:15 -0700

On 07/16/10 13:27, Paolo Bonzini wrote:

> I fixed it by using instead mkostemp(template, 0).  From a quick "git
> grep", it seems like sort and tac are affected by the bug in coreutils.

tac access the temp file in binary mode, so there's no problem there.

I don't see the problem with 'sort' offhand.  Why would the user care whether
line endings in sort's temp files are \r\n or \n?  Using binary mode is a
bit faster and more reliable, surely.

Or is there some problem if the file descriptor is created with O_BINARY
and then fdopen is called with "w" (and not "wb")?  I guess "sort" does
that on Cygwin now.

Message #11 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Paolo Bonzini <bonzini <at> gnu.org>
To: Paul Eggert <eggert <at> CS.UCLA.EDU>
Cc: 6657 <at> debbugs.gnu.org
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Sat, 17 Jul 2010 08:13:59 +0200

On 07/16/2010 11:47 PM, Paul Eggert wrote:
> On 07/16/10 13:27, Paolo Bonzini wrote:
>
>> I fixed it by using instead mkostemp(template, 0).  From a quick "git
>> grep", it seems like sort and tac are affected by the bug in coreutils.
>
> tac access the temp file in binary mode, so there's no problem there.
>
> I don't see the problem with 'sort' offhand.  Why would the user care whether
> line endings in sort's temp files are \r\n or \n?  Using binary mode is a
> bit faster and more reliable, surely.
>
> Or is there some problem if the file descriptor is created with O_BINARY
> and then fdopen is called with "w" (and not "wb")?  I guess "sort" does
> that on Cygwin now.

No idea.  I just thought a heads-up was in order...

Paolo

Message #14 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Paolo Bonzini <bonzini <at> gnu.org>
Cc: 6657 <at> debbugs.gnu.org, Paul Eggert <eggert <at> CS.UCLA.EDU>
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Sat, 17 Jul 2010 17:54:25 -0600

[Message part 1 (text/plain, inline)]

On 07/17/2010 12:13 AM, Paolo Bonzini wrote:
>> I don't see the problem with 'sort' offhand.  Why would the user care
>> whether
>> line endings in sort's temp files are \r\n or \n?  Using binary mode is a
>> bit faster and more reliable, surely.

Correct.  I see no reason to change sort.

>>
>> Or is there some problem if the file descriptor is created with O_BINARY
>> and then fdopen is called with "w" (and not "wb")?  I guess "sort" does
>> that on Cygwin now.
> 
> No idea.  I just thought a heads-up was in order...

fdopen(,"w") on cygwin honors the existing O_TEXT or O_BINARY flag of
the underlying fd; while only fdopen(,"wt") and fdopen(,"wb") force the
flag (and changes the flag of the underlying fd, if needed).

The problem with sed was that the temp file was then being rename()d
into the original file name, regardless of whether the final file should
be binary or text.  sort does not have the problem, because the
temporary file was truly temporary - it is not renamed into a
user-visible file after it is used.

By the way, newer cygwin provides mkostemp() - did you only fix the
problem for older cygwin that lacks mkostemp and thus gets the gnulib
fallback that doesn't force binary?  And what's wrong with using
setmode() (from <io.h>, or from gnulib's "binary-io.h") on the fd
created by mkstemp() to ensure the desired mode, rather than having to
use mkostemp?

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #17 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Paolo Bonzini <bonzini <at> gnu.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: 6657 <at> debbugs.gnu.org, Paul Eggert <eggert <at> CS.UCLA.EDU>
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Sun, 18 Jul 2010 09:19:34 +0200

On 07/18/2010 01:54 AM, Eric Blake wrote:
> By the way, newer cygwin provides mkostemp() - did you only fix the
> problem for older cygwin that lacks mkostemp and thus gets the gnulib
> fallback that doesn't force binary?

mkostemp also forces binary?  That's a bug IMO, since the caller can 
tell Cygwin if it wants binary or text.

> And what's wrong with using
> setmode() (from<io.h>, or from gnulib's "binary-io.h") on the fd
> created by mkstemp() to ensure the desired mode, rather than having to
> use mkostemp?

I want the default type given by the mount point (using the mount type 
of /tmp is fine), can I get that?

Paolo

Message #20 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Paolo Bonzini <bonzini <at> gnu.org>
Cc: 6657 <at> debbugs.gnu.org, Paul Eggert <eggert <at> CS.UCLA.EDU>
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Mon, 19 Jul 2010 07:28:32 -0600

[Message part 1 (text/plain, inline)]

On 07/18/2010 01:19 AM, Paolo Bonzini wrote:
> On 07/18/2010 01:54 AM, Eric Blake wrote:
>> By the way, newer cygwin provides mkostemp() - did you only fix the
>> problem for older cygwin that lacks mkostemp and thus gets the gnulib
>> fallback that doesn't force binary?
> 
> mkostemp also forces binary?  That's a bug IMO, since the caller can
> tell Cygwin if it wants binary or text.

In looking at it more, cygwin 1.7.5 does not provide mkostemp (although
it's planned for the future).  So I will make sure that when cygwin adds
it, that it will allow the user to request O_TEXT or O_BINARY (forced
mode), or 0 (underlying mode of mount point where file is being
created); in addition it supports the O_CLOEXEC/0 option that justifies
mkostemp on glibc.

> 
>> And what's wrong with using
>> setmode() (from<io.h>, or from gnulib's "binary-io.h") on the fd
>> created by mkstemp() to ensure the desired mode, rather than having to
>> use mkostemp?
> 
> I want the default type given by the mount point (using the mount type
> of /tmp is fine), can I get that?

Well, the mount type of /tmp is fine only if you will be rename()ing the
final file into /tmp - really, you want the default type given by the
mount point of where the final file will live.  But you are correct that
mkostemp(,0) is the right way to get that behavior, and since mkostemp
can create a file in any directory (not just /tmp), and since you can
only guarantee an atomic rename() if you don't cross device boundaries
(that is, you should already be creating the temporary file in the same
directory as the final destination), you've convinced me that your
approach of using mkostemp in sed is correct for cygwin.

By the way, I don't see your patch for using mkostemp on cygwin in
git://git.sv.gnu.org/sed.git; am I missing something, or is that not the
latest git repository for sed?

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #23 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Paolo Bonzini <bonzini <at> gnu.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: 6657 <at> debbugs.gnu.org, Paul Eggert <eggert <at> CS.UCLA.EDU>
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Mon, 19 Jul 2010 18:31:02 +0200

On 07/19/2010 03:28 PM, Eric Blake wrote:
> By the way, I don't see your patch for using mkostemp on cygwin in
> git://git.sv.gnu.org/sed.git; am I missing something, or is that not the
> latest git repository for sed?

I wanted to make sure you liked it before pushing. :)  I'll now push it.

I also experimented with forcing "wt" or "wb" depending on the presence 
or absence of --binary, but I convinced myself that at least using "wt" 
is wrong.

For now, I won't change the behavior of building the file in /tmp. 
While this may make rename not atomic, it can be "corrected" anyway by 
setting TMPDIR=. in the environment.  By comparison, "perl -i" is 
implemented using unlink+open which is not atomic so it's unlikely that 
anyone is relying on atomicity (and also wastes quota like TMPDIR=. 
would do).

Paolo

Message #26 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Paolo Bonzini <bonzini <at> gnu.org>
Cc: 6657 <at> debbugs.gnu.org, Paul Eggert <eggert <at> CS.UCLA.EDU>
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Mon, 19 Jul 2010 10:33:32 -0600

[Message part 1 (text/plain, inline)]

On 07/19/2010 10:31 AM, Paolo Bonzini wrote:
> On 07/19/2010 03:28 PM, Eric Blake wrote:
>> By the way, I don't see your patch for using mkostemp on cygwin in
>> git://git.sv.gnu.org/sed.git; am I missing something, or is that not the
>> latest git repository for sed?
> 
> I wanted to make sure you liked it before pushing. :)  I'll now push it.
> 
> I also experimented with forcing "wt" or "wb" depending on the presence
> or absence of --binary, but I convinced myself that at least using "wt"
> is wrong.
> 
> For now, I won't change the behavior of building the file in /tmp. While
> this may make rename not atomic, it can be "corrected" anyway by setting
> TMPDIR=. in the environment.

Yuck - that means if /tmp is mounted differently than ., then using
mkostemp(,0) will force the wrong line endings (converting binary to
text, or converting text to binary, depending on which direction the
mismatch is between the mount modes).  If you aren't creating the temp
file in the same mount point as the target, then you cannot blindly rely
on automatic mount point file modes to do the right thing.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #29 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Paolo Bonzini <bonzini <at> gnu.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: 6657 <at> debbugs.gnu.org, Paul Eggert <eggert <at> CS.UCLA.EDU>
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Mon, 19 Jul 2010 18:48:24 +0200

On 07/19/2010 06:33 PM, Eric Blake wrote:
> Yuck - that means if /tmp is mounted differently than ., then using
> mkostemp(,0) will force the wrong line endings (converting binary to
> text, or converting text to binary, depending on which direction the
> mismatch is between the mount modes).  If you aren't creating the temp
> file in the same mount point as the target, then you cannot blindly rely
> on automatic mount point file modes to do the right thing.

It's buggy anyway in all released versions of sed.  I'll probably make 
the change---just, not yet.

Paolo

Message #32 received at 6657 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Paolo Bonzini <bonzini <at> gnu.org>
Cc: 6657 <at> debbugs.gnu.org, Paul Eggert <eggert <at> CS.UCLA.EDU>
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Mon, 19 Jul 2010 10:51:56 -0600

[Message part 1 (text/plain, inline)]

On 07/19/2010 10:48 AM, Paolo Bonzini wrote:
> On 07/19/2010 06:33 PM, Eric Blake wrote:
>> Yuck - that means if /tmp is mounted differently than ., then using
>> mkostemp(,0) will force the wrong line endings (converting binary to
>> text, or converting text to binary, depending on which direction the
>> mismatch is between the mount modes).  If you aren't creating the temp
>> file in the same mount point as the target, then you cannot blindly rely
>> on automatic mount point file modes to do the right thing.
> 
> It's buggy anyway in all released versions of sed.  I'll probably make
> the change---just, not yet.

As a compromise, it is faster to at least document that the choice of
TMPDIR=. can help fix line ending issues with 'sed -i' on cygwin, until
such time as you actually do make some change on either the default
in-place location or more work on explicit mode matching when creating
the temp file.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #37 received at 6657-done <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Paolo Bonzini <bonzini <at> gnu.org>
Cc: 6657-done <at> debbugs.gnu.org
Subject: Re: bug#6657: mkstemp on cygwin creates binary files
Date: Tue, 13 Sep 2011 14:04:31 +0200

tags 6657 notabug
thanks

Paolo Bonzini wrote:
> I recently got by private email a report that "sed -i" changed the
> line endings of the file to bare linefeeds on cygwin.  The reason for
> this is that mkstemp on cygwin hardcodes the flags to O_EXCL|O_BINARY:
>
> http://www.cygwin.com/ml/cygwin-patches/2006-q2/msg00013.html
>
> I fixed it by using instead mkostemp(template, 0).  From a quick "git
> grep", it seems like sort and tac are affected by the bug in
> coreutils.

Thanks for the heads up.
This turned out not to be a problem with coreutils,
so I'm marking this as "done".

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.