Subject: Re: [gnu-prog-discuss] Writing unwritable files

From: Pádraig Brady <[email-masked]>
Date: Thu, 06 Sep 2012 13:30:08 +0100

On 09/06/2012 01:12 PM, Paolo Bonzini wrote:
> [For bug-coreutils: the context here is that sed -i, just like perl -i,
> breaks hard links and thus destroys the content of files with 0400
> permission].
>
> Il 06/09/2012 12:38, John Darrington ha scritto:
>>       That's expected of programs that break hard links.
>>
>> I wonder how many users who are not hackers expect it?  I suspect most
>> would not.
>>
>> Why is it not possible or not desirable to check the mode of the file
>> before renaming?
>
> Because there are valid use cases in which you _want_ to break hard links.
>
> Also because it would be twice as slow and require twice as much disk
> space.  The choices would be:
>
> 1) copy the input file to a temporary file, then write to the original
> file while processing the copy; required space = input_size +
> max(input_size, output_size)
>
> 2) write to a temporary file while processing the original, then copy it
> over to the input; required space = output_size + max(input_size,
> output_size).
>
> 3) same as (1) or (2) with memory replacing a temporary file.  Goes
> totally against the idea of sed as a _stream_ editor.
>
>>       Here is what the sed manual thinks.
>>
>> Thanks for pointing that out (it must be a recent addition; my installed
>> version doesn't have this text).
>
> Actually it is older than the version control repository, so 2004 or
> older.  But it is under "Reporting bugs", not under the -i option
> (patches welcome ;)).
>
> You'll also find it under /usr/share/doc/sed-4.2.1/BUGS.
>
>>       `-i' clobbers read-only files
>>            In short, `sed -i' will let you delete the contents of a read-only
>>            file, and in general the `-i' option (*note Invocation: Invoking
>>            sed.) lets you clobber protected files.  This is not a bug, but
>>            rather a consequence of how the Unix filesystem works.
>>
>>            The permissions on a file say what can happen to the data in that
>>            file, while the permissions on a directory say what can happen to
>>            the list of files in that directory.  `sed -i' will not ever open
>>            for writing  a file that is already on disk.  Rather, it will work
>>            on a temporary file that is finally renamed to the original name:
>>            if you rename or delete files, you're actually modifying the
>>            contents of the directory, so the operation depends on the
>>            permissions of the directory, not of the file.  For this same
>>            reason, `sed' does not let you use `-i' on a writeable file in a
>>            read-only directory, and will break hard or symbolic links when
>>            `-i' is used on such a file.
>>
>> I don't think that this addresses the issue at hand.  The program does something
>> non-intuitive that can lead to loss of data, and it wouldn't be reasonable to blame
>> the user in that instance.
>
> I agree, but it's not me who designed the Unix filesystem permissions.
>
>> (Some) other GNU programs don't behave like this.  For
>> example "truncate foo" on a readonly foo will exit with an error
>
> Truncate does not process foo for input at the same time, so it isn't't
> really relevant.
>
>> , as will  "dd if=foo of=foo count=10".
>
> dd has a well-defined behavior for overlapping input and output, and
> this well-defined behavior in fact mandates that dd doesn't break hard
> links.
>
>> Likewise, "shuf foo -o foo".
>
> I consider "shuf foo -o foo" (on a read-write file) to be insecure.
> Besides, it works by chance just because it reads everything in memory
> first.  If it used mmap to process the input file, "shuf foo -o foo"
> would be broken, and the only way to fix it would be to do the same as
> "sed -i".
>
> shuf could in fact introduce a "shuf -i" mode that would be consistent
> with the way "sed -i" works, including the ability to create a backup
> file _and_ the breaking of hard links.

Well `sort` and `shuf` need to read all their input before
generating output. This is traditional behavior and
POSIX also states that -o can refer to one of the input files.

Also related to this is having a seperate "replace" wrapper,
that would handle all the atomic, backup, permission, ... issues.
http://www.pixelbeat.org/docs/unix_file_replacement.html
I takes advantage of the existing coreutils to handle the above.
I really need to polish that off and submit it
(translations in the shell script was one thing that was bothering me).

I notice David Wheeler proposed much the same thing
with the "rewrite" util (I like that name too):
http://permalink.gmane.org/gmane.comp.standards.posix.austin.general/5348

cheers,
Pádraig