From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 08:13:38 2012 Received: (at submit) by debbugs.gnu.org; 6 Sep 2012 12:13:38 +0000 Received: from localhost ([127.0.0.1]:42496 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9axl-0001sW-BD for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:13:37 -0400 Received: from eggs.gnu.org ([208.118.235.92]:48634) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9axi-0001sN-Ok for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:13:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1T9axV-0008Ug-JT for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:13:22 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:40327) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9axV-0008Ub-F1 for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:13:21 -0400 Received: from eggs.gnu.org ([208.118.235.92]:48214) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9axJ-0004bK-J5 for bug-coreutils@gnu.org; Thu, 06 Sep 2012 08:13:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1T9ax9-0008Ic-7A for bug-coreutils@gnu.org; Thu, 06 Sep 2012 08:13:09 -0400 Received: from mail-pz0-f41.google.com ([209.85.210.41]:36530) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9ax0-0008HC-Ew; Thu, 06 Sep 2012 08:12:50 -0400 Received: by dadi14 with SMTP id i14so1173181dad.0 for ; Thu, 06 Sep 2012 05:12:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=sGvdfvXIgXZnk/yXOyLeaiuXwzAYUqZB9QlfAxI1FPk=; b=t58NdlUP1g0fQBdl3CahTgLxbIrNy7Gjk1tr2o6d3IuBxh/XnpgpTz8L+SjkLEhiX+ 8jiIfTjM7nlrMOFKR+bXK2T/t/Vosp1hrjqVY2wnUbDgEIgIZxn27JN0WnetR5+zQs/+ oyV4f3JyMsBSdzP/lLO8CGQ3eYw9XAWxsy7B4vLJcN8esFilh8fpQBQpL+tzxXPPIeBN Ay5TySw54SiYbnex7dbhc9VFgINoVmYsFHe3m4+HFazro6KBOdko9Jic/pIZK1g6+J6U RsekzXIprjf2W/E0sVqULseUpclECCp3n3wtEqDT8LdMuq9+nrZ8tyv6y2wbR49UOjtF Z5RQ== Received: by 10.68.211.105 with SMTP id nb9mr4545083pbc.67.1346933568728; Thu, 06 Sep 2012 05:12:48 -0700 (PDT) Received: from yakj.usersys.redhat.com (93-34-169-1.ip50.fastwebnet.it. [93.34.169.1]) by mx.google.com with ESMTPS id gt2sm1239902pbc.62.2012.09.06.05.12.45 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 06 Sep 2012 05:12:47 -0700 (PDT) Message-ID: <5048933A.7020105@gnu.org> Date: Thu, 06 Sep 2012 14:12:42 +0200 From: Paolo Bonzini User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120717 Thunderbird/14.0 MIME-Version: 1.0 To: John Darrington Subject: Re: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> In-Reply-To: <20120906103841.GA13245@cellform.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 208.118.235.17 X-Spam-Score: -6.1 (------) X-Debbugs-Envelope-To: submit Cc: Bug-coreutils , gnu-prog-discuss@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.1 (------) [For bug-coreutils: the context here is that sed -i, just like perl -i, breaks hard links and thus destroys the content of files with 0400 permission]. Il 06/09/2012 12:38, John Darrington ha scritto: > That's expected of programs that break hard links. > > I wonder how many users who are not hackers expect it? I suspect most > would not. > > Why is it not possible or not desirable to check the mode of the file > before renaming? Because there are valid use cases in which you _want_ to break hard links. Also because it would be twice as slow and require twice as much disk space. The choices would be: 1) copy the input file to a temporary file, then write to the original file while processing the copy; required space = input_size + max(input_size, output_size) 2) write to a temporary file while processing the original, then copy it over to the input; required space = output_size + max(input_size, output_size). 3) same as (1) or (2) with memory replacing a temporary file. Goes totally against the idea of sed as a _stream_ editor. > Here is what the sed manual thinks. > > Thanks for pointing that out (it must be a recent addition; my installed > version doesn't have this text). Actually it is older than the version control repository, so 2004 or older. But it is under "Reporting bugs", not under the -i option (patches welcome ;)). You'll also find it under /usr/share/doc/sed-4.2.1/BUGS. > `-i' clobbers read-only files > In short, `sed -i' will let you delete the contents of a read-only > file, and in general the `-i' option (*note Invocation: Invoking > sed.) lets you clobber protected files. This is not a bug, but > rather a consequence of how the Unix filesystem works. > > The permissions on a file say what can happen to the data in that > file, while the permissions on a directory say what can happen to > the list of files in that directory. `sed -i' will not ever open > for writing a file that is already on disk. Rather, it will work > on a temporary file that is finally renamed to the original name: > if you rename or delete files, you're actually modifying the > contents of the directory, so the operation depends on the > permissions of the directory, not of the file. For this same > reason, `sed' does not let you use `-i' on a writeable file in a > read-only directory, and will break hard or symbolic links when > `-i' is used on such a file. > > I don't think that this addresses the issue at hand. The program does something > non-intuitive that can lead to loss of data, and it wouldn't be reasonable to blame > the user in that instance. I agree, but it's not me who designed the Unix filesystem permissions. > (Some) other GNU programs don't behave like this. For > example "truncate foo" on a readonly foo will exit with an error Truncate does not process foo for input at the same time, so it isn't't really relevant. > , as will "dd if=foo of=foo count=10". dd has a well-defined behavior for overlapping input and output, and this well-defined behavior in fact mandates that dd doesn't break hard links. > Likewise, "shuf foo -o foo". I consider "shuf foo -o foo" (on a read-write file) to be insecure. Besides, it works by chance just because it reads everything in memory first. If it used mmap to process the input file, "shuf foo -o foo" would be broken, and the only way to fix it would be to do the same as "sed -i". shuf could in fact introduce a "shuf -i" mode that would be consistent with the way "sed -i" works, including the ability to create a backup file _and_ the breaking of hard links. Paolo From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 08:31:07 2012 Received: (at submit) by debbugs.gnu.org; 6 Sep 2012 12:31:07 +0000 Received: from localhost ([127.0.0.1]:42520 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9bEg-0002Hj-La for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:31:07 -0400 Received: from eggs.gnu.org ([208.118.235.92]:58255) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9bEc-0002Hb-Po for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:31:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1T9bEJ-0006UC-Oc for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:30:50 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:43860) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9bEJ-0006Tl-Gs for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:30:43 -0400 Received: from eggs.gnu.org ([208.118.235.92]:48301) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9bEE-0008TV-Op for bug-coreutils@gnu.org; Thu, 06 Sep 2012 08:30:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1T9bEA-0006MU-7m for bug-coreutils@gnu.org; Thu, 06 Sep 2012 08:30:38 -0400 Received: from mx1.redhat.com ([209.132.183.28]:20828) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9bE7-0006K3-DR; Thu, 06 Sep 2012 08:30:31 -0400 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q86CUDv7009482 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 6 Sep 2012 08:30:13 -0400 Received: from [10.36.116.62] (ovpn-116-62.ams2.redhat.com [10.36.116.62]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q86CU9Zw007478 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 6 Sep 2012 08:30:11 -0400 Message-ID: <50489750.6080502@draigBrady.com> Date: Thu, 06 Sep 2012 13:30:08 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 To: Paolo Bonzini Subject: Re: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> In-Reply-To: <5048933A.7020105@gnu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id q86CUDv7009482 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 208.118.235.17 X-Spam-Score: -6.9 (------) X-Debbugs-Envelope-To: submit Cc: gnu-prog-discuss@gnu.org, "David A. Wheeler" , Bug-coreutils , John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.9 (------) On 09/06/2012 01:12 PM, Paolo Bonzini wrote: > [For bug-coreutils: the context here is that sed -i, just like perl -i, > breaks hard links and thus destroys the content of files with 0400 > permission]. > > Il 06/09/2012 12:38, John Darrington ha scritto: >> That's expected of programs that break hard links. >> >> I wonder how many users who are not hackers expect it? I suspect most >> would not. >> >> Why is it not possible or not desirable to check the mode of the file >> before renaming? > > Because there are valid use cases in which you _want_ to break hard lin= ks. > > Also because it would be twice as slow and require twice as much disk > space. The choices would be: > > 1) copy the input file to a temporary file, then write to the original > file while processing the copy; required space =3D input_size + > max(input_size, output_size) > > 2) write to a temporary file while processing the original, then copy i= t > over to the input; required space =3D output_size + max(input_size, > output_size). > > 3) same as (1) or (2) with memory replacing a temporary file. Goes > totally against the idea of sed as a _stream_ editor. > >> Here is what the sed manual thinks. >> >> Thanks for pointing that out (it must be a recent addition; my install= ed >> version doesn't have this text). > > Actually it is older than the version control repository, so 2004 or > older. But it is under "Reporting bugs", not under the -i option > (patches welcome ;)). > > You'll also find it under /usr/share/doc/sed-4.2.1/BUGS. > >> `-i' clobbers read-only files >> In short, `sed -i' will let you delete the contents of a re= ad-only >> file, and in general the `-i' option (*note Invocation: Inv= oking >> sed.) lets you clobber protected files. This is not a bug,= but >> rather a consequence of how the Unix filesystem works. >> >> The permissions on a file say what can happen to the data i= n that >> file, while the permissions on a directory say what can hap= pen to >> the list of files in that directory. `sed -i' will not eve= r open >> for writing a file that is already on disk. Rather, it wi= ll work >> on a temporary file that is finally renamed to the original= name: >> if you rename or delete files, you're actually modifying th= e >> contents of the directory, so the operation depends on the >> permissions of the directory, not of the file. For this sa= me >> reason, `sed' does not let you use `-i' on a writeable file= in a >> read-only directory, and will break hard or symbolic links = when >> `-i' is used on such a file. >> >> I don't think that this addresses the issue at hand. The program does= something >> non-intuitive that can lead to loss of data, and it wouldn't be reason= able to blame >> the user in that instance. > > I agree, but it's not me who designed the Unix filesystem permissions. > >> (Some) other GNU programs don't behave like this. For >> example "truncate foo" on a readonly foo will exit with an error > > Truncate does not process foo for input at the same time, so it isn't't > really relevant. > >> , as will "dd if=3Dfoo of=3Dfoo count=3D10". > > dd has a well-defined behavior for overlapping input and output, and > this well-defined behavior in fact mandates that dd doesn't break hard > links. > >> Likewise, "shuf foo -o foo". > > I consider "shuf foo -o foo" (on a read-write file) to be insecure. > Besides, it works by chance just because it reads everything in memory > first. If it used mmap to process the input file, "shuf foo -o foo" > would be broken, and the only way to fix it would be to do the same as > "sed -i". > > shuf could in fact introduce a "shuf -i" mode that would be consistent > with the way "sed -i" works, including the ability to create a backup > file _and_ the breaking of hard links. Well `sort` and `shuf` need to read all their input before generating output. This is traditional behavior and POSIX also states that -o can refer to one of the input files. Also related to this is having a seperate "replace" wrapper, that would handle all the atomic, backup, permission, ... issues. http://www.pixelbeat.org/docs/unix_file_replacement.html I takes advantage of the existing coreutils to handle the above. I really need to polish that off and submit it (translations in the shell script was one thing that was bothering me). I notice David Wheeler proposed much the same thing with the "rewrite" util (I like that name too): http://permalink.gmane.org/gmane.comp.standards.posix.austin.general/5348 cheers, P=E1draig From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 08:40:53 2012 Received: (at submit) by debbugs.gnu.org; 6 Sep 2012 12:40:54 +0000 Received: from localhost ([127.0.0.1]:42555 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9bO3-0002XK-R6 for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:40:51 -0400 Received: from eggs.gnu.org ([208.118.235.92]:48814) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9bNv-0002X7-Ax for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:40:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1T9bNa-0001US-2j for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:40:27 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:48177) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9bNZ-0001Tp-Bq for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:40:17 -0400 Received: from eggs.gnu.org ([208.118.235.92]:59592) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9bNO-0003H3-IG for bug-coreutils@gnu.org; Thu, 06 Sep 2012 08:40:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1T9bNH-0001A3-8k for bug-coreutils@gnu.org; Thu, 06 Sep 2012 08:40:05 -0400 Received: from mail-pb0-f41.google.com ([209.85.160.41]:58991) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1T9bNH-00019x-33; Thu, 06 Sep 2012 08:39:59 -0400 Received: by pbbro12 with SMTP id ro12so2741656pbb.0 for ; Thu, 06 Sep 2012 05:39:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=+D/CSuydH1eZy72hY4s7FUEpVfKcLfj4e+7vZmQz4DE=; b=ekUSkojB+NAOhGgYwIZcsiVYtXWI7trA+aIV0lgho9t322WLsWnNcFEzj1guT9To1N 3w0jOpEXR8dDnXUweakJoPkOuSVgM3CoZizrc1FFUtMB49BC2iHsAfDeOPL90WryXLIU 1E76BXRDGlmupCnHzffiK8dcHqhrvP76yIAaZrRRK2G0NIqJrFz2adyXUfM8gq8YPVzY DFsgaxcgUC8tmh6PVI2OlN9e4fa1j8I6qibWyS4FkSO4/tU2085ziAvBO+j9xJEl/JaY 7KHIfLjj59EYmds3llQgKukxqZBthe1qhaVeOA+XZrfSv83hgrlREG1gwGuVnXyFV0cN 0M3w== Received: by 10.68.221.42 with SMTP id qb10mr4403541pbc.155.1346935198246; Thu, 06 Sep 2012 05:39:58 -0700 (PDT) Received: from yakj.usersys.redhat.com (93-34-169-1.ip50.fastwebnet.it. [93.34.169.1]) by mx.google.com with ESMTPS id gh7sm1284564pbc.29.2012.09.06.05.39.54 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 06 Sep 2012 05:39:56 -0700 (PDT) Message-ID: <50489997.7010609@gnu.org> Date: Thu, 06 Sep 2012 14:39:51 +0200 From: Paolo Bonzini User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120717 Thunderbird/14.0 MIME-Version: 1.0 To: =?ISO-8859-1?Q?P=E1draig_Brady?= Subject: Re: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <50489750.6080502@draigBrady.com> In-Reply-To: <50489750.6080502@draigBrady.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 208.118.235.17 X-Spam-Score: -6.1 (------) X-Debbugs-Envelope-To: submit Cc: gnu-prog-discuss@gnu.org, "David A. Wheeler" , Bug-coreutils , John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.1 (------) Il 06/09/2012 14:30, Pádraig Brady ha scritto: >> >> I consider "shuf foo -o foo" (on a read-write file) to be insecure. >> Besides, it works by chance just because it reads everything in memory >> first. If it used mmap to process the input file, "shuf foo -o foo" >> would be broken, and the only way to fix it would be to do the same as >> "sed -i". >> >> shuf could in fact introduce a "shuf -i" mode that would be consistent >> with the way "sed -i" works, including the ability to create a backup >> file _and_ the breaking of hard links. > > Well `sort` and `shuf` need to read all their input before > generating output. Yes, but they could use mmap instead of a single large buffer to cope with files that are bigger than the available memory, but smaller than the address space. > This is traditional behavior and > POSIX also states that -o can refer to one of the input files. Interesting, thanks! Paolo From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 08:44:28 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 12:44:28 +0000 Received: from localhost ([127.0.0.1]:42559 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9bRZ-0002cM-W6 for submit@debbugs.gnu.org; Thu, 06 Sep 2012 08:44:27 -0400 Received: from moutng.kundenserver.de ([212.227.17.9]:65237) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9bRQ-0002c6-0N for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 08:44:21 -0400 Received: from oxbaltgw11.schlund.de (oxbaltgw11.schlund.de [172.19.246.17]) by mrelayeu.kundenserver.de (node=mrbap3) with ESMTP (Nemesis) id 0MeBiQ-1Svto1406A-00PvJn; Thu, 06 Sep 2012 14:44:01 +0200 Date: Thu, 6 Sep 2012 14:43:59 +0200 (CEST) From: Bernhard Voelker To: Paolo Bonzini , John Darrington Message-ID: <496941318.1140203.1346935439589.JavaMail.open-xchange@email.1und1.de> In-Reply-To: <5048933A.7020105@gnu.org> References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Priority: 3 Importance: Medium X-Mailer: Open-Xchange Mailer v- X-Provags-ID: V02:K0:4GmDY+rG91Qn2yFUgaod8WZl0vcBgdxIwKqp91M9fNA Dr/ikZ9EBikyoUqjYguszDbex0sJYlAELMz3CCWNYbS2Nnor93 643SjXjLuhFU3wSMD4Nq0FGU8SXfADFBdrqgfp/XJT92Gam/GN tpv+kWqG9nXr6fWGxZ+mS08EldrSSQk7EfV0G5+TCk0azS14UD oWqEu1mWWHSlBX5nGnURnsdMG3ZOUcUz2gD8I9SD8D1opfQlEo bzvJB8Rqu2jbwnHQ4GQp7TZdoipneAjsJkloorfoxjItADBxIz UCgLnfMHo/k92XL/LA+rgC0AVDdvPoKwJh315QclyDzSsYKtdT LHZ1PyQl/ZYtvZEUlppPl/C4gVEPqhw2M9qepc3tpNOtjUcXxb z8LpfuBGEWgG0owX/NnFMCaRPvVMKSCoZdHtJHrM+23TwywLg5 Spy85 X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list Reply-To: Bernhard Voelker List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On September 6, 2012 at 2:12 PM Paolo Bonzini wrote: > [For bug-coreutils: the context here is that sed -i, just like perl -i, > breaks hard links and thus destroys the content of files with 0400 > permission]. I consider this being 2 different cases: * 'sed -i' breaks hard links: That's because it places the output at place where the input file was (by unlink+rename). That's okay IMO. * 'sed -i' destroys the content of files with 0400 perms: That's a bug IMHO. sed should open the input file read-write. If that fails, then the input won't change with a nice diagnostic. In 'sort -o', we recently added a similar check right at the beginning to avoid useless processing possibly leading to an error afterwards: see http://bugs.gnu.org/11816 with the commit http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commit;h=44fbd3fd862e34d42006f8b74cb11c9c56346417 Have a nice day, Berny From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 09:27:23 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 13:27:23 +0000 Received: from localhost ([127.0.0.1]:42626 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9c77-0003bi-Lz for submit@debbugs.gnu.org; Thu, 06 Sep 2012 09:27:22 -0400 Received: from mx.meyering.net ([88.168.87.75]:46222) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9c70-0003bU-MK for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 09:27:18 -0400 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id 386A4600B3; Thu, 6 Sep 2012 15:27:01 +0200 (CEST) From: Jim Meyering To: Paolo Bonzini Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files In-Reply-To: <5048933A.7020105@gnu.org> (Paolo Bonzini's message of "Thu, 06 Sep 2012 14:12:42 +0200") References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> Date: Thu, 06 Sep 2012 15:27:01 +0200 Message-ID: <87lignxowq.fsf@rho.meyering.net> Lines: 14 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Paolo Bonzini wrote: > [For bug-coreutils: the context here is that sed -i, just like perl -i, > breaks hard links and thus destroys the content of files with 0400 > permission]. Did I misunderstand how "destroy" is used above? $ echo important > k $ chmod a-w k $ sed -i s/./X/ k $ cat k XXXXXXXXX $ ls -og k -r--------. 1 10 Sep 6 15:23 k From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 12:12:13 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 16:12:14 +0000 Received: from localhost ([127.0.0.1]:44374 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9egf-0000Bj-K9 for submit@debbugs.gnu.org; Thu, 06 Sep 2012 12:12:13 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:49718) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9egd-0000Ba-1J for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 12:12:12 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 5CD4539E800D; Thu, 6 Sep 2012 09:11:57 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rsWMUfrv0Un7; Thu, 6 Sep 2012 09:11:57 -0700 (PDT) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 012C939E800A; Thu, 6 Sep 2012 09:11:56 -0700 (PDT) Message-ID: <5048CB48.2060203@cs.ucla.edu> Date: Thu, 06 Sep 2012 09:11:52 -0700 From: Paul Eggert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120828 Thunderbird/15.0 MIME-Version: 1.0 To: Paolo Bonzini Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> In-Reply-To: <5048933A.7020105@gnu.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 09/06/2012 05:12 AM, Paolo Bonzini wrote: > I consider "shuf foo -o foo" (on a read-write file) to be insecure. > Besides, it works by chance It's not by chance. shuf is designed to let you shuffle a file in-place, and is documented to work, by analogy with "sort -o foo foo". If we ever change "shuf" to use mmap, we'll make sure it continues to work in-place. I'm not sure what is meant by "insecure" here. Of course there are race conditions if other processes modify a file when "shuf" reads or writes it, but that's true for pretty much any program that reads or writes any file, including sed -i. From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 12:24:53 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 16:24:53 +0000 Received: from localhost ([127.0.0.1]:44408 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9esu-0000Tw-Th for submit@debbugs.gnu.org; Thu, 06 Sep 2012 12:24:53 -0400 Received: from mail-pb0-f44.google.com ([209.85.160.44]:49514) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9ess-0000To-L2 for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 12:24:51 -0400 Received: by pbbrr4 with SMTP id rr4so2750134pbb.3 for <12366@debbugs.gnu.org>; Thu, 06 Sep 2012 09:24:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=hQn6NIxuECYNAOQ96jmqAyQ45veFkGaN+6REyvxV4pM=; b=BxdOVYe7fql1gAJ11xXdmyOOUwBO9DmLncZ3E7IHhdixlvc6LwfGS6rzeTdlyssvf3 Hn5L58tiqQaBvbyfxak1X1L00egP/x+NtSjo013G9Ul0FXLZtHbvmeyLN+z4+7Uzv1Q6 s3uCM3rW0YKS7pf/wh346V/4I2I79bkp3YgSwDvLzXKZJ6PAy0Pt5ESxLcuvozxx0ZF0 kpDLMAJHVrjnHlij6zN7NSptq/hRkOLuY7mZ6IfCHKGVXxnzT4Q/0ff0LZ+50slNtPct XxUbBLWe+bagpKh340jkR1+4aRYdIrLbgykAaX040PwI0rKR4G83uAJsvPEUzf9k1r8L Dezg== Received: by 10.68.134.99 with SMTP id pj3mr5412992pbb.13.1346948676851; Thu, 06 Sep 2012 09:24:36 -0700 (PDT) Received: from yakj.usersys.redhat.com (93-34-169-1.ip50.fastwebnet.it. [93.34.169.1]) by mx.google.com with ESMTPS id th6sm1586433pbc.0.2012.09.06.09.24.32 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 06 Sep 2012 09:24:34 -0700 (PDT) Message-ID: <5048CE3D.8060903@gnu.org> Date: Thu, 06 Sep 2012 18:24:29 +0200 From: Paolo Bonzini User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120717 Thunderbird/14.0 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> In-Reply-To: <5048CB48.2060203@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) Il 06/09/2012 18:11, Paul Eggert ha scritto: >> > I consider "shuf foo -o foo" (on a read-write file) to be insecure. >> > Besides, it works by chance > It's not by chance. shuf is designed to let you > shuffle a file in-place, and is documented to work, > by analogy with "sort -o foo foo". If we ever > change "shuf" to use mmap, we'll make > sure it continues to work in-place. Yeah, I read that from Padraig. I stand corrected. > I'm not sure what is meant by "insecure" here. > Of course there are race conditions if other > processes modify a file when "shuf" > reads or writes it, but that's true for pretty > much any program that reads or writes any file, > including sed -i. No, unlink/rename "sed -i" replaces the file atomically. A program that reads the target file will never be able to observe an intermediate result. This is not true of "shuf -o foo foo". (In addition, the temporary file for "sed -i" is opened with 0400 permissions for the user running sed, and will not have the same owner/group/ACL/context as the target file until just before renaming to the destination). It's mostly paranoia, but the race window _is_ there unless you use rename and break hard links. Paolo From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 12:36:10 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 16:36:10 +0000 Received: from localhost ([127.0.0.1]:44426 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9f3q-0000k9-Jy for submit@debbugs.gnu.org; Thu, 06 Sep 2012 12:36:10 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:51097) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9f3p-0000k2-2I for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 12:36:09 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 9EF99A60001; Thu, 6 Sep 2012 09:35:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6Jmry4Gk04oZ; Thu, 6 Sep 2012 09:35:55 -0700 (PDT) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 47B7439E8014; Thu, 6 Sep 2012 09:35:55 -0700 (PDT) Message-ID: <5048D0EA.1050407@cs.ucla.edu> Date: Thu, 06 Sep 2012 09:35:54 -0700 From: Paul Eggert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120828 Thunderbird/15.0 MIME-Version: 1.0 To: Paolo Bonzini Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> In-Reply-To: <5048CE3D.8060903@gnu.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 09/06/2012 09:24 AM, Paolo Bonzini wrote: > A program that reads the target file will never > be able to observe an intermediate result. Sure, but that doesn't fix the race condition I mentioned. If some other process is writing F while I run 'sed -i F', F is not replaced atomically. That's true even if the other process is another instance of 'sed'. While 'sed -i' solves some race conditions, it doesn't even come close to solving them all. Fixing this problem in general is above sed's pay grade, just as it's above shuf's. From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 12:39:07 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 16:39:07 +0000 Received: from localhost ([127.0.0.1]:44433 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9f6h-0000qE-Bc for submit@debbugs.gnu.org; Thu, 06 Sep 2012 12:39:07 -0400 Received: from mail-bk0-f44.google.com ([209.85.214.44]:48912) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9f6e-0000q1-NP for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 12:39:05 -0400 Received: by bkty12 with SMTP id y12so889877bkt.3 for <12366@debbugs.gnu.org>; Thu, 06 Sep 2012 09:38:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=RxwHO51r5y5bOB5FQIkW4u7LYxNAq5sK1vTPQ65+sNE=; b=p3tMrVOU6Cm54444MY6uJQbzfRFWv53Er5YPcb4QssUPZL2Eqx8UTGmnCTlwGRydY7 nBwr+ecp6ch/BMfzI3TWpSOquSCn3smnuSo2UU7pbEmpJb/LZKUsg2aOieceb7mT6U3t AHIpjIlWgVi1Q7ko9EzT91FWYAEs2bLAC5gCbPBwQBqo3NS3GORer3q2JOrkdXDbIEiW efkoyXgIBNH9yVV6CcUP51E3+qx5AmZze8kV2Lj+6GZ9GlBZN1vfTaJKAEwcyJRn9Usu 3rf8MeTAZwsq49A8PBHxU3xFdUrkOQcUkZyMbL+0zTglG2dKtOWKsji7gVeGcvDtEzyw koow== Received: by 10.205.135.6 with SMTP id ie6mr1388154bkc.139.1346949529731; Thu, 06 Sep 2012 09:38:49 -0700 (PDT) Received: from yakj.usersys.redhat.com (93-34-169-1.ip50.fastwebnet.it. [93.34.169.1]) by mx.google.com with ESMTPS id hs2sm1778978bkc.1.2012.09.06.09.38.48 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 06 Sep 2012 09:38:48 -0700 (PDT) Message-ID: <5048D196.9070308@gnu.org> Date: Thu, 06 Sep 2012 18:38:46 +0200 From: Paolo Bonzini User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120717 Thunderbird/14.0 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> In-Reply-To: <5048D0EA.1050407@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) Il 06/09/2012 18:35, Paul Eggert ha scritto: >> > A program that reads the target file will never >> > be able to observe an intermediate result. > Sure, but that doesn't fix the race condition I > mentioned. If some other process is writing F > while I run 'sed -i F', F is not replaced atomically. How not so? Paolo > That's true even if the other process is another > instance of 'sed'. From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 13:00:54 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 17:00:54 +0000 Received: from localhost ([127.0.0.1]:44483 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9fRj-0001Ln-Jz for submit@debbugs.gnu.org; Thu, 06 Sep 2012 13:00:52 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:52503) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9fRh-0001Lf-Cz for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 13:00:50 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 9EFF2A60002; Thu, 6 Sep 2012 10:00:35 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6iD4CeI-GV6v; Thu, 6 Sep 2012 10:00:35 -0700 (PDT) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 191C239E800A; Thu, 6 Sep 2012 10:00:35 -0700 (PDT) Message-ID: <5048D6B2.6090104@cs.ucla.edu> Date: Thu, 06 Sep 2012 10:00:34 -0700 From: Paul Eggert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120828 Thunderbird/15.0 MIME-Version: 1.0 To: Paolo Bonzini Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> In-Reply-To: <5048D196.9070308@gnu.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) >> If some other process is writing F >> while I run 'sed -i F', F is not replaced atomically. > How not so? For example: echo ac >f sed -i 's/a/b/' f & sed -i 's/c/d/' f wait cat f If 'sed' were truly atomic, then the output of this would always be 'bd'. But it's not. From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 13:12:40 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 17:12:41 +0000 Received: from localhost ([127.0.0.1]:44513 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9fdA-0001by-K3 for submit@debbugs.gnu.org; Thu, 06 Sep 2012 13:12:40 -0400 Received: from joseki.proulx.com ([216.17.153.58]:57154) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9fd9-0001br-8m for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 13:12:39 -0400 Received: from hysteria.proulx.com (hysteria.proulx.com [192.168.230.119]) by joseki.proulx.com (Postfix) with ESMTP id 7E67721448; Thu, 6 Sep 2012 11:12:25 -0600 (MDT) Received: by hysteria.proulx.com (Postfix, from userid 1000) id 6911C2DCC2; Thu, 6 Sep 2012 11:12:25 -0600 (MDT) Date: Thu, 6 Sep 2012 11:12:25 -0600 From: Bob Proulx To: gnu-prog-discuss@gnu.org, John Darrington , 12366@debbugs.gnu.org Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files Message-ID: <20120906171225.GB19913@hysteria.proulx.com> Mail-Followup-To: gnu-prog-discuss@gnu.org, John Darrington , 12366@debbugs.gnu.org References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5048D6B2.6090104@cs.ucla.edu> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Paul Eggert wrote: > >> If some other process is writing F > >> while I run 'sed -i F', F is not replaced atomically. > > > How not so? > > For example: > > echo ac >f > sed -i 's/a/b/' f & > sed -i 's/c/d/' f > wait > cat f > > If 'sed' were truly atomic, then the output of this would > always be 'bd'. But it's not. The file replacement is atomic. The reading of the file is not. Bob From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 13:23:49 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 17:23:49 +0000 Received: from localhost ([127.0.0.1]:44576 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9fnx-00022F-0g for submit@debbugs.gnu.org; Thu, 06 Sep 2012 13:23:49 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:53729) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9fnu-000223-EC for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 13:23:47 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id B29C239E800D; Thu, 6 Sep 2012 10:23:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7V+Gv7lFWgO9; Thu, 6 Sep 2012 10:23:32 -0700 (PDT) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 67AB5A60002; Thu, 6 Sep 2012 10:23:32 -0700 (PDT) Message-ID: <5048DC14.1060001@cs.ucla.edu> Date: Thu, 06 Sep 2012 10:23:32 -0700 From: Paul Eggert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120828 Thunderbird/15.0 MIME-Version: 1.0 To: gnu-prog-discuss@gnu.org, John Darrington , 12366@debbugs.gnu.org Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> <20120906171225.GB19913@hysteria.proulx.com> In-Reply-To: <20120906171225.GB19913@hysteria.proulx.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 09/06/2012 10:12 AM, Bob Proulx wrote: > The file replacement is atomic. The reading of the file is not. Sure, but the point is that from the end user's point of view, 'sed -i' is not atomic, and can't be expected to be atomic. 'sed -i' and 'sort -o' both use some atomic operations internally, but neither is atomic overall. Users who want atomicity must look elsewhere, or implement it themselves. From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 13:35:47 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 17:35:47 +0000 Received: from localhost ([127.0.0.1]:44588 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9fzW-0002WR-Mg for submit@debbugs.gnu.org; Thu, 06 Sep 2012 13:35:47 -0400 Received: from moutng.kundenserver.de ([212.227.17.9]:62937) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9fzU-0002WF-LN for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 13:35:45 -0400 Received: from oxbaltgw11.schlund.de (oxbaltgw11.schlund.de [172.19.246.17]) by mrelayeu.kundenserver.de (node=mreu1) with ESMTP (Nemesis) id 0MJoZ6-1TAluw0XUy-001jVw; Thu, 06 Sep 2012 19:35:30 +0200 Date: Thu, 6 Sep 2012 19:35:29 +0200 (CEST) From: Bernhard Voelker To: 12366@debbugs.gnu.org, Paul Eggert , gnu-prog-discuss@gnu.org, John Darrington Message-ID: <1012365886.1155479.1346952929840.JavaMail.open-xchange@email.1und1.de> In-Reply-To: <5048DC14.1060001@cs.ucla.edu> References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> <20120906171225.GB19913@hysteria.proulx.com> <5048DC14.1060001@cs.ucla.edu> Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Priority: 3 Importance: Medium X-Mailer: Open-Xchange Mailer v- X-Provags-ID: V02:K0:wTJkPfcEzG9jirRjbWQD+Ngu6gp1nWXJ0unIpHw9tTu 8ky3m7K6HFO4xtI5PcgWgTUhN+7RNmjqw9Ae/DCKRu2vm8A8Lz b29ITqMGl/XMZ/YZWIsiB/4GY+HyzcsUgcPiwdyoKgjnx0A2Dh ILvykUPgujNLycjxO9sFuBjzecADodGbwodcR7aupd656ZPqiU A4xiZNBYdWO1h39ZySnBT7FBwda03FJjLPVpRb3fe+vpKG7h6a SNHiyH5nwQ2P7bJIu/fixcsYjxznzCK7huhVt2iFGEZ7ICs4O+ DiP6cBwAD3EFQEpa46UYmBubP9FCOyR7wPe6Jx5mu9VMrKiVcn sDqQg3AFV5EaeVf7wsbKgjw6hBOhOApkWCWy7jnQJCyDxqDYV1 TqMj/x7ZCtzl7lFB8HPg2ubcKjJg3/qR1g0492k0dVYknVjXgB WId7e X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list Reply-To: Bernhard Voelker List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On September 6, 2012 at 7:23 PM Paul Eggert wrote: > On 09/06/2012 10:12 AM, Bob Proulx wrote: > > The file replacement is atomic. The reading of the file is not. > > Sure, but the point is that from the end user's > point of view, 'sed -i' is not atomic, and can't > be expected to be atomic. > > 'sed -i' and 'sort -o' both use some atomic operations > internally, but neither is atomic overall. Users who > want atomicity must look elsewhere, or implement it > themselves. Why can't 'sed -i' be made atomic for the user? Today, it creates a temporary file for the output. At the end, it calls rename(). What if it instead rewinds the input and that temporary file and copies it's content to the input file? Okay, this is slower than a rename(), but it would write into the same inode. To preserve today's behaviour, this could be done with a new option like --in-place-same. Just a thought. Have a nice day, Berny From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 14:21:49 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 18:21:49 +0000 Received: from localhost ([127.0.0.1]:44677 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9gi4-0003ds-Ii for submit@debbugs.gnu.org; Thu, 06 Sep 2012 14:21:49 -0400 Received: from blade.simplesystems.org ([65.66.246.74]:42591) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9gi2-0003dk-8E for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 14:21:47 -0400 Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q86ILVjs020651; Thu, 6 Sep 2012 13:21:31 -0500 (CDT) Date: Thu, 6 Sep 2012 13:21:31 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Paolo Bonzini Subject: Re: [gnu-prog-discuss] bug#12366: Writing unwritable files In-Reply-To: <5048CE3D.8060903@gnu.org> Message-ID: References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Thu, 06 Sep 2012 13:21:31 -0500 (CDT) X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On Thu, 6 Sep 2012, Paolo Bonzini wrote: > >> I'm not sure what is meant by "insecure" here. >> Of course there are race conditions if other >> processes modify a file when "shuf" >> reads or writes it, but that's true for pretty >> much any program that reads or writes any file, >> including sed -i. > > No, unlink/rename "sed -i" replaces the file atomically. A program that POSIX rename assures that the destination path always exists if it already existed. If unlink/ln was used, then the destination path would temporarily be missing. While 'rename' is occuring, a second (parallel) reader/writer has no idea which version will be accessed. Microsoft Windows and other operating systems might not support the POSIX sematic. Certain filesystems (or their implementation) might not support atomic 'rename'. > It's mostly paranoia, but the race window _is_ there unless you use > rename and break hard links. Yes, you must use rename, and rename would need to work as per the POSIX specification. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 14:23:39 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 18:23:39 +0000 Received: from localhost ([127.0.0.1]:44682 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9gjq-0003gm-Lw for submit@debbugs.gnu.org; Thu, 06 Sep 2012 14:23:38 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:57001) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9gjo-0003gf-91 for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 14:23:37 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 5A168A60003; Thu, 6 Sep 2012 11:23:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gw0gzCqedYlr; Thu, 6 Sep 2012 11:23:21 -0700 (PDT) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id D8053A60002; Thu, 6 Sep 2012 11:23:21 -0700 (PDT) Message-ID: <5048EA19.1000905@cs.ucla.edu> Date: Thu, 06 Sep 2012 11:23:21 -0700 From: Paul Eggert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120828 Thunderbird/15.0 MIME-Version: 1.0 To: Bernhard Voelker Subject: Re: bug#12366: [gnu-prog-discuss] Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> <20120906171225.GB19913@hysteria.proulx.com> <5048DC14.1060001@cs.ucla.edu> <1012365886.1155479.1346952929840.JavaMail.open-xchange@email.1und1.de> In-Reply-To: <1012365886.1155479.1346952929840.JavaMail.open-xchange@email.1und1.de> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 09/06/2012 10:35 AM, Bernhard Voelker wrote: > Why can't 'sed -i' be made atomic for the user? > Today, it creates a temporary file for the output. > At the end, it calls rename(). What if it instead > rewinds the input and that temporary file and copies > it's content to the input file? That's kind of what 'sort -o' does, and it also has race conditions. For example, in that last phase while it's copying the content to the input file, some other process might be reading the input file. There is no good general and portable atomic solution to this sort of problem, not in POSIX anyway. Practical implementations of utilities like 'sed' and 'sort' and 'shuf' all involve races of some sort or another. From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 16:06:45 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 20:06:45 +0000 Received: from localhost ([127.0.0.1]:44798 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9iLc-00062w-Od for submit@debbugs.gnu.org; Thu, 06 Sep 2012 16:06:45 -0400 Received: from cellform.com ([203.82.214.165]:50557 helo=cloacina.cellform.com) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9haN-0004wq-Mw for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 15:17:58 -0400 Received: from cloacina.cellform.com (localhost [127.0.0.1]) by cloacina.cellform.com (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id q86JHI6t025327; Thu, 6 Sep 2012 19:17:18 GMT Received: (from john@localhost) by cloacina.cellform.com (8.14.3/8.14.3/Submit) id q86JHGnk025326; Thu, 6 Sep 2012 19:17:16 GMT Date: Thu, 6 Sep 2012 19:17:15 +0000 From: John Darrington To: Paul Eggert Subject: Re: [gnu-prog-discuss] bug#12366: Writing unwritable files Message-ID: <20120906191715.GA25303@cellform.com> References: <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> <20120906171225.GB19913@hysteria.proulx.com> <5048DC14.1060001@cs.ucla.edu> <1012365886.1155479.1346952929840.JavaMail.open-xchange@email.1und1.de> <5048EA19.1000905@cs.ucla.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0OAP2g/MAC+5xKAE" Content-Disposition: inline In-Reply-To: <5048EA19.1000905@cs.ucla.edu> User-Agent: Mutt/1.5.18 (2008-05-17) X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 X-Mailman-Approved-At: Thu, 06 Sep 2012 16:06:44 -0400 Cc: Bernhard Voelker , gnu-prog-discuss@gnu.org, 12366@debbugs.gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) --0OAP2g/MAC+5xKAE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Sep 06, 2012 at 11:23:21AM -0700, Paul Eggert wrote: On 09/06/2012 10:35 AM, Bernhard Voelker wrote: > Why can't 'sed -i' be made atomic for the user? > Today, it creates a temporary file for the output. > At the end, it calls rename(). What if it instead > rewinds the input and that temporary file and copies > it's content to the input file? =20 That's kind of what 'sort -o' does, and it also has race conditions. For example, in that last phase while it's copying the content to the input file, some other process might be reading the input file. I don't think that matters. In fact I like to be able to use=20 tail -f to see what's being written to a file, and find it the mozilla like behaviour, where I have to wait until the=20 entire file is downloaded in order to see the first byte,=20 rather annoying. =20 J' --=20 PGP Public key ID: 1024D/2DE827B3=20 fingerprint =3D 8797 A26D 0854 2EAB 0285 A290 8A67 719C 2DE8 27B3 See http://keys.gnupg.net or any PGP keyserver for public key. --0OAP2g/MAC+5xKAE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iD8DBQFQSPa7imdxnC3oJ7MRAo52AJ94OrCpwIyYoRCNsdG06G7KDTtXUgCfc8DM wiJjQgFLGiegq2cPgtIX/Wg= =LSA9 -----END PGP SIGNATURE----- --0OAP2g/MAC+5xKAE-- From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 06 17:01:56 2012 Received: (at 12366) by debbugs.gnu.org; 6 Sep 2012 21:01:56 +0000 Received: from localhost ([127.0.0.1]:44860 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9jD1-0007Ld-VG for submit@debbugs.gnu.org; Thu, 06 Sep 2012 17:01:56 -0400 Received: from mail-wg0-f46.google.com ([74.125.82.46]:60924) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1T9jD0-0007LW-9y for 12366@debbugs.gnu.org; Thu, 06 Sep 2012 17:01:54 -0400 Received: by wgi16 with SMTP id 16so1666854wgi.15 for <12366@debbugs.gnu.org>; Thu, 06 Sep 2012 14:01:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=dImqlTmxgnCuDJT6pm0yfXFVPpZt1/G5w0JgP+uBoKM=; b=it39o1ke2ex6EiXVEnc7+ysHRF1eDEEC0H9WOH88c8WY/4rZgIErcKqR4zZgUyq4VH kOUECqhF+XlGaBnMasS+gC30zf/8+QeUNT2izHrXBRj06/CVXrWqMmq6YSRs16hq7nqT FtweRcRBUkS9fVAmiTm+0JKc7wqJApcZJuXhwVn5r9CaviGhKV1ktnZjS2rWH4qjATNr rWc/kNSB7I5U5HA/NYccwCvNg7UKSPE22DoVDUkl3I+A74y3PftC51sQhiwxLyM0HO6y pHuQTZQmJvWu7n6DOaekvv8TmE5+HncBF7lIN/Z6Fz3xoYgoz9zCjS1FDSrnU64Hcve2 e8wA== Received: by 10.180.107.103 with SMTP id hb7mr7411972wib.3.1346965299561; Thu, 06 Sep 2012 14:01:39 -0700 (PDT) Received: from yakj.usersys.redhat.com (93-34-169-1.ip50.fastwebnet.it. [93.34.169.1]) by mx.google.com with ESMTPS id r9sm8493301wia.2.2012.09.06.14.01.38 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 06 Sep 2012 14:01:38 -0700 (PDT) Message-ID: <50490F31.5030602@gnu.org> Date: Thu, 06 Sep 2012 23:01:37 +0200 From: Paolo Bonzini User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120717 Thunderbird/14.0 MIME-Version: 1.0 To: Bob Friesenhahn Subject: Re: [gnu-prog-discuss] bug#12366: Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) Il 06/09/2012 20:21, Bob Friesenhahn ha scritto: >> >> No, unlink/rename "sed -i" replaces the file atomically. A program that > > POSIX rename assures that the destination path always exists if it > already existed. My bad, I meant link-breaking/rename. Of course you must not unlink first. Paolo From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 07 12:38:34 2012 Received: (at 12366) by debbugs.gnu.org; 7 Sep 2012 16:38:34 +0000 Received: from localhost ([127.0.0.1]:46935 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TA1Zi-0003Te-Gy for submit@debbugs.gnu.org; Fri, 07 Sep 2012 12:38:34 -0400 Received: from mail-pb0-f44.google.com ([209.85.160.44]:45108) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TA1Zb-0003TR-93 for 12366@debbugs.gnu.org; Fri, 07 Sep 2012 12:38:32 -0400 Received: by pbbrr4 with SMTP id rr4so4280038pbb.3 for <12366@debbugs.gnu.org>; Fri, 07 Sep 2012 09:38:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=cghnoqHKt0SY4OVPPTP1hJYqQbk7IW3oVv7DC9f1FcU=; b=OAtvsZtfZjJmQjM2J4PQn0ikeKn1DgdF/Byu9qS3PJaWAYbff+0UXgrcW0lwCdFmLx Llo7ORbJsSuGXEB8PCRF0bv2d4bZjvb5WaV0oWwhfrNw1ij75nJhN7rzCgy0Rttlk8+I vWlPeojZDlsIPBW4Goqwd73yBIefqLTwWJnIqoaPrMKZ3YT5DDbR8q5vUJemaGPVgi0T Y36JoEGUNr/LEZej8bCRztDCjQ4JjbmbtfjoF1O/U1rD1s3tf0656WAOicVxImvoHePR 706iSGG9Ux5kWA02nF6ED0qmLFw7aBy/BGfJ9wLTAGhka4psQiJtFYROMLI4IuXJE0jH hLGg== Received: by 10.68.218.101 with SMTP id pf5mr10541778pbc.60.1347035887940; Fri, 07 Sep 2012 09:38:07 -0700 (PDT) Received: from yakj.usersys.redhat.com (93-34-169-1.ip50.fastwebnet.it. [93.34.169.1]) by mx.google.com with ESMTPS id ty1sm3379584pbc.76.2012.09.07.09.38.03 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 07 Sep 2012 09:38:06 -0700 (PDT) Message-ID: <504A22E8.1070608@gnu.org> Date: Fri, 07 Sep 2012 18:38:00 +0200 From: Paolo Bonzini User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120717 Thunderbird/14.0 MIME-Version: 1.0 To: Paul Eggert Subject: Re: [gnu-prog-discuss] bug#12366: Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> <20120906171225.GB19913@hysteria.proulx.com> <5048DC14.1060001@cs.ucla.edu> In-Reply-To: <5048DC14.1060001@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) Il 06/09/2012 19:23, Paul Eggert ha scritto: >> > The file replacement is atomic. The reading of the file is not. > Sure, but the point is that from the end user's > point of view, 'sed -i' is not atomic, and can't > be expected to be atomic. Atomic file replacement is what matters for security. Paolo From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 07 15:47:05 2012 Received: (at 12366) by debbugs.gnu.org; 7 Sep 2012 19:47:05 +0000 Received: from localhost ([127.0.0.1]:47150 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TA4W7-0007lV-PT for submit@debbugs.gnu.org; Fri, 07 Sep 2012 15:47:04 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:55579) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TA4W4-0007l4-HD for 12366@debbugs.gnu.org; Fri, 07 Sep 2012 15:47:01 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 9C96639E8016; Fri, 7 Sep 2012 12:46:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TiVniW0cqPfn; Fri, 7 Sep 2012 12:46:40 -0700 (PDT) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 27DAD39E800D; Fri, 7 Sep 2012 12:46:40 -0700 (PDT) Message-ID: <504A4F1F.3000802@cs.ucla.edu> Date: Fri, 07 Sep 2012 12:46:39 -0700 From: Paul Eggert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120828 Thunderbird/15.0 MIME-Version: 1.0 To: Paolo Bonzini Subject: Re: bug#12366: [gnu-prog-discuss] bug#12366: Writing unwritable files References: <20120906092905.GA7603@cellform.com> <50487157.9000907@gnu.org> <20120906103841.GA13245@cellform.com> <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> <20120906171225.GB19913@hysteria.proulx.com> <5048DC14.1060001@cs.ucla.edu> <504A22E8.1070608@gnu.org> In-Reply-To: <504A22E8.1070608@gnu.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 Cc: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 09/07/2012 09:38 AM, Paolo Bonzini wrote: > Atomic file replacement is what matters for security. Unfortunately, 'sed's use of atomic file replacement does not suffice for security. For example, suppose sysadmins (mistakenly) followed the practice of using 'sed -i' to remove users from /etc/passwd. And suppose there are two misbehaving users moe and larry, and two sysadmins bonzini and eggert. bonzini discovers that moe's misbehaving, and types: sed -i '/^moe:/d' /etc/passwd and thinks, "Great! moe can't log in any more." Similarly eggert discovers that larry's misbehaving, and types: sed -i '/^larry:/d' /etc/passwd and thinks, "All right! I've done my job too." Unfortunately, it could be that moe can still log in afterwards. Or maybe larry can. We don't know, because 'sed -i' is not atomic, which means /etc/passwd might contain moe afterwards, or maybe larry. Of course one could wrap 'sed -i' inside a larger script, that arranges for atomicity at the end-user level. But the same is true for 'sort -o'. Perhaps the method of 'sed -i' buys the user *something*, but whatever that something is, isn't immediately obvious. When it comes to security mechanisms, simplicity and clarity are critical, and unfortunately 'sed -i' has problems in this area, just as 'sort -o' does. From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 07 16:43:07 2012 Received: (at 12366) by debbugs.gnu.org; 7 Sep 2012 20:43:07 +0000 Received: from localhost ([127.0.0.1]:47226 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TA5OM-0000ff-Rm for submit@debbugs.gnu.org; Fri, 07 Sep 2012 16:43:07 -0400 Received: from joseki.proulx.com ([216.17.153.58]:35560) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TA5OK-0000fX-RB for 12366@debbugs.gnu.org; Fri, 07 Sep 2012 16:43:06 -0400 Received: from hysteria.proulx.com (hysteria.proulx.com [192.168.230.119]) by joseki.proulx.com (Postfix) with ESMTP id 86A7D211D6; Fri, 7 Sep 2012 14:42:44 -0600 (MDT) Received: by hysteria.proulx.com (Postfix, from userid 1000) id 38F882DCC2; Fri, 7 Sep 2012 14:42:44 -0600 (MDT) Date: Fri, 7 Sep 2012 14:42:44 -0600 From: Bob Proulx To: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington Subject: Re: bug#12366: [gnu-prog-discuss] bug#12366: Writing unwritable files Message-ID: <20120907204244.GA12635@hysteria.proulx.com> Mail-Followup-To: 12366@debbugs.gnu.org, gnu-prog-discuss@gnu.org, John Darrington References: <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> <20120906171225.GB19913@hysteria.proulx.com> <5048DC14.1060001@cs.ucla.edu> <504A22E8.1070608@gnu.org> <504A4F1F.3000802@cs.ucla.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <504A4F1F.3000802@cs.ucla.edu> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 12366 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Paul Eggert wrote: > Paolo Bonzini wrote: > > Atomic file replacement is what matters for security. > > Unfortunately, 'sed's use of atomic file replacement does not > suffice for security. > > For example, suppose sysadmins (mistakenly) followed the practice of > using 'sed -i' to remove users from /etc/passwd. And suppose there > are two misbehaving users moe and larry, and two sysadmins bonzini and > eggert. bonzini discovers that moe's misbehaving, and types: > > sed -i '/^moe:/d' /etc/passwd Using /etc/passwd isn't a good example because system convention dictates that a /etc/passwd.lock must be observed for any edits there specifically for the problem you are illustrating. The above would not be correct even if sed were fully atomic overall. > Of course one could wrap 'sed -i' inside a larger script, that > arranges for atomicity at the end-user level. Right. The 'vipw' script for example. :-) [I have abused the EDITOR variable for that purpose many times. Set it to either an inline script or to a real script and use it to safely edit these types of files. More with 'visudo' though.] Bob From debbugs-submit-bounces@debbugs.gnu.org Sun Oct 07 05:13:41 2012 Received: (at 12366-done) by debbugs.gnu.org; 7 Oct 2012 09:13:42 +0000 Received: from localhost ([127.0.0.1]:59016 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TKmvd-0001FW-Q9 for submit@debbugs.gnu.org; Sun, 07 Oct 2012 05:13:41 -0400 Received: from mx.meyering.net ([88.168.87.75]:40008) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TKmvc-0001FQ-Io for 12366-done@debbugs.gnu.org; Sun, 07 Oct 2012 05:13:40 -0400 Received: from hx.meyering.net (hx.meyering.net [192.168.0.33]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id 0079E600DB; Sun, 7 Oct 2012 11:13:18 +0200 (CEST) From: Jim Meyering To: 12366-done@debbugs.gnu.org Subject: Re: bug#12366: [gnu-prog-discuss] bug#12366: Writing unwritable files References: <5048933A.7020105@gnu.org> <5048CB48.2060203@cs.ucla.edu> <5048CE3D.8060903@gnu.org> <5048D0EA.1050407@cs.ucla.edu> <5048D196.9070308@gnu.org> <5048D6B2.6090104@cs.ucla.edu> <20120906171225.GB19913@hysteria.proulx.com> <5048DC14.1060001@cs.ucla.edu> <504A22E8.1070608@gnu.org> <504A4F1F.3000802@cs.ucla.edu> <20120907204244.GA12635@hysteria.proulx.com> Date: Sun, 07 Oct 2012 11:13:18 +0200 In-Reply-To: <20120907204244.GA12635@hysteria.proulx.com> (Bob Proulx's message of "Fri, 7 Sep 2012 14:42:44 -0600") Message-ID: <871uhahcgh.fsf@hx.meyering.net> Lines: 4 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12366-done Cc: gnu-prog-discuss@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) This has been beaten to death, and is IMHO not indicative of a coreutils problem, so I've marked this issue as "done". If anyone wants to add a TODO item or something like that, please be precise and reopen the issue. From unknown Sat Jun 21 10:39:37 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sun, 04 Nov 2012 12:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator