From debbugs-submit-bounces@debbugs.gnu.org Mon Feb 22 11:01:11 2016 Received: (at submit) by debbugs.gnu.org; 22 Feb 2016 16:01:11 +0000 Received: from localhost ([127.0.0.1]:38237 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aXsv8-0005qz-E7 for submit@debbugs.gnu.org; Mon, 22 Feb 2016 11:01:11 -0500 Received: from eggs.gnu.org ([208.118.235.92]:55441) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aXmJe-0007J9-Ja for submit@debbugs.gnu.org; Mon, 22 Feb 2016 03:58:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aXmJY-00022A-B7 for submit@debbugs.gnu.org; Mon, 22 Feb 2016 03:57:57 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: **** X-Spam-Status: No, score=4.4 required=5.0 tests=BAYES_50,FREEMAIL_FROM, HTML_MESSAGE,RECEIVED_FROM_WINDOWS_HOST,URI_HEX autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:56164) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXmJY-000226-7c for submit@debbugs.gnu.org; Mon, 22 Feb 2016 03:57:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46410) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXmJX-0006Dr-5T for bug-gzip@gnu.org; Mon, 22 Feb 2016 03:57:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aXmJT-00021d-Tu for bug-gzip@gnu.org; Mon, 22 Feb 2016 03:57:55 -0500 Received: from blu004-omc3s25.hotmail.com ([65.55.116.100]:64714) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXmJT-00021Z-P0 for bug-gzip@gnu.org; Mon, 22 Feb 2016 03:57:51 -0500 Received: from BLU437-SMTP20 ([65.55.116.74]) by BLU004-OMC3S25.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Mon, 22 Feb 2016 00:57:50 -0800 X-TMN: [ZCLqXNxaqdG5N2L0bOSZaKNGBBQszv/k] X-Originating-Email: [jiangyy@outlook.com] Message-ID: From: Yanyan Jiang Content-Type: multipart/alternative; boundary="Apple-Mail=_09F57D67-1B14-4C94-9EAA-C87343C99813" Subject: Crash safety Date: Mon, 22 Feb 2016 03:57:47 -0500 To: bug-gzip@gnu.org MIME-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) X-Mailer: Apple Mail (2.3112) X-OriginalArrivalTime: 22 Feb 2016 08:57:49.0243 (UTC) FILETIME=[1AF2F8B0:01D16D4F] X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -2.7 (--) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Mon, 22 Feb 2016 11:01:09 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.7 (--) --Apple-Mail=_09F57D67-1B14-4C94-9EAA-C87343C99813 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hi gzip developers, Gzip version: 1.6 I am developing a tool to validate crash safety of application software. = I have just found that the file deletion has a potential safety = venerability: if only a prefix of I/O operations are flushed to disk, = after reboot, the file-system would only contain a 0-byte file (the data = is not reached to disk yet). A paper FYI: = http://0b4af6cdc2f0c5998459-c0245c5c937c5dedcca3f1764ecc9b2f.r43.cf2.rackc= dn.com/17780-osdi14-paper-pillai.pdf = (Table 1 on Page 440). Data = append can be (virtually) reordered with any operation at default ext3 = and ext4 settings. I recommend to use fsync() to persist the .gz file = before deletion. =E2=80=94 strace log =E2=80=94 36 open("a", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_NOFOLLOW) =3D 3 37 fstat64(3, {st_mode=3DS_IFREG|0664, st_size=3D19730, ...}) =3D 0 38 rt_sigprocmask(SIG_BLOCK, [HUP INT PIPE TERM XCPU XFSZ], [], 8) =3D = 0 39 open("a.gz", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0600) =3D 4 40 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) =3D 0 41 read(3, "10017649652034232324895361757801"..., 65536) =3D 19730 42 read(3, "", 45806) =3D 0 43 write(4, = "\37\213\10\10\24\312\312V\0\3a\0-\334m\226\234\274\22\3\340\377Y\r = \30l\354\375o,z\324"..., 9954) =3D 9954 44 close(3) =3D 0 45 utimensat(4, NULL, {{1456130580, 76955623}, {1456130580, = 128955620}}, 0) =3D 0 46 fchown32(4, 1000, 1000) =3D 0 47 fchmod(4, 0664) =3D 0 48 close(4) =3D 0 49 rt_sigprocmask(SIG_BLOCK, [HUP INT PIPE TERM XCPU XFSZ], [], 8) =3D = 0 50 unlink("a") =3D 0 Thank you for your attention! Regards, Yanyan Jiang =E8=92=8B=E7=82=8E=E5=B2=A9 Institute of Computer Software, Dept. of Computer Science, Nanjing University --Apple-Mail=_09F57D67-1B14-4C94-9EAA-C87343C99813 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"
Hi gzip developers,

Gzip version: 1.6

I am developing a tool = to validate crash safety of application software. I have just found that = the file deletion has a potential safety venerability: if only a prefix = of I/O operations are flushed to disk, after reboot, the file-system = would only contain a 0-byte file (the data is not reached to disk = yet).

A paper = FYI: http://0b4af6cdc2f0c5998459-c0245c5c937c5dedcca3f1764ecc9b2f.r4= 3.cf2.rackcdn.com/17780-osdi14-paper-pillai.pdf (Table 1 on = Page 440). Data append can be (virtually) reordered with any operation = at default ext3 and ext4 settings. I recommend to use fsync() to persist = the .gz file before deletion.

=E2=80=94 strace log =E2=80=94

 36 open("a", = O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_NOFOLLOW) =3D 3
 37 fstat64(3, {st_mode=3DS_IFREG|0664, st_size=3D19730, = ...}) =3D 0
 38 rt_sigprocmask(SIG_BLOCK, [HUP = INT PIPE TERM XCPU XFSZ], [], 8) =3D 0
 39 = open("a.gz", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0600) =3D 4
 40 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) =3D = 0
 41 read(3, = "10017649652034232324895361757801"..., 65536) =3D 19730
 42 read(3, "", 45806)         =              =3D 0
 43 write(4, = "\37\213\10\10\24\312\312V\0\3a\0-\334m\226\234\274\22\3\340\377Y\r =    \30l\354\375o,z\324"..., 9954) =3D 9954
 44 close(3)             =                    =3D = 0
 45 utimensat(4, NULL, {{1456130580, = 76955623}, {1456130580, 128955620}}, 0) =3D 0
 46 fchown32(4, 1000, 1000)         =         =3D 0
 47 = fchmod(4, 0664)                 =         =3D 0
 48 close(4) =                     =            =3D 0
 49 rt_sigprocmask(SIG_BLOCK, [HUP INT PIPE TERM XCPU = XFSZ], [], 8) =3D 0
 50 unlink("a")   =                     =       =3D 0

Thank you for your attention!

Regards,
Yanyan = Jiang =E8=92=8B=E7=82=8E=E5=B2=A9
Institute of = Computer Software,
Dept. of Computer Science, = Nanjing University

= --Apple-Mail=_09F57D67-1B14-4C94-9EAA-C87343C99813-- From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 23 02:16:15 2016 Received: (at control) by debbugs.gnu.org; 23 Feb 2016 07:16:15 +0000 Received: from localhost ([127.0.0.1]:38901 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aY7Cg-0002jH-UN for submit@debbugs.gnu.org; Tue, 23 Feb 2016 02:16:15 -0500 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:33336) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aY7Ce-0002iz-Jb for control@debbugs.gnu.org; Tue, 23 Feb 2016 02:16:13 -0500 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id A767916083D for ; Mon, 22 Feb 2016 23:16:06 -0800 (PST) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id qJkGIjXgVW4e for ; Mon, 22 Feb 2016 23:16:06 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 00331160FD3 for ; Mon, 22 Feb 2016 23:16:05 -0800 (PST) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id sBcwqPeqEETY for ; Mon, 22 Feb 2016 23:16:05 -0800 (PST) Received: from [192.168.1.9] (pool-100-32-155-148.lsanca.fios.verizon.net [100.32.155.148]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id C6B1016083D for ; Mon, 22 Feb 2016 23:16:05 -0800 (PST) To: control@debbugs.gnu.org From: Paul Eggert Subject: 22768 and 22770 are duplicates Organization: UCLA Computer Science Department Message-ID: <56CC0735.1030503@cs.ucla.edu> Date: Mon, 22 Feb 2016 23:16:05 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) merge 22768 22770 From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 23 02:28:17 2016 Received: (at 22768-done) by debbugs.gnu.org; 23 Feb 2016 07:28:17 +0000 Received: from localhost ([127.0.0.1]:38911 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aY7OK-00031m-NS for submit@debbugs.gnu.org; Tue, 23 Feb 2016 02:28:16 -0500 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:33673) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aY7OJ-00031Y-5V for 22768-done@debbugs.gnu.org; Tue, 23 Feb 2016 02:28:15 -0500 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id D909516083D; Mon, 22 Feb 2016 23:28:08 -0800 (PST) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id NM20hYKG5nS7; Mon, 22 Feb 2016 23:28:07 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id B54CA160FD3; Mon, 22 Feb 2016 23:28:07 -0800 (PST) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id LdSPYeQoouK4; Mon, 22 Feb 2016 23:28:07 -0800 (PST) Received: from [192.168.1.9] (pool-100-32-155-148.lsanca.fios.verizon.net [100.32.155.148]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 93D4F16083D; Mon, 22 Feb 2016 23:28:07 -0800 (PST) Subject: Re: bug#22768: Crash safety To: Yanyan Jiang , 22768-done@debbugs.gnu.org References: From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <56CC0A07.1080403@cs.ucla.edu> Date: Mon, 22 Feb 2016 23:28:07 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/mixed; boundary="------------060100060409040907000508" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) This is a multi-part message in MIME format. --------------060100060409040907000508 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Thanks for reporting the problem. It's annoying that gzip must invoke fsync, as that's way overkill compared to the write-ordering that is needed and fsync will slow gzip down, but I don't see any safe and reasonably portable alternative so I installed the attached patch on Savannah, here: http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=22aac8f8a616a72dbbe0e4119db8ddda0f076c04 --------------060100060409040907000508 Content-Type: text/plain; charset=UTF-8; name="0001-fsync-output-file-before-closing.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="0001-fsync-output-file-before-closing.txt" RnJvbSBjODM2N2Q3NjYwMjM5YjFhZWRjZTc4NTBjNzY2YzBjMGY4OTM4ZDFjIE1vbiBTZXAg MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBQYXVsIEVnZ2VydCA8ZWdnZXJ0QGNzLnVjbGEuZWR1 PgpEYXRlOiBNb24sIDIyIEZlYiAyMDE2IDIzOjIxOjQ5IC0wODAwClN1YmplY3Q6IFtQQVRD SF0gZnN5bmMgb3V0cHV0IGZpbGUgYmVmb3JlIGNsb3NpbmcKTUlNRS1WZXJzaW9uOiAxLjAK Q29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFyc2V0PVVURi04CkNvbnRlbnQtVHJhbnNm ZXItRW5jb2Rpbmc6IDhiaXQKClByb2JsZW0gcmVwb3J0ZWQgYnkgWWFueWFuIEppYW5nIOiS i+eCjuWyqSBpbjogaHR0cDovL2J1Z3MuZ251Lm9yZy8yMjc2OAoqIE5FV1M6IERvY3VtZW50 IHRoaXMuCiogYm9vdHN0cmFwLmNvbmYgKGdudWxpYl9tb2R1bGVzKTogQWRkIGZzeW5jLgoq IGd6aXAuYyAodHJlYXRfZmlsZSk6IENhbGwgZnN5bmMganVzdCBiZWZvcmUgY2xvc2luZyB0 aGUgb3V0cHV0LgoqIGxpYi8uZ2l0aWdub3JlLCBtNC8uZ2l0aWdub3JlOiBBZGQgZnN5bmMt cmVsYXRlZCBnbnVsaWIgZmlsZXMuCi0tLQogTkVXUyAgICAgICAgICAgfCAgNCArKysrCiBi b290c3RyYXAuY29uZiB8ICAxICsKIGd6aXAuYyAgICAgICAgIHwgMTEgKysrKysrKysrKy0K IGxpYi8uZ2l0aWdub3JlIHwgIDEgKwogbTQvLmdpdGlnbm9yZSAgfCAgMSArCiA1IGZpbGVz IGNoYW5nZWQsIDE3IGluc2VydGlvbnMoKyksIDEgZGVsZXRpb24oLSkKCmRpZmYgLS1naXQg YS9ORVdTIGIvTkVXUwppbmRleCBhMWM2NjhmLi4zMTQ3MmNjIDEwMDY0NAotLS0gYS9ORVdT CisrKyBiL05FV1MKQEAgLTQsNiArNCwxMCBAQCBHTlUgZ3ppcCBORVdTICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgLSotIG91dGxpbmUgLSotCiAKICoqIENoYW5nZXMg aW4gYmVoYXZpb3IKIAorICBXaGVuIGFjdGluZyBpbi1wbGFjZSwgZ3ppcCBub3cgZnN5bmNz IHRoZSBvdXRwdXQgYmVmb3JlIGNsb3NpbmcgaXQuCisgIFRoaXMgaXMgc2xvd2VyLCBidXQg b24gbWFueSBmaWxlIHN5c3RlbXMgaXQgaXMgc2FmZXIgaWYgdGhlIHN5c3RlbQorICBpcyBh Ym91dCB0byBjcmFzaC4KKwogICBUaGUgR1pJUCBlbnZpcm9ubWVudCB2YXJpYWJsZSBpcyBu b3cgb2Jzb2xlc2NlbnQ7IGd6aXAgbm93IHdhcm5zIGlmCiAgIGl0IGlzIHVzZWQsIGFuZCBy ZWplY3RzIGF0dGVtcHRzIHRvIHVzZSBkYW5nZXJvdXMgb3B0aW9ucyBvciBvcGVyYW5kcy4K ICAgWW91IGNhbiB1c2UgYW4gYWxpYXMgb3Igc2NyaXB0IGluc3RlYWQuCmRpZmYgLS1naXQg YS9ib290c3RyYXAuY29uZiBiL2Jvb3RzdHJhcC5jb25mCmluZGV4IDEzYTQ4NWQuLmIxNWNh YTMgMTAwNjQ0Ci0tLSBhL2Jvb3RzdHJhcC5jb25mCisrKyBiL2Jvb3RzdHJhcC5jb25mCkBA IC0zMSw2ICszMSw3IEBAIGZjbnRsLXNhZmVyCiBmZGwKIGZkb3BlbmRpcgogZnByaW50Zi1w b3NpeAorZnN5bmMKIGdldG9wdC1nbnUKIGdpdC12ZXJzaW9uLWdlbgogZ2l0bG9nLXRvLWNo YW5nZWxvZwpkaWZmIC0tZ2l0IGEvZ3ppcC5jIGIvZ3ppcC5jCmluZGV4IGEwMTM1NDAuLmI4 NzIzODMgMTAwNjQ0Ci0tLSBhL2d6aXAuYworKysgYi9nemlwLmMKQEAgLTkyNiw4ICs5MjYs MTcgQEAgbG9jYWwgdm9pZCB0cmVhdF9maWxlKGluYW1lKQogCiAgICAgaWYgKCF0b19zdGRv dXQpCiAgICAgICB7Ci0KICAgICAgICAgY29weV9zdGF0ICgmaXN0YXQpOworCisgICAgICAg IC8qIFRyYW5zZmVyIG91dHB1dCBkYXRhIHRvIHRoZSBvdXRwdXQgZmlsZSdzIHN0b3JhZ2Ug ZGV2aWNlLgorICAgICAgICAgICBPdGhlcndpc2UsIGlmIHRoZSBzeXN0ZW0gY3Jhc2hlZCBu b3cgdGhlIHVzZXIgbWlnaHQgbG9zZQorICAgICAgICAgICBib3RoIGlucHV0IGFuZCBvdXRw dXQgZGF0YS4gIFNlZTogUGlsbGFpIFRTIGV0IGFsLiAgQWxsCisgICAgICAgICAgIGZpbGUg c3lzdGVtcyBhcmUgbm90IGNyZWF0ZWQgZXF1YWw6IG9uIHRoZSBjb21wbGV4aXR5IG9mCisg ICAgICAgICAgIGNyYWZ0aW5nIGNyYXNoLWNvbnNpc3RlbnQgYXBwbGljYXRpb25zLiBPU0RJ JzE0LiAyMDE0OjQzMy00OC4KKyAgICAgICAgICAgaHR0cHM6Ly93d3cudXNlbml4Lm9yZy9j b25mZXJlbmNlL29zZGkxNC90ZWNobmljYWwtc2Vzc2lvbnMvcHJlc2VudGF0aW9uL3BpbGxh aSAgKi8KKyAgICAgICAgaWYgKCFrZWVwICYmIGZzeW5jIChvZmQpICE9IDAgJiYgZXJybm8g IT0gRUlOVkFMKQorICAgICAgICAgIHdyaXRlX2Vycm9yICgpOworCiAgICAgICAgIGlmIChj bG9zZSAob2ZkKSAhPSAwKQogICAgICAgICAgIHdyaXRlX2Vycm9yICgpOwogCmRpZmYgLS1n aXQgYS9saWIvLmdpdGlnbm9yZSBiL2xpYi8uZ2l0aWdub3JlCmluZGV4IDgxZDk0ZmYuLmEz NjhhMjYgMTAwNjQ0Ci0tLSBhL2xpYi8uZ2l0aWdub3JlCisrKyBiL2xpYi8uZ2l0aWdub3Jl CkBAIC0yMjUsMyArMjI1LDQgQEAKIC94c2l6ZS5oCiAveWVzbm8uYwogL3llc25vLmgKKy9m c3luYy5jCmRpZmYgLS1naXQgYS9tNC8uZ2l0aWdub3JlIGIvbTQvLmdpdGlnbm9yZQppbmRl eCA2NjBiOTI2Li4zMmY1NTY2IDEwMDY0NAotLS0gYS9tNC8uZ2l0aWdub3JlCisrKyBiL200 Ly5naXRpZ25vcmUKQEAgLTE1MCwzICsxNTAsNCBAQAogL3hhbGxvYy5tNAogL3hzaXplLm00 CiAveWVzbm8ubTQKKy9mc3luYy5tNAotLSAKMi41LjAKCg== --------------060100060409040907000508-- From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 26 06:29:18 2016 Received: (at 22768) by debbugs.gnu.org; 26 Feb 2016 11:29:18 +0000 Received: from localhost ([127.0.0.1]:46965 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZGaE-0005jT-0p for submit@debbugs.gnu.org; Fri, 26 Feb 2016 06:29:18 -0500 Received: from eggs.gnu.org ([208.118.235.92]:44690) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZGaB-0005jF-Jt for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 06:29:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aZGa1-0006CJ-Fu for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 06:29:10 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:39124) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZGa1-0006CF-C4; Fri, 26 Feb 2016 06:29:05 -0500 Received: from 231.red-217-124-67.dialup.dynamic.ccgg.telefonica.net ([217.124.67.231]:39067) by fencepost.gnu.org with esmtpa (Exim 4.82) (envelope-from ) id 1aZGZz-0008L2-Ni; Fri, 26 Feb 2016 06:29:05 -0500 Message-ID: <56D03848.5060108@gnu.org> Date: Fri, 26 Feb 2016 12:34:32 +0100 From: Antonio Diaz Diaz User-Agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.8.1.4) Gecko/20070601 SeaMonkey/1.1.2 MIME-Version: 1.0 To: 22768@debbugs.gnu.org, Yanyan Jiang Subject: Re: bug#22768: Crash safety References: <56CC0A07.1080403@cs.ucla.edu> In-Reply-To: <56CC0A07.1080403@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Paul Eggert wrote: > Thanks for reporting the problem. It's annoying that gzip must invoke > fsync, as that's way overkill compared to the write-ordering that is > needed and fsync will slow gzip down, but I don't see any safe and > reasonably portable alternative I 100% agree. I am considering a different approach for lzip; adding a new option, say '-y, --fsync', to call fsync when acting in-place. This provides safety to those needing it without slowing down the work of everybody else. Especially of those (de)compressing a lot of replaceable files in-place. (Ddrescue provides the option '-y, --synchronous' because calling fsync by default is too slow). What do you think? From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 26 06:53:13 2016 Received: (at 22768) by debbugs.gnu.org; 26 Feb 2016 11:53:13 +0000 Received: from localhost ([127.0.0.1]:46983 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZGxN-0006Lz-J0 for submit@debbugs.gnu.org; Fri, 26 Feb 2016 06:53:13 -0500 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:37881) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZGxM-0006Li-6U for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 06:53:12 -0500 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 6F153160FD0; Fri, 26 Feb 2016 03:53:05 -0800 (PST) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id WOjbe8sXMgCG; Fri, 26 Feb 2016 03:53:04 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id BF481160FD5; Fri, 26 Feb 2016 03:53:04 -0800 (PST) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id u1F44oGyYpV2; Fri, 26 Feb 2016 03:53:04 -0800 (PST) Received: from [192.168.1.9] (pool-100-32-155-148.lsanca.fios.verizon.net [100.32.155.148]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 9C740160FD0; Fri, 26 Feb 2016 03:53:04 -0800 (PST) Subject: Re: bug#22768: Crash safety To: Antonio Diaz Diaz , 22768@debbugs.gnu.org, Yanyan Jiang References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <56D03CA0.2020100@cs.ucla.edu> Date: Fri, 26 Feb 2016 03:53:04 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <56D03848.5060108@gnu.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Antonio Diaz Diaz wrote: > I am considering a different approach for lzip; adding a new option, say '-y, > --fsync', to call fsync when acting in-place. This provides safety to those > needing it without slowing down the work of everybody else. Especially of those > (de)compressing a lot of replaceable files in-place. Yes, I considered an --fsync option as well, but worried about making the default unsafe. How about if gzip and lzip instead add a --no-fsync option for people who don't need the safety? From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 26 07:29:42 2016 Received: (at 22768) by debbugs.gnu.org; 26 Feb 2016 12:29:42 +0000 Received: from localhost ([127.0.0.1]:46989 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZHWg-0007DZ-HL for submit@debbugs.gnu.org; Fri, 26 Feb 2016 07:29:42 -0500 Received: from eggs.gnu.org ([208.118.235.92]:59769) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZHWe-0007DL-IQ for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 07:29:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aZHWV-0007mN-Cp for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 07:29:35 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:43215) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZHWV-0007mI-9C; Fri, 26 Feb 2016 07:29:31 -0500 Received: from 231.red-217-124-67.dialup.dynamic.ccgg.telefonica.net ([217.124.67.231]:53059) by fencepost.gnu.org with esmtpa (Exim 4.82) (envelope-from ) id 1aZHWU-0003vJ-B4; Fri, 26 Feb 2016 07:29:31 -0500 Message-ID: <56D04673.2070405@gnu.org> Date: Fri, 26 Feb 2016 13:34:59 +0100 From: Antonio Diaz Diaz User-Agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.8.1.4) Gecko/20070601 SeaMonkey/1.1.2 MIME-Version: 1.0 To: 22768@debbugs.gnu.org Subject: Re: bug#22768: Crash safety References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> In-Reply-To: <56D03CA0.2020100@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Paul Eggert wrote: > Yes, I considered an --fsync option as well, but worried about making > the default unsafe. How about if gzip and lzip instead add a --no-fsync > option for people who don't need the safety? I think it is a good idea. The people who know that don't need the safety are probably the most inclined to read the documentation. Is it ok to use '-y' as short option name? From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 26 16:28:27 2016 Received: (at 22768) by debbugs.gnu.org; 26 Feb 2016 21:28:27 +0000 Received: from localhost ([127.0.0.1]:47922 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZPw3-0008HR-3P for submit@debbugs.gnu.org; Fri, 26 Feb 2016 16:28:27 -0500 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:34012) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZPw1-0008HF-N6 for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 16:28:26 -0500 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id C937B160FD0; Fri, 26 Feb 2016 13:28:18 -0800 (PST) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id SD4tnJ8V2QYL; Fri, 26 Feb 2016 13:28:18 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 20EEE160FD5; Fri, 26 Feb 2016 13:28:18 -0800 (PST) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 9JdMI4269SdU; Fri, 26 Feb 2016 13:28:18 -0800 (PST) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 0804D160FD0; Fri, 26 Feb 2016 13:28:18 -0800 (PST) Subject: Re: bug#22768: Crash safety To: Antonio Diaz Diaz References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <56D0C371.8030408@cs.ucla.edu> Date: Fri, 26 Feb 2016 13:28:17 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56D04673.2070405@gnu.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang , 22768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) On 02/26/2016 04:34 AM, Antonio Diaz Diaz wrote: > Is it ok to use '-y' as short option name? Why -y? And why a short name at all? I don't expect this to be something that people will want to type by hand. Come to think of it, gzip should also do an fsync, or at least an fdatasync, on the destination's directory before removing the source. And this suggests that any long option name shouldn't be something syscall-specific like '--no-fsync', but should instead be something more general and easy to remember, e.g., '--hasty'. From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 26 19:12:07 2016 Received: (at 22768) by debbugs.gnu.org; 27 Feb 2016 00:12:07 +0000 Received: from localhost ([127.0.0.1]:48004 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZSUQ-0003bH-WF for submit@debbugs.gnu.org; Fri, 26 Feb 2016 19:12:07 -0500 Received: from eggs.gnu.org ([208.118.235.92]:58506) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZSUP-0003an-M7 for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 19:12:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aZSUF-0006Za-D3 for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 19:12:00 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:60680) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZSUF-0006ZN-9K; Fri, 26 Feb 2016 19:11:55 -0500 Received: from 208.red-217-124-67.dialup.dynamic.ccgg.telefonica.net ([217.124.67.208]:59077) by fencepost.gnu.org with esmtpa (Exim 4.82) (envelope-from ) id 1aZSUD-0000Uy-QR; Fri, 26 Feb 2016 19:11:54 -0500 Message-ID: <56D0EB15.3060504@gnu.org> Date: Sat, 27 Feb 2016 01:17:25 +0100 From: Antonio Diaz Diaz User-Agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.8.1.4) Gecko/20070601 SeaMonkey/1.1.2 MIME-Version: 1.0 To: 22768@debbugs.gnu.org Subject: Re: bug#22768: Crash safety References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> In-Reply-To: <56D0C371.8030408@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Paul Eggert wrote: > Why -y? And why a short name at all? I don't expect this to be > something that people will want to type by hand. Because ddrescue already provides the option '-y, --synchronous' for a somewhat similar functionality. (It is in my first message). I find short option names handy, but I have no problem implementing this as long-only. > Come to think of it, gzip should also do an fsync, or at least an > fdatasync, on the destination's directory before removing the source. Doing it right in all circumstances may be impossible. Also fsync may be a no-op in some systems and very expensive in others. And some systems are safe without the need of fsync. This is why my first idea was to leave the safety to the system by default and add an option to enable the "maybe safer but slower" behavior. > And this suggests that any long option name shouldn't be something > syscall-specific like '--no-fsync', but should instead be something more > general and easy to remember, e.g., '--hasty'. I had to search 'hasty' in the dictionary, so I think it is perhaps not so good and easy to remember for non-English speakers. OTOH, many people using a CLI know what 'fsync' or 'sync' mean. Just now my preference is to make the behavior optional and call the option --fsync. I think both points meet the principle of least surprise. Best regards, Antonio. From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 26 19:45:33 2016 Received: (at 22768) by debbugs.gnu.org; 27 Feb 2016 00:45:34 +0000 Received: from localhost ([127.0.0.1]:48009 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZT0n-0004L5-OS for submit@debbugs.gnu.org; Fri, 26 Feb 2016 19:45:33 -0500 Received: from eggs.gnu.org ([208.118.235.92]:35912) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZT0m-0004Ks-Ih for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 19:45:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aZT0e-00082i-0A for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 19:45:27 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:32779) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZT0d-00082e-Sw; Fri, 26 Feb 2016 19:45:23 -0500 Received: from 208.red-217-124-67.dialup.dynamic.ccgg.telefonica.net ([217.124.67.208]:34635) by fencepost.gnu.org with esmtpa (Exim 4.82) (envelope-from ) id 1aZT0c-0003vY-R9; Fri, 26 Feb 2016 19:45:23 -0500 Message-ID: <56D0F2EE.5010000@gnu.org> Date: Sat, 27 Feb 2016 01:50:54 +0100 From: Antonio Diaz Diaz User-Agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.8.1.4) Gecko/20070601 SeaMonkey/1.1.2 MIME-Version: 1.0 To: 22768@debbugs.gnu.org Subject: Re: bug#22768: Crash safety References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> In-Reply-To: <56D0EB15.3060504@gnu.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Antonio Diaz Diaz wrote: > Just now my preference is to make the behavior optional and call the > option --fsync. I think both points meet the principle of least surprise. An additional reason to make the behavior optional is that people find the performance penalty of fsync so annoying that they even write libraries to disable it[1]. "This package contains a small LD_PRELOAD library (libeatmydata) and a couple of helper utilities designed to transparently disable fsync and friends (like open(O_SYNC)). This has two side-effects: making software that writes data safely to disk a lot quicker and making this software no longer crash safe." [1] http://packages.debian.org/testing/utils/eatmydata Best regards, Antonio. From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 26 19:47:02 2016 Received: (at 22768) by debbugs.gnu.org; 27 Feb 2016 00:47:02 +0000 Received: from localhost ([127.0.0.1]:48013 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZT2E-0004Ne-2h for submit@debbugs.gnu.org; Fri, 26 Feb 2016 19:47:02 -0500 Received: from havoc.proulx.com ([96.88.95.61]:56533) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZT2C-0004NF-7e for 22768@debbugs.gnu.org; Fri, 26 Feb 2016 19:47:00 -0500 Received: from joseki.proulx.com (localhost [127.0.0.1]) by havoc.proulx.com (Postfix) with ESMTP id 5BCF81031; Fri, 26 Feb 2016 17:46:59 -0700 (MST) Received: from hysteria.proulx.com (hysteria.proulx.com [192.168.230.119]) by joseki.proulx.com (Postfix) with ESMTP id E93472185F; Fri, 26 Feb 2016 17:46:58 -0700 (MST) Received: by hysteria.proulx.com (Postfix, from userid 1000) id BBA232DC5B; Fri, 26 Feb 2016 17:46:58 -0700 (MST) Date: Fri, 26 Feb 2016 17:46:58 -0700 From: Bob Proulx To: 22768@debbugs.gnu.org Subject: Re: bug#22768: Crash safety Message-ID: <20160226173235198655396@bob.proulx.com> References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56D0EB15.3060504@gnu.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang , Antonio Diaz Diaz , Paul Eggert X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Antonio Diaz Diaz wrote: > Paul Eggert wrote: > >And this suggests that any long option name shouldn't be something > >syscall-specific like '--no-fsync', but should instead be something more > >general and easy to remember, e.g., '--hasty'. > > I had to search 'hasty' in the dictionary, so I think it is perhaps not so > good and easy to remember for non-English speakers. OTOH, many people using > a CLI know what 'fsync' or 'sync' mean. Worse is that one program chooses "hasty". Another chooses "quick". Another "hurried", "fast", "rapid", "swift". It becomes impossible to remember what each program uses. Keeping to what it does seems least surprising and in this case it is either --fsync or --sync. > Just now my preference is to make the behavior optional and call the option > --fsync. I think both points meet the principle of least surprise. I would much prefer the above of an option to enable it rather than one to disable it. Otherwise I have to go through workarounds to avoid it in order to have the performance that used to be the default. It has been many decades with the cached behavior and apparently without significant issues due to it. Large changes should be made slowly as an option rather than abruptly as the default. Bob From debbugs-submit-bounces@debbugs.gnu.org Sat Feb 27 10:41:43 2016 Received: (at 22768) by debbugs.gnu.org; 27 Feb 2016 15:41:43 +0000 Received: from localhost ([127.0.0.1]:49013 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZh03-0007M9-6h for submit@debbugs.gnu.org; Sat, 27 Feb 2016 10:41:43 -0500 Received: from eggs.gnu.org ([208.118.235.92]:36211) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZh01-0007Lx-Vu for 22768@debbugs.gnu.org; Sat, 27 Feb 2016 10:41:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aZgzs-0000Z1-Ot for 22768@debbugs.gnu.org; Sat, 27 Feb 2016 10:41:36 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=BAYES_50,MISSING_HEADERS, RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:46606) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZgzs-0000Yq-E1; Sat, 27 Feb 2016 10:41:32 -0500 Received: from 184.red-217-124-67.dialup.dynamic.ccgg.telefonica.net ([217.124.67.184]:51060) by fencepost.gnu.org with esmtpa (Exim 4.82) (envelope-from ) id 1aZgzr-0003Kt-71; Sat, 27 Feb 2016 10:41:32 -0500 Message-ID: <56D1C4FA.6000704@gnu.org> Date: Sat, 27 Feb 2016 16:47:06 +0100 From: Antonio Diaz Diaz User-Agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.8.1.4) Gecko/20070601 SeaMonkey/1.1.2 MIME-Version: 1.0 Subject: Re: bug#22768: Crash safety References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> <20160226173235198655396@bob.proulx.com> In-Reply-To: <20160226173235198655396@bob.proulx.com> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang , 22768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.8 (---) Bob Proulx wrote: >> Just now my preference is to make the behavior optional and call the option >> --fsync. I think both points meet the principle of least surprise. > > I would much prefer the above of an option to enable it rather than > one to disable it. Otherwise I have to go through workarounds to > avoid it in order to have the performance that used to be the default. After thinking about it, I think that the right thing to do is to not implement any kind of fsync functionality in gzip/lzip. First, it may be a cause of feature creep. If gzip fsyncs the output file it might also test it, or even compare it with the input file, before deleting the input file. Second, as doing it right in all circumstances may be impossible, it may become an endless source of bug reports. (fsyncing also the destination's directory, opening the output with O_DIRECT,...). Third, it fights against other layers of the system, like the filesystem, instead of collaborating with them. Fourth, it fights against user's wishes instead of obeying them. If the user chooses a fast-but-unsafe configuration for the filesystem, gzip should not try to circumvent the user's choice, because gzip does not know if the file being compressed is worth the trouble or not. I think that the best way of guarding an important file against all bugs and crashes is a extended version of the procedure already documented in the manual of lzip: 1) gzip --keep file # don't delete input 2) sync # commit output and directory to disk 3) zcmp file file.gz # verify output 4) rm file # then remove input Best regards, Antonio. From debbugs-submit-bounces@debbugs.gnu.org Sun Feb 28 03:27:23 2016 Received: (at 22768) by debbugs.gnu.org; 28 Feb 2016 08:27:23 +0000 Received: from localhost ([127.0.0.1]:49496 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZwhA-0001rw-VW for submit@debbugs.gnu.org; Sun, 28 Feb 2016 03:27:23 -0500 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:57260) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZwh1-0001rb-5s for 22768@debbugs.gnu.org; Sun, 28 Feb 2016 03:27:15 -0500 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 6C086160FD5; Sun, 28 Feb 2016 00:27:01 -0800 (PST) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id HJI1spvvwgdl; Sun, 28 Feb 2016 00:26:58 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 97A44160FD0; Sun, 28 Feb 2016 00:26:58 -0800 (PST) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id k3DFYAvy1SgH; Sun, 28 Feb 2016 00:26:58 -0800 (PST) Received: from [192.168.1.9] (pool-100-32-155-148.lsanca.fios.verizon.net [100.32.155.148]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 62353160FD5; Sun, 28 Feb 2016 00:26:58 -0800 (PST) Subject: Re: bug#22768: Crash safety To: Antonio Diaz Diaz , 22768@debbugs.gnu.org References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <56D2AF52.9080702@cs.ucla.edu> Date: Sun, 28 Feb 2016 00:26:58 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <56D0EB15.3060504@gnu.org> Content-Type: multipart/mixed; boundary="------------030907090002050303050204" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) This is a multi-part message in MIME format. --------------030907090002050303050204 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Antonio Diaz Diaz wrote: > ddrescue already provides the option '-y, --synchronous' for a somewhat > similar functionality. OK, let's do it as --synchronous, long-only. If the need keeps growing we can add -y. > Just now my preference is to make the behavior optional On second thought, as Bob Proulx suggested, this is a better approach. I tried the a synchronous gzip on a contrived example (compressing 1000 empty files on an ext4 file system on an actual hard drive with options relatime, seclabel, data=ordered) and synchronizing made gzip 700x slower. Most people will prefer the old behavior, where gzip is faster and is unsafe mostly just in theory. I'm attaching the patches I installed recently in this area, to help fix this problem. I'll follow up on your other recent email in another message soon. --------------030907090002050303050204 Content-Type: text/x-diff; name="0001-gzip-fdatasync-output-dir-before-unlinking.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-gzip-fdatasync-output-dir-before-unlinking.patch" >From 8604a5f86e8d1d83594b722a6e401f2e853cafa2 Mon Sep 17 00:00:00 2001 From: Paul Eggert Date: Sat, 27 Feb 2016 14:12:46 -0800 Subject: [PATCH 1/4] gzip: fdatasync output dir before unlinking This follows up on the earlier patch to avoid data loss near the system crashes. See: http://bugs.gnu.org/22768 * bootstrap.conf (gnulib_modules): Add dirname-lgpl, fdatasync, openat-safer, unistd-safer, unlinkat. * gzip.c: Include stddef.h, dirname.h. Include fcntl--.h instead of fcntl-safer.h. (RW_USER): Remove; no longer needed. (dfname, dfd): New static vars. (dot): New static const. (atdir_eq, atdir_set): New functions. (treat_file): Also fdatasync the output directory, if !keep. (treat_file, create_outfile, open_and_stat): Use dir fd for unlinkat and openat, if possible. (open_and_stat): Omit mode argument, since it was always the same. All callers changed. * lib/.gitignore, m4/.gitignore: Add new gnulib files. * tailor.h (PROTO, NO_STDIN_FSTAT, OPEN): Remove. Remove MACOS section, as this stuff would not work anyway now, and circa 2001 Apple stopped supporting Mac OS 9 and earlier. * zip.c: Do not include unistd.h and fcntl.h, as this file does not directly use any symbols defined by those headers. --- bootstrap.conf | 5 +++ gzip.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++--------- lib/.gitignore | 10 +++++- m4/.gitignore | 7 +++- tailor.h | 21 ------------ zip.c | 3 -- 6 files changed, 109 insertions(+), 42 deletions(-) diff --git a/bootstrap.conf b/bootstrap.conf index b15caa3..308dc5e 100644 --- a/bootstrap.conf +++ b/bootstrap.conf @@ -25,10 +25,12 @@ announce-gen calloc close closein +dirname-lgpl fclose fcntl fcntl-safer fdl +fdatasync fdopendir fprintf-posix fsync @@ -47,6 +49,7 @@ lstat maintainer-makefile malloc-gnu manywarnings +openat-safer perror printf-posix readme-release @@ -55,6 +58,8 @@ savedir stat-time sys_stat time +unistd-safer +unlinkat update-copyright utimens xalloc diff --git a/gzip.c b/gzip.c index b872383..15bcf78 100644 --- a/gzip.c +++ b/gzip.c @@ -59,6 +59,7 @@ static char const *const license_msg[] = { #include #include #include +#include #include #include @@ -70,7 +71,8 @@ static char const *const license_msg[] = { #include "revision.h" #include "timespec.h" -#include "fcntl-safer.h" +#include "dirname.h" +#include "fcntl--.h" #include "getopt.h" #include "ignore-value.h" #include "stat-time.h" @@ -79,7 +81,6 @@ static char const *const license_msg[] = { /* configuration */ -#include #include #include #include @@ -97,8 +98,6 @@ static char const *const license_msg[] = { # include #endif -#define RW_USER (S_IRUSR | S_IWUSR) /* creation mode for open() */ - #ifndef MAX_PATH_LEN # define MAX_PATH_LEN 1024 /* max pathname length */ #endif @@ -205,9 +204,11 @@ static off_t total_in; /* input bytes for all files */ static off_t total_out; /* output bytes for all files */ char ifname[MAX_PATH_LEN]; /* input file name */ char ofname[MAX_PATH_LEN]; /* output file name */ +static char dfname[MAX_PATH_LEN]; /* name of dir containing output file */ static struct stat istat; /* status for input file */ int ifd; /* input file descriptor */ int ofd; /* output file descriptor */ +static int dfd = -1; /* output directory file descriptor */ unsigned insize; /* valid bytes in inbuf */ unsigned inptr; /* index of next byte to be processed in inbuf */ unsigned outcnt; /* bytes in output buffer */ @@ -769,6 +770,48 @@ local void treat_stdin() } } +static char const dot = '.'; + +/* True if the cached directory for calls to openat etc. is DIR, with + length DIRLEN. DIR need not be null-terminated. DIRLEN must be + less than MAX_PATH_LEN. */ +static bool +atdir_eq (char const *dir, ptrdiff_t dirlen) +{ + if (dirlen == 0) + dir = &dot, dirlen = 1; + return memcmp (dfname, dir, dirlen) == 0 && !dfname[dirlen]; +} + +/* Set the directory used for calls to openat etc. to be the directory + DIR, with length DIRLEN. DIR need not be null-terminated. + DIRLEN must be less than MAX_PATH_LEN. Return a file descriptor for + the directory, or -1 if one could not be obtained. */ +static int +atdir_set (char const *dir, ptrdiff_t dirlen) +{ + /* Don't bother opening directories on older systems that + lack openat and unlinkat. It's not worth the porting hassle. */ + #if HAVE_OPENAT && HAVE_UNLINKAT + enum { try_opening_directories = true }; + #else + enum { try_opening_directories = false }; + #endif + + if (try_opening_directories && ! atdir_eq (dir, dirlen)) + { + if (0 <= dfd) + close (dfd); + if (dirlen == 0) + dir = &dot, dirlen = 1; + memcpy (dfname, dir, dirlen); + dfname[dirlen] = '\0'; + dfd = open (dfname, O_SEARCH | O_DIRECTORY); + } + + return dfd; +} + /* ======================================================================== * Compress or decompress the given file */ @@ -928,26 +971,30 @@ local void treat_file(iname) { copy_stat (&istat); - /* Transfer output data to the output file's storage device. + /* If KEEP, transfer output data to the output file's storage device. Otherwise, if the system crashed now the user might lose both input and output data. See: Pillai TS et al. All file systems are not created equal: on the complexity of crafting crash-consistent applications. OSDI'14. 2014:433-48. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai */ - if (!keep && fsync (ofd) != 0 && errno != EINVAL) - write_error (); - - if (close (ofd) != 0) + if ((!keep + && ((0 <= dfd && fdatasync (dfd) != 0 && errno != EINVAL) + || (fsync (ofd) != 0 && errno != EINVAL))) + || close (ofd) != 0) write_error (); if (!keep) { sigset_t oldset; int unlink_errno; + char *ifbase = last_component (ifname); + int ufd = atdir_eq (ifname, ifbase - ifname) ? dfd : -1; + int res; sigprocmask (SIG_BLOCK, &caught_signals, &oldset); remove_ofname_fd = -1; - unlink_errno = xunlink (ifname) == 0 ? 0 : errno; + res = ufd < 0 ? xunlink (ifname) : unlinkat (ufd, ifbase, 0); + unlink_errno = res == 0 ? 0 : errno; sigprocmask (SIG_SETMASK, &oldset, NULL); if (unlink_errno) @@ -998,6 +1045,19 @@ local int create_outfile() int name_shortened = 0; int flags = (O_WRONLY | O_CREAT | O_EXCL | (ascii && decompress ? 0 : O_BINARY)); + char const *base = ofname; + int atfd = AT_FDCWD; + + if (!keep) + { + char const *b = last_component (ofname); + int f = atdir_set (ofname, b - ofname); + if (0 <= f) + { + base = b; + atfd = f; + } + } for (;;) { @@ -1005,7 +1065,7 @@ local int create_outfile() sigset_t oldset; sigprocmask (SIG_BLOCK, &caught_signals, &oldset); - remove_ofname_fd = ofd = OPEN (ofname, flags, RW_USER); + remove_ofname_fd = ofd = openat (atfd, base, flags, S_IRUSR | S_IWUSR); open_errno = errno; sigprocmask (SIG_SETMASK, &oldset, NULL); @@ -1115,13 +1175,15 @@ local char *get_suffix(name) } -/* Open file NAME with the given flags and mode and store its status +/* Open file NAME with the given flags and store its status into *ST. Return a file descriptor to the newly opened file, or -1 (setting errno) on failure. */ static int -open_and_stat (char *name, int flags, mode_t mode, struct stat *st) +open_and_stat (char *name, int flags, struct stat *st) { int fd; + int atfd = AT_FDCWD; + char const *base = name; /* Refuse to follow symbolic links unless -c or -f. */ if (!to_stdout && !force) @@ -1142,7 +1204,18 @@ open_and_stat (char *name, int flags, mode_t mode, struct stat *st) } } - fd = OPEN (name, flags, mode); + if (!keep) + { + char const *b = last_component (name); + int f = atdir_set (name, b - name); + if (0 <= f) + { + base = b; + atfd = f; + } + } + + fd = openat (atfd, base, flags); if (0 <= fd && fstat (fd, st) != 0) { int e = errno; @@ -1186,7 +1259,7 @@ open_input_file (iname, sbuf) strcpy(ifname, iname); /* If input file exists, return OK. */ - fd = open_and_stat (ifname, open_flags, RW_USER, sbuf); + fd = open_and_stat (ifname, open_flags, sbuf); if (0 <= fd) return fd; @@ -1227,7 +1300,7 @@ open_input_file (iname, sbuf) if (sizeof ifname <= ilen + strlen (s)) goto name_too_long; strcat(ifname, s); - fd = open_and_stat (ifname, open_flags, RW_USER, sbuf); + fd = open_and_stat (ifname, open_flags, sbuf); if (0 <= fd) return fd; if (errno != ENOENT) diff --git a/lib/.gitignore b/lib/.gitignore index a368a26..b26889f 100644 --- a/lib/.gitignore +++ b/lib/.gitignore @@ -5,6 +5,7 @@ /alloca.in.h /asnprintf.c /assure.h +/at-func.c /basename-lgpl.c /c-ctype.c /c-ctype.h @@ -59,6 +60,7 @@ /fd-hook.c /fd-hook.h /fd-safer.c +/fdatasync.c /fdopendir.c /fflush.c /filename.h @@ -84,6 +86,8 @@ /fseterr.c /fseterr.h /fstat.c +/fstatat.c +/fsync.c /ftell.c /ftello.c /getcwd-lgpl.c @@ -132,6 +136,7 @@ /openat-die.c /openat-priv.h /openat-proc.c +/openat-safer.c /openat.c /openat.h /opendir-safer.c @@ -157,6 +162,7 @@ /ref-add.sin /ref-del.sed /ref-del.sin +/rmdir.c /save-cwd.c /save-cwd.h /savedir.c @@ -169,6 +175,7 @@ /stat-time.c /stat-time.h /stat.c +/statat.c /stdbool.h /stdbool.in.h /stddef.h @@ -205,6 +212,8 @@ /unistd.c /unistd.h /unistd.in.h +/unlink.c +/unlinkat.c /unused-parameter.h /utimens.c /utimens.h @@ -225,4 +234,3 @@ /xsize.h /yesno.c /yesno.h -/fsync.c diff --git a/m4/.gitignore b/m4/.gitignore index 32f5566..4641515 100644 --- a/m4/.gitignore +++ b/m4/.gitignore @@ -32,6 +32,7 @@ /fcntl-safer.m4 /fcntl.m4 /fcntl_h.m4 +/fdatasync.m4 /fdopendir.m4 /fflush.m4 /filenamecat.m4 @@ -48,6 +49,8 @@ /fseeko.m4 /fseterr.m4 /fstat.m4 +/fstatat.m4 +/fsync.m4 /ftell.m4 /ftello.m4 /getcwd.m4 @@ -109,6 +112,7 @@ /quotearg.m4 /readdir.m4 /realloc.m4 +/rmdir.m4 /save-cwd.m4 /savedir.m4 /signbit.m4 @@ -136,6 +140,8 @@ /timespec.m4 /unistd-safer.m4 /unistd_h.m4 +/unlink.m4 +/unlinkat.m4 /utimbuf.m4 /utimens.m4 /utimes.m4 @@ -150,4 +156,3 @@ /xalloc.m4 /xsize.m4 /yesno.m4 -/fsync.m4 diff --git a/tailor.h b/tailor.h index 9d2399d..1feb807 100644 --- a/tailor.h +++ b/tailor.h @@ -56,7 +56,6 @@ # define NO_MULTIPLE_DOTS # define MAX_EXT_CHARS 3 # define Z_SUFFIX "z" -# define PROTO # define STDC_HEADERS # define NO_SIZE_CHECK # define UNLINK_READONLY_BUG @@ -81,7 +80,6 @@ # define Z_SUFFIX "z" # define casemap(c) tolow(c) # endif -# define PROTO # define STDC_HEADERS # define UNLINK_READONLY_BUG # include @@ -114,7 +112,6 @@ # define PATH_SEP2 '\\' # define PATH_SEP3 ':' # define MAX_PATH_LEN 260 -# define PROTO # define STDC_HEADERS # define SET_BINARY_MODE(fd) setmode(fd, O_BINARY) # define UNLINK_READONLY_BUG @@ -179,7 +176,6 @@ # define HAVE_CHOWN # define HAVE_LSTAT # else /* SASC */ -# define NO_STDIN_FSTAT # define HAVE_SYS_DIR_H # include /* for read() and write() */ # define direct dirent @@ -202,19 +198,6 @@ # endif #endif -#ifdef MACOS -# define PATH_SEP ':' -# define DYN_ALLOC -# define PROTO -# define NO_STDIN_FSTAT -# define chmod(file, mode) (0) -# define OPEN(name, flags, mode) open(name, flags) -# define OS_CODE 0x07 -# ifdef MPW -# define isatty(fd) ((fd) <= 2) -# endif -#endif - #ifdef TOPS20 # define OS_CODE 0x0a #endif @@ -276,7 +259,3 @@ #ifndef SET_BINARY_MODE # define SET_BINARY_MODE(fd) #endif - -#ifndef OPEN -# define OPEN(name, flags, mode) open_safer (name, flags, mode) -#endif diff --git a/zip.c b/zip.c index 7f1117c..178bfd0 100644 --- a/zip.c +++ b/zip.c @@ -23,9 +23,6 @@ #include "tailor.h" #include "gzip.h" -#include -#include - local ulg crc; /* crc on uncompressed file data */ off_t header_bytes; /* number of bytes in gzip header */ -- 2.5.0 --------------030907090002050303050204 Content-Type: text/x-diff; name="0002-gzip-use-constants-not-fileno.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-gzip-use-constants-not-fileno.patch" >From 94757ff442298042a5613a4515612a7ad1ee4597 Mon Sep 17 00:00:00 2001 From: Paul Eggert Date: Sat, 27 Feb 2016 14:33:17 -0800 Subject: [PATCH 2/4] gzip: use constants, not fileno * gzip.c (main, treat_stdin, treat_file, get_method) (check_ofname): Prefer STDIN_FILENO to fileno (stdin), and similarly for STDOUT_FILENO. --- gzip.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/gzip.c b/gzip.c index 15bcf78..6cfc561 100644 --- a/gzip.c +++ b/gzip.c @@ -634,7 +634,7 @@ int main (int argc, char **argv) /* And get to work */ if (file_count != 0) { if (to_stdout && !test && !list && (!decompress || !ascii)) { - SET_BINARY_MODE(fileno(stdout)); + SET_BINARY_MODE (STDOUT_FILENO); } while (optind < argc) { treat_file(argv[optind++]); @@ -675,7 +675,7 @@ local void treat_stdin() { if (!force && !list && (presume_input_tty - || isatty(fileno((FILE *)(decompress ? stdin : stdout))))) { + || isatty (decompress ? STDIN_FILENO : STDOUT_FILENO))) { /* Do not send compressed data to the terminal or read it from * the terminal. We get here when user invoked the program * without parameters, so be helpful. According to the GNU standards: @@ -701,16 +701,16 @@ local void treat_stdin() } if (decompress || !ascii) { - SET_BINARY_MODE(fileno(stdin)); + SET_BINARY_MODE (STDIN_FILENO); } if (!test && !list && (!decompress || !ascii)) { - SET_BINARY_MODE(fileno(stdout)); + SET_BINARY_MODE (STDOUT_FILENO); } strcpy(ifname, "stdin"); strcpy(ofname, "stdout"); /* Get the file's time stamp and size. */ - if (fstat (fileno (stdin), &istat) != 0) + if (fstat (STDIN_FILENO, &istat) != 0) { progerror ("standard input"); do_exit (ERROR); @@ -728,7 +728,7 @@ local void treat_stdin() clear_bufs(); /* clear input and output buffers */ to_stdout = 1; part_nb = 0; - ifd = fileno(stdin); + ifd = STDIN_FILENO; if (decompress) { method = get_method(ifd); @@ -744,7 +744,8 @@ local void treat_stdin() /* Actually do the compression/decompression. Loop over zipped members. */ for (;;) { - if ((*work)(fileno(stdin), fileno(stdout)) != OK) return; + if (work (STDIN_FILENO, STDOUT_FILENO) != OK) + return; if (input_eof ()) break; @@ -931,7 +932,7 @@ local void treat_file(iname) * a new ofname and save the original name in the compressed file. */ if (to_stdout) { - ofd = fileno(stdout); + ofd = STDOUT_FILENO; /* Keep remove_ofname_fd negative. */ } else { if (create_outfile() != OK) return; @@ -1630,7 +1631,7 @@ local int get_method(in) inptr--; last_member = 1; if (imagic0 != EOF) { - write_buf(fileno(stdout), magic, 1); + write_buf (STDOUT_FILENO, magic, 1); bytes_out++; } } @@ -1844,7 +1845,7 @@ local int check_ofname() if (!force) { int ok = 0; fprintf (stderr, "%s: %s already exists;", program_name, ofname); - if (foreground && (presume_input_tty || isatty(fileno(stdin)))) { + if (foreground && (presume_input_tty || isatty (STDIN_FILENO))) { fprintf(stderr, " do you wish to overwrite (y or n)? "); fflush(stderr); ok = yesno(); -- 2.5.0 --------------030907090002050303050204 Content-Type: text/x-diff; name="0003-gzip-new-option-synchronous.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0003-gzip-new-option-synchronous.patch" >From d09376128d896a3bbd79eea3d7b77fe6e0b9ea35 Mon Sep 17 00:00:00 2001 From: Paul Eggert Date: Sat, 27 Feb 2016 23:58:54 -0800 Subject: [PATCH 3/4] gzip: new option --synchronous This follows up on the earlier patch to avoid data loss near the system crashes. It makes the new behavior optional, with the default off. See: http://bugs.gnu.org/22768 * NEWS, doc/gzip.texi (Sample, Invoking gzip), gunzip.in (usage): * gzip.1, zcat.in (usage): Document this. * gzip.c (synchronous): New static var. (SYNCHRONOUS_OPTION): New constant. (longopts, help, main, treat_file): Add support for --synchronous. --- NEWS | 12 ++++++++---- doc/gzip.texi | 11 +++++++++++ gunzip.in | 1 + gzip.1 | 6 ++++++ gzip.c | 28 +++++++++++++++++++++------- zcat.in | 1 + 6 files changed, 48 insertions(+), 11 deletions(-) diff --git a/NEWS b/NEWS index 31472cc..a3989af 100644 --- a/NEWS +++ b/NEWS @@ -4,14 +4,18 @@ GNU gzip NEWS -*- outline -*- ** Changes in behavior - When acting in-place, gzip now fsyncs the output before closing it. - This is slower, but on many file systems it is safer if the system - is about to crash. - The GZIP environment variable is now obsolescent; gzip now warns if it is used, and rejects attempts to use dangerous options or operands. You can use an alias or script instead. +** New features + + gzip now accepts the --synchronous option, which causes it to use + fsync and similar primitives to transfer output data to the output + file's storage device when the file system supports this. Although + this option makes gzip safer in the presence of system crashes, it + can make gzip considerably slower. + ** Bug fixes gzip -k -v no longer reports that files are replaced. diff --git a/doc/gzip.texi b/doc/gzip.texi index 978b919..fa94b84 100644 --- a/doc/gzip.texi +++ b/doc/gzip.texi @@ -204,6 +204,7 @@ Mandatory arguments to long options are mandatory for short options too. -q, --quiet suppress all warnings -r, --recursive operate recursively on directories -S, --suffix=SUF use suffix SUF on compressed files + --synchronous synchronous output (safer if system crashes, but slower) -t, --test test compressed file integrity -v, --verbose verbose mode -V, --version display version number @@ -374,6 +375,16 @@ gunzip -S "" * (*.* for MSDOS) Previous versions of gzip used the @samp{.z} suffix. This was changed to avoid a conflict with @command{pack}. +@item --synchronous +Use synchronous output, by transferring output data to the output +file's storage device when the file system supports this. Because +file system data can be cached, without this option if the system +crashes around the time a command like @samp{gzip FOO} is run the user +might lose both @file{FOO} and @file{FOO.gz}; this is the default with +@command{gzip}, just as it is the default with most applications that +move data. When this option is used, @command{gzip} is safer but can +be considerably slower. + @item --test @itemx -t Test. Check the compressed file integrity. diff --git a/gunzip.in b/gunzip.in index 1296834..d0efd2d 100644 --- a/gunzip.in +++ b/gunzip.in @@ -45,6 +45,7 @@ Mandatory arguments to long options are mandatory for short options too. -q, --quiet suppress all warnings -r, --recursive operate recursively on directories -S, --suffix=SUF use suffix SUF on compressed files + --synchronous synchronous output (safer if system crashes, but slower) -t, --test test compressed file integrity -v, --verbose verbose mode --help display this help and exit diff --git a/gzip.1 b/gzip.1 index 07995ea..3262a87 100644 --- a/gzip.1 +++ b/gzip.1 @@ -296,6 +296,12 @@ are transferred to other systems. When decompressing, add .suf to the beginning of the list of suffixes to try, when deriving an output file name from an input file name. .TP +.B --synchronous +Use synchronous output. With this option, +.I gzip +is less likely to lose data during a system crash, but it can be +considerably slower. +.TP .B \-t --test Test. Check the compressed file integrity. .TP diff --git a/gzip.c b/gzip.c index 6cfc561..d9cdfaa 100644 --- a/gzip.c +++ b/gzip.c @@ -159,6 +159,14 @@ DECLARE(uch, window, 2L*WSIZE); is deliberately not documented, and only for testing. */ static bool presume_input_tty; +/* If true, transfer output data to the output file's storage device + when supported. Otherwise, if the system crashes around the time + gzip is run, the user might lose both input and output data. See: + Pillai TS et al. All file systems are not created equal: on the + complexity of crafting crash-consistent applications. OSDI'14. 2014:433-48. + https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai */ +static bool synchronous; + static int ascii = 0; /* convert end-of-lines to local OS conventions */ int to_stdout = 0; /* output to stdout (-c) */ static int decompress = 0; /* decompress (-d) */ @@ -240,6 +248,7 @@ static int handled_sig[] = enum { PRESUME_INPUT_TTY_OPTION = CHAR_MAX + 1, + SYNCHRONOUS_OPTION, /* A value greater than all valid long options, used as a flag to distinguish options derived from the GZIP environment variable. */ @@ -268,6 +277,7 @@ static const struct option longopts[] = {"-presume-input-tty", no_argument, NULL, PRESUME_INPUT_TTY_OPTION}, {"quiet", 0, 0, 'q'}, /* quiet mode */ {"silent", 0, 0, 'q'}, /* quiet mode */ + {"synchronous",0, 0, SYNCHRONOUS_OPTION}, {"recursive", 0, 0, 'r'}, /* recurse through directories */ {"suffix", 1, 0, 'S'}, /* use given suffix instead of .gz */ {"test", 0, 0, 't'}, /* test compressed file integrity */ @@ -353,6 +363,7 @@ local void help() " -r, --recursive operate recursively on directories", #endif " -S, --suffix=SUF use suffix SUF on compressed files", + " --synchronous synchronous output (safer if system crashes, but slower)", " -t, --test test compressed file integrity", " -v, --verbose verbose mode", " -V, --version display version number", @@ -551,6 +562,9 @@ int main (int argc, char **argv) z_len = strlen(optarg); z_suffix = optarg; break; + case SYNCHRONOUS_OPTION: + synchronous = true; + break; case 't': test = decompress = to_stdout = 1; break; @@ -645,6 +659,12 @@ int main (int argc, char **argv) if (list && !quiet && file_count > 1) { do_list(-1, -1); /* print totals */ } + if (to_stdout + && ((synchronous + && (fdatasync (STDOUT_FILENO) != 0 && errno != EINVAL)) + || close (STDOUT_FILENO) != 0) + && errno != EBADF) + write_error (); do_exit(exit_code); return exit_code; /* just to avoid lint warning */ } @@ -972,13 +992,7 @@ local void treat_file(iname) { copy_stat (&istat); - /* If KEEP, transfer output data to the output file's storage device. - Otherwise, if the system crashed now the user might lose - both input and output data. See: Pillai TS et al. All - file systems are not created equal: on the complexity of - crafting crash-consistent applications. OSDI'14. 2014:433-48. - https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai */ - if ((!keep + if ((synchronous && ((0 <= dfd && fdatasync (dfd) != 0 && errno != EINVAL) || (fsync (ofd) != 0 && errno != EINVAL))) || close (ofd) != 0) diff --git a/zcat.in b/zcat.in index be6577f..75660e5 100644 --- a/zcat.in +++ b/zcat.in @@ -39,6 +39,7 @@ Uncompress FILEs to standard output. -q, --quiet suppress all warnings -r, --recursive operate recursively on directories -S, --suffix=SUF use suffix SUF on compressed files + --synchronous synchronous output (safer if system crashes, but slower) -t, --test test compressed file integrity -v, --verbose verbose mode --help display this help and exit -- 2.5.0 --------------030907090002050303050204 Content-Type: text/x-diff; name="0004-misc-update-version-copyright.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0004-misc-update-version-copyright.patch" >From e441ca6547c4034104a144395e1e3f3bffca93a5 Mon Sep 17 00:00:00 2001 From: Paul Eggert Date: Sun, 28 Feb 2016 00:01:50 -0800 Subject: [PATCH 4/4] misc: update --version copyright * gunzip.in, gzexe.in, zcat.in, zdiff.in, zforce.in, zgrep.in: * zless.in, zmore.in, znew.in: Update copyright year in --version output to 2016. --- gunzip.in | 2 +- gzexe.in | 2 +- zcat.in | 2 +- zdiff.in | 2 +- zforce.in | 2 +- zgrep.in | 2 +- zless.in | 2 +- zmore.in | 2 +- znew.in | 2 +- 9 files changed, 9 insertions(+), 9 deletions(-) diff --git a/gunzip.in b/gunzip.in index d0efd2d..d7edb53 100644 --- a/gunzip.in +++ b/gunzip.in @@ -24,7 +24,7 @@ esac PATH=$bindir:$PATH version="gunzip (gzip) @VERSION@ -Copyright (C) 2007, 2011-2015 Free Software Foundation, Inc. +Copyright (C) 2007, 2011-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. diff --git a/gzexe.in b/gzexe.in index d5b82a2..6b61ec4 100644 --- a/gzexe.in +++ b/gzexe.in @@ -37,7 +37,7 @@ nl=' IFS=" $tab$nl" version='gzexe (gzip) @VERSION@ -Copyright (C) 2007, 2011-2015 Free Software Foundation, Inc. +Copyright (C) 2007, 2011-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. diff --git a/zcat.in b/zcat.in index 75660e5..d8ffd85 100644 --- a/zcat.in +++ b/zcat.in @@ -24,7 +24,7 @@ esac PATH=$bindir:$PATH version="zcat (gzip) @VERSION@ -Copyright (C) 2007, 2011-2015 Free Software Foundation, Inc. +Copyright (C) 2007, 2011-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. diff --git a/zdiff.in b/zdiff.in index 60905d5..f1fa95f 100644 --- a/zdiff.in +++ b/zdiff.in @@ -31,7 +31,7 @@ case $1 in esac version="z$prog (gzip) @VERSION@ -Copyright (C) 2009, 2011-2015 Free Software Foundation, Inc. +Copyright (C) 2009, 2011-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. diff --git a/zforce.in b/zforce.in index e5d66ad..5baff07 100644 --- a/zforce.in +++ b/zforce.in @@ -30,7 +30,7 @@ esac PATH=$bindir:$PATH; export PATH version="zforce (gzip) @VERSION@ -Copyright (C) 2010-2015 Free Software Foundation, Inc. +Copyright (C) 2010-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. diff --git a/zgrep.in b/zgrep.in index 9668fae..d94b8d8 100644 --- a/zgrep.in +++ b/zgrep.in @@ -31,7 +31,7 @@ PATH=$bindir:$PATH grep='${GREP-'\''@GREP@'\''}' version='zgrep (gzip) @VERSION@ -Copyright (C) 2010-2015 Free Software Foundation, Inc. +Copyright (C) 2010-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. diff --git a/zless.in b/zless.in index 9c7d163..a3edcc7 100644 --- a/zless.in +++ b/zless.in @@ -23,7 +23,7 @@ esac PATH=$bindir:$PATH; export PATH version="zless (gzip) @VERSION@ -Copyright (C) 2007, 2011-2015 Free Software Foundation, Inc. +Copyright (C) 2007, 2011-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. diff --git a/zmore.in b/zmore.in index 5f5bb26..b1bb68f 100644 --- a/zmore.in +++ b/zmore.in @@ -24,7 +24,7 @@ esac PATH=$bindir:$PATH; export PATH version="zmore (gzip) @VERSION@ -Copyright (C) 2010-2015 Free Software Foundation, Inc. +Copyright (C) 2010-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. diff --git a/znew.in b/znew.in index 7f4a8eb..ec025d2 100644 --- a/znew.in +++ b/znew.in @@ -25,7 +25,7 @@ esac PATH=$bindir:$PATH; export PATH version="znew (gzip) @VERSION@ -Copyright (C) 2010-2015 Free Software Foundation, Inc. +Copyright (C) 2010-2016 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. -- 2.5.0 --------------030907090002050303050204-- From debbugs-submit-bounces@debbugs.gnu.org Sun Feb 28 03:31:08 2016 Received: (at 22768) by debbugs.gnu.org; 28 Feb 2016 08:31:08 +0000 Received: from localhost ([127.0.0.1]:49500 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZwkt-0001z1-TL for submit@debbugs.gnu.org; Sun, 28 Feb 2016 03:31:08 -0500 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:57319) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aZwks-0001yX-9f for 22768@debbugs.gnu.org; Sun, 28 Feb 2016 03:31:06 -0500 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id EBDE4160FD0; Sun, 28 Feb 2016 00:31:00 -0800 (PST) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id CWnR00YuMosI; Sun, 28 Feb 2016 00:30:59 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id E665A160FD5; Sun, 28 Feb 2016 00:30:59 -0800 (PST) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 9qD807eCMHcT; Sun, 28 Feb 2016 00:30:59 -0800 (PST) Received: from [192.168.1.9] (pool-100-32-155-148.lsanca.fios.verizon.net [100.32.155.148]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id C2E59160FD0; Sun, 28 Feb 2016 00:30:59 -0800 (PST) Subject: Re: bug#22768: Crash safety To: Antonio Diaz Diaz References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> <20160226173235198655396@bob.proulx.com> <56D1C4FA.6000704@gnu.org> From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <56D2B043.20909@cs.ucla.edu> Date: Sun, 28 Feb 2016 00:30:59 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <56D1C4FA.6000704@gnu.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang , 22768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Antonio Diaz Diaz wrote: > it may be a cause of feature creep. If gzip fsyncs the output file it > might also test it, or even compare it with the input file, before deleting the > input file. Feature creep is something we should avoid. Here, though, it's a real pain to synchronize correctly and many people will get it wrong. (See my commentary at the end of this email for one example of getting it wrong.) By comparison, comparing the decompressed output with the input file is something that most people will probably get right, so it's less useful to add a gzip option for that. > Second, as doing it right in all circumstances may be impossible Sure, as some file systems do not support fsync. Still, gzip should do what it can. > it may become an endless source of bug reports. I doubt it. gzip has run unsafely for decades, and this is the first bug report about it -- one discovered by code inspection, not by actual failure. > (fsyncing also the destination's directory, Yes, that needs fixing. Done in the patches I just now emailed to you. > opening the output with O_DIRECT,...). I doubt whether that feature will be needed or useful for gzip. > Third, it fights against other layers of the system, like the filesystem, > instead of collaborating with them. True, fsync is a bad design. But that is no excuse for gzip losing data. > Fourth, it fights against user's wishes instead of obeying them. This should not be a problem if --synchronous is a new option, defaulting to the old (unsynchronized) behavior. > I think that the best way of guarding an important file against all bugs and > crashes is a extended version of the procedure already documented in the manual > of lzip: > > 1) gzip --keep file # don't delete input > 2) sync # commit output and directory to disk > 3) zcmp file file.gz # verify output > 4) rm file # then remove input That approach does not suffice, because 'sync' does not guarantee that the output data has been synchronized to disk. See: http://pubs.opengroup.org/onlinepubs/9699919799/functions/sync.html With GNU 'sync' there is a workaround, but it is not portable to non-GNU systems; besides, the workaround is not obvious. From debbugs-submit-bounces@debbugs.gnu.org Mon Feb 29 12:07:18 2016 Received: (at 22768) by debbugs.gnu.org; 29 Feb 2016 17:07:19 +0000 Received: from localhost ([127.0.0.1]:53899 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aaRHy-0007e1-ME for submit@debbugs.gnu.org; Mon, 29 Feb 2016 12:07:18 -0500 Received: from eggs.gnu.org ([208.118.235.92]:41747) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aaRHw-0007dn-R7 for 22768@debbugs.gnu.org; Mon, 29 Feb 2016 12:07:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aaRHl-0007cJ-Qf for 22768@debbugs.gnu.org; Mon, 29 Feb 2016 12:07:11 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:34048) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aaRHl-0007cD-ML; Mon, 29 Feb 2016 12:07:05 -0500 Received: from 249.red-217-124-67.dialup.dynamic.ccgg.telefonica.net ([217.124.67.249]:47113) by fencepost.gnu.org with esmtpa (Exim 4.82) (envelope-from ) id 1aaRHj-00011s-TE; Mon, 29 Feb 2016 12:07:05 -0500 Message-ID: <56D47C13.1080207@gnu.org> Date: Mon, 29 Feb 2016 18:12:51 +0100 From: Antonio Diaz Diaz User-Agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.8.1.4) Gecko/20070601 SeaMonkey/1.1.2 MIME-Version: 1.0 To: 22768@debbugs.gnu.org Subject: Re: bug#22768: Crash safety References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> <20160226173235198655396@bob.proulx.com> <56D1C4FA.6000704@gnu.org> <56D2B043.20909@cs.ucla.edu> In-Reply-To: <56D2B043.20909@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Paul Eggert wrote: > Feature creep is something we should avoid. Here, though, it's a real > pain to synchronize correctly and many people will get it wrong. The problem is that it is impossible to get it right unless one does something as extreme as unmounting-then-remounting the filesystem, or even making a backup copy on a removable device and then verifying the copy on a different computer. > Sure, as some file systems do not support fsync. Still, gzip should do > what it can. I am not so sure. There seems to be an arms race between tools that do what they can and ways of preventing those tools from doing it. Just search for "disable fsync". Even if invoked optionally, all this complication to perhaps achieve nothing but a false sense of safety goes against my KISS philosophy. Imagine if some backup tool begins calling 'gzip --synchronous', and users are forced to install libeatmydata to disable it. >> it may become an endless source of bug reports. > > I doubt it. gzip has run unsafely for decades, and this is the first > bug report about it -- one discovered by code inspection, not by actual > failure. Publish or perish may be the cause of such an endless source of bug reports. The next one might be titled "On why 'gzip --synchronous' does not work on some filesystems". >> (fsyncing also the destination's directory, > > Yes, that needs fixing. Done in the patches I just now emailed to you. Thanks. But I was not asking for a fix. Just pointing out what others might ask. > True, fsync is a bad design. But that is no excuse for gzip losing data. As I see it, it is not gzip the one losing the data, but the filesystem that does not respect the write order, or even the user that chose such filesystem (perhaps because of a good reason). >> 1) gzip --keep file # don't delete input >> 2) sync # commit output and directory to disk >> 3) zcmp file file.gz # verify output >> 4) rm file # then remove input > > That approach does not suffice, because 'sync' does not guarantee that > the output data has been synchronized to disk. I know, but how can you guarantee that 'gzip --synchronous' will work on a system where the 'sync' above does not even guarantee that 'file.gz' is written to disk before 'file' is deleted? I still think that the right thing to do is to not implement any kind of fsync functionality in gzip/lzip, and achieve permanence (when it is needed) by some other means. As you said, gzip has run unsafely for decades without a failure. Best regards, Antonio. From debbugs-submit-bounces@debbugs.gnu.org Mon Feb 29 14:43:51 2016 Received: (at 22768) by debbugs.gnu.org; 29 Feb 2016 19:43:51 +0000 Received: from localhost ([127.0.0.1]:54068 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aaTjS-0004hC-PS for submit@debbugs.gnu.org; Mon, 29 Feb 2016 14:43:50 -0500 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:57226) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aaTjR-0004h1-VD for 22768@debbugs.gnu.org; Mon, 29 Feb 2016 14:43:50 -0500 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 70B80160FDA; Mon, 29 Feb 2016 11:43:44 -0800 (PST) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id UX6WLqjlQzaP; Mon, 29 Feb 2016 11:43:43 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id C5851160FD2; Mon, 29 Feb 2016 11:43:41 -0800 (PST) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7_9078ttclbj; Mon, 29 Feb 2016 11:43:41 -0800 (PST) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id AA6AE160F53; Mon, 29 Feb 2016 11:43:41 -0800 (PST) Subject: Re: bug#22768: Crash safety To: Antonio Diaz Diaz , 22768@debbugs.gnu.org References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> <20160226173235198655396@bob.proulx.com> <56D1C4FA.6000704@gnu.org> <56D2B043.20909@cs.ucla.edu> <56D47C13.1080207@gnu.org> From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <56D49F6D.8030200@cs.ucla.edu> Date: Mon, 29 Feb 2016 11:43:41 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56D47C13.1080207@gnu.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) On 02/29/2016 09:12 AM, Antonio Diaz Diaz wrote: >>> 1) gzip --keep file # don't delete input >>> 2) sync # commit output and directory to disk >>> 3) zcmp file file.gz # verify output >>> 4) rm file # then remove input >> >> That approach does not suffice, because 'sync' does not guarantee >> that the output data has been synchronized to disk. > > I know, but how can you guarantee that 'gzip --synchronous' will work > on a system where the 'sync' above does not even guarantee that > 'file.gz' is written to disk before 'file' is deleted? Yes, I can guarantee that 'gzip --synchronous' will not lose data on any system conforming to POSIX with the Synchronized Input and Output option. No such guarantee can be made for the above shell script, because the 'sync' command does not make the same guarantees that the 'fsync' function does. Putting the above shell script into the documentation would give users a false sense of security. (Or maybe we should put the above shell script into the documentation as an example of what *not* to do. :-) > The next one might be titled "On why 'gzip --synchronous' does not > work on some filesystems". :-) Of course the problem can still exist on file systems that do not conform to POSIX, and there are many of those. Still, there are people who take these things seriously, and who use file systems that are safe in the presence of crashes, and for these people grep --synchronous should work. > As you said, gzip has run unsafely for decades without a failure. I did not say that! And I am skeptical that it's true. I think it's quite possible that gzip has lost data when an operating system crashed at the wrong moment. From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 01 13:07:36 2016 Received: (at 22768) by debbugs.gnu.org; 1 Mar 2016 18:07:36 +0000 Received: from localhost ([127.0.0.1]:56654 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aaohs-0001kc-7P for submit@debbugs.gnu.org; Tue, 01 Mar 2016 13:07:36 -0500 Received: from eggs.gnu.org ([208.118.235.92]:44203) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aaohq-0001kP-De for 22768@debbugs.gnu.org; Tue, 01 Mar 2016 13:07:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aaohg-0007gm-5M for 22768@debbugs.gnu.org; Tue, 01 Mar 2016 13:07:29 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:37613) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aaohg-0007gi-1d; Tue, 01 Mar 2016 13:07:24 -0500 Received: from 154.red-217-124-67.dialup.dynamic.ccgg.telefonica.net ([217.124.67.154]:35440) by fencepost.gnu.org with esmtpa (Exim 4.82) (envelope-from ) id 1aaohe-0005YD-OK; Tue, 01 Mar 2016 13:07:23 -0500 Message-ID: <56D5DBBB.9040400@gnu.org> Date: Tue, 01 Mar 2016 19:13:15 +0100 From: Antonio Diaz Diaz User-Agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.8.1.4) Gecko/20070601 SeaMonkey/1.1.2 MIME-Version: 1.0 To: 22768@debbugs.gnu.org Subject: Re: bug#22768: Crash safety References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> <20160226173235198655396@bob.proulx.com> <56D1C4FA.6000704@gnu.org> <56D2B043.20909@cs.ucla.edu> <56D47C13.1080207@gnu.org> <56D49F6D.8030200@cs.ucla.edu> In-Reply-To: <56D49F6D.8030200@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Paul Eggert wrote: >> I know, but how can you guarantee that 'gzip --synchronous' will work >> on a system where the 'sync' above does not even guarantee that >> 'file.gz' is written to disk before 'file' is deleted? > > Yes, I can guarantee that 'gzip --synchronous' will not lose data on any > system conforming to POSIX with the Synchronized Input and Output > option. Unless someone has somehow disabled fsync, I guess. :-) I am not questioning you. You know very well what you do. It is simply that I find the situation so chaotic that I think maybe better methods to ensure data permanence have yet to be developed. > Still, there are people who take these things seriously, and who use > file systems that are safe in the presence of crashes, and for these > people grep --synchronous should work. What I ask myself is, are those people better served by adding --synchronous options to every tool, or by using a crash-tolerant file system[1]? "At the ACM Symposium on Operating Systems Principles in October, MIT researchers will present the first file system that is mathematically guaranteed not to lose track of data during crashes." [1] http://news.mit.edu/2015/crash-tolerant-data-storage-0824 >> As you said, gzip has run unsafely for decades without a failure. > > I did not say that! Sorry, I meant "without a reported failure". My point is that if gzip has run unsafely for decades without a reported failure, maybe all those who take these things seriously are already using file systems safe enough to guarantee that well behaved tools like gzip do not lose data. Best regards, Antonio. (No need to CC me. I am subscribed to bug-gzip). From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 01 15:57:15 2016 Received: (at 22768) by debbugs.gnu.org; 1 Mar 2016 20:57:15 +0000 Received: from localhost ([127.0.0.1]:56746 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aarM3-0008Ef-K8 for submit@debbugs.gnu.org; Tue, 01 Mar 2016 15:57:15 -0500 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:37794) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aarM2-0008ER-24 for 22768@debbugs.gnu.org; Tue, 01 Mar 2016 15:57:14 -0500 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id CEA31160633; Tue, 1 Mar 2016 12:57:07 -0800 (PST) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id W5BsvSRQT-iy; Tue, 1 Mar 2016 12:57:05 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 61418160FDB; Tue, 1 Mar 2016 12:57:05 -0800 (PST) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 6OEPokXCIrLp; Tue, 1 Mar 2016 12:57:05 -0800 (PST) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 46610160633; Tue, 1 Mar 2016 12:57:05 -0800 (PST) Subject: Re: bug#22768: Crash safety To: Antonio Diaz Diaz , 22768@debbugs.gnu.org References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> <20160226173235198655396@bob.proulx.com> <56D1C4FA.6000704@gnu.org> <56D2B043.20909@cs.ucla.edu> <56D47C13.1080207@gnu.org> <56D49F6D.8030200@cs.ucla.edu> <56D5DBBB.9040400@gnu.org> From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <56D60221.9010005@cs.ucla.edu> Date: Tue, 1 Mar 2016 12:57:05 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56D5DBBB.9040400@gnu.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) On 03/01/2016 10:13 AM, Antonio Diaz Diaz wrote: > [1] http://news.mit.edu/2015/crash-tolerant-data-storage-0824 FSCQ is not even close to ready for prime-time, I'm afraid. Its prototype is slow compared to conventional file systems (it assumes a single-threaded kernel, it issues many more writes than ext4 does to implement a commit, its has been publicly tested only on flash drives, etc.). The FSQC authors would like to add support for fsync/fdatasync to get some of that performance back, which seems reasonable -- but at that point, applications like gzip would still need to call fsync/fdatasync to avoid losing data. You may well be right that eventually file system designers will figure this stuff out so that well-written POSIX applications will not lose data even if they don't use fsync/fdatasync. However, if FSCQ is any indication, we're many years away from that. In the meantime fsync/fdatasync is all we have. > My point is that if gzip has run unsafely for decades without a > reported failure, maybe all those who take these things seriously are > already using file systems safe enough to guarantee that well behaved > tools like gzip do not lose data. That will be true for many users. Still, I imagine that non-experts would have a good deal of trouble connecting the dots between lost data and any gzip invocation that lost the data, and could chalk it up to a system crash losing data for other reasons. (After all, things are somewhat chaotic during a crash...) One can find examples on the net like "How to recover lost/deleted Gzip compressed gz file" (this is for BYclouder, a commercial tool) that talk about system crashes, and which indicate (though do not prove) that a real problem exists with gzip. My sources: Chen H, Ziegler D, Chajed T, Chlipapa A, Kaashoek MF, Zeldovich N. Using Crash Hoare logic for certifying the FSCQ file system. SOSP 2015. https://people.csail.mit.edu/nickolai/papers/chen-fscq.pdf How to recover lost/deleted Gzip compressed gz file. BYclouder. 2013-04-23. http://www.byclouder.com/help/recovery/file/archive/how-to-recover-gzip-gz.html From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 03 08:02:18 2016 Received: (at 22768) by debbugs.gnu.org; 3 Mar 2016 13:02:18 +0000 Received: from localhost ([127.0.0.1]:58721 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1abStW-0006mS-6l for submit@debbugs.gnu.org; Thu, 03 Mar 2016 08:02:18 -0500 Received: from eggs.gnu.org ([208.118.235.92]:54911) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1abStV-0006mG-6K for 22768@debbugs.gnu.org; Thu, 03 Mar 2016 08:02:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1abStM-0002vV-3x for 22768@debbugs.gnu.org; Thu, 03 Mar 2016 08:02:12 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:52772) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abStM-0002vG-0v; Thu, 03 Mar 2016 08:02:08 -0500 Received: from 184.red-217-124-67.dialup.dynamic.ccgg.telefonica.net ([217.124.67.184]:57396) by fencepost.gnu.org with esmtpa (Exim 4.82) (envelope-from ) id 1abStK-0000x2-WB; Thu, 03 Mar 2016 08:02:07 -0500 Message-ID: <56D83739.7070907@gnu.org> Date: Thu, 03 Mar 2016 14:08:09 +0100 From: Antonio Diaz Diaz User-Agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.8.1.4) Gecko/20070601 SeaMonkey/1.1.2 MIME-Version: 1.0 To: 22768@debbugs.gnu.org Subject: Re: bug#22768: Crash safety References: <56CC0A07.1080403@cs.ucla.edu> <56D03848.5060108@gnu.org> <56D03CA0.2020100@cs.ucla.edu> <56D04673.2070405@gnu.org> <56D0C371.8030408@cs.ucla.edu> <56D0EB15.3060504@gnu.org> <20160226173235198655396@bob.proulx.com> <56D1C4FA.6000704@gnu.org> <56D2B043.20909@cs.ucla.edu> <56D47C13.1080207@gnu.org> <56D49F6D.8030200@cs.ucla.edu> <56D5DBBB.9040400@gnu.org> <56D60221.9010005@cs.ucla.edu> In-Reply-To: <56D60221.9010005@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22768 Cc: Yanyan Jiang X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Paul Eggert wrote: > You may well be right that eventually file system designers will figure > this stuff out so that well-written POSIX applications will not lose > data even if they don't use fsync/fdatasync. However, if FSCQ is any > indication, we're many years away from that. In the meantime > fsync/fdatasync is all we have. Thanks for the explanation. I have already started the last round of release candidates of the lzip family, but just after releasing lzip-1.18, in a month or so, i'll implement --synchronous in a way compatible with what you implement in gzip. Best regards, Antonio. From unknown Mon Aug 18 15:39:38 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 01 Apr 2016 11:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator From debbugs-submit-bounces@debbugs.gnu.org Mon May 09 18:36:37 2016 Received: (at control) by debbugs.gnu.org; 9 May 2016 22:36:37 +0000 Received: from localhost ([127.0.0.1]:44848 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aztn2-0008SU-QP for submit@debbugs.gnu.org; Mon, 09 May 2016 18:36:37 -0400 Received: from nm14-vm1.bullet.mail.bf1.yahoo.com ([98.139.213.132]:44672) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aztHd-0007h0-0L for control@debbugs.gnu.org; Mon, 09 May 2016 18:04:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1462831442; bh=Dfbau/J/QEszFjnR1g04Um9KbFlUzrYw892dHtplDTY=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=tHRGWf1P25lV+4ZTrsCuZmxsnxtUojei5ncXmu03DY7im//jFOc+8lCM/G3tZqBk3aKkXIuMXYfBBnQeqV44qehT1LMHH6QOaI+A+OX5DqJy4Gp7hsFYnmShkRB0rCr9xAiwqIQRZfJ3gDNgtLP8oxmuhkCzEkAIk7JP+nvrHE39OIE+KfJsHtZmf9pfCi7tfDt+esfV4xlO2k3wgMCkm5N5VBxRaefWhIm3r/61BTt7dgIYoaOJYy2L4Cq6M+oHhj0TI8FTHEQvnhdT2QaXCsBNSzm7AFB/Z6dtHzrKcdaVR0nAgfNCBrDWgNuAkY1qFY42aXtdp/+VXNBHKl2AUg== Received: from [98.139.215.141] by nm14.bullet.mail.bf1.yahoo.com with NNFMP; 09 May 2016 22:04:02 -0000 Received: from [98.139.212.233] by tm12.bullet.mail.bf1.yahoo.com with NNFMP; 09 May 2016 22:04:02 -0000 Received: from [127.0.0.1] by omp1042.mail.bf1.yahoo.com with NNFMP; 09 May 2016 22:04:02 -0000 X-Yahoo-Newman-Property: ymail-5 X-Yahoo-Newman-Id: 230691.44189.bm@omp1042.mail.bf1.yahoo.com X-YMail-OSG: OtfktxsVM1ma.cQWJWTGTFP6keOAdUtGpXTcZAZurtiainVeP5T5.hMkfRbD6CM agFDTfgLanaCXcb0IbJIgyQWhQUOvv5arDftcH36glGeu5MQzZ22TT.cGTr8X2BpoNk4NUufMAaR ZOphW6tleI7sPO9axCaOKcBMG3nRiXd_sV6yfQTNiPy3jZdl82jthG9zbNNcEC_RhpSreAcpPHsY 0EJPAhqbud7iQcwB52w4JuqbAR3q2cM3IbdwFbKirto6RGD35V4JWelohW2R1XinGBdO1UeFU0tU ZjNl5DY5833aTdWewQJ.PrFqKvAgb0DxvRw00hAi1.ZD72sGcfoTAQccyYYdFfQw_4zmD8Nu6eSf 7eW.RjOeIJ_DjfEu8b0WI0_OJjh1abwDoPaniPkcMUMk1G.DkZsZfuZeWJUBoDVj1_HMdyc4XS7x Bv24I_b4vxmkcAfyeyk8CLesUYZAfFc2rq1AVEAZtDuwUrxDpJbbnVPadlz9fDsndzfPVU0itXVI H7.TqGg-- Received: from jws10693.mail.bf1.yahoo.com by sendmailws161.mail.bf1.yahoo.com; Mon, 09 May 2016 22:04:01 +0000; 1462831441.748 Date: Mon, 9 May 2016 22:04:01 +0000 (UTC) From: John Wiersba To: GNU bug tracker automated control server Message-ID: <1538121635.1423254.1462831441443.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: Subject: unarchive 22768 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1423253_1957983009.1462831441441" Content-Length: 1291 X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-Mailman-Approved-At: Mon, 09 May 2016 18:36:36 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: John Wiersba Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) ------=_Part_1423253_1957983009.1462831441441 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit unarchive 22768 ------=_Part_1423253_1957983009.1462831441441 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit
unarchive 22768


------=_Part_1423253_1957983009.1462831441441-- From unknown Mon Aug 18 15:39:38 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 07 Jun 2016 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator