From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Roman Rybalko Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: help-debbugs@gnu.org Resent-Date: Sat, 23 Jul 2011 22:42:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 9157 X-GNU-PR-Package: debbugs.gnu.org X-GNU-PR-Keywords: patch To: undisclosed-recipients:; Received: via spool by submit@debbugs.gnu.org id=B.131146089328668 (code B ref -1); Sat, 23 Jul 2011 22:42:04 +0000 Received: (at submit) by debbugs.gnu.org; 23 Jul 2011 22:41:33 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qkkt0-0007SI-HY for submit@debbugs.gnu.org; Sat, 23 Jul 2011 18:41:32 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qkf87-0007lA-AA for submit@debbugs.gnu.org; Sat, 23 Jul 2011 12:32:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qkf80-0003qJ-QE for submit@debbugs.gnu.org; Sat, 23 Jul 2011 12:32:38 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:49617) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qkf80-0003qF-Oi for submit@debbugs.gnu.org; Sat, 23 Jul 2011 12:32:36 -0400 Received: from eggs.gnu.org ([140.186.70.92]:60155) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qkf7z-0001sK-0B for bug-coreutils@gnu.org; Sat, 23 Jul 2011 12:32:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qkf7w-0003q1-QQ for bug-coreutils@gnu.org; Sat, 23 Jul 2011 12:32:34 -0400 Received: from host.romanr.info ([82.146.61.98]:45136) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qkf7w-0003pp-CB for bug-coreutils@gnu.org; Sat, 23 Jul 2011 12:32:32 -0400 Received: from firewall.localdomain (withheld) by host.romanr.info (Postfix) with ESMTPSA id 2EF15641CE1A for ; Sat, 23 Jul 2011 20:32:21 +0400 (MSD) Received: from roma.localdomain (withheld) by firewall.localdomain (Postfix) with ESMTP id DABDF8685 for ; Sat, 23 Jul 2011 20:32:24 +0400 (MSD) Received: from test (withheld) by roma.localdomain (Postfix) with SMTP id 3C04B69E5E for ; Sat, 23 Jul 2011 20:32:27 +0400 (MSD) X-Mailbox-Line: From dd75e573a541d3fe046fb56d482d6419dd92b958 Mon Sep 17 00:00:00 2001 From: Roman Rybalko Date: Sat, 23 Jul 2011 20:08:01 +0400 Message-Id: <20110723163227.3C04B69E5E@roma.localdomain> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -3.9 (---) X-Mailman-Approved-At: Sat, 23 Jul 2011 18:41:29 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.9 (---) --- doc/coreutils.texi | 4 +++ src/dd.c | 22 +++++++++++++++- tests/Makefile.am | 1 + tests/dd/sparse | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 96 insertions(+), 1 deletions(-) create mode 100755 tests/dd/sparse diff --git a/doc/coreutils.texi b/doc/coreutils.texi index 424446c..761c698 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. When used with @samp{block} or @samp{unblock}, pad with spaces instead of zero bytes. +@item sparse +@opindex sparse +Make sparse output file. + @end table The following ``conversions'' are really file flags diff --git a/src/dd.c b/src/dd.c index 0824f6c..49847f5 100644 --- a/src/dd.c +++ b/src/dd.c @@ -126,7 +126,8 @@ enum C_NOCREAT = 010000, C_EXCL = 020000, C_FDATASYNC = 040000, - C_FSYNC = 0100000 + C_FSYNC = 0100000, + C_SPARSE = 0200000 }; /* Status bit masks. */ @@ -268,6 +269,7 @@ static struct symbol_value const conversions[] = {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ {"fdatasync", C_FDATASYNC}, /* Synchronize output data before finishing. */ {"fsync", C_FSYNC}, /* Also synchronize output metadata. */ + {"sparse", C_SPARSE}, /* Make sparse output file. */ {"", 0} }; @@ -533,6 +535,9 @@ Each CONV symbol may be:\n\ fsync likewise, but also write metadata\n\ "), stdout); fputs (_("\ + sparse make sparse output file\n\ +"), stdout); + fputs (_("\ \n\ Each FLAG symbol may be:\n\ \n\ @@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size) { ssize_t nwritten; process_signals (); + if (conversions_mask & C_SPARSE) + { + off_t seek_size = 0; + while (buf[total_written + seek_size] == 0) + ++seek_size; + if (seek_size) + { + off_t cur_off = 0; + cur_off = lseek(fd, seek_size, SEEK_CUR); + if (cur_off < 0) + break; + total_written += seek_size; + continue; + } + } nwritten = write (fd, buf + total_written, size - total_written); if (nwritten < 0) { diff --git a/tests/Makefile.am b/tests/Makefile.am index ebd1b11..0f1376a 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -364,6 +364,7 @@ TESTS = \ dd/skip-seek \ dd/skip-seek2 \ dd/skip-seek-past-file \ + dd/sparse \ dd/stderr \ dd/unblock \ dd/unblock-sync \ diff --git a/tests/dd/sparse b/tests/dd/sparse new file mode 100755 index 0000000..f0e0806 --- /dev/null +++ b/tests/dd/sparse @@ -0,0 +1,70 @@ +#!/bin/sh +# Ensure that dd conv=sparse works. + +# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/init.sh"; path_prepend_ ../src +print_ver_ dd + +# sometimes we may read less than 1M +dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample0`" = "1048576" ] || fail=1 +dd if=/dev/urandom of=sample1 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample1`" = "1048576" ] || fail=1 + +# test 1 +dd if=sample1 of=test1 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test1 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test1 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 1-1 +dd if=test1 of=out1-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-1`" = "`stat -c '%s %b %B' out1-1-check`" ] || fail=1 + +# test 1-2 +dd if=test1 of=out1-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=3 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-2`" = "`stat -c '%s %b %B' out1-2-check`" ] || fail=1 + +# test 1-3 +dd if=test1 of=out1-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-3-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-3`" = "`stat -c '%s %b %B' out1-3-check`" ] || fail=1 + +# test 2 +dd if=sample0 of=test2 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test2 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test2 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 2-1 +dd if=test2 of=out2-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-1-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-1`" = "`stat -c '%s %b %B' out2-1-check`" ] || fail=1 + +# test 2-2 +dd if=test2 of=out2-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-2-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-2`" = "`stat -c '%s %b %B' out2-2-check`" ] || fail=1 + +# test 2-3 +dd if=test2 of=out2-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-3-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-3`" = "`stat -c '%s %b %B' out2-3-check`" ] || fail=1 + +Exit $fail -- 1.7.0.4 From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag References: <20110723163227.3C04B69E5E@roma.localdomain> In-Reply-To: <20110723163227.3C04B69E5E@roma.localdomain> Resent-From: Roman Rybalko Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: help-debbugs@gnu.org Resent-Date: Sun, 24 Jul 2011 18:07:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: debbugs.gnu.org X-GNU-PR-Keywords: patch To: undisclosed-recipients:; Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.131153080024594 (code B ref 9157); Sun, 24 Jul 2011 18:07:02 +0000 Received: (at 9157) by debbugs.gnu.org; 24 Jul 2011 18:06:40 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ql34X-0006Oa-A5 for submit@debbugs.gnu.org; Sun, 24 Jul 2011 14:06:40 -0400 Received: from host.romanr.info ([82.146.61.98]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ql1l7-0008Bw-0o for 9157@debbugs.gnu.org; Sun, 24 Jul 2011 12:42:31 -0400 Received: from firewall.localdomain (withheld) by host.romanr.info (Postfix) with ESMTPSA id 5639764256B4 for <9157@debbugs.gnu.org>; Sun, 24 Jul 2011 20:42:21 +0400 (MSD) Received: from roma.localdomain (withheld) by firewall.localdomain (Postfix) with ESMTP id 422609D9F for <9157@debbugs.gnu.org>; Sun, 24 Jul 2011 20:42:19 +0400 (MSD) Received: from test (withheld) by roma.localdomain (Postfix) with SMTP id E96DE7CA9 for <9157@debbugs.gnu.org>; Sun, 24 Jul 2011 20:42:18 +0400 (MSD) X-Mailbox-Line: From 06b0e06096664f62e707d3cdb4e6377ab4909517 Mon Sep 17 00:00:00 2001 From: Roman Rybalko Date: Sat, 23 Jul 2011 20:08:01 +0400 Message-Id: <20110724164218.E96DE7CA9@roma.localdomain> X-Spam-Score: -1.6 (-) X-Mailman-Approved-At: Sun, 24 Jul 2011 14:06:33 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -0.9 (/) --- doc/coreutils.texi | 4 +++ src/dd.c | 22 +++++++++++++++- tests/Makefile.am | 1 + tests/dd/sparse | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 96 insertions(+), 1 deletions(-) create mode 100755 tests/dd/sparse diff --git a/doc/coreutils.texi b/doc/coreutils.texi index 424446c..761c698 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. When used with @samp{block} or @samp{unblock}, pad with spaces instead of zero bytes. +@item sparse +@opindex sparse +Make sparse output file. + @end table The following ``conversions'' are really file flags diff --git a/src/dd.c b/src/dd.c index 0824f6c..0393740 100644 --- a/src/dd.c +++ b/src/dd.c @@ -126,7 +126,8 @@ enum C_NOCREAT = 010000, C_EXCL = 020000, C_FDATASYNC = 040000, - C_FSYNC = 0100000 + C_FSYNC = 0100000, + C_SPARSE = 0200000 }; /* Status bit masks. */ @@ -268,6 +269,7 @@ static struct symbol_value const conversions[] = {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ {"fdatasync", C_FDATASYNC}, /* Synchronize output data before finishing. */ {"fsync", C_FSYNC}, /* Also synchronize output metadata. */ + {"sparse", C_SPARSE}, /* Make sparse output file. */ {"", 0} }; @@ -533,6 +535,9 @@ Each CONV symbol may be:\n\ fsync likewise, but also write metadata\n\ "), stdout); fputs (_("\ + sparse make sparse output file\n\ +"), stdout); + fputs (_("\ \n\ Each FLAG symbol may be:\n\ \n\ @@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size) { ssize_t nwritten; process_signals (); + if (conversions_mask & C_SPARSE) + { + off_t seek_size = 0; + while (total_written + seek_size < size && buf[total_written + seek_size] == 0) + ++seek_size; + if (seek_size) + { + off_t cur_off = 0; + cur_off = lseek(fd, seek_size, SEEK_CUR); + if (cur_off < 0) + break; + total_written += seek_size; + continue; + } + } nwritten = write (fd, buf + total_written, size - total_written); if (nwritten < 0) { diff --git a/tests/Makefile.am b/tests/Makefile.am index ebd1b11..0f1376a 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -364,6 +364,7 @@ TESTS = \ dd/skip-seek \ dd/skip-seek2 \ dd/skip-seek-past-file \ + dd/sparse \ dd/stderr \ dd/unblock \ dd/unblock-sync \ diff --git a/tests/dd/sparse b/tests/dd/sparse new file mode 100755 index 0000000..f0e0806 --- /dev/null +++ b/tests/dd/sparse @@ -0,0 +1,70 @@ +#!/bin/sh +# Ensure that dd conv=sparse works. + +# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/init.sh"; path_prepend_ ../src +print_ver_ dd + +# sometimes we may read less than 1M +dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample0`" = "1048576" ] || fail=1 +dd if=/dev/urandom of=sample1 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample1`" = "1048576" ] || fail=1 + +# test 1 +dd if=sample1 of=test1 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test1 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test1 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 1-1 +dd if=test1 of=out1-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-1`" = "`stat -c '%s %b %B' out1-1-check`" ] || fail=1 + +# test 1-2 +dd if=test1 of=out1-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=3 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-2`" = "`stat -c '%s %b %B' out1-2-check`" ] || fail=1 + +# test 1-3 +dd if=test1 of=out1-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-3-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-3`" = "`stat -c '%s %b %B' out1-3-check`" ] || fail=1 + +# test 2 +dd if=sample0 of=test2 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test2 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test2 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 2-1 +dd if=test2 of=out2-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-1-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-1`" = "`stat -c '%s %b %B' out2-1-check`" ] || fail=1 + +# test 2-2 +dd if=test2 of=out2-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-2-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-2`" = "`stat -c '%s %b %B' out2-2-check`" ] || fail=1 + +# test 2-3 +dd if=test2 of=out2-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-3-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-3`" = "`stat -c '%s %b %B' out2-3-check`" ] || fail=1 + +Exit $fail -- 1.7.0.4 From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Glenn Morris Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Sat, 06 Aug 2011 20:54:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: 9157@debbugs.gnu.org Cc: Roman Rybalko Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.131266402422793 (code B ref 9157); Sat, 06 Aug 2011 20:54:01 +0000 Received: (at 9157) by debbugs.gnu.org; 6 Aug 2011 20:53:44 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QpnsN-0005va-SU for submit@debbugs.gnu.org; Sat, 06 Aug 2011 16:53:44 -0400 Received: from fencepost.gnu.org ([140.186.70.10]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QpnsK-0005vS-NU for 9157@debbugs.gnu.org; Sat, 06 Aug 2011 16:53:42 -0400 Received: from localhost ([127.0.0.1]:57971) by fencepost.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QpnrT-0007IZ-AU; Sat, 06 Aug 2011 16:52:47 -0400 From: Glenn Morris References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> X-Spook: red noise Legion of Doom South Africa Mahmoud Ahmadinejad X-Ran: .N3OC-E3p+ozvDt1D;wGq)4Qx:931kq)[l/ng83!GuOoafXh4gjCNNEliLmyx#QeB(%_&3 X-Hue: white X-Attribution: GM Date: Sat, 06 Aug 2011 16:52:46 -0400 In-Reply-To: <20110724164218.E96DE7CA9@roma.localdomain> (Roman Rybalko's message of "Sat, 23 Jul 2011 20:08:01 +0400") Message-ID: User-Agent: Gnus (www.gnus.org), GNU Emacs (www.gnu.org/software/emacs/) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Spam-Score: -6.4 (------) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.4 (------) This bug was reported via bcc. Because of this, the bug tracking system did not know which package it was associated with and assigned it to debbugs.gnu.org. I have reassigned it to the coreutils package and am sending this so that the bug-coreutils mailing list sees the report. See the complete report at http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9157 Roman Rybalko wrote: > --- > doc/coreutils.texi | 4 +++ > src/dd.c | 22 +++++++++++++++- > tests/Makefile.am | 1 + > tests/dd/sparse | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > 4 files changed, 96 insertions(+), 1 deletions(-) > create mode 100755 tests/dd/sparse > > diff --git a/doc/coreutils.texi b/doc/coreutils.texi > index 424446c..761c698 100644 > --- a/doc/coreutils.texi > +++ b/doc/coreutils.texi > @@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. > When used with @samp{block} or @samp{unblock}, pad with spaces instead of > zero bytes. > > +@item sparse > +@opindex sparse > +Make sparse output file. > + > @end table > > The following ``conversions'' are really file flags > diff --git a/src/dd.c b/src/dd.c > index 0824f6c..0393740 100644 > --- a/src/dd.c > +++ b/src/dd.c > @@ -126,7 +126,8 @@ enum > C_NOCREAT = 010000, > C_EXCL = 020000, > C_FDATASYNC = 040000, > - C_FSYNC = 0100000 > + C_FSYNC = 0100000, > + C_SPARSE = 0200000 > }; > > /* Status bit masks. */ > @@ -268,6 +269,7 @@ static struct symbol_value const conversions[] = > {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ > {"fdatasync", C_FDATASYNC}, /* Synchronize output data before finishing. */ > {"fsync", C_FSYNC}, /* Also synchronize output metadata. */ > + {"sparse", C_SPARSE}, /* Make sparse output file. */ > {"", 0} > }; > > @@ -533,6 +535,9 @@ Each CONV symbol may be:\n\ > fsync likewise, but also write metadata\n\ > "), stdout); > fputs (_("\ > + sparse make sparse output file\n\ > +"), stdout); > + fputs (_("\ > \n\ > Each FLAG symbol may be:\n\ > \n\ > @@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size) > { > ssize_t nwritten; > process_signals (); > + if (conversions_mask & C_SPARSE) > + { > + off_t seek_size = 0; > + while (total_written + seek_size < size && buf[total_written + seek_size] == 0) > + ++seek_size; > + if (seek_size) > + { > + off_t cur_off = 0; > + cur_off = lseek(fd, seek_size, SEEK_CUR); > + if (cur_off < 0) > + break; > + total_written += seek_size; > + continue; > + } > + } > nwritten = write (fd, buf + total_written, size - total_written); > if (nwritten < 0) > { > diff --git a/tests/Makefile.am b/tests/Makefile.am > index ebd1b11..0f1376a 100644 > --- a/tests/Makefile.am > +++ b/tests/Makefile.am > @@ -364,6 +364,7 @@ TESTS = \ > dd/skip-seek \ > dd/skip-seek2 \ > dd/skip-seek-past-file \ > + dd/sparse \ > dd/stderr \ > dd/unblock \ > dd/unblock-sync \ > diff --git a/tests/dd/sparse b/tests/dd/sparse > new file mode 100755 > index 0000000..f0e0806 > --- /dev/null > +++ b/tests/dd/sparse > @@ -0,0 +1,70 @@ > +#!/bin/sh > +# Ensure that dd conv=sparse works. > + > +# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc. > + > +# This program is free software: you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation, either version 3 of the License, or > +# (at your option) any later version. > + > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > + > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see . > + > +. "${srcdir=.}/init.sh"; path_prepend_ ../src > +print_ver_ dd > + > +# sometimes we may read less than 1M > +dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c %s sample0`" = "1048576" ] || fail=1 > +dd if=/dev/urandom of=sample1 count=1 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c %s sample1`" = "1048576" ] || fail=1 > + > +# test 1 > +dd if=sample1 of=test1 seek=0 bs=1M 2> /dev/null || fail=1 > +dd if=sample0 of=test1 seek=1 bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=test1 seek=2 bs=1M 2> /dev/null || fail=1 > + > +# test 1-1 > +dd if=test1 of=out1-1 conv=sparse bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out1-1-check bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out1-1-check seek=2 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c '%s %b %B' out1-1`" = "`stat -c '%s %b %B' out1-1-check`" ] || fail=1 > + > +# test 1-2 > +dd if=test1 of=out1-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out1-2-check seek=1 bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out1-2-check seek=3 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c '%s %b %B' out1-2`" = "`stat -c '%s %b %B' out1-2-check`" ] || fail=1 > + > +# test 1-3 > +dd if=test1 of=out1-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out1-3-check seek=2 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c '%s %b %B' out1-3`" = "`stat -c '%s %b %B' out1-3-check`" ] || fail=1 > + > +# test 2 > +dd if=sample0 of=test2 seek=0 bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=test2 seek=1 bs=1M 2> /dev/null || fail=1 > +dd if=sample0 of=test2 seek=2 bs=1M 2> /dev/null || fail=1 > + > +# test 2-1 > +dd if=test2 of=out2-1 conv=sparse bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out2-1-check seek=1 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c '%s %b %B' out2-1`" = "`stat -c '%s %b %B' out2-1-check`" ] || fail=1 > + > +# test 2-2 > +dd if=test2 of=out2-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out2-2-check seek=2 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c '%s %b %B' out2-2`" = "`stat -c '%s %b %B' out2-2-check`" ] || fail=1 > + > +# test 2-3 > +dd if=test2 of=out2-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out2-3-check seek=1 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c '%s %b %B' out2-3`" = "`stat -c '%s %b %B' out2-3-check`" ] || fail=1 > + > +Exit $fail From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Jim Meyering Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Sat, 10 Dec 2011 10:11:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Roman Rybalko Cc: 9157@debbugs.gnu.org Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.132351182425293 (code B ref 9157); Sat, 10 Dec 2011 10:11:03 +0000 Received: (at 9157) by debbugs.gnu.org; 10 Dec 2011 10:10:24 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RZJsu-0006Zu-6s for submit@debbugs.gnu.org; Sat, 10 Dec 2011 05:10:24 -0500 Received: from mx.meyering.net ([88.168.87.75]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RZJsp-0006Zc-Kz for 9157@debbugs.gnu.org; Sat, 10 Dec 2011 05:10:22 -0500 Received: from rho.meyering.net (localhost.localdomain [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id AB77860092; Sat, 10 Dec 2011 11:09:18 +0100 (CET) From: Jim Meyering In-Reply-To: <20110724164218.E96DE7CA9@roma.localdomain> (Roman Rybalko's message of "Sat, 23 Jul 2011 20:08:01 +0400") References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> Date: Sat, 10 Dec 2011 11:09:18 +0100 Message-ID: <87iplohh69.fsf@rho.meyering.net> Lines: 168 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.7 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.7 (--) Roman Rybalko wrote: > --- > doc/coreutils.texi | 4 +++ > src/dd.c | 22 +++++++++++++++- > tests/Makefile.am | 1 + > tests/dd/sparse | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > 4 files changed, 96 insertions(+), 1 deletions(-) > create mode 100755 tests/dd/sparse Thank you for the patch. It's nice to see accompanying test suite changes. I see that copyright paperwork is now on file with the FSF, so... Did you notice that cp now has the capability (using extents) to preserve holes efficiently on some file system types? Recent kernels also have SEEK_HOLE/SEEK_DATA support with which it is more efficient to detect holes. Here are some things we'll have to consider before adding a new hole-punching option to dd: Your patch may create a hole in the destination for each sequence of length seek_size or greater of zero bytes in the input. As you may have seen in the cp-related discussion, one may want different options: - preserve a file's hole/non-hole structure - efficiently detect existing holes and fill them with explicit zeros in dest - efficiently detect existing holes and seek-in-dest for each sequence of zeros (longer than some minimum) in non-hole input > diff --git a/doc/coreutils.texi b/doc/coreutils.texi > index 424446c..761c698 100644 > --- a/doc/coreutils.texi > +++ b/doc/coreutils.texi > @@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. > When used with @samp{block} or @samp{unblock}, pad with spaces instead of > zero bytes. > > +@item sparse > +@opindex sparse > +Make sparse output file. Please say a little more here. I.e., when might a hole be introduced? When is this option useful? > @end table > > The following ``conversions'' are really file flags > diff --git a/src/dd.c b/src/dd.c > index 0824f6c..0393740 100644 > --- a/src/dd.c > +++ b/src/dd.c > @@ -126,7 +126,8 @@ enum > C_NOCREAT = 010000, > C_EXCL = 020000, > C_FDATASYNC = 040000, > - C_FSYNC = 0100000 > + C_FSYNC = 0100000, > + C_SPARSE = 0200000 > }; > > /* Status bit masks. */ > @@ -268,6 +269,7 @@ static struct symbol_value const conversions[] = > {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ > {"fdatasync", C_FDATASYNC}, /* Synchronize output data before finishing. */ > {"fsync", C_FSYNC}, /* Also synchronize output metadata. */ > + {"sparse", C_SPARSE}, /* Make sparse output file. */ > {"", 0} > }; > > @@ -533,6 +535,9 @@ Each CONV symbol may be:\n\ > fsync likewise, but also write metadata\n\ > "), stdout); > fputs (_("\ > + sparse make sparse output file\n\ > +"), stdout); > + fputs (_("\ > \n\ > Each FLAG symbol may be:\n\ > \n\ > @@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size) > { > ssize_t nwritten; > process_signals (); > + if (conversions_mask & C_SPARSE) > + { > + off_t seek_size = 0; > + while (total_written + seek_size < size && buf[total_written + seek_size] == 0) > + ++seek_size; > + if (seek_size) > + { > + off_t cur_off = 0; > + cur_off = lseek(fd, seek_size, SEEK_CUR); > + if (cur_off < 0) > + break; dd must not ignore lseek failure. > + total_written += seek_size; > + continue; > + } > + } > nwritten = write (fd, buf + total_written, size - total_written); > if (nwritten < 0) > { > diff --git a/tests/Makefile.am b/tests/Makefile.am > index ebd1b11..0f1376a 100644 > --- a/tests/Makefile.am > +++ b/tests/Makefile.am > @@ -364,6 +364,7 @@ TESTS = \ > dd/skip-seek \ > dd/skip-seek2 \ > dd/skip-seek-past-file \ > + dd/sparse \ > dd/stderr \ > dd/unblock \ > dd/unblock-sync \ > diff --git a/tests/dd/sparse b/tests/dd/sparse > new file mode 100755 > index 0000000..f0e0806 > --- /dev/null > +++ b/tests/dd/sparse > @@ -0,0 +1,70 @@ > +#!/bin/sh > +# Ensure that dd conv=sparse works. > + > +# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc. Use only 2011 as the copyright year. > +# This program is free software: you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation, either version 3 of the License, or > +# (at your option) any later version. > + > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > + > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see . > + > +. "${srcdir=.}/init.sh"; path_prepend_ ../src > +print_ver_ dd > + > +# sometimes we may read less than 1M > +dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c %s sample0`" = "1048576" ] || fail=1 We'd write that like this instead: (note use of test, not "[...]", use of $(...), not `...`) test "$(stat -c %s sample0)" = 1048576 || fail=1 > +dd if=/dev/urandom of=sample1 count=1 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c %s sample1`" = "1048576" ] || fail=1 > + > +# test 1 > +dd if=sample1 of=test1 seek=0 bs=1M 2> /dev/null || fail=1 > +dd if=sample0 of=test1 seek=1 bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=test1 seek=2 bs=1M 2> /dev/null || fail=1 > + > +# test 1-1 > +dd if=test1 of=out1-1 conv=sparse bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out1-1-check bs=1M 2> /dev/null || fail=1 > +dd if=sample1 of=out1-1-check seek=2 bs=1M 2> /dev/null || fail=1 > +[ "`stat -c '%s %b %B' out1-1`" = "`stat -c '%s %b %B' out1-1-check`" ] || fail=1 From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: "Roman Rybalko (devel)" Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Mon, 12 Dec 2011 00:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Jim Meyering Cc: 9157@debbugs.gnu.org Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.132364797230763 (code B ref 9157); Mon, 12 Dec 2011 00:00:02 +0000 Received: (at 9157) by debbugs.gnu.org; 11 Dec 2011 23:59:32 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RZtIq-000807-3m for submit@debbugs.gnu.org; Sun, 11 Dec 2011 18:59:32 -0500 Received: from host.romanr.info ([82.146.61.98]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RZpd1-0002mP-Sg for 9157@debbugs.gnu.org; Sun, 11 Dec 2011 15:04:09 -0500 Received: from (withheld) by host.romanr.info (Postfix) with ESMTPSA id BA5496BA28A7; Mon, 12 Dec 2011 00:02:55 +0400 (MSK) Message-ID: <4EE50C6B.9000403@romanr.info> Date: Mon, 12 Dec 2011 00:02:51 +0400 From: "Roman Rybalko (devel)" User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> In-Reply-To: <87iplohh69.fsf@rho.meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -2.1 (--) X-Mailman-Approved-At: Sun, 11 Dec 2011 18:59:30 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.2 (--) On 10.12.2011 14:09, Jim Meyering wrote: > Here are some things we'll have to consider before > adding a new hole-punching option to dd: > > Your patch may create a hole in the destination for each sequence of > length seek_size or greater of zero bytes in the input. > As you may have seen in the cp-related discussion, one may > want different options: > - preserve a file's hole/non-hole structure > - efficiently detect existing holes and fill them with explicit zeros in dest > - efficiently detect existing holes and seek-in-dest for each sequence of > zeros (longer than some minimum) in non-hole input Okay, I'll think about that. That's a clear task. >> diff --git a/doc/coreutils.texi b/doc/coreutils.texi >> index 424446c..761c698 100644 >> --- a/doc/coreutils.texi >> +++ b/doc/coreutils.texi >> @@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. >> When used with @samp{block} or @samp{unblock}, pad with spaces instead of >> zero bytes. >> >> +@item sparse >> +@opindex sparse >> +Make sparse output file. > Please say a little more here. > I.e., when might a hole be introduced? > When is this option useful? Okay. >> @@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size) >> { >> ssize_t nwritten; >> process_signals (); >> + if (conversions_mask & C_SPARSE) >> + { >> + off_t seek_size = 0; >> + while (total_written + seek_size < size && buf[total_written + seek_size] == 0) >> + ++seek_size; >> + if (seek_size) >> + { >> + off_t cur_off = 0; >> + cur_off = lseek(fd, seek_size, SEEK_CUR); >> + if (cur_off < 0) >> + break; > dd must not ignore lseek failure. That's a problem for me. How would be suitable to handle lseek failure? Perhaps with new kernel API this code may be obsoleted. >> @@ -0,0 +1,70 @@ >> +#!/bin/sh >> +# Ensure that dd conv=sparse works. >> + >> +# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc. > Use only 2011 as the copyright year. Okay. >> +# sometimes we may read less than 1M >> +dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1 >> +[ "`stat -c %s sample0`" = "1048576" ] || fail=1 > We'd write that like this instead: > (note use of test, not "[...]", use of $(...), not `...`) > > test "$(stat -c %s sample0)" = 1048576 || fail=1 Okay. -- WBR, Roman Rybalko From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Mon, 27 Feb 2012 15:23:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: "Roman Rybalko (devel)" Cc: 9157@debbugs.gnu.org, Jim Meyering Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.13303561791925 (code B ref 9157); Mon, 27 Feb 2012 15:23:01 +0000 Received: (at 9157) by debbugs.gnu.org; 27 Feb 2012 15:22:59 +0000 Received: from localhost ([127.0.0.1]:59937 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S22Pi-0000Uz-Hn for submit@debbugs.gnu.org; Mon, 27 Feb 2012 10:22:59 -0500 Received: from mail2.vodafone.ie ([213.233.128.44]:3449) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S22Pe-0000Up-N8 for 9157@debbugs.gnu.org; Mon, 27 Feb 2012 10:22:56 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvgCAN6dS09tThHq/2dsb2JhbAAMNrIfA4N+AQEBBHkQCw0EAwECChYPCQMCAQIBPQgGDQEFAgEBFod3t2YEixSBZBYCBQZNDwEKhRMDBzIjg0EEjk+MbYxv Received: from unknown (HELO [192.168.1.79]) ([109.78.17.234]) by mail2.vodafone.ie with ESMTP; 27 Feb 2012 15:20:00 +0000 Message-ID: <4F4B9F20.2070001@draigBrady.com> Date: Mon, 27 Feb 2012 15:20:00 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> In-Reply-To: <4EE50C6B.9000403@romanr.info> X-Enigmail-Version: 1.3.2 Content-Type: multipart/mixed; boundary="------------080706030007030807060302" X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) This is a multi-part message in MIME format. --------------080706030007030807060302 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit I recently noticed a use case for this feature: https://review.openstack.org/#change,4435 But I managed to forget we had a patch pending for this, and spent a couple of hours writing my own version :p Well at least it's good practise for reviewing Roman's... I notice Roman's seeks() the output for any run of NULs, while mine only considers blocks of the full output block size. Checking the full block is a bit more CPU efficient, and gives a bit more control, so I'm marginally leaning towards doing that? Also Roman's doesn't handle the case where a seek is done at the end of the file. In that case an ftruncate() or write() is needed to correctly set the size. Notes on my version attached are: I first need to refactor is_nul() for use by cp too. My version is advisory also We may need to coalesce seeks to larger ones? something like cache_round()? I thought it better to keep the code simple in this regard though as it's probably not of practical concern. I used conv= for bsd compat, rather than oflag=. Needs tests and docs yet. cheers, P�draig. --------------080706030007030807060302 Content-Type: text/x-patch; name="dd-sparse.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="dd-sparse.patch" >From 8ea0d03c3dc6f77a25c77467a20da0eb9944ee9f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?P=C3=A1draig=20Brady?= Date: Mon, 27 Feb 2012 13:53:07 +0000 Subject: [PATCH] dd: add support for the conv=sparse option Small seeks are not coalesced to larger ones (like is done in cache_round() for example, for the moment at least. conv= is used rather then oflag= for FreeBSD compatibility. * src/dd.c (last_seek): A new global boolean to flag whether the last "write" was converted to a seek. (usage): Describe the new conf=sparse option. (iwrite): Convert a write of a NUL block to a seek if requested. (do_copy): Initialize the output buffer to have a sentinel, to allow for efficient testing for NUL output blocks. If the last block in the file was converted to a seek, then convert back to a write so the size ip updated. * NEWS: Mention the new feature. --- NEWS | 3 ++ src/dd.c | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 84 insertions(+), 3 deletions(-) diff --git a/NEWS b/NEWS index e2e8fc5..8006669 100644 --- a/NEWS +++ b/NEWS @@ -7,6 +7,9 @@ GNU coreutils NEWS -*- outline -*- dd now accepts the count_bytes, skip_bytes iflags and the seek_bytes oflag, to more easily allow processing portions of a file. + dd now accepts the conv=sparse flag to attempt to create sparse + output, by seeking rather than writing to the output file. + split now accepts an optional "from" argument to --numeric-suffixes, which changes the start number from the default of 0. diff --git a/src/dd.c b/src/dd.c index fe44a30..3c84567 100644 --- a/src/dd.c +++ b/src/dd.c @@ -126,7 +126,8 @@ enum C_NOCREAT = 010000, C_EXCL = 020000, C_FDATASYNC = 040000, - C_FSYNC = 0100000 + C_FSYNC = 0100000, + C_SPARSE = 0200000 }; /* Status bit masks. */ @@ -167,6 +168,9 @@ static uintmax_t seek_records = 0; output. */ static uintmax_t seek_bytes = 0; +/* Whether the last output was done with a seek (rather than a write). */ +static bool last_seek; + /* Copy only this many records. The default is effectively infinity. */ static uintmax_t max_records = (uintmax_t) -1; @@ -271,6 +275,7 @@ static struct symbol_value const conversions[] = {"unblock", C_UNBLOCK | C_TWOBUFS}, /* Fixed to variable length records. */ {"lcase", C_LCASE | C_TWOBUFS}, /* Translate upper to lower case. */ {"ucase", C_UCASE | C_TWOBUFS}, /* Translate lower to upper case. */ + {"sparse", C_SPARSE}, /* Sparsify output. */ {"swab", C_SWAB | C_TWOBUFS}, /* Swap bytes of input. */ {"noerror", C_NOERROR}, /* Ignore i/o errors. */ {"nocreat", C_NOCREAT}, /* Do not create output file. */ @@ -548,6 +553,7 @@ Each CONV symbol may be:\n\ unblock replace trailing spaces in cbs-size records with newline\n\ lcase change upper case to lower case\n\ ucase change lower case to upper case\n\ + sparse try to seek rather than write the output for NUL input blocks\n\ swab swap every pair of input bytes\n\ sync pad every input block with NULs to ibs-size; when used\n\ with block or unblock, pad with spaces rather than NULs\n\ @@ -989,6 +995,27 @@ iread_fullblock (int fd, char *buf, size_t size) return nread; } +/* Return whether the buffer consists entirely of NULs. + Note the word after the buffer must be non NUL. */ + +static bool _GL_ATTRIBUTE_PURE +is_nul (const char* buf, size_t bufsize) +{ + typedef uintptr_t word; + + /* Find first nonzero *word*, or the word with the sentinel. */ + word *wp = (word *) buf; + while (*wp++ == 0) + continue; + + /* Find the first nonzero *byte*, or the sentinel. */ + char *cp = (char *) (wp - 1); + while (*cp++ == 0) + continue; + + return cp > buf + bufsize; +} + /* Write to FD the buffer BUF of size SIZE, processing any signals that arrive. Return the number of bytes written, setting errno if this is less than SIZE. Keep trying if there are partial @@ -1020,9 +1047,30 @@ iwrite (int fd, char const *buf, size_t size) while (total_written < size) { - ssize_t nwritten; + ssize_t nwritten = 0; process_signals (); - nwritten = write (fd, buf + total_written, size - total_written); + + last_seek = false; + if ((conversions_mask & C_SPARSE)) + { + if (is_nul (buf, size)) + { + if (lseek (fd, size, SEEK_CUR) < 0) + { + conversions_mask &= ~C_SPARSE; + /* Don't warn about the advisory sparse request. */ + } + else + { + last_seek = true; + nwritten = size; + } + } + } + + if (!nwritten) + nwritten = write (fd, buf + total_written, size - total_written); + if (nwritten < 0) { if (errno != EINTR) @@ -1861,6 +1909,10 @@ dd_copy (void) obuf = ibuf; } + /* write sentinel to slop after the buffer, + to allow efficient checking for NUL blocks. */ + memset (obuf + output_blocksize, 1, sizeof (uintptr_t)); + if (skip_records != 0 || skip_bytes != 0) { uintmax_t us_bytes = input_offset + (skip_records * input_blocksize) @@ -2072,6 +2124,32 @@ dd_copy (void) } } + /* if the last write was converted to a seek, then for a regular file, + write NUL to set the size. */ + if (last_seek) + { + struct stat stdout_stat; + if (fstat (STDOUT_FILENO, &stdout_stat) != 0) + { + error (0, errno, _("cannot fstat %s"), quote (output_file)); + return EXIT_FAILURE; + } + if (S_ISREG (stdout_stat.st_mode)) + { + if (lseek (STDOUT_FILENO, -1, SEEK_CUR) < 0) + { + error (0, errno, _("rewinding %s"), quote (output_file)); + return EXIT_FAILURE; + } + conversions_mask &= ~C_SPARSE; + if (iwrite (STDOUT_FILENO, "\0", 1) != 1) + { + error (0, errno, _("writing %s"), quote (output_file)); + return EXIT_FAILURE; + } + } + } + if ((conversions_mask & C_FDATASYNC) && fdatasync (STDOUT_FILENO) != 0) { if (errno != ENOSYS && errno != EINVAL) -- 1.7.6.4 --------------080706030007030807060302-- From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Paul Eggert Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Mon, 27 Feb 2012 15:54:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: =?UTF-8?Q?P=C3=A1draig?= Brady Cc: 9157@debbugs.gnu.org, "Roman Rybalko \(devel\)" Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.13303580004685 (code B ref 9157); Mon, 27 Feb 2012 15:54:02 +0000 Received: (at 9157) by debbugs.gnu.org; 27 Feb 2012 15:53:20 +0000 Received: from localhost ([127.0.0.1]:59961 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S22t6-0001DW-2l for submit@debbugs.gnu.org; Mon, 27 Feb 2012 10:53:20 -0500 Received: from smtp.cs.ucla.edu ([131.179.128.62]:41371) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S22t2-0001DN-Rs for 9157@debbugs.gnu.org; Mon, 27 Feb 2012 10:53:18 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 6B81A39E800D; Mon, 27 Feb 2012 07:50:24 -0800 (PST) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id t5MCBPjlpoUR; Mon, 27 Feb 2012 07:50:24 -0800 (PST) Received: from [192.168.1.10] (pool-71-189-109-235.lsanca.fios.verizon.net [71.189.109.235]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 08E3239E800C; Mon, 27 Feb 2012 07:50:24 -0800 (PST) Message-ID: <4F4BA638.8080106@cs.ucla.edu> Date: Mon, 27 Feb 2012 07:50:16 -0800 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux i686; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> In-Reply-To: <4F4B9F20.2070001@draigBrady.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Thanks for looking into this! Some nits: On 02/27/2012 07:20 AM, P=C3=A1draig Brady wrote: > +is_nul (const char* buf, size_t bufsize) The usual spacing style is "const char *buf", since the "*" really belongs to the "buf" in C. > + memset (obuf + output_blocksize, 1, sizeof (uintptr_t)); This constraint on OUTPUT_BLOCK_SLOP should be documented, thus: #define OUTPUT_BLOCK_SLOP MAX (sizeof (uintptr_t), page_size - 1) > + /* write sentinel to slop after the buffer, > + to allow efficient checking for NUL blocks. */ Initial cap for sentence. The next comment has a similar issue. From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Mon, 27 Feb 2012 16:14:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Paul Eggert Cc: 9157@debbugs.gnu.org, "Roman Rybalko \(devel\)" Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.13303592126438 (code B ref 9157); Mon, 27 Feb 2012 16:14:02 +0000 Received: (at 9157) by debbugs.gnu.org; 27 Feb 2012 16:13:32 +0000 Received: from localhost ([127.0.0.1]:59971 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S23Cd-0001fl-Oq for submit@debbugs.gnu.org; Mon, 27 Feb 2012 11:13:31 -0500 Received: from mail2.vodafone.ie ([213.233.128.44]:22661) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S23Ca-0001fX-Se for 9157@debbugs.gnu.org; Mon, 27 Feb 2012 11:13:30 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjsDALeqS09tThHq/2dsb2JhbAAMNw6FK6sWgWIDg34BAQEEIw8BRhALDQsCAgUWCwICCQMCAQIBRQYNAQcBAbYFiiCBL4llgWQkEAsGBAcDAQQDBQkHDwEKAwMDAoUIAweDAIEWBJs8jDg3 Received: from unknown (HELO [192.168.1.79]) ([109.78.17.234]) by mail2.vodafone.ie with ESMTP; 27 Feb 2012 16:10:24 +0000 Message-ID: <4F4BAAEF.5060606@draigBrady.com> Date: Mon, 27 Feb 2012 16:10:23 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> In-Reply-To: <4F4BA638.8080106@cs.ucla.edu> X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 02/27/2012 03:50 PM, Paul Eggert wrote: > Thanks for looking into this! Some nits: > > On 02/27/2012 07:20 AM, Pádraig Brady wrote: >> +is_nul (const char* buf, size_t bufsize) > > The usual spacing style is "const char *buf", since the > "*" really belongs to the "buf" in C. > >> + memset (obuf + output_blocksize, 1, sizeof (uintptr_t)); > > This constraint on OUTPUT_BLOCK_SLOP should be documented, thus: > > #define OUTPUT_BLOCK_SLOP MAX (sizeof (uintptr_t), page_size - 1) > >> + /* write sentinel to slop after the buffer, >> + to allow efficient checking for NUL blocks. */ > > Initial cap for sentence. The next comment has a similar issue. > Thanks Paul. Adjustments made. I'm also thinking I should warn once when the new lseek fails, but only after the fall back write() succeeds. cheers, Pádraig. From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Paul Eggert Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Mon, 27 Feb 2012 17:18:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: =?UTF-8?Q?P=C3=A1draig?= Brady Cc: 9157@debbugs.gnu.org, "Roman Rybalko \(devel\)" Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.133036302912116 (code B ref 9157); Mon, 27 Feb 2012 17:18:01 +0000 Received: (at 9157) by debbugs.gnu.org; 27 Feb 2012 17:17:09 +0000 Received: from localhost ([127.0.0.1]:59997 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S24CC-00039N-N5 for submit@debbugs.gnu.org; Mon, 27 Feb 2012 12:17:09 -0500 Received: from smtp.cs.ucla.edu ([131.179.128.62]:42403) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S24CB-00039E-4b for 9157@debbugs.gnu.org; Mon, 27 Feb 2012 12:17:08 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 2AEB739E800D; Mon, 27 Feb 2012 09:14:14 -0800 (PST) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZA7DJhZsxhBM; Mon, 27 Feb 2012 09:14:13 -0800 (PST) Received: from [192.168.1.10] (pool-71-189-109-235.lsanca.fios.verizon.net [71.189.109.235]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id BCD3639E800B; Mon, 27 Feb 2012 09:14:13 -0800 (PST) Message-ID: <4F4BB9E6.3020806@cs.ucla.edu> Date: Mon, 27 Feb 2012 09:14:14 -0800 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux i686; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> In-Reply-To: <4F4BAAEF.5060606@draigBrady.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 02/27/2012 08:10 AM, P=C3=A1draig Brady wrote: > I'm also thinking I should warn once when > the new lseek fails, but only after the fall back > write() succeeds. I wouldn't bother. Either diagnostic should suffice, and warning about the lseek would slow the app down a bit and make it more complicated. From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Mon, 27 Feb 2012 18:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Paul Eggert Cc: 9157@debbugs.gnu.org Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.133036614917200 (code B ref 9157); Mon, 27 Feb 2012 18:10:02 +0000 Received: (at 9157) by debbugs.gnu.org; 27 Feb 2012 18:09:09 +0000 Received: from localhost ([127.0.0.1]:60080 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S250X-0004TN-GA for submit@debbugs.gnu.org; Mon, 27 Feb 2012 13:09:09 -0500 Received: from mail2.vodafone.ie ([213.233.128.44]:57039) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S250V-0004TD-0K for 9157@debbugs.gnu.org; Mon, 27 Feb 2012 13:09:07 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvkCAL/FS09tThHq/2dsb2JhbAAMNw6FGasoggsDg38BAQEEIw8BRhALDQsCAgUWCwICCQMCAQIBRQYNAQcBAa4pkhWBL4llgWQkEAsGBAcDAQQDBQkHDwEKAwMBAgKFCAMHgwCBFgSbPIw4Nw Received: from unknown (HELO [192.168.1.79]) ([109.78.17.234]) by mail2.vodafone.ie with ESMTP; 27 Feb 2012 18:06:13 +0000 Message-ID: <4F4BC614.8080506@draigBrady.com> Date: Mon, 27 Feb 2012 18:06:12 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> In-Reply-To: <4F4BB9E6.3020806@cs.ucla.edu> X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 02/27/2012 05:14 PM, Paul Eggert wrote: > On 02/27/2012 08:10 AM, Pádraig Brady wrote: >> I'm also thinking I should warn once when >> the new lseek fails, but only after the fall back >> write() succeeds. > > I wouldn't bother. Either diagnostic should suffice, > and warning about the lseek would slow the app down > a bit and make it more complicated. Yep, I meant defer to the write() diagnostic. What I was wondering was whether to warn about the case where the seek fails but the write succeeds like here: dd conv=sparse bs=1 count=10 | ... I'll leave it as advisory for now, and not have a warning for the above case. cheers, Pádraig. From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 01:14:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Paul Eggert , roman.rybalko@romanr.info Cc: 9157@debbugs.gnu.org Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.133039159425726 (code B ref 9157); Tue, 28 Feb 2012 01:14:01 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 01:13:14 +0000 Received: from localhost ([127.0.0.1]:60469 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2Bcv-0006gf-Ve for submit@debbugs.gnu.org; Mon, 27 Feb 2012 20:13:14 -0500 Received: from mail2.vodafone.ie ([213.233.128.44]:18704) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2Bcj-0006fx-Q2 for 9157@debbugs.gnu.org; Mon, 27 Feb 2012 20:13:02 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApcBABooTE9tThHq/2dsb2JhbAAMNw6FGas2hg4BAQEEIw8BRhALDQsCAgUWCwICCQMCAQIBRQYBDAEHAQGuW5I0gS+LQggQDBALBgQHAwEEAwUJBw8BCgYDAoUIAweDAIEWBJs8jDg3 Received: from unknown (HELO [192.168.1.79]) ([109.78.17.234]) by mail2.vodafone.ie with ESMTP; 28 Feb 2012 01:09:51 +0000 Message-ID: <4F4C295F.7040400@draigBrady.com> Date: Tue, 28 Feb 2012 01:09:51 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> In-Reply-To: <4F4BC614.8080506@draigBrady.com> X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 02/27/2012 06:06 PM, Pádraig Brady wrote: > On 02/27/2012 05:14 PM, Paul Eggert wrote: >> On 02/27/2012 08:10 AM, Pádraig Brady wrote: >>> I'm also thinking I should warn once when >>> the new lseek fails, but only after the fall back >>> write() succeeds. >> >> I wouldn't bother. Either diagnostic should suffice, >> and warning about the lseek would slow the app down >> a bit and make it more complicated. > > Yep, I meant defer to the write() diagnostic. > > What I was wondering was whether to warn about > the case where the seek fails but the write succeeds like here: > > dd conv=sparse bs=1 count=10 | ... > > I'll leave it as advisory for now, and not have > a warning for the above case. Another adjustment I'll need to make is how to handle existing regular files with conv=trunc. I.E. seeking over existing possible non NUL data. It's too dangerous/inconsistent to do this for files I think. With devices we can support auto allocating virtual devices as mentioned elsewhere in the thread. We could do any of these for existing files with conv=sparse,notrunc 1. disallow conv=sparse,notrunc 2. ignore sparse (with warning) 3. allow sparse in the additional part of output file 4. detect holes in output file and maintain. I'm leaning towards 2 as that will allow us to change to 3 or 4 if deemed necessary in future. cheers, Pádraig. From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Paul Eggert Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 08:55:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: =?UTF-8?Q?P=C3=A1draig?= Brady Cc: 9157@debbugs.gnu.org, roman.rybalko@romanr.info Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.13304192483330 (code B ref 9157); Tue, 28 Feb 2012 08:55:01 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 08:54:08 +0000 Received: from localhost ([127.0.0.1]:60747 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2Ioy-0000r9-4B for submit@debbugs.gnu.org; Tue, 28 Feb 2012 03:54:08 -0500 Received: from smtp.cs.ucla.edu ([131.179.128.62]:39418) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2Ioc-0000qc-Ja for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 03:53:56 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 31B1339E800F; Tue, 28 Feb 2012 00:50:35 -0800 (PST) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x0vm0u1EB2fP; Tue, 28 Feb 2012 00:50:34 -0800 (PST) Received: from [192.168.1.10] (pool-71-189-109-235.lsanca.fios.verizon.net [71.189.109.235]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id A7DAA39E800C; Tue, 28 Feb 2012 00:50:34 -0800 (PST) Message-ID: <4F4C955C.4040500@cs.ucla.edu> Date: Tue, 28 Feb 2012 00:50:36 -0800 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux i686; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> In-Reply-To: <4F4C295F.7040400@draigBrady.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 02/27/2012 05:09 PM, P=C3=A1draig Brady wrote: > how > to handle existing regular files with conv=3Dtrunc. > I.E. seeking over existing possible non NUL data. > It's too dangerous/inconsistent to do this for files I think. Why? This is *dd* we're talking about here. It's *supposed* to be used for tricky stuff like this. If one interprets conv=3Dsparse to mean "write sparsely", rather than "create a sparse file that exactly mimics the input's sparseness", then everything should be clear, no? From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 11:07:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Paul Eggert Cc: 9157@debbugs.gnu.org, roman.rybalko@romanr.info Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.133042717614967 (code B ref 9157); Tue, 28 Feb 2012 11:07:02 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 11:06:16 +0000 Received: from localhost ([127.0.0.1]:60808 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2Ksp-0003t8-LF for submit@debbugs.gnu.org; Tue, 28 Feb 2012 06:06:15 -0500 Received: from mx1.redhat.com ([209.132.183.28]:19448) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2KsU-0003sI-BN for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 06:06:04 -0500 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q1SB2Zwo002001 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 28 Feb 2012 06:02:35 -0500 Received: from [10.36.116.68] (ovpn-116-68.ams2.redhat.com [10.36.116.68]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q1SB2VIF006655 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 28 Feb 2012 06:02:34 -0500 Message-ID: <4F4CB447.4050300@draigBrady.com> Date: Tue, 28 Feb 2012 11:02:31 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> In-Reply-To: <4F4C955C.4040500@cs.ucla.edu> X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=UTF-8 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id q1SB2Zwo002001 X-Spam-Score: -6.9 (------) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.9 (------) On 02/28/2012 08:50 AM, Paul Eggert wrote: > On 02/27/2012 05:09 PM, P=C3=A1draig Brady wrote: >> how >> to handle existing regular files with conv=3Dtrunc. >> I.E. seeking over existing possible non NUL data. >> It's too dangerous/inconsistent to do this for files I think. >=20 > Why? This is *dd* we're talking about here. > It's *supposed* to be used for tricky stuff like this. >=20 > If one interprets conv=3Dsparse to mean "write sparsely", > rather than "create a sparse file that exactly mimics > the input's sparseness", then everything should be clear, > no? Well I wasn't considering reproducing sparseness. I was thinking it would be more beneficial to be able to update a file in place, but do so sparsely if possible. I.E. support something like: truncate -r src.img backup.img && dd if=3Dsrc.img conv=3Dnotrunc,sparse of=3Dbackup.img To do that, one could not simply skip over the output for NUL input. Doing that would be "dangerous" as I said above, or in other words, surprising to users to not update possibly non NUL data in the output file. As for the "inconsistent" point I mentioned. The last patch currently just seeks the output for NUL input, _except_ for the last byte of the file if part of a NUL write. So that inconsistency means one couldn't use NUL input as a write mask so to speak, if one ever did want such an esoteric feature. I guess we could address the consistency aspect by using ftruncate() rather than the write(outfd, "\0", 1) technique (only doing that if we've extended the file). However I'm not sure such functionality is useful for files, and that we should try to align with the first operation? Note ftruncate() might be good to do anyway to avoid writes completely to a fully sparse file. Also I'm just thought about oflag=3Dappend. That will cause writes to ignore the seek beyond end of file. I.E. with the last patch, this will create a file.test with just "ab" printf "a\000\000b" | src/dd conv=3Dsparse bs=3D1 count=3D10 oflag=3Dappend > file.test I guess an ftruncate() would cater for that too, except in the case where another process is writing to the file, in which case you would might want to disallow the conv=3Dsparse oflag=3Dappend combo. We can't really ftruncate as we go as that would be racy, so I guess we should disallow the combo? cheers, P=C3=A1draig. From unknown Tue Sep 23 17:51:33 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.428 (Entity 5.428) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Roman Rybalko Subject: bug#9157: closed (Re: bug#9157: [PATCH] dd: sparse conv flag) Message-ID: References: <4F4D4BF7.6020404@draigBrady.com> <20110723163227.3C04B69E5E@roma.localdomain> X-Gnu-PR-Message: they-closed 9157 X-Gnu-PR-Package: coreutils X-Gnu-PR-Keywords: patch Reply-To: 9157@debbugs.gnu.org Date: Tue, 28 Feb 2012 21:51:03 +0000 Content-Type: multipart/mixed; boundary="----------=_1330465863-1177-1" This is a multi-part message in MIME format... ------------=_1330465863-1177-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #9157: [PATCH] dd: sparse conv flag which was filed against the coreutils package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 9157@debbugs.gnu.org. --=20 9157: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D9157 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1330465863-1177-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 9157-done) by debbugs.gnu.org; 28 Feb 2012 21:50:34 +0000 Received: from localhost ([127.0.0.1]:54663 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2UwL-0000I5-Au for submit@debbugs.gnu.org; Tue, 28 Feb 2012 16:50:34 -0500 Received: from mx1.redhat.com ([209.132.183.28]:13590) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2Uvv-0000H7-HO for 9157-done@debbugs.gnu.org; Tue, 28 Feb 2012 16:50:17 -0500 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q1SLnjl4021494 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 28 Feb 2012 16:49:46 -0500 Received: from [10.36.116.40] (ovpn-116-40.ams2.redhat.com [10.36.116.40]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q1SLngRq002436 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 28 Feb 2012 16:49:44 -0500 Message-ID: <4F4D4BF7.6020404@draigBrady.com> Date: Tue, 28 Feb 2012 21:49:43 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#9157: [PATCH] dd: sparse conv flag References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> <4F4CB447.4050300@draigBrady.com> <4F4D084E.6090907@draigBrady.com> <4F4D1BF8.8020902@cs.ucla.edu> In-Reply-To: <4F4D1BF8.8020902@cs.ucla.edu> X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=UTF-8 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id q1SLnjl4021494 X-Spam-Score: -6.9 (------) X-Debbugs-Envelope-To: 9157-done Cc: 9157-done@debbugs.gnu.org, roman.rybalko@romanr.info X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.9 (------) On 02/28/2012 06:24 PM, Paul Eggert wrote: > On 02/28/2012 09:01 AM, P=C3=A1draig Brady wrote: >=20 >> Doing that would be "dangerous" as I said above, >> or in other words, surprising to users to not update >> possibly non NUL data in the output file. >=20 > It's not surprising at all. It's what I want and > expect. For example, suppose every nonzero byte of file A > has an offset that is in a hole of file B. Then I should > be able to overlay A and B into another file C, as follows: >=20 > dd if=3DA of=3DC conv=3Dsparse > dd if=3DB of=3DC conv=3Dsparse,notrunc >=20 > This is akin to the "or" operation on files, and it'd be a nice > operation to have. Even if A's contents don't fit in B's holes, > I still might want to do the above, to allow B's non-holes to > override A's contents. Why disable this useful functionality? >=20 > Any surprise issues can be dealt with by documenting dd's > behavior appropriately. Fair enough. So this is the "write mask" functionality I referred to, and which is now supported consistently since the write(fd,"\0",1) to ftruncate() change. >> +Try to seek rather than write @sc{nul} output blocks. >> +This will create sparse output when extending. >> +This option is ignored in conjunction with >> +@samp{conv=3Dnotrunc} or @samp{oflag=3Dappend}. >=20 > I still dubious about this level of handholding. > dd is meant for low-level use, and as far as possible > options should be orthogonal. For example, with dd, > oflag=3Dappend does not disable seek=3DN -- both flags operate, > which means that the seek is ineffective unless it > is past the end of file. conv=3Dsparse oflag=3Dappend > should be similar: all it should mean is that the > writes are sparse when they're past the end of > the file (this latter functionality should work, but > doesn't work with the proposed patch). OK I'll remove the 2 restrictions, and essentially move the comments for them from code to texinfo. cheers, P=C3=A1draig. ------------=_1330465863-1177-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 23 Jul 2011 22:41:33 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qkkt0-0007SI-HY for submit@debbugs.gnu.org; Sat, 23 Jul 2011 18:41:32 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qkf87-0007lA-AA for submit@debbugs.gnu.org; Sat, 23 Jul 2011 12:32:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qkf80-0003qJ-QE for submit@debbugs.gnu.org; Sat, 23 Jul 2011 12:32:38 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:49617) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qkf80-0003qF-Oi for submit@debbugs.gnu.org; Sat, 23 Jul 2011 12:32:36 -0400 Received: from eggs.gnu.org ([140.186.70.92]:60155) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qkf7z-0001sK-0B for bug-coreutils@gnu.org; Sat, 23 Jul 2011 12:32:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qkf7w-0003q1-QQ for bug-coreutils@gnu.org; Sat, 23 Jul 2011 12:32:34 -0400 Received: from host.romanr.info ([82.146.61.98]:45136) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qkf7w-0003pp-CB for bug-coreutils@gnu.org; Sat, 23 Jul 2011 12:32:32 -0400 Received: from firewall.localdomain (withheld) by host.romanr.info (Postfix) with ESMTPSA id 2EF15641CE1A for ; Sat, 23 Jul 2011 20:32:21 +0400 (MSD) Received: from roma.localdomain (withheld) by firewall.localdomain (Postfix) with ESMTP id DABDF8685 for ; Sat, 23 Jul 2011 20:32:24 +0400 (MSD) Received: from test (withheld) by roma.localdomain (Postfix) with SMTP id 3C04B69E5E for ; Sat, 23 Jul 2011 20:32:27 +0400 (MSD) X-Mailbox-Line: From dd75e573a541d3fe046fb56d482d6419dd92b958 Mon Sep 17 00:00:00 2001 From: Roman Rybalko Date: Sat, 23 Jul 2011 20:08:01 +0400 Subject: [PATCH] dd: sparse conv flag Message-Id: <20110723163227.3C04B69E5E@roma.localdomain> To: undisclosed-recipients:; X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -3.9 (---) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Sat, 23 Jul 2011 18:41:29 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.9 (---) --- doc/coreutils.texi | 4 +++ src/dd.c | 22 +++++++++++++++- tests/Makefile.am | 1 + tests/dd/sparse | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 96 insertions(+), 1 deletions(-) create mode 100755 tests/dd/sparse diff --git a/doc/coreutils.texi b/doc/coreutils.texi index 424446c..761c698 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. When used with @samp{block} or @samp{unblock}, pad with spaces instead of zero bytes. +@item sparse +@opindex sparse +Make sparse output file. + @end table The following ``conversions'' are really file flags diff --git a/src/dd.c b/src/dd.c index 0824f6c..49847f5 100644 --- a/src/dd.c +++ b/src/dd.c @@ -126,7 +126,8 @@ enum C_NOCREAT = 010000, C_EXCL = 020000, C_FDATASYNC = 040000, - C_FSYNC = 0100000 + C_FSYNC = 0100000, + C_SPARSE = 0200000 }; /* Status bit masks. */ @@ -268,6 +269,7 @@ static struct symbol_value const conversions[] = {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ {"fdatasync", C_FDATASYNC}, /* Synchronize output data before finishing. */ {"fsync", C_FSYNC}, /* Also synchronize output metadata. */ + {"sparse", C_SPARSE}, /* Make sparse output file. */ {"", 0} }; @@ -533,6 +535,9 @@ Each CONV symbol may be:\n\ fsync likewise, but also write metadata\n\ "), stdout); fputs (_("\ + sparse make sparse output file\n\ +"), stdout); + fputs (_("\ \n\ Each FLAG symbol may be:\n\ \n\ @@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size) { ssize_t nwritten; process_signals (); + if (conversions_mask & C_SPARSE) + { + off_t seek_size = 0; + while (buf[total_written + seek_size] == 0) + ++seek_size; + if (seek_size) + { + off_t cur_off = 0; + cur_off = lseek(fd, seek_size, SEEK_CUR); + if (cur_off < 0) + break; + total_written += seek_size; + continue; + } + } nwritten = write (fd, buf + total_written, size - total_written); if (nwritten < 0) { diff --git a/tests/Makefile.am b/tests/Makefile.am index ebd1b11..0f1376a 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -364,6 +364,7 @@ TESTS = \ dd/skip-seek \ dd/skip-seek2 \ dd/skip-seek-past-file \ + dd/sparse \ dd/stderr \ dd/unblock \ dd/unblock-sync \ diff --git a/tests/dd/sparse b/tests/dd/sparse new file mode 100755 index 0000000..f0e0806 --- /dev/null +++ b/tests/dd/sparse @@ -0,0 +1,70 @@ +#!/bin/sh +# Ensure that dd conv=sparse works. + +# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/init.sh"; path_prepend_ ../src +print_ver_ dd + +# sometimes we may read less than 1M +dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample0`" = "1048576" ] || fail=1 +dd if=/dev/urandom of=sample1 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample1`" = "1048576" ] || fail=1 + +# test 1 +dd if=sample1 of=test1 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test1 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test1 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 1-1 +dd if=test1 of=out1-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-1`" = "`stat -c '%s %b %B' out1-1-check`" ] || fail=1 + +# test 1-2 +dd if=test1 of=out1-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=3 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-2`" = "`stat -c '%s %b %B' out1-2-check`" ] || fail=1 + +# test 1-3 +dd if=test1 of=out1-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-3-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-3`" = "`stat -c '%s %b %B' out1-3-check`" ] || fail=1 + +# test 2 +dd if=sample0 of=test2 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test2 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test2 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 2-1 +dd if=test2 of=out2-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-1-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-1`" = "`stat -c '%s %b %B' out2-1-check`" ] || fail=1 + +# test 2-2 +dd if=test2 of=out2-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-2-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-2`" = "`stat -c '%s %b %B' out2-2-check`" ] || fail=1 + +# test 2-3 +dd if=test2 of=out2-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-3-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-3`" = "`stat -c '%s %b %B' out2-3-check`" ] || fail=1 + +Exit $fail -- 1.7.0.4 ------------=_1330465863-1177-1-- From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 22:25:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Paul Eggert Cc: 9157@debbugs.gnu.org, roman.rybalko@romanr.info Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.13304678704258 (code B ref 9157); Tue, 28 Feb 2012 22:25:02 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 22:24:30 +0000 Received: from localhost ([127.0.0.1]:54730 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2VTB-00016J-Gh for submit@debbugs.gnu.org; Tue, 28 Feb 2012 17:24:30 -0500 Received: from mx1.redhat.com ([209.132.183.28]:49569) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2VSn-00015O-Oh for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 17:24:17 -0500 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q1SH17da012642 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 28 Feb 2012 12:01:07 -0500 Received: from [10.36.116.41] (ovpn-116-41.ams2.redhat.com [10.36.116.41]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q1SH12Yv027492 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 28 Feb 2012 12:01:05 -0500 Message-ID: <4F4D084E.6090907@draigBrady.com> Date: Tue, 28 Feb 2012 17:01:02 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> <4F4CB447.4050300@draigBrady.com> In-Reply-To: <4F4CB447.4050300@draigBrady.com> X-Enigmail-Version: 1.3.2 Content-Type: multipart/mixed; boundary="------------050605020305080706050805" X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11 X-Spam-Score: -6.9 (------) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.9 (------) This is a multi-part message in MIME format. --------------050605020305080706050805 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id q1SH17da012642 On 02/28/2012 11:02 AM, P=C3=A1draig Brady wrote: > On 02/28/2012 08:50 AM, Paul Eggert wrote: >> On 02/27/2012 05:09 PM, P=C3=A1draig Brady wrote: >>> how >>> to handle existing regular files with conv=3Dtrunc. >>> I.E. seeking over existing possible non NUL data. >>> It's too dangerous/inconsistent to do this for files I think. >> >> Why? This is *dd* we're talking about here. >> It's *supposed* to be used for tricky stuff like this. >> >> If one interprets conv=3Dsparse to mean "write sparsely", >> rather than "create a sparse file that exactly mimics >> the input's sparseness", then everything should be clear, >> no? >=20 > Well I wasn't considering reproducing sparseness. > I was thinking it would be more beneficial to be able to > update a file in place, but do so sparsely if possible. > I.E. support something like: >=20 > truncate -r src.img backup.img && > dd if=3Dsrc.img conv=3Dnotrunc,sparse of=3Dbackup.img >=20 > To do that, one could not simply skip over the output > for NUL input. Doing that would be "dangerous" as I said above, > or in other words, surprising to users to not update > possibly non NUL data in the output file. >=20 > As for the "inconsistent" point I mentioned. The last patch > currently just seeks the output for NUL input, > _except_ for the last byte of the file if part of a NUL write. > So that inconsistency means one couldn't use NUL input as > a write mask so to speak, if one ever did want such an > esoteric feature. I guess we could address the consistency aspect > by using ftruncate() rather than the write(outfd, "\0", 1) technique > (only doing that if we've extended the file). > However I'm not sure such functionality is useful for files, > and that we should try to align with the first operation? > Note ftruncate() might be good to do anyway to avoid > writes completely to a fully sparse file. >=20 > Also I'm just thought about oflag=3Dappend. > That will cause writes to ignore the seek beyond end of file. > I.E. with the last patch, this will create a file.test with just "ab" >=20 > printf "a\000\000b" | > src/dd conv=3Dsparse bs=3D1 count=3D10 oflag=3Dappend > file.test >=20 > I guess an ftruncate() would cater for that too, except in the case > where another process is writing to the file, in which case > you would might want to disallow the conv=3Dsparse oflag=3Dappend combo. > We can't really ftruncate as we go as that would be racy, > so I guess we should disallow the combo? Ok the attached should be pretty much complete. It disables conv=3Dsparse when used with conv=3Dnotrunc or oflag=3Dappend. Comments appreciated. Roman are you around? cheers, P=C3=A1draig. --------------050605020305080706050805 Content-Type: text/x-patch; name="dd-sparse.patch" Content-Disposition: attachment; filename="dd-sparse.patch" Content-Transfer-Encoding: 7bit >From 4698cf3cb3981c013e412a6080fe3ab1cab34656 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?P=C3=A1draig=20Brady?= Date: Mon, 27 Feb 2012 13:53:07 +0000 Subject: [PATCH] dd: add support for the conv=sparse option Small seeks are not coalesced to larger ones (like is done in cache_round() for example, for the moment at least. conv= is used rather then oflag= for FreeBSD compatibility. * src/dd.c (last_seek): A new global boolean to flag whether the last "write" was converted to a seek. (usage): Describe the new conf=sparse option. (scanargs): Ignore conv=sparse in some combinations. (iwrite): Convert a write of a NUL block to a seek if requested. (do_copy): Initialize the output buffer to have a sentinel, to allow for efficient testing for NUL output blocks. If the last block in the file was converted to a seek, then convert back to a write so the size ip updated. * NEWS: Mention the new feature. * tests/dd/sparse: A new test for the feature. * tests/Makefile.am: Reference the new test. --- NEWS | 3 + doc/coreutils.texi | 7 +++ src/dd.c | 104 ++++++++++++++++++++++++++++++++++++++++++++++++++-- tests/Makefile.am | 1 + tests/dd/sparse | 57 ++++++++++++++++++++++++++++ 5 files changed, 168 insertions(+), 4 deletions(-) create mode 100755 tests/dd/sparse diff --git a/NEWS b/NEWS index e2e8fc5..8006669 100644 --- a/NEWS +++ b/NEWS @@ -7,6 +7,9 @@ GNU coreutils NEWS -*- outline -*- dd now accepts the count_bytes, skip_bytes iflags and the seek_bytes oflag, to more easily allow processing portions of a file. + dd now accepts the conv=sparse flag to attempt to create sparse + output, by seeking rather than writing to the output file. + split now accepts an optional "from" argument to --numeric-suffixes, which changes the start number from the default of 0. diff --git a/doc/coreutils.texi b/doc/coreutils.texi index 414626d..f22e7d2 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -8140,6 +8140,13 @@ Change lowercase letters to uppercase. The @samp{lcase} and @samp{ucase} conversions are mutually exclusive. +@item sparse +@opindex sparse +Try to seek rather than write @sc{nul} output blocks. +This will create sparse output when extending. +This option is ignored in conjunction with +@samp{conv=notrunc} or @samp{oflag=append}. + @item swab @opindex swab @r{(byte-swapping)} @cindex byte-swapping diff --git a/src/dd.c b/src/dd.c index fe44a30..903346f 100644 --- a/src/dd.c +++ b/src/dd.c @@ -94,7 +94,7 @@ malloc. See dd_copy for details. INPUT_BLOCK_SLOP must be no less than OUTPUT_BLOCK_SLOP. */ #define INPUT_BLOCK_SLOP (2 * SWAB_ALIGN_OFFSET + 2 * page_size - 1) -#define OUTPUT_BLOCK_SLOP (page_size - 1) +#define OUTPUT_BLOCK_SLOP MAX (sizeof (uintptr_t), page_size - 1) /* Maximum blocksize for the given SLOP. Keep it smaller than SIZE_MAX - SLOP, so that we can @@ -126,7 +126,9 @@ enum C_NOCREAT = 010000, C_EXCL = 020000, C_FDATASYNC = 040000, - C_FSYNC = 0100000 + C_FSYNC = 0100000, + + C_SPARSE = 0200000 }; /* Status bit masks. */ @@ -167,6 +169,9 @@ static uintmax_t seek_records = 0; output. */ static uintmax_t seek_bytes = 0; +/* Whether the last output was done with a seek (rather than a write). */ +static bool last_seek; + /* Copy only this many records. The default is effectively infinity. */ static uintmax_t max_records = (uintmax_t) -1; @@ -271,6 +276,7 @@ static struct symbol_value const conversions[] = {"unblock", C_UNBLOCK | C_TWOBUFS}, /* Fixed to variable length records. */ {"lcase", C_LCASE | C_TWOBUFS}, /* Translate upper to lower case. */ {"ucase", C_UCASE | C_TWOBUFS}, /* Translate lower to upper case. */ + {"sparse", C_SPARSE}, /* Try to sparsely write output. */ {"swab", C_SWAB | C_TWOBUFS}, /* Swap bytes of input. */ {"noerror", C_NOERROR}, /* Ignore i/o errors. */ {"nocreat", C_NOCREAT}, /* Do not create output file. */ @@ -548,6 +554,7 @@ Each CONV symbol may be:\n\ unblock replace trailing spaces in cbs-size records with newline\n\ lcase change upper case to lower case\n\ ucase change lower case to upper case\n\ + sparse try to seek rather than write the output for NUL input blocks\n\ swab swap every pair of input bytes\n\ sync pad every input block with NULs to ibs-size; when used\n\ with block or unblock, pad with spaces rather than NULs\n\ @@ -989,6 +996,27 @@ iread_fullblock (int fd, char *buf, size_t size) return nread; } +/* Return whether the buffer consists entirely of NULs. + Note the word after the buffer must be non NUL. */ + +static bool _GL_ATTRIBUTE_PURE +is_nul (const char *buf, size_t bufsize) +{ + typedef uintptr_t word; + + /* Find first nonzero *word*, or the word with the sentinel. */ + word *wp = (word *) buf; + while (*wp++ == 0) + continue; + + /* Find the first nonzero *byte*, or the sentinel. */ + char *cp = (char *) (wp - 1); + while (*cp++ == 0) + continue; + + return cp > buf + bufsize; +} + /* Write to FD the buffer BUF of size SIZE, processing any signals that arrive. Return the number of bytes written, setting errno if this is less than SIZE. Keep trying if there are partial @@ -1020,9 +1048,28 @@ iwrite (int fd, char const *buf, size_t size) while (total_written < size) { - ssize_t nwritten; + ssize_t nwritten = 0; process_signals (); - nwritten = write (fd, buf + total_written, size - total_written); + + /* Perform a seek for a NUL block if sparse output is enabled. */ + last_seek = false; + if ((conversions_mask & C_SPARSE) && is_nul (buf, size)) + { + if (lseek (fd, size, SEEK_CUR) < 0) + { + conversions_mask &= ~C_SPARSE; + /* Don't warn about the advisory sparse request. */ + } + else + { + last_seek = true; + nwritten = size; + } + } + + if (!nwritten) + nwritten = write (fd, buf + total_written, size - total_written); + if (nwritten < 0) { if (errno != EINTR) @@ -1321,6 +1368,24 @@ scanargs (int argc, char *const *argv) || multiple_bits_set (output_flags & (O_DIRECT | O_NOCACHE))) error (EXIT_FAILURE, 0, _("cannot combine direct and nocache")); + if ((output_flags & O_APPEND) && (conversions_mask & C_SPARSE)) + { + /* Disable sparse seeks as they'll be noops with O_APPEND. + Also they will be relative to the last write + (start of file if no writes) rather than EOF. + This might be catered for with ftruncate and SEEK_END, + but not in an atomic way across processes. */ + error (0, 0, _("oflag=append disables conv=sparse")); + conversions_mask &= ~C_SPARSE; + } + if ((conversions_mask & C_NOTRUNC) && (conversions_mask & C_SPARSE)) + { + /* Disable sparse writing when not truncating the output file, + so that non NUL parts of the output are overwritten by NUL input. */ + error (0, 0, _("conv=notrunc disables conv=sparse")); + conversions_mask &= ~C_SPARSE; + } + if (input_flags & O_NOCACHE) { i_nocache = true; @@ -1861,6 +1926,10 @@ dd_copy (void) obuf = ibuf; } + /* Write sentinel to slop after the buffer, + to allow efficient checking for NUL blocks. */ + memset (obuf + output_blocksize, 1, sizeof (uintptr_t)); + if (skip_records != 0 || skip_bytes != 0) { uintmax_t us_bytes = input_offset + (skip_records * input_blocksize) @@ -2072,6 +2141,33 @@ dd_copy (void) } } + /* If the last write was converted to a seek, then for a regular file, + ftruncate to extend the size. */ + if (last_seek) + { + struct stat stdout_stat; + if (fstat (STDOUT_FILENO, &stdout_stat) != 0) + { + error (0, errno, _("cannot fstat %s"), quote (output_file)); + return EXIT_FAILURE; + } + if (S_ISREG (stdout_stat.st_mode)) + { + off_t output_offset = lseek (STDOUT_FILENO, 0, SEEK_CUR); + if (output_offset > stdout_stat.st_size) + { + if (ftruncate (STDOUT_FILENO, output_offset) != 0) + { + error (0, errno, + _("failed to truncate to %"PRIuMAX" bytes" + " in output file %s"), + output_offset, quote (output_file)); + return EXIT_FAILURE; + } + } + } + } + if ((conversions_mask & C_FDATASYNC) && fdatasync (STDOUT_FILENO) != 0) { if (errno != ENOSYS && errno != EINVAL) diff --git a/tests/Makefile.am b/tests/Makefile.am index 74ff470..5e184ac 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -373,6 +373,7 @@ TESTS = \ dd/skip-seek2 \ dd/bytes \ dd/skip-seek-past-file \ + dd/sparse \ dd/stderr \ dd/unblock \ dd/unblock-sync \ diff --git a/tests/dd/sparse b/tests/dd/sparse new file mode 100755 index 0000000..254cf08 --- /dev/null +++ b/tests/dd/sparse @@ -0,0 +1,57 @@ +#!/bin/sh + +# Copyright (C) 2012 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/init.sh"; path_prepend_ ../src +print_ver_ dd + +# Ensure basic sparse generation works +truncate -s1M sparse +dd bs=32K if=sparse of=sparse.dd conv=sparse +test $(stat -c %s sparse) = $(stat -c %s sparse.dd) || fail=1 + +# Ensure conv=sparse ignored with oflag=append, +# or otherwise seeks in output are not done +printf 'a\000\000b' > file.in +dd if=file.in bs=1 conv=sparse oflag=append > file.out +cmp file.in file.out || fail=1 + +# Ensure conv=sparse ignored with conv=notrunc, +# or otherwise non NUL data in file.out is not overwritten with NULs +printf 'a\001\001b' > file.out +dd if=file.in bs=1 conv=sparse,notrunc > file.out +cmp file.in file.out || fail=1 + +# Ensure we fall back to write if seek fails +dd if=file.in bs=1 conv=sparse | cat > file.out +cmp file.in file.out || fail=1 + +# Setup for block size tests +dd if=/dev/urandom of=file.in bs=1M count=1 +truncate -s+1M file.in +dd if=/dev/urandom of=file.in bs=1M count=1 conv=notrunc oflag=append + +# Ensure NUL blocks smaller than the block size are not made sparse +dd if=file.in of=file.out bs=2M conv=sparse +test $(stat -c %s file.in) = $(stat -c %s file.out) || fail=1 +test $(stat -c %b file.in) = $(stat -c %b file.out) && fail=1 + +# Ensure NUL blocks >= block size are made sparse +dd if=file.in of=file.out bs=1M conv=sparse +test $(stat -c %s file.in) = $(stat -c %s file.out) || fail=1 +test $(stat -c %b file.in) = $(stat -c %b file.out) || fail=1 + +Exit $fail -- 1.7.6.4 --------------050605020305080706050805-- From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Paul Eggert Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 22:40:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: =?UTF-8?Q?P=C3=A1draig?= Brady Cc: 9157@debbugs.gnu.org, roman.rybalko@romanr.info Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.13304687895690 (code B ref 9157); Tue, 28 Feb 2012 22:40:02 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 22:39:49 +0000 Received: from localhost ([127.0.0.1]:54770 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2Vi1-0001TZ-72 for submit@debbugs.gnu.org; Tue, 28 Feb 2012 17:39:49 -0500 Received: from smtp.cs.ucla.edu ([131.179.128.62]:51929) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2Vhf-0001Sw-Um for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 17:39:38 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id B81F1A6000B; Tue, 28 Feb 2012 10:24:57 -0800 (PST) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XanoOTjjekan; Tue, 28 Feb 2012 10:24:57 -0800 (PST) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 218DAA60008; Tue, 28 Feb 2012 10:24:57 -0800 (PST) Message-ID: <4F4D1BF8.8020902@cs.ucla.edu> Date: Tue, 28 Feb 2012 10:24:56 -0800 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.1) Gecko/20120209 Thunderbird/10.0.1 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> <4F4CB447.4050300@draigBrady.com> <4F4D084E.6090907@draigBrady.com> In-Reply-To: <4F4D084E.6090907@draigBrady.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 02/28/2012 09:01 AM, P=C3=A1draig Brady wrote: > Doing that would be "dangerous" as I said above, > or in other words, surprising to users to not update > possibly non NUL data in the output file. It's not surprising at all. It's what I want and expect. For example, suppose every nonzero byte of file A has an offset that is in a hole of file B. Then I should be able to overlay A and B into another file C, as follows: dd if=3DA of=3DC conv=3Dsparse dd if=3DB of=3DC conv=3Dsparse,notrunc This is akin to the "or" operation on files, and it'd be a nice operation to have. Even if A's contents don't fit in B's holes, I still might want to do the above, to allow B's non-holes to override A's contents. Why disable this useful functionality? Any surprise issues can be dealt with by documenting dd's behavior appropriately. > +Try to seek rather than write @sc{nul} output blocks. > +This will create sparse output when extending. > +This option is ignored in conjunction with > +@samp{conv=3Dnotrunc} or @samp{oflag=3Dappend}. I still dubious about this level of handholding. dd is meant for low-level use, and as far as possible options should be orthogonal. For example, with dd, oflag=3Dappend does not disable seek=3DN -- both flags operate, which means that the seek is ineffective unless it is past the end of file. conv=3Dsparse oflag=3Dappend should be similar: all it should mean is that the writes are sparse when they're past the end of the file (this latter functionality should work, but doesn't work with the proposed patch). From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Roman Rybalko Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 23:00:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: =?UTF-8?Q?P=C3=A1draig?= Brady Cc: 9157@debbugs.gnu.org, Paul Eggert Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.13304699827457 (code B ref 9157); Tue, 28 Feb 2012 23:00:01 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 22:59:42 +0000 Received: from localhost ([127.0.0.1]:54794 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2W1F-0001w3-Hp for submit@debbugs.gnu.org; Tue, 28 Feb 2012 17:59:41 -0500 Received: from host.romanr.info ([82.146.61.98]:59765) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2VqR-0001fv-A0 for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 17:48:32 -0500 Received: from (withheld) by host.romanr.info (Postfix) with ESMTPSA id A9693372BB9D; Tue, 28 Feb 2012 22:00:24 +0400 (MSK) Message-ID: <4F4D1637.2070508@romanr.name> Date: Tue, 28 Feb 2012 22:00:23 +0400 From: Roman Rybalko User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24) Gecko/20111108 Thunderbird/3.1.16 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> <4F4CB447.4050300@draigBrady.com> <4F4D084E.6090907@draigBrady.com> In-Reply-To: <4F4D084E.6090907@draigBrady.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.9 (-) X-Mailman-Approved-At: Tue, 28 Feb 2012 17:59:29 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 28.02.2012 21:01, P=C3=A1draig Brady wrote: > Ok the attached should be pretty much complete. > It disables conv=3Dsparse when used with > conv=3Dnotrunc or oflag=3Dappend. > Comments appreciated. About "sparse" stuff everything's ok, imo. Actually I can live with truncated files due to absence of last write, but for user it's really convenient to have output file with matched size :) so it worths having additional static var with some logic around. I have another idea about INPUT_BLOCK_SLOP/OUTPUT_BLOCK_SLOP: what about not hardcoding them but getting them out of the filesystem, or out of ibs/obs specified by user - that last would be the best case. But that's probably not for this feature request. --=20 WBR, Roman Rybalko From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Jim Meyering Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 23:14:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: =?UTF-8?Q?P=C3=A1draig?= Brady Cc: 9157@debbugs.gnu.org, roman.rybalko@romanr.info, Paul Eggert Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.13304708368693 (code B ref 9157); Tue, 28 Feb 2012 23:14:01 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 23:13:56 +0000 Received: from localhost ([127.0.0.1]:54803 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2WF2-0002Fx-CK for submit@debbugs.gnu.org; Tue, 28 Feb 2012 18:13:56 -0500 Received: from mx.meyering.net ([88.168.87.75]:42104) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2WEp-0002FX-67 for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 18:13:45 -0500 Received: from rho.meyering.net (localhost.localdomain [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id 65FEE60094; Wed, 29 Feb 2012 00:13:26 +0100 (CET) From: Jim Meyering In-Reply-To: <4F4D084E.6090907@draigBrady.com> ("=?UTF-8?Q?P=C3=A1draig?= Brady"'s message of "Tue, 28 Feb 2012 17:01:02 +0000") References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> <4F4CB447.4050300@draigBrady.com> <4F4D084E.6090907@draigBrady.com> Date: Wed, 29 Feb 2012 00:13:26 +0100 Message-ID: <87399ulfft.fsf@rho.meyering.net> Lines: 117 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) P=E1draig Brady wrote: ... Thanks for working on that. All I would adjust are a few nits and add require_sparse_support_ in the test script: > Subject: [PATCH] dd: add support for the conv=3Dsparse option > > Small seeks are not coalesced to larger ones (like is > done in cache_round() for example, for the moment at least. > > conv=3D is used rather then oflag=3D for FreeBSD compatibility. > > * src/dd.c (last_seek): A new global boolean to flag "last" is ambiguous. Does it mean "final" or "preceding"? >From the context below, (not the comment, since it too uses "last") I see it means "final". "final_op_was_seek" is longer but ultra clear. Maybe worth the length. > whether the last "write" was converted to a seek. > (usage): Describe the new conf=3Dsparse option. > (scanargs): Ignore conv=3Dsparse in some combinations. > (iwrite): Convert a write of a NUL block to a seek if requested. > (do_copy): Initialize the output buffer to have a sentinel, > to allow for efficient testing for NUL output blocks. > If the last block in the file was converted to a seek, > then convert back to a write so the size ip updated. s/ip/is/ > * NEWS: Mention the new feature. > * tests/dd/sparse: A new test for the feature. > * tests/Makefile.am: Reference the new test. > --- > NEWS | 3 + > doc/coreutils.texi | 7 +++ > src/dd.c | 104 ++++++++++++++++++++++++++++++++++++++++++++++= ++++-- > tests/Makefile.am | 1 + > tests/dd/sparse | 57 ++++++++++++++++++++++++++++ > 5 files changed, 168 insertions(+), 4 deletions(-) > create mode 100755 tests/dd/sparse > > diff --git a/NEWS b/NEWS > index e2e8fc5..8006669 100644 > --- a/NEWS > +++ b/NEWS > @@ -7,6 +7,9 @@ GNU coreutils NEWS -*-= outline -*- > dd now accepts the count_bytes, skip_bytes iflags and the seek_bytes > oflag, to more easily allow processing portions of a file. > > + dd now accepts the conv=3Dsparse flag to attempt to create sparse > + output, by seeking rather than writing to the output file. > + > split now accepts an optional "from" argument to --numeric-suffixes, > which changes the start number from the default of 0. > > diff --git a/doc/coreutils.texi b/doc/coreutils.texi > index 414626d..f22e7d2 100644 > --- a/doc/coreutils.texi > +++ b/doc/coreutils.texi > @@ -8140,6 +8140,13 @@ Change lowercase letters to uppercase. > > The @samp{lcase} and @samp{ucase} conversions are mutually exclusive. > > +@item sparse > +@opindex sparse > +Try to seek rather than write @sc{nul} output blocks. > +This will create sparse output when extending. Maybe add this: s/\./on a file system that support sparse files./ ? > +This option is ignored in conjunction with > +@samp{conv=3Dnotrunc} or @samp{oflag=3Dappend}. > + > @item swab > @opindex swab @r{(byte-swapping)} > @cindex byte-swapping > diff --git a/src/dd.c b/src/dd.c > index fe44a30..903346f 100644 > --- a/src/dd.c > +++ b/src/dd.c > @@ -94,7 +94,7 @@ > malloc. See dd_copy for details. INPUT_BLOCK_SLOP must be no less t= han > OUTPUT_BLOCK_SLOP. */ > #define INPUT_BLOCK_SLOP (2 * SWAB_ALIGN_OFFSET + 2 * page_size - 1) > -#define OUTPUT_BLOCK_SLOP (page_size - 1) > +#define OUTPUT_BLOCK_SLOP MAX (sizeof (uintptr_t), page_size - 1) I haven't seen justification for why you're making the above change. Can sizeof uintptr_t really be larger than page_size-1 (getpagesize()-1)? This seems so unlikely that it'd deserve an assertion in main where page_size is set, even though there are only two uses of OUTPUT_BLOCK_SLOP. If you leave the sizeof, please omit the parentheses. They are unnecessary here. Same below. ... > + /* Write sentinel to slop after the buffer, > + to allow efficient checking for NUL blocks. */ > + memset (obuf + output_blocksize, 1, sizeof (uintptr_t)); ... > diff --git a/tests/dd/sparse b/tests/dd/sparse ... > + > +. "${srcdir=3D.}/init.sh"; path_prepend_ ../src > +print_ver_ dd require_sparse_support_ > +# Ensure basic sparse generation works > +truncate -s1M sparse > +dd bs=3D32K if=3Dsparse of=3Dsparse.dd conv=3Dsparse > +test $(stat -c %s sparse) =3D $(stat -c %s sparse.dd) || fail=3D1 From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 23:35:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Jim Meyering Cc: 9157@debbugs.gnu.org, roman.rybalko@romanr.info, Paul Eggert Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.133047206410595 (code B ref 9157); Tue, 28 Feb 2012 23:35:02 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 23:34:24 +0000 Received: from localhost ([127.0.0.1]:54851 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2WYp-0002kX-7Z for submit@debbugs.gnu.org; Tue, 28 Feb 2012 18:34:24 -0500 Received: from mx1.redhat.com ([209.132.183.28]:54945) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2WYT-0002jg-76 for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 18:34:11 -0500 Received: from int-mx02.intmail.prod.int.phx2.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q1SNXhIl010804 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 28 Feb 2012 18:33:43 -0500 Received: from [10.36.116.40] (ovpn-116-40.ams2.redhat.com [10.36.116.40]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q1SNXehe021350 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 28 Feb 2012 18:33:42 -0500 Message-ID: <4F4D6454.7040508@draigBrady.com> Date: Tue, 28 Feb 2012 23:33:40 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> <4F4CB447.4050300@draigBrady.com> <4F4D084E.6090907@draigBrady.com> <87399ulfft.fsf@rho.meyering.net> In-Reply-To: <87399ulfft.fsf@rho.meyering.net> X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=ISO-8859-1 X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id q1SNXhIl010804 X-Spam-Score: -6.9 (------) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.9 (------) On 02/28/2012 11:13 PM, Jim Meyering wrote: > P=E1draig Brady wrote: > ... >=20 > Thanks for working on that. > All I would adjust are a few nits and > add require_sparse_support_ in the test script: >=20 >> Subject: [PATCH] dd: add support for the conv=3Dsparse option >> >> Small seeks are not coalesced to larger ones (like is >> done in cache_round() for example, for the moment at least. >> >> conv=3D is used rather then oflag=3D for FreeBSD compatibility. >> >> * src/dd.c (last_seek): A new global boolean to flag >=20 > "last" is ambiguous. Does it mean "final" or "preceding"? >>>From the context below, (not the comment, since it too uses "last") > I see it means "final". "final_op_was_seek" is longer but ultra clear. > Maybe worth the length. yes I was too terse >> -#define OUTPUT_BLOCK_SLOP (page_size - 1) >> +#define OUTPUT_BLOCK_SLOP MAX (sizeof (uintptr_t), page_size - 1) >=20 > I haven't seen justification for why you're making the above change. > Can sizeof uintptr_t really be larger than page_size-1 (getpagesize()-1= )? > This seems so unlikely that it'd deserve an assertion in main where > page_size is set, even though there are only two uses of OUTPUT_BLOCK_S= LOP. Yes it's never going to trigger. Paul suggested MAX(...) so as to doc/enforce the new constraint. assert() is not used in dd.c yet I'll leave as is unless Paul comments otherwise (modulo the extraneous () of course). cheers, P=E1draig From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: Paul Eggert Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 28 Feb 2012 23:41:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Jim Meyering Cc: 9157@debbugs.gnu.org, roman.rybalko@romanr.info, =?UTF-8?Q?P=C3=A1draig?= Brady Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.133047240811224 (code B ref 9157); Tue, 28 Feb 2012 23:41:01 +0000 Received: (at 9157) by debbugs.gnu.org; 28 Feb 2012 23:40:08 +0000 Received: from localhost ([127.0.0.1]:54868 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2WeN-0002uO-QG for submit@debbugs.gnu.org; Tue, 28 Feb 2012 18:40:08 -0500 Received: from smtp.cs.ucla.edu ([131.179.128.62]:58695) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2WeB-0002ts-O8 for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 18:39:56 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id BE2C239E8010; Tue, 28 Feb 2012 15:39:39 -0800 (PST) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rvzc3Xr42hgC; Tue, 28 Feb 2012 15:39:39 -0800 (PST) Received: from penguin.cs.ucla.edu (Penguin.CS.UCLA.EDU [131.179.64.200]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 4162839E800C; Tue, 28 Feb 2012 15:39:39 -0800 (PST) Message-ID: <4F4D65BA.5020109@cs.ucla.edu> Date: Tue, 28 Feb 2012 15:39:38 -0800 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.1) Gecko/20120209 Thunderbird/10.0.1 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> <4F4CB447.4050300@draigBrady.com> <4F4D084E.6090907@draigBrady.com> <87399ulfft.fsf@rho.meyering.net> In-Reply-To: <87399ulfft.fsf@rho.meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 02/28/2012 03:13 PM, Jim Meyering wrote: >> -#define OUTPUT_BLOCK_SLOP (page_size - 1) >> > +#define OUTPUT_BLOCK_SLOP MAX (sizeof (uintptr_t), page_size - 1) > I haven't seen justification for why you're making the above change. > Can sizeof uintptr_t really be larger than page_size-1 (getpagesize()-1)? > This seems so unlikely that it'd deserve an assertion in main where > page_size is set, even though there are only two uses of OUTPUT_BLOCK_SLOP. Theoretically getpagesize could return 1, no? Admittedly it's unlikely. I guess it'd be OK to omit this change, and to just put in a comment saying something like "We use the output slop to store a uintptr_t value, so we assume that getpagesize returns a value greater than sizeof (uintptr_t)." Perhaps the comment should go right next to the place where the uintptr_t sentinel is inserted. > If you leave the sizeof, please omit the parentheses. But "sizeof uintptr_t" wouldn't work, as uintptr_t is a type. From unknown Tue Sep 23 17:51:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#9157: [PATCH] dd: sparse conv flag Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Wed, 29 Feb 2012 00:37:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 9157 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Paul Eggert Cc: 9157@debbugs.gnu.org, roman.rybalko@romanr.info, Jim Meyering Received: via spool by 9157-submit@debbugs.gnu.org id=B9157.133047581216209 (code B ref 9157); Wed, 29 Feb 2012 00:37:02 +0000 Received: (at 9157) by debbugs.gnu.org; 29 Feb 2012 00:36:52 +0000 Received: from localhost ([127.0.0.1]:54904 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2XXG-0004DB-S7 for submit@debbugs.gnu.org; Tue, 28 Feb 2012 19:36:51 -0500 Received: from mx1.redhat.com ([209.132.183.28]:12076) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1S2XX2-0004Ck-VW for 9157@debbugs.gnu.org; Tue, 28 Feb 2012 19:36:38 -0500 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q1T0aInk018305 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 28 Feb 2012 19:36:18 -0500 Received: from [10.36.116.40] (ovpn-116-40.ams2.redhat.com [10.36.116.40]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q1T0aF3T019029 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 28 Feb 2012 19:36:17 -0500 Message-ID: <4F4D72FF.1030806@draigBrady.com> Date: Wed, 29 Feb 2012 00:36:15 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: <20110723163227.3C04B69E5E@roma.localdomain> <20110724164218.E96DE7CA9@roma.localdomain> <87iplohh69.fsf@rho.meyering.net> <4EE50C6B.9000403@romanr.info> <4F4B9F20.2070001@draigBrady.com> <4F4BA638.8080106@cs.ucla.edu> <4F4BAAEF.5060606@draigBrady.com> <4F4BB9E6.3020806@cs.ucla.edu> <4F4BC614.8080506@draigBrady.com> <4F4C295F.7040400@draigBrady.com> <4F4C955C.4040500@cs.ucla.edu> <4F4CB447.4050300@draigBrady.com> <4F4D084E.6090907@draigBrady.com> <87399ulfft.fsf@rho.meyering.net> <4F4D65BA.5020109@cs.ucla.edu> In-Reply-To: <4F4D65BA.5020109@cs.ucla.edu> X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=ISO-8859-1 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id q1T0aInk018305 X-Spam-Score: -6.9 (------) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.9 (------) On 02/28/2012 11:39 PM, Paul Eggert wrote: > On 02/28/2012 03:13 PM, Jim Meyering wrote: >>> -#define OUTPUT_BLOCK_SLOP (page_size - 1) >>>> +#define OUTPUT_BLOCK_SLOP MAX (sizeof (uintptr_t), page_size - 1) >> I haven't seen justification for why you're making the above change. >> Can sizeof uintptr_t really be larger than page_size-1 (getpagesize()-= 1)? >> This seems so unlikely that it'd deserve an assertion in main where >> page_size is set, even though there are only two uses of OUTPUT_BLOCK_= SLOP. >=20 > Theoretically getpagesize could return 1, no? > Admittedly it's unlikely. I guess it'd be OK > to omit this change, and to just put in a comment > saying something like "We use the output slop > to store a uintptr_t value, so we assume that > getpagesize returns a value greater than sizeof (uintptr_t)." > Perhaps the comment should go right next to the place where > the uintptr_t sentinel is inserted. >=20 >> If you leave the sizeof, please omit the parentheses. >=20 > But "sizeof uintptr_t" wouldn't work, as uintptr_t is a type. >=20 Ok I'm going with the version with the assert, since that's better than a comment. thanks for all the comments, P=E1draig. p.s. I can't commit at the moment due to authentication error. I'll look into it in the morning.