Subject: bug#9157: closed (Re: bug#9157: [PATCH] dd: sparse conv flag)

From: [email-masked] (GNU bug Tracking System)
Date: Tue, 28 Feb 2012 21:51:03 +0000

Your bug report

#9157: [PATCH] dd: sparse conv flag

which was filed against the coreutils package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
9157: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9157
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#9157: [PATCH] dd: sparse conv flag
From: Pádraig Brady <[email-masked]>
To: Paul Eggert <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Tue, 28 Feb 2012 21:49:43 +0000
List-Id: <debbugs-submit.debbugs.gnu.org>

On 02/28/2012 06:24 PM, Paul Eggert wrote:
> On 02/28/2012 09:01 AM, Pádraig Brady wrote:
> 
>> Doing that would be "dangerous" as I said above,
>> or in other words, surprising to users to not update
>> possibly non NUL data in the output file.
> 
> It's not surprising at all.  It's what I want and
> expect.  For example, suppose every nonzero byte of file A
> has an offset that is in a hole of file B.  Then I should
> be able to overlay A and B into another file C, as follows:
> 
> dd if=A of=C conv=sparse
> dd if=B of=C conv=sparse,notrunc
> 
> This is akin to the "or" operation on files, and it'd be a nice
> operation to have.  Even if A's contents don't fit in B's holes,
> I still might want to do the above, to allow B's non-holes to
> override A's contents.  Why disable this useful functionality?
> 
> Any surprise issues can be dealt with by documenting dd's
> behavior appropriately.

Fair enough. So this is the "write mask" functionality
I referred to, and which is now supported consistently
since the write(fd,"\0",1) to ftruncate() change.

>> +Try to seek rather than write @sc{nul} output blocks.
>> +This will create sparse output when extending.
>> +This option is ignored in conjunction with
>> +@samp{conv=notrunc} or @samp{oflag=append}.
> 
> I still dubious about this level of handholding.
> dd is meant for low-level use, and as far as possible
> options should be orthogonal.  For example, with dd,
> oflag=append does not disable seek=N -- both flags operate,
> which means that the seek is ineffective unless it
> is past the end of file.  conv=sparse oflag=append
> should be similar: all it should mean is that the
> writes are sparse when they're past the end of
> the file (this latter functionality should work, but
> doesn't work with the proposed patch).

OK I'll remove the 2 restrictions, and essentially
move the comments for them from code to texinfo.

cheers,
Pádraig.

Subject: [PATCH] dd: sparse conv flag
From: Roman Rybalko <[email-masked]>
To: undisclosed-recipients:;
Date: Sat, 23 Jul 2011 20:08:01 +0400
List-Id: <debbugs-submit.debbugs.gnu.org>

---
 doc/coreutils.texi |    4 +++
 src/dd.c           |   22 +++++++++++++++-
 tests/Makefile.am  |    1 +
 tests/dd/sparse    |   70 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 96 insertions(+), 1 deletions(-)
 create mode 100755 tests/dd/sparse

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 424446c..761c698 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes.
 When used with @samp{block} or @samp{unblock}, pad with spaces instead of
 zero bytes.
 
+@item sparse
+@opindex sparse
+Make sparse output file.
+
 @end table
 
 The following ``conversions'' are really file flags
diff --git a/src/dd.c b/src/dd.c
index 0824f6c..49847f5 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -126,7 +126,8 @@ enum
     C_NOCREAT = 010000,
     C_EXCL = 020000,
     C_FDATASYNC = 040000,
-    C_FSYNC = 0100000
+    C_FSYNC = 0100000,
+    C_SPARSE = 0200000
   };
 
 /* Status bit masks.  */
@@ -268,6 +269,7 @@ static struct symbol_value const conversions[] =
   {"sync", C_SYNC},		/* Pad input records to ibs with NULs. */
   {"fdatasync", C_FDATASYNC},	/* Synchronize output data before finishing.  */
   {"fsync", C_FSYNC},		/* Also synchronize output metadata.  */
+  {"sparse", C_SPARSE},		/* Make sparse output file. */
   {"", 0}
 };
 
@@ -533,6 +535,9 @@ Each CONV symbol may be:\n\
   fsync     likewise, but also write metadata\n\
 "), stdout);
       fputs (_("\
+  sparse    make sparse output file\n\
+"), stdout);
+      fputs (_("\
 \n\
 Each FLAG symbol may be:\n\
 \n\
@@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size)
     {
       ssize_t nwritten;
       process_signals ();
+      if (conversions_mask & C_SPARSE)
+        {
+          off_t seek_size = 0;
+          while (buf[total_written + seek_size] == 0)
+            ++seek_size;
+          if (seek_size)
+            {
+              off_t cur_off = 0;
+              cur_off = lseek(fd, seek_size, SEEK_CUR);
+              if (cur_off < 0)
+                break;
+              total_written += seek_size;
+              continue;
+            }
+        }
       nwritten = write (fd, buf + total_written, size - total_written);
       if (nwritten < 0)
         {
diff --git a/tests/Makefile.am b/tests/Makefile.am
index ebd1b11..0f1376a 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -364,6 +364,7 @@ TESTS =						\
   dd/skip-seek					\
   dd/skip-seek2					\
   dd/skip-seek-past-file			\
+  dd/sparse                                     \
   dd/stderr					\
   dd/unblock					\
   dd/unblock-sync				\
diff --git a/tests/dd/sparse b/tests/dd/sparse
new file mode 100755
index 0000000..f0e0806
--- /dev/null
+++ b/tests/dd/sparse
@@ -0,0 +1,70 @@
+#!/bin/sh
+# Ensure that dd conv=sparse works.
+
+# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. "${srcdir=.}/init.sh"; path_prepend_ ../src
+print_ver_ dd
+
+# sometimes we may read less than 1M
+dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1
+[ "`stat -c %s sample0`" = "1048576" ] || fail=1
+dd if=/dev/urandom of=sample1 count=1 bs=1M 2> /dev/null || fail=1
+[ "`stat -c %s sample1`" = "1048576" ] || fail=1
+
+# test 1
+dd if=sample1 of=test1 seek=0 bs=1M 2> /dev/null || fail=1
+dd if=sample0 of=test1 seek=1 bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=test1 seek=2 bs=1M 2> /dev/null || fail=1
+
+# test 1-1
+dd if=test1 of=out1-1 conv=sparse bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=out1-1-check bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=out1-1-check seek=2 bs=1M 2> /dev/null || fail=1
+[ "`stat -c '%s %b %B' out1-1`" = "`stat -c '%s %b %B' out1-1-check`" ] || fail=1
+
+# test 1-2
+dd if=test1 of=out1-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=out1-2-check seek=1 bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=out1-2-check seek=3 bs=1M 2> /dev/null || fail=1
+[ "`stat -c '%s %b %B' out1-2`" = "`stat -c '%s %b %B' out1-2-check`" ] || fail=1
+
+# test 1-3
+dd if=test1 of=out1-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=out1-3-check seek=2 bs=1M 2> /dev/null || fail=1
+[ "`stat -c '%s %b %B' out1-3`" = "`stat -c '%s %b %B' out1-3-check`" ] || fail=1
+
+# test 2
+dd if=sample0 of=test2 seek=0 bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=test2 seek=1 bs=1M 2> /dev/null || fail=1
+dd if=sample0 of=test2 seek=2 bs=1M 2> /dev/null || fail=1
+
+# test 2-1
+dd if=test2 of=out2-1 conv=sparse bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=out2-1-check seek=1 bs=1M 2> /dev/null || fail=1
+[ "`stat -c '%s %b %B' out2-1`" = "`stat -c '%s %b %B' out2-1-check`" ] || fail=1
+
+# test 2-2
+dd if=test2 of=out2-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=out2-2-check seek=2 bs=1M 2> /dev/null || fail=1
+[ "`stat -c '%s %b %B' out2-2`" = "`stat -c '%s %b %B' out2-2-check`" ] || fail=1
+
+# test 2-3
+dd if=test2 of=out2-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1
+dd if=sample1 of=out2-3-check seek=1 bs=1M 2> /dev/null || fail=1
+[ "`stat -c '%s %b %B' out2-3`" = "`stat -c '%s %b %B' out2-3-check`" ] || fail=1
+
+Exit $fail
-- 
1.7.0.4