Package: coreutils;
Reported by: Roman Rybalko <devel <at> romanr.info>
Date: Sat, 23 Jul 2011 22:42:04 UTC
Severity: normal
Tags: patch
Done: Pádraig Brady <P <at> draigBrady.com>
Bug is archived. No further changes may be made.
View this message in rfc822 format
From: help-debbugs <at> gnu.org (GNU bug Tracking System) To: Pádraig Brady <P <at> draigBrady.com> Cc: tracker <at> debbugs.gnu.org Subject: bug#9157: closed ([PATCH] dd: sparse conv flag) Date: Tue, 28 Feb 2012 21:51:02 +0000
[Message part 1 (text/plain, inline)]
Your message dated Tue, 28 Feb 2012 21:49:43 +0000 with message-id <4F4D4BF7.6020404 <at> draigBrady.com> and subject line Re: bug#9157: [PATCH] dd: sparse conv flag has caused the debbugs.gnu.org bug report #9157, regarding [PATCH] dd: sparse conv flag to be marked as done. (If you believe you have received this mail in error, please contact help-debbugs <at> gnu.org.) -- 9157: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9157 GNU Bug Tracking System Contact help-debbugs <at> gnu.org with problems
[Message part 2 (message/rfc822, inline)]
From: Roman Rybalko <devel <at> romanr.info> To: undisclosed-recipients:; Subject: [PATCH] dd: sparse conv flag Date: Sat, 23 Jul 2011 20:08:01 +0400--- doc/coreutils.texi | 4 +++ src/dd.c | 22 +++++++++++++++- tests/Makefile.am | 1 + tests/dd/sparse | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 96 insertions(+), 1 deletions(-) create mode 100755 tests/dd/sparse diff --git a/doc/coreutils.texi b/doc/coreutils.texi index 424446c..761c698 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. When used with @samp{block} or @samp{unblock}, pad with spaces instead of zero bytes. +@item sparse +@opindex sparse +Make sparse output file. + @end table The following ``conversions'' are really file flags diff --git a/src/dd.c b/src/dd.c index 0824f6c..49847f5 100644 --- a/src/dd.c +++ b/src/dd.c @@ -126,7 +126,8 @@ enum C_NOCREAT = 010000, C_EXCL = 020000, C_FDATASYNC = 040000, - C_FSYNC = 0100000 + C_FSYNC = 0100000, + C_SPARSE = 0200000 }; /* Status bit masks. */ @@ -268,6 +269,7 @@ static struct symbol_value const conversions[] = {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ {"fdatasync", C_FDATASYNC}, /* Synchronize output data before finishing. */ {"fsync", C_FSYNC}, /* Also synchronize output metadata. */ + {"sparse", C_SPARSE}, /* Make sparse output file. */ {"", 0} }; @@ -533,6 +535,9 @@ Each CONV symbol may be:\n\ fsync likewise, but also write metadata\n\ "), stdout); fputs (_("\ + sparse make sparse output file\n\ +"), stdout); + fputs (_("\ \n\ Each FLAG symbol may be:\n\ \n\ @@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size) { ssize_t nwritten; process_signals (); + if (conversions_mask & C_SPARSE) + { + off_t seek_size = 0; + while (buf[total_written + seek_size] == 0) + ++seek_size; + if (seek_size) + { + off_t cur_off = 0; + cur_off = lseek(fd, seek_size, SEEK_CUR); + if (cur_off < 0) + break; + total_written += seek_size; + continue; + } + } nwritten = write (fd, buf + total_written, size - total_written); if (nwritten < 0) { diff --git a/tests/Makefile.am b/tests/Makefile.am index ebd1b11..0f1376a 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -364,6 +364,7 @@ TESTS = \ dd/skip-seek \ dd/skip-seek2 \ dd/skip-seek-past-file \ + dd/sparse \ dd/stderr \ dd/unblock \ dd/unblock-sync \ diff --git a/tests/dd/sparse b/tests/dd/sparse new file mode 100755 index 0000000..f0e0806 --- /dev/null +++ b/tests/dd/sparse @@ -0,0 +1,70 @@ +#!/bin/sh +# Ensure that dd conv=sparse works. + +# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +. "${srcdir=.}/init.sh"; path_prepend_ ../src +print_ver_ dd + +# sometimes we may read less than 1M +dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample0`" = "1048576" ] || fail=1 +dd if=/dev/urandom of=sample1 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample1`" = "1048576" ] || fail=1 + +# test 1 +dd if=sample1 of=test1 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test1 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test1 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 1-1 +dd if=test1 of=out1-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-1`" = "`stat -c '%s %b %B' out1-1-check`" ] || fail=1 + +# test 1-2 +dd if=test1 of=out1-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=3 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-2`" = "`stat -c '%s %b %B' out1-2-check`" ] || fail=1 + +# test 1-3 +dd if=test1 of=out1-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-3-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-3`" = "`stat -c '%s %b %B' out1-3-check`" ] || fail=1 + +# test 2 +dd if=sample0 of=test2 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test2 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test2 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 2-1 +dd if=test2 of=out2-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-1-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-1`" = "`stat -c '%s %b %B' out2-1-check`" ] || fail=1 + +# test 2-2 +dd if=test2 of=out2-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-2-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-2`" = "`stat -c '%s %b %B' out2-2-check`" ] || fail=1 + +# test 2-3 +dd if=test2 of=out2-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-3-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-3`" = "`stat -c '%s %b %B' out2-3-check`" ] || fail=1 + +Exit $fail -- 1.7.0.4
[Message part 3 (message/rfc822, inline)]
From: Pádraig Brady <P <at> draigBrady.com> To: Paul Eggert <eggert <at> cs.ucla.edu> Cc: 9157-done <at> debbugs.gnu.org, roman.rybalko <at> romanr.info Subject: Re: bug#9157: [PATCH] dd: sparse conv flag Date: Tue, 28 Feb 2012 21:49:43 +0000On 02/28/2012 06:24 PM, Paul Eggert wrote: > On 02/28/2012 09:01 AM, Pádraig Brady wrote: > >> Doing that would be "dangerous" as I said above, >> or in other words, surprising to users to not update >> possibly non NUL data in the output file. > > It's not surprising at all. It's what I want and > expect. For example, suppose every nonzero byte of file A > has an offset that is in a hole of file B. Then I should > be able to overlay A and B into another file C, as follows: > > dd if=A of=C conv=sparse > dd if=B of=C conv=sparse,notrunc > > This is akin to the "or" operation on files, and it'd be a nice > operation to have. Even if A's contents don't fit in B's holes, > I still might want to do the above, to allow B's non-holes to > override A's contents. Why disable this useful functionality? > > Any surprise issues can be dealt with by documenting dd's > behavior appropriately. Fair enough. So this is the "write mask" functionality I referred to, and which is now supported consistently since the write(fd,"\0",1) to ftruncate() change. >> +Try to seek rather than write @sc{nul} output blocks. >> +This will create sparse output when extending. >> +This option is ignored in conjunction with >> +@samp{conv=notrunc} or @samp{oflag=append}. > > I still dubious about this level of handholding. > dd is meant for low-level use, and as far as possible > options should be orthogonal. For example, with dd, > oflag=append does not disable seek=N -- both flags operate, > which means that the seek is ineffective unless it > is past the end of file. conv=sparse oflag=append > should be similar: all it should mean is that the > writes are sparse when they're past the end of > the file (this latter functionality should work, but > doesn't work with the proposed patch). OK I'll remove the 2 restrictions, and essentially move the comments for them from code to texinfo. cheers, Pádraig.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.