Package: coreutils;
Reported by: Roman Rybalko <devel <at> romanr.info>
Date: Sat, 23 Jul 2011 22:42:04 UTC
Severity: normal
Tags: patch
Done: Pádraig Brady <P <at> draigBrady.com>
Bug is archived. No further changes may be made.
View this message in rfc822 format
From: help-debbugs <at> gnu.org (GNU bug Tracking System) To: Roman Rybalko <devel <at> romanr.info> Subject: bug#9157: closed (Re: bug#9157: [PATCH] dd: sparse conv flag) Date: Tue, 28 Feb 2012 21:51:03 +0000
[Message part 1 (text/plain, inline)]
Your bug report #9157: [PATCH] dd: sparse conv flag which was filed against the coreutils package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 9157 <at> debbugs.gnu.org. -- 9157: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9157 GNU Bug Tracking System Contact help-debbugs <at> gnu.org with problems
[Message part 2 (message/rfc822, inline)]
From: Pádraig Brady <P <at> draigBrady.com> To: Paul Eggert <eggert <at> cs.ucla.edu> Cc: 9157-done <at> debbugs.gnu.org, roman.rybalko <at> romanr.info Subject: Re: bug#9157: [PATCH] dd: sparse conv flag Date: Tue, 28 Feb 2012 21:49:43 +0000On 02/28/2012 06:24 PM, Paul Eggert wrote: > On 02/28/2012 09:01 AM, Pádraig Brady wrote: > >> Doing that would be "dangerous" as I said above, >> or in other words, surprising to users to not update >> possibly non NUL data in the output file. > > It's not surprising at all. It's what I want and > expect. For example, suppose every nonzero byte of file A > has an offset that is in a hole of file B. Then I should > be able to overlay A and B into another file C, as follows: > > dd if=A of=C conv=sparse > dd if=B of=C conv=sparse,notrunc > > This is akin to the "or" operation on files, and it'd be a nice > operation to have. Even if A's contents don't fit in B's holes, > I still might want to do the above, to allow B's non-holes to > override A's contents. Why disable this useful functionality? > > Any surprise issues can be dealt with by documenting dd's > behavior appropriately. Fair enough. So this is the "write mask" functionality I referred to, and which is now supported consistently since the write(fd,"\0",1) to ftruncate() change. >> +Try to seek rather than write @sc{nul} output blocks. >> +This will create sparse output when extending. >> +This option is ignored in conjunction with >> +@samp{conv=notrunc} or @samp{oflag=append}. > > I still dubious about this level of handholding. > dd is meant for low-level use, and as far as possible > options should be orthogonal. For example, with dd, > oflag=append does not disable seek=N -- both flags operate, > which means that the seek is ineffective unless it > is past the end of file. conv=sparse oflag=append > should be similar: all it should mean is that the > writes are sparse when they're past the end of > the file (this latter functionality should work, but > doesn't work with the proposed patch). OK I'll remove the 2 restrictions, and essentially move the comments for them from code to texinfo. cheers, Pádraig.
[Message part 3 (message/rfc822, inline)]
From: Roman Rybalko <devel <at> romanr.info> To: undisclosed-recipients:; Subject: [PATCH] dd: sparse conv flag Date: Sat, 23 Jul 2011 20:08:01 +0400--- doc/coreutils.texi | 4 +++ src/dd.c | 22 +++++++++++++++- tests/Makefile.am | 1 + tests/dd/sparse | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 96 insertions(+), 1 deletions(-) create mode 100755 tests/dd/sparse diff --git a/doc/coreutils.texi b/doc/coreutils.texi index 424446c..761c698 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -8127,6 +8127,10 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. When used with @samp{block} or @samp{unblock}, pad with spaces instead of zero bytes. +@item sparse +@opindex sparse +Make sparse output file. + @end table The following ``conversions'' are really file flags diff --git a/src/dd.c b/src/dd.c index 0824f6c..49847f5 100644 --- a/src/dd.c +++ b/src/dd.c @@ -126,7 +126,8 @@ enum C_NOCREAT = 010000, C_EXCL = 020000, C_FDATASYNC = 040000, - C_FSYNC = 0100000 + C_FSYNC = 0100000, + C_SPARSE = 0200000 }; /* Status bit masks. */ @@ -268,6 +269,7 @@ static struct symbol_value const conversions[] = {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ {"fdatasync", C_FDATASYNC}, /* Synchronize output data before finishing. */ {"fsync", C_FSYNC}, /* Also synchronize output metadata. */ + {"sparse", C_SPARSE}, /* Make sparse output file. */ {"", 0} }; @@ -533,6 +535,9 @@ Each CONV symbol may be:\n\ fsync likewise, but also write metadata\n\ "), stdout); fputs (_("\ + sparse make sparse output file\n\ +"), stdout); + fputs (_("\ \n\ Each FLAG symbol may be:\n\ \n\ @@ -985,6 +990,21 @@ iwrite (int fd, char const *buf, size_t size) { ssize_t nwritten; process_signals (); + if (conversions_mask & C_SPARSE) + { + off_t seek_size = 0; + while (buf[total_written + seek_size] == 0) + ++seek_size; + if (seek_size) + { + off_t cur_off = 0; + cur_off = lseek(fd, seek_size, SEEK_CUR); + if (cur_off < 0) + break; + total_written += seek_size; + continue; + } + } nwritten = write (fd, buf + total_written, size - total_written); if (nwritten < 0) { diff --git a/tests/Makefile.am b/tests/Makefile.am index ebd1b11..0f1376a 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -364,6 +364,7 @@ TESTS = \ dd/skip-seek \ dd/skip-seek2 \ dd/skip-seek-past-file \ + dd/sparse \ dd/stderr \ dd/unblock \ dd/unblock-sync \ diff --git a/tests/dd/sparse b/tests/dd/sparse new file mode 100755 index 0000000..f0e0806 --- /dev/null +++ b/tests/dd/sparse @@ -0,0 +1,70 @@ +#!/bin/sh +# Ensure that dd conv=sparse works. + +# Copyright (C) 2003, 2005-2011 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +. "${srcdir=.}/init.sh"; path_prepend_ ../src +print_ver_ dd + +# sometimes we may read less than 1M +dd if=/dev/zero of=sample0 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample0`" = "1048576" ] || fail=1 +dd if=/dev/urandom of=sample1 count=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c %s sample1`" = "1048576" ] || fail=1 + +# test 1 +dd if=sample1 of=test1 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test1 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test1 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 1-1 +dd if=test1 of=out1-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-1-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-1`" = "`stat -c '%s %b %B' out1-1-check`" ] || fail=1 + +# test 1-2 +dd if=test1 of=out1-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-2-check seek=3 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-2`" = "`stat -c '%s %b %B' out1-2-check`" ] || fail=1 + +# test 1-3 +dd if=test1 of=out1-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out1-3-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out1-3`" = "`stat -c '%s %b %B' out1-3-check`" ] || fail=1 + +# test 2 +dd if=sample0 of=test2 seek=0 bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=test2 seek=1 bs=1M 2> /dev/null || fail=1 +dd if=sample0 of=test2 seek=2 bs=1M 2> /dev/null || fail=1 + +# test 2-1 +dd if=test2 of=out2-1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-1-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-1`" = "`stat -c '%s %b %B' out2-1-check`" ] || fail=1 + +# test 2-2 +dd if=test2 of=out2-2 seek=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-2-check seek=2 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-2`" = "`stat -c '%s %b %B' out2-2-check`" ] || fail=1 + +# test 2-3 +dd if=test2 of=out2-3 seek=1 skip=1 conv=sparse bs=1M 2> /dev/null || fail=1 +dd if=sample1 of=out2-3-check seek=1 bs=1M 2> /dev/null || fail=1 +[ "`stat -c '%s %b %B' out2-3`" = "`stat -c '%s %b %B' out2-3-check`" ] || fail=1 + +Exit $fail -- 1.7.0.4
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.