GNU bug report logs - #8171
[dd] bs=blocksize doesn't work as expected

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 8171 in the body.
You can then email your comments to 8171 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Станислав <ginermail <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: [dd] bs=blocksize doesn't work as expected
Date: Fri, 4 Mar 2011 08:02:56 +0300

[Message part 1 (text/plain, inline)]

[dd] bs=blocksize doesn't work as expected, for example:
tar -c /somedir | dd bs=256K of=/dev/nst0
In this example backup record will be recorded to a tape with block size =
10K instead of 256K because 10K is default for tar.
There is workaround:
tar -c /somedir | dd ibs=256K obs=256K of=/dev/nst0
If we set block size for input and output separatly - it works as expected.

lucid <http://packages.ubuntu.com/lucid/coreutils> (utils): The GNU core
utilities
7.4-2ubuntu2: amd64 i386

[Message part 2 (text/html, inline)]

Message #8 received at 8171 <at> debbugs.gnu.org (full text, mbox):

From: Ondrej Vasik <ovasik <at> redhat.com>
To: Станислав
	<ginermail <at> gmail.com>
Cc: 8171 <at> debbugs.gnu.org
Subject: Re: bug#8171: [dd] bs=blocksize doesn't work as expected
Date: Fri, 04 Mar 2011 14:30:38 +0100

On Fri, 2011-03-04 at 08:02 +0300, Станислав wrote:
> [dd] bs=blocksize doesn't work as expected, for example:
> tar -c /somedir | dd bs=256K of=/dev/nst0
> In this example backup record will be recorded to a tape with block size =
> 10K instead of 256K because 10K is default for tar.
> There is workaround:
> tar -c /somedir | dd ibs=256K obs=256K of=/dev/nst0
> If we set block size for input and output separatly - it works as expected.

Even partial block (short read) is a block - especially when you read
via pipe. If you want to have only fullblock, dd provides
iflag=fullblock for you. See info documentation (or online GNU coreutils
manual) for details.

> lucid <http://packages.ubuntu.com/lucid/coreutils> (utils): The GNU core
> utilities
> 7.4-2ubuntu2: amd64 i386

Current version is 8.10, but iflag=fullblock should be present there -
added in 7.0

Message #13 received at 8171-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Станислав <ginermail <at> gmail.com>
Cc: 8171-done <at> debbugs.gnu.org
Subject: Re: bug#8171: [dd] bs=blocksize doesn't work as expected
Date: Fri, 04 Mar 2011 14:02:35 +0000

On 04/03/11 05:02, Станислав wrote:
> [dd] bs=blocksize doesn't work as expected, for example:
> tar -c /somedir | dd bs=256K of=/dev/nst0
> In this example backup record will be recorded to a tape with block size =
> 10K instead of 256K because 10K is default for tar.
> There is workaround:
> tar -c /somedir | dd ibs=256K obs=256K of=/dev/nst0
> If we set block size for input and output separatly - it works as expected.

This operation is expected and dictated by POSIX.
It is surprising though, and I tried to clarify it in the docs:
http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=d5ab0599

Note also that we're currently considering warning
that iflag=fullblock should be used, but only
currently in the case were a count is specified,
as then you get the wrong amount of data as well
as the wrong sized writes.
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7362

cheers,
Pádraig.

Message #16 received at 8171 <at> debbugs.gnu.org (full text, mbox):

From: Станислав <ginermail <at> gmail.com>
To: 8171 <at> debbugs.gnu.org
Date: Fri, 4 Mar 2011 18:27:38 +0300

[Message part 1 (text/plain, inline)]

Thanks a lot for clarifying!

I also think it would be nice to notice this behaviour clearer in
documentation. man dd says almost nothing about. info dd says about 'short
reads' but it seems not so easy to understand what it means. 'dd' is
populary utility in scripts so many people can bumb into this behaviour. As
for me it took two months to catch 'slow' problem with our tape drive.

PS: Is it good work around that I mentioned before (I mean 'ibs=256K
obs=256K') or it can be changed in future?

Best regards,
Stanislav

[Message part 2 (text/html, inline)]

Message #19 received at 8171 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Станислав <ginermail <at> gmail.com>
Cc: 8171 <at> debbugs.gnu.org
Subject: Re: bug#8171:
Date: Fri, 04 Mar 2011 08:41:40 -0700

[Message part 1 (text/plain, inline)]

On 03/04/2011 08:27 AM, Станислав wrote:
> Thanks a lot for clarifying!
> 
> I also think it would be nice to notice this behaviour clearer in
> documentation. man dd says almost nothing about. info dd says about 'short
> reads' but it seems not so easy to understand what it means. 'dd' is
> populary utility in scripts so many people can bumb into this behaviour. As
> for me it took two months to catch 'slow' problem with our tape drive.
> 
> PS: Is it good work around that I mentioned before (I mean 'ibs=256K
> obs=256K') or it can be changed in future?

POSIX requires that 'ibs=n obs=n' behave the way you want; it is _only_
the 'bs=n' behavior where a short read results in surprising behavior.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #22 received at 8171 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Станислав <ginermail <at> gmail.com>
Cc: 8171 <at> debbugs.gnu.org
Subject: Re: bug#8171:
Date: Fri, 04 Mar 2011 15:45:04 +0000

On 04/03/11 15:27, Станислав wrote:
> Thanks a lot for clarifying!
> 
> I also think it would be nice to notice this behaviour clearer in
> documentation. man dd says almost nothing about. info dd says about 'short
> reads' but it seems not so easy to understand what it means. 'dd' is
> populary utility in scripts so many people can bumb into this behaviour. As
> for me it took two months to catch 'slow' problem with our tape drive.

I'll think about improving the man page, maybe by doing:

-  bs=BYTES        read and write BYTES bytes at a time (also see ibs=,obs=)
+  bs=BYTES        read and write up to BYTES bytes at a time

-  ibs=BYTES       read BYTES bytes at a time (default: 512)
-  ibs=BYTES       read up to BYTES bytes at a time (default: 512)

> PS: Is it good work around that I mentioned before (I mean 'ibs=256K
> obs=256K') or it can be changed in future?

That behavior will not change and is a valid workaround,
that works on all flavors of dd

cheers,
Pádraig.

Message #25 received at 8171 <at> debbugs.gnu.org (full text, mbox):

From: Станислав <ginermail <at> gmail.com>
To: 8171 <at> debbugs.gnu.org
Date: Sat, 5 Mar 2011 11:10:46 +0300

[Message part 1 (text/plain, inline)]

Thank you very much!

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.