GNU bug report logs - #7362
dd strangeness

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Pádraig Brady <P <at> draigBrady.com>
Cc: tracker <at> debbugs.gnu.org
Subject: bug#7362: closed (dd strangeness)
Date: Wed, 10 Nov 2010 15:26:01 +0000

[Message part 1 (text/plain, inline)]

Your message dated Wed, 10 Nov 2010 15:29:41 +0000
with message-id <4CDABA65.9030808 <at> draigBrady.com>
and subject line Re: bug#7362: dd strangeness
has caused the GNU bug report #7362,
regarding dd strangeness
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
7362: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7362
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Lucia Rotger <lucia <at> aircomp.aero>
To: bug-coreutils <at> gnu.org
Subject: dd strangeness
Date: Wed, 10 Nov 2010 11:22:25 +0100

I see this behavior in Solaris, Linux and BSD dd: if I send a big enough 
file they all read it short at the end of the stream.

This works as expected:

# cat /dev/zero | dd bs=512 count=293601280 | wc

I get the expected results, dd reads exactly 293601280 blocks and wc 
sees 150323855360 characters, 140 GB

Whereas substituting cat for zfs send doesn't:

# zfs send <backup> | dd bs=512 count=293601280 | wc

The output of one of the runs is

293590463+10817 records in
293590463+10817 records out

and the bytes counted by wc are < 140 GB. The zfs command sends 600 GB, 
so obviously dd should not run short.

BSD and Linux dd were used on BSD and Linux machines, respectively, 
piping the stream with nc.

Since this happens with three different implementations of dd I'm 
thinking of a design flaw but I've never ecountered it before. I'm 
testing sdd (a dd replacement) and will see what happens, but it'll take 
5 hours still. There seems to be something going on in dd with different 
input and output block sizes since both sdd and this 
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/517773 hint at 
it: "The dd process requires a ridiculous amount of CPU during startup, 
though, since it is running with bs=1 to not miss stuff". But I don't 
know if that's what's happening here. According to man dd, bs sets ibs 
and obs.

bs=512 is the last attempt I made but I've tried combinations of the bs 
and count parameters (always to make a size of 140 GB) to no avail, 
nothing seems to work with a big stream. I still haven't tried bs=1 as I 
think it would take weeks to go through but maybe I'm wrong. If I try 
with smaller files, up to hundreds of MBs dd works fine, but I can't 
tell at what size it breaks or under which circumstances or why.

[Message part 3 (message/rfc822, inline)]

From: Pádraig Brady <P <at> draigBrady.com>
To: Lucia Rotger <lucia <at> aircomp.aero>
Cc: 7362-done <at> debbugs.gnu.org
Subject: Re: bug#7362: dd strangeness
Date: Wed, 10 Nov 2010 15:29:41 +0000

On 10/11/10 10:22, Lucia Rotger wrote:
> I see this behavior in Solaris, Linux and BSD dd: if I send a big enough
> file they all read it short at the end of the stream.
> 
> This works as expected:
> 
> # cat /dev/zero | dd bs=512 count=293601280 | wc
> 
> I get the expected results, dd reads exactly 293601280 blocks and wc
> sees 150323855360 characters, 140 GB
> 
> Whereas substituting cat for zfs send doesn't:
> 
> # zfs send <backup> | dd bs=512 count=293601280 | wc

different write sizes to the pipe mean
in the later case, dd will get short reads.
IMHO dd is doing the wrong/most surprising thing here,
but it can't be changed for compatibility reasons.
You can get coreutils dd to do what you want with:

dd iflag=fullblock

cheers,
Pádraig.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.