Subject: Re: bug#7362: dd strangeness

From: Pádraig Brady <[email-masked]>
Date: Fri, 12 Nov 2010 15:20:07 +0000

On 12/11/10 14:53, Lucia Rotger wrote:
> On 10/11/2010 16:29, Pádraig Brady wrote:
>> On 10/11/10 10:22, Lucia Rotger wrote:
>>> I see this behavior in Solaris, Linux and BSD dd: if I send a big enough
>>> file they all read it short at the end of the stream.
>>>
>>> This works as expected:
>>>
>>> # cat /dev/zero | dd bs=512 count=293601280 | wc
>>>
>>> I get the expected results, dd reads exactly 293601280 blocks and wc
>>> sees 150323855360 characters, 140 GB
>>>
>>> Whereas substituting cat for zfs send doesn't:
>>>
>>> # zfs send<backup>  | dd bs=512 count=293601280 | wc
>>
>> different write sizes to the pipe mean
>> in the later case, dd will get short reads.
>> IMHO dd is doing the wrong/most surprising thing here,
>> but it can't be changed for compatibility reasons.
>> You can get coreutils dd to do what you want with:
>>
>> dd iflag=fullblock
> 
> Thanks for the explanation. Now, if I may abuse your time a bit more,
> how exactly is a block size determined when reading from a pipe? is it
> the size returned by each read()? And why does dd stop short at the last
> block?
> 
> Sorry about those problems in the man page being already fixed. I
> checked and I have dd version 6.1, these Ubuntu guys don't update very
> well.
> 
> I have somewhat solved it in a more portable way doing
> 
> # zfs send<backup> | dd obs=512 | dd bs=512 count=293601280 | wc

The above works because the middle dd coalesces the short reads,
and so writes 512 bytes at a time, and so the next dd will not
have short reads. This will not generally work as you increase the size.

To illustrate what the middle dd is doing:

$ (echo 1; sleep 1; echo 2) | strace -e read,write dd obs=512
read(0, "1\n", 512)                     = 2
read(0, "2\n", 512)                     = 2
read(0, "", 512)                        = 0
write(1, "1\n2\n", 4)                   = 4

> I suppose adding ibs=1 to the first dd would work with everything but it
> would probably take ages to finish.

right. The crux of your problem is specifying a count.
If you didn't the short reads wouldn't matter.
If you do need to specify a count, then the iflag=fullblock
available since 7.0 will work.

cheers,
Pádraig.