GNU bug report logs - #22277
'dd' - stats are not what expected

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22277 in the body.
You can then email your comments to 22277 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mike Fiedler <micfied <at> yandex.com>
To: bug-coreutils <at> gnu.org
Subject: 'dd' - stats are not what expected
Date: Wed, 30 Dec 2015 20:11:19 -0500

[Message part 1 (text/html, inline)]

Message #8 received at 22277 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Mike Fiedler <micfied <at> yandex.com>, 22277 <at> debbugs.gnu.org
Subject: Re: bug#22277: 'dd' - stats are not what expected
Date: Thu, 31 Dec 2015 10:18:10 +0000

unarchive 17505
forcemerge 17505 22277
stop

On 31/12/15 01:11, Mike Fiedler wrote:
>  
> Hi,
>  
> I ran one of my favorite utilities 'dd' again this evening, this time with bs=1G ( IEC ) - I usually do 1M but this time I dealt with more data to be copied...
> I had to copy about 215 GiB of data from one to another drive ( offset 215 GiB was about the end of the last partition ).
> So I did:
>  
> $  dd if=/dev/sdb of=/dev/sda bs=*1G* count=*222*
> 222+0 records in
> 222+0 records out
> 238370684928 bytes (*238 GB*) copied, 1275.03 s, 187 MB/s
> 
> When it finished, I got a bit confused, and I asked myself a question if the data I requested did really get copied..  of course it did, but I was not expecting 238 GB to be shown.
> To make sure I calculated the 512 byte sector end number out of the 238370684928 bytes 'dd' result and compared it with the output of fdisk showing the last sector of the last partition... I was fine.
>  
> I think, and many others have a same opinion, 1kB = 1000B, etc, should be banned from use in the IT world, and banned from use by the sales people.
>  
> The point is, as you probably noticed, if dd is told to use IEC, let's stick to IEC and not get the results in whatever artificial decimal crap....
> It can not only confuse, but utility like 'dd' should be 100% specific about handling the units, and there should be not a bit of doubt when it spits out the results.
> If I would use 1K in this case, I would not notice the difference - my brain is simply too simple, and small, but 1G should at least result in displaying 222 GiB and for sure not GB.

I have to agree, and this has come up a few times now.

The number in brackets is not exact and informational for human consumption,
so we should make an effort to be less confusing.
There was a proposed patch at:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17505#37
which auto determines the appropriate base from the amount output,
to output the number with the least amount of info loss.

There were some issues noted with that,
but IMHO they were lesser than the current issue.

We will have to be careful to not corrupt output
when switching with status=progress (due to possibly shorter status line).

I'll have another look.

thanks,
Pádraig.

Message #13 received at 22277 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>,
 Mike Fiedler <micfied <at> yandex.com>, 22277 <at> debbugs.gnu.org
Subject: Re: bug#22277: 'dd' - stats are not what expected
Date: Thu, 31 Dec 2015 14:14:48 -0800

[Message part 1 (text/plain, inline)]

Paul Eggert wrote:

> Ah. Is this due to glitches when the size grows from (say) "1023 KiB" to "1.0
> MiB", or is it something else?

Assuming that's the problem, I installed the attached further patches to try to 
fix it. The gnulib update fixes an off-by-one bug I encountered in gnulib while 
testing this.

[0001-dd-append-spaces-to-shorter-status-progress-line.patch (text/x-diff, attachment)]

[0002-build-update-gnulib-submodule-to-latest.patch (text/x-diff, attachment)]

Message #16 received at 22277 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>,
 Mike Fiedler <micfied <at> yandex.com>, 22277 <at> debbugs.gnu.org
Subject: Re: bug#22277: 'dd' - stats are not what expected
Date: Thu, 31 Dec 2015 11:56:40 -0800

[Message part 1 (text/plain, inline)]

Pádraig Brady wrote:
> There were some issues noted with that,
> but IMHO they were lesser than the current issue.

No matter which format we pick, fans of the other format will complain. Instead, 
let's output the information in both formats. I did that with the attached patch.

> We will have to be careful to not corrupt output
> when switching with status=progress (due to possibly shorter status line).

I didn't know about this problem. Could you explain it?

Ah. Is this due to glitches when the size grows from (say) "1023 KiB" to "1.0 
MiB", or is it something else?

[0001-dd-summarize-in-human-readable-format-too.patch (text/x-diff, attachment)]

Message #19 received at 22277 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Mike Fiedler <micfied <at> yandex.com>, 22277 <at> debbugs.gnu.org
Subject: Re: bug#22277: 'dd' - stats are not what expected
Date: Thu, 31 Dec 2015 21:51:19 +0000

[Message part 1 (text/plain, inline)]

On 31/12/15 10:18, Pádraig Brady wrote:
> unarchive 17505
> forcemerge 17505 22277
> stop
> 
> On 31/12/15 01:11, Mike Fiedler wrote:
>>  
>> Hi,
>>  
>> I ran one of my favorite utilities 'dd' again this evening, this time with bs=1G ( IEC ) - I usually do 1M but this time I dealt with more data to be copied...
>> I had to copy about 215 GiB of data from one to another drive ( offset 215 GiB was about the end of the last partition ).
>> So I did:
>>  
>> $  dd if=/dev/sdb of=/dev/sda bs=*1G* count=*222*
>> 222+0 records in
>> 222+0 records out
>> 238370684928 bytes (*238 GB*) copied, 1275.03 s, 187 MB/s
>>
>> When it finished, I got a bit confused, and I asked myself a question if the data I requested did really get copied..  of course it did, but I was not expecting 238 GB to be shown.
>> To make sure I calculated the 512 byte sector end number out of the 238370684928 bytes 'dd' result and compared it with the output of fdisk showing the last sector of the last partition... I was fine.
>>  
>> I think, and many others have a same opinion, 1kB = 1000B, etc, should be banned from use in the IT world, and banned from use by the sales people.
>>  
>> The point is, as you probably noticed, if dd is told to use IEC, let's stick to IEC and not get the results in whatever artificial decimal crap....
>> It can not only confuse, but utility like 'dd' should be 100% specific about handling the units, and there should be not a bit of doubt when it spits out the results.
>> If I would use 1K in this case, I would not notice the difference - my brain is simply too simple, and small, but 1G should at least result in displaying 222 GiB and for sure not GB.
> 
> I have to agree, and this has come up a few times now.
> 
> The number in brackets is not exact and informational for human consumption,
> so we should make an effort to be less confusing.
> There was a proposed patch at:
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17505#37
> which auto determines the appropriate base from the amount output,
> to output the number with the least amount of info loss.
> 
> There were some issues noted with that,
> but IMHO they were lesser than the current issue.
> 
> We will have to be careful to not corrupt output
> when switching with status=progress (due to possibly shorter status line).

Resending as previous email seems dropped somewhere:

The attached auto sets the units.
For status=progress this is done based on output block size,
for the final transfer stats, it's done based on the transferred byte count.

cheers,
Pádraig

[dd-stats-units.patch (text/x-patch, attachment)]

Message #24 received at 22277 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 22277 <at> debbugs.gnu.org
Subject: Re: bug#22277: 'dd' - stats are not what expected
Date: Thu, 31 Dec 2015 17:27:46 -0800

Pádraig Brady wrote:
> I had proposed a patch to erase that extra
> output with \e[K.

Sorry, I didn't notice that proposal (maybe mailing list problems?).

Is \e[K portable? If so, the code in my recently-installed patch could be 
simplified, since it wouldn't need to count columns.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.