GNU bug report logs - #68216
An instance of DD receiving data from another instance with bs=1M reports the wrong record count.

Previous Next

To reply to this bug, email your comments to 68216 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "jared <at> johnstone.net.au" <jared <at> johnstone.net.au>
To: bug-coreutils <at> gnu.org
Subject: An instance of DD receiving data from another instance with bs=1M
 reports the wrong record count.
Date: Tue, 2 Jan 2024 21:45:36 +1100

[Message part 1 (text/plain, inline)]

Hi all,

It seems an instance of `dd`  with only an output file doesn't report 
the count of records accurately despite bs= being described as 
influencing both ibs= and obs=

dd'ing from an unstable remote over ssh today I had it continue from the 
last received block between crashes in preparation for a P2V. The remote 
had bs=1M set with its in=device and the local instance had bs=1M too 
with the destination of= device but upon referring to the receiving 
end's stable count of records I noticed it ignored my bs=1M setting. 
This were worked around using obs=1M to make the counter show a value I 
could reliable skip=/seek= with to resume.

I have an example here:

   /$ dd if=/dev/zero bs=1M count=1 | dd bs=1M of=/dev/null
   1+0 records in
   1+0 records out
   1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0010021 s, 1.0 GB/s
   *0+16 records in
   0+16 records out*
   1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.000771722 s, 1.4 GB/s/

This is being experienced on Archlinux with kernel '6.6.4' and package 
'core/coreutils 9.4-2'

dd's man page for this distribution shows:

   /bs=BYTES
                  read and write up to BYTES bytes at a time (default:
   512); overrides ibs and obs/

So I feel it not overriding obs= when invoked with only of= and bs= 
arguments could be a bug.

Running a single dd with /if=source of=dest bs=1M/ works as intended.

While in my case this isn't a fatal bug working with virtual block 
devices it could be problematic for use cases which require a proper 
treatment of block sizes such as tape devices.

Best regards,
Jared

[Message part 2 (text/html, inline)]

[OpenPGP_0xB648A7603ED419B7.asc (application/pgp-keys, attachment)]

[OpenPGP_signature.asc (application/pgp-signature, attachment)]

Message #8 received at 68216 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: "jared <at> johnstone.net.au" <jared <at> johnstone.net.au>, 68216 <at> debbugs.gnu.org
Subject: Re: An instance of DD receiving data from another instance with bs=1M
 reports the wrong record count.
Date: Tue, 2 Jan 2024 17:48:17 +0000

forcemerge 8171 68216
stop

On 02/01/2024 10:45, jared <at> johnstone.net.au wrote:
> Hi all,
> 
> It seems an instance of `dd`  with only an output file doesn't report
> the count of records accurately despite bs= being described as
> influencing both ibs= and obs=
> 
> dd'ing from an unstable remote over ssh today I had it continue from the
> last received block between crashes in preparation for a P2V. The remote
> had bs=1M set with its in=device and the local instance had bs=1M too
> with the destination of= device but upon referring to the receiving
> end's stable count of records I noticed it ignored my bs=1M setting.
> This were worked around using obs=1M to make the counter show a value I
> could reliable skip=/seek= with to resume.
> 
> I have an example here:
> 
>      /$ dd if=/dev/zero bs=1M count=1 | dd bs=1M of=/dev/null
>      1+0 records in
>      1+0 records out
>      1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0010021 s, 1.0 GB/s
>      *0+16 records in
>      0+16 records out*
>      1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.000771722 s, 1.4 GB/s/
> 
> This is being experienced on Archlinux with kernel '6.6.4' and package
> 'core/coreutils 9.4-2'
> 
> dd's man page for this distribution shows:
> 
>      /bs=BYTES
>                     read and write up to BYTES bytes at a time (default:
>      512); overrides ibs and obs/
> 
> So I feel it not overriding obs= when invoked with only of= and bs=
> arguments could be a bug.
> 
> Running a single dd with /if=source of=dest bs=1M/ works as intended.
> 
> While in my case this isn't a fatal bug working with virtual block
> devices it could be problematic for use cases which require a proper
> treatment of block sizes such as tape devices.

Well the behavior is expected and documented:

‘bs=BYTES’
     Set both input and output block sizes to BYTES.  This makes ‘dd’
     read and write BYTES per block, overriding any ‘ibs’ and ‘obs’
     settings.  In addition, if no data-transforming ‘conv’ operand is
     specified, input is copied to the output as soon as it’s read, even
     if it is smaller than the block size.

For details please see https://bugs.gnu.org/8171

thanks,
Pádraig.

Message #11 received at 68216 <at> debbugs.gnu.org (full text, mbox):

From: "jared <at> johnstone.net.au" <jared <at> johnstone.net.au>
To: 68216 <at> debbugs.gnu.org
Subject: Re: Re: An instance of DD receiving data from another instance with
 bs=1M, reports the wrong record count.
Date: Wed, 3 Jan 2024 09:45:56 +1100

[Message part 1 (text/plain, inline)]

Hi Pádraig,

iflag=fullblock seems to resolve this experience and the man page 
excerpt provided makes sense though those lines make no appearance in 
the man pages on Archlinux. I will follow up with the distro maintainers.

Thanks,
Jared

[OpenPGP_0xB648A7603ED419B7.asc (application/pgp-keys, attachment)]

[OpenPGP_signature.asc (application/pgp-signature, attachment)]

Message #14 received at 68216 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: "jared <at> johnstone.net.au" <jared <at> johnstone.net.au>, 68216 <at> debbugs.gnu.org
Subject: Re: An instance of DD receiving data from another instance with
 bs=1M, reports the wrong record count.
Date: Tue, 2 Jan 2024 23:04:38 +0000

On 02/01/2024 22:45, jared <at> johnstone.net.au wrote:
> Hi Pádraig,
> 
> iflag=fullblock seems to resolve this experience and the man page
> excerpt provided makes sense though those lines make no appearance in
> the man pages on Archlinux. I will follow up with the distro maintainers.

As would explicitly specifying obs on the second dd instance.

cheers,
Pádraig

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.