GNU bug report logs -
#8490
dd reads random number of records from pipes - named or otherwise - coreutils 8.9
Previous Next
Reported by: Jesse Gordon <jesseg <at> nikola.com>
Date: Wed, 13 Apr 2011 02:46:02 UTC
Severity: normal
Done: Eric Blake <eblake <at> redhat.com>
Bug is archived. No further changes may be made.
Full log
View this message in rfc822 format
[Message part 1 (text/plain, inline)]
Your message dated Wed, 13 Apr 2011 08:07:39 -0600
with message-id <4DA5AE2B.2070508 <at> redhat.com>
and subject line Re: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9
has caused the GNU bug report #8490,
regarding dd reads random number of records from pipes - named or otherwise - coreutils 8.9
to be marked as done.
(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)
--
8490: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8490
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems
[Message part 2 (message/rfc822, inline)]
I can't believe such an obvious bug would exist this long, but on
the other hand the test is so simple I can't see where it's user
error.
dd, when reading from stdin or from a named pipe sometimes (but
not always) reads a random number of records a bit less then what
it should.
I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and
even mkfifo pip; yes > pipe & dd if=pipe -- and all sometimes
failed.
However, if=arealfile seems to always work perfectly.
To replicate:
yes|dd bs=1000 count=1000|wc -c
cat /dev/zero |dd bs=1000 count=1000 |wc -c
If it works perfectly the first time, just keep trying. For me,
it randomly works and doesn't work. Some conditions are more
likely to work, and others to fail.
For me, using the "yes" method above almost always reads the
incorrect number of bytes, while the /dev/zero method usually
works correctly but occasionally reads the wrong number of bytes.
The problem exists on the following coronations:
Slackware 12.0.0 (2.6.21.5-smp) / coreutils 8.9
Slackware 12.0.0 (2.6.21.5-smp) / coreutils 6.9
Slackware 13.0.0 (2.6.29.6) / coreutils 7.9
Kubuntu lenny/sid (2.6.24-27-generic SMP) / coreutils 6.10
(I only mention the older versions of coreutils because it may be
helpful. I'm only filing the bug report for 8.9!)
Thanks,
Jesse
For example:
root <at> stats:~# dd --version
dd (coreutils) 8.9
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Paul Rubin, David MacKenzie, and Stuart Kemp.
root <at> stats:~# yes|dd bs=1000 count=1000|wc -c
597+403 records in
597+403 records out
608232 bytes (608 kB) copied, 0.0263828 s, 23.1 MB/s
608232
root <at> stats:~# yes|dd bs=1000 count=1000|wc -c
885+115 records in
885+115 records out
887784 bytes (888 kB) copied, 0.054488 s, 16.3 MB/s
887784
root <at> stats:~# yes|dd bs=1000 count=1000|wc -c
696+304 records in
696+304 records out
705536 bytes (706 kB) copied, 0.0271789 s, 26.0 MB/s
705536
root <at> stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
1000000
1000+0 records in
1000+0 records out
1000000 bytes (1.0 MB) copied, 0.00879434 s, 114 MB/s
root <at> stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
972+28 records in
972+28 records out
993040 bytes (993 kB) copied, 0.00582009 s, 171 MB/s
993040
root <at> stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
983+17 records in
983+17 records out
996040 bytes (996 kB) copied, 0.0102457 s, 97.2 MB/s
996040
root <at> stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
1000000
1000+0 records in
1000+0 records out
1000000 bytes (1.0 MB) copied, 0.0181759 s, 55.0 MB/s
root <at> stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c
1000+0 records in
1000+0 records out
1000000 bytes (1.0 MB) copied, 0.010386 s, 96.3 MB/s
1000000
root <at> stats:~#
[Message part 3 (message/rfc822, inline)]
[Message part 4 (text/plain, inline)]
On 04/12/2011 03:02 PM, Jesse Gordon wrote:
> I can't believe such an obvious bug would exist this long, but on the
> other hand the test is so simple I can't see where it's user error.
Thanks for the report. And you are correct in surmising that it is user
error and not a bug in dd.
> dd, when reading from stdin or from a named pipe sometimes (but not
> always) reads a random number of records a bit less then what it should.
Rather, dd reads as many bytes as possible, but unless that is less than
PIPE_MAX, it is not guaranteed to be an atomic read. In turn, if you
have asked dd to pad out partial reads into complete writes, then that
explains your problem. Unfortunately, it is rather easy to do this
without realizing it; the POSIX wording on how dd behaves is rather
detailed.
> I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and even
> mkfifo pip; yes > pipe & dd if=pipe -- and all sometimes failed.
> However, if=arealfile seems to always work perfectly.
>
> To replicate:
>
> yes|dd bs=1000 count=1000|wc -c
There's your problem. bs=1000 is the key that tells dd to always write
1000 byte output blocks, even if the input block hit a short read, and
stop after 1000 reads.
Instead, try:
yes|dd ibs=1000 obs=1000 count=1000|wc -c
which tells dd to explicitly read in input blocks of 1000 bytes, even if
it requires multiple reads, prior to doing output blocks of 1000 bytes,
and stop after 1000 writes.
>
> The problem exists on the following coronations:
It exists everywhere that dd complies with POSIX, even with non-GNU dd.
Because POSIX requires the difference in behavior between bs=nnn vs.
ibs=nnn obs=nnn.
--
Eric Blake eblake <at> redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
[signature.asc (application/pgp-signature, attachment)]
This bug report was last modified 14 years and 99 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.