Subject: Re: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9

From: Eric Blake <[email-masked]>
Date: Thu, 14 Apr 2011 15:45:51 -0600

On 04/14/2011 02:53 PM, Jesse Gordon wrote:
> But, look at POSIX: It says under INPUT FILES that "The input file can
> be any file type" -- if we take that to include blocks and pipes, then
> we can see that dd is not compliant because it cannot reliably read from
> a pipe.

dd _is_ reliably reading from a pipe.  The problem is that POSIX has no
option like iflags=fullblock to consolidate short reads into a single
buffer before doing output, so the reliable read ends shy of the amount
of data that you were hoping for.

The solution to this is not to rant on this list, but to propose an
enhancement to POSIX to standardize iflags=fullblock.

> Fact: Here's a prime example: Let's say I happen to want to read exactly
> 999983 blocks of 2099 bytes each from a pipe. With purely POSIX DD, it's
> impossible. Doesn't that violate POSIX?

No.  Just because POSIX doesn't have the command line options _in dd
alone_ to do what you want doesn't mean that dd is broken (just that the
POSIX standard is not as useful as it could be, by the fact that you
have to rely on extensions to the standard), nor does it mean that you
can't do what you want with other POSIX tools.

Remember, the problem is not that you can't read from pipes, but that
when you couple the count=nnn with short reads you don't get full data.
 So the solution, using only POSIX, is to get rid of count=nnn, and
isolate the input counting from the output blocking.  Unfortunately,
POSIX also states that 'head -c' is not portable, but that other
utilities in the standard provide the same functionality, without
stating what those other utilities are (I'm guessing that whoever wrote
that sentence was referring to dd).

If you don't care about efficiency, then you can avoid the problem by
avoiding short reads (read from a blocking pipe will always return at
least one byte, since returning 0 implies EOF):

yes | dd ibs=1 count=$((999983*2099)) | dd obs=1000 | wc -c

proves that you can read exactly 999983*2099 bytes, with only one
partial write in the second dd.

> Doesn't a program's ability to function as described in POSIX preempt
> technical details about short reads from read()?

Coreutils' dd _does_ function as described in POSIX.

> 
> And if POSIX really intends for pipe as input file to be not guaranteed
> usable, then it ought to mention something about that, but it doesn't.

Raise that as a bug against POSIX, then:
http://austingroupbugs.net/main_page.php

> Any thinking observer would, I believe, have to conclude that dd is
> supposed to work perfectly to read 999983 blocks of 2099 bytes each from
> a pipe.

There's nothing in POSIX that requires that, unfortunately.

> Do you really think that the POSIX folks meant for dd to be unusable for
> reading from pipes?

POSIX merely intended for the default behavior of dd to match its
historical behavior, which really was unusable for reading from pipes as
historically implemented.  That said, there is nothing wrong with
proposing that POSIX add a new option to make dd more useful when
reading from pipes.

> I notice that the POSIX documents we're reading are copyright through
> 2008 -- is there a newer one out now?

2008 is the newest version.  Technical Corrigendum 1 to that revision
will probably come out later this year.  Membership in the Austin Group
(the committee that maintains POSIX) is free to anyone who wants to join.

> Do you have any suggestions for getting in touch with IEEE?

Join the Austin Group - the IEEE generally blanket-approves whatever the
Austin Group says when it comes to POSIX.

-- 
Eric Blake   [email-masked]    +1-801-349-2682
Libvirt virtualization library http://libvirt.org