On 04/14/2011 02:53 PM, Jesse Gordon wrote: > But, look at POSIX: It says under INPUT FILES that "The input file can > be any file type" -- if we take that to include blocks and pipes, then > we can see that dd is not compliant because it cannot reliably read from > a pipe. dd _is_ reliably reading from a pipe. The problem is that POSIX has no option like iflags=fullblock to consolidate short reads into a single buffer before doing output, so the reliable read ends shy of the amount of data that you were hoping for. The solution to this is not to rant on this list, but to propose an enhancement to POSIX to standardize iflags=fullblock. > Fact: Here's a prime example: Let's say I happen to want to read exactly > 999983 blocks of 2099 bytes each from a pipe. With purely POSIX DD, it's > impossible. Doesn't that violate POSIX? No. Just because POSIX doesn't have the command line options _in dd alone_ to do what you want doesn't mean that dd is broken (just that the POSIX standard is not as useful as it could be, by the fact that you have to rely on extensions to the standard), nor does it mean that you can't do what you want with other POSIX tools. Remember, the problem is not that you can't read from pipes, but that when you couple the count=nnn with short reads you don't get full data. So the solution, using only POSIX, is to get rid of count=nnn, and isolate the input counting from the output blocking. Unfortunately, POSIX also states that 'head -c' is not portable, but that other utilities in the standard provide the same functionality, without stating what those other utilities are (I'm guessing that whoever wrote that sentence was referring to dd). If you don't care about efficiency, then you can avoid the problem by avoiding short reads (read from a blocking pipe will always return at least one byte, since returning 0 implies EOF): yes | dd ibs=1 count=$((999983*2099)) | dd obs=1000 | wc -c proves that you can read exactly 999983*2099 bytes, with only one partial write in the second dd. > Doesn't a program's ability to function as described in POSIX preempt > technical details about short reads from read()? Coreutils' dd _does_ function as described in POSIX. > > And if POSIX really intends for pipe as input file to be not guaranteed > usable, then it ought to mention something about that, but it doesn't. Raise that as a bug against POSIX, then: http://austingroupbugs.net/main_page.php > Any thinking observer would, I believe, have to conclude that dd is > supposed to work perfectly to read 999983 blocks of 2099 bytes each from > a pipe. There's nothing in POSIX that requires that, unfortunately. > Do you really think that the POSIX folks meant for dd to be unusable for > reading from pipes? POSIX merely intended for the default behavior of dd to match its historical behavior, which really was unusable for reading from pipes as historically implemented. That said, there is nothing wrong with proposing that POSIX add a new option to make dd more useful when reading from pipes. > I notice that the POSIX documents we're reading are copyright through > 2008 -- is there a newer one out now? 2008 is the newest version. Technical Corrigendum 1 to that revision will probably come out later this year. Membership in the Austin Group (the committee that maintains POSIX) is free to anyone who wants to join. > Do you have any suggestions for getting in touch with IEEE? Join the Austin Group - the IEEE generally blanket-approves whatever the Austin Group says when it comes to POSIX. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org