GNU bug report logs - #8490
dd reads random number of records from pipes - named or otherwise - coreutils 8.9

Previous Next

Full log

Message #21 received at 8490-done <at> debbugs.gnu.org (full text, mbox):

From: Jesse Gordon <jesseg <at> nikola.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 8490-done <at> debbugs.gnu.org
Subject: Re: bug#8490: dd reads random number of records from pipes - named
	or	otherwise - coreutils 8.9
Date: Wed, 13 Apr 2011 11:28:02 -0700

On 4/13/2011 7:07 AM, Eric Blake wrote:
> On 04/12/2011 03:02 PM, Jesse Gordon wrote:
>> I can't believe such an obvious bug would exist this long, but on the
>> other hand the test is so simple I can't see where it's user error.
> Thanks for the report.  And you are correct in surmising that it is user
> error and not a bug in dd.
>
>> dd, when reading from stdin or from a named pipe sometimes (but not
>> always) reads a random number of records a bit less then what it should.
> Rather, dd reads as many bytes as possible, but unless that is less than
> PIPE_MAX, it is not guaranteed to be an atomic read.  In turn, if you
> have asked dd to pad out partial reads into complete writes, then that
> explains your problem.  Unfortunately, it is rather easy to do this
> without realizing it; the POSIX wording on how dd behaves is rather
> detailed.
>
How have I asked dd to pad out partial read? I'm not specifying 
pad or sync or anything.
And why is reading from a pipe a partial read when there is 
neither EOF or error?


Behold:  ifp=stdin; readbytes=fread(buffer, 1, ibs, ifp);

That reads in ibs bytes quite nicely from a pipe. It waits for 
all the data to fill into buffer, and only bails for the 
legitimate reasons -- like EOF, or some real error.

If POSIX really requires dd to abort a read for any reason other 
then EOF or an error, then I'm dumbfounded. To me, it seems 
obvious that the rule should be "When asked to read from a pipe, 
don't quit till it's done or becomes impossible."

dd doesn't seem to abort early when reading hard drives, even if 
the block size isn't the same as the hard drive IO read size, or 
even if it has to wait a few ms for the drive to seek.

I guess it's a good thing that POSIX doesn't say "dd must perform 
a segfault every 10th run on a full moon" ha ha :=)

It sure looks like poorly implemented non-blocking IO to me.

However, iflags=fullblock seems to fix it.

>> I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and even
>> mkfifo pip; yes>  pipe&  dd if=pipe -- and all sometimes failed.
>> However, if=arealfile seems to always work perfectly.
>>
>> To replicate:
>>
>> yes|dd bs=1000 count=1000|wc -c
> There's your problem.  bs=1000 is the key that tells dd to always write
> 1000 byte output blocks, even if the input block hit a short read, and
> stop after 1000 reads.
>
> Instead, try:
>
> yes|dd ibs=1000 obs=1000 count=1000|wc -c
>
I tried this but it still stops early. I copied and pasted 
exactly your command:

root <at> stats:~# yes|dd ibs=1000 obs=1000 count=1000|wc -c
564+436 records in
574+1 records out
574464 bytes (574 kB) copied, 0.0180807 s, 31.8 MB/s
574464

> which tells dd to explicitly read in input blocks of 1000 bytes, even if
> it requires multiple reads, prior to doing output blocks of 1000 bytes,
> and stop after 1000 writes.
>
>> The problem exists on the following coronations:
> It exists everywhere that dd complies with POSIX, even with non-GNU dd.
>   Because POSIX requires the difference in behavior between bs=nnn vs.
> ibs=nnn obs=nnn.
>
I still cannot fathom why it would ever be acceptable to abort 
early when there's no error and no EOF and the pipe is still 
sending data. That just goes against all good programming common 
sense as far as my tiny brain cell can tell.

Is there actually EVER a real reason for dd to need to abort a 
read when there's no EOF and no error? Why would POSIX require this?

~~~~~

Okay, I poked around in the source code a bit. Now I'll tell you 
I'm not great programmer so don't be afraid to say "Jesse, here's 
where you're wrong..."

However, I added in some fprintf(stderr, statements in various 
places to help me understand where and why dd is bailing.
I put an fprintf() at each break inside of the while(1){} main 
loop, and it's breaking at the first one:


  while (1)
    {
      if ((r_partial + r_full) >= max_records)
        {
                static uintmax_t temp;
                temp=r_partial + r_full;
                fprintf(stderr,"break 001: ");
                fprintf(stderr,"r_partial=%d ",r_partial);
                fprintf(stderr,"r_full=%d ",r_full);
                fprintf(stderr,"max_records=%d ",max_records);
                fprintf(stderr,"temp=%d\n",temp);
                break;
        }


And when I run it, I get:

root <at> stats:/big/src/coreutils-8.9# yes|src/dd bs=1000 
count=1000|wc -c
Setting max_records=1000 due to count=1000 being specified.
break 001: r_partial=394 r_full=606 max_records=1000 temp=1000
606+394 records in
606+394 records out
618304 bytes (618 kB) copied, 0.0757696 s, 8.2 MB/s
618304


Okay, so that makes sense enough: If it's read count records 
(whole or partially) then it's done reading because it's 
satisfied count=.

So the question is why is it reading partial records?
r_partial gets incremented whenever iread() returns less then 
input_blocksize. So why should iread() return less then ibs from 
a stream when simply trying again should finish the read?

static ssize_t
iread (int fd, char *buf, size_t size)
{
  while (true)
    {
      ssize_t nread;
      process_signals ();
      nread = read (fd, buf, size);
      if (! (nread < 0 && errno == EINTR))
        return nread;
    }
}

As you can see, it is supposed to keep trying until it gets some 
data. It does block if there is no data to read yet. The problem 
is when it happens to retry when some but not all of the data has 
arrived at the pipe.

I added in more fprintf()s and in my case, read() is always 
setting errno to 29=ESPIPE (something about an error seeking on a 
pipe.)

In trying to understand the inverted logic of the return code, 
I'm re ordering it here:

      if(nread < 0 && errno == EINTR) //If read returned -1 
(ERROR) and that error is "Interrupted by Signal, then"
      {
          //keep trying
       }
        else //If we read 0 or more bytes, OR there was an error 
OTHER then EINTR (Interrupted by Signal) then
        {
            return nread;
        }

So why don't we want to return on EINTR? or am I all confused?
~~~~
Anyway, of course just below iread() is iread_fullblock() which 
it appears does exactly how it should for reading from pipes.

Maybe iread_fullblock() should always be used for reading from pipes.

Ohwell.

I really have a hard time believing that posix requries DD to 
abort a pipe read just because the data wasn't ready quick enough.

What use of dd would actually be broken if iread() didn't abort  
for ESPIPE when reading a pipe?
It should be as simple as making it to keep reading and appending 
of errno==ESPIPE and read() returned other then 0.

Can someone point me to where POSIX requires this current 
behavior of dd? or am I just hopelessly confused?
Is the requirement actually specific to reading from pipes? or is 
reading from pipes broken due to trying to satisfy some 
requirement relating to the reading of block devices or whatever?

But at least I have a work around -- iflags=fullblock.

Thank you all very much,

Jesse

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.