GNU bug report logs - #12626
Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine?

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 12626 in the body.
You can then email your comments to 12626 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
Subject: Bug?: dd  limited to <2G read size (2G-8K) on 64 bit machine?
Date: Thu, 11 Oct 2012 20:11:53 -0700

I'm running on a 64-bit version of openSuse 12.1 with a 3.6 vanilla kernel.

I find that if I try to use a read size of > (2G-8K), I get partial read 
errors.

Ishtar:> free
            total       used       free     shared    buffers     cached
Mem:      49487868   31923108   17564760          0    1077920   18871456
-/+ buffers/cache:   11973732   37514136
Swap:      8393924      18308    8375616

---
Should have plenty of memory.  I do have huge page support turned on.
ulimit doesn't look to be the problem (tried as root)
core file size          (blocks, -c) 100000
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 386074
max locked memory       (kbytes, -l) 512
max memory size         (kbytes, -m) 42064712
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 386074
virtual memory          (kbytes, -v) 46305440
file locks                      (-x) unlimited

Message #8 received at 12626 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 12626 <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Thu, 11 Oct 2012 21:51:05 -0700

On 10/11/2012 08:11 PM, Linda Walsh wrote:
> I find that if I try to use a read size of > (2G-8K), I get partial read errors.

My guess is that it's your kernel, or maybe your
file system, and not dd per se.  Try running 'strace'.

Message #11 received at 12626 <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 12626 <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Thu, 11 Oct 2012 23:39:16 -0700

[Message part 1 (text/plain, inline)]

Paul Eggert wrote:
> On 10/11/2012 08:11 PM, Linda Walsh wrote:
>   
>> I find that if I try to use a read size of > (2G-8K), I get partial read errors.
>>     
>
> My guess is that it's your kernel, or maybe your
> file system, and not dd per se.  Try running 'strace'.
>   

open("/dev/Home+Space/Home", O_RDONLY|O_DIRECT) = 3
dup2(3, 0)                              = 0
close(3)                                = 0
lseek(0, 0, SEEK_CUR)                   = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
dup2(3, 1)                              = 1
close(3)                                = 0
mmap(NULL, 2147495936, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 
-1, 0) = 0x7f1e5efae000
read(0, 
"XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
2147483648) = 2147479552
read(0, 
"XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
2147483648) = 2147479552
write(1, 
"XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
2147479552) = 2147479552
read(0, "   this.context.sourceBoxHighlig"..., 2147483648) = 2147479552
open("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/coreutils.mo", 
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) 
= -1 ENOENT (No such file or directory)
----

Maybe has something to do with lvm...am guessing...I was trying to dd 
read from an lvm device raw....

I'll look into it...but I'm not even getting error messages if I try to 
do 4G @ a time...

dd if=/dev/Home+Space/Home bs=2G count=4 of=/dev/null iflag=direct
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+4 records in
0+4 records out
8589918208 bytes (8.6 GB) copied, 9.1148 s, 942 MB/s
dd if=/dev/Home+Space/Home bs=4G count=2 of=/dev/null iflag=direct
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+2 records in
0+2 records out
4294959104 bytes (4.3 GB) copied, 5.4533 s, 788 MB/s
dd if=/dev/Home+Space/Home bs=8G count=1 of=/dev/null iflag=direct
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 3.58152 s, 600 MB/s
...

The larger the block size asked for... the smaller I got???

[Message part 2 (text/html, inline)]

Message #14 received at 12626 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 12626 <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Thu, 11 Oct 2012 23:42:33 -0700

On 10/11/2012 11:39 PM, Linda Walsh wrote:
> I'm not even getting error messages if I try to do 4G @ a time

That's what dd is supposed to do.  It requests 4 GB,
the kernel gives it only 2 GB, so it goes with 2 GB.
dd has always worked that way, and POSIX requires it
to work that way.

Message #19 received at 12626-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Linda Walsh <coreutils <at> tlinx.org>, 12626-done <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 10:14:13 +0100

tag 12626 notabug
stop

On 10/12/2012 07:42 AM, Paul Eggert wrote:
> On 10/11/2012 11:39 PM, Linda Walsh wrote:
>> I'm not even getting error messages if I try to do 4G @ a time
>
> That's what dd is supposed to do.  It requests 4 GB,
> the kernel gives it only 2 GB, so it goes with 2 GB.
> dd has always worked that way, and POSIX requires it
> to work that way.

As my mother would say, that's a lazy man's load.
Please use a more reasonable buffer size.

Also note the iflag=fullblock option, which
will ensure the internal buffer is filled.
Then count= will refer to the number of buffers
processed, rather than the number of read()s.

Also note if you specify ibs and obs seperately,
then you will get writes of the size you requested at least.
But be careful with using this behavior in conjunction
with count=, as that will limit the number of read()s.

cheers,
Pádraig.

Message #22 received at 12626-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Linda Walsh <coreutils <at> tlinx.org>, 12626-done <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 07:34:54 -0700

On 10/12/2012 02:14 AM, Pádraig Brady wrote:
> if you specify ibs and obs seperately,
> then you will get writes of the size you requested at least.

Won't there be similar problems with the write system
call too?  Perhaps not on Linda's system, but on other
systems.  Once the buffer size exceeds 2 GB or so,
things get pretty dicey in the wild, not due to any
problem in dd itself, but due to the kernels or
file systems that dd relies on.

Message #25 received at 12626-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Linda Walsh <coreutils <at> tlinx.org>, 12626-done <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 15:37:15 +0100

On 10/12/2012 03:34 PM, Paul Eggert wrote:
> On 10/12/2012 02:14 AM, Pádraig Brady wrote:
>> if you specify ibs and obs seperately,
>> then you will get writes of the size you requested at least.
>
> Won't there be similar problems with the write system
> call too?  Perhaps not on Linda's system, but on other
> systems.  Once the buffer size exceeds 2 GB or so,
> things get pretty dicey in the wild, not due to any
> problem in dd itself, but due to the kernels or
> file systems that dd relies on.

True, but at least you'd get errors in that case.

cheers,
Pádraig.

Message #28 received at 12626-done <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Pádraig Brady <P <at> draigBrady.com>, 12626-done <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 08:14:37 -0700

[Message part 1 (text/plain, inline)]

I was writing to /dev/null.  It doesn't seem to have an issue.

This appears to be related to "iflag=direct" more than anything.

W/o direct it works, but, for example I can't use iflag=direct with 
/dev/zero
for even a 4K block size.

Hmmm....




Paul Eggert wrote:
> On 10/12/2012 02:14 AM, Pádraig Brady wrote:
>   
>> if you specify ibs and obs seperately,
>> then you will get writes of the size you requested at least.
>>     
>
> Won't there be similar problems with the write system
> call too?  Perhaps not on Linda's system, but on other
> systems.  Once the buffer size exceeds 2 GB or so,
> things get pretty dicey in the wild, not due to any
> problem in dd itself, but due to the kernels or
> file systems that dd relies on.
>   

[Message part 2 (text/html, inline)]

Message #31 received at 12626-done <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Pádraig Brady <P <at> draigBrady.com>, 12626-done <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 08:25:10 -0700

[Message part 1 (text/plain, inline)]

It's not 'direct':
Ishtar:dev/shm# dd if=/dev/zero of=4G bs=2G count=2
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+2 records in
0+2 records out
4294959104 bytes (4.3 GB) copied, 4.29234 s, 1.0 GB/s
Ishtar:dev/shm# dd if=/dev/zero of=4G bs=2G count=2 iflag=fullblock
2+0 records in
2+0 records out
4294967296 bytes (4.3 GB) copied, 5.41603 s, 793 MB/s
---
From /dev/shm (file to file)
Ishtar:dev/shm# dd if=4G of=4Ga bs=2G count=2
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+2 records in
0+2 records out
4294959104 bytes (4.3 GB) copied, 5.52481 s, 777 MB/s
-----------------


Hey guys, this is still a bug though:

Ishtar:dev/shm# dd if=4G of=4Ga bs=4G count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 4.0274 s, 533 MB/s
(note no error message....)




Paul Eggert wrote:
> On 10/12/2012 02:14 AM, Pádraig Brady wrote:
>   
>> if you specify ibs and obs seperately,
>> then you will get writes of the size you requested at least.
>>     
>
> Won't there be similar problems with the write system
> call too?  Perhaps not on Linda's system, but on other
> systems.  Once the buffer size exceeds 2 GB or so,
> things get pretty dicey in the wild, not due to any
> problem in dd itself, but due to the kernels or
> file systems that dd relies on.
>   

[Message part 2 (text/html, inline)]

Message #34 received at 12626-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>,
	Pádraig Brady <P <at> draigBrady.com>,
	12626-done <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 09:35:20 -0600

[Message part 1 (text/plain, inline)]

On 10/12/2012 09:25 AM, Linda Walsh wrote:

[please don't top-post on technical lists]

> 
> Hey guys, this is still a bug though:
> 
> Ishtar:dev/shm# dd if=4G of=4Ga bs=4G count=1
> 0+1 records in
> 0+1 records out
> 2147479552 bytes (2.1 GB) copied, 4.0274 s, 533 MB/s
> (note no error message....)

Nope, that's not a bug, but behavior required by POSIX.  You asked dd to
read _up to 4G_ for a count of exactly 1.  Just because the read was
short does not make it an error.

-- 
Eric Blake   eblake <at> redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #37 received at 12626-done <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Pádraig Brady <P <at> draigBrady.com>, 12626-done <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 08:40:27 -0700

[Message part 1 (text/plain, inline)]

It is a bug if there is a check for short reads
and that message doesn't get triggered.

There is a check for short reads .... the message isn't
being written out.

You are a complete idiot if you think that is not a bug.

POSIX compliance isn't the only standard of whether or not something is
a bug.  If code is included to check for an error condition
(short reads), and DOES trigger in some cases, but not in others,
that is the essence of a bug in the code -- regardless of specs.

If you don't like my posts, Erik, you are free to ignore them.
Otherwise, stop adding off-topic comments like this:

Eric Blake wrote:
> On 10/12/2012 09:25 AM, Linda Walsh wrote:
>
> [please don't top-post on technical lists]
>   

>   
>> Hey guys, this is still a bug though:
>>
>> Ishtar:dev/shm# dd if=4G of=4Ga bs=4G count=1
>> 0+1 records in
>> 0+1 records out
>> 2147479552 bytes (2.1 GB) copied, 4.0274 s, 533 MB/s
>> (note no error message....)
>>     
>
> Nope, that's not a bug, but behavior required by POSIX.  You asked dd to
> read _up to 4G_ for a count of exactly 1.  Just because the read was
> short does not make it an error
>   

[Message part 2 (text/html, inline)]

Message #40 received at 12626-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>,
	Pádraig Brady <P <at> draigBrady.com>,
	12626-done <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 09:44:55 -0600

[Message part 1 (text/plain, inline)]

On 10/12/2012 09:40 AM, Linda Walsh wrote:
> It is a bug if there is a check for short reads
> and that message doesn't get triggered.

Short reads are NOT an error in dd.  If you want to pretend that short
reads didn't happen, by re-reading until the buffer is full, then use
iflag=fullblock, as has already been mentioned to you earlier in this
thread.

-- 
Eric Blake   eblake <at> redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #43 received at 12626 <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 12626 <at> debbugs.gnu.org
Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit
	machine?
Date: Fri, 12 Oct 2012 10:17:42 -0700

[Message part 1 (text/plain, inline)]

I tried using gvim instead... ;-)

I found:
 /* Warn about partial reads if bs=SIZE is given and iflag=fullblock
    is not, and if counting or skipping bytes or using direct I/O.
    This helps to avoid confusion with miscounts, and to avoid issues
    with direct I/O on GNU/Linux.  */
 warn_partial_read =
   (! (conversions_mask & C_TWOBUFS) && ! (input_flags & O_FULLBLOCK)
    && (skip_records
        || (0 < max_records && max_records < (uintmax_t) -1)
        || (input_flags | output_flags) & O_DIRECT));


------------
I'm not doing conversions and didn't have fullblock set.
I'm not skipping records
input has o_direct set...

but the troublesome line:
        || (0 < max_records && max_records < (uintmax_t) -1)
I asked to copy 1,2 or 4 records

uintmax -1 =  0xffff fffe --- I don't understand, if max_records is >0
and less than ~4G-1, set this flag?

I'm assuming it's a flag to display the message or not, as I know it 
doesn't display the message most of the time...

Is that right uintmax?   or should that be an unsigned long int max?



But I don't think that's the root cause of what I am seeing.  But that 
statement doesn't look right....
It acts more like something (maybe not dd), is running with a 32-bit 
word size.

ldd shows dd linking with lib64 targets:

>  ldd dd
       linux-vdso.so.1 (0x00007fff6d5ff000)
       librt.so.1 => /lib64/librt.so.1 (0x0000003001800000)
       libc.so.6 => /lib64/libc.so.6 (0x0000003000400000)
       libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003001000000)
       /lib64/ld-linux-x86-64.so.2 (0x0000003000000000)

---
Does dd have a 32-bit limit on numb blocks?




Paul Eggert wrote:
> On 10/11/2012 08:11 PM, Linda Walsh wrote:
>   
>> I find that if I try to use a read size of > (2G-8K), I get partial read errors.
>>     
>
> My guess is that it's your kernel, or maybe your
> file system, and not dd per se.  Try running 'strace'.
>   

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.