From unknown Fri Jun 20 07:14:56 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#12626 <12626@debbugs.gnu.org> To: bug#12626 <12626@debbugs.gnu.org> Subject: Status: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? Reply-To: bug#12626 <12626@debbugs.gnu.org> Date: Fri, 20 Jun 2025 14:14:56 +0000 retitle 12626 Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? reassign 12626 coreutils submitter 12626 Linda Walsh severity 12626 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Thu Oct 11 23:12:59 2012 Received: (at submit) by debbugs.gnu.org; 12 Oct 2012 03:12:59 +0000 Received: from localhost ([127.0.0.1]:40129 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMVgH-0006Nb-0E for submit@debbugs.gnu.org; Thu, 11 Oct 2012 23:12:58 -0400 Received: from eggs.gnu.org ([208.118.235.92]:45043) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMVgC-0006NM-V2 for submit@debbugs.gnu.org; Thu, 11 Oct 2012 23:12:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TMVfL-00006Z-3T for submit@debbugs.gnu.org; Thu, 11 Oct 2012 23:11:59 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:56270) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TMVfL-00006V-0U for submit@debbugs.gnu.org; Thu, 11 Oct 2012 23:11:59 -0400 Received: from eggs.gnu.org ([208.118.235.92]:57483) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TMVfK-0008Eg-7O for bug-coreutils@gnu.org; Thu, 11 Oct 2012 23:11:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TMVfJ-00006K-CY for bug-coreutils@gnu.org; Thu, 11 Oct 2012 23:11:58 -0400 Received: from ishtar.tlinx.org ([173.164.175.65]:43752) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TMVfJ-00006G-1v for bug-coreutils@gnu.org; Thu, 11 Oct 2012 23:11:57 -0400 Received: from [192.168.3.12] (Athenae [192.168.3.12]) by Ishtar.tlinx.org (8.14.5/8.14.4/SuSE Linux 0.8) with ESMTP id q9C3BqUa029990 for ; Thu, 11 Oct 2012 20:11:55 -0700 Message-ID: <50778A79.5070106@tlinx.org> Date: Thu, 11 Oct 2012 20:11:53 -0700 From: Linda Walsh User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.24) Gecko/20100228 Lightning/0.9 Thunderbird/2.0.0.24 Mnenhy/0.7.6.666 MIME-Version: 1.0 To: bug-coreutils@gnu.org Subject: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 208.118.235.17 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) I'm running on a 64-bit version of openSuse 12.1 with a 3.6 vanilla kernel. I find that if I try to use a read size of > (2G-8K), I get partial read errors. Ishtar:> free total used free shared buffers cached Mem: 49487868 31923108 17564760 0 1077920 18871456 -/+ buffers/cache: 11973732 37514136 Swap: 8393924 18308 8375616 --- Should have plenty of memory. I do have huge page support turned on. ulimit doesn't look to be the problem (tried as root) core file size (blocks, -c) 100000 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 386074 max locked memory (kbytes, -l) 512 max memory size (kbytes, -m) 42064712 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 386074 virtual memory (kbytes, -v) 46305440 file locks (-x) unlimited From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 00:52:04 2012 Received: (at 12626) by debbugs.gnu.org; 12 Oct 2012 04:52:04 +0000 Received: from localhost ([127.0.0.1]:40166 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMXEB-0000B7-I2 for submit@debbugs.gnu.org; Fri, 12 Oct 2012 00:52:04 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:58095) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMXE7-0000Aa-M8 for 12626@debbugs.gnu.org; Fri, 12 Oct 2012 00:52:01 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 7A14BA6000A; Thu, 11 Oct 2012 21:51:05 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oxquDmyCLJ+0; Thu, 11 Oct 2012 21:51:05 -0700 (PDT) Received: from [192.168.1.3] (pool-108-23-119-2.lsanca.fios.verizon.net [108.23.119.2]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 1FE7DA60008; Thu, 11 Oct 2012 21:51:05 -0700 (PDT) Message-ID: <5077A1B9.8010402@cs.ucla.edu> Date: Thu, 11 Oct 2012 21:51:05 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux i686; rv:15.0) Gecko/20120912 Thunderbird/15.0.1 MIME-Version: 1.0 To: Linda Walsh Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> In-Reply-To: <50778A79.5070106@tlinx.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12626 Cc: 12626@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) On 10/11/2012 08:11 PM, Linda Walsh wrote: > I find that if I try to use a read size of > (2G-8K), I get partial read errors. My guess is that it's your kernel, or maybe your file system, and not dd per se. Try running 'strace'. From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 02:40:24 2012 Received: (at 12626) by debbugs.gnu.org; 12 Oct 2012 06:40:24 +0000 Received: from localhost ([127.0.0.1]:40232 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMYv1-0002eX-Lg for submit@debbugs.gnu.org; Fri, 12 Oct 2012 02:40:24 -0400 Received: from ishtar.tlinx.org ([173.164.175.65]:60089) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMYuv-0002eL-Eq for 12626@debbugs.gnu.org; Fri, 12 Oct 2012 02:40:19 -0400 Received: from [192.168.3.12] (Athenae [192.168.3.12]) by Ishtar.tlinx.org (8.14.5/8.14.4/SuSE Linux 0.8) with ESMTP id q9C6dFNg012215; Thu, 11 Oct 2012 23:39:17 -0700 Message-ID: <5077BB14.7010701@tlinx.org> Date: Thu, 11 Oct 2012 23:39:16 -0700 From: Linda Walsh User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.24) Gecko/20100228 Lightning/0.9 Thunderbird/2.0.0.24 Mnenhy/0.7.6.666 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> In-Reply-To: <5077A1B9.8010402@cs.ucla.edu> Content-Type: multipart/alternative; boundary="------------090503000105060604010304" X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12626 Cc: 12626@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) This is a multi-part message in MIME format. --------------090503000105060604010304 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Paul Eggert wrote: > On 10/11/2012 08:11 PM, Linda Walsh wrote: > >> I find that if I try to use a read size of > (2G-8K), I get partial read errors. >> > > My guess is that it's your kernel, or maybe your > file system, and not dd per se. Try running 'strace'. > open("/dev/Home+Space/Home", O_RDONLY|O_DIRECT) = 3 dup2(3, 0) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = 0 open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 dup2(3, 1) = 1 close(3) = 0 mmap(NULL, 2147495936, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1e5efae000 read(0, "XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2147483648) = 2147479552 read(0, "XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2147483648) = 2147479552 write(1, "XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2147479552) = 2147479552 read(0, " this.context.sourceBoxHighlig"..., 2147483648) = 2147479552 open("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) ---- Maybe has something to do with lvm...am guessing...I was trying to dd read from an lvm device raw.... I'll look into it...but I'm not even getting error messages if I try to do 4G @ a time... dd if=/dev/Home+Space/Home bs=2G count=4 of=/dev/null iflag=direct dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock 0+4 records in 0+4 records out 8589918208 bytes (8.6 GB) copied, 9.1148 s, 942 MB/s dd if=/dev/Home+Space/Home bs=4G count=2 of=/dev/null iflag=direct dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock 0+2 records in 0+2 records out 4294959104 bytes (4.3 GB) copied, 5.4533 s, 788 MB/s dd if=/dev/Home+Space/Home bs=8G count=1 of=/dev/null iflag=direct 0+1 records in 0+1 records out 2147479552 bytes (2.1 GB) copied, 3.58152 s, 600 MB/s ... The larger the block size asked for... the smaller I got??? --------------090503000105060604010304 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by Ishtar.tlinx.org id q9C6dFNg012215

Paul Eggert wrote: 
On 10/11/2012 08:11 PM, Linda Walsh wrote:
  
I find that if I try to use a read size of > (2G-8K=
), I get partial read errors.
    

My guess is that it's your kernel, or maybe your
file system, and not dd per se.  Try running 'strace'.

open("/dev/Home+Space/Home", O_RDONLY|O_DIRECT) =3D 3
dup2(3, 0)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0
close(3)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0
lseek(0, 0, SEEK_CUR)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) =3D 3
dup2(3, 1)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 1
close(3)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0
mmap(NULL, 2147495936, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D 0x7f1e5efae000
read(0, "XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2147483648) =3D 2147479552
read(0, "XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2147483648) =3D 2147479552
write(1, "XFSB\0\0\20\0\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2147479552) =3D 2147479552
read(0, "=C2=A0=C2=A0 this.context.sourceBoxHighlig"..., 2147483648) =3D = 2147479552
open("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) =3D -1 ENOENT (No such file or directory)
----

Maybe has something to do with lvm...am guessing...I was trying to dd read from an lvm device raw....

I'll look into it...but I'm not even getting error messages if I try to do 4G @ a time...

dd if=3D/dev/Home+Space/Home bs=3D2G count=3D4 of=3D/dev/null iflag=3Ddir= ect
dd: warning: partial read (2147479552 bytes); suggest iflag=3Dfullblock 0+4 records in
0+4 records out
8589918208 bytes (8.6 GB) copied, 9.1148 s, 942 MB/s
dd if=3D/dev/Home+Space/Home bs=3D4G count=3D2 of=3D/dev/null iflag=3Ddir= ect
dd: warning: partial read (2147479552 bytes); suggest iflag=3Dfullblock 0+2 records in
0+2 records out
4294959104 bytes (4.3 GB) copied, 5.4533 s, 788 MB/s
dd if=3D/dev/Home+Space/Home bs=3D8G count=3D1 of=3D/dev/null iflag=3Ddir= ect
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 3.58152 s, 600 MB/s
...

The larger the block size asked for... the smaller I got???






--------------090503000105060604010304-- From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 02:43:37 2012 Received: (at 12626) by debbugs.gnu.org; 12 Oct 2012 06:43:37 +0000 Received: from localhost ([127.0.0.1]:40236 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMYy8-0002iz-JP for submit@debbugs.gnu.org; Fri, 12 Oct 2012 02:43:36 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:33943) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMYy5-0002il-0t for 12626@debbugs.gnu.org; Fri, 12 Oct 2012 02:43:34 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 93FA8A6000C; Thu, 11 Oct 2012 23:42:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PX2g+h52KJj4; Thu, 11 Oct 2012 23:42:38 -0700 (PDT) Received: from [192.168.1.3] (pool-108-23-119-2.lsanca.fios.verizon.net [108.23.119.2]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 3BF5DA60008; Thu, 11 Oct 2012 23:42:38 -0700 (PDT) Message-ID: <5077BBD9.3040508@cs.ucla.edu> Date: Thu, 11 Oct 2012 23:42:33 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux i686; rv:15.0) Gecko/20120912 Thunderbird/15.0.1 MIME-Version: 1.0 To: Linda Walsh Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> In-Reply-To: <5077BB14.7010701@tlinx.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12626 Cc: 12626@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) On 10/11/2012 11:39 PM, Linda Walsh wrote: > I'm not even getting error messages if I try to do 4G @ a time That's what dd is supposed to do. It requests 4 GB, the kernel gives it only 2 GB, so it goes with 2 GB. dd has always worked that way, and POSIX requires it to work that way. From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 05:15:18 2012 Received: (at 12626-done) by debbugs.gnu.org; 12 Oct 2012 09:15:18 +0000 Received: from localhost ([127.0.0.1]:40429 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMbKw-0007uj-07 for submit@debbugs.gnu.org; Fri, 12 Oct 2012 05:15:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:10805) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMbKs-0007ua-G2 for 12626-done@debbugs.gnu.org; Fri, 12 Oct 2012 05:15:15 -0400 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q9C9EGbD009398 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 12 Oct 2012 05:14:16 -0400 Received: from [10.36.116.62] (ovpn-116-62.ams2.redhat.com [10.36.116.62]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q9C9EDVi007901 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 12 Oct 2012 05:14:14 -0400 Message-ID: <5077DF65.5030207@draigBrady.com> Date: Fri, 12 Oct 2012 10:14:13 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> <5077BBD9.3040508@cs.ucla.edu> In-Reply-To: <5077BBD9.3040508@cs.ucla.edu> Content-Type: text/plain; charset=UTF-8; format=flowed X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id q9C9EGbD009398 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: 12626-done Cc: Linda Walsh , 12626-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) tag 12626 notabug stop On 10/12/2012 07:42 AM, Paul Eggert wrote: > On 10/11/2012 11:39 PM, Linda Walsh wrote: >> I'm not even getting error messages if I try to do 4G @ a time > > That's what dd is supposed to do. It requests 4 GB, > the kernel gives it only 2 GB, so it goes with 2 GB. > dd has always worked that way, and POSIX requires it > to work that way. As my mother would say, that's a lazy man's load. Please use a more reasonable buffer size. Also note the iflag=3Dfullblock option, which will ensure the internal buffer is filled. Then count=3D will refer to the number of buffers processed, rather than the number of read()s. Also note if you specify ibs and obs seperately, then you will get writes of the size you requested at least. But be careful with using this behavior in conjunction with count=3D, as that will limit the number of read()s. cheers, P=C3=A1draig. From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 10:36:02 2012 Received: (at 12626-done) by debbugs.gnu.org; 12 Oct 2012 14:36:02 +0000 Received: from localhost ([127.0.0.1]:41138 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMgLJ-0007qF-OG for submit@debbugs.gnu.org; Fri, 12 Oct 2012 10:36:02 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:49610) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMgLH-0007pl-El for 12626-done@debbugs.gnu.org; Fri, 12 Oct 2012 10:36:00 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 36257A6000C; Fri, 12 Oct 2012 07:35:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MKBU6aUvzxL5; Fri, 12 Oct 2012 07:35:02 -0700 (PDT) Received: from [192.168.1.3] (pool-108-23-119-2.lsanca.fios.verizon.net [108.23.119.2]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id A7521A60008; Fri, 12 Oct 2012 07:35:02 -0700 (PDT) Message-ID: <50782A8E.2010706@cs.ucla.edu> Date: Fri, 12 Oct 2012 07:34:54 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux i686; rv:16.0) Gecko/20121011 Thunderbird/16.0.1 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> <5077BBD9.3040508@cs.ucla.edu> <5077DF65.5030207@draigBrady.com> In-Reply-To: <5077DF65.5030207@draigBrady.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12626-done Cc: Linda Walsh , 12626-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) On 10/12/2012 02:14 AM, P=C3=A1draig Brady wrote: > if you specify ibs and obs seperately, > then you will get writes of the size you requested at least. Won't there be similar problems with the write system call too? Perhaps not on Linda's system, but on other systems. Once the buffer size exceeds 2 GB or so, things get pretty dicey in the wild, not due to any problem in dd itself, but due to the kernels or file systems that dd relies on. From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 10:38:20 2012 Received: (at 12626-done) by debbugs.gnu.org; 12 Oct 2012 14:38:20 +0000 Received: from localhost ([127.0.0.1]:41154 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMgNV-0007vK-Gl for submit@debbugs.gnu.org; Fri, 12 Oct 2012 10:38:20 -0400 Received: from mx1.redhat.com ([209.132.183.28]:17532) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMgNP-0007v7-VY for 12626-done@debbugs.gnu.org; Fri, 12 Oct 2012 10:38:14 -0400 Received: from int-mx02.intmail.prod.int.phx2.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q9CEbIh7003829 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 12 Oct 2012 10:37:18 -0400 Received: from [10.36.116.62] (ovpn-116-62.ams2.redhat.com [10.36.116.62]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q9CEbGwg003089 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 12 Oct 2012 10:37:17 -0400 Message-ID: <50782B1B.9050506@draigBrady.com> Date: Fri, 12 Oct 2012 15:37:15 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> <5077BBD9.3040508@cs.ucla.edu> <5077DF65.5030207@draigBrady.com> <50782A8E.2010706@cs.ucla.edu> In-Reply-To: <50782A8E.2010706@cs.ucla.edu> Content-Type: text/plain; charset=UTF-8; format=flowed X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id q9CEbIh7003829 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: 12626-done Cc: Linda Walsh , 12626-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) On 10/12/2012 03:34 PM, Paul Eggert wrote: > On 10/12/2012 02:14 AM, P=C3=A1draig Brady wrote: >> if you specify ibs and obs seperately, >> then you will get writes of the size you requested at least. > > Won't there be similar problems with the write system > call too? Perhaps not on Linda's system, but on other > systems. Once the buffer size exceeds 2 GB or so, > things get pretty dicey in the wild, not due to any > problem in dd itself, but due to the kernels or > file systems that dd relies on. True, but at least you'd get errors in that case. cheers, P=C3=A1draig. From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 11:15:35 2012 Received: (at 12626-done) by debbugs.gnu.org; 12 Oct 2012 15:15:35 +0000 Received: from localhost ([127.0.0.1]:41186 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMgxb-0000Ky-Ij for submit@debbugs.gnu.org; Fri, 12 Oct 2012 11:15:35 -0400 Received: from ishtar.tlinx.org ([173.164.175.65]:46966) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMgxY-0000Kq-WA for 12626-done@debbugs.gnu.org; Fri, 12 Oct 2012 11:15:34 -0400 Received: from [192.168.3.12] (Athenae [192.168.3.12]) by Ishtar.tlinx.org (8.14.5/8.14.4/SuSE Linux 0.8) with ESMTP id q9CFEbVl011728; Fri, 12 Oct 2012 08:14:39 -0700 Message-ID: <507833DD.5070506@tlinx.org> Date: Fri, 12 Oct 2012 08:14:37 -0700 From: Linda Walsh User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.24) Gecko/20100228 Lightning/0.9 Thunderbird/2.0.0.24 Mnenhy/0.7.6.666 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> <5077BBD9.3040508@cs.ucla.edu> <5077DF65.5030207@draigBrady.com> <50782A8E.2010706@cs.ucla.edu> In-Reply-To: <50782A8E.2010706@cs.ucla.edu> Content-Type: multipart/alternative; boundary="------------060307000306070802090200" X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12626-done Cc: Pádraig Brady , 12626-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) This is a multi-part message in MIME format. --------------060307000306070802090200 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by Ishtar.tlinx.org id q9CFEbVl011728 I was writing to /dev/null. It doesn't seem to have an issue. This appears to be related to "iflag=3Ddirect" more than anything. W/o direct it works, but, for example I can't use iflag=3Ddirect with=20 /dev/zero for even a 4K block size. Hmmm.... Paul Eggert wrote: > On 10/12/2012 02:14 AM, P=C3=A1draig Brady wrote: > =20 >> if you specify ibs and obs seperately, >> then you will get writes of the size you requested at least. >> =20 > > Won't there be similar problems with the write system > call too? Perhaps not on Linda's system, but on other > systems. Once the buffer size exceeds 2 GB or so, > things get pretty dicey in the wild, not due to any > problem in dd itself, but due to the kernels or > file systems that dd relies on. > =20 --------------060307000306070802090200 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by Ishtar.tlinx.org id q9CFEbVl011728 I was writing to /dev/null.=C2=A0 It doesn't seem to have an issue.

This appears to be related to "iflag=3Ddirect" more than anything.

W/o direct it works, but, for example I can't use iflag=3Ddirect with /dev/zero
for even a 4K block size.

Hmmm....




Paul Eggert wrote: 
On 10/12/2012 02:14 AM, P=C3=A1draig Brady wrote:
  
if you specify ibs and obs seperately,
then you will get writes of the size you requested at least.
    

Won't there be similar problems with the write system
call too?  Perhaps not on Linda's system, but on other
systems.  Once the buffer size exceeds 2 GB or so,
things get pretty dicey in the wild, not due to any
problem in dd itself, but due to the kernels or
file systems that dd relies on.
--------------060307000306070802090200-- From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 11:26:07 2012 Received: (at 12626-done) by debbugs.gnu.org; 12 Oct 2012 15:26:07 +0000 Received: from localhost ([127.0.0.1]:41191 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMh7m-0000ZQ-M1 for submit@debbugs.gnu.org; Fri, 12 Oct 2012 11:26:06 -0400 Received: from ishtar.tlinx.org ([173.164.175.65]:47118) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMh7k-0000ZG-Fd for 12626-done@debbugs.gnu.org; Fri, 12 Oct 2012 11:26:05 -0400 Received: from [192.168.3.12] (Athenae [192.168.3.12]) by Ishtar.tlinx.org (8.14.5/8.14.4/SuSE Linux 0.8) with ESMTP id q9CFPAep012522; Fri, 12 Oct 2012 08:25:12 -0700 Message-ID: <50783656.1040900@tlinx.org> Date: Fri, 12 Oct 2012 08:25:10 -0700 From: Linda Walsh User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.24) Gecko/20100228 Lightning/0.9 Thunderbird/2.0.0.24 Mnenhy/0.7.6.666 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> <5077BBD9.3040508@cs.ucla.edu> <5077DF65.5030207@draigBrady.com> <50782A8E.2010706@cs.ucla.edu> In-Reply-To: <50782A8E.2010706@cs.ucla.edu> Content-Type: multipart/alternative; boundary="------------000908000700070005000102" X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12626-done Cc: Pádraig Brady , 12626-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) This is a multi-part message in MIME format. --------------000908000700070005000102 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by Ishtar.tlinx.org id q9CFPAep012522 It's not 'direct': Ishtar:dev/shm# dd if=3D/dev/zero of=3D4G bs=3D2G count=3D2 dd: warning: partial read (2147479552 bytes); suggest iflag=3Dfullblock 0+2 records in 0+2 records out 4294959104 bytes (4.3 GB) copied, 4.29234 s, 1.0 GB/s Ishtar:dev/shm# dd if=3D/dev/zero of=3D4G bs=3D2G count=3D2 iflag=3Dfullb= lock 2+0 records in 2+0 records out 4294967296 bytes (4.3 GB) copied, 5.41603 s, 793 MB/s --- From /dev/shm (file to file) Ishtar:dev/shm# dd if=3D4G of=3D4Ga bs=3D2G count=3D2 dd: warning: partial read (2147479552 bytes); suggest iflag=3Dfullblock 0+2 records in 0+2 records out 4294959104 bytes (4.3 GB) copied, 5.52481 s, 777 MB/s ----------------- Hey guys, this is still a bug though: Ishtar:dev/shm# dd if=3D4G of=3D4Ga bs=3D4G count=3D1 0+1 records in 0+1 records out 2147479552 bytes (2.1 GB) copied, 4.0274 s, 533 MB/s (note no error message....) Paul Eggert wrote: > On 10/12/2012 02:14 AM, P=C3=A1draig Brady wrote: > =20 >> if you specify ibs and obs seperately, >> then you will get writes of the size you requested at least. >> =20 > > Won't there be similar problems with the write system > call too? Perhaps not on Linda's system, but on other > systems. Once the buffer size exceeds 2 GB or so, > things get pretty dicey in the wild, not due to any > problem in dd itself, but due to the kernels or > file systems that dd relies on. > =20 --------------000908000700070005000102 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by Ishtar.tlinx.org id q9CFPAep012522 It's not 'direct':
Ishtar:dev/shm# dd if=3D/dev/zero of=3D4G bs=3D2G count=3D2
dd: warning: partial read (2147479552 bytes); suggest iflag=3Dfullblock 0+2 records in
0+2 records out
4294959104 bytes (4.3 GB) copied, 4.29234 s, 1.0 GB/s
Ishtar:dev/shm# dd if=3D/dev/zero of=3D4G bs=3D2G count=3D2 iflag=3Dfullb= lock
2+0 records in
2+0 records out
4294967296 bytes (4.3 GB) copied, 5.41603 s, 793 MB/s
---
>From /dev/shm (file to file)
Ishtar:dev/shm# dd if=3D4G of=3D4Ga bs=3D2G count=3D2
dd: warning: partial read (2147479552 bytes); suggest iflag=3Dfullblock 0+2 records in
0+2 records out
4294959104 bytes (4.3 GB) copied, 5.52481 s, 777 MB/s
-----------------


Hey guys, this is still a bug though:

Ishtar:dev/shm# dd if=3D4G of=3D4Ga bs=3D4G count=3D1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 4.0274 s, 533 MB/s
(note no error message....)




Paul Eggert wrote: 
On 10/12/2012 02:14 AM, P=C3=A1draig Brady wrote:
  
if you specify ibs and obs seperately,
then you will get writes of the size you requested at least.
    

Won't there be similar problems with the write system
call too?  Perhaps not on Linda's system, but on other
systems.  Once the buffer size exceeds 2 GB or so,
things get pretty dicey in the wild, not due to any
problem in dd itself, but due to the kernels or
file systems that dd relies on.
--------------000908000700070005000102-- From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 11:36:18 2012 Received: (at 12626-done) by debbugs.gnu.org; 12 Oct 2012 15:36:18 +0000 Received: from localhost ([127.0.0.1]:41213 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMhHe-0000ol-3W for submit@debbugs.gnu.org; Fri, 12 Oct 2012 11:36:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43244) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMhHa-0000ob-8K for 12626-done@debbugs.gnu.org; Fri, 12 Oct 2012 11:36:16 -0400 Received: from int-mx12.intmail.prod.int.phx2.redhat.com (int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.25]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q9CFZLjX032760 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 12 Oct 2012 11:35:22 -0400 Received: from [10.3.113.23] (ovpn-113-23.phx2.redhat.com [10.3.113.23]) by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q9CFZLHT001514; Fri, 12 Oct 2012 11:35:21 -0400 Message-ID: <507838B8.5000704@redhat.com> Date: Fri, 12 Oct 2012 09:35:20 -0600 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120911 Thunderbird/15.0.1 MIME-Version: 1.0 To: Linda Walsh Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> <5077BBD9.3040508@cs.ucla.edu> <5077DF65.5030207@draigBrady.com> <50782A8E.2010706@cs.ucla.edu> <50783656.1040900@tlinx.org> In-Reply-To: <50783656.1040900@tlinx.org> X-Enigmail-Version: 1.4.4 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig8C30D1986DFA4168E3EA0663" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.25 X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: 12626-done Cc: Paul Eggert , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , 12626-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig8C30D1986DFA4168E3EA0663 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 10/12/2012 09:25 AM, Linda Walsh wrote: [please don't top-post on technical lists] >=20 > Hey guys, this is still a bug though: >=20 > Ishtar:dev/shm# dd if=3D4G of=3D4Ga bs=3D4G count=3D1 > 0+1 records in > 0+1 records out > 2147479552 bytes (2.1 GB) copied, 4.0274 s, 533 MB/s > (note no error message....) Nope, that's not a bug, but behavior required by POSIX. You asked dd to read _up to 4G_ for a count of exactly 1. Just because the read was short does not make it an error. --=20 Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --------------enig8C30D1986DFA4168E3EA0663 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBCAAGBQJQeDi4AAoJEKeha0olJ0NqV/UH/iWk2aEb36TaXqvxrNcfgpxP 1fHTfUy8WYC+XsSYjgOBwSoV2xHuolJV6ZQX6qqIXLN50LbRbeDC7ovodlXsJOoJ JB3rW3V5d5jDyp6q9IC+9QGjejtLMZeiQDpzWQpBqMmG2o5MxUfx9flO3a33oQvk zl2u71qfu8qb+VTnAhXpCxZSM0gmksJrNT8kQAu/j/1tLPnqCM+hn/0XYiDk7IKq 7w+2mab0U/RV0jQFb5VZzMX3MSQ0Gs0KS5Oo8LEanfnY86M/QV/W2Ya7LGYEFIF1 Ex0SROCxBR0q2yRJTrcLmUkANiK5n7Ab0MbW4Rv3YD7Lz1Ypjv9KDoKQDkt9k4k= =u+1/ -----END PGP SIGNATURE----- --------------enig8C30D1986DFA4168E3EA0663-- From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 11:41:22 2012 Received: (at 12626-done) by debbugs.gnu.org; 12 Oct 2012 15:41:22 +0000 Received: from localhost ([127.0.0.1]:41225 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMhMY-0000wG-He for submit@debbugs.gnu.org; Fri, 12 Oct 2012 11:41:22 -0400 Received: from ishtar.tlinx.org ([173.164.175.65]:47384) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMhMW-0000w9-VF for 12626-done@debbugs.gnu.org; Fri, 12 Oct 2012 11:41:21 -0400 Received: from [192.168.3.12] (Athenae [192.168.3.12]) by Ishtar.tlinx.org (8.14.5/8.14.4/SuSE Linux 0.8) with ESMTP id q9CFeQ1F013616; Fri, 12 Oct 2012 08:40:28 -0700 Message-ID: <507839EB.6010207@tlinx.org> Date: Fri, 12 Oct 2012 08:40:27 -0700 From: Linda Walsh User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.24) Gecko/20100228 Lightning/0.9 Thunderbird/2.0.0.24 Mnenhy/0.7.6.666 MIME-Version: 1.0 To: Eric Blake Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> <5077BBD9.3040508@cs.ucla.edu> <5077DF65.5030207@draigBrady.com> <50782A8E.2010706@cs.ucla.edu> <50783656.1040900@tlinx.org> <507838B8.5000704@redhat.com> In-Reply-To: <507838B8.5000704@redhat.com> Content-Type: multipart/alternative; boundary="------------090604060204080108030903" X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12626-done Cc: Paul Eggert , Pádraig Brady , 12626-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) This is a multi-part message in MIME format. --------------090604060204080108030903 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit It is a bug if there is a check for short reads and that message doesn't get triggered. There is a check for short reads .... the message isn't being written out. You are a complete idiot if you think that is not a bug. POSIX compliance isn't the only standard of whether or not something is a bug. If code is included to check for an error condition (short reads), and DOES trigger in some cases, but not in others, that is the essence of a bug in the code -- regardless of specs. If you don't like my posts, Erik, you are free to ignore them. Otherwise, stop adding off-topic comments like this: Eric Blake wrote: > On 10/12/2012 09:25 AM, Linda Walsh wrote: > > [please don't top-post on technical lists] > > >> Hey guys, this is still a bug though: >> >> Ishtar:dev/shm# dd if=4G of=4Ga bs=4G count=1 >> 0+1 records in >> 0+1 records out >> 2147479552 bytes (2.1 GB) copied, 4.0274 s, 533 MB/s >> (note no error message....) >> > > Nope, that's not a bug, but behavior required by POSIX. You asked dd to > read _up to 4G_ for a count of exactly 1. Just because the read was > short does not make it an error > --------------090604060204080108030903 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by Ishtar.tlinx.org id q9CFeQ1F013616 It is a bug if there is a check for short reads
and that message doesn't get triggered.

There is a check for short reads .... the message isn't
being written out.

You are a complete idiot if you think that is not a bug.

POSIX compliance isn't the only standard of whether or not something is a bug.=C2=A0 If code is included to check for an error condition
(short reads), and DOES trigger in some cases, but not in others,
that is the essence of a bug in the code -- regardless of specs.

If you don't like my posts, Erik, you are free to ignore them.
Otherwise, stop adding off-topic comments like this:

Eric Blake wrote: 
On 10/12/2012 09:25 AM, Linda Walsh wrote:

[please don't top-post on technical lists]
  

  
Hey guys, this is still a bug though:

Ishtar:dev/shm# dd if=3D4G of=3D4Ga bs=3D4G count=3D1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 4.0274 s, 533 MB/s
(note no error message....)
    

Nope, that's not a bug, but behavior required by POSIX.  You asked dd to
read _up to 4G_ for a count of exa=
ctly 1.  Just because the read was
short does not make it an error
--------------090604060204080108030903-- From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 11:45:53 2012 Received: (at 12626-done) by debbugs.gnu.org; 12 Oct 2012 15:45:53 +0000 Received: from localhost ([127.0.0.1]:41241 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMhQs-00013K-Mt for submit@debbugs.gnu.org; Fri, 12 Oct 2012 11:45:51 -0400 Received: from mx1.redhat.com ([209.132.183.28]:9400) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMhQq-00013B-Nh for 12626-done@debbugs.gnu.org; Fri, 12 Oct 2012 11:45:49 -0400 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q9CFiuaS015483 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 12 Oct 2012 11:44:56 -0400 Received: from [10.3.113.23] (ovpn-113-23.phx2.redhat.com [10.3.113.23]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q9CFitiW025850; Fri, 12 Oct 2012 11:44:56 -0400 Message-ID: <50783AF7.1000709@redhat.com> Date: Fri, 12 Oct 2012 09:44:55 -0600 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120911 Thunderbird/15.0.1 MIME-Version: 1.0 To: Linda Walsh Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> <5077BB14.7010701@tlinx.org> <5077BBD9.3040508@cs.ucla.edu> <5077DF65.5030207@draigBrady.com> <50782A8E.2010706@cs.ucla.edu> <50783656.1040900@tlinx.org> <507838B8.5000704@redhat.com> <507839EB.6010207@tlinx.org> In-Reply-To: <507839EB.6010207@tlinx.org> X-Enigmail-Version: 1.4.4 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigA64C4CCDE98200ACAD31AE91" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: 12626-done Cc: Paul Eggert , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , 12626-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA64C4CCDE98200ACAD31AE91 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 10/12/2012 09:40 AM, Linda Walsh wrote: > It is a bug if there is a check for short reads > and that message doesn't get triggered. Short reads are NOT an error in dd. If you want to pretend that short reads didn't happen, by re-reading until the buffer is full, then use iflag=3Dfullblock, as has already been mentioned to you earlier in this thread. --=20 Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --------------enigA64C4CCDE98200ACAD31AE91 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBCAAGBQJQeDr3AAoJEKeha0olJ0NqzdUIAKcX8RIXQdOTI/yBjnlt9Fgn OSh9V4If+WsOX/5dXXruizy4hdvfJWRYGeLstV4/aDtLFBWmh1UzA69JOj/5khN7 YGTnqEwCVzrRXvPp/DHXFkCFOVLg3IM6OznImDiUxhPJRY7fX0TOqvl58IugE4ep Os22+pjrxviEVN0YBIB8NY59fRmf43A50getXYhqWbZWE2EMsdfgykaYj+uhixWG HY036N8C10N1z/FOAoIcuyK3+vzP9yv+dsTCErjLuaouQQL0zkp6O2nM2og+Zg+k XSQ5YSjqSxs8aA3W5kRUfR6N5ef9qeekSWJU9sN8A+c/1+e2qtyE4xCUYntp3AM= =/TKE -----END PGP SIGNATURE----- --------------enigA64C4CCDE98200ACAD31AE91-- From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 12 13:18:41 2012 Received: (at 12626) by debbugs.gnu.org; 12 Oct 2012 17:18:41 +0000 Received: from localhost ([127.0.0.1]:41275 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMisj-0003A2-3w for submit@debbugs.gnu.org; Fri, 12 Oct 2012 13:18:41 -0400 Received: from ishtar.tlinx.org ([173.164.175.65]:50362) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TMisg-00039r-2v for 12626@debbugs.gnu.org; Fri, 12 Oct 2012 13:18:39 -0400 Received: from [192.168.3.12] (Athenae [192.168.3.12]) by Ishtar.tlinx.org (8.14.5/8.14.4/SuSE Linux 0.8) with ESMTP id q9CHHgft020467; Fri, 12 Oct 2012 10:17:44 -0700 Message-ID: <507850B6.2090508@tlinx.org> Date: Fri, 12 Oct 2012 10:17:42 -0700 From: Linda Walsh User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.24) Gecko/20100228 Lightning/0.9 Thunderbird/2.0.0.24 Mnenhy/0.7.6.666 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#12626: Bug?: dd limited to <2G read size (2G-8K) on 64 bit machine? References: <50778A79.5070106@tlinx.org> <5077A1B9.8010402@cs.ucla.edu> In-Reply-To: <5077A1B9.8010402@cs.ucla.edu> Content-Type: multipart/alternative; boundary="------------040003020501010902060402" X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: 12626 Cc: 12626@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.3 (-) This is a multi-part message in MIME format. --------------040003020501010902060402 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit I tried using gvim instead... ;-) I found: /* Warn about partial reads if bs=SIZE is given and iflag=fullblock is not, and if counting or skipping bytes or using direct I/O. This helps to avoid confusion with miscounts, and to avoid issues with direct I/O on GNU/Linux. */ warn_partial_read = (! (conversions_mask & C_TWOBUFS) && ! (input_flags & O_FULLBLOCK) && (skip_records || (0 < max_records && max_records < (uintmax_t) -1) || (input_flags | output_flags) & O_DIRECT)); ------------ I'm not doing conversions and didn't have fullblock set. I'm not skipping records input has o_direct set... but the troublesome line: || (0 < max_records && max_records < (uintmax_t) -1) I asked to copy 1,2 or 4 records uintmax -1 = 0xffff fffe --- I don't understand, if max_records is >0 and less than ~4G-1, set this flag? I'm assuming it's a flag to display the message or not, as I know it doesn't display the message most of the time... Is that right uintmax? or should that be an unsigned long int max? But I don't think that's the root cause of what I am seeing. But that statement doesn't look right.... It acts more like something (maybe not dd), is running with a 32-bit word size. ldd shows dd linking with lib64 targets: > ldd dd linux-vdso.so.1 (0x00007fff6d5ff000) librt.so.1 => /lib64/librt.so.1 (0x0000003001800000) libc.so.6 => /lib64/libc.so.6 (0x0000003000400000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003001000000) /lib64/ld-linux-x86-64.so.2 (0x0000003000000000) --- Does dd have a 32-bit limit on numb blocks? Paul Eggert wrote: > On 10/11/2012 08:11 PM, Linda Walsh wrote: > >> I find that if I try to use a read size of > (2G-8K), I get partial read errors. >> > > My guess is that it's your kernel, or maybe your > file system, and not dd per se. Try running 'strace'. > --------------040003020501010902060402 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by Ishtar.tlinx.org id q9CHHgft020467 I tried using gvim instead... ;-)

I found:
=C2=A0 /* Warn about partial reads if bs=3DSIZE is given and iflag=3Dfull= block
=C2=A0=C2=A0=C2=A0=C2=A0 is not, and if counting or skipping bytes or usi= ng direct I/O.
=C2=A0=C2=A0=C2=A0=C2=A0 This helps to avoid confusion with miscounts, an= d to avoid issues
=C2=A0=C2=A0=C2=A0=C2=A0 with direct I/O on GNU/Linux.=C2=A0 */
=C2=A0 warn_partial_read =3D
=C2=A0=C2=A0=C2=A0 (! (conversions_mask & C_TWOBUFS) && ! (in= put_flags & O_FULLBLOCK)
=C2=A0=C2=A0=C2=A0=C2=A0 && (skip_records
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 || (0 < max_records &= amp;& max_records < (uintmax_t) -1)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 || (input_flags | output= _flags) & O_DIRECT));


------------
I'm not doing conversions and didn't have fullblock set.
I'm not skipping records
input has o_direct set...

but the troublesome line:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 || (0 < max_records &= amp;& max_records < (uintmax_t) -1)
I asked to copy 1,2 or 4 records

uintmax -1 =3D=C2=A0 0xffff fffe --- I don't understand, if max_records i= s >0
and less than ~4G-1, set this flag?

I'm assuming it's a flag to display the message or not, as I know it doesn't display the message most of the time...

Is that right uintmax?=C2=A0=C2=A0 or should that be an unsigned long int= max?



But I don't think that's the root cause of what I am seeing.=C2=A0 But th= at statement doesn't look right....
It acts more like something (maybe not dd), is running with a 32-bit word size.

ldd shows dd linking with lib64 targets:

> ldd dd
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 linux-vdso.so.1 (0x00007fff6d5= ff000)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 librt.so.1 =3D> /lib64/libr= t.so.1 (0x0000003001800000)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 libc.so.6 =3D> /lib64/libc.= so.6 (0x0000003000400000)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 libpthread.so.0 =3D> /lib64= /libpthread.so.0 (0x0000003001000000)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /lib64/ld-linux-x86-64.so.2 (0= x0000003000000000)

---
Does dd have a 32-bit limit on numb blocks?




Paul Eggert wrote: 
On 10/11/2012 08:11 PM, Linda Walsh wrote:
  
I find that if I try to use a read size of > (2G-8K=
), I get partial read errors.
    

My guess is that it's your kernel, or maybe your
file system, and not dd per se.  Try running 'strace'.
--------------040003020501010902060402-- From unknown Fri Jun 20 07:14:56 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sat, 10 Nov 2012 12:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator