From unknown Fri Jun 20 07:25:34 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#13135 <13135@debbugs.gnu.org> To: bug#13135 <13135@debbugs.gnu.org> Subject: Status: Loss of data while copying Reply-To: bug#13135 <13135@debbugs.gnu.org> Date: Fri, 20 Jun 2025 14:25:34 +0000 retitle 13135 Loss of data while copying reassign 13135 coreutils submitter 13135 xojoc@gmx.com severity 13135 normal tag 13135 notabug thanks From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 10 09:59:43 2012 Received: (at submit) by debbugs.gnu.org; 10 Dec 2012 14:59:44 +0000 Received: from localhost ([127.0.0.1]:35711 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti4pb-0004u6-59 for submit@debbugs.gnu.org; Mon, 10 Dec 2012 09:59:43 -0500 Received: from eggs.gnu.org ([208.118.235.92]:34681) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti4pZ-0004tz-5x for submit@debbugs.gnu.org; Mon, 10 Dec 2012 09:59:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ti4os-0001vY-3x for submit@debbugs.gnu.org; Mon, 10 Dec 2012 09:59:05 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, RP_MATCHES_RCVD autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:40870) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ti4os-0001vU-0f for submit@debbugs.gnu.org; Mon, 10 Dec 2012 09:58:58 -0500 Received: from eggs.gnu.org ([208.118.235.92]:45900) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ti4oW-0000re-SS for bug-coreutils@gnu.org; Mon, 10 Dec 2012 09:58:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ti4oP-0001ma-BH for bug-coreutils@gnu.org; Mon, 10 Dec 2012 09:58:36 -0500 Received: from mailout-eu.gmx.com ([213.165.64.42]:36690) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1Ti4oP-0001mF-0x for bug-coreutils@gnu.org; Mon, 10 Dec 2012 09:58:29 -0500 Received: (qmail invoked by alias); 10 Dec 2012 14:58:26 -0000 Received: from unknown (EHLO smag-R59-R60-R61) [151.65.149.48] by mail.gmx.com (mp-eu005) with SMTP; 10 Dec 2012 15:58:26 +0100 X-Authenticated: #130707387 X-Provags-ID: V01U2FsdGVkX1/S6DMPGR/ShJzG0uIRvaZevkcMc4XKWmk7vXWL6N rc4i4rM3I9aIAG Date: Mon, 10 Dec 2012 15:58:48 +0100 From: Cojocaru Alexandru To: bug-coreutils@gnu.org Subject: Loss of data while copying Message-Id: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> X-Mailer: Sylpheed 3.3.0 (GTK+ 2.24.13; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 208.118.235.17 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list Reply-To: xojoc@gmx.com List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) bash$ yes $(for i in $(seq 1 100000); do echo -n a; done) | dd of=big-lines ibs=100001 count=10000 9924+76 records in 1951183+1 records out *999005921* bytes (999 MB) copied, 21.6135 s, 46.2 MB/s bash$ yes $(for i in $(seq 1 100000); do echo -n a; done) | dd of=big-lines ibs=100001 count=10000 9887+113 records in 1950792+1 records out *998805894* bytes (999 MB) copied, 21.3409 s, 46.8 MB/s bash$ yes $(for i in $(seq 1 100000); do echo -n a; done) | dd of=big-lines ibs=100001 count=10000 9890+110 records in 1950792+1 records out *998805919* bytes (999 MB) copied, 21.5801 s, 46.3 MB/s bash$ yes $(for i in $(seq 1 100000); do echo -n a; done) | dd of=big-lines ibs=100001 count=10000 9884+116 records in 1950011+1 records out *998405915* bytes (998 MB) copied, 25.2695 s, 39.5 MB/s WTF?! Best regards, Cojocaru Alexandru From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 10 10:10:27 2012 Received: (at control) by debbugs.gnu.org; 10 Dec 2012 15:10:27 +0000 Received: from localhost ([127.0.0.1]:35719 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti4zy-00059c-UC for submit@debbugs.gnu.org; Mon, 10 Dec 2012 10:10:27 -0500 Received: from mx1.redhat.com ([209.132.183.28]:1507) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti4zu-00059K-Fd; Mon, 10 Dec 2012 10:10:24 -0500 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id qBAF9jp1018620 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 10 Dec 2012 10:09:45 -0500 Received: from [10.3.112.2] ([10.3.112.2]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id qBAF9iTq017048; Mon, 10 Dec 2012 10:09:45 -0500 Message-ID: <50C5FB38.6040209@redhat.com> Date: Mon, 10 Dec 2012 08:09:44 -0700 From: Eric Blake Organization: Red Hat, Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: xojoc@gmx.com Subject: Re: bug#13135: Loss of data while copying References: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> In-Reply-To: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigC5045D40783175953EFE7361" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: control Cc: 13135-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigC5045D40783175953EFE7361 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable tag 13135 notabug thanks On 12/10/2012 07:58 AM, Cojocaru Alexandru wrote: > bash$ yes $(for i in $(seq 1 100000); do echo -n a; done) | dd of=3Dbig= -lines ibs=3D100001 count=3D10000 > 9924+76 records in Thanks for the report. Based on this output, short reads occurred. dd transferred exactly 10000 reads as requested, but since some of those were short, it transferred less than 10000*100001 bytes. This is expected (and the behavior is described in that way by POSIX); the solution you are looking for is to _also_ use the iconv=3Dfullblock option, to force dd to re-read until it has a full input block rather than immediately transferring short input reads to output. As such, I'm closing this as not a bug. Do feel free to add further comments to this thread, though, if you have more questions about why dd does this. Oh, and by the way, 'echo -n' is not portable. You want to use printf(1) instead. --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --------------enigC5045D40783175953EFE7361 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQEcBAEBCAAGBQJQxfs4AAoJEKeha0olJ0NqQkcH/1BeRVnaQmsR0tt9hRHVY+kR 54yVgbfpStID+rpvzpUOJRAU31I3PheoBtxn3i+c5FFO2I8XnkaWA8c1B6TCDtUO y5iflPDPzmOZtR9kK7tiJmn8FHGodVBfHMUCqLkyjHe5DVRxaq50wq4v8lXq/yh6 pjDvziLcltg6mYcTVj8zENECAftJl/B+bblpJAug9WM6VgU1DwI3tmACYTn4XguQ M9VbChnc/OFFOVpxbJQH0yJxGfzDPMdAdrTY8v7XYAh5PHOqLPkPZjtsZP1Z+dd0 29WoTwTu0DyAO3gSNexReAL0lVUHNrszfl+CQie+t6UBSmuWa7rtViiZN2tAUpc= =/PTD -----END PGP SIGNATURE----- --------------enigC5045D40783175953EFE7361-- From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 10 10:22:26 2012 Received: (at 13135) by debbugs.gnu.org; 10 Dec 2012 15:22:26 +0000 Received: from localhost ([127.0.0.1]:35731 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti5Ba-0006Jl-4S for submit@debbugs.gnu.org; Mon, 10 Dec 2012 10:22:26 -0500 Received: from mx1.redhat.com ([209.132.183.28]:55197) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti5BI-0006JN-SR for 13135@debbugs.gnu.org; Mon, 10 Dec 2012 10:22:24 -0500 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id qBAFLV17030867 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 10 Dec 2012 10:21:31 -0500 Received: from [10.36.116.74] (ovpn-116-74.ams2.redhat.com [10.36.116.74]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id qBAFLTkU019246 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 10 Dec 2012 10:21:31 -0500 Message-ID: <50C5FDF9.2070903@draigBrady.com> Date: Mon, 10 Dec 2012 15:21:29 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: 13135@debbugs.gnu.org, xojoc@gmx.com Subject: Re: bug#13135: Loss of data while copying References: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> <50C5FB38.6040209@redhat.com> In-Reply-To: <50C5FB38.6040209@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id qBAFLV17030867 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: 13135 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) On 12/10/2012 03:09 PM, Eric Blake wrote: > tag 13135 notabug > thanks > > On 12/10/2012 07:58 AM, Cojocaru Alexandru wrote: >> bash$ yes $(for i in $(seq 1 100000); do echo -n a; done) | dd of=3Dbi= g-lines ibs=3D100001 count=3D10000 >> 9924+76 records in > > Thanks for the report. Based on this output, short reads occurred. dd > transferred exactly 10000 reads as requested, but since some of those > were short, it transferred less than 10000*100001 bytes. This is > expected (and the behavior is described in that way by POSIX); the > solution you are looking for is to _also_ use the iconv=3Dfullblock > option, to force dd to re-read until it has a full input block rather > than immediately transferring short input reads to output. > > As such, I'm closing this as not a bug. Do feel free to add further > comments to this thread, though, if you have more questions about why d= d > does this. > > Oh, and by the way, 'echo -n' is not portable. You want to use > printf(1) instead. > Yes, because a count was specified, dd will operate in its default awkward but POSIX specified mode of counting each read() call, even if it returned less than specified. This is especially noticeable with pipes: yes blah | dd of=3D/dev/null ibs=3D100001 count=3D10000 To avoid that you can use iflag=3Dfullblock (not iconv as Eric mentioned = above): yes blah | dd of=3D/dev/null ibs=3D100001 count=3D10000 iflag=3Dfullbl= ock cheers, P=C3=A1draig. From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 10 12:26:17 2012 Received: (at submit) by debbugs.gnu.org; 10 Dec 2012 17:26:17 +0000 Received: from localhost ([127.0.0.1]:35839 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti77Q-0000ii-OZ for submit@debbugs.gnu.org; Mon, 10 Dec 2012 12:26:17 -0500 Received: from eggs.gnu.org ([208.118.235.92]:37974) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti77P-0000iW-5Z for submit@debbugs.gnu.org; Mon, 10 Dec 2012 12:26:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ti76m-00020e-Q1 for submit@debbugs.gnu.org; Mon, 10 Dec 2012 12:25:38 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-101.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD, USER_IN_WHITELIST autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:38828) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ti76m-00020Y-LN for submit@debbugs.gnu.org; Mon, 10 Dec 2012 12:25:36 -0500 Received: from eggs.gnu.org ([208.118.235.92]:49274) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ti76d-0001F8-4l for bug-coreutils@gnu.org; Mon, 10 Dec 2012 12:25:36 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ti76X-0001xT-62 for bug-coreutils@gnu.org; Mon, 10 Dec 2012 12:25:27 -0500 Received: from bitwagon.com ([74.82.39.175]:50765) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1Ti76W-0001xF-Ud for bug-coreutils@gnu.org; Mon, 10 Dec 2012 12:25:21 -0500 Received: from f17-64.local ([24.21.156.164]) by bitwagon.com for ; Mon, 10 Dec 2012 09:05:11 -0800 Message-ID: <50C61681.20301@bitwagon.com> Date: Mon, 10 Dec 2012 09:06:09 -0800 From: John Reiser Organization: - User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: bug-coreutils@gnu.org Subject: Re: bug#13135: Loss of data while copying References: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> <50C5FB38.6040209@redhat.com> <50C5FDF9.2070903@draigBrady.com> In-Reply-To: <50C5FDF9.2070903@draigBrady.com> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 208.118.235.17 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) On 12/10/2012 07:21 AM, Pádraig Brady wrote: >> On 12/10/2012 07:58 AM, Cojocaru Alexandru wrote: >>> bash$ yes $(for i in $(seq 1 100000); do echo -n a; done) | dd of=big-lines ibs=100001 count=10000 >>> 9924+76 records in The original poster should know better: the best bug reports include not only what actually happened, but also the version number of the components, what the original poster expected to happen, and an explicit identification of the differences. For instance: (I am running dd from coreutils-8.15.) The 'yes' will generate an infinite stream of lines, each containing one hundred thousand 'a' characters followed by a terminating newline. I expect that "ibs=100001" causes dd to read each entire line (including the newline) in one operation, and that "count=10000" causes dd to stop after copying ten thousand lines. Thus the dd summary should say: 10000 records in 10000 records out 1000010000 bytes (1000 MB) copied > Yes, because a count was specified, > dd will operate in its default awkward but POSIX specified mode > of counting each read() call, even if it returned less than specified. > This is especially noticeable with pipes: So this bug report is really about the execrable documentation for 'dd'. Despite similar complaints appearing yearly [or so], the text of "info dd" does not contain the string "pipe". SHAME ON COREUTILS. Explaining the most common error, and how to avoid it, certainly does belong in the documentation. The purpose of documentation is to *FACILITATE* the correct use of the tool, and not merely to erect the minimal legal defense of the code. -- From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 10 13:03:56 2012 Received: (at 13135) by debbugs.gnu.org; 10 Dec 2012 18:03:56 +0000 Received: from localhost ([127.0.0.1]:35856 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti7hr-0001bf-1a for submit@debbugs.gnu.org; Mon, 10 Dec 2012 13:03:56 -0500 Received: from mx1.redhat.com ([209.132.183.28]:21060) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ti7hm-0001bV-UR for 13135@debbugs.gnu.org; Mon, 10 Dec 2012 13:03:54 -0500 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id qBAI3Bao026201 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 10 Dec 2012 13:03:12 -0500 Received: from [10.3.112.2] ([10.3.112.2]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id qBAI3APs008255; Mon, 10 Dec 2012 13:03:11 -0500 Message-ID: <50C623DE.4070502@redhat.com> Date: Mon, 10 Dec 2012 11:03:10 -0700 From: Eric Blake Organization: Red Hat, Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: John Reiser Subject: Re: bug#13135: Loss of data while copying References: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> <50C5FB38.6040209@redhat.com> <50C5FDF9.2070903@draigBrady.com> <50C61681.20301@bitwagon.com> In-Reply-To: <50C61681.20301@bitwagon.com> X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig62F3672CB8A042E56B60710A" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: 13135 Cc: 13135@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig62F3672CB8A042E56B60710A Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 12/10/2012 10:06 AM, John Reiser wrote: >> Yes, because a count was specified, >> dd will operate in its default awkward but POSIX specified mode >> of counting each read() call, even if it returned less than specified.= >> This is especially noticeable with pipes: >=20 > So this bug report is really about the execrable documentation for 'dd'= =2E > Despite similar complaints appearing yearly [or so], > the text of "info dd" does not contain the string "pipe". SHAME ON COR= EUTILS. > Explaining the most common error, and how to avoid it, certainly does > belong in the documentation. The purpose of documentation is to *FACIL= ITATE* > the correct use of the tool, and not merely to erect the minimal legal = defense > of the code. Rather than complaining, how about you submit a patch to improve the documentation? --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --------------enig62F3672CB8A042E56B60710A Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQEcBAEBCAAGBQJQxiPeAAoJEKeha0olJ0NqscMH/2c727fFnjBWa5ce3cwnMMv6 JKAGhFK0z802scGDIk5C9vCu8LGHI3DL7TfEuatQxBIKIOalaKiTWR4ogwAPMiyd DOCAuu6rD9dvD+GPwkm7TpE5tQVJ/zY/Jgsic02OH2/LNkxyEZ6bqatjYCli18Ag 3gq/yGc9Flzt0dpPEHird0ybxxvo7NYqaqFFvX6+9AcGF8mO66DgjtSH6KhaD73I EaV8rd/UkDKTsPDtN839j6dkBIVJb+9jZxfBkEYCtYDrHJr4D+Lnw4Rsuiqip+hW 6r/SRK6HeM8ARbLmCeYeuBwdeJjNrMr7f/TRjMOCfW8rt8rj3bGXVgdj33PwzVY= =8xEk -----END PGP SIGNATURE----- --------------enig62F3672CB8A042E56B60710A-- From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 10 15:43:10 2012 Received: (at submit) by debbugs.gnu.org; 10 Dec 2012 20:43:10 +0000 Received: from localhost ([127.0.0.1]:35928 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TiABy-0005Gn-3F for submit@debbugs.gnu.org; Mon, 10 Dec 2012 15:43:10 -0500 Received: from eggs.gnu.org ([208.118.235.92]:47376) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TiABw-0005Gg-8Z for submit@debbugs.gnu.org; Mon, 10 Dec 2012 15:43:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TiABJ-0001bF-AJ for submit@debbugs.gnu.org; Mon, 10 Dec 2012 15:42:31 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-101.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD, USER_IN_WHITELIST autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:50050) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TiABJ-0001bA-74 for submit@debbugs.gnu.org; Mon, 10 Dec 2012 15:42:29 -0500 Received: from eggs.gnu.org ([208.118.235.92]:58732) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TiABI-0003Ll-6l for bug-coreutils@gnu.org; Mon, 10 Dec 2012 15:42:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TiABH-0001ac-2R for bug-coreutils@gnu.org; Mon, 10 Dec 2012 15:42:28 -0500 Received: from bitwagon.com ([2001:470:0:e6::4a52:27af]:51323) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1TiABG-0001aT-QC for bug-coreutils@gnu.org; Mon, 10 Dec 2012 15:42:26 -0500 Received: from f17-64.local ([24.21.156.164]) by bitwagon.com for ; Mon, 10 Dec 2012 12:42:24 -0800 Message-ID: <50C64971.9030900@bitwagon.com> Date: Mon, 10 Dec 2012 12:43:29 -0800 From: John Reiser Organization: - User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: bug-coreutils@gnu.org Subject: Re: bug#13135: Loss of data while copying References: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> <50C5FB38.6040209@redhat.com> <50C5FDF9.2070903@draigBrady.com> <50C61681.20301@bitwagon.com> <50C623DE.4070502@redhat.com> In-Reply-To: <50C623DE.4070502@redhat.com> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 208.118.235.17 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: submit Cc: 13135@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) On 12/10/2012 10:03 AM, Eric Blake wrote: > On 12/10/2012 10:06 AM, John Reiser wrote: >>> Yes, because a count was specified, >>> dd will operate in its default awkward but POSIX specified mode >>> of counting each read() call, even if it returned less than specified. >>> This is especially noticeable with pipes: >> >> So this bug report is really about the execrable documentation for 'dd'. >> Despite similar complaints appearing yearly [or so], >> the text of "info dd" does not contain the string "pipe". SHAME ON COREUTILS. >> Explaining the most common error, and how to avoid it, certainly does >> belong in the documentation. The purpose of documentation is to *FACILITATE* >> the correct use of the tool, and not merely to erect the minimal legal defense >> of the code. > > Rather than complaining, how about you submit a patch to improve the > documentation? > diff --git a/doc/coreutils.texi b/doc/coreutils.texi index 21400ad..c2282eb 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -8055,6 +8055,7 @@ OS/360 JCL. @item if=@var{file} @opindex if Read from @var{file} instead of standard input. +(If the input is a pipe then see @samp{fullblock} below.) @item of=@var{file} @opindex of @@ -8397,6 +8398,9 @@ may return early if a full block is not available. When that happens, continue calling @code{read} to fill the remainder of the block. This flag can be used only with @code{iflag}. +If the input is a pipe and argument @samp{count=} also is specified, +then probably @samp{iflag=fullblock} should be used +in order to prevent surprises caused by short reads. @item count_bytes @opindex count_bytes -- From debbugs-submit-bounces@debbugs.gnu.org Fri Dec 14 22:16:42 2012 Received: (at 13135) by debbugs.gnu.org; 15 Dec 2012 03:16:42 +0000 Received: from localhost ([127.0.0.1]:43162 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TjiEz-0005oF-VX for submit@debbugs.gnu.org; Fri, 14 Dec 2012 22:16:42 -0500 Received: from mx1.redhat.com ([209.132.183.28]:18860) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TjiEt-0005o2-Uf for 13135@debbugs.gnu.org; Fri, 14 Dec 2012 22:16:37 -0500 Received: from int-mx12.intmail.prod.int.phx2.redhat.com (int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.25]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id qBF3FW7S001859 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 14 Dec 2012 22:15:32 -0500 Received: from [10.36.116.69] (ovpn-116-69.ams2.redhat.com [10.36.116.69]) by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id qBF3FTwp023130 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 14 Dec 2012 22:15:31 -0500 Message-ID: <50CBEB51.7090902@draigBrady.com> Date: Sat, 15 Dec 2012 03:15:29 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: John Reiser Subject: Re: bug#13135: Loss of data while copying References: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> <50C5FB38.6040209@redhat.com> <50C5FDF9.2070903@draigBrady.com> <50C61681.20301@bitwagon.com> <50C623DE.4070502@redhat.com> <50C64971.9030900@bitwagon.com> In-Reply-To: <50C64971.9030900@bitwagon.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.25 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: 13135 Cc: 13135@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.9 (------) On 12/10/2012 08:43 PM, John Reiser wrote: > On 12/10/2012 10:03 AM, Eric Blake wrote: >> On 12/10/2012 10:06 AM, John Reiser wrote: >>>> Yes, because a count was specified, >>>> dd will operate in its default awkward but POSIX specified mode >>>> of counting each read() call, even if it returned less than specified. >>>> This is especially noticeable with pipes: >>> >>> So this bug report is really about the execrable documentation for 'dd'. >>> Despite similar complaints appearing yearly [or so], >>> the text of "info dd" does not contain the string "pipe". SHAME ON COREUTILS. >>> Explaining the most common error, and how to avoid it, certainly does >>> belong in the documentation. The purpose of documentation is to *FACILITATE* >>> the correct use of the tool, and not merely to erect the minimal legal defense >>> of the code. We've tried really hard to make this issue obvious. Even going to the effort of auto prompting the user to use iflag=fullblock. The full discussion of the awkward auto suggestion logic can be seen in http://bugs.gnu.org/7362 In more "normal" cases users will get the warning: $ yes blah | src/dd of=/dev/null bs=100001 count=10000 dd: warning: partial read (53248 bytes); suggest iflag=fullblock We didn't prompt in this case because it's a bit of an edge case in that ibs is specified rather than bs. So since there is write aggregation in that case and to support use cases like the following, we don't warn here: $ (echo part1; sleep 1; echo part2; sleep 1; echo discard) | dd count=2 ibs=4096 obs=1 2>/dev/null >> Rather than complaining, how about you submit a patch to improve the >> documentation? >> > > diff --git a/doc/coreutils.texi b/doc/coreutils.texi > index 21400ad..c2282eb 100644 > --- a/doc/coreutils.texi > +++ b/doc/coreutils.texi > @@ -8055,6 +8055,7 @@ OS/360 JCL. > @item if=@var{file} > @opindex if > Read from @var{file} instead of standard input. > +(If the input is a pipe then see @samp{fullblock} below.) I think I'll move the warning to count= as it's mostly an issue when that is specified. > > @item of=@var{file} > @opindex of > @@ -8397,6 +8398,9 @@ may return early if a full block is not available. > When that happens, continue calling @code{read} to fill the remainder > of the block. > This flag can be used only with @code{iflag}. > +If the input is a pipe and argument @samp{count=} also is specified, > +then probably @samp{iflag=fullblock} should be used > +in order to prevent surprises caused by short reads. How about this instead? diff --git a/doc/coreutils.texi b/doc/coreutils.texi index 5f8fad7..b916a86 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -8117,6 +8117,11 @@ Copy @var{n} @samp{ibs}-byte blocks from the input file, of everything until the end of the file. if @samp{iflag=count_bytes} is specified, @var{n} is interpreted as a byte count rather than a block count. +Note if the input may return short reads as could be the case +when reading from a pipe for example, @samp{iflag=fullblock} +will ensure that @samp{count=} corresponds to complete input blocks +rather than the traditional POSIX specified behavior of counting +input read operations. @item status=@var{which} @opindex status @@ -8397,6 +8402,10 @@ may return early if a full block is not available. When that happens, continue calling @code{read} to fill the remainder of the block. This flag can be used only with @code{iflag}. +This flag is useful with pipes for example +as they may return short reads. I that case, +this flag is needed to ensure that a @samp{count=} argument is +interpreted as a block count rather than a count of read operations. @item count_bytes @opindex count_bytes From debbugs-submit-bounces@debbugs.gnu.org Fri Dec 14 23:17:29 2012 Received: (at 13135) by debbugs.gnu.org; 15 Dec 2012 04:17:29 +0000 Received: from localhost ([127.0.0.1]:43174 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TjjBp-0007Ct-Im for submit@debbugs.gnu.org; Fri, 14 Dec 2012 23:17:29 -0500 Received: from mx.meyering.net ([88.168.87.75]:52930) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TjjBY-0007CV-R6 for 13135@debbugs.gnu.org; Fri, 14 Dec 2012 23:17:28 -0500 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id A2BDD600C4; Sat, 15 Dec 2012 05:16:09 +0100 (CET) From: Jim Meyering To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#13135: Loss of data while copying In-Reply-To: <50CBEB51.7090902@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?= =?iso-8859-1?Q?g?= Brady"'s message of "Sat, 15 Dec 2012 03:15:29 +0000") References: <20121210155848.a8c564f8517d9b25469b294e@gmx.com> <50C5FB38.6040209@redhat.com> <50C5FDF9.2070903@draigBrady.com> <50C61681.20301@bitwagon.com> <50C623DE.4070502@redhat.com> <50C64971.9030900@bitwagon.com> <50CBEB51.7090902@draigBrady.com> Date: Sat, 15 Dec 2012 05:16:09 +0100 Message-ID: <87r4ms2bna.fsf@rho.meyering.net> Lines: 33 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.5 (-) X-Debbugs-Envelope-To: 13135 Cc: John Reiser , 13135@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.5 (-) P=E1draig Brady wrote: ... > I think I'll move the warning to count=3D > as it's mostly an issue when that is specified. Good idea. >> @item of=3D@var{file} >> @opindex of >> @@ -8397,6 +8398,9 @@ may return early if a full block is not available. >> When that happens, continue calling @code{read} to fill the remainder >> of the block. >> This flag can be used only with @code{iflag}. >> +If the input is a pipe and argument @samp{count=3D} also is specified, >> +then probably @samp{iflag=3Dfullblock} should be used >> +in order to prevent surprises caused by short reads. > > How about this instead? Looks good. Thanks. > diff --git a/doc/coreutils.texi b/doc/coreutils.texi ... > @@ -8397,6 +8402,10 @@ may return early if a full block is not available. > When that happens, continue calling @code{read} to fill the remainder > of the block. > This flag can be used only with @code{iflag}. > +This flag is useful with pipes for example > +as they may return short reads. I that case, s/I/In/ ... From unknown Fri Jun 20 07:25:34 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sat, 12 Jan 2013 12:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator