From unknown Sun Aug 17 01:59:47 2025 X-Loop: help-debbugs@gnu.org Subject: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9 Resent-From: Jesse Gordon Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Wed, 13 Apr 2011 02:46:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 8490 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: 8490@debbugs.gnu.org X-Debbugs-Original-To: bug-coreutils@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.13026627162668 (code B ref -1); Wed, 13 Apr 2011 02:46:02 +0000 Received: (at submit) by debbugs.gnu.org; 13 Apr 2011 02:45:16 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Q9q4x-0000gz-Og for submit@debbugs.gnu.org; Tue, 12 Apr 2011 22:45:16 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Q9q3x-0000f3-Ho for submit@debbugs.gnu.org; Tue, 12 Apr 2011 22:44:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Q9q3r-0007MV-4k for submit@debbugs.gnu.org; Tue, 12 Apr 2011 22:44:08 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.6 required=5.0 tests=BAYES_00, RECEIVED_FROM_WINDOWS_HOST, T_DKIM_INVALID, T_RP_MATCHES_RCVD autolearn=no version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:57541) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q9q3r-0007MQ-0C for submit@debbugs.gnu.org; Tue, 12 Apr 2011 22:44:07 -0400 Received: from eggs.gnu.org ([140.186.70.92]:49338) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q9pEn-0003wW-FP for bug-coreutils@gnu.org; Tue, 12 Apr 2011 21:51:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Q9kj5-0002li-JR for bug-coreutils@gnu.org; Tue, 12 Apr 2011 17:02:21 -0400 Received: from mail2.nikola.com ([64.146.180.253]:24492 helo=nikola.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q9kj5-0002lc-Bj for bug-coreutils@gnu.org; Tue, 12 Apr 2011 17:02:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=nikola.com; s=MDaemon; t=1302642135; x=1303246935; q=dns/txt; h=Received: Message-ID:Date:From:User-Agent:MIME-Version:To:Subject: Content-Type:Content-Transfer-Encoding; bh=oX23IHLdTPWe15H/a7OLk BILCeS2rJm/fAd0kyeaNnI=; b=sAcIY/1az6Z2c+/7MAkHUJOoX+T1Q7/RAv/OS Lll8GIHULKH0Lhd9Zm5lZG11tDqZSd6I+uhyvMXWIiXdYWxN3z7BLcrbM9gytoUg jIsJ3AiAeUxSQjwUtyzeGRyGMP5WBPLOfd4fS+yTqPR5aU5Q4VVyAB5YF+EZLC5n xk2l0w= X-MDAV-Processed: nikola.com, Tue, 12 Apr 2011 14:02:15 -0700 Received: from [10.0.0.8] by nikola.com (MDaemon PRO v11.0.3) with ESMTP id md50006682981.msg for ; Tue, 12 Apr 2011 14:02:14 -0700 X-Authenticated-Sender: jesseg@nikola.com X-MDRemoteIP: 10.0.0.8 X-Return-Path: jesseg@nikola.com X-Envelope-From: jesseg@nikola.com X-MDaemon-Deliver-To: bug-coreutils@gnu.org Message-ID: <4DA4BDD4.9010802@nikola.com> Date: Tue, 12 Apr 2011 14:02:12 -0700 From: Jesse Gordon User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Windows 2000 SP4, XP SP1+ X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -5.9 (-----) X-Mailman-Approved-At: Tue, 12 Apr 2011 22:45:14 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) I can't believe such an obvious bug would exist this long, but on the other hand the test is so simple I can't see where it's user error. dd, when reading from stdin or from a named pipe sometimes (but not always) reads a random number of records a bit less then what it should. I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and even mkfifo pip; yes > pipe & dd if=pipe -- and all sometimes failed. However, if=arealfile seems to always work perfectly. To replicate: yes|dd bs=1000 count=1000|wc -c cat /dev/zero |dd bs=1000 count=1000 |wc -c If it works perfectly the first time, just keep trying. For me, it randomly works and doesn't work. Some conditions are more likely to work, and others to fail. For me, using the "yes" method above almost always reads the incorrect number of bytes, while the /dev/zero method usually works correctly but occasionally reads the wrong number of bytes. The problem exists on the following coronations: Slackware 12.0.0 (2.6.21.5-smp) / coreutils 8.9 Slackware 12.0.0 (2.6.21.5-smp) / coreutils 6.9 Slackware 13.0.0 (2.6.29.6) / coreutils 7.9 Kubuntu lenny/sid (2.6.24-27-generic SMP) / coreutils 6.10 (I only mention the older versions of coreutils because it may be helpful. I'm only filing the bug report for 8.9!) Thanks, Jesse For example: root@stats:~# dd --version dd (coreutils) 8.9 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Paul Rubin, David MacKenzie, and Stuart Kemp. root@stats:~# yes|dd bs=1000 count=1000|wc -c 597+403 records in 597+403 records out 608232 bytes (608 kB) copied, 0.0263828 s, 23.1 MB/s 608232 root@stats:~# yes|dd bs=1000 count=1000|wc -c 885+115 records in 885+115 records out 887784 bytes (888 kB) copied, 0.054488 s, 16.3 MB/s 887784 root@stats:~# yes|dd bs=1000 count=1000|wc -c 696+304 records in 696+304 records out 705536 bytes (706 kB) copied, 0.0271789 s, 26.0 MB/s 705536 root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 1000000 1000+0 records in 1000+0 records out 1000000 bytes (1.0 MB) copied, 0.00879434 s, 114 MB/s root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 972+28 records in 972+28 records out 993040 bytes (993 kB) copied, 0.00582009 s, 171 MB/s 993040 root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 983+17 records in 983+17 records out 996040 bytes (996 kB) copied, 0.0102457 s, 97.2 MB/s 996040 root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 1000000 1000+0 records in 1000+0 records out 1000000 bytes (1.0 MB) copied, 0.0181759 s, 55.0 MB/s root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 1000+0 records in 1000+0 records out 1000000 bytes (1.0 MB) copied, 0.010386 s, 96.3 MB/s 1000000 root@stats:~# From unknown Sun Aug 17 01:59:47 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.427 (Entity 5.427) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Jesse Gordon Subject: bug#8490: closed (Re: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9) Message-ID: References: <4DA5AE2B.2070508@redhat.com> <4DA4BDD4.9010802@nikola.com> X-Gnu-PR-Message: they-closed 8490 X-Gnu-PR-Package: coreutils Reply-To: 8490@debbugs.gnu.org Date: Wed, 13 Apr 2011 14:08:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1302703682-30846-1" This is a multi-part message in MIME format... ------------=_1302703682-30846-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #8490: dd reads random number of records from pipes - named or otherwise - = coreutils 8.9 which was filed against the coreutils package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 8490@debbugs.gnu.org. --=20 8490: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8490 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1302703682-30846-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 8490-done) by debbugs.gnu.org; 13 Apr 2011 14:07:53 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QA0jY-00081C-K6 for submit@debbugs.gnu.org; Wed, 13 Apr 2011 10:07:52 -0400 Received: from mx1.redhat.com ([209.132.183.28]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QA0jW-00080z-Iq for 8490-done@debbugs.gnu.org; Wed, 13 Apr 2011 10:07:51 -0400 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3DE7iSK015762 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 13 Apr 2011 10:07:44 -0400 Received: from [10.3.113.17] ([10.3.113.17]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id p3DE7h7u017424; Wed, 13 Apr 2011 10:07:44 -0400 Message-ID: <4DA5AE2B.2070508@redhat.com> Date: Wed, 13 Apr 2011 08:07:39 -0600 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110318 Red Hat/3.1.9-3.el6_0 Mnenhy/0.8.3 Thunderbird/3.1.9 MIME-Version: 1.0 To: Jesse Gordon Subject: Re: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9 References: <4DA4BDD4.9010802@nikola.com> In-Reply-To: <4DA4BDD4.9010802@nikola.com> X-Enigmail-Version: 1.1.2 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig16DD5B04EBAE3656144FE267" X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11 X-Spam-Score: -10.2 (----------) X-Debbugs-Envelope-To: 8490-done Cc: 8490-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -10.2 (----------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig16DD5B04EBAE3656144FE267 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 04/12/2011 03:02 PM, Jesse Gordon wrote: > I can't believe such an obvious bug would exist this long, but on the > other hand the test is so simple I can't see where it's user error. Thanks for the report. And you are correct in surmising that it is user error and not a bug in dd. > dd, when reading from stdin or from a named pipe sometimes (but not > always) reads a random number of records a bit less then what it should= =2E Rather, dd reads as many bytes as possible, but unless that is less than PIPE_MAX, it is not guaranteed to be an atomic read. In turn, if you have asked dd to pad out partial reads into complete writes, then that explains your problem. Unfortunately, it is rather easy to do this without realizing it; the POSIX wording on how dd behaves is rather detailed. > I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and even=20 > mkfifo pip; yes > pipe & dd if=3Dpipe -- and all sometimes failed. > However, if=3Darealfile seems to always work perfectly. >=20 > To replicate: >=20 > yes|dd bs=3D1000 count=3D1000|wc -c There's your problem. bs=3D1000 is the key that tells dd to always write= 1000 byte output blocks, even if the input block hit a short read, and stop after 1000 reads. Instead, try: yes|dd ibs=3D1000 obs=3D1000 count=3D1000|wc -c which tells dd to explicitly read in input blocks of 1000 bytes, even if it requires multiple reads, prior to doing output blocks of 1000 bytes, and stop after 1000 writes. >=20 > The problem exists on the following coronations: It exists everywhere that dd complies with POSIX, even with non-GNU dd. Because POSIX requires the difference in behavior between bs=3Dnnn vs. ibs=3Dnnn obs=3Dnnn. --=20 Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org --------------enig16DD5B04EBAE3656144FE267 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJNpa4vAAoJEKeha0olJ0Nq254H+wSggK9/6KUihbSJOrBIyCtb yv2wD1xLwtMQBOWzm0GqaOAqJjgOiaJw4KQ7GTuwfeTgkhjiOyzDMMBem3oOfx5M U001k/1ItKTifuD3nf7RtaOBRDgPpA3LDddkw/cGa9l9X+oHRHLZXzXZ/FFTCsd9 tockru+uu3DV7oheGf67G7EkCRkp19CiDGAxHuz26hrANPTe2BBXs1yh0CEAVwb0 d231hPMH63mP/eZ/H4dfb6I+RZZSO0cYOvRdN2XbfGdGnmtdavbNPqpeXLvjRCg7 gw5kSsrKirRPsA/0gRlC4C1nnf9lWQmMjXhYs4io4ZAaaXX0XMDEvK1AmWY4RK4= =FH59 -----END PGP SIGNATURE----- --------------enig16DD5B04EBAE3656144FE267-- ------------=_1302703682-30846-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 13 Apr 2011 02:45:16 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Q9q4x-0000gz-Og for submit@debbugs.gnu.org; Tue, 12 Apr 2011 22:45:16 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Q9q3x-0000f3-Ho for submit@debbugs.gnu.org; Tue, 12 Apr 2011 22:44:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Q9q3r-0007MV-4k for submit@debbugs.gnu.org; Tue, 12 Apr 2011 22:44:08 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.6 required=5.0 tests=BAYES_00, RECEIVED_FROM_WINDOWS_HOST, T_DKIM_INVALID, T_RP_MATCHES_RCVD autolearn=no version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:57541) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q9q3r-0007MQ-0C for submit@debbugs.gnu.org; Tue, 12 Apr 2011 22:44:07 -0400 Received: from eggs.gnu.org ([140.186.70.92]:49338) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q9pEn-0003wW-FP for bug-coreutils@gnu.org; Tue, 12 Apr 2011 21:51:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Q9kj5-0002li-JR for bug-coreutils@gnu.org; Tue, 12 Apr 2011 17:02:21 -0400 Received: from mail2.nikola.com ([64.146.180.253]:24492 helo=nikola.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q9kj5-0002lc-Bj for bug-coreutils@gnu.org; Tue, 12 Apr 2011 17:02:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=nikola.com; s=MDaemon; t=1302642135; x=1303246935; q=dns/txt; h=Received: Message-ID:Date:From:User-Agent:MIME-Version:To:Subject: Content-Type:Content-Transfer-Encoding; bh=oX23IHLdTPWe15H/a7OLk BILCeS2rJm/fAd0kyeaNnI=; b=sAcIY/1az6Z2c+/7MAkHUJOoX+T1Q7/RAv/OS Lll8GIHULKH0Lhd9Zm5lZG11tDqZSd6I+uhyvMXWIiXdYWxN3z7BLcrbM9gytoUg jIsJ3AiAeUxSQjwUtyzeGRyGMP5WBPLOfd4fS+yTqPR5aU5Q4VVyAB5YF+EZLC5n xk2l0w= X-MDAV-Processed: nikola.com, Tue, 12 Apr 2011 14:02:15 -0700 Received: from [10.0.0.8] by nikola.com (MDaemon PRO v11.0.3) with ESMTP id md50006682981.msg for ; Tue, 12 Apr 2011 14:02:14 -0700 X-Authenticated-Sender: jesseg@nikola.com X-MDRemoteIP: 10.0.0.8 X-Return-Path: jesseg@nikola.com X-Envelope-From: jesseg@nikola.com X-MDaemon-Deliver-To: bug-coreutils@gnu.org Message-ID: <4DA4BDD4.9010802@nikola.com> Date: Tue, 12 Apr 2011 14:02:12 -0700 From: Jesse Gordon User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: bug-coreutils@gnu.org Subject: dd reads random number of records from pipes - named or otherwise - coreutils 8.9 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Windows 2000 SP4, XP SP1+ X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Tue, 12 Apr 2011 22:45:14 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) I can't believe such an obvious bug would exist this long, but on the other hand the test is so simple I can't see where it's user error. dd, when reading from stdin or from a named pipe sometimes (but not always) reads a random number of records a bit less then what it should. I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and even mkfifo pip; yes > pipe & dd if=pipe -- and all sometimes failed. However, if=arealfile seems to always work perfectly. To replicate: yes|dd bs=1000 count=1000|wc -c cat /dev/zero |dd bs=1000 count=1000 |wc -c If it works perfectly the first time, just keep trying. For me, it randomly works and doesn't work. Some conditions are more likely to work, and others to fail. For me, using the "yes" method above almost always reads the incorrect number of bytes, while the /dev/zero method usually works correctly but occasionally reads the wrong number of bytes. The problem exists on the following coronations: Slackware 12.0.0 (2.6.21.5-smp) / coreutils 8.9 Slackware 12.0.0 (2.6.21.5-smp) / coreutils 6.9 Slackware 13.0.0 (2.6.29.6) / coreutils 7.9 Kubuntu lenny/sid (2.6.24-27-generic SMP) / coreutils 6.10 (I only mention the older versions of coreutils because it may be helpful. I'm only filing the bug report for 8.9!) Thanks, Jesse For example: root@stats:~# dd --version dd (coreutils) 8.9 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Paul Rubin, David MacKenzie, and Stuart Kemp. root@stats:~# yes|dd bs=1000 count=1000|wc -c 597+403 records in 597+403 records out 608232 bytes (608 kB) copied, 0.0263828 s, 23.1 MB/s 608232 root@stats:~# yes|dd bs=1000 count=1000|wc -c 885+115 records in 885+115 records out 887784 bytes (888 kB) copied, 0.054488 s, 16.3 MB/s 887784 root@stats:~# yes|dd bs=1000 count=1000|wc -c 696+304 records in 696+304 records out 705536 bytes (706 kB) copied, 0.0271789 s, 26.0 MB/s 705536 root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 1000000 1000+0 records in 1000+0 records out 1000000 bytes (1.0 MB) copied, 0.00879434 s, 114 MB/s root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 972+28 records in 972+28 records out 993040 bytes (993 kB) copied, 0.00582009 s, 171 MB/s 993040 root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 983+17 records in 983+17 records out 996040 bytes (996 kB) copied, 0.0102457 s, 97.2 MB/s 996040 root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 1000000 1000+0 records in 1000+0 records out 1000000 bytes (1.0 MB) copied, 0.0181759 s, 55.0 MB/s root@stats:~# cat /dev/zero |dd bs=1000 count=1000 |wc -c 1000+0 records in 1000+0 records out 1000000 bytes (1.0 MB) copied, 0.010386 s, 96.3 MB/s 1000000 root@stats:~# ------------=_1302703682-30846-1-- From unknown Sun Aug 17 01:59:47 2025 X-Loop: help-debbugs@gnu.org Subject: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9 Resent-From: Bjartur Thorlacius Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Wed, 13 Apr 2011 14:40:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 8490 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Jesse Gordon Cc: 8490@debbugs.gnu.org Received: via spool by 8490-submit@debbugs.gnu.org id=B8490.13027055891279 (code B ref 8490); Wed, 13 Apr 2011 14:40:02 +0000 Received: (at 8490) by debbugs.gnu.org; 13 Apr 2011 14:39:49 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QA1ET-0000KZ-4f for submit@debbugs.gnu.org; Wed, 13 Apr 2011 10:39:49 -0400 Received: from mail-vw0-f44.google.com ([209.85.212.44]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QA1EQ-0000KM-Ps for 8490@debbugs.gnu.org; Wed, 13 Apr 2011 10:39:47 -0400 Received: by vws12 with SMTP id 12so506331vws.3 for <8490@debbugs.gnu.org>; Wed, 13 Apr 2011 07:39:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=JJyGHgxkvsd2TcEBnTFRmyB1X6WxrXLa19hZ/scPHF0=; b=PIJW55g5YjUpqhhsLGfVI/uFmnbxWwIjIoQVoE682kPzWCBe0jqRhnm0At7ga5+YZx uIf5HvL1rNiCWuc7yjlZXKG60rqsj6AXL8T9B8uGt0R9L+tARQ/pebqc7GoNOWVL+4bW N5tmgdCNAPU3Bzf/qwsWbhnoyILOUdCbpCUyA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=x5URsGNjew+MfsJ1GgdM0FxWvMraL3ydFpalWYQlKBt7NwZwPE9NNFi1GjlMAGjufc i/k7VCK42MDVxpt9ORdcSwpX5p24YpyiTvBw9I8BQ0dRyo9fk7oVWDvlnS4p6t8CmW6F YbXkHoUtELzeeDVVW4Ocs9P6cQle6/WTclOMg= MIME-Version: 1.0 Received: by 10.52.99.39 with SMTP id en7mr6162136vdb.170.1302705581285; Wed, 13 Apr 2011 07:39:41 -0700 (PDT) Received: by 10.52.167.228 with HTTP; Wed, 13 Apr 2011 07:39:41 -0700 (PDT) In-Reply-To: <4DA4BDD4.9010802@nikola.com> References: <4DA4BDD4.9010802@nikola.com> Date: Wed, 13 Apr 2011 14:39:41 +0000 Message-ID: From: Bjartur Thorlacius Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -3.6 (---) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.6 (---) Have you looked into fullblock? If you only specify bs and count (and not ibs or obs) dd may fill the buffer partially. It'll do try to do count copies, but each copy may contain less data than expected. This sort of makes sense on HDDs or tapes with variable block sizes (where a read would return a whole block, but the block would be smaller than user specified bs). In this case dd will preserve the original block size. I've never encountered such an odditie =E2=80=94 Or maybe I have, without noticing. I think about noone that hasn't been involved in the development, in one way or another, gets this wrong (I don't quite get it yet). I think this should be changed, unless the user provids a hypothetical partblock option. From unknown Sun Aug 17 01:59:47 2025 X-Loop: help-debbugs@gnu.org Subject: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9 Resent-From: John Reiser Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Wed, 13 Apr 2011 15:03:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 8490 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: bug-coreutils@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.13027069563265 (code B ref -1); Wed, 13 Apr 2011 15:03:01 +0000 Received: (at submit) by debbugs.gnu.org; 13 Apr 2011 15:02:36 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QA1aV-0000qc-I2 for submit@debbugs.gnu.org; Wed, 13 Apr 2011 11:02:35 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QA1aT-0000qP-9O for submit@debbugs.gnu.org; Wed, 13 Apr 2011 11:02:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QA1aI-0000hJ-QX for submit@debbugs.gnu.org; Wed, 13 Apr 2011 11:02:28 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:43335) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QA1aI-0000hF-P9 for submit@debbugs.gnu.org; Wed, 13 Apr 2011 11:02:22 -0400 Received: from eggs.gnu.org ([140.186.70.92]:58757) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QA1aG-0001PG-St for bug-coreutils@gnu.org; Wed, 13 Apr 2011 11:02:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QA1aB-0000fp-CT for bug-coreutils@gnu.org; Wed, 13 Apr 2011 11:02:20 -0400 Received: from bitwagon.com ([74.82.39.175]:32843) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1QA1aA-0000fa-TZ for bug-coreutils@gnu.org; Wed, 13 Apr 2011 11:02:15 -0400 Received: from f14-64.local ([67.171.188.169]) by bitwagon.com for ; Wed, 13 Apr 2011 08:02:11 -0700 Message-ID: <4DA5BAEE.50000@bitwagon.com> Date: Wed, 13 Apr 2011 08:02:06 -0700 From: John Reiser Organization: - User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14 Thunderbird/3.1.9 MIME-Version: 1.0 References: <4DA4BDD4.9010802@nikola.com> In-Reply-To: <4DA4BDD4.9010802@nikola.com> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -4.5 (----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.6 (----) > dd, when reading from stdin or from a named pipe sometimes (but not always) reads a random number of records a bit less then what it should. > yes|dd bs=1000 count=1000|wc -c > cat /dev/zero |dd bs=1000 count=1000 |wc -c dd does "read(fd,buf,bs)" for 'count' times, and writes whatever it gets each time. If the operating system does not deliver 'bs' bytes each time, then the total output will be less than bs*count bytes. Because neither /usr/bin/yes nor /dev/zero generates records with a blocksize of 1000 bytes (nor divisible by 1000 bytes) then you are at the mercy of multiprocessing delays and pipe buffering. read() on a pipe waits only for non-empty, not necessarily for the size requested. -- From unknown Sun Aug 17 01:59:47 2025 X-Loop: help-debbugs@gnu.org Subject: bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9 Resent-From: Eric Blake Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Wed, 13 Apr 2011 15:15:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 8490 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Bjartur Thorlacius Cc: 8490@debbugs.gnu.org, Jesse Gordon Received: via spool by 8490-submit@debbugs.gnu.org id=B8490.13027076534292 (code B ref 8490); Wed, 13 Apr 2011 15:15:02 +0000 Received: (at 8490) by debbugs.gnu.org; 13 Apr 2011 15:14:13 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QA1lk-00017B-QG for submit@debbugs.gnu.org; Wed, 13 Apr 2011 11:14:13 -0400 Received: from mx1.redhat.com ([209.132.183.28]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QA1lf-00016e-7O for 8490@debbugs.gnu.org; Wed, 13 Apr 2011 11:14:11 -0400 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3DFE1eH012938 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 13 Apr 2011 11:14:01 -0400 Received: from [10.3.113.17] ([10.3.113.17]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id p3DFE0Y6028577; Wed, 13 Apr 2011 11:14:00 -0400 Message-ID: <4DA5BDB7.3010000@redhat.com> Date: Wed, 13 Apr 2011 09:13:59 -0600 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110318 Red Hat/3.1.9-3.el6_0 Mnenhy/0.8.3 Thunderbird/3.1.9 MIME-Version: 1.0 References: <4DA4BDD4.9010802@nikola.com> In-Reply-To: X-Enigmail-Version: 1.1.2 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigA7D611D964A9F8F942D79FA8" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Spam-Score: -10.2 (----------) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -10.2 (----------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA7D611D964A9F8F942D79FA8 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 04/13/2011 08:39 AM, Bjartur Thorlacius wrote: > Have you looked into fullblock? If you only specify bs and count (and > not ibs or obs) dd may fill the buffer partially. It'll do try to do > count copies, but each copy may contain less data than expected. This > sort of makes sense on HDDs or tapes with variable block sizes (where > a read would return a whole block, but the block would be smaller than > user specified bs). In this case dd will preserve the original block > size. I've never encountered such an odditie =E2=80=94 Or maybe I have,= > without noticing. >=20 > I think about noone that hasn't been involved in the development, in > one way or another, gets this wrong (I don't quite get it yet). I > think this should be changed, unless the user provids a hypothetical > partblock option. It can't be changed without changing POSIX. --=20 Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org --------------enigA7D611D964A9F8F942D79FA8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJNpb23AAoJEKeha0olJ0Nqon0H/0ShlafBT1GTOxDNveDydghJ ZLqXhJk+Rgh/rpyNSgZQnl8uVDkR0FE93GpiyU/3mJZalfcCH3utnz1yk/twACRi B1y1fikA+gQqnnv0FSq+4vgYNh048/vxEYOnpDYUXcm/mkvfO3rm6OOgl4gdosQL YCqtf/cDLG1ihw1wktUlJXZh8gOYE/e4jsBImV/SYSDfJWN3qKEY2Pf3/Tgpxy+F 8+ElJ4epXuw+42FV9bJ8kksEDiVuj0J8FgLmPMOgIDAhkC2pAtIIO6kI5XP/NOq3 6Y+TY11gC1qITu3ar6O2kjlOYX7OUvVUZ0TN9r8KVS5L2KsCV1TeuIdQmK4J18I= =2qiN -----END PGP SIGNATURE----- --------------enigA7D611D964A9F8F942D79FA8--