GNU bug report logs - #6417
[Bug 591969] Re: dd bs=3G || ubuntu-bug coreutils

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 6417 in the body.
You can then email your comments to 6417 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Bug 591969 <591969 <at> bugs.launchpad.net>
Cc: bug-coreutils <at> gnu.org
Subject: Re: [Bug 591969] Re: dd bs=3G || ubuntu-bug coreutils
Date: Sun, 13 Jun 2010 21:19:46 +0200

Dave Gilbert wrote:
> You're on 32bit, on 64bit this seems to work.  My guess is that there is
> a limit to the block size available on 32bit (if that limit was 2G or
> just under it wouldn't surprise me), on 64bit there is a similar limit
> somewhere :
>
> dg <at> major:~$ true | dd bs=3G count=1
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 2.1008e-05 s, 0.0 kB/s
> dg <at> major:~$ true | dd bs=3E count=1
> dd: memory exhausted
> dg <at> major:~$ true | dd bs=3Z count=1
> dd: invalid number `3Z'
>
> I can see an argument that there might want to put a better error message in, or state that the bs is dependent on memory
> size.

Thanks for the report.

dd is trying to tell you not to specify such a large buffer size.
A buffer size larger than a few megabytes is not normally useful.

That diagnostic is not helpful indeed, but it was easy to fix,
so I'll probably apply something like the following upstream.
With the new diagnostic, you can see that dd tries (and fails)
to allocate output buffers of the specified size:

    $ :|./dd bs=3EiB
    ./dd: failed to allocate an input buffer of size 3.0 EiB
    [Exit 1]
    $ :|./dd obs=3EiB
    ./dd: failed to allocate an output buffer of size 3.0 EiB
    [Exit 1]

Making dd give a better diagnostic for e.g., "bs=3G" on an i686
system might be more work than it's worth.


From 899f17359bf4990888b54c5ae90c41a74529d04d Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering <at> redhat.com>
Date: Sun, 13 Jun 2010 17:34:45 +0200
Subject: [PATCH] dd: print a better diagnostic for an invalid block size
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* src/dd.c (dd_copy): Give a better diagnostic than
"dd: memory exhausted" for an over-large bs= block size setting.
Same for ibs= and obs=.  Reported by Imre Péntek in
http://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/591969
---
 src/dd.c |   25 +++++++++++++++++++++++--
 1 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/src/dd.c b/src/dd.c
index 30d3d28..c9b8cbd 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -1557,6 +1557,16 @@ set_fd_flags (int fd, int add_flags, char const *name)
     }
 }

+static char *
+human_size (size_t n)
+{
+  static char hbuf[LONGEST_HUMAN_READABLE + 1];
+  int human_opts =
+    (human_autoscale | human_round_to_nearest | human_base_1024
+     | human_space_before_unit | human_SI | human_B);
+  return human_readable (n, hbuf, human_opts, 1, 1);
+}
+
 /* The main loop.  */

 static int
@@ -1593,7 +1603,13 @@ dd_copy (void)
      It is necessary when accessing raw (i.e. character special) disk
      devices on Unixware or other SVR4-derived system.  */

-  real_buf = xmalloc (input_blocksize + INPUT_BLOCK_SLOP);
+  size_t sz = input_blocksize + INPUT_BLOCK_SLOP;
+  real_buf = malloc (sz);
+  if (!real_buf)
+    error (EXIT_FAILURE, 0,
+           _("failed to allocate an input buffer of size %s"),
+           human_size (sz));
+
   ibuf = real_buf;
   ibuf += SWAB_ALIGN_OFFSET;	/* allow space for swab */

@@ -1602,7 +1618,12 @@ dd_copy (void)
   if (conversions_mask & C_TWOBUFS)
     {
       /* Page-align the output buffer, too.  */
-      real_obuf = xmalloc (output_blocksize + OUTPUT_BLOCK_SLOP);
+      sz = output_blocksize + OUTPUT_BLOCK_SLOP;
+      real_obuf = malloc (sz);
+      if (!real_obuf)
+        error (EXIT_FAILURE, 0,
+               _("failed to allocate an output buffer of size %s"),
+               human_size (sz));
       obuf = ptr_align (real_obuf, page_size);
     }
   else
--
1.7.1.511.g2f531

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> CS.UCLA.EDU>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#6417: [Bug 591969] Re: dd bs=3G || ubuntu-bug coreutils
Date: Mon, 14 Jun 2010 13:24:40 -0700

On 06/13/2010 12:19 PM, Jim Meyering wrote:

> +  size_t sz = input_blocksize + INPUT_BLOCK_SLOP;
> +  real_buf = malloc (sz);
> +  if (!real_buf)
> +    error (EXIT_FAILURE, 0,
> +           _("failed to allocate an input buffer of size %s"),
> +           human_size (sz));

Wouldn't it be better to output the exact buffer size,
rather than an approximation?  Usually it's better to
output more information, rather than less, to help
debug problems more precisely.

One way to do this would be to output the same string
that the user specified: if the user says "bs=3EiB" then
output "3EiB" in the diagnostic.  That would be exact,
and would be a bit friendlier than "3458764513820540928 bytes".
Or, we could output both, as in "3EiB (3458764513820540928 bytes)",
where the first is the string and the second is the integer that
we converted it to.

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> CS.UCLA.EDU>
Cc: bug-coreutils <at> gnu.org
Subject: Re: bug#6417: [Bug 591969] Re: dd bs=3G || ubuntu-bug coreutils
Date: Mon, 14 Jun 2010 22:35:59 +0200

Paul Eggert wrote:
> On 06/13/2010 12:19 PM, Jim Meyering wrote:
>
>> +  size_t sz = input_blocksize + INPUT_BLOCK_SLOP;
>> +  real_buf = malloc (sz);
>> +  if (!real_buf)
>> +    error (EXIT_FAILURE, 0,
>> +           _("failed to allocate an input buffer of size %s"),
>> +           human_size (sz));
>
> Wouldn't it be better to output the exact buffer size,
> rather than an approximation?  Usually it's better to

Hi Paul,

Thanks for the feedback.

I debated that, but figured that most of the time
people would realize what has gone wrong (number too large).

> output more information, rather than less, to help
> debug problems more precisely.
>
> One way to do this would be to output the same string
> that the user specified: if the user says "bs=3EiB" then

That would be best, but as you see above, it's not the same
number, since what we try to allocate is larger by INPUT_BLOCK_SLOP.

> output "3EiB" in the diagnostic.  That would be exact,
> and would be a bit friendlier than "3458764513820540928 bytes".
> Or, we could output both, as in "3EiB (3458764513820540928 bytes)",
> where the first is the string and the second is the integer that
> we converted it to.

That would be an improvement, indeed.
I've gone ahead and pushed the patch above.
You're welcome to patch further.

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> CS.UCLA.EDU>
To: Jim Meyering <jim <at> meyering.net>
Cc: bug-coreutils <at> gnu.org
Subject: Re: bug#6417: [Bug 591969] Re: dd bs=3G || ubuntu-bug coreutils
Date: Mon, 12 Jul 2010 17:22:17 -0700

On 06/14/10 13:35, Jim Meyering wrote:
> Paul Eggert wrote:
>> Or, we could output both, as in "3EiB (3458764513820540928 bytes)",
>> where the first is the string and the second is the integer that
>> we converted it to.
> 
> That would be an improvement, indeed.
> ...
> You're welcome to patch further.

Thanks, I tried it out, and discovered that outputting the exact
number isn't right, since the user might complain "I asked for 1 MB;
why is it complaining about 1005000 bytes"?  Since the slop is small,
it's better to simply output the user-requested size and omit the
overhead, which is plausibly small enough to be like malloc's internal
overhead anyway.  Also, from a technical-wording point of view it's
more accurate that way, since the message talks about the buffer size,
not the malloc request size.

Anyway, I installed this; of course further improvements are welcome.

From 335b59b9625758f92ba82cd6b7138f7423df60ef Mon Sep 17 00:00:00 2001
From: Paul R. Eggert <eggert <at> cs.ucla.edu>
Date: Mon, 12 Jul 2010 17:12:43 -0700
Subject: [PATCH] dd: also spell out size on memory exhaustion

* src/dd.c (dd_copy): Use requested blocksize (not adjusted) in
diagnostic, to forestall user complaints that the numbers don't
match exactly.  Report both exact and human-readable sizes, using
a message format that is consistent with both "BBBB bytes (N XB)
copied" in dd.c and "memory exhausted" in lib/xmalloc.c.
---
 src/dd.c |   14 ++++++--------
 1 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/src/dd.c b/src/dd.c
index c9b8cbd..ab8b84c 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -1603,12 +1603,11 @@ dd_copy (void)
      It is necessary when accessing raw (i.e. character special) disk
      devices on Unixware or other SVR4-derived system.  */
 
-  size_t sz = input_blocksize + INPUT_BLOCK_SLOP;
-  real_buf = malloc (sz);
+  real_buf = malloc (input_blocksize + INPUT_BLOCK_SLOP);
   if (!real_buf)
     error (EXIT_FAILURE, 0,
-           _("failed to allocate an input buffer of size %s"),
-           human_size (sz));
+           _("memory exhausted by input buffer of size %zu bytes (%s)"),
+           input_blocksize, human_size (input_blocksize));
 
   ibuf = real_buf;
   ibuf += SWAB_ALIGN_OFFSET;	/* allow space for swab */
@@ -1618,12 +1617,11 @@ dd_copy (void)
   if (conversions_mask & C_TWOBUFS)
     {
       /* Page-align the output buffer, too.  */
-      sz = output_blocksize + OUTPUT_BLOCK_SLOP;
-      real_obuf = malloc (sz);
+      real_obuf = malloc (output_blocksize + OUTPUT_BLOCK_SLOP);
       if (!real_obuf)
         error (EXIT_FAILURE, 0,
-               _("failed to allocate an output buffer of size %s"),
-               human_size (sz));
+               _("memory exhausted by output buffer of size %zu bytes (%s)"),
+               output_blocksize, human_size (output_blocksize));
       obuf = ptr_align (real_obuf, page_size);
     }
   else
-- 
1.7.1

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.