GNU bug report logs - #13530
head: memory exhausted when printing all from stdin but last P/E bytes

Previous Next

Full log

View this message in rfc822 format

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: lei.zhang <at> uwaterloo.ca, 13530 <at> debbugs.gnu.org, Pádraig Brady <P <at> draigBrady.com>, coreutils <at> gnu.org
Subject: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes
Date: Tue, 28 May 2013 03:38:55 +0200

Paul Eggert wrote:
> On 05/27/2013 06:04 PM, Jim Meyering wrote:
>> +lim=$(echo $SIZE_MAX | subtract_one_)
>> +lim=$(expr $lim - $max_BUFSIZ)
>
> Sorry, I don't see how this will work either.
> It's common for a GMP-less expr to handle values
> only up to SIZE_MAX / 2, and subtracting just 1
> won't work around that problem.

I was concentrating on the 32-bit case, where GMP-less expr
works fine on $SIZE_MAX.  It's on 64-bit that it's a problem, so...

> Maybe divide by 10 instead?  That's easy to do
> textually.  (I don't know what the test is about
> so I'm not sure what to suggest.)

I'd rather not divide by 10 all around.
That'd decrease it too much in the 32-bit case.
I took a different tack:


From 0d81799b18bef79b49d9042111f9dee3312796a7 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering <at> fb.com>
Date: Mon, 27 May 2013 17:01:14 -0700
Subject: [PATCH] tests: head-c: avoid spurious failure with a 32-bit SIZE_MAX

* tests/misc/head-c.sh: When eliding N bytes from a non-seekable
input, N must be slightly smaller than SIZE_MAX in order to handle
input longer than N bytes, since the current implementation buffers
N bytes in memory.  This command would fail on 32-bit systems,
where SIZE_MAX < 1E:
  head --bytes=-E < /dev/null
Instead of "E", use a value slightly smaller than SIZE_MAX.
---
 tests/misc/head-c.sh | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh
index 37a86ce..a81754e 100755
--- a/tests/misc/head-c.sh
+++ b/tests/misc/head-c.sh
@@ -19,6 +19,7 @@
 . "${srcdir=.}/tests/init.sh"; path_prepend_ ./src
 print_ver_ head
 require_ulimit_v_
+getlimits_

 # exercise the fix of 2001-08-18, based on test case from Ian Bruce
 echo abc > in || framework_failure_
@@ -28,9 +29,27 @@ case "$(cat out)" in
   *) fail=1 ;;
 esac

+# Use a limit of N = SIZE_MAX - max_BUFSIZ
+# The "- max_BUFSIZ" term is because head must be able to add BUFSIZ
+# to the selected value of N without exceeding SIZE_MAX.
+# Since we've seen BUFSIZ up to 128K, use 256K to be safe.
+max_BUFSIZ=$(expr 256 '*' 1024)
+
+# It's ok to use a 10-digit $SIZE_MAX, because expr uses wider intmax_t.
+# However, when $SIZE_MAX is longer, it's a 20-digit quantity that is
+# too large for GMP-disabled expr to handle, so use $SSIZE_MAX there.
+# We'd use $SSIZE_MAX in both cases, but want to keep the number as
+# large as possible for the 32-bit case, since there, it may barely
+# exceed the size of available RAM, while SSIZE_MAX probably will not.
+test $(printf $SIZE_MAX|wc -c) -lt 11 \
+  && big=$SIZE_MAX \
+  || big=$SSIZE_MAX
+
+lim=$(expr $big - $max_BUFSIZ) \
+
 # Only allocate memory as needed.
 # Coreutils <= 8.21 would allocate memory up front
 # based on the value passed to -c
-(ulimit -v 20000; head --bytes=-E < /dev/null) || fail=1
+(ulimit -v 20000; head --bytes=-$lim < /dev/null) || fail=1

 Exit $fail
--
1.8.3

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.