From unknown Sat Aug 09 13:13:54 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#13530 <13530@debbugs.gnu.org> To: bug#13530 <13530@debbugs.gnu.org> Subject: Status: head: memory exhausted when printing all from stdin but last P/E bytes Reply-To: bug#13530 <13530@debbugs.gnu.org> Date: Sat, 09 Aug 2025 20:13:54 +0000 retitle 13530 head: memory exhausted when printing all from stdin but last = P/E bytes reassign 13530 coreutils submitter 13530 Lei Zhang severity 13530 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 22 21:42:27 2013 Received: (at submit) by debbugs.gnu.org; 23 Jan 2013 02:42:27 +0000 Received: from localhost ([127.0.0.1]:45369 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TxqID-0005ei-Uc for submit@debbugs.gnu.org; Tue, 22 Jan 2013 21:42:26 -0500 Received: from eggs.gnu.org ([208.118.235.92]:39518) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TxqAr-0005MB-GO for submit@debbugs.gnu.org; Tue, 22 Jan 2013 21:34:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Txq9b-0000Os-BW for submit@debbugs.gnu.org; Tue, 22 Jan 2013 21:33:36 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, HTML_MESSAGE,RCVD_IN_DNSWL_LOW,T_DKIM_INVALID autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:50195) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Txq9b-0000Of-8w for submit@debbugs.gnu.org; Tue, 22 Jan 2013 21:33:31 -0500 Received: from eggs.gnu.org ([208.118.235.92]:50845) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Txq9V-0004YM-8y for bug-coreutils@gnu.org; Tue, 22 Jan 2013 21:33:31 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Txq9Q-0000GU-H6 for bug-coreutils@gnu.org; Tue, 22 Jan 2013 21:33:25 -0500 Received: from mail-da0-f47.google.com ([209.85.210.47]:56646) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Txq9Q-0000G6-4o for bug-coreutils@gnu.org; Tue, 22 Jan 2013 21:33:20 -0500 Received: by mail-da0-f47.google.com with SMTP id s35so3530734dak.20 for ; Tue, 22 Jan 2013 18:33:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:sender:from:date:x-google-sender-auth :message-id:subject:to:content-type; bh=3HG+y2WFob3n29gxYMrKwpYybBuxlBTQeFM8hR35jBY=; b=YDvNeejb+6sWaXebIP/eOHtWy8xz5YD4weUZH9zRU1pJH/iylFwtnxOGTEwkw9Ablz OUE6KbZweg6ua4oImGf9Q3UPEuHlBdxc6F/C7kCXXT8kOr0Wp+B9bf0v+O0aTcaomPSj TdvWdPX2wbvA2fkUmk8l7euhYAZTEcTKNa7jzu532X1/c0kBEbQF8KZCSKZ8kwZ5FHlV DeIMsCezG/o7MB6lnNJPvbR3SzU+qaZZxb66r1XM/YWtI6Uk7arlEeouCt47CMBYe+Av t7RQWNL6P8kYnODu+LMWtFwAOIAP6Dw2pUBoCMEsStPx6NLB2fIRJEOpLJ6/mu090mx+ Cd9g== X-Received: by 10.68.135.40 with SMTP id pp8mr64079137pbb.63.1358908397791; Tue, 22 Jan 2013 18:33:17 -0800 (PST) MIME-Version: 1.0 Received: by 10.66.79.161 with HTTP; Tue, 22 Jan 2013 18:32:57 -0800 (PST) From: Lei Zhang Date: Tue, 22 Jan 2013 21:32:57 -0500 X-Google-Sender-Auth: GhC3EPcYBbaQNchyQnw3M46M-hw Message-ID: Subject: head: memory exhausted when printing all from stdin but last P/E bytes To: bug-coreutils@gnu.org Content-Type: multipart/alternative; boundary=047d7b10d17b5528f504d3eb8420 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 208.118.235.17 X-Spam-Score: -3.4 (---) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Tue, 22 Jan 2013 21:42:24 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.1 (------) --047d7b10d17b5528f504d3eb8420 Content-Type: text/plain; charset=UTF-8 Hi All, We found a bug in the `head' program of coreutils 8.20: Invoking `head -c -P' or `head -c -E' will cause memory exhaustion. However, smaller units (e.g., b, K, M) work fine; bigger units (e.g., Z, Y) fail properly (by outputing "number of bytes is so large that it is not representable"). And `-n' also works fine. A bit dig reveals that the bug is introduced at line 322 of head.c (coreutils 8.20): 204: elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0) 205: { 322: b = xcalloc (n_bufs, sizeof *b); /** this statement fails **/ 398: } We also examined n_elide and n_bufs before that statement. Actually, they are very large: n_elide: 1125899906842624 n_bufs: 137438953474 According to the following comment in the source file: > CAUTION: do not fail (out of memory) when asked to elide > a ridiculous amount, but when given only a small input. */ We think this is a bug and bring this issue to your attention. Thanks! Environments: $ uname -a Linux anti-think 3.7.3-1-ARCH #1 SMP PREEMPT Thu Jan 17 18:52:30 CET 2013 x86_64 GNU/Linux $ pacman -Qi coreutils Name : coreutils Version : 8.20-1 URL : http://www.gnu.org/software/coreutils Licenses : GPL3 Groups : base Provides : None Depends On : glibc pam acl gmp libcap Optional Deps : None Required By : ca-certificates dbus filesystem linux mkinitcpio perl sysvinit-tools util-linux Conflicts With : None Replaces : None Installed Size : 13820.00 KiB Packager : Allan McRae Architecture : x86_64 Build Date : Wed 24 Oct 2012 03:57:11 AM EDT Install Date : Sun 28 Oct 2012 01:51:06 PM EDT Install Reason : Explicitly installed Install Script : Yes Description : The basic file, shell and text manipulation utilities of the GNU operating system CPU: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz CPU memory: 4GB Best regards, -- Lei Zhang Department of Electrical and Computer Engineering University of Waterloo 200 University Avenue West Waterloo, Ontario, Canada N2L 3G1 --047d7b10d17b5528f504d3eb8420 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi All,

We found a bug in the `head' program of coreutils 8.20:<= br>
Invoking `head -c -P' or `head -c -E' will cause memory exha= ustion.

However, smaller units (e.g., b, K, M) work fine; bigger uni= ts (e.g., Z, Y) fail properly
(by outputing "number of bytes is so large that it is not representabl= e"). And `-n' also
works fine.

A bit dig reveals that th= e bug is introduced at line 322 of head.c (coreutils 8.20):

204: eli= de_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0)
205: {
322: =C2=A0 =C2=A0 b =3D xcalloc (n_bufs, sizeof *b); /** this st= atement fails **/
398: }

We also examined n_elide and n_bufs befo= re that statement. Actually, they are very
large:

n_elide: 112589= 9906842624
n_bufs: 137438953474

According to the following comment in the sourc= e file:

> CAUTION: do not fail (out of memory) when asked to elid= e
> a ridiculous amount, but when given only a small input. =C2=A0*/<= br>
We think this is a bug and bring this issue to your attention. Thanks!<= br>

Environments:

$ uname -a
Linux anti-think 3.7.3-1-ARCH= #1 SMP PREEMPT Thu Jan 17 18:52:30 CET 2013 x86_64 GNU/Linux

$ pacm= an -Qi coreutils
Name =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : coreutils
Version =C2=A0 =C2= =A0 =C2=A0 =C2=A0: 8.20-1
URL =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:= http:/= /www.gnu.org/software/coreutils
Licenses =C2=A0 =C2=A0 =C2=A0 : GPL3=
Groups =C2=A0 =C2=A0 =C2=A0 =C2=A0 : base
Provides =C2=A0 =C2=A0 =C2=A0 : None
Depends On =C2=A0 =C2=A0 : glibc =C2=A0pam =C2=A0acl =C2=A0gmp =C2=A0libcap=
Optional Deps =C2=A0: None
Required By =C2=A0 =C2=A0: ca-certificate= s =C2=A0dbus =C2=A0filesystem =C2=A0linux =C2=A0mkinitcpio =C2=A0perl =C2= =A0sysvinit-tools =C2=A0util-linux
Conflicts With : None
Replaces =C2= =A0 =C2=A0 =C2=A0 : None
Installed Size : 13820.00 KiB
Packager =C2=A0 =C2=A0 =C2=A0 : Allan McRa= e <allan@archli= nux.org>
Architecture =C2=A0 : x86_64
Build Date =C2=A0 =C2=A0= : Wed 24 Oct 2012 03:57:11 AM EDT
Install Date =C2=A0 : Sun 28 Oct 2012 01:51:06 PM EDT
Install Reason : Explicitly installed
Install Script : Yes
Descriptio= n =C2=A0 =C2=A0: The basic file, shell and text manipulation utilities of t= he GNU operating system

CPU: Intel(R) Core(TM) i5-3320M CPU @ 2.60GH= z CPU
memory: 4GB

Best regards,

--
Lei Zhang
<= span>Department of Electrical and Computer Engineering
Univ= ersity of Waterloo
200 University Avenue West
Waterloo, Ontario, Canada N2L 3G1
--047d7b10d17b5528f504d3eb8420-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 23 07:36:01 2013 Received: (at 13530) by debbugs.gnu.org; 23 Jan 2013 12:36:01 +0000 Received: from localhost ([127.0.0.1]:45818 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TxzYe-0005vx-AD for submit@debbugs.gnu.org; Wed, 23 Jan 2013 07:36:00 -0500 Received: from mail1.vodafone.ie ([213.233.128.43]:32440) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TxzYa-0005va-GT for 13530@debbugs.gnu.org; Wed, 23 Jan 2013 07:35:58 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApMBAHXY/1BtTQK8/2dsb2JhbAANN4ZFtBCDaoMRAQEBBCMPAQVBEAsNCwICBRYLAgIJAwIBAgFFBg0BBwEBsVlwkm2BI459gRMDnBmNMQ Received: from unknown (HELO [192.168.1.79]) ([109.77.2.188]) by mail1.vodafone.ie with ESMTP; 23 Jan 2013 12:34:39 +0000 Message-ID: <50FFD8DF.2090102@draigBrady.com> Date: Wed, 23 Jan 2013 12:34:39 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: Lei Zhang Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 13530 Cc: 13530@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: 0.8 (/) On 01/23/2013 02:32 AM, Lei Zhang wrote: > Hi All, > > We found a bug in the `head' program of coreutils 8.20: > > Invoking `head -c -P' or `head -c -E' will cause memory exhaustion. > > However, smaller units (e.g., b, K, M) work fine; bigger units (e.g., Z, Y) > fail properly > (by outputing "number of bytes is so large that it is not representable"). > And `-n' also > works fine. > > A bit dig reveals that the bug is introduced at line 322 of head.c > (coreutils 8.20): > > 204: elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t > n_elide_0) > 205: { > 322: b = xcalloc (n_bufs, sizeof *b); /** this statement fails **/ > 398: } > > We also examined n_elide and n_bufs before that statement. Actually, they > are very > large: > > n_elide: 1125899906842624 > n_bufs: 137438953474 > > According to the following comment in the source file: > >> CAUTION: do not fail (out of memory) when asked to elide >> a ridiculous amount, but when given only a small input. */ > > We think this is a bug and bring this issue to your attention. Thanks! There is the argument that we _should_ allocate everything up front to indicate immediately that the system can't (currently) support the requested operation, but given the 'currently' caveat above I guess it's more general to fail when we actually run out of mem? So to do that we can either realloc our pointer array in chunks or use the simpler approach in the patch below, and take advantage of kernel overcommit strategies. (calloc is problematic for overcommit as zeroing the memory fully allocates it to the process). The caveat with that though is that it's dependent on the overcommit config for the kernel and possibly current memory conditions. thanks, Pádraig. diff --git a/src/head.c b/src/head.c index cf84a95..909be04 100644 --- a/src/head.c +++ b/src/head.c @@ -197,7 +197,7 @@ copy_fd (int src_fd, FILE *o_stream, uintmax_t n_bytes) return COPY_FD_OK; } -/* Print all but the last N_ELIDE lines from the input available via +/* Print all but the last N_ELIDE bytes from the input available via the non-seekable file descriptor FD. Return true upon success. Give a diagnostic and return false upon error. */ static bool @@ -314,18 +314,22 @@ elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0) size_t n_read; bool buffered_enough; size_t i, i_next; + size_t n_alloc = 0; char **b; /* Round n_elide up to a multiple of READ_BUFSIZE. */ size_t rem = READ_BUFSIZE - (n_elide % READ_BUFSIZE); size_t n_elide_round = n_elide + rem; size_t n_bufs = n_elide_round / READ_BUFSIZE + 1; - b = xcalloc (n_bufs, sizeof *b); + b = xnmalloc (n_bufs, sizeof *b); buffered_enough = false; for (i = 0, i_next = 1; !eof; i = i_next, i_next = (i_next + 1) % n_bufs) { - if (b[i] == NULL) - b[i] = xmalloc (READ_BUFSIZE); + if (! buffered_enough) + { + b[i] = xmalloc (READ_BUFSIZE); + n_alloc = i + 1; + } n_read = full_read (fd, b[i], READ_BUFSIZE); if (n_read < READ_BUFSIZE) { @@ -389,7 +393,7 @@ elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0) } free_mem: - for (i = 0; i < n_bufs; i++) + for (i = 0; i < n_alloc; i++) free (b[i]); free (b); From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 23 07:55:11 2013 Received: (at 13530) by debbugs.gnu.org; 23 Jan 2013 12:55:11 +0000 Received: from localhost ([127.0.0.1]:45830 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TxzrC-0007BY-Un for submit@debbugs.gnu.org; Wed, 23 Jan 2013 07:55:11 -0500 Received: from moutng.kundenserver.de ([212.227.126.171]:59501) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TxzrB-0007BJ-BC for 13530@debbugs.gnu.org; Wed, 23 Jan 2013 07:55:10 -0500 Received: from [192.168.1.11] (p5083F782.dip.t-dialin.net [80.131.247.130]) by mrelayeu.kundenserver.de (node=mrbap1) with ESMTP (Nemesis) id 0LiX1g-1TLLhA0avk-00ctNC; Wed, 23 Jan 2013 13:53:48 +0100 Message-ID: <50FFDD5B.9010105@bernhard-voelker.de> Date: Wed, 23 Jan 2013 13:53:47 +0100 From: Bernhard Voelker User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130105 Thunderbird/17.0.2 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> In-Reply-To: <50FFD8DF.2090102@draigBrady.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Provags-ID: V02:K0:OwtGYLY+WJSsqoMKa6enBvViLg9FVruxBkybXk+7mbI s6mQJO3OUE2z9wyDI+gl1FxzAP3WjEZYtmhhc1q042kP2i16Ys RGSZj2vyon5OsBS4UGqBAFhx2KCh2mZb+ozyvk61vm7ehA2Qrb j//ErYkrzu7jKl8iUwQ3bRLpjTtTrxy1kOZZm3AqTjyoEsCdHo 6hwtk1t5PU/diLZXsutkeSRChVNSwYEUP0WfT2Dzol/wAyhsTh FPJVcMtSATufAe02yT/uQF+aUlEGAbm3UYr66XpkYRRtvHWBJj 6GkWNsZIWYSikkpZQ9JJQ7NCcfhRlef1YVDap5C3BHQOtukLlp y9MF6JQBKSayfiTAihvSR/4TCFFsrjgurE6JNZDuU X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 13530 Cc: Lei Zhang , 13530@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: 0.8 (/) On 01/23/2013 01:34 PM, Pádraig Brady wrote: > There is the argument that we _should_ allocate > everything up front to indicate immediately > that the system can't (currently) support the requested operation, > but given the 'currently' caveat above I guess it's > more general to fail when we actually run out of mem? head doesn't "allocate everything up front" - instead, it only allocates the pointer array which would hold the actual data. The strategy in elide_tail_lines_pipe() seems more robust: it allocates memory when needed. Or another (probably martian) idea: what about a tmpfile()? > free_mem: > - for (i = 0; i < n_bufs; i++) > + for (i = 0; i < n_alloc; i++) > free (b[i]); > free (b); BTW: both in the old and the new version, the loop can break if (b[i] == 0) because the array is filled from the beginning. - for (i = 0; i < n_bufs; i++) + for (i = 0; i < n_bufs && b[i]; i++) This makes "echo 123 | head -c -T" ~40% faster here on my PC. Have a nice day, Berny From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 23 08:05:20 2013 Received: (at 13530) by debbugs.gnu.org; 23 Jan 2013 13:05:20 +0000 Received: from localhost ([127.0.0.1]:45857 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ty00z-0007tN-Ex for submit@debbugs.gnu.org; Wed, 23 Jan 2013 08:05:18 -0500 Received: from mail1.vodafone.ie ([213.233.128.43]:38046) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ty00w-0007t6-Rn for 13530@debbugs.gnu.org; Wed, 23 Jan 2013 08:05:16 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApQBAOHd/1BtTQK8/2dsb2JhbAANN4ZFtBCDaoMRAQEBAwEjDwEFQQULCw0BCgICBRYLAgIJAwIBAgFFBg0BBwEBiA+pVnCSb4Ejjn2BEwOcGY0x Received: from unknown (HELO [192.168.1.79]) ([109.77.2.188]) by mail1.vodafone.ie with ESMTP; 23 Jan 2013 13:03:58 +0000 Message-ID: <50FFDFBB.2050901@draigBrady.com> Date: Wed, 23 Jan 2013 13:03:55 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: Bernhard Voelker Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <50FFDD5B.9010105@bernhard-voelker.de> In-Reply-To: <50FFDD5B.9010105@bernhard-voelker.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 13530 Cc: Lei Zhang , 13530@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: 0.8 (/) On 01/23/2013 12:53 PM, Bernhard Voelker wrote: > On 01/23/2013 01:34 PM, Pádraig Brady wrote: >> There is the argument that we _should_ allocate >> everything up front to indicate immediately >> that the system can't (currently) support the requested operation, >> but given the 'currently' caveat above I guess it's >> more general to fail when we actually run out of mem? > > head doesn't "allocate everything up front" - instead, it only > allocates the pointer array which would hold the actual data. Sure. I was wondering whether that should change to allocate everything up front so as to exit early. > The strategy in elide_tail_lines_pipe() seems more robust: > it allocates memory when needed. Yes probably. > Or another (probably martian) idea: > what about a tmpfile()? Not worth it I think. >> free_mem: >> - for (i = 0; i < n_bufs; i++) >> + for (i = 0; i < n_alloc; i++) >> free (b[i]); >> free (b); > > BTW: both in the old and the new version, the loop can break > if (b[i] == 0) because the array is filled from the beginning. The new version only frees what's allocated and so gets this benefit too. > - for (i = 0; i < n_bufs; i++) > + for (i = 0; i < n_bufs && b[i]; i++) > > This makes "echo 123 | head -c -T" ~40% faster here on my PC. nice thanks, Pádraig From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 23 08:22:51 2013 Received: (at 13530) by debbugs.gnu.org; 23 Jan 2013 13:22:51 +0000 Received: from localhost ([127.0.0.1]:45903 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ty0Hy-0000fv-OR for submit@debbugs.gnu.org; Wed, 23 Jan 2013 08:22:51 -0500 Received: from moutng.kundenserver.de ([212.227.17.10]:53202) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ty0Hv-0000fb-W1 for 13530@debbugs.gnu.org; Wed, 23 Jan 2013 08:22:48 -0500 Received: from [192.168.1.11] (p5083F782.dip.t-dialin.net [80.131.247.130]) by mrelayeu.kundenserver.de (node=mreu2) with ESMTP (Nemesis) id 0M2Yvh-1T5czv2nbR-00ruRB; Wed, 23 Jan 2013 14:21:30 +0100 Message-ID: <50FFE3D9.1020102@bernhard-voelker.de> Date: Wed, 23 Jan 2013 14:21:29 +0100 From: Bernhard Voelker User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130105 Thunderbird/17.0.2 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <50FFDD5B.9010105@bernhard-voelker.de> <50FFDFBB.2050901@draigBrady.com> In-Reply-To: <50FFDFBB.2050901@draigBrady.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Provags-ID: V02:K0:k+TbkxWzXsEAIq95t8CX557QKcjzP7PQHA/Nv8LQr9y qEcdMxud7bW6bpDQXdpLckF8slTCKdVDVQV9v+HYUqoEIosE2X nSpI3/TFYWJcLxNVH5gBEB9c9uRgZKnQ8MUp1zqBtlaLzTqC7f p09qqs9tw5fOQB93J8qFyM9I/DHhKlTGkuJEgnjzzcjtcLaXR0 cWe8r4ode42mVh/sNP7/kMj6SS7qELxsvmOF3QHf+RM30NJD72 lKTwOg1kxwJTo1jPMGrtYk9+Fz3mzNkWdACZr+3slrPxyfPtUw KlzlM95FIDkAr0juoo36pUR+uwl9okDSOeCnFSRx4JIjAq7BO4 2sOgkD0fkW9GRbK7mFHDmkHjZMbiCDNivnSBiNsJB X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 13530 Cc: Lei Zhang , 13530@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -0.0 (/) On 01/23/2013 02:03 PM, Pádraig Brady wrote: > On 01/23/2013 12:53 PM, Bernhard Voelker wrote: >> head doesn't "allocate everything up front" - instead, it only >> allocates the pointer array which would hold the actual data. > > Sure. I was wondering whether that should change > to allocate everything up front so as to exit early. I think that's not a good idea because head would ENOMEM even if it doesn't wouldn't need the memory (depending on the input). Why should the following fail? $ echo 123 | head -c -T I think there's no reason to fail until we really know it would fail. In this case, we can only know it when we actually come to the point when we need the memory. It doesn't matter if we have $ echo 123 | head -c -10 or $ echo 123 | head -c -P >>> free_mem: >>> - for (i = 0; i < n_bufs; i++) >>> + for (i = 0; i < n_alloc; i++) >>> free (b[i]); >>> free (b); >> >> BTW: both in the old and the new version, the loop can break >> if (b[i] == 0) because the array is filled from the beginning. > > The new version only frees what's allocated and so gets this benefit too. ah yes, sorry, I didn't look close enough. Have a nice day, Berny From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 23 08:28:40 2013 Received: (at 13530) by debbugs.gnu.org; 23 Jan 2013 13:28:40 +0000 Received: from localhost ([127.0.0.1]:45918 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ty0NZ-00014Q-JK for submit@debbugs.gnu.org; Wed, 23 Jan 2013 08:28:39 -0500 Received: from mail1.vodafone.ie ([213.233.128.43]:61982) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ty0NT-00013v-Hn for 13530@debbugs.gnu.org; Wed, 23 Jan 2013 08:28:33 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApMBAAjk/1BtTQK8/2dsb2JhbAANN4ZFtBCDaoMRAQEBBCMPAQVBEAsNAQoCAgUWCwICCQMCAQIBRQYNAQcBAbFkcJJvgSOOfYETA5wZjTE Received: from unknown (HELO [192.168.1.79]) ([109.77.2.188]) by mail1.vodafone.ie with ESMTP; 23 Jan 2013 13:27:15 +0000 Message-ID: <50FFE533.8030409@draigBrady.com> Date: Wed, 23 Jan 2013 13:27:15 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: Bernhard Voelker Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <50FFDD5B.9010105@bernhard-voelker.de> <50FFDFBB.2050901@draigBrady.com> <50FFE3D9.1020102@bernhard-voelker.de> In-Reply-To: <50FFE3D9.1020102@bernhard-voelker.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 13530 Cc: Lei Zhang , 13530@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: 0.8 (/) On 01/23/2013 01:21 PM, Bernhard Voelker wrote: > On 01/23/2013 02:03 PM, Pádraig Brady wrote: >> On 01/23/2013 12:53 PM, Bernhard Voelker wrote: >>> head doesn't "allocate everything up front" - instead, it only >>> allocates the pointer array which would hold the actual data. >> >> Sure. I was wondering whether that should change >> to allocate everything up front so as to exit early. > > I think that's not a good idea because head would ENOMEM > even if it doesn't wouldn't need the memory (depending on the > input). Why should the following fail? > > $ echo 123 | head -c -T > > I think there's no reason to fail until we really know it would > fail. In this case, we can only know it when we actually come > to the point when we need the memory. > > It doesn't matter if we have > $ echo 123 | head -c -10 > or > $ echo 123 | head -c -P Yes my point is that the above could be a problem at: today | head -c -P # This noop is OK tomorrow | head -c -P # If we actually send lots of data we fail cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Fri Apr 12 14:22:52 2013 Received: (at 13530-done) by debbugs.gnu.org; 12 Apr 2013 18:22:53 +0000 Received: from localhost ([127.0.0.1]:47700 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UQice-0003dQ-8M for submit@debbugs.gnu.org; Fri, 12 Apr 2013 14:22:52 -0400 Received: from mail3.vodafone.ie ([213.233.128.45]:33937) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UQicZ-0003cb-8T for 13530-done@debbugs.gnu.org; Fri, 12 Apr 2013 14:22:49 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjoDAPROaFFtTW87/2dsb2JhbAANQ4M8RIJqhV64UQMBgSKDEwEBAQQjVhALDQQDAQIBCRYLAgIJAwIBAgE9CAYNAQUCAQGIF6k+cZJSjjhOBgsHCYIlgRMDj1KIUYUHhVCGboE3 Received: from unknown (HELO [192.168.1.79]) ([109.77.111.59]) by mail3.vodafone.ie with ESMTP; 12 Apr 2013 19:18:49 +0100 Message-ID: <51685009.3000409@draigBrady.com> Date: Fri, 12 Apr 2013 19:18:49 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Lei Zhang Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> In-Reply-To: <50FFD8DF.2090102@draigBrady.com> X-Enigmail-Version: 1.5.1 Content-Type: multipart/mixed; boundary="------------070600000601070403060507" X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 13530-done Cc: 13530-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) This is a multi-part message in MIME format. --------------070600000601070403060507 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 01/23/2013 12:34 PM, Pádraig Brady wrote: > On 01/23/2013 02:32 AM, Lei Zhang wrote: >> Hi All, >> >> We found a bug in the `head' program of coreutils 8.20: >> >> Invoking `head -c -P' or `head -c -E' will cause memory exhaustion. >> >> However, smaller units (e.g., b, K, M) work fine; bigger units (e.g., Z, Y) >> fail properly >> (by outputing "number of bytes is so large that it is not representable"). >> And `-n' also >> works fine. >> >> A bit dig reveals that the bug is introduced at line 322 of head.c >> (coreutils 8.20): >> >> 204: elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t >> n_elide_0) >> 205: { >> 322: b = xcalloc (n_bufs, sizeof *b); /** this statement fails **/ >> 398: } >> >> We also examined n_elide and n_bufs before that statement. Actually, they >> are very >> large: >> >> n_elide: 1125899906842624 >> n_bufs: 137438953474 >> >> According to the following comment in the source file: >> >>> CAUTION: do not fail (out of memory) when asked to elide >>> a ridiculous amount, but when given only a small input. */ >> >> We think this is a bug and bring this issue to your attention. Thanks! > > There is the argument that we _should_ allocate > everything up front to indicate immediately > that the system can't (currently) support the requested operation, > but given the 'currently' caveat above I guess it's > more general to fail when we actually run out of mem? > > So to do that we can either realloc our pointer array > in chunks or use the simpler approach in the patch below, > and take advantage of kernel overcommit strategies. > (calloc is problematic for overcommit as zeroing the > memory fully allocates it to the process). > The caveat with that though is that it's dependent > on the overcommit config for the kernel and possibly > current memory conditions. > > thanks, > Pádraig. > > diff --git a/src/head.c b/src/head.c > index cf84a95..909be04 100644 > --- a/src/head.c > +++ b/src/head.c > @@ -197,7 +197,7 @@ copy_fd (int src_fd, FILE *o_stream, uintmax_t n_bytes) > return COPY_FD_OK; > } > > -/* Print all but the last N_ELIDE lines from the input available via > +/* Print all but the last N_ELIDE bytes from the input available via > the non-seekable file descriptor FD. Return true upon success. > Give a diagnostic and return false upon error. */ > static bool > @@ -314,18 +314,22 @@ elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0) > size_t n_read; > bool buffered_enough; > size_t i, i_next; > + size_t n_alloc = 0; > char **b; > /* Round n_elide up to a multiple of READ_BUFSIZE. */ > size_t rem = READ_BUFSIZE - (n_elide % READ_BUFSIZE); > size_t n_elide_round = n_elide + rem; > size_t n_bufs = n_elide_round / READ_BUFSIZE + 1; > - b = xcalloc (n_bufs, sizeof *b); > + b = xnmalloc (n_bufs, sizeof *b); > > buffered_enough = false; > for (i = 0, i_next = 1; !eof; i = i_next, i_next = (i_next + 1) % n_bufs) > { > - if (b[i] == NULL) > - b[i] = xmalloc (READ_BUFSIZE); > + if (! buffered_enough) > + { > + b[i] = xmalloc (READ_BUFSIZE); > + n_alloc = i + 1; > + } > n_read = full_read (fd, b[i], READ_BUFSIZE); > if (n_read < READ_BUFSIZE) > { > @@ -389,7 +393,7 @@ elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0) > } > > free_mem: > - for (i = 0; i < n_bufs; i++) > + for (i = 0; i < n_alloc; i++) > free (b[i]); > free (b); I expect to push soon, the attached more complete fix to realloc the array. thanks, Pádraig. --------------070600000601070403060507 Content-Type: text/x-patch; name="head-elide-tail-pipe-mem.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="head-elide-tail-pipe-mem.patch" >From 57d55485f0d2014c076f2cbfe0b340d1f2046952 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?P=C3=A1draig=20Brady?= Date: Wed, 23 Jan 2013 12:34:46 +0000 Subject: [PATCH] head: with --bytes=-N only allocate memory as needed * src/head.c (elide_tail_bytes_pipe): Don't use calloc as that bypasses memory overcommit due to the zeroing requirement. Also realloc rather than malloc the pointer array to avoid dependence on overcommit entirely. * tests/misc/head-c.sh: Add a test case. Fixes http://bugs.gnu.org/13530 --- src/head.c | 28 ++++++++++++++++++++++------ tests/misc/head-c.sh | 9 +++++++-- 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/src/head.c b/src/head.c index d79d5f7..00e1be1 100644 --- a/src/head.c +++ b/src/head.c @@ -196,7 +196,7 @@ copy_fd (int src_fd, FILE *o_stream, uintmax_t n_bytes) return COPY_FD_OK; } -/* Print all but the last N_ELIDE lines from the input available via +/* Print all but the last N_ELIDE bytes from the input available via the non-seekable file descriptor FD. Return true upon success. Give a diagnostic and return false upon error. */ static bool @@ -313,18 +313,34 @@ elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0) size_t n_read; bool buffered_enough; size_t i, i_next; - char **b; + char **b = NULL; /* Round n_elide up to a multiple of READ_BUFSIZE. */ size_t rem = READ_BUFSIZE - (n_elide % READ_BUFSIZE); size_t n_elide_round = n_elide + rem; size_t n_bufs = n_elide_round / READ_BUFSIZE + 1; - b = xcalloc (n_bufs, sizeof *b); + size_t n_alloc = 0; + size_t n_array_alloc = 0; buffered_enough = false; for (i = 0, i_next = 1; !eof; i = i_next, i_next = (i_next + 1) % n_bufs) { - if (b[i] == NULL) - b[i] = xmalloc (READ_BUFSIZE); + if (n_array_alloc == i) + { + /* reallocate between 16 and n_bufs entries. */ + if (n_array_alloc == 0) + n_array_alloc = MIN (n_bufs, 16); + else if (n_array_alloc <= n_bufs / 2) + n_array_alloc *= 2; + else + n_array_alloc = n_bufs; + b = xnrealloc (b, n_array_alloc, sizeof *b); + } + + if (! buffered_enough) + { + b[i] = xmalloc (READ_BUFSIZE); + n_alloc = i + 1; + } n_read = full_read (fd, b[i], READ_BUFSIZE); if (n_read < READ_BUFSIZE) { @@ -388,7 +404,7 @@ elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0) } free_mem: - for (i = 0; i < n_bufs; i++) + for (i = 0; i < n_alloc; i++) free (b[i]); free (b); diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh index 6807c4d..eada8d5 100755 --- a/tests/misc/head-c.sh +++ b/tests/misc/head-c.sh @@ -1,5 +1,5 @@ #!/bin/sh -# exercise the fix of 2001-08-18, based on test case from Ian Bruce +# exercise head -c # Copyright (C) 2001-2013 Free Software Foundation, Inc. @@ -19,12 +19,17 @@ . "${srcdir=.}/tests/init.sh"; path_prepend_ ./src print_ver_ head +# exercise the fix of 2001-08-18, based on test case from Ian Bruce echo abc > in || framework_failure_ - (head -c1; head -c1) < in > out || fail=1 case "$(cat out)" in ab) ;; *) fail=1 ;; esac +# Only allocate memory as needed. +# Coreutils <= 8.21 would allocate memory up front +# based on the value passed to -c +(ulimit -v 20000; head --bytes=-E < /dev/null) || fail=1 + Exit $fail -- 1.7.7.6 --------------070600000601070403060507-- From debbugs-submit-bounces@debbugs.gnu.org Fri Apr 12 15:10:29 2013 Received: (at 13530) by debbugs.gnu.org; 12 Apr 2013 19:10:29 +0000 Received: from localhost ([127.0.0.1]:47763 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UQjMi-0007ox-TO for submit@debbugs.gnu.org; Fri, 12 Apr 2013 15:10:29 -0400 Received: from mx.meyering.net ([88.168.87.75]:53153) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UQjMg-0007oX-CF for 13530@debbugs.gnu.org; Fri, 12 Apr 2013 15:10:27 -0400 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id B15C860103; Fri, 12 Apr 2013 21:06:28 +0200 (CEST) From: Jim Meyering To: 13530@debbugs.gnu.org Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes In-Reply-To: <51685009.3000409@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?= =?iso-8859-1?Q?g?= Brady"'s message of "Fri, 12 Apr 2013 19:18:49 +0100") References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> Date: Fri, 12 Apr 2013 21:06:28 +0200 Message-ID: <87ehefzhzf.fsf@rho.meyering.net> Lines: 5 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -3.9 (---) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, P@draigBrady.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.7 (----) P=E1draig Brady wrote: ... > I expect to push soon, the attached more complete fix to realloc the arra= y. Thanks! That change looks fine and passed tests here, too. From debbugs-submit-bounces@debbugs.gnu.org Fri Apr 12 15:24:12 2013 Received: (at 13530) by debbugs.gnu.org; 12 Apr 2013 19:24:12 +0000 Received: from localhost ([127.0.0.1]:47797 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UQjZz-0000Zf-7l for submit@debbugs.gnu.org; Fri, 12 Apr 2013 15:24:12 -0400 Received: from moutng.kundenserver.de ([212.227.126.186]:63473) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UQjZw-0000ZL-C5 for 13530@debbugs.gnu.org; Fri, 12 Apr 2013 15:24:09 -0400 Received: from [192.168.1.11] (p4FC3E1E8.dip.t-dialin.net [79.195.225.232]) by mrelayeu.kundenserver.de (node=mrbap4) with ESMTP (Nemesis) id 0MPGl6-1UV7E523jk-004iyr; Fri, 12 Apr 2013 21:19:57 +0200 Message-ID: <51685E5C.1060800@bernhard-voelker.de> Date: Fri, 12 Apr 2013 21:19:56 +0200 From: Bernhard Voelker User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130329 Thunderbird/17.0.5 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87ehefzhzf.fsf@rho.meyering.net> In-Reply-To: <87ehefzhzf.fsf@rho.meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Provags-ID: V02:K0:YJMgUM0BRNY9isjQlgNbEDwAN18YS8pFqIb0W1WLUNo uAy4bIsE9Gi7bNcltLkadbbWkQ1B0w9DdLfnJ2DNzcuQzgXgw6 trje7M2oF8uuYfpD5/Datj5Z6gKIjTIsLncDPhZL234mli2ET0 XSGBW3vxyHOs9FLZTJ22f8x207aM2Co3irH7I5KS8Wth00LvWr gqgEDKJk+ohDR0fQkZEc/Zt20vjZUFR9VVrcvpF6qkcwpFkhJ/ gXL8Xhyb6IApqsjSneBM2dosg8y8RcK9JGxwFZF1aej6O9RAdW JJR9dmNT5E27x4/oASmeGVMeghfKhewC7Lqw/8syxtd/LYSONN bzS71AChVcssxmb2auGvFbboLb7O1D+lLOqejQtI2 X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, 13530@debbugs.gnu.org, P@draigBrady.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 04/12/2013 09:06 PM, Jim Meyering wrote: > Pádraig Brady wrote: > ... >> I expect to push soon, the attached more complete fix to realloc the array. > > Thanks! That change looks fine and passed tests here, too. +1 Have a nice day, Berny From unknown Sat Aug 09 13:13:54 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sat, 11 May 2013 11:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator From debbugs-submit-bounces@debbugs.gnu.org Sun May 26 22:26:50 2013 Received: (at control) by debbugs.gnu.org; 27 May 2013 02:26:50 +0000 Received: from localhost ([127.0.0.1]:35930 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ugn97-0007xl-KY for submit@debbugs.gnu.org; Sun, 26 May 2013 22:26:49 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:10376) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Ugn95-0007xG-MC for control@debbugs.gnu.org; Sun, 26 May 2013 22:26:48 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Aj0DAN/ColFtTsJq/2dsb2JhbAANTYZzhHGDa7VmBAMdfYM3CipUDQIFFAILAgsDAgECATkGAgIVCAEBsyRykHCBJo0YfIIrgRMDnXyODw Received: from unknown (HELO [192.168.1.79]) ([109.78.194.106]) by mail2.vodafone.ie with ESMTP; 27 May 2013 03:25:31 +0100 Message-ID: <51A2C41A.2010308@draigBrady.com> Date: Mon, 27 May 2013 03:25:30 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: control@debbugs.gnu.org X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: 2.8 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 13530 [...] Content analysis details: (2.8 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [213.233.128.44 listed in list.dnswl.org] 0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60% [score: 0.4953] 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 13530 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [213.233.128.44 listed in list.dnswl.org] -0.0 BAYES_20 BODY: Bayes spam probability is 5 to 20% [score: 0.1560] 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject 0.0 TVD_SPACE_RATIO TVD_SPACE_RATIO unarchive 13530 From debbugs-submit-bounces@debbugs.gnu.org Sun May 26 22:37:26 2013 Received: (at 13530) by debbugs.gnu.org; 27 May 2013 02:37:26 +0000 Received: from localhost ([127.0.0.1]:35939 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UgnJK-0000pe-Uq for submit@debbugs.gnu.org; Sun, 26 May 2013 22:37:25 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:7596) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UgnIP-0000mz-QO for 13530@debbugs.gnu.org; Sun, 26 May 2013 22:36:46 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Aj0DAB3FolFtTsJq/2dsb2JhbAANTcJPgmoDAYEZgxcBAQEDATIBRgULCw0LCRQCDwkDAgECAUUGDQEHAQGIA6sWkWSNZlhfB4NUA518jFeBOIFnASM Received: from unknown (HELO [192.168.1.79]) ([109.78.194.106]) by mail2.vodafone.ie with ESMTP; 27 May 2013 03:35:01 +0100 Message-ID: <51A2C655.5090404@draigBrady.com> Date: Mon, 27 May 2013 03:35:01 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <8761y5s1sm.fsf@rho.meyering.net> In-Reply-To: <8761y5s1sm.fsf@rho.meyering.net> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, 13530@debbugs.gnu.org, coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 05/27/2013 01:29 AM, Jim Meyering wrote: > Pádraig Brady wrote: > >> unarchive 13530 >> stop > > Thanks. > > ... >>> @@ -358,6 +356,14 @@ elide_tail_bytes_pipe (const char *filename, int fd, uintmax_t n_elide_0) >>> >>> if (buffered_enough) >>> { >>> + if (n_elide_0 != n_elide) >>> + { >>> + error (0, 0, _("memory exhausted while reading %s"), >>> + quote (filename)); >>> + ok = false; >>> + goto free_mem; >>> + } >>> + > ... >> Oh right it's coming back to me a bit now. >> So by removing these upfront checks where possible, >> it only makes sense of the program could under different >> free mem available, fulfil the operation up to the specified limit. >> In this case though, the program could never fulfil the request, >> so it's better to fail early as is the case in the code currently? > > Well, it *can* fulfill the request whenever the request is degenerate, > i.e., when the size of the input is smaller than N and also small enough > to be read into memory. Sure, but... > Technically, we could handle this case the same way we handle it > in tac.c: read data from nonseekable FD and write it to a temporary file. > I'm not sure it's worth the effort here, though. Yes ideally we could avoid this limit, though I don't see it as a priority either. > ... >>> -(ulimit -v 20000; head --bytes=-E < /dev/null) || fail=1 >>> +(ulimit -v 20000; head --bytes=-$OFF_T_MAX < /dev/null) || fail=1 > > I'm inclined to make the above (nonseekable input) cases succeed, > for consistency with the seekable-input case like this: > > : > empty > head --bytes=-E empty > > I confess that I did not like the way my manual test ended up > using so much memory... but it couldn't know if it was going > to be able to succeed without actually reading/allocating all > of that space. > > If we give up immediately, we fail unnecessarily in cases like > the above where the input is smaller than N. > > What do you think? ... I'm inclined to treat a value that could never be fulfilled in total as invalid. Otherwise one might run into unexpected limits in _future_. This is the same way we immediately error on values over UINTMAX_MAX, rather truncating the values silently to something we can. thanks, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 20:08:40 2013 Received: (at 13530) by debbugs.gnu.org; 28 May 2013 00:08:40 +0000 Received: from localhost ([127.0.0.1]:36930 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh7Sx-0006EE-WA for submit@debbugs.gnu.org; Mon, 27 May 2013 20:08:40 -0400 Received: from mx.meyering.net ([88.168.87.75]:46336) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh7Su-0006E1-Pd for 13530@debbugs.gnu.org; Mon, 27 May 2013 20:08:38 -0400 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id 995EC601B2; Tue, 28 May 2013 02:07:20 +0200 (CEST) From: Jim Meyering To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes In-Reply-To: <51A2A1DB.6030304@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?= =?iso-8859-1?Q?g?= Brady"'s message of "Mon, 27 May 2013 00:59:23 +0100") References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> Date: Tue, 28 May 2013 02:07:20 +0200 Message-ID: <874ndootl3.fsf@rho.meyering.net> Lines: 68 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, 13530@debbugs.gnu.org, coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) P=E1draig Brady wrote: ... >> -(ulimit -v 20000; head --bytes=3D-E < /dev/null) || fail=3D1 >> +(ulimit -v 20000; head --bytes=3D-$OFF_T_MAX < /dev/null) || fail=3D1 >> >> Exit $fail >> -- >> 1.8.3 >> > > So the test would have to be adjusted to take the min(SIZE_MAX, OFF_T_MAX= )? > > HEAD_TAIL_LIMIT=3D$(printf '%s\n' $SIZE_MAX $OFF_T_MAX | sort -g | head -= n1) I didn't see a reason to include OFF_T_MAX, since this has to do solely with in-memory buffering, and no seeking. Also, I had to subtract BUFSIZ, so made another adjustment. This works for me: >From 6b20bb5c9c464277f843a1d3f418824275f25f6b Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 27 May 2013 17:01:14 -0700 Subject: [PATCH] tests: head-c: avoid spurious failure with a 32-bit SIZE_M= AX * tests/misc/head-c.sh: When eliding N bytes from a non-seekable input, N must be slightly smaller than SIZE_MAX in order to handle input longer than N bytes, since the current implementation buffers N bytes in memory. This command would fail on 32-bit systems, where SIZE_MAX < 1E: head --bytes=3D-E < /dev/null Instead of "E", use a value slightly smaller than SIZE_MAX. --- tests/misc/head-c.sh | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh index 37a86ce..70a6ccc 100755 --- a/tests/misc/head-c.sh +++ b/tests/misc/head-c.sh @@ -19,6 +19,7 @@ . "${srcdir=3D.}/tests/init.sh"; path_prepend_ ./src print_ver_ head require_ulimit_v_ +getlimits_ # exercise the fix of 2001-08-18, based on test case from Ian Bruce echo abc > in || framework_failure_ @@ -28,9 +29,16 @@ case "$(cat out)" in *) fail=3D1 ;; esac +# Use a limit of N =3D SIZE_MAX - max_BUFSIZ +# The "- max_BUFSIZ" term is because head must be able to add BUFSIZ +# to the selected value of N without exceeding SIZE_MAX. +# Since we've seen BUFSIZ up to 128K, use 256K to be safe. +max_BUFSIZ=3D$(expr 256 '*' 1024) +lim=3D$(expr $SIZE_MAX - $max_BUFSIZ) + # Only allocate memory as needed. # Coreutils <=3D 8.21 would allocate memory up front # based on the value passed to -c -(ulimit -v 20000; head --bytes=3D-E < /dev/null) || fail=3D1 +(ulimit -v 20000; head --bytes=3D-$lim < /dev/null) || fail=3D1 Exit $fail -- 1.8.3 From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 20:27:03 2013 Received: (at 13530) by debbugs.gnu.org; 28 May 2013 00:27:03 +0000 Received: from localhost ([127.0.0.1]:36940 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh7kk-0006sC-LN for submit@debbugs.gnu.org; Mon, 27 May 2013 20:27:03 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:35501) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh7ki-0006rh-21 for 13530@debbugs.gnu.org; Mon, 27 May 2013 20:27:01 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 83AE3A60001; Mon, 27 May 2013 17:25:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iKgGt-VgUdJz; Mon, 27 May 2013 17:25:37 -0700 (PDT) Received: from [192.168.1.9] (pool-71-108-49-126.lsanca.fios.verizon.net [71.108.49.126]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 600C439E8100; Mon, 27 May 2013 17:25:37 -0700 (PDT) Message-ID: <51A3F980.3050504@cs.ucla.edu> Date: Mon, 27 May 2013 17:25:36 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> In-Reply-To: <874ndootl3.fsf@rho.meyering.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, 13530@debbugs.gnu.org, =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) On 05/27/2013 05:07 PM, Jim Meyering wrote: > +max_BUFSIZ=$(expr 256 '*' 1024) > +lim=$(expr $SIZE_MAX - $max_BUFSIZ) Can't this code fail, due to overflow, on non-GMP hosts? See: http://lists.gnu.org/archive/html/coreutils/2013-05/msg00060.html and look for "$SIZE_MAX". From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 20:55:48 2013 Received: (at 13530) by debbugs.gnu.org; 28 May 2013 00:55:48 +0000 Received: from localhost ([127.0.0.1]:36948 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh8CZ-0007qp-Qv for submit@debbugs.gnu.org; Mon, 27 May 2013 20:55:48 -0400 Received: from mx.meyering.net ([88.168.87.75]:46388) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh8CX-0007qg-33 for 13530@debbugs.gnu.org; Mon, 27 May 2013 20:55:46 -0400 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id 0F0C3601B2; Tue, 28 May 2013 02:54:28 +0200 (CEST) From: Jim Meyering To: Paul Eggert Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes In-Reply-To: <51A3F980.3050504@cs.ucla.edu> (Paul Eggert's message of "Mon, 27 May 2013 17:25:36 -0700") References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> <51A3F980.3050504@cs.ucla.edu> Date: Tue, 28 May 2013 02:54:28 +0200 Message-ID: <87wqqjorej.fsf@rho.meyering.net> Lines: 38 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, 13530@debbugs.gnu.org, =?iso-8859-1?Q?P=E1draig?= Brady , coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Paul Eggert wrote: > On 05/27/2013 05:07 PM, Jim Meyering wrote: > >> +max_BUFSIZ=$(expr 256 '*' 1024) >> +lim=$(expr $SIZE_MAX - $max_BUFSIZ) > > Can't this code fail, due to overflow, on non-GMP hosts? See: > > http://lists.gnu.org/archive/html/coreutils/2013-05/msg00060.html > > and look for "$SIZE_MAX". Thanks for mentioning that. I propose to move your subtract_one variable into init.cfg and then use it like this: diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh index 70a6ccc..41ea52b 100755 --- a/tests/misc/head-c.sh +++ b/tests/misc/head-c.sh @@ -34,7 +34,15 @@ esac # to the selected value of N without exceeding SIZE_MAX. # Since we've seen BUFSIZ up to 128K, use 256K to be safe. max_BUFSIZ=$(expr 256 '*' 1024) -lim=$(expr $SIZE_MAX - $max_BUFSIZ) + +# Normally we would just write this, +# lim=$(expr $SIZE_MAX - $max_BUFSIZ) +# But that fails for non-GMP expr. See this: +# https://lists.gnu.org/archive/html/coreutils/2013-05/msg00060.html +# Instead, use that same approach to obtain SIZE_MAX-1, and *then* +# subtract $max_BUFSIZ. +lim=$(echo $SIZE_MAX | sed "$subtract_one") +lim=$(expr $lim - $max_BUFSIZ) # Only allocate memory as needed. # Coreutils <= 8.21 would allocate memory up front From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 21:05:41 2013 Received: (at 13530) by debbugs.gnu.org; 28 May 2013 01:05:41 +0000 Received: from localhost ([127.0.0.1]:36953 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh8M8-0008DY-FG for submit@debbugs.gnu.org; Mon, 27 May 2013 21:05:41 -0400 Received: from mx.meyering.net ([88.168.87.75]:46404) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh8M5-0008DK-Tf for 13530@debbugs.gnu.org; Mon, 27 May 2013 21:05:39 -0400 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id ADBB0600BB; Tue, 28 May 2013 03:04:21 +0200 (CEST) From: Jim Meyering To: Paul Eggert Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes In-Reply-To: <51A3F980.3050504@cs.ucla.edu> (Paul Eggert's message of "Mon, 27 May 2013 17:25:36 -0700") References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> <51A3F980.3050504@cs.ucla.edu> Date: Tue, 28 May 2013 03:04:21 +0200 Message-ID: <87r4groqy2.fsf@rho.meyering.net> Lines: 163 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, 13530@debbugs.gnu.org, =?iso-8859-1?Q?P=E1draig?= Brady , coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Paul Eggert wrote: > On 05/27/2013 05:07 PM, Jim Meyering wrote: > >> +max_BUFSIZ=$(expr 256 '*' 1024) >> +lim=$(expr $SIZE_MAX - $max_BUFSIZ) > > Can't this code fail, due to overflow, on non-GMP hosts? See: > > http://lists.gnu.org/archive/html/coreutils/2013-05/msg00060.html > > and look for "$SIZE_MAX". Here are two patches. The first factors out the definition into a new function. The second uses it in the revised head-c-fixing patch. Both tests still pass, though I haven't yet run them against a GMP-free expr. >From f97095c19244d61af4172ab457b5bc79081ada79 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 27 May 2013 17:59:41 -0700 Subject: [PATCH 1/2] maint: factor out new subtract_one_ function * tests/misc/cut-huge-range.sh (subtract_one): Move definition of this sed script to init.cfg so we can use it from another test. * init.cfg (subtract_one_): New function, from that variable. --- init.cfg | 24 ++++++++++++++++++++++++ tests/misc/cut-huge-range.sh | 22 +--------------------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/init.cfg b/init.cfg index c48607c..b013c4e 100644 --- a/init.cfg +++ b/init.cfg @@ -596,4 +596,28 @@ require_gnu_() || skip_ 'not running on GNU/Hurd' } +subtract_one_() +{ + # sed script to subtract one from the input. + # Each input line should consist of a positive decimal number. + # Each output line's number is one less than the input's. + # There is no limit (other than line length) on the number's magnitude. + local subtract_one=' + s/$/@/ + : again + s/0@/@9/ + s/1@/0/ + s/2@/1/ + s/3@/2/ + s/4@/3/ + s/5@/4/ + s/6@/5/ + s/7@/6/ + s/8@/7/ + s/9@/8/ + t again + ' + sed "$subtract_one" +} + sanitize_path_ diff --git a/tests/misc/cut-huge-range.sh b/tests/misc/cut-huge-range.sh index 7816577..ae7cc70 100755 --- a/tests/misc/cut-huge-range.sh +++ b/tests/misc/cut-huge-range.sh @@ -21,31 +21,11 @@ print_ver_ cut require_ulimit_v_ getlimits_ -# sed script to subtract one from the input. -# Each input line should consist of a positive decimal number. -# Each output line's number is one less than the input's. -# There's no limit (other than line length) on the number's magnitude. -subtract_one=' - s/$/@/ - : again - s/0@/@9/ - s/1@/0/ - s/2@/1/ - s/3@/2/ - s/4@/3/ - s/5@/4/ - s/6@/5/ - s/7@/6/ - s/8@/7/ - s/9@/8/ - t again -' - # Ensure we can cut up to our sentinel value. # This is currently SIZE_MAX, but could be raised to UINTMAX_MAX # if we didn't allocate memory for each line as a unit. # Don't use expr to subtract one, since SIZE_MAX may exceed its maximum value. -CUT_MAX=$(echo $SIZE_MAX | sed "$subtract_one") +CUT_MAX=$(echo $SIZE_MAX | subtract_one_ ) # From coreutils-8.10 through 8.20, this would make cut try to allocate # a 256MiB bit vector. With a 20MB limit on VM, the following would fail. -- 1.8.3 >From 905cd2b5c503c82894b433767810ff2f1e40b69d Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 27 May 2013 17:01:14 -0700 Subject: [PATCH 2/2] tests: head-c: avoid spurious failure with a 32-bit SIZE_MAX * tests/misc/head-c.sh: When eliding N bytes from a non-seekable input, N must be slightly smaller than SIZE_MAX in order to handle input longer than N bytes, since the current implementation buffers N bytes in memory. This command would fail on 32-bit systems, where SIZE_MAX < 1E: head --bytes=-E < /dev/null Instead of "E", use a value slightly smaller than SIZE_MAX. --- tests/misc/head-c.sh | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh index 37a86ce..155c7f6 100755 --- a/tests/misc/head-c.sh +++ b/tests/misc/head-c.sh @@ -19,6 +19,7 @@ . "${srcdir=.}/tests/init.sh"; path_prepend_ ./src print_ver_ head require_ulimit_v_ +getlimits_ # exercise the fix of 2001-08-18, based on test case from Ian Bruce echo abc > in || framework_failure_ @@ -28,9 +29,24 @@ case "$(cat out)" in *) fail=1 ;; esac +# Use a limit of N = SIZE_MAX - max_BUFSIZ +# The "- max_BUFSIZ" term is because head must be able to add BUFSIZ +# to the selected value of N without exceeding SIZE_MAX. +# Since we've seen BUFSIZ up to 128K, use 256K to be safe. +max_BUFSIZ=$(expr 256 '*' 1024) + +# Normally we would just write this, +# lim=$(expr $SIZE_MAX - $max_BUFSIZ) +# But that fails for non-GMP expr. See this: +# https://lists.gnu.org/archive/html/coreutils/2013-05/msg00060.html +# Instead, use that same approach to obtain SIZE_MAX-1, and *then* +# subtract $max_BUFSIZ. +lim=$(echo $SIZE_MAX | subtract_one_) +lim=$(expr $lim - $max_BUFSIZ) + # Only allocate memory as needed. # Coreutils <= 8.21 would allocate memory up front # based on the value passed to -c -(ulimit -v 20000; head --bytes=-E < /dev/null) || fail=1 +(ulimit -v 20000; head --bytes=-$lim < /dev/null) || fail=1 Exit $fail -- 1.8.3 From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 21:15:58 2013 Received: (at 13530) by debbugs.gnu.org; 28 May 2013 01:15:58 +0000 Received: from localhost ([127.0.0.1]:36960 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh8W5-0000Cd-RO for submit@debbugs.gnu.org; Mon, 27 May 2013 21:15:58 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:36924) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh8W2-0000CF-9u for 13530@debbugs.gnu.org; Mon, 27 May 2013 21:15:55 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 09C5CA60002; Mon, 27 May 2013 18:14:33 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ij9gj-D6gOVl; Mon, 27 May 2013 18:14:32 -0700 (PDT) Received: from [192.168.1.9] (pool-71-108-49-126.lsanca.fios.verizon.net [71.108.49.126]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 96326A60001; Mon, 27 May 2013 18:14:32 -0700 (PDT) Message-ID: <51A404F8.3040702@cs.ucla.edu> Date: Mon, 27 May 2013 18:14:32 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> <51A3F980.3050504@cs.ucla.edu> <87r4groqy2.fsf@rho.meyering.net> In-Reply-To: <87r4groqy2.fsf@rho.meyering.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, 13530@debbugs.gnu.org, =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) On 05/27/2013 06:04 PM, Jim Meyering wrote: > +lim=$(echo $SIZE_MAX | subtract_one_) > +lim=$(expr $lim - $max_BUFSIZ) Sorry, I don't see how this will work either. It's common for a GMP-less expr to handle values only up to SIZE_MAX / 2, and subtracting just 1 won't work around that problem. Maybe divide by 10 instead? That's easy to do textually. (I don't know what the test is about so I'm not sure what to suggest.) From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 21:40:14 2013 Received: (at 13530) by debbugs.gnu.org; 28 May 2013 01:40:14 +0000 Received: from localhost ([127.0.0.1]:36968 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh8tZ-00014P-QV for submit@debbugs.gnu.org; Mon, 27 May 2013 21:40:14 -0400 Received: from mx.meyering.net ([88.168.87.75]:46438) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uh8tX-00014E-Ch for 13530@debbugs.gnu.org; Mon, 27 May 2013 21:40:12 -0400 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id 3973D600BB; Tue, 28 May 2013 03:38:55 +0200 (CEST) From: Jim Meyering To: Paul Eggert Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes In-Reply-To: <51A404F8.3040702@cs.ucla.edu> (Paul Eggert's message of "Mon, 27 May 2013 18:14:32 -0700") References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> <51A3F980.3050504@cs.ucla.edu> <87r4groqy2.fsf@rho.meyering.net> <51A404F8.3040702@cs.ucla.edu> Date: Tue, 28 May 2013 03:38:55 +0200 Message-ID: <87ip23opcg.fsf@rho.meyering.net> Lines: 81 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, 13530@debbugs.gnu.org, =?iso-8859-1?Q?P=E1draig?= Brady , coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Paul Eggert wrote: > On 05/27/2013 06:04 PM, Jim Meyering wrote: >> +lim=$(echo $SIZE_MAX | subtract_one_) >> +lim=$(expr $lim - $max_BUFSIZ) > > Sorry, I don't see how this will work either. > It's common for a GMP-less expr to handle values > only up to SIZE_MAX / 2, and subtracting just 1 > won't work around that problem. I was concentrating on the 32-bit case, where GMP-less expr works fine on $SIZE_MAX. It's on 64-bit that it's a problem, so... > Maybe divide by 10 instead? That's easy to do > textually. (I don't know what the test is about > so I'm not sure what to suggest.) I'd rather not divide by 10 all around. That'd decrease it too much in the 32-bit case. I took a different tack: >From 0d81799b18bef79b49d9042111f9dee3312796a7 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 27 May 2013 17:01:14 -0700 Subject: [PATCH] tests: head-c: avoid spurious failure with a 32-bit SIZE_MAX * tests/misc/head-c.sh: When eliding N bytes from a non-seekable input, N must be slightly smaller than SIZE_MAX in order to handle input longer than N bytes, since the current implementation buffers N bytes in memory. This command would fail on 32-bit systems, where SIZE_MAX < 1E: head --bytes=-E < /dev/null Instead of "E", use a value slightly smaller than SIZE_MAX. --- tests/misc/head-c.sh | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh index 37a86ce..a81754e 100755 --- a/tests/misc/head-c.sh +++ b/tests/misc/head-c.sh @@ -19,6 +19,7 @@ . "${srcdir=.}/tests/init.sh"; path_prepend_ ./src print_ver_ head require_ulimit_v_ +getlimits_ # exercise the fix of 2001-08-18, based on test case from Ian Bruce echo abc > in || framework_failure_ @@ -28,9 +29,27 @@ case "$(cat out)" in *) fail=1 ;; esac +# Use a limit of N = SIZE_MAX - max_BUFSIZ +# The "- max_BUFSIZ" term is because head must be able to add BUFSIZ +# to the selected value of N without exceeding SIZE_MAX. +# Since we've seen BUFSIZ up to 128K, use 256K to be safe. +max_BUFSIZ=$(expr 256 '*' 1024) + +# It's ok to use a 10-digit $SIZE_MAX, because expr uses wider intmax_t. +# However, when $SIZE_MAX is longer, it's a 20-digit quantity that is +# too large for GMP-disabled expr to handle, so use $SSIZE_MAX there. +# We'd use $SSIZE_MAX in both cases, but want to keep the number as +# large as possible for the 32-bit case, since there, it may barely +# exceed the size of available RAM, while SSIZE_MAX probably will not. +test $(printf $SIZE_MAX|wc -c) -lt 11 \ + && big=$SIZE_MAX \ + || big=$SSIZE_MAX + +lim=$(expr $big - $max_BUFSIZ) \ + # Only allocate memory as needed. # Coreutils <= 8.21 would allocate memory up front # based on the value passed to -c -(ulimit -v 20000; head --bytes=-E < /dev/null) || fail=1 +(ulimit -v 20000; head --bytes=-$lim < /dev/null) || fail=1 Exit $fail -- 1.8.3 From debbugs-submit-bounces@debbugs.gnu.org Tue May 28 05:13:10 2013 Received: (at 13530) by debbugs.gnu.org; 28 May 2013 09:13:10 +0000 Received: from localhost ([127.0.0.1]:37195 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UhFxu-000210-1k for submit@debbugs.gnu.org; Tue, 28 May 2013 05:13:10 -0400 Received: from mail3.vodafone.ie ([213.233.128.45]:5094) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UhFxq-00020G-Jx for 13530@debbugs.gnu.org; Tue, 28 May 2013 05:13:08 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AnYDAMdzpFFtTJ4j/2dsb2JhbAANTIM4wg4DAYEbgxcBAQEEMgFGEAsNBAMBAgEJFg8JAwIBAgE9CAYNAQUCAQGyFZF0jVkNWF8Hg1QDk2qEeoUYjFeBOIFnAQ Received: from unknown (HELO [192.168.1.79]) ([109.76.158.35]) by mail3.vodafone.ie with ESMTP; 28 May 2013 10:11:42 +0100 Message-ID: <51A474CE.9040403@draigBrady.com> Date: Tue, 28 May 2013 10:11:42 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> <51A3F980.3050504@cs.ucla.edu> <87r4groqy2.fsf@rho.meyering.net> <51A404F8.3040702@cs.ucla.edu> <87ip23opcg.fsf@rho.meyering.net> In-Reply-To: <87ip23opcg.fsf@rho.meyering.net> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, Paul Eggert , 13530@debbugs.gnu.org, coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 05/28/2013 02:38 AM, Jim Meyering wrote: > Paul Eggert wrote: >> On 05/27/2013 06:04 PM, Jim Meyering wrote: >>> +lim=$(echo $SIZE_MAX | subtract_one_) >>> +lim=$(expr $lim - $max_BUFSIZ) >> >> Sorry, I don't see how this will work either. >> It's common for a GMP-less expr to handle values >> only up to SIZE_MAX / 2, and subtracting just 1 >> won't work around that problem. > > I was concentrating on the 32-bit case, where GMP-less expr > works fine on $SIZE_MAX. It's on 64-bit that it's a problem, so... > >> Maybe divide by 10 instead? That's easy to do >> textually. (I don't know what the test is about >> so I'm not sure what to suggest.) > > I'd rather not divide by 10 all around. > That'd decrease it too much in the 32-bit case. > I took a different tack: > > >>>From 0d81799b18bef79b49d9042111f9dee3312796a7 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Mon, 27 May 2013 17:01:14 -0700 > Subject: [PATCH] tests: head-c: avoid spurious failure with a 32-bit SIZE_MAX > > * tests/misc/head-c.sh: When eliding N bytes from a non-seekable > input, N must be slightly smaller than SIZE_MAX in order to handle > input longer than N bytes, since the current implementation buffers > N bytes in memory. This command would fail on 32-bit systems, > where SIZE_MAX < 1E: > head --bytes=-E < /dev/null > Instead of "E", use a value slightly smaller than SIZE_MAX. > --- > tests/misc/head-c.sh | 21 ++++++++++++++++++++- > 1 file changed, 20 insertions(+), 1 deletion(-) > > diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh > index 37a86ce..a81754e 100755 > --- a/tests/misc/head-c.sh > +++ b/tests/misc/head-c.sh > @@ -19,6 +19,7 @@ > . "${srcdir=.}/tests/init.sh"; path_prepend_ ./src > print_ver_ head > require_ulimit_v_ > +getlimits_ > > # exercise the fix of 2001-08-18, based on test case from Ian Bruce > echo abc > in || framework_failure_ > @@ -28,9 +29,27 @@ case "$(cat out)" in > *) fail=1 ;; > esac > > +# Use a limit of N = SIZE_MAX - max_BUFSIZ > +# The "- max_BUFSIZ" term is because head must be able to add BUFSIZ > +# to the selected value of N without exceeding SIZE_MAX. > +# Since we've seen BUFSIZ up to 128K, use 256K to be safe. > +max_BUFSIZ=$(expr 256 '*' 1024) > + > +# It's ok to use a 10-digit $SIZE_MAX, because expr uses wider intmax_t. > +# However, when $SIZE_MAX is longer, it's a 20-digit quantity that is > +# too large for GMP-disabled expr to handle, so use $SSIZE_MAX there. > +# We'd use $SSIZE_MAX in both cases, but want to keep the number as > +# large as possible for the 32-bit case, since there, it may barely > +# exceed the size of available RAM, while SSIZE_MAX probably will not. > +test $(printf $SIZE_MAX|wc -c) -lt 11 \ > + && big=$SIZE_MAX \ > + || big=$SSIZE_MAX > + > +lim=$(expr $big - $max_BUFSIZ) \ trailing \ is not needed. The above assumes intmax_t is always wider than size_t, which is might be OK? Also since OFF_T_MAX is checked earlier in the code, it assumes that off_t is wider than size_t which isn't the case if _FILE_OFFSET_BITS is not 64. You could take the approach to just use $SSIZE_MAX, and assume 64 bit testing covers the 32 bit case implicitly? That would also assume that off_t was never narrower than size_t, but that's probably OK. These limits are painful. Sorry for the churn :( thanks, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 01 23:48:44 2013 Received: (at 13530) by debbugs.gnu.org; 2 Jun 2013 03:48:44 +0000 Received: from localhost ([127.0.0.1]:47201 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UizHf-0008Kg-KO for submit@debbugs.gnu.org; Sat, 01 Jun 2013 23:48:44 -0400 Received: from mx.meyering.net ([88.168.87.75]:55929) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UizHb-0008KF-HQ for 13530@debbugs.gnu.org; Sat, 01 Jun 2013 23:48:41 -0400 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id 61A5D600B1; Sun, 2 Jun 2013 05:46:53 +0200 (CEST) From: Jim Meyering To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes In-Reply-To: <51A474CE.9040403@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?= =?iso-8859-1?Q?g?= Brady"'s message of "Tue, 28 May 2013 10:11:42 +0100") References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> <51A3F980.3050504@cs.ucla.edu> <87r4groqy2.fsf@rho.meyering.net> <51A404F8.3040702@cs.ucla.edu> <87ip23opcg.fsf@rho.meyering.net> <51A474CE.9040403@draigBrady.com> Date: Sun, 02 Jun 2013 05:46:53 +0200 Message-ID: <874ndhdviq.fsf@rho.meyering.net> Lines: 82 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, Paul Eggert , 13530@debbugs.gnu.org, coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) P=E1draig Brady wrote: ... >> +# Use a limit of N =3D SIZE_MAX - max_BUFSIZ >> +# The "- max_BUFSIZ" term is because head must be able to add BUFSIZ >> +# to the selected value of N without exceeding SIZE_MAX. >> +# Since we've seen BUFSIZ up to 128K, use 256K to be safe. >> +max_BUFSIZ=3D$(expr 256 '*' 1024) >> + >> +# It's ok to use a 10-digit $SIZE_MAX, because expr uses wider intmax_t. >> +# However, when $SIZE_MAX is longer, it's a 20-digit quantity that is >> +# too large for GMP-disabled expr to handle, so use $SSIZE_MAX there. >> +# We'd use $SSIZE_MAX in both cases, but want to keep the number as >> +# large as possible for the 32-bit case, since there, it may barely >> +# exceed the size of available RAM, while SSIZE_MAX probably will not. >> +test $(printf $SIZE_MAX|wc -c) -lt 11 \ >> + && big=3D$SIZE_MAX \ >> + || big=3D$SSIZE_MAX >> + >> +lim=3D$(expr $big - $max_BUFSIZ) \ > > trailing \ is not needed. > > The above assumes intmax_t is always wider than size_t, > which is might be OK? > Also since OFF_T_MAX is checked earlier in the code, > it assumes that off_t is wider than size_t which isn't > the case if _FILE_OFFSET_BITS is not 64. > > You could take the approach to just use $SSIZE_MAX, > and assume 64 bit testing covers the 32 bit case implicitly? > That would also assume that off_t was never narrower than size_t, > but that's probably OK. > > These limits are painful. Sorry for the churn :( This started because the test passed on 64-bit, yet failed on 32-bit. If it were to use $SSIZE_MAX, the test could mistakenly pass when run without the underlying patch, because on a 32-bit system, it's not uncommon to have 2^31 bytes of RAM, so the allocation could succeed. However, that's so much simpler that I now think it's the way to go. Thanks! >From aeaeb87c134ce748527ba99e749b7369fcba2438 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 27 May 2013 17:01:14 -0700 Subject: [PATCH] tests: head-c: avoid spurious failure with a 32-bit size_t * tests/misc/head-c.sh: Don't try to elide 1 exabytes, since on 32-bit systems, that number is not representable as a size_t. This command would fail on 32-bit systems, where SIZE_MAX < 1E: head --bytes=3D-E < /dev/null Instead of "E", use $SSIZE_MAX. For discussion, see http://bugs.gnu.org/13530 --- tests/misc/head-c.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh index 37a86ce..a5b6c6a 100755 --- a/tests/misc/head-c.sh +++ b/tests/misc/head-c.sh @@ -19,6 +19,7 @@ . "${srcdir=3D.}/tests/init.sh"; path_prepend_ ./src print_ver_ head require_ulimit_v_ +getlimits_ # exercise the fix of 2001-08-18, based on test case from Ian Bruce echo abc > in || framework_failure_ @@ -31,6 +32,6 @@ esac # Only allocate memory as needed. # Coreutils <=3D 8.21 would allocate memory up front # based on the value passed to -c -(ulimit -v 20000; head --bytes=3D-E < /dev/null) || fail=3D1 +(ulimit -v 20000; head --bytes=3D-$SSIZE_MAX < /dev/null) || fail=3D1 Exit $fail -- 1.8.3.101.g727a46b From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 02 06:53:52 2013 Received: (at 13530) by debbugs.gnu.org; 2 Jun 2013 10:53:52 +0000 Received: from localhost ([127.0.0.1]:47373 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uj5v5-0005aP-BS for submit@debbugs.gnu.org; Sun, 02 Jun 2013 06:53:52 -0400 Received: from mail3.vodafone.ie ([213.233.128.45]:57199) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Uj5v1-0005Zj-Tr for 13530@debbugs.gnu.org; Sun, 02 Jun 2013 06:53:49 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlwDAFIjq1FtTT3N/2dsb2JhbAANTYM5gnW8BwMBgRKDFwEBAQQyAUYQCw0LCRYPCQMCAQIBRQYNAQcBAbACkXaOSF8Hg1gDmGeFGIxXgTg Received: from unknown (HELO [192.168.1.79]) ([109.77.61.205]) by mail3.vodafone.ie with ESMTP; 02 Jun 2013 11:51:55 +0100 Message-ID: <51AB23CB.9000208@draigBrady.com> Date: Sun, 02 Jun 2013 11:51:55 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> <51A3F980.3050504@cs.ucla.edu> <87r4groqy2.fsf@rho.meyering.net> <51A404F8.3040702@cs.ucla.edu> <87ip23opcg.fsf@rho.meyering.net> <51A474CE.9040403@draigBrady.com> <874ndhdviq.fsf@rho.meyering.net> In-Reply-To: <874ndhdviq.fsf@rho.meyering.net> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, Paul Eggert , 13530@debbugs.gnu.org, coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 06/02/2013 04:46 AM, Jim Meyering wrote: > diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh > index 37a86ce..a5b6c6a 100755 > --- a/tests/misc/head-c.sh > +++ b/tests/misc/head-c.sh > @@ -19,6 +19,7 @@ > . "${srcdir=.}/tests/init.sh"; path_prepend_ ./src > print_ver_ head > require_ulimit_v_ > +getlimits_ > > # exercise the fix of 2001-08-18, based on test case from Ian Bruce > echo abc > in || framework_failure_ > @@ -31,6 +32,6 @@ esac > # Only allocate memory as needed. > # Coreutils <= 8.21 would allocate memory up front > # based on the value passed to -c > -(ulimit -v 20000; head --bytes=-E < /dev/null) || fail=1 > +(ulimit -v 20000; head --bytes=-$SSIZE_MAX < /dev/null) || fail=1 +1 thanks! Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 02 12:27:00 2013 Received: (at 13530) by debbugs.gnu.org; 2 Jun 2013 16:27:00 +0000 Received: from localhost ([127.0.0.1]:47972 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UjB7R-0007Cp-NI for submit@debbugs.gnu.org; Sun, 02 Jun 2013 12:26:59 -0400 Received: from mx.meyering.net ([88.168.87.75]:56839) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1UjB7M-0007Cd-Oe for 13530@debbugs.gnu.org; Sun, 02 Jun 2013 12:26:54 -0400 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id E41AB60179; Sun, 2 Jun 2013 18:25:03 +0200 (CEST) From: Jim Meyering To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes In-Reply-To: <51AB23CB.9000208@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?= =?iso-8859-1?Q?g?= Brady"'s message of "Sun, 02 Jun 2013 11:51:55 +0100") References: <50FFD8DF.2090102@draigBrady.com> <51685009.3000409@draigBrady.com> <87a9nhu0tl.fsf@rho.meyering.net> <51A2595D.2090606@draigBrady.com> <87y5b1sdi9.fsf@rho.meyering.net> <87ppwds8cp.fsf@rho.meyering.net> <51A2A1DB.6030304@draigBrady.com> <874ndootl3.fsf@rho.meyering.net> <51A3F980.3050504@cs.ucla.edu> <87r4groqy2.fsf@rho.meyering.net> <51A404F8.3040702@cs.ucla.edu> <87ip23opcg.fsf@rho.meyering.net> <51A474CE.9040403@draigBrady.com> <874ndhdviq.fsf@rho.meyering.net> <51AB23CB.9000208@draigBrady.com> Date: Sun, 02 Jun 2013 18:25:03 +0200 Message-ID: <87bo7ocwf4.fsf@rho.meyering.net> Lines: 24 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -4.7 (----) X-Debbugs-Envelope-To: 13530 Cc: lei.zhang@uwaterloo.ca, Paul Eggert , 13530@debbugs.gnu.org, coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.7 (----) P=E1draig Brady wrote: > On 06/02/2013 04:46 AM, Jim Meyering wrote: >> diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh >> index 37a86ce..a5b6c6a 100755 >> --- a/tests/misc/head-c.sh >> +++ b/tests/misc/head-c.sh >> @@ -19,6 +19,7 @@ >> . "${srcdir=3D.}/tests/init.sh"; path_prepend_ ./src >> print_ver_ head >> require_ulimit_v_ >> +getlimits_ >> >> # exercise the fix of 2001-08-18, based on test case from Ian Bruce >> echo abc > in || framework_failure_ >> @@ -31,6 +32,6 @@ esac >> # Only allocate memory as needed. >> # Coreutils <=3D 8.21 would allocate memory up front >> # based on the value passed to -c >> -(ulimit -v 20000; head --bytes=3D-E < /dev/null) || fail=3D1 >> +(ulimit -v 20000; head --bytes=3D-$SSIZE_MAX < /dev/null) || fail=3D1 > > +1 Pushed. From unknown Sat Aug 09 13:13:54 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 01 Jul 2013 11:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator