From unknown Thu Aug 14 12:24:30 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#16662 <16662@debbugs.gnu.org> To: bug#16662 <16662@debbugs.gnu.org> Subject: Status: Libtool execute mode is DOSing my system Reply-To: bug#16662 <16662@debbugs.gnu.org> Date: Thu, 14 Aug 2025 19:24:30 +0000 retitle 16662 Libtool execute mode is DOSing my system reassign 16662 libtool submitter 16662 Nick Bowler severity 16662 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 05 21:07:03 2014 Received: (at submit) by debbugs.gnu.org; 6 Feb 2014 02:07:03 +0000 Received: from localhost ([127.0.0.1]:51763 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WBEMo-0004Ck-Ki for submit@debbugs.gnu.org; Wed, 05 Feb 2014 21:07:02 -0500 Received: from eggs.gnu.org ([208.118.235.92]:43994) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WBEMn-0004CM-1c for submit@debbugs.gnu.org; Wed, 05 Feb 2014 21:07:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WBEMl-0003Ao-RH for submit@debbugs.gnu.org; Wed, 05 Feb 2014 21:07:00 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:37562) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WBEMl-0003Aj-OP for submit@debbugs.gnu.org; Wed, 05 Feb 2014 21:06:59 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45706) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WBEMk-0002xp-PF for bug-libtool@gnu.org; Wed, 05 Feb 2014 21:06:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WBEMj-0003AD-Qc for bug-libtool@gnu.org; Wed, 05 Feb 2014 21:06:58 -0500 Received: from mail-qa0-x233.google.com ([2607:f8b0:400d:c00::233]:39981) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WBEMj-00039x-MG for bug-libtool@gnu.org; Wed, 05 Feb 2014 21:06:57 -0500 Received: by mail-qa0-f51.google.com with SMTP id f11so1849832qae.38 for ; Wed, 05 Feb 2014 18:06:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=3CpEoeXKsHVFNIS0wRVI2tpYY4XOv1TLNTBl+XXuSr8=; b=i8KzyGQXvzvH700+DCkogTlcGv/PI1ytvl1Jk+pv57UHMtKhf75BUY89QM7gDbXb6U Dv5eQk34iKltQ2wfGYmzqnKgcJ9Ia/Gzt1c8q2b0GgtcRjO/lfoeuhGr4Vma5Q2uf30V s6bfO1fqjyIfExt8esH3WmNxXaJ87OAl5L0+3+RyuxtYNQ/xb66OwHbWMPPUT1FU6wbz 15nqQOfwQOrLJMrytFymD8rukVeK36/TLYzn4OQONWbHjgC706H9wRZYJCW/t3TGGlRk dTnWV9gCuWiaiwO8EVX8m2LJemq/DeecB+GAV1CjCH+EU7ywrIFEsK2v2g/8asTP6Vwk aaiQ== MIME-Version: 1.0 X-Received: by 10.224.5.69 with SMTP id 5mr8257995qau.11.1391652417124; Wed, 05 Feb 2014 18:06:57 -0800 (PST) Received: by 10.96.170.35 with HTTP; Wed, 5 Feb 2014 18:06:57 -0800 (PST) Date: Wed, 5 Feb 2014 21:06:57 -0500 X-Google-Sender-Auth: aJTtc2f_wd6D9EEx6LzGgmake9w Message-ID: Subject: Libtool execute mode is DOSing my system From: Nick Bowler To: bug-libtool@gnu.org Content-Type: text/plain; charset=UTF-8 X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) Hi, I was surprised to discover that in one of my test cases, running the program via libtool --mode=execute is consuming huge amounts of memory, apparently because of a runaway sed program it runs on my program's input files. This is a problem because these input files are not plain text files, are several gigabytes in size and GNU sed balloons out to consume all available memory. It does eventually complete and the test passes, but not before spending several minutes pushing the entirety of system memory out to disk. This can be easily reproduced by running something like the following: % truncate -s 16G /tmp/bigfile % libtool --mode=execute /usr/bin/wc -c /tmp/bigfile Running sh -x libtool, we see... [...] + for file in '"$@"' + case $file in + func_ltwrapper_script_p /tmp/bigfile + func_lalib_p /tmp/bigfile + test -f /tmp/bigfile + /bin/grep '^# Generated by .*libtool' + /bin/sed -e 4q /tmp/bigfile [thrashing begins here] and sure enough, running just that sed command by itself exhibits the same behaviour. I guess I can see why libtool's execute mode has to search to see which input files are libtool wrapper scripts, but that sed program is obviously a poor choice for large binary files. Frustratingly, putting "--" before the input files does not appear to affect the behaviour of libtool at all in this matter. So it would be nice if libtool didn't consume all available memory before it even starts running my program :) For reference, I'm running: libtool 2.4.2 GNU sed 4.2.1 Cheers, Nick From debbugs-submit-bounces@debbugs.gnu.org Thu Feb 06 02:31:31 2014 Received: (at submit) by debbugs.gnu.org; 6 Feb 2014 07:31:31 +0000 Received: from localhost ([127.0.0.1]:52012 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WBJQo-0006gV-EW for submit@debbugs.gnu.org; Thu, 06 Feb 2014 02:31:30 -0500 Received: from eggs.gnu.org ([208.118.235.92]:38027) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WBJQl-0006gM-0e for submit@debbugs.gnu.org; Thu, 06 Feb 2014 02:31:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WBJQf-00012D-6U for submit@debbugs.gnu.org; Thu, 06 Feb 2014 02:31:26 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:37944) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WBJQf-000129-3b for submit@debbugs.gnu.org; Thu, 06 Feb 2014 02:31:21 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39740) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WBJQZ-0002lP-ND for bug-libtool@gnu.org; Thu, 06 Feb 2014 02:31:21 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WBJQU-00010R-Ca for bug-libtool@gnu.org; Thu, 06 Feb 2014 02:31:15 -0500 Received: from mail.lysator.liu.se ([2001:6b0:17:f0a0::3]:56535) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WBJQU-00010L-5h for bug-libtool@gnu.org; Thu, 06 Feb 2014 02:31:10 -0500 Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id EE9F74002D for ; Thu, 6 Feb 2014 08:31:05 +0100 (CET) Received: from [192.168.0.68] (90-227-119-221-no95.business.telia.com [90.227.119.221]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id CD5B840012 for ; Thu, 6 Feb 2014 08:31:05 +0100 (CET) Message-ID: <52F33A2E.40408@lysator.liu.se> Date: Thu, 06 Feb 2014 08:30:54 +0100 From: Peter Rosin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: bug-libtool@gnu.org Subject: Re: bug#16662: Libtool execute mode is DOSing my system References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.3 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.3 (----) On 2014-02-06 03:06, Nick Bowler wrote: > Hi, > > I was surprised to discover that in one of my test cases, running the > program via libtool --mode=execute is consuming huge amounts of memory, > apparently because of a runaway sed program it runs on my program's > input files. This is a problem because these input files are not plain > text files, are several gigabytes in size and GNU sed balloons out to > consume all available memory. Dup of bug#13472, please see http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13472 Cheers, Peter From debbugs-submit-bounces@debbugs.gnu.org Tue May 06 03:44:53 2014 Received: (at 16662) by debbugs.gnu.org; 6 May 2014 07:44:53 +0000 Received: from localhost ([127.0.0.1]:52313 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wha3W-0003G9-3e for submit@debbugs.gnu.org; Tue, 06 May 2014 03:44:53 -0400 Received: from mail.lysator.liu.se ([130.236.254.3]:50871) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wha3M-0003Ff-7l; Tue, 06 May 2014 03:44:44 -0400 Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id B4AC940042; Tue, 6 May 2014 09:44:38 +0200 (CEST) Received: from [192.168.0.68] (90-227-119-221-no95.business.telia.com [90.227.119.221]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 7A52940040; Tue, 6 May 2014 09:44:36 +0200 (CEST) Message-ID: <536892E2.5040604@lysator.liu.se> Date: Tue, 06 May 2014 09:44:34 +0200 From: Peter Rosin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Nick Bowler Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input by dd References: <5107F2FB.50103@lysator.liu.se> <683610949.2m6zIZog4y@unused-4-121.brq.redhat.com> <2406035.DfkmgaF2Wx@unused-4-121.brq.redhat.com> <1706515.fgxXn01m9q@praiskup.usersys.redhat.com> <52FA61F2.9030009@lysator.liu.se> <52FD0781.7070006@lysator.liu.se> <52FD229B.5020300@lysator.liu.se> In-Reply-To: Content-Type: multipart/mixed; boundary="------------030401060304080109010303" X-Virus-Scanned: ClamAV using ClamSMTP X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 16662 Cc: Pavel Raiskup , control@debbugs.gnu.org, 13472@debbugs.gnu.org, 16662@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) This is a multi-part message in MIME format. --------------030401060304080109010303 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit close 13472 close 16662 stop On 2014-02-14 17:43, Nick Bowler wrote: > On 2/13/14, Peter Rosin wrote: >> On 2014-02-13 18:57, Peter Rosin wrote: >>> Ok, everybody seems happy. But not me, not 100% anyway. This introduces >>> an extra fork, and AFAICT, the "extra protection" is only needed when >>> func_lalib_p is called from func_ltwrapper_script_p. Should we perhaps >>> have a separate implementation in func_ltwrapper_script_p instead of >>> simply calling func_lalib_p? >>> >>> Maybe we could also lose the "sed -e 4q" part when dd limits the size? >> >> *snip* outline, real patch attached instead... >> >> Now, which patch should I push? > > The new looks a bit nicer to me although I have not tested it. > > One comment I have that applies to both patches... you should probably > be using AC_CHECK_PROGS instead of AC_PATH_PROG because the latter > requires an absolute filename. > > Specifically, AC_PATH_PROG will ignore any user-set value for DD if > it's not absolute. Ok, I'm reviving this old thing once again. There were two minor complaints with the last patch, this patch addresses both. I.e. 1. DD=dd now works to override the dd check, you don't need to specify a full path. 2. The code duplication between func_lalib_p and func_ltwrapper_script_p has been factored out. However, to address the first point, I made use of the AC_PATH_PROGS_FEATURE_CHECK macro, which was added in Autoconf 2.62, which Libtool didn't require before. But 2.62 is 6 years old by now. If anyone has problems with that, we will have to fix it later as I'm now pushing this thing. It's been brewing for far too long already... Final version attached. Cheers, Peter --------------030401060304080109010303 Content-Type: text/x-patch; name="0001-libtool-speed-up-ltwrapper_script-detection-in-execu.patch" Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename*0="0001-libtool-speed-up-ltwrapper_script-detection-in-execu.pa"; filename*1="tch" >From da30ce4dc9554c80f1931600af2b8bbab486476e Mon Sep 17 00:00:00 2001 From: Peter Rosin Date: Tue, 6 May 2014 00:03:19 +0200 Subject: [PATCH] libtool: speed up ltwrapper_script detection in execute mode Execute mode is slow and might even DOS the computer in extreme cases when a parameter is a big binary file without newlines. Work around this with different truncation if a suitable dd utility is found. Fixes bug#13472 and bug#16662. Reported by Pavel Raiskup and Nick Bowler. * m4/libtool.m4 (_LT_PATH_DD): New macro, for finding a dd utility that works for the below purpose. (_LT_CMD_TRUNCATE): New macro, for finding out how to truncate binary pipes (fallback to the old sed truncation if no suitable dd is found in _LT_PATH_DD). (_LT_SETUP): Require _LT_CMD_TRUNCATE. (LT_INIT): Require Autoconf 2.62, as needed by _LT_PATH_DD. * build_aux/ltmain.in (func_lalib_p): Factor out the actual "generated by libtool" test into... (func_generated_by_libtool_p): ...this new function... (func_ltwrapper_script_p): ...so that it can be reused here, when truncating the pipe according to _LT_CMD_TRUNCATE. * THANKS: Update. Signed-off-by: Peter Rosin --- THANKS | 2 ++ build-aux/ltmain.in | 14 +++++++++++--- m4/libtool.m4 | 40 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 52 insertions(+), 4 deletions(-) diff --git a/THANKS b/THANKS index e895aee..85c5150 100644 --- a/THANKS +++ b/THANKS @@ -150,6 +150,7 @@ Mike Gorchak mike@malva.ua Mike Frysinger vapier@gentoo.org Mike Miller mtmiller@ieee.org + Nick Bowler nbowler@draconx.ca Nix nix@esperi.org.uk Olaf Lenz olenz@fias.uni-frankfurt.de Olly Betts olly@muscat.co.uk @@ -161,6 +162,7 @@ Paul Eggert eggert@twinsun.com Paul Laight plaight@quantxautomation.co.uk Paul Seidler sepek@lavabit.com + Pavel Raiskup praiskup@redhat.com Paweł Daniluk pawel@bioexploratorium.pl Peter Eisentraut peter_e@gmx.net Peter Fritzsche peter.fritzsche@gmx.de diff --git a/build-aux/ltmain.in b/build-aux/ltmain.in index 3a0fb0c..6af4087 100644 --- a/build-aux/ltmain.in +++ b/build-aux/ltmain.in @@ -570,6 +570,14 @@ $1 _LTECHO_EOF' } +# func_generated_by_libtool +# True iff stdin has been generated by Libtool. This function is only +# a basic sanity check; it will hardly flush out determined imposters. +func_generated_by_libtool_p () +{ + $GREP "^# Generated by .*$PACKAGE" > /dev/null 2>&1 +} + # func_lalib_p file # True iff FILE is a libtool '.la' library or '.lo' object file. # This function is only a basic sanity check; it will hardly flush out @@ -577,8 +585,7 @@ _LTECHO_EOF' func_lalib_p () { test -f "$1" && - $SED -e 4q "$1" 2>/dev/null \ - | $GREP "^# Generated by .*$PACKAGE" > /dev/null 2>&1 + $SED -e 4q "$1" 2>/dev/null | func_generated_by_libtool_p } # func_lalib_unsafe_p file @@ -610,7 +617,8 @@ func_lalib_unsafe_p () # determined imposters. func_ltwrapper_script_p () { - func_lalib_p "$1" + test -f "$1" && + $lt_truncate_bin < "$1" 2>/dev/null | func_generated_by_libtool_p } # func_ltwrapper_executable_p file diff --git a/m4/libtool.m4 b/m4/libtool.m4 index ce58f31..0454030 100644 --- a/m4/libtool.m4 +++ b/m4/libtool.m4 @@ -59,7 +59,7 @@ esac # LT_INIT([OPTIONS]) # ------------------ AC_DEFUN([LT_INIT], -[AC_PREREQ([2.58])dnl We use AC_INCLUDES_DEFAULT +[AC_PREREQ([2.62])dnl We use AC_PATH_PROGS_FEATURE_CHECK AC_REQUIRE([AC_CONFIG_AUX_DIR_DEFAULT])dnl AC_BEFORE([$0], [LT_LANG])dnl AC_BEFORE([$0], [LT_OUTPUT])dnl @@ -169,6 +169,7 @@ m4_require([_LT_CHECK_SHAREDLIB_FROM_LINKLIB])dnl m4_require([_LT_CMD_OLD_ARCHIVE])dnl m4_require([_LT_CMD_GLOBAL_SYMBOLS])dnl m4_require([_LT_WITH_SYSROOT])dnl +m4_require([_LT_CMD_TRUNCATE])dnl _LT_CONFIG_LIBTOOL_INIT([ # See if we are running on zsh, and set the options that allow our @@ -3216,6 +3217,43 @@ _LT_TAGDECL([], [reload_cmds], [2])dnl ])# _LT_CMD_RELOAD +# _LT_PATH_DD +# ----------- +# find a working dd +m4_defun([_LT_PATH_DD], +[AC_CACHE_CHECK([for a working dd], [ac_cv_path_lt_DD], +[printf 0123456789abcdef0123456789abcdef >conftest.i +cat conftest.i conftest.i >conftest2.i +: ${lt_DD:=$DD} +AC_PATH_PROGS_FEATURE_CHECK([lt_DD], [dd], +[if "$ac_path_lt_DD" bs=32 count=1 conftest.out 2>/dev/null; then + cmp -s conftest.i conftest.out \ + && ac_cv_path_lt_DD="$ac_path_lt_DD" ac_path_lt_DD_found=: +fi]) +rm -f conftest.i conftest2.i conftest.out]) +])# _LT_PATH_DD + + +# _LT_CMD_TRUNCATE +# ---------------- +# find command to truncate a binary pipe +m4_defun([_LT_CMD_TRUNCATE], +[m4_require([_LT_PATH_DD]) +AC_CACHE_CHECK([how to truncate binary pipes], [lt_cv_truncate_bin], +[printf 0123456789abcdef0123456789abcdef >conftest.i +cat conftest.i conftest.i >conftest2.i +lt_cv_truncate_bin= +if "$ac_cv_path_lt_DD" bs=32 count=1 conftest.out 2>/dev/null; then + cmp -s conftest.i conftest.out \ + && lt_cv_truncate_bin="$ac_cv_path_lt_DD bs=4096 count=1" +fi +rm -f conftest.i conftest2.i conftest.out +test -z "$lt_cv_truncate_bin" && lt_cv_truncate_bin="$SED -e 4q"]) +_LT_DECL([lt_truncate_bin], [lt_cv_truncate_bin], [1], + [Command to truncate a binary pipe]) +])# _LT_CMD_TRUNCATE + + # _LT_CHECK_MAGIC_METHOD # ---------------------- # how to check for library dependencies -- 1.7.9 --------------030401060304080109010303-- From unknown Thu Aug 14 12:24:30 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 03 Jun 2014 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator