GNU bug report logs - #16662
Libtool execute mode is DOSing my system

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 16662 in the body.
You can then email your comments to 16662 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> draconx.ca>
To: bug-libtool <at> gnu.org
Subject: Libtool execute mode is DOSing my system
Date: Wed, 5 Feb 2014 21:06:57 -0500

Hi,

I was surprised to discover that in one of my test cases, running the
program via libtool --mode=execute is consuming huge amounts of memory,
apparently because of a runaway sed program it runs on my program's
input files.  This is a problem because these input files are not plain
text files, are several gigabytes in size and GNU sed balloons out to
consume all available memory.

It does eventually complete and the test passes, but not before spending
several minutes pushing the entirety of system memory out to disk.

This can be easily reproduced by running something like the following:

  % truncate -s 16G /tmp/bigfile
  % libtool --mode=execute /usr/bin/wc -c /tmp/bigfile

Running sh -x libtool, we see...

  [...]
  + for file in '"$@"'
  + case $file in
  + func_ltwrapper_script_p /tmp/bigfile
  + func_lalib_p /tmp/bigfile
  + test -f /tmp/bigfile
  + /bin/grep '^# Generated by .*libtool'
  + /bin/sed -e 4q /tmp/bigfile
  [thrashing begins here]

and sure enough, running just that sed command by itself exhibits the
same behaviour.

I guess I can see why libtool's execute mode has to search to see
which input files are libtool wrapper scripts, but that sed program
is obviously a poor choice for large binary files.

Frustratingly, putting "--" before the input files does not appear to
affect the behaviour of libtool at all in this matter.

So it would be nice if libtool didn't consume all available memory
before it even starts running my program :)

For reference, I'm running:

  libtool 2.4.2
  GNU sed 4.2.1

Cheers,
  Nick

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: bug-libtool <at> gnu.org
Subject: Re: bug#16662: Libtool execute mode is DOSing my system
Date: Thu, 06 Feb 2014 08:30:54 +0100

On 2014-02-06 03:06, Nick Bowler wrote:
> Hi,
> 
> I was surprised to discover that in one of my test cases, running the
> program via libtool --mode=execute is consuming huge amounts of memory,
> apparently because of a runaway sed program it runs on my program's
> input files.  This is a problem because these input files are not plain
> text files, are several gigabytes in size and GNU sed balloons out to
> consume all available memory.

Dup of bug#13472, please see

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13472

Cheers,
Peter

Message #11 received at 16662 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Nick Bowler <nbowler <at> draconx.ca>
Cc: Pavel Raiskup <pavel <at> raiskup.cz>, control <at> debbugs.gnu.org,
 13472 <at> debbugs.gnu.org, 16662 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Tue, 06 May 2014 09:44:34 +0200

[Message part 1 (text/plain, inline)]

close 13472
close 16662
stop

On 2014-02-14 17:43, Nick Bowler wrote:
> On 2/13/14, Peter Rosin <peda <at> lysator.liu.se> wrote:
>> On 2014-02-13 18:57, Peter Rosin wrote:
>>> Ok, everybody seems happy. But not me, not 100% anyway. This introduces
>>> an extra fork, and AFAICT, the "extra protection" is only needed when
>>> func_lalib_p is called from func_ltwrapper_script_p. Should we perhaps
>>> have a separate implementation in func_ltwrapper_script_p instead of
>>> simply calling func_lalib_p?
>>>
>>> Maybe we could also lose the "sed -e 4q" part when dd limits the size?
>>
>> *snip* outline, real patch attached instead...
>>
>> Now, which patch should I push?
> 
> The new looks a bit nicer to me although I have not tested it.
> 
> One comment I have that applies to both patches... you should probably
> be using AC_CHECK_PROGS instead of AC_PATH_PROG because the latter
> requires an absolute filename.
> 
> Specifically, AC_PATH_PROG will ignore any user-set value for DD if
> it's not absolute.

Ok, I'm reviving this old thing once again. There were two minor complaints
with the last patch, this patch addresses both. I.e.

1. DD=dd now works to override the dd check, you don't need to specify
a full path.
2. The code duplication between func_lalib_p and func_ltwrapper_script_p
has been factored out.

However, to address the first point, I made use of the
AC_PATH_PROGS_FEATURE_CHECK macro, which was added in Autoconf 2.62, which
Libtool didn't require before. But 2.62 is 6 years old by now. If anyone
has problems with that, we will have to fix it later as I'm now pushing
this thing. It's been brewing for far too long already...

Final version attached.

Cheers,
Peter

[0001-libtool-speed-up-ltwrapper_script-detection-in-execu.patch (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.