GNU bug report logs - #13472
Execution mode can hang pc when parameter refers to big file

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 13472 in the body.
You can then email your comments to 13472 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: bug-libtool <at> gnu.org
Subject: Execution mode can hang pc when parameter refers to big file
Date: Thu, 17 Jan 2013 12:09:44 +0100

Hi liboolers,

the problem — e.g. running the:

  libtool execute ls BIGfile

could stack computer atm (if user has swap enabled and) if the BIGfile
refers to really big file not containing new-lines.

For more info see the bug:

  http://bugzilla.redhat.com/636045

=====

The problem is that libtool is trying to guess whether the 'BIGfile' is
ltwrapper or not (I guess).  One step in this check is that libtool runs
sed for 'head' purposes:

  sed -e 4q BIGfile

The problem of this sed command is that it is probably buffering whole
"binary lines" (which may have gigabytes) into memory.  Don't know "how"
it is doing there, but 'head' utility hes _smaller_ space complexity (two
runs?).  Unfortunately, both sed & head could go through whole really big
file and it can take too long (+ sed costs too much memory).

If you see the Comment #7 in referenced bug, Paolo suggest to use 'dd'
utility.  It would be easy to use dd as sed's "limiter":

diff --git a/build-aux/ltmain.in b/build-aux/ltmain.in
index c8cdb9c..a92b3bb 100644
--- a/build-aux/ltmain.in
+++ b/build-aux/ltmain.in
@@ -575,7 +575,7 @@ _LTECHO_EOF'
 func_lalib_p ()
 {
     test -f "$1" &&
-      $SED -e 4q "$1" 2>/dev/null \
+      dd if="$1" bs=16K count=1 2>/dev/null | $SED -e 4q 2>/dev/null \
         | $GREP "^# Generated by .*$PACKAGE" > /dev/null 2>&1
 }

Using sed as the head utility is quite common in libtool project, probably
it would be fine to generalize head task that (or other) way.  I am able
to prepare patch if you are interested.

Pavel

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
To: Pavel Raiskup <praiskup <at> redhat.com>
Cc: 13472 <at> debbugs.gnu.org, bug-libtool <at> gnu.org
Subject: Re: bug#13472: Execution mode can hang pc when parameter refers to
	big file
Date: Thu, 17 Jan 2013 09:29:47 -0600 (CST)

[Message part 1 (text/plain, inline)]

On Thu, 17 Jan 2013, Pavel Raiskup wrote:

> Hi liboolers,
>
> the problem — e.g. running the:
>
>  libtool execute ls BIGfile
>
> could stack computer atm (if user has swap enabled and) if the BIGfile
> refers to really big file not containing new-lines.
>
> For more info see the bug:
>
>  http://bugzilla.redhat.com/636045
>
> =====
>
> The problem is that libtool is trying to guess whether the 'BIGfile' is
> ltwrapper or not (I guess).  One step in this check is that libtool runs
> sed for 'head' purposes:
>
>  sed -e 4q BIGfile
>
> The problem of this sed command is that it is probably buffering whole
> "binary lines" (which may have gigabytes) into memory.  Don't know "how"

This seems like a problem even on larger machines.  Executables and 
libraries can be rather large files.  If sed does absorb the whole 
file (reading tens of megabytes, or even over 100 MB, for debug 
builds) then there is an efficiency loss.

Since libtool has a configure script, it can test for whatever 
features are needed and provide a portable fallback in case 'dd' is 
not available.

Everything would be easier if libtool would arrange to build a small 
helper executable in the build tree which it can used instead of 
system tools.  Libtool already builds a native executable in some 
cases.  This would diminish portability concerns and improve 
performance.  The main issue with this is that for cross-builds, a 
system native compiler needs to be used to build the helper 
executable.

Bob
-- 
Bob Friesenhahn
bfriesen <at> simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Petr Hracek <phracek <at> redhat.com>
To: bug-libtool <at> gnu.org
Cc: praiskup <at> redhat.com, bfriesen <at> simple.dallas.tx.us
Subject: bug#13472: Execution mode can hang pc when parameter refers to big
	file
Date: Tue, 22 Jan 2013 16:03:02 +0100

[Message part 1 (text/plain, inline)]

Hello Bob,

I have created simple helper binary which can be replaced with sed
in case that dd is not available.
Patch is in attachment including new c source file.

best regards
Petr
--
Greetings,
Petr Hracek

BaseOS Core Services Brno

[helper_dd.patch (text/x-patch, attachment)]

Message #17 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Cc: Pavel Raiskup <praiskup <at> redhat.com>, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: Execution mode can hang pc when parameter refers to
	big file
Date: Tue, 29 Jan 2013 15:43:06 +0100

Hi Bob,

On 2013-01-17 16:29, Bob Friesenhahn wrote:
> Everything would be easier if libtool would arrange to build a small helper
> executable in the build tree which it can used instead of system tools.
> Libtool already builds a native executable in some cases.  This would
> diminish portability concerns and improve performance.  The main issue
> with this is that for cross-builds, a system native compiler needs to be
> used to build the helper executable.

Forgive me for my ignorance, but when does libtool build something with
the $build compiler in the cross case? The only thing I can find that
looks like an alternative compiler is $LTCC, but that's just another
name for the normal $host C compiler. What am I missing?

Cheers,
Peter

Message #20 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
To: Peter Rosin <peda <at> lysator.liu.se>
Cc: Pavel Raiskup <praiskup <at> redhat.com>, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: Execution mode can hang pc when parameter refers to
	big file
Date: Tue, 29 Jan 2013 09:30:07 -0600 (CST)

On Tue, 29 Jan 2013, Peter Rosin wrote:

> Hi Bob,
>
> On 2013-01-17 16:29, Bob Friesenhahn wrote:
>> Everything would be easier if libtool would arrange to build a small helper
>> executable in the build tree which it can used instead of system tools.
>> Libtool already builds a native executable in some cases.  This would
>> diminish portability concerns and improve performance.  The main issue
>> with this is that for cross-builds, a system native compiler needs to be
>> used to build the helper executable.
>
> Forgive me for my ignorance, but when does libtool build something with
> the $build compiler in the cross case? The only thing I can find that
> looks like an alternative compiler is $LTCC, but that's just another
> name for the normal $host C compiler. What am I missing?

As I recall, libtool builds an execution test wrapper for a Windows 
native build.  It would (should) not do that for a cross-build. 
However, the C test wrapper was written using portable interfaces so 
it could compile under POSIX (even if we don't currently do that).

Autoconf/libtool in general does not seem to support the notion of 
supporting both a cross-compiler for the target and a native compiler 
for producing programs supporting the build, or remote/emulated 
testing of the "target".  If this hurdle was solved in a clean way, 
then many implementation opportunities would appear.

Bob
-- 
Bob Friesenhahn
bfriesen <at> simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Message #23 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Cc: Pavel Raiskup <praiskup <at> redhat.com>, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: Execution mode can hang pc when parameter refers to
	big file
Date: Tue, 29 Jan 2013 17:04:11 +0100

On 2013-01-29 16:30, Bob Friesenhahn wrote:
> On Tue, 29 Jan 2013, Peter Rosin wrote:
>> On 2013-01-17 16:29, Bob Friesenhahn wrote:
>>> Everything would be easier if libtool would arrange to build a small helper
>>> executable in the build tree which it can used instead of system tools.
>>> Libtool already builds a native executable in some cases.  This would
>>> diminish portability concerns and improve performance.  The main issue
>>> with this is that for cross-builds, a system native compiler needs to be
>>> used to build the helper executable.
>>
>> Forgive me for my ignorance, but when does libtool build something with
>> the $build compiler in the cross case? The only thing I can find that
>> looks like an alternative compiler is $LTCC, but that's just another
>> name for the normal $host C compiler. What am I missing?
> 
> As I recall, libtool builds an execution test wrapper for a Windows native
> build.  It would (should) not do that for a cross-build. However, the C
> test wrapper was written using portable interfaces so it could compile
> under POSIX (even if we don't currently do that).

The cwrapper compiles under POSIX because it is needed on Cygwin and
it compiles under Win32 because it is needed on MinGW. The cwrapper is
compiled with $LTCC, i.e. the standard $host C compiler.

> Autoconf/libtool in general does not seem to support the notion of
> supporting both a cross-compiler for the target and a native compiler for
> producing programs supporting the build, or remote/emulated testing of
> the "target".  If this hurdle was solved in a clean way, then many
> implementation opportunities would appear.

Agreed.

The conclusion is that there is no existing infrastructure for building
helper executables in libtool.

BTW, here's an alternative to "dd if="$1" bs=16K count=1"

	od -bvN 16K "$1" |
	sed 's/^[^ ]*//;/^ *$/d;s/  */\\\\/g' |
	while read l; do printf $l; done

Sadly "od -bvN" isn't portable, an alternative seems to be "hexdump -bvn".

Cheers,
Peter

Message #26 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Roumen Petrov <bugtrack <at> roumenpetrov.info>
To: 13472 <at> debbugs.gnu.org
Cc: Pavel Raiskup <praiskup <at> redhat.com>, Peter Rosin <peda <at> lysator.liu.se>,
	Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Subject: Re: bug#13472: Execution mode can hang pc when parameter refers to
	big file
Date: Tue, 29 Jan 2013 22:18:42 +0200

Bob Friesenhahn wrote:
> On Tue, 29 Jan 2013, Peter Rosin wrote:
>
>> Hi Bob,
>>
>> On 2013-01-17 16:29, Bob Friesenhahn wrote:
>>> Everything would be easier if libtool would arrange to build a small 
>>> helper
>>> executable in the build tree which it can used instead of system tools.
>>> Libtool already builds a native executable in some cases. This would
>>> diminish portability concerns and improve performance.  The main issue
>>> with this is that for cross-builds, a system native compiler needs 
>>> to be
>>> used to build the helper executable.
>>
>> Forgive me for my ignorance, but when does libtool build something with
>> the $build compiler in the cross case? The only thing I can find that
>> looks like an alternative compiler is $LTCC, but that's just another
>> name for the normal $host C compiler. What am I missing?
>
> As I recall, libtool builds an execution test wrapper for a Windows 
> native build.  It would (should) not do that for a cross-build. 
> However, the C test wrapper was written using portable interfaces so 
> it could compile under POSIX (even if we don't currently do that).

In windows case I would like to disagree. this is a way to run test in 
emulated environment. Build system run in emulator a native executable 
(libtool wrapper executable) , then wrapper set environment and spawn 
real executable.
This is specific for this case and know emulator. I would like to inform 
you that shell-wrapper from pre-libtool  2(.2?)  era can not be used in 
this case.
May be I misunderstand you idea .
Please detail you case.

> [SNIP]
>
> Bob

Roumen

Message #29 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: 13472 <at> debbugs.gnu.org
Cc: peda <at> lysator.liu.se, bfriesen <at> simple.dallas.tx.us
Subject: [PATCH 0/1] Execution mode can hang pc when parameter refers to big
	file
Date: Thu, 25 Apr 2013 13:48:34 +0200

Hi, sorry for the delay, here comes my (quite simple) proposal,

>> Autoconf/libtool in general does not seem to support the notion of
>> supporting both a cross-compiler for the target and a native compiler for
>> producing programs supporting the build, or remote/emulated testing of
>> the "target".  If this hurdle was solved in a clean way, then many
>> implementation opportunities would appear.
>
> Agreed.
>
> The conclusion is that there is no existing infrastructure for building
> helper executables in libtool.

this idea is not included, sadly - I'm unable to provide enough error
prone solution and design such architecture.

> BTW, here's an alternative to "dd if="$1" bs=16K count=1"
>
>    od -bvN 16K "$1" |
>    sed 's/^[^ ]*//;/^ *$/d;s/  */\\\\/g' |
>    while read l; do printf $l; done
>
> Sadly "od -bvN" isn't portable, an alternative seems to be "hexdump -bvn".

Thanks for this suggestion!

I was thinking about it and it looks that the 'dd' utility should be
enough portable for POSIX systems as it will be used with basic arguments.
And if the 'dd' is not found, no so big issue happens;  libtool has the
same behaviour as before — read the whole file.

Anyway, use of 'hexdump' or 'od' (which is at least on linux usually part
of coreutils (same as dd)) may easily be incorporated when necessary which
will cost just another one bash if-branch.

[PATCH] libtool: speed up by pre-cutting sed's input by dd

Pavel

Message #32 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: 13472 <at> debbugs.gnu.org
Cc: peda <at> lysator.liu.se, bfriesen <at> simple.dallas.tx.us
Subject: [PATCH] libtool: speed up by pre-cutting sed's input by dd
Date: Thu, 25 Apr 2013 13:48:35 +0200

The execute mode was too slow when any parameter of executed
command was a big binary file not having any newlines inside:

    $ libtool --mode=execute ls BIG_FILE_WITHOUT_NEWLINES

This was because of the lalib detection among parameters.  Every
such file is pre-filtered by sed uitlity.  That big slowdown
is there because the sed utility tries to read and cache
_unlimitedly_ the first four lines of these files.

The new approach is pre-cutting relevant files by dd utility at
4kB size.

* build-aux/ltmain.in (func_try_sizelim): New function.
(func_lalib_p): Cut the input file at 4kB before it is passed to
sed utility.
* gl/build-aux/funclib.sh: New variable $DD.
* m4/libtool.m4 (_LT_PROG_DD): New macro.
(_LT_SETUP): Require _LT_PROG_DD for the dd detection.
---
 build-aux/ltmain.in     | 14 +++++++++++++-
 gl/build-aux/funclib.sh |  1 +
 m4/libtool.m4           | 14 ++++++++++++++
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/build-aux/ltmain.in b/build-aux/ltmain.in
index 4c56b98..4a21b58 100644
--- a/build-aux/ltmain.in
+++ b/build-aux/ltmain.in
@@ -568,6 +568,18 @@ $1
 _LTECHO_EOF'
 }
 
+# func_try_sizelim file n
+# try to write at most the first N bytes from FILE to the standard output when
+# possible, otherwise put whole file
+func_try_sizelim ()
+{
+  if [ -n "$DD" ]; then
+    $DD if="$1" bs=$2 count=1 2>/dev/null
+  else
+    cat "$1"
+  fi
+}
+
 # func_lalib_p file
 # True iff FILE is a libtool '.la' library or '.lo' object file.
 # This function is only a basic sanity check; it will hardly flush out
@@ -575,7 +587,7 @@ _LTECHO_EOF'
 func_lalib_p ()
 {
     test -f "$1" &&
-      $SED -e 4q "$1" 2>/dev/null \
+        func_try_sizelim "$1" 4096 | $SED -e 4q 2>/dev/null \
         | $GREP "^# Generated by .*$PACKAGE" > /dev/null 2>&1
 }
 
diff --git a/gl/build-aux/funclib.sh b/gl/build-aux/funclib.sh
index 8a212d1..ff9680d 100644
--- a/gl/build-aux/funclib.sh
+++ b/gl/build-aux/funclib.sh
@@ -135,6 +135,7 @@ fi
 # in the command search PATH.
 
 : ${CP="cp -f"}
+: ${DD="$LTDD"}
 : ${ECHO="$bs_echo"}
 : ${EGREP="grep -E"}
 : ${FGREP="grep -F"}
diff --git a/m4/libtool.m4 b/m4/libtool.m4
index 3f50b0c..7fa3d9b 100644
--- a/m4/libtool.m4
+++ b/m4/libtool.m4
@@ -175,6 +175,7 @@ m4_require([_LT_CHECK_SHAREDLIB_FROM_LINKLIB])dnl
 m4_require([_LT_CMD_OLD_ARCHIVE])dnl
 m4_require([_LT_CMD_GLOBAL_SYMBOLS])dnl
 m4_require([_LT_WITH_SYSROOT])dnl
+m4_require([_LT_PROG_DD])dnl
 
 _LT_CONFIG_LIBTOOL_INIT([
 # See if we are running on zsh, and set the options which allow our
@@ -898,6 +899,19 @@ dnl AC_DEFUN([AC_LIBTOOL_GCJ], [])
 dnl AC_DEFUN([AC_LIBTOOL_RC], [])
 
 
+# _LT_PROG_DD
+# -----------
+# Try to find dd program and set the $LTDD variable to point to it, otherwise
+# set the $LTDD variable to be empty.
+m4_defun([_LT_PROG_DD],[
+m4_require([_LT_DECL_DLLTOOL])
+LTDD=dd
+$LTDD if=/dev/zero of=/dev/null bs=1 count=16K >/dev/null 2>/dev/null
+test $? -ne 0 && LTDD=
+_LT_DECL([], [LTDD], [1], [The dd program])dnl
+]) # _LT_PROG_DD
+
+
 # _LT_TAG_COMPILER
 # ----------------
 m4_defun([_LT_TAG_COMPILER],
-- 
1.8.1.4

Message #35 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: bug-libtool <at> gnu.org
Cc: peda <at> lysator.liu.se, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
	by dd
Date: Mon, 29 Apr 2013 08:37:49 +0200

Ping & update :),

Pavel

> +# func_try_sizelim file n
> +# try to write at most the first N bytes from FILE to the standard output when
> +# possible, otherwise put whole file
> +func_try_sizelim ()
> +{
> +  if [ -n "$DD" ]; then

Here should be 'test' rather than '['.

> +    $DD if="$1" bs=$2 count=1 2>/dev/null
> +  else
> +    cat "$1"

If the file is not readable (and possibly other?) ~> 2>/dev/null

> +  fi
> +}

I found that the first indentation is 4 spaces..

>  # func_lalib_p file
>  # True iff FILE is a libtool '.la' library or '.lo' object file.
>  # This function is only a basic sanity check; it will hardly flush out
> @@ -575,7 +587,7 @@ _LTECHO_EOF'
>  func_lalib_p ()
>  {
>      test -f "$1" &&
> -      $SED -e 4q "$1" 2>/dev/null \
> +        func_try_sizelim "$1" 4096 | $SED -e 4q 2>/dev/null \

Fixed indent.

>          | $GREP "^# Generated by .*$PACKAGE" > /dev/null 2>&1
>  }

------------------------------------------------------------------------------

From 38d952a1378e58655fba7cdb1e7f8edcd052e3ec Mon Sep 17 00:00:00 2001
From: Pavel Raiskup <praiskup <at> redhat.com>
Date: Thu, 25 Apr 2013 10:24:33 +0200
Subject: [PATCH] libtool: speed up by pre-cutting sed's input by dd

The execute mode was too slow when any parameter of executed
command was a big binary file not having any newlines inside:

    $ libtool --mode=execute ls BIG_FILE_WITHOUT_NEWLINES

This was because of the lalib detection among parameters.  Every
such file is pre-filtered by sed uitlity.  That big slowdown
is there because the sed utility tries to read and cache
_unlimitedly_ the first four lines of these files.

The new approach is pre-cutting relevant files by dd utility at
4kB size.

* build-aux/ltmain.in (func_try_sizelim): New function.
(func_lalib_p): Cut the input file at 4kB before it is passed to
sed utility.
* gl/build-aux/funclib.sh: New variable $DD.
* m4/libtool.m4 (_LT_PROG_DD): New macro.
(_LT_SETUP): Require _LT_PROG_DD for the dd detection.
---
 build-aux/ltmain.in     | 14 +++++++++++++-
 gl/build-aux/funclib.sh |  1 +
 m4/libtool.m4           | 14 ++++++++++++++
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/build-aux/ltmain.in b/build-aux/ltmain.in
index 4c56b98..60d7192 100644
--- a/build-aux/ltmain.in
+++ b/build-aux/ltmain.in
@@ -568,6 +568,18 @@ $1
 _LTECHO_EOF'
 }
 
+# func_try_sizelim file n
+# try to write at most the first N bytes from FILE to the standard output when
+# possible, otherwise put whole file
+func_try_sizelim ()
+{
+    if test -n "$DD"; then
+      $DD if="$1" bs=$2 count=1 2>/dev/null
+    else
+      cat "$1" 2>/dev/null
+    fi
+}
+
 # func_lalib_p file
 # True iff FILE is a libtool '.la' library or '.lo' object file.
 # This function is only a basic sanity check; it will hardly flush out
@@ -575,7 +587,7 @@ _LTECHO_EOF'
 func_lalib_p ()
 {
     test -f "$1" &&
-      $SED -e 4q "$1" 2>/dev/null \
+      func_try_sizelim "$1" 4096 | $SED -e 4q 2>/dev/null \
         | $GREP "^# Generated by .*$PACKAGE" > /dev/null 2>&1
 }
 
diff --git a/gl/build-aux/funclib.sh b/gl/build-aux/funclib.sh
index 8a212d1..ff9680d 100644
--- a/gl/build-aux/funclib.sh
+++ b/gl/build-aux/funclib.sh
@@ -135,6 +135,7 @@ fi
 # in the command search PATH.
 
 : ${CP="cp -f"}
+: ${DD="$LTDD"}
 : ${ECHO="$bs_echo"}
 : ${EGREP="grep -E"}
 : ${FGREP="grep -F"}
diff --git a/m4/libtool.m4 b/m4/libtool.m4
index 3f50b0c..7fa3d9b 100644
--- a/m4/libtool.m4
+++ b/m4/libtool.m4
@@ -175,6 +175,7 @@ m4_require([_LT_CHECK_SHAREDLIB_FROM_LINKLIB])dnl
 m4_require([_LT_CMD_OLD_ARCHIVE])dnl
 m4_require([_LT_CMD_GLOBAL_SYMBOLS])dnl
 m4_require([_LT_WITH_SYSROOT])dnl
+m4_require([_LT_PROG_DD])dnl
 
 _LT_CONFIG_LIBTOOL_INIT([
 # See if we are running on zsh, and set the options which allow our
@@ -898,6 +899,19 @@ dnl AC_DEFUN([AC_LIBTOOL_GCJ], [])
 dnl AC_DEFUN([AC_LIBTOOL_RC], [])
 
 
+# _LT_PROG_DD
+# -----------
+# Try to find dd program and set the $LTDD variable to point to it, otherwise
+# set the $LTDD variable to be empty.
+m4_defun([_LT_PROG_DD],[
+m4_require([_LT_DECL_DLLTOOL])
+LTDD=dd
+$LTDD if=/dev/zero of=/dev/null bs=1 count=16K >/dev/null 2>/dev/null
+test $? -ne 0 && LTDD=
+_LT_DECL([], [LTDD], [1], [The dd program])dnl
+]) # _LT_PROG_DD
+
+
 # _LT_TAG_COMPILER
 # ----------------
 m4_defun([_LT_TAG_COMPILER],
-- 
1.8.1.4

Message #41 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: bug-libtool <at> gnu.org
Cc: peda <at> lysator.liu.se, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
	by dd
Date: Mon, 13 May 2013 14:03:51 +0200

On Monday, April 29, 2013 08:37:49 Pavel Raiskup wrote:
> Ping & update :),

Another ping.  Is there some obvious problem with the patch?  Should I fix
something?  The 'dd' utility is checked during './configure' phase, and if
not found - whole file is read as it is now.

Can this somehow broke the cross compilation?  If yes, could you give me
an example (somehow reproducible).  Honestly, I don't see much into cross
compilation use-cases.

Thanks, Pavel

Message #47 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <pavel <at> raiskup.cz>
To: bug-libtool <at> gnu.org
Cc: peda <at> lysator.liu.se, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
	by dd
Date: Wed, 05 Jun 2013 10:20:16 +0200

On Monday, May 13, 2013 02:03:51 PM Pavel Raiskup wrote:
> On Monday, April 29, 2013 08:37:49 Pavel Raiskup wrote:
> > Ping & update :),
> 
> Another ping.

... and another, should I let it be?  Thanks, Pavel

Context catcher:
http://lists.gnu.org/archive/html/bug-libtool/2013-04/msg00009.html

Message #53 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: bug-libtool <at> gnu.org
Cc: Peter Rosin <peda <at> lysator.liu.se>, 13472 <at> debbugs.gnu.org,
 Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Subject: Re: bug#13472: Execution mode can hang pc when parameter refers to
 big file
Date: Tue, 14 Jan 2014 14:22:01 +0100

As the 'dd' [1] solution does not seem to be very popular, I would like to
discuss the 'native' compiler solution.  For getting a context of this
thread [2].

> Autoconf/libtool in general does not seem to support the notion of
> supporting both a cross-compiler for the target and a native compiler
> for producing programs supporting the build, or remote/emulated
> testing of the "target".  If this hurdle was solved in a clean way,
> then many implementation opportunities would appear.

If I understand it, we need something like LT_NATIVE_CC?  Is that correct?
If I tried to look at this, would this topic be an interesting for libtool
upstream (that means patch review/rejecting/discussing)?

Pavel

[1] http://article.gmane.org/gmane.comp.gnu.libtool.bugs/8161
[1] http://thread.gmane.org/gmane.comp.gnu.libtool.bugs/8113/focus=8130

Message #59 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Pavel Raiskup <pavel <at> raiskup.cz>, Nick Bowler <nbowler <at> draconx.ca>, 
 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Tue, 11 Feb 2014 18:46:26 +0100

[Message part 1 (text/plain, inline)]

On 2013-06-05 10:20, Pavel Raiskup wrote:
> On Monday, May 13, 2013 02:03:51 PM Pavel Raiskup wrote:
>> On Monday, April 29, 2013 08:37:49 Pavel Raiskup wrote:
>>> Ping & update :),
>>
>> Another ping.
> 
> ... and another, should I let it be?  Thanks, Pavel
> 
> Context catcher:
> http://lists.gnu.org/archive/html/bug-libtool/2013-04/msg00009.html

Ok Pavel, first of all, sorry for the "smallish" delay in handling
your request, and Nick, are you able to test this patch?

I didn't like some aspects of Pavels patch, it seemed convoluted and
the dd check didn't actually check if dd behaved as expected. But, I
rewrote the patch instead of describing in words exactly what I mean
and I hope I'm not stepping on any toes by that action... Anyway, this
has the dubious benefit that we do not have to wait for Pavel to assign
copyrights (I couldn't find him in any list).

Does my version work? The testsuite behaves OK for me, but other than
that I haven't tested.

Cheers,
Peter

[0001-libtool-speed-up-lalib-detection-in-execute-mode.patch (text/x-patch, attachment)]

Message #62 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> draconx.ca>
To: Peter Rosin <peda <at> lysator.liu.se>
Cc: Pavel Raiskup <pavel <at> raiskup.cz>, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Tue, 11 Feb 2014 14:15:37 -0500

On 2/11/14, Peter Rosin <peda <at> lysator.liu.se> wrote:
> On 2013-06-05 10:20, Pavel Raiskup wrote:
>> On Monday, May 13, 2013 02:03:51 PM Pavel Raiskup wrote:
>>> On Monday, April 29, 2013 08:37:49 Pavel Raiskup wrote:
>>>> Ping & update :),
>>>
>>> Another ping.
>>
>> ... and another, should I let it be?  Thanks, Pavel
>>
>> Context catcher:
>> http://lists.gnu.org/archive/html/bug-libtool/2013-04/msg00009.html
>
> Ok Pavel, first of all, sorry for the "smallish" delay in handling
> your request, and Nick, are you able to test this patch?
>
> I didn't like some aspects of Pavels patch, it seemed convoluted and
> the dd check didn't actually check if dd behaved as expected. But, I
> rewrote the patch instead of describing in words exactly what I mean
> and I hope I'm not stepping on any toes by that action... Anyway, this
> has the dubious benefit that we do not have to wait for Pavel to assign
> copyrights (I couldn't find him in any list).
>
> Does my version work? The testsuite behaves OK for me, but other than
> that I haven't tested.

Thanks both of you for looking at this.  I will test it tonight...  I
was also going to comment that I didn't really like the configure test
in Pavel's version.

Cheers,
  Nick

Message #65 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <pavel <at> raiskup.cz>
To: bug-libtool <at> gnu.org
Cc: Nick Bowler <nbowler <at> draconx.ca>, Peter Rosin <peda <at> lysator.liu.se>,
 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Wed, 12 Feb 2014 13:03:59 +0100

On Tuesday, February 11, 2014 18:46:26 Peter Rosin wrote:
> On 2013-06-05 10:20, Pavel Raiskup wrote:
> > On Monday, May 13, 2013 02:03:51 PM Pavel Raiskup wrote:
> >> On Monday, April 29, 2013 08:37:49 Pavel Raiskup wrote:
> >>> Ping & update :),
> >>
> >> Another ping.
> > 
> > ... and another, should I let it be?  Thanks, Pavel
> > 
> > Context catcher:
> > http://lists.gnu.org/archive/html/bug-libtool/2013-04/msg00009.html
> 
> Ok Pavel, first of all, sorry for the "smallish" delay in handling
> your request, and Nick, are you able to test this patch?

Peter, tested & it works for me, thanks a lot for looking at this!

> .. I rewrote the patch instead of describing in words exactly what I
> mean and I hope I'm not stepping on any toes by that action... Anyway,
> this has the dubious benefit that we do not have to wait for Pavel to
> assign copyrights (I couldn't find him in any list).

I'm completely OK with your patch.

Pavel

Message #71 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> draconx.ca>
To: Peter Rosin <peda <at> lysator.liu.se>
Cc: Pavel Raiskup <pavel <at> raiskup.cz>, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Wed, 12 Feb 2014 10:22:17 -0500

On 2/11/14, Peter Rosin <peda <at> lysator.liu.se> wrote:
> Ok Pavel, first of all, sorry for the "smallish" delay in handling
> your request, and Nick, are you able to test this patch?
[...]
> Does my version work? The testsuite behaves OK for me, but other than
> that I haven't tested.

Patch seems to work!

Thanks,
  Nick

Message #74 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Nick Bowler <nbowler <at> draconx.ca>
Cc: Pavel Raiskup <pavel <at> raiskup.cz>, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Thu, 13 Feb 2014 18:57:21 +0100

On 2014-02-12 16:22, Nick Bowler wrote:
> On 2/11/14, Peter Rosin <peda <at> lysator.liu.se> wrote:
>> Ok Pavel, first of all, sorry for the "smallish" delay in handling
>> your request, and Nick, are you able to test this patch?
> [...]
>> Does my version work? The testsuite behaves OK for me, but other than
>> that I haven't tested.
> 
> Patch seems to work!

Ok, everybody seems happy. But not me, not 100% anyway. This introduces
an extra fork, and AFAICT, the "extra protection" is only needed when
func_lalib_p is called from func_ltwrapper_script_p. Should we perhaps
have a separate implementation in func_ltwrapper_script_p instead of
simply calling func_lalib_p?

Maybe we could also lose the "sed -e 4q" part when dd limits the size?

I.e. keep func_lalib_p as before:

func_lalib_p ()
{
    test -f "$1" &&
      $SED -e 4q "$1" 2>/dev/null \
        | $GREP "^# Generated by .*$PACKAGE" > /dev/null 2>&1
}

but change func_ltwrapper_script_p from:

func_ltwrapper_script_p ()
{
    func_lalib_p "$1"
}

into:

func_ltwrapper_script_p ()
{
    test -f "$1" &&
      $lt_truncate_bin < "$1" 2>/dev/null \
        | $GREP "^# Generated by .*$PACKAGE" > /dev/null 2>&1
}

where lt_truncate_bin is either "/usr/bin/dd bs=4096 count=1" or
"$SED -e 4q" depending on the presence of a suitable dd.

That way we shave off a fork, which can't be bad...

Thoughts? Or should I just push out the prior patch?

Cheers,
Peter

Message #77 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Nick Bowler <nbowler <at> draconx.ca>
Cc: Pavel Raiskup <pavel <at> raiskup.cz>, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Thu, 13 Feb 2014 20:52:59 +0100

[Message part 1 (text/plain, inline)]

On 2014-02-13 18:57, Peter Rosin wrote:
> On 2014-02-12 16:22, Nick Bowler wrote:
>> On 2/11/14, Peter Rosin <peda <at> lysator.liu.se> wrote:
>>> Ok Pavel, first of all, sorry for the "smallish" delay in handling
>>> your request, and Nick, are you able to test this patch?
>> [...]
>>> Does my version work? The testsuite behaves OK for me, but other than
>>> that I haven't tested.
>>
>> Patch seems to work!
> 
> Ok, everybody seems happy. But not me, not 100% anyway. This introduces
> an extra fork, and AFAICT, the "extra protection" is only needed when
> func_lalib_p is called from func_ltwrapper_script_p. Should we perhaps
> have a separate implementation in func_ltwrapper_script_p instead of
> simply calling func_lalib_p?
> 
> Maybe we could also lose the "sed -e 4q" part when dd limits the size?

*snip* outline, real patch attached instead...

Now, which patch should I push?

The old [1]:
libtool: speed up lalib detection in execute mode

or the new?
libtool: speed up ltwrapper_script detection in execute mode

I like the latter better and will push that soonish. Unless...

Cheers,
Peter

[1] http://lists.gnu.org/archive/html/bug-libtool/2014-02/msg00005.html

[0001-libtool-speed-up-ltwrapper_script-detection-in-execu.patch (text/x-patch, attachment)]

Message #80 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> draconx.ca>
To: Peter Rosin <peda <at> lysator.liu.se>
Cc: Pavel Raiskup <pavel <at> raiskup.cz>, 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Fri, 14 Feb 2014 11:43:31 -0500

On 2/13/14, Peter Rosin <peda <at> lysator.liu.se> wrote:
> On 2014-02-13 18:57, Peter Rosin wrote:
>> Ok, everybody seems happy. But not me, not 100% anyway. This introduces
>> an extra fork, and AFAICT, the "extra protection" is only needed when
>> func_lalib_p is called from func_ltwrapper_script_p. Should we perhaps
>> have a separate implementation in func_ltwrapper_script_p instead of
>> simply calling func_lalib_p?
>>
>> Maybe we could also lose the "sed -e 4q" part when dd limits the size?
>
> *snip* outline, real patch attached instead...
>
> Now, which patch should I push?

The new looks a bit nicer to me although I have not tested it.

One comment I have that applies to both patches... you should probably
be using AC_CHECK_PROGS instead of AC_PATH_PROG because the latter
requires an absolute filename.

Specifically, AC_PATH_PROG will ignore any user-set value for DD if
it's not absolute.

Cheers,
  Nick

Message #83 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <pavel <at> raiskup.cz>
To: bug-libtool <at> gnu.org
Cc: Nick Bowler <nbowler <at> draconx.ca>, Peter Rosin <peda <at> lysator.liu.se>,
 13472 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Fri, 21 Feb 2014 15:03:24 +0100

Peter, sorry for the delay,

On Thursday, February 13, 2014 20:52:59 Peter Rosin wrote:
> On 2014-02-13 18:57, Peter Rosin wrote:
> > On 2014-02-12 16:22, Nick Bowler wrote:
> >> On 2/11/14, Peter Rosin <peda <at> lysator.liu.se> wrote:
> >>> Ok Pavel, first of all, sorry for the "smallish" delay in handling
> >>> your request, and Nick, are you able to test this patch?
> >> [...]
> >>> Does my version work? The testsuite behaves OK for me, but other than
> >>> that I haven't tested.
> >>
> >> Patch seems to work!
> > 
> > Ok, everybody seems happy. But not me, not 100% anyway. This introduces
> > an extra fork, and AFAICT, the "extra protection" is only needed when
> > func_lalib_p is called from func_ltwrapper_script_p. Should we perhaps
> > have a separate implementation in func_ltwrapper_script_p instead of
> > simply calling func_lalib_p?
> > 
> > Maybe we could also lose the "sed -e 4q" part when dd limits the size?

IMHO very good idea ^^^.  Cutting some line in the middle should not be a
problem after 4kB of characters, so why not.

> *snip* outline, real patch attached instead...
> 
> Now, which patch should I push?
> 
> The old [1]:
> libtool: speed up lalib detection in execute mode
> 
> or the new?
> libtool: speed up ltwrapper_script detection in execute mode
> 
> I like the latter better and will push that soonish. Unless...

I would use  something in the middle.  I really don't like code
duplication and looking at the two functions, the grep work is clearly
duplicated in [v2].  So as you proposed, use either 'dd' or 'sed'.  But
basically, I would take any of the two patches (as they work) rather than
make you to work on this too much - so I am OK if you choose what you like
better.

Thanks for your work,
Pavel

Message #89 received at 13472 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Nick Bowler <nbowler <at> draconx.ca>
Cc: Pavel Raiskup <pavel <at> raiskup.cz>, control <at> debbugs.gnu.org,
 13472 <at> debbugs.gnu.org, 16662 <at> debbugs.gnu.org
Subject: Re: bug#13472: [PATCH] libtool: speed up by pre-cutting sed's input
 by dd
Date: Tue, 06 May 2014 09:44:34 +0200

[Message part 1 (text/plain, inline)]

close 13472
close 16662
stop

On 2014-02-14 17:43, Nick Bowler wrote:
> On 2/13/14, Peter Rosin <peda <at> lysator.liu.se> wrote:
>> On 2014-02-13 18:57, Peter Rosin wrote:
>>> Ok, everybody seems happy. But not me, not 100% anyway. This introduces
>>> an extra fork, and AFAICT, the "extra protection" is only needed when
>>> func_lalib_p is called from func_ltwrapper_script_p. Should we perhaps
>>> have a separate implementation in func_ltwrapper_script_p instead of
>>> simply calling func_lalib_p?
>>>
>>> Maybe we could also lose the "sed -e 4q" part when dd limits the size?
>>
>> *snip* outline, real patch attached instead...
>>
>> Now, which patch should I push?
> 
> The new looks a bit nicer to me although I have not tested it.
> 
> One comment I have that applies to both patches... you should probably
> be using AC_CHECK_PROGS instead of AC_PATH_PROG because the latter
> requires an absolute filename.
> 
> Specifically, AC_PATH_PROG will ignore any user-set value for DD if
> it's not absolute.

Ok, I'm reviving this old thing once again. There were two minor complaints
with the last patch, this patch addresses both. I.e.

1. DD=dd now works to override the dd check, you don't need to specify
a full path.
2. The code duplication between func_lalib_p and func_ltwrapper_script_p
has been factored out.

However, to address the first point, I made use of the
AC_PATH_PROGS_FEATURE_CHECK macro, which was added in Autoconf 2.62, which
Libtool didn't require before. But 2.62 is 6 years old by now. If anyone
has problems with that, we will have to fix it later as I'm now pushing
this thing. It's been brewing for far too long already...

Final version attached.

Cheers,
Peter

[0001-libtool-speed-up-ltwrapper_script-detection-in-execu.patch (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.