GNU bug report logs - #19370
LT 2.4.4 regression (vs. 2.4.2)

From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "bug-libtool <at> gnu.org" <bug-libtool <at> gnu.org>
Subject: LT 2.4.4 regression (vs. 2.4.2)
Date: Sat, 13 Dec 2014 17:59:52 +0000

[Message part 1 (text/plain, inline)]
I have found what appears to be a regression in Libtool 2.4.4 vs. 2.4.2 (I did not test 2.4.3).

When embedding LT 2.4.4 libltdl in a larger project, the first time you invoke any "make" target (e.g., even "make clean"), "make" decides to run aclocal in the embedded libltdl directory.

This did not happen in LT 2.4.2 and earlier.

This behavior causes other problems in the Open MPI project, but I think the fact that "make" decides to invoke aclocal at all is the root cause.

Attached is a simple reproducer (if the tarball doesn't make it through anti-virus scanners, the same source is on github: https://github.com/jsquyres/libtool-2.4.4-bug):

- untar it
- run "./autogen.sh" (which runs autoreconf)
- run "./configure"
- run "make clean"

You'll see aclocal invoked in the embedded-libltdl directory.

The problem appears to be that make is checking for ../m4/libtool.m4 file as a dependency.  This file file -- and the entire ../m4 directory, for that matter -- does not exist.  So make decides to fire the "run the aclocal" rule.

In LT 2.4.2, make appears to check for m4/libtool.m4 (note the lack of ../), and somehow decides that even though this directory/file does not exist, it does not need to fire the "run the aclocal" rule.

Can someone have a look at this?  This behavior is preventing the Open MPI project from upgrading beyond Libtool 2.4.2.

Thank you!

-- 
Jeff Squyres
jsquyres <at> cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

[libtool-2.4.4-bug.tar.bz2 (application/x-bzip2, attachment)]
From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Fri, 19 Dec 2014 20:03:30 +0000

Any comments on this, perchance?

It's a blocker for us in the Open MPI project; it prevents us from upgrading from 2.4.2.

It's a bit of a problem because some software projects, such as mac-ports and home-brew are shipping LT >= 2.4.3.

On Dec 13, 2014, at 1:01 PM, GNU bug Tracking System <help-debbugs <at> gnu.org> wrote:

> Thank you for filing a new bug report with debbugs.gnu.org.
> 
> This is an automatically generated reply to let you know your message
> has been received.
> 
> Your message is being forwarded to the package maintainers and other
> interested parties for their attention; they will reply in due course.
> 
> Your message has been sent to the package maintainer(s):
> bug-libtool <at> gnu.org
> 
> If you wish to submit further information on this problem, please
> send it to 19370 <at> debbugs.gnu.org.
> 
> Please do not send mail to help-debbugs <at> gnu.org unless you wish
> to report a problem with the Bug-tracking system.
> 
> -- 
> 19370: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19370
> GNU Bug Tracking System
> Contact help-debbugs <at> gnu.org with problems

-- 
Jeff Squyres
jsquyres <at> cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

From: "Gary V. Vaughan" <gary <at> vaughan.pe>
To: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Fri, 19 Dec 2014 20:19:45 +0000

Hi Jeff,

I'm sorry, I didn't yet have chance to work on this... I'll try to reproduce it over the holidays,
and depending on whether that makes it obvious what's happening, a fix may or not be
straight forward and forthcoming.

It would certainly speed things along if you could help produce an analysis, a small self
contained reproducer, a test case and/or propose a patch.

Sorry I can't be of more help for the moment,
-- 
Gary V. Vaughan (gary AT vaughan DOT pe)

> On 19 Dec 2014, at 20:03, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
> 
> Any comments on this, perchance?
> 
> It's a blocker for us in the Open MPI project; it prevents us from upgrading from 2.4.2.
> 
> It's a bit of a problem because some software projects, such as mac-ports and home-brew are shipping LT >= 2.4.3.
> 
> 
>> On Dec 13, 2014, at 1:01 PM, GNU bug Tracking System <help-debbugs <at> gnu.org> wrote:
>> 
>> Thank you for filing a new bug report with debbugs.gnu.org.
>> 
>> This is an automatically generated reply to let you know your message
>> has been received.
>> 
>> Your message is being forwarded to the package maintainers and other
>> interested parties for their attention; they will reply in due course.
>> 
>> Your message has been sent to the package maintainer(s):
>> bug-libtool <at> gnu.org
>> 
>> If you wish to submit further information on this problem, please
>> send it to 19370 <at> debbugs.gnu.org.
>> 
>> Please do not send mail to help-debbugs <at> gnu.org unless you wish
>> to report a problem with the Bug-tracking system.
>> 
>> -- 
>> 19370: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19370
>> GNU Bug Tracking System
>> Contact help-debbugs <at> gnu.org with problems
> 
> 
> -- 
> Jeff Squyres
> jsquyres <at> cisco.com
> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> 
> 
> _______________________________________________
> Bug-libtool mailing list
> Bug-libtool <at> gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-libtool

From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "Gary V. Vaughan" <gary <at> vaughan.pe>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Sat, 20 Dec 2014 11:18:55 +0000

Thanks for replying, Gary. 

I did include what analysis I was able to do in my first email: I tracked down that the problem is that the "make" rules decide to invoke aclocal in the embedded libltdl because it's looking for non-existent files as dependencies (it looks like the wrong path is being used somehow?). I didn't go beyond that - I don't know the internals of libtool (this is a regression compared to 2.4.2). 

I also included a reproducer, both as a tarball and as a link to a github repo. 

Hopefully that's enough to get you going in the right direction. 

Sent from my phone. No type good. 

> On Dec 19, 2014, at 3:19 PM, Gary V. Vaughan <gary <at> vaughan.pe> wrote:
> 
> Hi Jeff,
> 
> I'm sorry, I didn't yet have chance to work on this... I'll try to reproduce it over the holidays,
> and depending on whether that makes it obvious what's happening, a fix may or not be
> straight forward and forthcoming.
> 
> It would certainly speed things along if you could help produce an analysis, a small self
> contained reproducer, a test case and/or propose a patch.
> 
> Sorry I can't be of more help for the moment,
> -- 
> Gary V. Vaughan (gary AT vaughan DOT pe)
> 
>> On 19 Dec 2014, at 20:03, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
>> 
>> Any comments on this, perchance?
>> 
>> It's a blocker for us in the Open MPI project; it prevents us from upgrading from 2.4.2.
>> 
>> It's a bit of a problem because some software projects, such as mac-ports and home-brew are shipping LT >= 2.4.3.
>> 
>> 
>>> On Dec 13, 2014, at 1:01 PM, GNU bug Tracking System <help-debbugs <at> gnu.org> wrote:
>>> 
>>> Thank you for filing a new bug report with debbugs.gnu.org.
>>> 
>>> This is an automatically generated reply to let you know your message
>>> has been received.
>>> 
>>> Your message is being forwarded to the package maintainers and other
>>> interested parties for their attention; they will reply in due course.
>>> 
>>> Your message has been sent to the package maintainer(s):
>>> bug-libtool <at> gnu.org
>>> 
>>> If you wish to submit further information on this problem, please
>>> send it to 19370 <at> debbugs.gnu.org.
>>> 
>>> Please do not send mail to help-debbugs <at> gnu.org unless you wish
>>> to report a problem with the Bug-tracking system.
>>> 
>>> -- 
>>> 19370: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19370
>>> GNU Bug Tracking System
>>> Contact help-debbugs <at> gnu.org with problems
>> 
>> 
>> -- 
>> Jeff Squyres
>> jsquyres <at> cisco.com
>> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Bug-libtool mailing list
>> Bug-libtool <at> gnu.org
>> https://lists.gnu.org/mailman/listinfo/bug-libtool

From: "Gary V. Vaughan" <gary <at> vaughan.pe>
To: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Mon, 22 Dec 2014 21:22:01 +0000

tags 19370 notabug
close 19730

Hi Jeff,

Sorry for the delay.

On Dec 20, 2014, at 11:18 AM, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
> 
> Thanks for replying, Gary. 

Even though I didn't read the original report carefully enough...

> I did include what analysis I was able to do in my first email: I tracked down that the problem is that the "make" rules decide to invoke aclocal in the embedded libltdl because it's looking for non-existent files as dependencies (it looks like the wrong path is being used somehow?).

...because you'd already included pretty much everything I asked for.

> I didn't go beyond that - I don't know the internals of libtool (this is a regression compared to 2.4.2). 
> 
> I also included a reproducer, both as a tarball and as a link to a github repo. 

Perfect!  So, even though your tarball does reproduce the bug you describe, I first converted it to a new autotest to protect against future reappearance of the bug, only to discover that inside the testsuite everything works as it should.  Hmm.

> Hopefully that's enough to get you going in the right direction. 

Indeed it is.  And the problem is that autoreconf, as called from the autogen.sh in the tarball, still runs the tools in the wrong order.  Autoreconf stupidly runs aclocal first, and then calls libtoolize which adds more m4 files to AC_CONFIG_MACRO_DIR, and that in turn causes aclocal.m4 to be out of date (because it needs to be regenerated to pick up the local versions of the libtoolize added m4 files added to ../config/ after it was first generated).

The bootstrap script in the libtool source tree fixes this (and many other problems with the autogen.sh/autoreconf approach), so if you care to write a bootstrap.conf (by copying and hacking nearly everything out of libtool-2.4.4/bootstrap.conf), things are then created in the right order and the bug disappears.

Alternatively, you can amend your autogen.sh to something like this:

  libtoolize --install --copy --ltdl
  LIBTOOLIZE=true autoreconf -fvi

If it worked for you in 2.4.2 in that order, then it was just a lucky combination of an empty local directory and installed versions of the macro files in the right place for aclocal.m4 to be valid on the initial too-early run.

In your original report, however, you said:

"The problem appears to be that make is checking for ../m4/libtool.m4 file as a dependency.  This file file -- and the entire ../m4 directory, for that matter -- does not exist.  So make decides to fire the "run the aclocal" rule."

...which seems odd to me, because for a subproject libltdl, the parent AC_CONFIG_MACRO_DIR/ACLOCAL_AMFLAGS directory is supposed to be merged in.  Did you mean to say "../config/libtool.m4" above?  If that substitution really isn't happening, then you've found a different bug - but I can't reproduce that one with 2.4.3, 2.4.4 nor current master.

HTH,
-- 
Gary V. Vaughan (gary AT vaughan DOT pe)

> Sent from my phone. No type good. 
> 
>> On Dec 19, 2014, at 3:19 PM, Gary V. Vaughan <gary <at> vaughan.pe> wrote:
>> 
>> Hi Jeff,
>> 
>> I'm sorry, I didn't yet have chance to work on this... I'll try to reproduce it over the holidays,
>> and depending on whether that makes it obvious what's happening, a fix may or not be
>> straight forward and forthcoming.
>> 
>> It would certainly speed things along if you could help produce an analysis, a small self
>> contained reproducer, a test case and/or propose a patch.
>> 
>> Sorry I can't be of more help for the moment,
>> -- 
>> Gary V. Vaughan (gary AT vaughan DOT pe)
>> 
>>> On 19 Dec 2014, at 20:03, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
>>> 
>>> Any comments on this, perchance?
>>> 
>>> It's a blocker for us in the Open MPI project; it prevents us from upgrading from 2.4.2.
>>> 
>>> It's a bit of a problem because some software projects, such as mac-ports and home-brew are shipping LT >= 2.4.3.
>>> 
>>> 
>>>> On Dec 13, 2014, at 1:01 PM, GNU bug Tracking System <help-debbugs <at> gnu.org> wrote:
>>>> 
>>>> Thank you for filing a new bug report with debbugs.gnu.org.
>>>> 
>>>> This is an automatically generated reply to let you know your message
>>>> has been received.
>>>> 
>>>> Your message is being forwarded to the package maintainers and other
>>>> interested parties for their attention; they will reply in due course.
>>>> 
>>>> Your message has been sent to the package maintainer(s):
>>>> bug-libtool <at> gnu.org
>>>> 
>>>> If you wish to submit further information on this problem, please
>>>> send it to 19370 <at> debbugs.gnu.org.
>>>> 
>>>> Please do not send mail to help-debbugs <at> gnu.org unless you wish
>>>> to report a problem with the Bug-tracking system.
>>>> 
>>>> -- 
>>>> 19370: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19370
>>>> GNU Bug Tracking System
>>>> Contact help-debbugs <at> gnu.org with problems
>>> 
>>> 
>>> -- 
>>> Jeff Squyres
>>> jsquyres <at> cisco.com
>>> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Bug-libtool mailing list
>>> Bug-libtool <at> gnu.org
>>> https://lists.gnu.org/mailman/listinfo/bug-libtool
> 
> 
> 
> _______________________________________________
> Bug-libtool mailing list
> Bug-libtool <at> gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-libtool

From: "Gary V. Vaughan" <gary <at> vaughan.pe>
To: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Mon, 22 Dec 2014 23:41:01 +0000

close  19370
reopen 19730

Just in case 19730 was mistakenly closed by my fat fingers earlier  :-/
-- 
Gary V. Vaughan (gary AT vaughan DOT pe)

> On 22 Dec 2014, at 21:22, Gary V. Vaughan <gary <at> vaughan.pe> wrote:
> 
> tags 19370 notabug
> close 19730
> 
> Hi Jeff,
> 
> Sorry for the delay.
> 
>> On Dec 20, 2014, at 11:18 AM, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
>> 
>> Thanks for replying, Gary.
> 
> Even though I didn't read the original report carefully enough...
> 
>> I did include what analysis I was able to do in my first email: I tracked down that the problem is that the "make" rules decide to invoke aclocal in the embedded libltdl because it's looking for non-existent files as dependencies (it looks like the wrong path is being used somehow?).
> 
> ...because you'd already included pretty much everything I asked for.
> 
>> I didn't go beyond that - I don't know the internals of libtool (this is a regression compared to 2.4.2). 
>> 
>> I also included a reproducer, both as a tarball and as a link to a github repo.
> 
> Perfect!  So, even though your tarball does reproduce the bug you describe, I first converted it to a new autotest to protect against future reappearance of the bug, only to discover that inside the testsuite everything works as it should.  Hmm.
> 
>> Hopefully that's enough to get you going in the right direction.
> 
> Indeed it is.  And the problem is that autoreconf, as called from the autogen.sh in the tarball, still runs the tools in the wrong order.  Autoreconf stupidly runs aclocal first, and then calls libtoolize which adds more m4 files to AC_CONFIG_MACRO_DIR, and that in turn causes aclocal.m4 to be out of date (because it needs to be regenerated to pick up the local versions of the libtoolize added m4 files added to ../config/ after it was first generated).
> 
> The bootstrap script in the libtool source tree fixes this (and many other problems with the autogen.sh/autoreconf approach), so if you care to write a bootstrap.conf (by copying and hacking nearly everything out of libtool-2.4.4/bootstrap.conf), things are then created in the right order and the bug disappears.
> 
> Alternatively, you can amend your autogen.sh to something like this:
> 
>  libtoolize --install --copy --ltdl
>  LIBTOOLIZE=true autoreconf -fvi
> 
> If it worked for you in 2.4.2 in that order, then it was just a lucky combination of an empty local directory and installed versions of the macro files in the right place for aclocal.m4 to be valid on the initial too-early run.
> 
> In your original report, however, you said:
> 
> "The problem appears to be that make is checking for ../m4/libtool.m4 file as a dependency.  This file file -- and the entire ../m4 directory, for that matter -- does not exist.  So make decides to fire the "run the aclocal" rule."
> 
> ...which seems odd to me, because for a subproject libltdl, the parent AC_CONFIG_MACRO_DIR/ACLOCAL_AMFLAGS directory is supposed to be merged in.  Did you mean to say "../config/libtool.m4" above?  If that substitution really isn't happening, then you've found a different bug - but I can't reproduce that one with 2.4.3, 2.4.4 nor current master.
> 
> HTH,
> -- 
> Gary V. Vaughan (gary AT vaughan DOT pe)
> 
>> Sent from my phone. No type good. 
>> 
>>> On Dec 19, 2014, at 3:19 PM, Gary V. Vaughan <gary <at> vaughan.pe> wrote:
>>> 
>>> Hi Jeff,
>>> 
>>> I'm sorry, I didn't yet have chance to work on this... I'll try to reproduce it over the holidays,
>>> and depending on whether that makes it obvious what's happening, a fix may or not be
>>> straight forward and forthcoming.
>>> 
>>> It would certainly speed things along if you could help produce an analysis, a small self
>>> contained reproducer, a test case and/or propose a patch.
>>> 
>>> Sorry I can't be of more help for the moment,
>>> -- 
>>> Gary V. Vaughan (gary AT vaughan DOT pe)
>>> 
>>>> On 19 Dec 2014, at 20:03, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
>>>> 
>>>> Any comments on this, perchance?
>>>> 
>>>> It's a blocker for us in the Open MPI project; it prevents us from upgrading from 2.4.2.
>>>> 
>>>> It's a bit of a problem because some software projects, such as mac-ports and home-brew are shipping LT >= 2.4.3.
>>>> 
>>>> 
>>>>> On Dec 13, 2014, at 1:01 PM, GNU bug Tracking System <help-debbugs <at> gnu.org> wrote:
>>>>> 
>>>>> Thank you for filing a new bug report with debbugs.gnu.org.
>>>>> 
>>>>> This is an automatically generated reply to let you know your message
>>>>> has been received.
>>>>> 
>>>>> Your message is being forwarded to the package maintainers and other
>>>>> interested parties for their attention; they will reply in due course.
>>>>> 
>>>>> Your message has been sent to the package maintainer(s):
>>>>> bug-libtool <at> gnu.org
>>>>> 
>>>>> If you wish to submit further information on this problem, please
>>>>> send it to 19370 <at> debbugs.gnu.org.
>>>>> 
>>>>> Please do not send mail to help-debbugs <at> gnu.org unless you wish
>>>>> to report a problem with the Bug-tracking system.
>>>>> 
>>>>> -- 
>>>>> 19370: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19370
>>>>> GNU Bug Tracking System
>>>>> Contact help-debbugs <at> gnu.org with problems
>>>> 
>>>> 
>>>> -- 
>>>> Jeff Squyres
>>>> jsquyres <at> cisco.com
>>>> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Bug-libtool mailing list
>>>> Bug-libtool <at> gnu.org
>>>> https://lists.gnu.org/mailman/listinfo/bug-libtool
>> 
>> 
>> 
>> _______________________________________________
>> Bug-libtool mailing list
>> Bug-libtool <at> gnu.org
>> https://lists.gnu.org/mailman/listinfo/bug-libtool
> 
> 
> 
> 
> _______________________________________________
> Bug-libtool mailing list
> Bug-libtool <at> gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-libtool

From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "Gary V. Vaughan" <gary <at> vaughan.pe>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Mon, 5 Jan 2015 21:30:48 +0000

On Dec 22, 2014, at 4:22 PM, Gary V. Vaughan <gary <at> vaughan.pe> wrote:

> Indeed it is.  And the problem is that autoreconf, as called from the autogen.sh in the tarball, still runs the tools in the wrong order. 

(first day back in the office today -- just seeing your reply now...)

Ah!  Ok.

>  Autoreconf stupidly runs aclocal first, and then calls libtoolize which adds more m4 files to AC_CONFIG_MACRO_DIR, and that in turn causes aclocal.m4 to be out of date (because it needs to be regenerated to pick up the local versions of the libtoolize added m4 files added to ../config/ after it was first generated).
> 
> The bootstrap script in the libtool source tree fixes this (and many other problems with the autogen.sh/autoreconf approach), so if you care to write a bootstrap.conf (by copying and hacking nearly everything out of libtool-2.4.4/bootstrap.conf), things are then created in the right order and the bug disappears.

That's a bummer.  We always thought that The Recommended Way to run the autootols was to use autoreconf.  Specifically: we used to have a magic incantation of a specific order of Autotools to bootstrap OMPI.  But that ordering was only "mostly" correct, meaning that upgrading Autotools sometimes broke it, because we didn't have the order exactly right...  My memory of the details is fuzzy here; I just remember it was a great relief when we trashed the whole thing and replaced it with a single invocation of autoreconf.

> Alternatively, you can amend your autogen.sh to something like this:
> 
>  libtoolize --install --copy --ltdl
>  LIBTOOLIZE=true autoreconf -fvi

Just to be clear: you're saying that I should invoke libtoolize *before* autoreconf, right?  (as opposed to appending those 2 lines at the end of my existing autogen.sh script)

I'm pretty sure that's what you're saying, and indeed, if I make my autogen.sh be this:

-----
$ more autogen.sh
#!/bin/sh

libtoolize --install --copy --ltdl
autoreconf -ivf --warnings=all,no-obsolete,no-override -I config
-----

...then the problem goes away.  Yay!

> If it worked for you in 2.4.2 in that order, then it was just a lucky combination of an empty local directory and installed versions of the macro files in the right place for aclocal.m4 to be valid on the initial too-early run.

Ever since we switched to invoking a single autoreconf (which was a loooong time ago; I'd have to go spelunking through history to find out when it was done, but it was probably on the order of years ago), we've not invoked libtoolize before autoreconf.

So just to be crystal clear: is the official guidance that we should run libtoolize and then autoreconf, and that should always work?

> In your original report, however, you said:
> 
> "The problem appears to be that make is checking for ../m4/libtool.m4 file as a dependency.  This file file -- and the entire ../m4 directory, for that matter -- does not exist.  So make decides to fire the "run the aclocal" rule."
> 
> ...which seems odd to me, because for a subproject libltdl, the parent AC_CONFIG_MACRO_DIR/ACLOCAL_AMFLAGS directory is supposed to be merged in.  Did you mean to say "../config/libtool.m4" above?  If that substitution really isn't happening, then you've found a different bug - but I can't reproduce that one with 2.4.3, 2.4.4 nor current master.

I don't remember, and since you can't reproduce it, let's assume that I made a user error and I really did mean "../config/libtool.m4".  :-)

-- 
Jeff Squyres
jsquyres <at> cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

From: "Gary V. Vaughan" <gary <at> vaughan.pe>
To: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Mon, 5 Jan 2015 22:10:56 +0000

Hi Jeff,

On Jan 5, 2015, at 9:30 PM, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
> 
> On Dec 22, 2014, at 4:22 PM, Gary V. Vaughan <gary <at> vaughan.pe> wrote:
> 
>> Indeed it is.  And the problem is that autoreconf, as called from the autogen.sh in the tarball, still runs the tools in the wrong order. 
> 
> (first day back in the office today -- just seeing your reply now...)
> 
> Ah!  Ok.
> 
>> Autoreconf stupidly runs aclocal first, and then calls libtoolize which adds more m4 files to AC_CONFIG_MACRO_DIR, and that in turn causes aclocal.m4 to be out of date (because it needs to be regenerated to pick up the local versions of the libtoolize added m4 files added to ../config/ after it was first generated).
>> 
>> The bootstrap script in the libtool source tree fixes this (and many other problems with the autogen.sh/autoreconf approach), so if you care to write a bootstrap.conf (by copying and hacking nearly everything out of libtool-2.4.4/bootstrap.conf), things are then created in the right order and the bug disappears.
> 
> That's a bummer.  We always thought that The Recommended Way to run the autootols was to use autoreconf.  Specifically: we used to have a magic incantation of a specific order of Autotools to bootstrap OMPI.  But that ordering was only "mostly" correct, meaning that upgrading Autotools sometimes broke it, because we didn't have the order exactly right...  My memory of the details is fuzzy here; I just remember it was a great relief when we trashed the whole thing and replaced it with a single invocation of autoreconf.

A bummer indeed.  I suppose this may very well be fixed in autoconf master... though I'm too lazy to check :-)

Getting the order right is a difficult error-prone process with hard to debug side-effects, so the fewer tools you have to invoke to do the bootstrap, the better.  And autoreconf is several fewer than aclocal/automake/autoconf, even though it calls libtoolize (and autopoint IIRC) too late! Of course, it gets out of hand fast when you have to run all the gettextize bits, and gnulib-tool and throw in some help2mans and the like :-(

>> Alternatively, you can amend your autogen.sh to something like this:
>> 
>> libtoolize --install --copy --ltdl
>> LIBTOOLIZE=true autoreconf -fvi
> 
> Just to be clear: you're saying that I should invoke libtoolize *before* autoreconf, right?  (as opposed to appending those 2 lines at the end of my existing autogen.sh script)
> 
> I'm pretty sure that's what you're saying, and indeed, if I make my autogen.sh be this:
> 
> -----
> $ more autogen.sh
> #!/bin/sh
> 
> libtoolize --install --copy --ltdl
> autoreconf -ivf --warnings=all,no-obsolete,no-override -I config
> -----
> 
> ...then the problem goes away.  Yay!

Pretty much, although without the LIBTOOLIZE=true setting before calling autoreconf, it will run wastefully run libtoolize a second time, which may or may not throw the timestamps out of sync again, depending how careful I was about preserving filestamps in generated files from the libtoolize code when their content does not change.  I'd recommend keeping that setting, just in case.

The crux of the matter is that if you run `aclocal -I m4` and then put more files that configure.ac calls out to into m4, then the next run of `aclocal -I m4` necessarily generates a new and different aclocal.m4 (with m4_includes for the new files replacing verbatim copies of the /usr/share/aclocal contents).

>> If it worked for you in 2.4.2 in that order, then it was just a lucky combination of an empty local directory and installed versions of the macro files in the right place for aclocal.m4 to be valid on the initial too-early run.
> 
> Ever since we switched to invoking a single autoreconf (which was a loooong time ago; I'd have to go spelunking through history to find out when it was done, but it was probably on the order of years ago), we've not invoked libtoolize before autoreconf.
> 
> So just to be crystal clear: is the official guidance that we should run libtoolize and then autoreconf, and that should always work?

Well, I hesitate to dub my word as "official"... but this is what my bootstrap script (and the gnulib bootstrap script) have been doing as a work-around for autoreconf brokenness for several years a piece with less weirdness than the olden days of trusting autoreconf.

Another option you have, should you worry about maintaining your own autogen.sh script to keep track of changes in upstream autotools dependencies and invocation ordering, is to use my bootstrap script (as used by libtool itself and m4 among others, and maintained separately at http://github.com/gvvaughan/bootstrap).  This nicely future-proofs you against upstream changes, or addition of internationalization or gnulib to your projects.

>> In your original report, however, you said:
>> 
>> "The problem appears to be that make is checking for ../m4/libtool.m4 file as a dependency.  This file file -- and the entire ../m4 directory, for that matter -- does not exist.  So make decides to fire the "run the aclocal" rule."
>> 
>> ...which seems odd to me, because for a subproject libltdl, the parent AC_CONFIG_MACRO_DIR/ACLOCAL_AMFLAGS directory is supposed to be merged in.  Did you mean to say "../config/libtool.m4" above?  If that substitution really isn't happening, then you've found a different bug - but I can't reproduce that one with 2.4.3, 2.4.4 nor current master.
> 
> I don't remember, and since you can't reproduce it, let's assume that I made a user error and I really did mean "../config/libtool.m4".  :-)

Agreed!

Cheers,
-- 
Gary V. Vaughan (gary AT gnu DOT org)

From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "Gary V. Vaughan" <gary <at> vaughan.pe>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Tue, 6 Jan 2015 11:44:18 +0000

On Jan 5, 2015, at 5:10 PM, Gary V. Vaughan <gary <at> vaughan.pe> wrote:

> A bummer indeed.  I suppose this may very well be fixed in autoconf master... though I'm too lazy to check :-)

:-)

Is this something that needs to be reported to the Autoconf devs, or is this already a known issue?

If it's not already a known issue, I'm guessing that Open MPI may not be the only project to run into this bug once other projects start upgrading to LT 2.4.4 (although, admittedly, there may not be many that embed libltdl).

>> Just to be clear: you're saying that I should invoke libtoolize *before* [snip]
> 
> Pretty much, although without the LIBTOOLIZE=true setting before calling autoreconf, it will run wastefully run libtoolize a second time, which may or may not throw the timestamps out of sync again, depending how careful I was about preserving filestamps in generated files from the libtoolize code when their content does not change.  I'd recommend keeping that setting, just in case.

Ahhh... I see.  I thought that the mailer had munged your previous mail and there were some quotes missing from your original suggestion.  Now I grok what you are suggesting: setting LIBTOOLIZE to effectively be a no-op so that autoreconf won't *actually* invoke libtoolize again.  Got it.

> The crux of the matter is that if you run `aclocal -I m4` and then put more files that configure.ac calls out to into m4, then the next run of `aclocal -I m4` necessarily generates a new and different aclocal.m4 (with m4_includes for the new files replacing verbatim copies of the /usr/share/aclocal contents).

Ok, I think I see.

> Another option you have, should you worry about maintaining your own autogen.sh script to keep track of changes in upstream autotools dependencies and invocation ordering, is to use my bootstrap script (as used by libtool itself and m4 among others, and maintained separately at http://github.com/gvvaughan/bootstrap).  This nicely future-proofs you against upstream changes, or addition of internationalization or gnulib to your projects.

That's a good suggestion; many thanks.

I'm a little hesitant to do it, however, simply because I'd prefer to get the Autotools fixed correctly such that autoreconf works properly.  That's (supposedly) the officially-recommended Way Of Doing Things, and it should work.  Perhaps that's naive, but I'd like to stick with the Office Way as much as possible.  As such, a 2-line workaround is much more attractive than a complicated non-office bootstrap script that we will potentially need to continually refresh from your github.

Make sense?

So I think the crux of this particular issue comes down to: do we need to report this to the Autoconf devs?

-- 
Jeff Squyres
jsquyres <at> cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

From: "Gary V. Vaughan" <gary <at> gnu.org>
To: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Tue, 6 Jan 2015 12:06:30 +0000

Hi Jeff,

On Jan 6, 2015, at 11:44 AM, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
> 
> Is this something that needs to be reported to the Autoconf devs, or is this already a known issue?

I don't really follow Autoconf development these days, but there's certainly no harm in reporting the issue upstream.  Worst case, it'll be marked as a duplicate and count as an upvote on what needs fixing next.

Cheers,
-- 
Gary V. Vaughan (gary AT gnu DOT org)

From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "Gary V. Vaughan" <gary <at> gnu.org>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Tue, 6 Jan 2015 16:57:03 +0000

On Jan 6, 2015, at 7:06 AM, Gary V. Vaughan <gary <at> gnu.org> wrote:

>> Is this something that needs to be reported to the Autoconf devs, or is this already a known issue?
> 
> I don't really follow Autoconf development these days, but there's certainly no harm in reporting the issue upstream.  Worst case, it'll be marked as a duplicate and count as an upvote on what needs fixing next.

Ok.

Sadly, however, your workaround was good enough for my toy example that I submitted, but it did not fix the issue in the Open MPI project (i.e., I run libtoolize .../LIBTOOLIZE=true autoreconf ..., but aclocal still decides to run in the libltdl directory, and Badness Ensues).

I'll have to investigate a little deeper and see what the difference is between my toy project and what Open MPI is doing.

-- 
Jeff Squyres
jsquyres <at> cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

From: "Gary V. Vaughan" <gary <at> gnu.org>
To: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Tue, 6 Jan 2015 17:33:57 +0000

Hi Jeff,

On Jan 6, 2015, at 4:57 PM, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
> 
> On Jan 6, 2015, at 7:06 AM, Gary V. Vaughan <gary <at> gnu.org> wrote:
> 
>>> Is this something that needs to be reported to the Autoconf devs, or is this already a known issue?
>> 
>> I don't really follow Autoconf development these days, but there's certainly no harm in reporting the issue upstream.  Worst case, it'll be marked as a duplicate and count as an upvote on what needs fixing next.
> 
> Ok.
> 
> Sadly, however, your workaround was good enough for my toy example that I submitted, but it did not fix the issue in the Open MPI project (i.e., I run libtoolize .../LIBTOOLIZE=true autoreconf ..., but aclocal still decides to run in the libltdl directory, and Badness Ensues).
> 
> I'll have to investigate a little deeper and see what the difference is between my toy project and what Open MPI is doing.

Even if you don't plan to adopt it, it might be instructive to write a bootstrap.conf for Open MPI and run my bootstrap in verbose mode to see what tools it runs and in what order, and whether that fixes the underlying issue.

It certainly looks like a dependency issue between the generated files, which may come from any of the autotools :-(

HTH,
-- 
Gary V. Vaughan (gary AT gnu DOT org)

From: Pavel Raiskup <praiskup <at> redhat.com>
To: bug-libtool <at> gnu.org
Cc: "Gary V. Vaughan" <gary <at> vaughan.pe>,
 "Jeff Squyres \(jsquyres\)" <jsquyres <at> cisco.com>,
 "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Fri, 16 Jan 2015 13:18:36 +0100

Hi Gary, quite long, sorry for that,

On Monday 22 of December 2014 21:22:01 Gary V. Vaughan wrote:
> tags 19370 notabug
> close 19730
>
> On Dec 20, 2014, at 11:18 AM, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
> > Hopefully that's enough to get you going in the right direction.
> 
> Indeed it is.  And the problem is that autoreconf, as called from the
> autogen.sh in the tarball, still runs the tools in the wrong order.
> Autoreconf stupidly runs aclocal first, and then calls libtoolize which
> adds more m4 files to AC_CONFIG_MACRO_DIR, and that in turn causes
> aclocal.m4 to be out of date (because it needs to be regenerated to pick
> up the local versions of the libtoolize added m4 files added to
> ../config/ after it was first generated).

actually, (at least modern enough) autoreconf runs the aclocal twice.
Once before libtoolize call (do detect whether it should call the
libtoolize tool at all) and second time [1] after libtoolize to
incorporate the macros.

I'd like to say in advance that IMO this should be fixed in libtoolize,
somehow, reasoning follows.

- autoreconf is really generally believed to be the "tool #1" for
  regenerating GNU buildsystem .. if we don't take the 'gnulib' or others
  into account of course;  then however bootstrap comes and autoreconf
  -vfi should still be OK

- thus either libtool or autoreconf should be fixed, ..

- when libltdl is included into project as 'convenience' library, it is
  treated like "subproject".  Then running autoreconf from project's
  $(top_srcdir) treats subprojects independently by recursing down into
  subdirectory (in our case libltdl) and autoreconfing there.

  The child autoreconf does not expect that something defined in parent
  directory will touch subdirectories after successful autoreconf.

  When parent autoreconf finished with autoreconfing of subdirectory, it
  continues in cwd - and that includes running libtoolize; _however_, this
  second run of libtoolize from top-level overwrites the _common_ macros.
  Now, the macros are more recent than the "subdirectory thinks".

  After that ^^^, the aclocal tool is run in top-level directory - but it
  has no idea about recursing down into (otherwise independent)
  subdirectory.  So everything happens as expected, two separate
  directories were autoreconfed independently.

> The bootstrap script in the libtool source tree fixes this (and many
> other problems with the autogen.sh/autoreconf approach), so if you care
> to write a bootstrap.conf (by copying and hacking nearly everything out
> of libtool-2.4.4/bootstrap.conf), things are then created in the right
> order and the bug disappears.
> 
> Alternatively, you can amend your autogen.sh to something like this:
> 
>   libtoolize --install --copy --ltdl
>   LIBTOOLIZE=true autoreconf -fvi

Yes, this work-arounds that.  Thanks!

> If it worked for you in 2.4.2 in that order, then it was just a lucky
> combination of an empty local directory and installed versions of the
> macro files in the right place for aclocal.m4 to be valid on the initial
> too-early run.

Hmm, I think sharing the macros with top-level is clear trigger here.

> In your original report, however, you said:
> 
> "The problem appears to be that make is checking for ../m4/libtool.m4
> file as a dependency.  This file file -- and the entire ../m4 directory,
> for that matter -- does not exist.  So make decides to fire the "run the
> aclocal" rule."
>
> ...which seems odd to me, because for a subproject libltdl, the parent
> AC_CONFIG_MACRO_DIR/ACLOCAL_AMFLAGS directory is supposed to be merged
> in.  Did you mean to say "../config/libtool.m4" above?  If that
> substitution really isn't happening, then you've found a different bug -
> but I can't reproduce that one with 2.4.3, 2.4.4 nor current master.

Reproduced.  I'll try to make automatized reproducer and post later
possibly.

To make it clear:  The top-level libtoolize re-initializes the
sub-directory macros without runing autoreconf.  Makefile.in _is already_
generated (distributed via libtool) and put on place.  The Makefile.in
distributed with libtool contains:

  am__aclocal_m4_deps = $(top_srcdir)/../m4/libtool.m4 \
  >-------$(top_srcdir)/../m4/ltargz.m4 $(top_srcdir)/../m4/ltdl.m4 \
  >-------$(top_srcdir)/../m4/ltoptions.m4 \
  >-------$(top_srcdir)/../m4/ltsugar.m4 \
  >-------$(top_srcdir)/../m4/ltversion.m4 \
  >-------$(top_srcdir)/../m4/lt~obsolete.m4 $(top_srcdir)/configure.ac

Without regenerating Makefile.in, bad things happen if user has
non-default AC_CONFIG_MACRO_DIR.  Also, proposed fix [2] is kind of
related.

-------

For the solution, I was thinking about something like detecting whether
libtoolize is run from $(srcdir) or not.  Something like:

  --- a/libtoolize.in
  +++ b/libtoolize.in
  @@ -898,6 +898,8 @@ func_install_pkgltdl_files ()
       $require_ltdl_dir
       $require_ltdl_mode

  +    test '.' = "$ltdl_dir" || return
  +
       # Remove any lingering files that my have been installed by some
       # previous libtoolize release:
       $opt_force && for file in $all_pkgltdl_files; do

.. or turning maintainer mode on in libltdl?  Documenting autoreconf-ing by
hand in subdirectory is also an option (after top-dir is done) but that is
counter-intuitive without 'autoreconf --no-recursive'.

That all because it seems to me that hacking this in autoreconf is (a) imo
too difficult and (b) too late as libtool-2.4.4 is out.

[1] http://git.savannah.gnu.org/cgit/autoconf.git/tree/bin/autoreconf.in?id=7b13e39a112309786ebb2fdb76e027b7eaa4f2f5#n486
[2] http://news.gmane.org/gmane.comp.gnu.libtool.patches

Pavel

From: "Gary V. Vaughan" <gary <at> vaughan.pe>
To: Pavel Raiskup <praiskup <at> redhat.com>
Cc: jsquyres <at> cisco.com, 19370 <at> debbugs.gnu.org
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Fri, 16 Jan 2015 13:26:00 +0000

> On Jan 16, 2015, at 12:18 PM, Pavel Raiskup <praiskup <at> redhat.com> wrote:
> 
> Hi Gary, quite long, sorry for that,

Hi Pavel,

Not in the least, the issue is convoluted, and I'd rather have long and
clear than terse and ambiguous! :-)  Thank you for making the time to
analyse the issue and propose some solutions!

> On Monday 22 of December 2014 21:22:01 Gary V. Vaughan wrote:
>> tags 19370 notabug
>> close 19730
>> 
>> On Dec 20, 2014, at 11:18 AM, Jeff Squyres (jsquyres) <jsquyres <at> cisco.com> wrote:
>>> Hopefully that's enough to get you going in the right direction.
>> 
>> Indeed it is.  And the problem is that autoreconf, as called from the
>> autogen.sh in the tarball, still runs the tools in the wrong order.
>> Autoreconf stupidly runs aclocal first, and then calls libtoolize which
>> adds more m4 files to AC_CONFIG_MACRO_DIR, and that in turn causes
>> aclocal.m4 to be out of date (because it needs to be regenerated to pick
>> up the local versions of the libtoolize added m4 files added to
>> ../config/ after it was first generated).
> 
> actually, (at least modern enough) autoreconf runs the aclocal twice.
> Once before libtoolize call (do detect whether it should call the
> libtoolize tool at all) and second time [1] after libtoolize to
> incorporate the macros.

That's good to know.  I stopped closely following autoconf development a
few years ago, and didn't realise this was finally cleaned up.  Some of
these corner case may be because of my slightly out of date view of how
these tools interact :-(

> I'd like to say in advance that IMO this should be fixed in libtoolize,
> somehow, reasoning follows.
> 
> - autoreconf is really generally believed to be the "tool #1" for
>  regenerating GNU buildsystem .. if we don't take the 'gnulib' or others
>  into account of course;  then however bootstrap comes and autoreconf
>  -vfi should still be OK

Agreed.

> - thus either libtool or autoreconf should be fixed, ..
> 
> - when libltdl is included into project as 'convenience' library, it is
>  treated like "subproject".  Then running autoreconf from project's
>  $(top_srcdir) treats subprojects independently by recursing down into
>  subdirectory (in our case libltdl) and autoreconfing there.

Not always.  There is the full subproject mode, but also recursive and
nonrecursive modes that hooks into the parent project's configure.ac (and
Makefile.am in the latter case).

Is it only subproject libltdl (the one with its own configure.ac) that
exhibits this bug?

>  The child autoreconf does not expect that something defined in parent
>  directory will touch subdirectories after successful autoreconf.
> 
>  When parent autoreconf finished with autoreconfing of subdirectory, it
>  continues in cwd - and that includes running libtoolize; _however_, this
>  second run of libtoolize from top-level overwrites the _common_ macros.
>  Now, the macros are more recent than the "subdirectory thinks".
> 
>  After that ^^^, the aclocal tool is run in top-level directory - but it
>  has no idea about recursing down into (otherwise independent)
>  subdirectory.  So everything happens as expected, two separate
>  directories were autoreconfed independently.

Eventually (and by "eventually" I mean I've been planning for a long time,
but haven't got to it yet) I'd like to get rid of the whole concept of
carrying a local libltdl.  Libtool is now pervasive enough in the ecosystem,
that it would be massively more straight forward if libltdl were just treated
like any other dependency... i.e. if your project needs it, just install the
proper version on your system first.

We've long ignored the problem of what happens when linking, say, libm4.so
with an internal copy of libltdl.so into an application that also uses the
system libltdl.so, or worse wants to link with it's own subproject version.
I'm sure distros have to jump through some hoops to make all of these clients
simply agree on the system libltdl.so, which really is the only sane solution.
The various subproject modes are an outdated idea from 20 years ago when
libltdl didn't have any traction yet.

[[snip]]

I think I'll try to release Libtool 2.4.5 later today, and then hack all the
multi-mode libltdl subproject support out entirely for the next release, which
neatly sidesteps the trickle of bugs caused all the little subtle interactions
between the various generated files.

Cheers,
-- 
Gary V. Vaughan (gary AT gnu DOT org)

From: Pavel Raiskup <praiskup <at> redhat.com>
To: "Gary V. Vaughan" <gary <at> vaughan.pe>
Cc: jsquyres <at> cisco.com, 19370 <at> debbugs.gnu.org
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Fri, 16 Jan 2015 15:34:54 +0100

On Friday 16 of January 2015 13:26:00 Gary V. Vaughan wrote:
> > On Monday 22 of December 2014 21:22:01 Gary V. Vaughan wrote:
> > - when libltdl is included into project as 'convenience' library, it is
> >  treated like "subproject".  Then running autoreconf from project's
> >  $(top_srcdir) treats subprojects independently by recursing down into
> >  subdirectory (in our case libltdl) and autoreconfing there.
> 
> Not always.  There is the full subproject mode, but also recursive and
> nonrecursive modes that hooks into the parent project's configure.ac (and
> Makefile.am in the latter case).
> 
> Is it only subproject libltdl (the one with its own configure.ac) that
> exhibits this bug?

I bet this particular bug yes.  But that is something I can not say
surely.  Switching to LTDL_INIT([nonrecursive]) works fine for me.

> >  The child autoreconf does not expect that something defined in parent
> >  directory will touch subdirectories after successful autoreconf.
> > 
> >  When parent autoreconf finished with autoreconfing of subdirectory, it
> >  continues in cwd - and that includes running libtoolize; _however_, this
> >  second run of libtoolize from top-level overwrites the _common_ macros.
> >  Now, the macros are more recent than the "subdirectory thinks".
> > 
> >  After that ^^^, the aclocal tool is run in top-level directory - but it
> >  has no idea about recursing down into (otherwise independent)
> >  subdirectory.  So everything happens as expected, two separate
> >  directories were autoreconfed independently.
> 
> Eventually (and by "eventually" I mean I've been planning for a long time,
> but haven't got to it yet) I'd like to get rid of the whole concept of
> carrying a local libltdl.  Libtool is now pervasive enough in the ecosystem,
> that it would be massively more straight forward if libltdl were just treated
> like any other dependency... i.e. if your project needs it, just install the
> proper version on your system first.
>
> We've long ignored the problem of what happens when linking, say, libm4.so
> with an internal copy of libltdl.so into an application that also uses the
> system libltdl.so, or worse wants to link with it's own subproject version.
> I'm sure distros have to jump through some hoops to make all of these clients
> simply agree on the system libltdl.so, which really is the only sane solution.
> The various subproject modes are an outdated idea from 20 years ago when
> libltdl didn't have any traction yet.
>
> [[snip]]
> 
> I think I'll try to release Libtool 2.4.5 later today, and then hack all the
> multi-mode libltdl subproject support out entirely for the next release, which
> neatly sidesteps the trickle of bugs caused all the little subtle interactions
> between the various generated files.

TBH, this is something I have no strong opinion on.  Seems like there is
no big benefit at least for GNU/Linux.  Maybe going through some
obsoleting process could help .. but I need to study more to discuss.

Pavel

From: "Gary V. Vaughan" <gary <at> vaughan.pe>
To: Autoconf Bugs List <bug-autoconf <at> gnu.org>
Cc: Pavel Raiskup <praiskup <at> redhat.com>, Jeff Squyres <jsquyres <at> cisco.com>,
 19370 <at> debbugs.gnu.org
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Fri, 16 Jan 2015 18:49:12 +0000

[[Adding Autoconf List, the home of autoreconf]]

On Jan 16, 2015, at 1:26 PM, Gary V. Vaughan <gary <at> vaughan.pe> wrote:
>>> And the problem is that autoreconf, as called from the
>>> autogen.sh in the tarball, still runs the tools in the wrong order.
>>> Autoreconf stupidly runs aclocal first, and then calls libtoolize which
>>> adds more m4 files to AC_CONFIG_MACRO_DIR, and that in turn causes
>>> aclocal.m4 to be out of date (because it needs to be regenerated to pick
>>> up the local versions of the libtoolize added m4 files added to
>>> ../config/ after it was first generated).
>> 
>> actually, (at least modern enough) autoreconf runs the aclocal twice.
>> Once before libtoolize call (do detect whether it should call the
>> libtoolize tool at all) and second time [1] after libtoolize to
>> incorporate the macros.
> 
> That's good to know.  I stopped closely following autoconf development a
> few years ago, and didn't realise this was finally cleaned up.  Some of
> these corner case may be because of my slightly out of date view of how
> these tools interact :-(

Now that I think about it, why is it necessary to run aclocal just to
find out whether LT_INIT, AM_PROG_LIBTOOL or AC_PROG_LIBTOOL is invoked?
I know that for a full m4 --trace run, one needs to have some (possibly
outdated) versions of required macros available, but it's very easy to
work around that: see the implementation of `func_require_libtoolize` in
the libtool bootstrap script (my clean rewrite of the gnulib bootstrap
script).

Further, now that autoconf is actively maintained again, why do we have
a vestigial autoreconf and a whole zoo of autogen.sh and bootstrap scripts?
Wouldn't it make more sense to centralize and maintain all of this in the
one true autoreconf?  Merging my bootstrap script with the latest autoreconf
eliminates the spurious rerun of aclocal, and brings support for gettext,
gnulib and per-project customizations.

It doesn't seem like a great deal of work to translate 700 lines of perl
into shell, fold it into bootstrap, and ship the result as autoreconf in the
next release of Autoconf.  Then everyone can go back to running `autoreconf
-fvi` and be done with the whole mess of wrapper scripts...

Cheers,
-- 
Gary V. Vaughan (gary AT gnu DOT org)

From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "Gary V. Vaughan" <gary <at> vaughan.pe>
Cc: Pavel Raiskup <praiskup <at> redhat.com>,
 "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Sat, 17 Jan 2015 03:23:47 +0000

On Jan 16, 2015, at 8:26 AM, Gary V. Vaughan <gary <at> vaughan.pe> wrote:
> 
> I think I'll try to release Libtool 2.4.5 later today, and then hack all the
> multi-mode libltdl subproject support out entirely for the next release, which
> neatly sidesteps the trickle of bugs caused all the little subtle interactions
> between the various generated files.

Is it really true that Linux systems all install libltdl-devel by default these days?

Without it, Open MPI can't compile its libltdl support, which is pretty key.

I ask because most people will install a Linux distro and then go grab the latest Open MPI tarball and build it from source (because OMPI releases far faster than distros, and we don't distribute binaries).

Plugins are pretty critical to Open MPI... that's why we embed libltdl.

It would be pretty bad for us if you prevent us from embedding libltdl.  We would be very, very sad pandas.  :-(

-- 
Jeff Squyres
jsquyres <at> cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "Gary V. Vaughan" <gary <at> vaughan.pe>
Cc: Pavel Raiskup <praiskup <at> redhat.com>,
 "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Sat, 17 Jan 2015 03:48:57 +0000

To be clear...

I'd still really like to embed libltdl in some way in OMPI.

If it's a different way than we've been doing before, that's fine.

E.g., if you give me a simple procedure to follow instead of using the magic m4 macros, that would also be fine.  I.e., something that would reduce your support burden by making it less of a general case for everyone to use (i.e., reduce that steady trickle of bugs you mentioned).  I'd even be fine if the bulk of libltdl's configure.ac script moved into a .m4 that I could call from my own, top-level configure.m4 (and I could copy all the relevant libltdl .m4 files into my own m4 macro directory).  That would be great/easy, for example.

...or any other solution that would be easy(ier) for you to keep maintaining some form of libltdl embedding option.

-- 
Jeff Squyres
jsquyres <at> cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

From: Sergei Steshenko <sergstesh <at> yahoo.com>
To: "Gary V. Vaughan" <gary <at> vaughan.pe>, 
 Autoconf Bugs List <bug-autoconf <at> gnu.org>
Cc: Pavel Raiskup <praiskup <at> redhat.com>, Jeff Squyres <jsquyres <at> cisco.com>,
 "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Sat, 17 Jan 2015 03:58:54 +0000 (UTC)

>________________________________
> From: Gary V. Vaughan <gary <at> vaughan.pe>
>To: Autoconf Bugs List <bug-autoconf <at> gnu.org> 
>Cc: Pavel Raiskup <praiskup <at> redhat.com>; Jeff Squyres <jsquyres <at> cisco.com>; 19370 <at> debbugs.gnu.org 
>Sent: Friday, January 16, 2015 8:49 PM
>Subject: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
> 
[snip]
>It doesn't seem like a great deal of work to translate 700 lines of perl
>into shell, fold it into bootstrap, and ship the result as autoreconf in the
>next release of Autoconf.  Then everyone can go back to running `autoreconf
>-fvi` and be done with the whole mess of wrapper scripts...
>
>
>
>
>
>Cheers,
>-- 
>Gary V. Vaughan (gary AT gnu DOT org)
>
>

Why to do this in the first place ?

At all, why to rely on shell scripting when Perl compiles on a gazillion of platforms and doesn't even require root privileges in order to be installed ? Furthermore, Perl is portable in the sense that properly compiled Perl tree can be moved to another location and it will still work - it is by design

--Sergei.

From: "Jeff Squyres (jsquyres)" <jsquyres <at> cisco.com>
To: "Gary V. Vaughan" <gary <at> gnu.org>
Cc: "19370 <at> debbugs.gnu.org" <19370 <at> debbugs.gnu.org>
Subject: Re: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))
Date: Tue, 10 Mar 2015 19:07:40 +0000

On Jan 6, 2015, at 12:33 PM, Gary V. Vaughan <gary <at> gnu.org> wrote:
> 
> It certainly looks like a dependency issue between the generated files, which may come from any of the autotools :-(

To close this chapter: this issue became a nightmare for me/Open MPI.  After trying a lot of different things and experiencing a lot of pain, we have stopped embedding libltdl in Open MPI on our git master, which will eventually translate into the Open MPI v1.9.0 release someday (see https://github.com/open-mpi/ompi/issues/311 and https://github.com/open-mpi/ompi/pull/410 if you care for the gory details).

-- 
Jeff Squyres
jsquyres <at> cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

GNU bug report logs - #19370 LT 2.4.4 regression (vs. 2.4.2)

GNU bug report logs - #19370
LT 2.4.4 regression (vs. 2.4.2)