Subject: bug#19370: Acknowledgement (LT 2.4.4 regression (vs. 2.4.2))

From: "Gary V. Vaughan" <[email-masked]>
Date: Mon, 5 Jan 2015 22:10:56 +0000

Hi Jeff,

On Jan 5, 2015, at 9:30 PM, Jeff Squyres (jsquyres) <[email-masked]> wrote:
> 
> On Dec 22, 2014, at 4:22 PM, Gary V. Vaughan <[email-masked]> wrote:
> 
>> Indeed it is.  And the problem is that autoreconf, as called from the autogen.sh in the tarball, still runs the tools in the wrong order. 
> 
> (first day back in the office today -- just seeing your reply now...)
> 
> Ah!  Ok.
> 
>> Autoreconf stupidly runs aclocal first, and then calls libtoolize which adds more m4 files to AC_CONFIG_MACRO_DIR, and that in turn causes aclocal.m4 to be out of date (because it needs to be regenerated to pick up the local versions of the libtoolize added m4 files added to ../config/ after it was first generated).
>> 
>> The bootstrap script in the libtool source tree fixes this (and many other problems with the autogen.sh/autoreconf approach), so if you care to write a bootstrap.conf (by copying and hacking nearly everything out of libtool-2.4.4/bootstrap.conf), things are then created in the right order and the bug disappears.
> 
> That's a bummer.  We always thought that The Recommended Way to run the autootols was to use autoreconf.  Specifically: we used to have a magic incantation of a specific order of Autotools to bootstrap OMPI.  But that ordering was only "mostly" correct, meaning that upgrading Autotools sometimes broke it, because we didn't have the order exactly right...  My memory of the details is fuzzy here; I just remember it was a great relief when we trashed the whole thing and replaced it with a single invocation of autoreconf.

A bummer indeed.  I suppose this may very well be fixed in autoconf master... though I'm too lazy to check :-)

Getting the order right is a difficult error-prone process with hard to debug side-effects, so the fewer tools you have to invoke to do the bootstrap, the better.  And autoreconf is several fewer than aclocal/automake/autoconf, even though it calls libtoolize (and autopoint IIRC) too late! Of course, it gets out of hand fast when you have to run all the gettextize bits, and gnulib-tool and throw in some help2mans and the like :-(

>> Alternatively, you can amend your autogen.sh to something like this:
>> 
>> libtoolize --install --copy --ltdl
>> LIBTOOLIZE=true autoreconf -fvi
> 
> Just to be clear: you're saying that I should invoke libtoolize *before* autoreconf, right?  (as opposed to appending those 2 lines at the end of my existing autogen.sh script)
> 
> I'm pretty sure that's what you're saying, and indeed, if I make my autogen.sh be this:
> 
> -----
> $ more autogen.sh
> #!/bin/sh
> 
> libtoolize --install --copy --ltdl
> autoreconf -ivf --warnings=all,no-obsolete,no-override -I config
> -----
> 
> ...then the problem goes away.  Yay!

Pretty much, although without the LIBTOOLIZE=true setting before calling autoreconf, it will run wastefully run libtoolize a second time, which may or may not throw the timestamps out of sync again, depending how careful I was about preserving filestamps in generated files from the libtoolize code when their content does not change.  I'd recommend keeping that setting, just in case.

The crux of the matter is that if you run `aclocal -I m4` and then put more files that configure.ac calls out to into m4, then the next run of `aclocal -I m4` necessarily generates a new and different aclocal.m4 (with m4_includes for the new files replacing verbatim copies of the /usr/share/aclocal contents).

>> If it worked for you in 2.4.2 in that order, then it was just a lucky combination of an empty local directory and installed versions of the macro files in the right place for aclocal.m4 to be valid on the initial too-early run.
> 
> Ever since we switched to invoking a single autoreconf (which was a loooong time ago; I'd have to go spelunking through history to find out when it was done, but it was probably on the order of years ago), we've not invoked libtoolize before autoreconf.
> 
> So just to be crystal clear: is the official guidance that we should run libtoolize and then autoreconf, and that should always work?

Well, I hesitate to dub my word as "official"... but this is what my bootstrap script (and the gnulib bootstrap script) have been doing as a work-around for autoreconf brokenness for several years a piece with less weirdness than the olden days of trusting autoreconf.

Another option you have, should you worry about maintaining your own autogen.sh script to keep track of changes in upstream autotools dependencies and invocation ordering, is to use my bootstrap script (as used by libtool itself and m4 among others, and maintained separately at http://github.com/gvvaughan/bootstrap).  This nicely future-proofs you against upstream changes, or addition of internationalization or gnulib to your projects.

>> In your original report, however, you said:
>> 
>> "The problem appears to be that make is checking for ../m4/libtool.m4 file as a dependency.  This file file -- and the entire ../m4 directory, for that matter -- does not exist.  So make decides to fire the "run the aclocal" rule."
>> 
>> ...which seems odd to me, because for a subproject libltdl, the parent AC_CONFIG_MACRO_DIR/ACLOCAL_AMFLAGS directory is supposed to be merged in.  Did you mean to say "../config/libtool.m4" above?  If that substitution really isn't happening, then you've found a different bug - but I can't reproduce that one with 2.4.3, 2.4.4 nor current master.
> 
> I don't remember, and since you can't reproduce it, let's assume that I made a user error and I really did mean "../config/libtool.m4".  :-)

Agreed!

Cheers,
-- 
Gary V. Vaughan (gary AT gnu DOT org)