GNU bug report logs - #16057
Non-parallel test suite API changes in 1.13

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 16057 in the body.
You can then email your comments to 16057 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Behdad Esfahbod <behdad <at> behdad.org>
To: bug-automake <at> gnu.org
Subject: Non-parallel test suite API changes in 1.13
Date: Wed, 04 Dec 2013 20:37:55 -0500

Hi,

Please advise how one is supposed to port their pre-1.13 test suite's
TESTS_ENVIRONMENT to 1.13.  Currently, in fontconfig and harfbuzz at least, we
cannot find a solution that works both with 1.11 and 1.13, and we cannot
require 1.13.

Example change that is not acceptable:


https://github.com/behdad/harfbuzz/commit/d913f98d88098fc0f4163dfbc54d8ca9ebe9dd81

Thanks,
-- 
behdad
http://behdad.org/

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Behdad Esfahbod <behdad <at> behdad.org>
To: bug-automake <at> gnu.org
Subject: Re: Non-parallel test suite API changes in 1.13
Date: Wed, 04 Dec 2013 20:48:32 -0500

FWIW, the serial-tests is useless since it makes automake < 1.13 to fail. :(

On 13-12-04 08:37 PM, Behdad Esfahbod wrote:
> Hi,
> 
> Please advise how one is supposed to port their pre-1.13 test suite's
> TESTS_ENVIRONMENT to 1.13.  Currently, in fontconfig and harfbuzz at least, we
> cannot find a solution that works both with 1.11 and 1.13, and we cannot
> require 1.13.
> 
> Example change that is not acceptable:
> 
> 
> https://github.com/behdad/harfbuzz/commit/d913f98d88098fc0f4163dfbc54d8ca9ebe9dd81
> 
> Thanks,
> 

-- 
behdad
http://behdad.org/

Message #11 received at 16057 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Behdad Esfahbod <behdad <at> behdad.org>
Cc: 16057 <at> debbugs.gnu.org
Subject: Re: bug#16057: Non-parallel test suite API changes in 1.13
Date: Thu, 05 Dec 2013 22:28:32 +0000

tags 16057 notabug
close 16057
stop

On 12/05/2013 01:37 AM, Behdad Esfahbod wrote:
> Hi,
>
Hello Behdad.

> Please advise how one is supposed to port their pre-1.13 test suite's
> TESTS_ENVIRONMENT to 1.13.  Currently, in fontconfig and harfbuzz at least, we
> cannot find a solution that works both with 1.11 and 1.13, and we cannot
> require 1.13.
>
Why not? 1.13 is almost one year old now...

> Example change that is not acceptable:
> 
> https://github.com/behdad/harfbuzz/commit/d913f98d88098fc0f4163dfbc54d8ca9ebe9dd81
>
Sorry but my suggestion would be exactly along the lines of that
commit.

You might be able to keep 1.11 compatibility with some more-or-less
hacky workaround, but I'm not going to try to work out the details,
because I believe that is the wrong approach.  Automake is only
required by developers, it's easy to install (and to install multiple
versions in parallel, if different level of APIs compatibilities are
needed), and only requires a very tiny amount of disk space.

I'm closing this bug as "works as intended".  Sorry.

Regards,
  Stefano

Message #18 received at 16057 <at> debbugs.gnu.org (full text, mbox):

From: Behdad Esfahbod <behdad <at> behdad.org>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 16057 <at> debbugs.gnu.org
Subject: Re: bug#16057: Non-parallel test suite API changes in 1.13
Date: Thu, 05 Dec 2013 17:38:20 -0500

On 13-12-05 05:28 PM, Stefano Lattarini wrote:
> Why not? 1.13 is almost one year old now...

Why not?  Because I don't require any feature new in 1.13, yet it breaks my
package.  Also because it's in no long-term-support distro like Ubuntu 12.04.
I personally installed 1.13 and released a tarball with it, just to discover
that travis-ci.org, our automated continuous-build server, doesn't have
automake 1.13.  So now I have to patch that system, and ask all my developer /
testers to install automake 1.13.  All these not even because we use any
feature new in 1.13...

At least to allow fixing similar issues in the future, make AM_INIT_AUTOMAKE
warn about unknown options but not err.

-- 
behdad
http://behdad.org/

Message #21 received at 16057 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>,
 Behdad Esfahbod <behdad <at> behdad.org>
Cc: 16057 <at> debbugs.gnu.org
Subject: Re: bug#16057: Non-parallel test suite API changes in 1.13
Date: Thu, 05 Dec 2013 15:49:18 -0700

[Message part 1 (text/plain, inline)]

On 12/05/2013 03:28 PM, Stefano Lattarini wrote:
> tags 16057 notabug
> close 16057
> stop
> 
> On 12/05/2013 01:37 AM, Behdad Esfahbod wrote:
>> Hi,
>>
> Hello Behdad.
> 
>> Please advise how one is supposed to port their pre-1.13 test suite's
>> TESTS_ENVIRONMENT to 1.13.  Currently, in fontconfig and harfbuzz at least, we
>> cannot find a solution that works both with 1.11 and 1.13, and we cannot
>> require 1.13.
>>
> Why not? 1.13 is almost one year old now...

Believe it or not, some people still like to make their project work
with out-of-the-box tools.  In the case of libvirt, we strive to make
our Makefile.am work on the software available in CentOS 5 (hello,
patched automake 1.9) - and it is not a coincidence that this is also
the oldest version still actively supported by gnulib.  Just because
newer releases are now a year old, and in spite of the fact that
automake is fairly easy to self-install from a tarball, does NOT imply
that everyone is willing to self-install.  For good or for bad, there
are enough developers that think of the autotools as magic black boxes
that they are unwilling to use any version not shipped by their distro.

So, anything that upstream automake can do to document HOW to write a
Makefile.am that works on both modern AND ancient automake is appreciated.

> You might be able to keep 1.11 compatibility with some more-or-less
> hacky workaround, but I'm not going to try to work out the details,
> because I believe that is the wrong approach.  Automake is only
> required by developers, it's easy to install (and to install multiple
> versions in parallel, if different level of APIs compatibilities are
> needed), and only requires a very tiny amount of disk space.
> 
> I'm closing this bug as "works as intended".  Sorry.

I'm sorry to see this attitude; it is in the best interest as a
developer tool to specifically document how to write code that still
works with older versions.  While it is okay to encourage the use of
newer tools, we can't be so blind as to assume that people won't (or
even can't) upgrade as fast as we'd like.

Is AM_INIT_AUTOMAKE in 1.11 smart enough to handle m4 conditionals in
the set of options requested?  And if so, can that be coupled with the
use of m4_ifdef or some other m4 code to probe for a feature that only
exists in 1.13 or newer?  If so, try something along the lines of:

AM_INIT_AUTOMAKE([1.11.1 gnits ...]m4_ifdef([1.13-witness], [ serial-tests])

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #24 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Peter Johansson <trojkan <at> gmail.com>
To: bug-automake <at> gnu.org
Subject: Re: bug#16057: Non-parallel test suite API changes in 1.13
Date: Fri, 06 Dec 2013 08:49:59 +1000

On 12/06/2013 08:49 AM, Eric Blake wrote:
>> Why not? 1.13 is almost one year old now...
> Believe it or not, some people still like to make their project work
> with out-of-the-box tools.
+1

-- 
Peter Johansson

Message #27 received at 16057 <at> debbugs.gnu.org (full text, mbox):

From: Behdad Esfahbod <behdad <at> behdad.org>
To: Eric Blake <eblake <at> redhat.com>, 
 Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 16057 <at> debbugs.gnu.org
Subject: Re: bug#16057: Non-parallel test suite API changes in 1.13
Date: Thu, 05 Dec 2013 18:22:31 -0500

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 13-12-05 05:49 PM, Eric Blake wrote:
> AM_INIT_AUTOMAKE([1.11.1 gnits ...]m4_ifdef([1.13-witness], [
> serial-tests])

I ended up working around it like this:


https://github.com/behdad/harfbuzz/commit/2646aec1e67cd6e09f5f7859c9d5898917acc2d5

- -- 
behdad
http://behdad.org/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKhCrcACgkQn+4E5dNTERXYOwCfVIqqFFXiAJRukm2KrE/8na+i
6nIAni8fKjqalln2irdsWKGCAZNt5int
=+utR
-----END PGP SIGNATURE-----

Message #30 received at 16057 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 16057 <at> debbugs.gnu.org, Behdad Esfahbod <behdad <at> behdad.org>
Subject: Re: bug#16057: Non-parallel test suite API changes in 1.13
Date: Thu, 05 Dec 2013 23:25:19 +0000

On 12/05/2013 10:49 PM, Eric Blake wrote:
> On 12/05/2013 03:28 PM, Stefano Lattarini wrote:
>> tags 16057 notabug
>> close 16057
>> stop
>>
>> On 12/05/2013 01:37 AM, Behdad Esfahbod wrote:
>>> Hi,
>>>
>> Hello Behdad.
>>
>>> Please advise how one is supposed to port their pre-1.13 test suite's
>>> TESTS_ENVIRONMENT to 1.13.  Currently, in fontconfig and harfbuzz at least, we
>>> cannot find a solution that works both with 1.11 and 1.13, and we cannot
>>> require 1.13.
>>>
>> Why not? 1.13 is almost one year old now...
> 
> Believe it or not, some people still like to make their project work
> with out-of-the-box tools.
>
I believe it, but I disagree with this approach, when the out-of-the-box
tools are too outdated (But see below, the bit about documentation).

> In the case of libvirt, we strive to make
> our Makefile.am work on the software available in CentOS 5 (hello,
> patched automake 1.9) - and it is not a coincidence that this is also
> the oldest version still actively supported by gnulib.  Just because
> newer releases are now a year old, and in spite of the fact that
> automake is fairly easy to self-install from a tarball, does NOT imply
> that everyone is willing to self-install.
>
And I'm not willing to support such unwillingness, unless it's very
easy do.  Sorry.

As I've already admitted, in retrospective, switching the default
testsuite behaviour in a minor release have been a bad idea, since I
broke the expectations of compatibility held by the users.  But I
can't change that mistake now now, and I believe the best way
forward from the situation we are in is to encourage developers to
update their tools.  (And I believe we should do so also in the
future -- but only by bumping major versions more frequently, not
by breaking compatibility between minor versions).

> For good or for bad, there
> are enough developers that think of the autotools as magic black boxes
> that they are unwilling to use any version not shipped by their distro.
>
I'm not willing to go to great lengths to support such attitude in the
developers, sorry.

> So, anything that upstream automake can do to document HOW to write a
> Makefile.am that works on both modern AND ancient automake is appreciated.
> 
OTOH, I see no harm in documenting possible workarounds; we already do so
so for other use cases we don't really support (e.g., keeping autotools
generated files in a Git repository): it's easy to do, doesn't require
us to maintain more code, and makes it clear to the user that it is not
an encouraged setup and that maintaining it is his responsibility, not
ours.  So, patches welcome.

>> You might be able to keep 1.11 compatibility with some more-or-less
>> hacky workaround, but I'm not going to try to work out the details,
>> because I believe that is the wrong approach.  Automake is only
>> required by developers, it's easy to install (and to install multiple
>> versions in parallel, if different level of APIs compatibilities are
>> needed), and only requires a very tiny amount of disk space.
>>
>> I'm closing this bug as "works as intended".  Sorry.
> 
> I'm sorry to see this attitude; it is in the best interest as a
> developer tool to specifically document how to write code that still
> works with older versions.  While it is okay to encourage the use of
> newer tools, we can't be so blind as to assume that people won't (or
> even can't) upgrade as fast as we'd like.
>
We don't completely disagree in this at least (see above).

> Is AM_INIT_AUTOMAKE in 1.11 smart enough to handle m4 conditionals in
> the set of options requested?
>
I think so (I'm not going to try it myself, but if somebody else does,
I won't throw away the insight he gets ;-)

> And if so, can that be coupled with the
> use of m4_ifdef or some other m4 code to probe for a feature that only
> exists in 1.13 or newer?  If so, try something along the lines of:
> 
> AM_INIT_AUTOMAKE([1.11.1 gnits ...]m4_ifdef([1.13-witness], [ serial-tests])
> 
The problem with this might be if the "1.13-witness" gets screwd up
by distro-specific patches somehow.  If you manage to avoid that, seems
a reasonable workaround.

Regards,
  Stefano

Message #33 received at 16057 <at> debbugs.gnu.org (full text, mbox):

From: Behdad Esfahbod <behdad <at> behdad.org>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>, 
 Eric Blake <eblake <at> redhat.com>
Cc: 16057 <at> debbugs.gnu.org
Subject: Re: bug#16057: Non-parallel test suite API changes in 1.13
Date: Thu, 05 Dec 2013 23:25:56 -0500

On 13-12-05 06:25 PM, Stefano Lattarini wrote:
>> For good or for bad, there
>> > are enough developers that think of the autotools as magic black boxes
>> > that they are unwilling to use any version not shipped by their distro.
>> >
> I'm not willing to go to great lengths to support such attitude in the
> developers, sorry.

You broke backwards compatibility.  That was a bug.  You could spend two hours
and fix it.  Instead, you decided to carry the bug forward and force every
maintainer to deal with it.

All you need to do to fix it is, if TESTS_ENVIRONMENT is defined but not
AM_TESTS_ENVIRONMENT, assume old behavior, and possibly print a warning.  I
don't see how that's going to "great lengths", and mind you, to fix a bug you
introduced, not to support any attitudes in other people whatsoever.

The irony is, now we need to write code to achieve portability over different
versions of automake, so we can enjoy portability to different versions of
makes and whatever else automake promises to do. :)

Anyway, I can find workaround for my own cases.  I just find the attitude a
step in the wrong direction.

Cheers,
-- 
behdad
http://behdad.org/

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.