GNU bug report logs - #11745
the new test-suite summary is confusing

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 11745 in the body.
You can then email your comments to 11745 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alexandre Duret-Lutz <adl <at> lrde.epita.fr>
To: bug-automake <at> gnu.org
Subject: the new test-suite summary is confusing
Date: Tue, 19 Jun 2012 18:47:07 +0200

Hi!

I have a project that contains 8 different test-suites, each in its
own directory.  Some test suite are big, other are small, but they
test different parts of the project and I like being able to run "make
check" in the relevant directory.  They are run sequentially (with
SUBDIRS) when "make check" is run from the top-level.

I upgraded to Automake 1.12.1 to discover the each of these test-suite
now displays a huge summary like:

============================================================================
Testsuite summary for spot 0.9.1a
============================================================================
# TOTAL: 2
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

Instead of the older one-line summary I'm used too.

I can live with the extra verbosity, but I find the initial header
saying "Testsuite summary for spot 0.9.1a" in each of the test suites
very confusing.  The above in definitely *not* a testsuite summary for
the project, it's only a test summary for one directory.

I think the easiest fix would be to change the header to just
"Testsuite summary", without mentioning the package.

Another option could be to let the test-suite be named in Makefile.am.
Maybe introduce
    TESTS_DISPLAY_NAME = $(PACKAGE_STRING)
and use
    echo "Testsuite summary for $(TESTS_DISPLAY_NAME)"
so that people with multiple test-suites can give them different names.

-- 
Alexandre Duret-Lutz

Message #8 received at 11745 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: adl <at> lrde.epita.fr
Cc: 11745 <at> debbugs.gnu.org
Subject: Re: the new test-suite summary is confusing
Date: Mon, 1 Jun 2020 19:44:23 -0600

[Message part 1 (text/plain, inline)]

Hi Alexandre and all - regarding
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=11745 (12 years ago, oh well).

    I upgraded to Automake 1.12.1 to discover the each of these test-suite
    [directories] now displays a huge summary like:
    ============================================================================
    Testsuite summary for spot 0.9.1a
    ...
    The above in definitely *not* a testsuite summary for
    the project, it's only a test summary for one directory.

Agreed, but:

    I think the easiest fix would be to change the header to just
    "Testsuite summary", without mentioning the package.

I don't think that's the right fix. In TeX Live, we have many different
packages, all in one Automake tree, and if every testsuite summary just
said "TeX Live", it would be difficult to discern what the summary was
reporting on. That is, the TL make check output starts like this:
Testsuite summary for Kpathsea 6.3.3/dev
..
Testsuite summary for zlib (TeX Live) 1.2.11
..
Testsuite summary for lua53 (TeX Live) 5.3.5

If all of those just said "TeX Live", it wouldn't be good.


I don't think we should change the default form of the Automake header
line at this late date, since some packages (and plenty of AM's own
tests) might well be expecting what's currently output.

So I propose to define a new variable in lib/am/check.am:
AM_TESTSUITE_SUMMARY_HEADER = " for $(PACKAGE_STRING)"
so the default is unchanged.

I made a simple test (t/testsuite-summary-header.sh) that checks various
settings of the new variable behave as expected, in particular, setting
it to the empty string for your case.

Proposed patch attached. make check and make distcheck still pass.
Jim, anyone, any comments? --thanks, karl.

[summary-hdr.patch (application/octet-stream, attachment)]

Message #11 received at 11745 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Karl Berry <karl <at> freefriends.org>
Cc: adl <at> lrde.epita.fr, 11745 <at> debbugs.gnu.org
Subject: Re: bug#11745: the new test-suite summary is confusing
Date: Fri, 5 Jun 2020 20:29:50 -0700

On Mon, Jun 1, 2020 at 6:46 PM Karl Berry <karl <at> freefriends.org> wrote:
> Hi Alexandre and all - regarding
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=11745 (12 years ago, oh well).
>
>     I upgraded to Automake 1.12.1 to discover the each of these test-suite
>     [directories] now displays a huge summary like:
>     ============================================================================
>     Testsuite summary for spot 0.9.1a
>     ...
>     The above in definitely *not* a testsuite summary for
>     the project, it's only a test summary for one directory.
>
> Agreed, but:
>
>     I think the easiest fix would be to change the header to just
>     "Testsuite summary", without mentioning the package.
>
> I don't think that's the right fix. In TeX Live, we have many different
> packages, all in one Automake tree, and if every testsuite summary just
> said "TeX Live", it would be difficult to discern what the summary was
> reporting on. That is, the TL make check output starts like this:
> Testsuite summary for Kpathsea 6.3.3/dev
> ..
> Testsuite summary for zlib (TeX Live) 1.2.11
> ..
> Testsuite summary for lua53 (TeX Live) 5.3.5
>
> If all of those just said "TeX Live", it wouldn't be good.
>
>
> I don't think we should change the default form of the Automake header
> line at this late date, since some packages (and plenty of AM's own
> tests) might well be expecting what's currently output.
>
> So I propose to define a new variable in lib/am/check.am:
> AM_TESTSUITE_SUMMARY_HEADER = " for $(PACKAGE_STRING)"
> so the default is unchanged.

Thanks, Karl!
This looks great. Only one minor suggestion: to use single quotes
there, rather than double:

  AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING)'

so that the replacement value is less likely to require additional escaping.

Message #14 received at 11745 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: jim <at> meyering.net
Cc: adl <at> lrde.epita.fr, 11745 <at> debbugs.gnu.org
Subject: Re: bug#11745: the new test-suite summary is confusing
Date: Sat, 6 Jun 2020 16:48:40 -0600

      AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING)'

I went back and forth on which quotes to use :).
Ok, accepted, pushed, thanks :). -k

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.