GNU bug report logs - #11814
The test logs lost their title

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 11814 in the body.
You can then email your comments to 11814 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Automake Bugs <bug-automake <at> gnu.org>
Subject: The test logs lost their title
Date: Fri, 29 Jun 2012 09:17:59 +0200

It seems that in recent changes, the test logs have lost
their title, which included the exit status.  Now, reading
a log, one can no longer know how the test exited.

Message #8 received at 11814 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: 11814 <at> debbugs.gnu.org
Subject: Re: bug#11814: The test logs lost their title
Date: Fri, 29 Jun 2012 19:01:09 +0200

tags notabug 11814
thanks

Hi Akim.

On 06/29/2012 09:17 AM, Akim Demaille wrote:
> It seems that in recent changes, the test logs have lost
> their title, which included the exit status.  Now, reading
> a log, one can no longer know how the test exited.
>
All of this is intended, because we now report "metadata" about the
test run in the '*.trs' file rather than in the '*.log' file.  So,
to know whether a test run has been successful, or which the results
of the individual subtests has been, you should look at that file
(it should be easily human readable as well as machine parseable,
although its format is still experimental and likely to change in
future Automake versions).

All of this should be explained in detail in the manual.  If you
fund something unclear there, feel free to open a new bug report.

Regards,
  Stefano

Message #11 received at 11814 <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 11814 <at> debbugs.gnu.org
Subject: Re: bug#11814: The test logs lost their title
Date: Thu, 5 Jul 2012 14:39:24 +0200

[Message part 1 (text/plain, inline)]

Le 29 juin 2012 à 19:01, Stefano Lattarini a écrit :

> Hi Akim.

Hi!

> On 06/29/2012 09:17 AM, Akim Demaille wrote:
>> It seems that in recent changes, the test logs have lost
>> their title, which included the exit status.  Now, reading
>> a log, one can no longer know how the test exited.
>> 
> All of this is intended, because we now report "metadata" about the
> test run in the '*.trs' file rather than in the '*.log' file.

This is not metadata, this is the result of the test case.
When reading the log of the test, I expect to get the result.
Of course I could do it myself, but it means addition complexity
with temporary files, trap etc. which I am very happy to avoid.
Was there a _problem_ that needed to be solved by the removal
of the result?


> So,
> to know whether a test run has been successful, or which the results
> of the individual subtests has been, you should look at that file
> (it should be easily human readable as well as machine parseable,
> although its format is still experimental and likely to change in
> future Automake versions).

I don't want to read several files.

Also, test-suite.log contains a new section which
is not compliant with rst:

> ==========================================
>    Vaucanson 2.0a: tests/test-suite.log
> ==========================================
> 
> # TOTAL: 16
> # PASS:  14
> # SKIP:  0
> # XFAIL: 1
> # FAIL:  1
> # XPASS: 0
> # ERROR: 0
> 
> .. contents:: :depth: 2
> 
> XFAIL: rat/wrong-weight-set
> ===========================

which results in:

[Message part 2 (text/html, inline)]

[Capture d’écran 2012-07-05 à 14.37.02.png (image/png, inline)]

Message #14 received at 11814 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>, 11814 <at> debbugs.gnu.org
Subject: Re: bug#11814: The test logs lost their title
Date: Thu, 05 Jul 2012 17:42:14 +0200

> Le 29 juin 2012 à 19:01, Stefano Lattarini a écrit :
> 
>> Hi Akim.
> 
> Hi!
> 
>> On 06/29/2012 09:17 AM, Akim Demaille wrote:
>>> It seems that in recent changes, the test logs have lost
>>> their title, which included the exit status.  Now, reading
>>> a log, one can no longer know how the test exited.
>>>
>> All of this is intended, because we now report "metadata" about the
>> test run in the '*.trs' file rather than in the '*.log' file.
> 
> This is not metadata, this is the result of the test case.
> When reading the log of the test, I expect to get the result.
> Of course I could do it myself, but it means addition complexity
> with temporary files, trap etc. which I am very happy to avoid.
> Was there a _problem_ that needed to be solved by the removal
> of the result?
>
The problem was that displaying the exit code at the beginning of the
log required one more 'cat' invocation and one more 'mv' invocation per
test.  So, in the heavy refactoring taking place during the introduction
of support for TAP and for user-defined test drivers, I zapped that
feature for the sake of performances, aware of the fact that the exit
status was still available in the new corresponding '.trs' file.  But I
see your point about the convenience of having it repeated in the log
file too.  So I will add a new option for the default driver, as well as
for the TAP driver, that will cause them to print a summary of the test
output at the end of the '.log' file.

> 
>> So,
>> to know whether a test run has been successful, or which the results
>> of the individual subtests has been, you should look at that file
>> (it should be easily human readable as well as machine parseable,
>> although its format is still experimental and likely to change in
>> future Automake versions).
> 
> I don't want to read several files.
> 
> Also, test-suite.log contains a new section which
> is not compliant with rst:
> 
>> ==========================================
>>    Vaucanson 2.0a: tests/test-suite.log
>> ==========================================
>>
>> # TOTAL: 16
>> # PASS:  14
>> # SKIP:  0
>> # XFAIL: 1
>> # FAIL:  1
>> # XPASS: 0
>> # ERROR: 0
>>
>> .. contents:: :depth: 2
>>
>> XFAIL: rat/wrong-weight-set
>> ===========================
> 
> which results in:
>
> [SNIP IMAGE]
>
I don't know reStructuredText well enough to fix this without too much
effort.  But of course, I'll gladly accept a patch (better if with a
test case, unless that would be too tricky or too hard to write).

Regards,
  Stefano

Message #17 received at 11814 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: 11814 <at> debbugs.gnu.org
Cc: Akim Demaille <akim.demaille <at> gmail.com>,
 "automake-patches <at> gnu.org" <automake-patches <at> gnu.org>
Subject: Re: The test logs lost their title
Date: Tue, 24 Dec 2013 22:05:10 +0100

[Message part 1 (text/plain, inline)]

tags 11814 +patch
close 11814
stop

Reference:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11814

Rather than adding yet another option, I've decided to simply enhance
the test-driver script to *append* the result and exit status of any
test it runs after the logged output of that test.  This is implemented
by the attached patch.  The change will appear in Automake 1.15 (since
I'm not sure it's 100% safe for a micro release),

Thanks, and sorry for the shameful delay,
  Stefano

[0001-testsuite-harness-report-test-exit-status-in-log-fil.patch (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.