GNU bug report logs - #20715
tap-driver.sh 'missing test plan' message contradicts the TAP Standard

Previous Next

To reply to this bug, email your comments to 20715 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Arthur Schwarz" <aschwarz1309 <at> att.net>
To: <bug-automake <at> gnu.org>
Subject: tap-driver.sh 'missing test plan' message contradicts the TAP Standard
Date: Mon, 1 Jun 2015 14:45:49 -0700

15.4.1 Introduction to TAP


http://testanything.org/

Explicitly specifies that a test plan is optional.

"The plan

The plan tells how many tests will be run, or how many tests have run. It's
a check that the test file hasn't stopped prematurely. It must appear once,
whether at the beginning or end of the output. The plan is usually the first
line of TAP output (although in future there may be a version line before
it) and it specifies how many test points are to follow. For example,

1..10

means you plan on running 10 tests. This is a safeguard in case your test
file dies silently in the middle of its run. The plan is optional but if
there is a plan before the test points it must be the first non-diagnostic
line output by the test file. In certain instances a test file may not know
how many test points it will ultimately be running. In this case the plan
can be the last non-diagnostic line in the output. The plan cannot appear in
the middle of the output, nor can it appear more than once."

The operand part of this paragraph is " The plan is optional ..." which
directly contradicts "It must appear once ...". Your diagnostic message is
either incorrect, or more likely, there should be a statement in the
Automake Manual providing guidance.

Message #8 received at 20715 <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> elliptictech.com>
To: Arthur Schwarz <aschwarz1309 <at> att.net>
Cc: 20715 <at> debbugs.gnu.org
Subject: Re: bug#20715: tap-driver.sh 'missing test plan' message contradicts
 the TAP Standard
Date: Tue, 2 Jun 2015 09:40:40 -0400

On 2015-06-01 14:45 -0700, Arthur Schwarz wrote:
> http://testanything.org/
> 
> Explicitly specifies that a test plan is optional.
[...]
> The operand part of this paragraph is " The plan is optional ..." which
> directly contradicts "It must appear once ...". Your diagnostic message is
> either incorrect, or more likely, there should be a statement in the
> Automake Manual providing guidance.

The plan line has to be mandatory for this feature to work as
intended (i.e., for the TAP consumer to determine whether a
producer has run to completion or not).  An optional plan would
be useless.

Other TAP consumers behave similarly to Automake's in this regard.
Take prove for example, the de-facto reference implementation of a
TAP consumer, which outputs a similar error message:

  % prove -ve sh test.sh
  test.sh .. 
  ok 1
  All 1 subtests passed 

  Test Summary Report
  -------------------
  test.sh (Wstat: 0 Tests: 1 Failed: 0)
    Parse errors: No plan found in TAP output
  Files=1, Tests=1,  0 wallclock secs ( 0.03 usr +  0.00 sys =  0.03 CPU)
  Result: FAIL

The bug is in the TAP documentation.

Cheers,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Message #11 received at 20715 <at> debbugs.gnu.org (full text, mbox):

From: "Arthur Schwarz" <aschwarz1309 <at> att.net>
To: "'Nick Bowler'" <nbowler <at> elliptictech.com>
Cc: 20715 <at> debbugs.gnu.org
Subject: RE: bug#20715: tap-driver.sh 'missing test plan' message contradicts
 the TAP Standard
Date: Tue, 2 Jun 2015 07:34:23 -0700

[Message part 1 (text/plain, inline)]

The issue is not intent but information. The Automake Manual says that their
TAP implementation is defined by the TAP references. The reference to the
TAP Standard is missing which is puzzling, and the remaining references can
not be considered definitive but only an instance of an implementation of
the standard. The 'fault' is that in the face of conflicting statements in
the standard, Automake does not declare which is to be used. No biggie
unless you plan to use TAP.

Here is an alternative description of the TAP area of the Automake document.
It is unproofed, unfinished, and inelegant. It is 'free' in the sense that I
have lost all my senses and assigned my GPL copyright to the FSF. Sigh.

art

-----Original Message-----
From: Nick Bowler [mailto:nbowler <at> elliptictech.com] 
Sent: Tuesday, June 02, 2015 6:41 AM
To: Arthur Schwarz
Cc: 20715 <at> debbugs.gnu.org
Subject: Re: bug#20715: tap-driver.sh 'missing test plan' message
contradicts the TAP Standard

On 2015-06-01 14:45 -0700, Arthur Schwarz wrote:
> http://testanything.org/
> 
> Explicitly specifies that a test plan is optional.
[...]
> The operand part of this paragraph is " The plan is optional ..." which
> directly contradicts "It must appear once ...". Your diagnostic message is
> either incorrect, or more likely, there should be a statement in the
> Automake Manual providing guidance.

The plan line has to be mandatory for this feature to work as
intended (i.e., for the TAP consumer to determine whether a
producer has run to completion or not).  An optional plan would
be useless.

Other TAP consumers behave similarly to Automake's in this regard.
Take prove for example, the de-facto reference implementation of a
TAP consumer, which outputs a similar error message:

  % prove -ve sh test.sh
  test.sh .. 
  ok 1
  All 1 subtests passed 

  Test Summary Report
  -------------------
  test.sh (Wstat: 0 Tests: 1 Failed: 0)
    Parse errors: No plan found in TAP output
  Files=1, Tests=1,  0 wallclock secs ( 0.03 usr +  0.00 sys =  0.03 CPU)
  Result: FAIL

The bug is in the TAP documentation.

Cheers,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

[TAP.pdf (application/pdf, attachment)]

Message #14 received at 20715 <at> debbugs.gnu.org (full text, mbox):

From: "Arthur Schwarz" <aschwarz1309 <at> att.net>
To: "'Arthur Schwarz'" <aschwarz1309 <at> att.net>,
 "'Nick Bowler'" <nbowler <at> elliptictech.com>
Cc: 20715 <at> debbugs.gnu.org
Subject: RE: bug#20715: tap-driver.sh 'missing test plan' message contradicts
 the TAP Standard
Date: Tue, 2 Jun 2015 09:40:15 -0700

    The plan line has to be mandatory for this feature to work as
    intended (i.e., for the TAP consumer to determine whether a
    producer has run to completion or not).  An optional plan would
    be useless.

Sorry to be such a pest.

Your statement that "... determine whether a producer has run to completion
or not" is puzzling. The TAP Standard specifically states that if the test
plan is 1..N and the number of test lines are k < N, then the k+1 .. N
missing test lines are to be considered as skipped. If this is your
standards conforming implementation then correct test completion is seen
when either N test lines are detected or an end of file is seen. So your
statement is confusing. Are you saying that the Automake TAP implementation
is non-conforming?

There is one issue not addressed in the TAP Standard and not addressed in
the Automake Manual. What happens if k > N? From your statement I would
assume that the TAP Automake implementation gives up and goes home, perhaps
with a diagnostic message (sorry, I just haven't checked this out yet so I
really don't know). But if the TAP implementation just gives up then
trailing Diagnostic lines are not read which violates the TAP Standard.

If I sound confused and look ugly I am. One the state of mind, the other the
state of being.

art

Message #17 received at 20715 <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> elliptictech.com>
To: Arthur Schwarz <aschwarz1309 <at> att.net>
Cc: 20715 <at> debbugs.gnu.org
Subject: Re: bug#20715: tap-driver.sh 'missing test plan' message contradicts
 the TAP Standard
Date: Tue, 2 Jun 2015 13:30:54 -0400

On 2015-06-02 09:40 -0700, Arthur Schwarz wrote:
> [Nick Bowler wrote]
>     The plan line has to be mandatory for this feature to work as
>     intended (i.e., for the TAP consumer to determine whether a
>     producer has run to completion or not).  An optional plan would
>     be useless.
> 
> Sorry to be such a pest.
> 
> Your statement that "... determine whether a producer has run to completion
> or not" is puzzling. The TAP Standard specifically states that if the test
> plan is 1..N and the number of test lines are k < N, then the k+1 .. N
> missing test lines are to be considered as skipped.

Where does it say that?  It is not correct to treat missing test results
as skips.

It is an error for the number of actual test results to differ from the
plan.  This is the whole point of the plan: the consumer always knows
how many test results are expected.  This enables the consumer to tell
the difference between a producer that crashed before completing all the
tests, and a producer that ran normally.

> There is one issue not addressed in the TAP Standard and not addressed in
> the Automake Manual. What happens if k > N?

If k > N, then the number of test cases does not match the plan, so
this is an error.

Regards,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Message #20 received at 20715 <at> debbugs.gnu.org (full text, mbox):

From: "Arthur Schwarz" <aschwarz1309 <at> att.net>
To: "'Nick Bowler'" <nbowler <at> elliptictech.com>
Cc: 20715 <at> debbugs.gnu.org
Subject: RE: bug#20715: tap-driver.sh 'missing test plan' message contradicts
 the TAP Standard
Date: Tue, 2 Jun 2015 11:08:25 -0700

On 2015-06-02 09:40 -0700, Arthur Schwarz wrote:
> [Nick Bowler wrote]
>     The plan line has to be mandatory for this feature to work as
>     intended (i.e., for the TAP consumer to determine whether a
>     producer has run to completion or not).  An optional plan would
>     be useless.
> 
> Sorry to be such a pest.
> 
> Your statement that "... determine whether a producer has run to
completion
> or not" is puzzling. The TAP Standard specifically states that if the test
> plan is 1..N and the number of test lines are k < N, then the k+1 .. N
> missing test lines are to be considered as skipped.

Where does it say that?  It is not correct to treat missing test results
as skips.

   From  http://testanything.org/tap-version-13-specification.html

   Skipping a few

   The following listing plans on running 5 tests. However, our program
decided to not run tests 2 thru 5 at all. To properly report this, the tests
are marked as being skipped.

   TAP version 13
   1..5
   ok 1 - approved operating system
   # $^0 is solaris
   ok 2 - # SKIP no /sys directory
   ok 3 - # SKIP no /sys directory
   ok 4 - # SKIP no /sys directory
   ok 5 - # SKIP no /sys directory

It is an error for the number of actual test results to differ from the
plan.  This is the whole point of the plan: the consumer always knows
how many test results are expected.  This enables the consumer to tell
the difference between a producer that crashed before completing all the
tests, and a producer that ran normally.

   See above.

> There is one issue not addressed in the TAP Standard and not addressed in
> the Automake Manual. What happens if k > N?

If k > N, then the number of test cases does not match the plan, so
this is an error.

  TAP allows k < N (see above). The question of what happens if k > N seems
reasonable. As an extension, the Automake Manual does not state what will
happen if k > N (nor if k < N). This also reflects of the TAP Protocol
statement that the test plan can be at the beginning or end of the test. The
issue is easy to address if the test plan is at the beginning of the
document but will require that an implementation use deferred test line
posting if the test plan is at the end of the document. I think that the TAP
Protocol is poor in this regard.

art

Regards,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Message #23 received at 20715 <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> elliptictech.com>
To: Arthur Schwarz <aschwarz1309 <at> att.net>
Cc: 20715 <at> debbugs.gnu.org
Subject: Re: bug#20715: tap-driver.sh 'missing test plan' message contradicts
 the TAP Standard
Date: Wed, 3 Jun 2015 10:25:13 -0400

On 2015-06-02 11:08 -0700, Arthur Schwarz wrote:
> [Nick Bowler wrote]
> > On 2015-06-02 09:40 -0700, Arthur Schwarz wrote:
> > > The TAP Standard specifically states that if the test plan is 1..N
> > > and the number of test lines are k < N, then the k+1 .. N missing
> > > test lines are to be considered as skipped.
> > 
> > Where does it say that?  It is not correct to treat missing test results
> > as skips.
> 
> From  http://testanything.org/tap-version-13-specification.html
[...]
> The following listing plans on running 5 tests. However, our program
> decided to not run tests 2 thru 5 at all. To properly report this, the
> tests are marked as being skipped.
> 
>    TAP version 13
>    1..5
>    ok 1 - approved operating system
>    # $^0 is solaris
>    ok 2 - # SKIP no /sys directory
>    ok 3 - # SKIP no /sys directory
>    ok 4 - # SKIP no /sys directory
>    ok 5 - # SKIP no /sys directory

This example has exactly 5 test lines, which matches the plan of 5.
So there is no error.

[...]
> TAP allows k < N (see above).

It does not.

Regards,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.