GNU bug report logs - #20693
SKIP not counted in TAP

Previous Next

To reply to this bug, email your comments to 20693 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Arthur Schwarz" <aschwarz1309 <at> att.net>
To: <bug-automake <at> gnu.org>
Subject: SKIP not counted in TAP
Date: Fri, 29 May 2015 08:15:15 -0700

I think this is an error.

skip causes the following issues:
1: It is not reported.
2: It is not summarized.
3: It must be subtracted from the plan (tests run), and
4: The .log file has TAP generated output and the test data intermixed.

I think this is against the TAP definition?

====================== output ======================

PASS: test2.tap 1
FAIL: test2.tap 2
===========================================
Testsuite summary for test 0.5
===========================================
# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
===========================================
See src/test-suite.log
===========================================

====================== test-suite.log ======================

==================================
   test 0.5: src/test-suite.log
==================================

# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test2
===========

skip
ok
PASS: test2.tap 1
not ok
FAIL: test2.tap 2
1..2

====================== program ======================
 
## Process this file with automake to produce Makefile.in

# select testing format
AUTOMAKE_OPTIONS = parallel-tests 
EXTRA_DIST       = tap-driver.sh


# test options
TEST_EXTENSIONS    = .tap
TAP_LOG_DRIVER     = env AM_TAP_AWK='$(AWK)' $(SHELL) ./tap-driver.sh

# Testing 
check_SCRIPTS      = test2.tap
TESTS              = test2.tap

test2.tap:
	echo '#!/bin/bash'                                  > test2.tap
	echo 'echo test2.tap "$$# [ $$@ ]"   >> test2.log' >> test2.tap
	echo "echo I am a tap script         >> test2.log" >> test2.tap
	echo echo '1..2'                                   >> test2.tap
	echo echo 'ok'                                     >> test2.tap
	echo echo 'not ok'                                 >> test2.tap
	echo echo 'skip'                                   >> test2.tap
	chmod +x test2.tap

CLEANFILES         = $(test_SOURCES)

Message #8 received at 20693 <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> elliptictech.com>
To: Arthur Schwarz <aschwarz1309 <at> att.net>
Cc: 20693 <at> debbugs.gnu.org
Subject: Re: bug#20693: SKIP not counted in TAP
Date: Fri, 29 May 2015 13:11:27 -0400

On 2015-05-29 08:15 -0700, Arthur Schwarz wrote:
> skip causes the following issues:
> 1: It is not reported.
> 2: It is not summarized.
> 3: It must be subtracted from the plan (tests run), and
> 4: The .log file has TAP generated output and the test data intermixed.
> 
> I think this is against the TAP definition?

In a way, because...

> skip
> ok
> PASS: test2.tap 1
> not ok
> FAIL: test2.tap 2
> 1..2

... it is your program's output that does not follow the specification.

There are two basic styles of skipping in TAP.  You can skip a single
test result:

  ok 1 # skip reason

or you can use the plan line to indicate that the entire test program is
to be skipped:

  1..0 # skip reason

Regards,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Message #11 received at 20693 <at> debbugs.gnu.org (full text, mbox):

From: "Arthur Schwarz" <aschwarz1309 <at> att.net>
To: "'Nick Bowler'" <nbowler <at> elliptictech.com>
Cc: 20693 <at> debbugs.gnu.org
Subject: RE: bug#20693: SKIP not counted in TAP
Date: Fri, 29 May 2015 14:49:37 -0700

Thanks Nick;

But doesn't this mean that tap-driver.sh didn't report a violation to the
TAP standard? And why is my output intermixed in the .trs file? If 'skip'
below is illegal, why isn't it tagged? (I think that multiline comments need
leading whitespace?)

As another issue, I looked at the awk code and have some potential revisions
below. I know very little about awk but I think the resultant code is faster
in that the 'is_first_read = 1' is not necessary and consequent checks for
each line input are unnecessary. 

As to your (sly and underhanded) comment about awk being defined (and I
should read it), well, in this case I agree. But in general where something
is important to the issue at hand obligating the user to read every relevant
manual bearing on the issue I feel is burdensome and unnecessary. I
understand the problems with following 'other peoples manuals' and burden
that that places on the maintenance staff, but requiring that readers of the
Automake User's Manual read the Autoconf, Automake, make, TAP standards,
bash for 'env' and etc. is needless. My opinion is that some of the material
in these (and other manuals) which have a direct bearing on Automake should
have more than a reference to Yet Another Manual. This issue directly bears
on, for example, TAP, where a few wise words from the Master's Of The
Universe will go a long way being productive. But, this is an opinion. Oh, I
am doing some writing for Section 15 of the Automake Manual to say that I am
putting some effort is providing an alternative viewpoint.

===================== tap-driver.sh =====================

   
   ## --------- ##
   ##  PARSING  ##
   ## --------- ##
   
   st = getline
   
   nextline = $0;  # I have no idea what $0 does.
   
   if (st < 0) # I/O error.
     abort("in input loop: only one input line")
   }
   
   do
     {
       # Involutions required so that we are able to read the exit status
       # from the last input line.
       # Copy any input line verbatim into the log file.
   
       o o o
       
       st = getline
       curline = nextline
       nextline = $0
       $0 = curline

   while(st >= 0)

   if (st < 0) # I/O error.
      fatal("I/O error while reading from input stream")
   }
   
On 2015-05-29 08:15 -0700, Arthur Schwarz wrote:
> skip causes the following issues:
> 1: It is not reported.
> 2: It is not summarized.
> 3: It must be subtracted from the plan (tests run), and
> 4: The .log file has TAP generated output and the test data intermixed.
> 
> I think this is against the TAP definition?

In a way, because...

> skip
> ok
> PASS: test2.tap 1
> not ok
> FAIL: test2.tap 2
> 1..2

... it is your program's output that does not follow the specification.

There are two basic styles of skipping in TAP.  You can skip a single
test result:

  ok 1 # skip reason

or you can use the plan line to indicate that the entire test program is
to be skipped:

  1..0 # skip reason

Regards,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Message #14 received at 20693 <at> debbugs.gnu.org (full text, mbox):

From: "Arthur Schwarz" <aschwarz1309 <at> att.net>
To: "'Nick Bowler'" <nbowler <at> elliptictech.com>
Cc: 20693 <at> debbugs.gnu.org
Subject: RE: bug#20693: SKIP not counted in TAP
Date: Fri, 29 May 2015 15:15:14 -0700

-----Original Message-----In a way, because...

> skip
> ok
> PASS: test2.tap 1
> not ok
> FAIL: test2.tap 2
> 1..2

... it is your program's output that does not follow the specification.

There are two basic styles of skipping in TAP.  You can skip a single
test result:

  ok 1 # skip reason

or you can use the plan line to indicate that the entire test program is
to be skipped:

  1..0 # skip reason


Sorry to be such a pimple, but I just tried:

========== script ==========

#!/bin/bash
echo test2.tap "$# [ $@ ]"   >> test2.log
echo I am a tap script         >> test2.log
echo 1..3
echo ok
echo not ok
echo ok 3 # skip reason 


========== which yielded ==========

PASS: test2.tap 1
FAIL: test2.tap 2
PASS: test2.tap 3
=================================
Testsuite summary for test 0.5
=================================
# TOTAL: 3
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
================================
See src/test-suite.log
================================

========== and  ==========

I would have expected a 'skip' not a PASS! And the comment isn't output w/wo
TAP_LOG_DRIVER_FLAGS=--comments. However, 1..0 works if there is no test
output, as in the output only containing 1..0 and nothing else. More than
one test yields errors (# UNPLANNED).

If this is an error on my part (which it may well be) then I am confused
about your comment above.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.