Subject: bug#11814: The test logs lost their title

From: Stefano Lattarini <[email-masked]>
Date: Tue, 24 Dec 2013 22:05:10 +0100

tags 11814 +patch
close 11814
stop

Reference:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11814

Rather than adding yet another option, I've decided to simply enhance
the test-driver script to *append* the result and exit status of any
test it runs after the logged output of that test.  This is implemented
by the attached patch.  The change will appear in Automake 1.15 (since
I'm not sure it's 100% safe for a micro release),

Thanks, and sorry for the shameful delay,
  Stefano
>From 329cbe0ab70f1e7ee3d2f7ae00b5997d7e0bcb55 Mon Sep 17 00:00:00 2001
Message-Id: <329cbe0ab[email-masked]>
From: Stefano Lattarini <[email-masked]>
Date: Tue, 24 Dec 2013 17:45:18 +0100
Subject: [PATCH] testsuite harness: report test exit status in log file

The exit status of a test should be reported in the test logs, so
that one can see at a glance whether the test has succeeded or failed,
without having to look also into the corresponding .trs file.

This fixes automake bug#11814.

* lib/test-driver: Also report the test script exit status in the
test log (as the last line).
* t/check-exit-status-reported.sh: Test this new behaviour.
* t/list-of-tests.mk: Add the new test.
* t/ax/test-lib.sh( am_exit_trap): No longer log the test exit status;
this has been made redundant by the change to 'test-driver'.  While at
it, fix an imperfect quoting.

Signed-off-by: Stefano Lattarini <[email-masked]>
---
 lib/test-driver                          | 13 +++++-
 t/ax/test-lib.sh                         |  3 +-
 t/list-of-tests.mk                       |  1 +
 t/parallel-tests-exit-status-reported.sh | 68 ++++++++++++++++++++++++++++++++
 4 files changed, 81 insertions(+), 4 deletions(-)
 create mode 100644 t/parallel-tests-exit-status-reported.sh

diff --git a/lib/test-driver b/lib/test-driver
index d306056..110eec4 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -106,11 +106,14 @@ trap "st=143; $do_exit" 15
 # Test script is run here.
 "$@" >$log_file 2>&1
 estatus=$?
+
 if test $enable_hard_errors = no && test $estatus -eq 99; then
-  estatus=1
+  tweaked_estatus=1
+else
+  tweaked_estatus=$estatus
 fi
 
-case $estatus:$expect_failure in
+case $tweaked_estatus:$expect_failure in
   0:yes) col=$red res=XPASS recheck=yes gcopy=yes;;
   0:*)   col=$grn res=PASS  recheck=no  gcopy=no;;
   77:*)  col=$blu res=SKIP  recheck=no  gcopy=yes;;
@@ -119,6 +122,12 @@ case $estatus:$expect_failure in
   *:*)   col=$red res=FAIL  recheck=yes gcopy=yes;;
 esac
 
+# Report the test outcome and exit status in the logs, so that one can
+# know whether the test passed or failed simply by looking at the '.log'
+# file, without the need of also peaking into the corresponding '.trs'
+# file (automake bug#11814).
+echo "$res $test_name (exit status: $estatus)" >>$log_file
+
 # Report outcome to console.
 echo "${col}${res}${std}: $test_name"
 
diff --git a/t/ax/test-lib.sh b/t/ax/test-lib.sh
index aa015d6..b8c7253 100644
--- a/t/ax/test-lib.sh
+++ b/t/ax/test-lib.sh
@@ -254,7 +254,7 @@ am_exit_trap ()
     # behaviour, while from time to time useful to developers, is not
     # meant to be enabled by default, as it could cause spurious failures
     # in the wild.  Thus it will be enabled only when the variable
-    # "am_explicit_skips" is set to a "true" value.
+    # 'am_explicit_skips' is set to a "true" value.
     case $am_explicit_skips in
       [yY]|[yY]es|1)
         if test $exit_status -eq 77 && test $am__test_skipped != yes; then
@@ -266,7 +266,6 @@ am_exit_trap ()
   fi
   am_keeping_testdirs || rm_rf_ $am_test_subdir
   set +x
-  echo "$me: exit $exit_status"
   # Spurious escaping to ensure we do not call our "exit" alias.
   \exit $exit_status
 }
diff --git a/t/list-of-tests.mk b/t/list-of-tests.mk
index 462497e..ba65789 100644
--- a/t/list-of-tests.mk
+++ b/t/list-of-tests.mk
@@ -765,6 +765,7 @@ t/parallel-tests-basics.sh \
 t/parallel-tests-concurrency.sh \
 t/parallel-tests-concurrency-2.sh \
 t/parallel-tests-empty.sh \
+t/parallel-tests-exit-status-reported.sh \
 t/parallel-tests-generated-and-distributed.sh \
 t/parallel-tests-recheck.sh \
 t/parallel-tests-trailing-whitespace.sh \
diff --git a/t/parallel-tests-exit-status-reported.sh b/t/parallel-tests-exit-status-reported.sh
new file mode 100644
index 0000000..d40f1f5
--- /dev/null
+++ b/t/parallel-tests-exit-status-reported.sh
@@ -0,0 +1,68 @@
+#! /bin/sh
+# Copyright (C) 2013 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# The exit status of a test should be reported in the test logs, so
+# that one can see at a glance whether the test has succeeded or failed,
+# without having to look also into the corresponding .trs file.
+# See automake bug#11814.
+
+. test-init.sh
+
+echo AC_OUTPUT >> configure.ac
+
+echo XFAIL_TESTS = t3.test t00.test > Makefile.am
+echo TESTS = t00.test >> Makefile.am
+for s in 0 1 2 3 5 77 78 99 100 126 127; do
+  echo "TESTS += t${s}.test" >> Makefile.am
+  cat > t${s}.test <<END
+#!/bin/sh
+printf "%s\\n%s\\n" 'random' 'will exit with status $s'
+exit $s
+END
+done
+cp t0.test t00.test
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+run_make -e FAIL check
+ls -l  # For debugging.
+
+match_result ()
+{
+  cat "$1.log"  # For debugging.
+  test $(wc -l <"$1.log") -eq 3
+  sed -n '$p' "$1.log" | grep "^$2 $1\\.test (exit status: $3)$"
+}
+
+match_result  t0   PASS  0
+match_result  t00  XPASS 0
+match_result  t1   FAIL  1
+match_result  t2   FAIL  2
+match_result  t3   XFAIL 3
+match_result  t5   FAIL  5
+match_result  t77  SKIP  77
+match_result  t78  FAIL  78
+match_result  t99  ERROR 99
+match_result  t100 FAIL  100
+match_result  t126 FAIL  126
+match_result  t127 FAIL  127
+
+:
-- 
1.8.5.rc0.23.gaa27064