Subject: wording for user on test suite failure?
From: Karl Berry <[email-masked]>
To: [email-masked]
Date: Wed, 31 Jan 2024 11:59:14 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

I was thinking about making the message when the test suite failure more
explicit about what to do, as in below. If any comments or criticisms,
let me know.

I was actually inspired to do this not so much to tweak these words, but
because I was thinking of adding yet more instructions about how to help
debug the subsecond mtime stuff (for automake only, of course), i.e.,
setting am_cv_sleep_fractional_seconds=false. But with the new
possibility that checking make's behavior will be enough to obviate
those random failures, that doesn't seem necessary/desirable now.

But having gone to all the trouble of figuring out how to change the
text and which tests need to be changed to keep them in sync, I guess I
still hope people think it's worth it, or at least ok, to make this
change. The output would look like this:

============================================================================
See tests/test-suite.log for debugging.
Some test(s) failed.  Please report this to <address>,
together with the test-suite.log file (gzipped) and your system
information.  Thanks.
============================================================================

Hopefully I did not break the colorization, but I don't have a good way
to test that. --thanks, karl.

-----------------------------------------------------------------------------
test: make bug-reporting message on test failure more explicit.

* NEWS: mention this.
* lib/am/check.am ($(TEST_SUITE_LOG)) <please_report>: new fn,
new wording.
* t/ax/test-defs.in (failure_footer_text): new wording.
(failure_footer_text_colorized): add red.
* t/ax/tap-summary-aux.sh (failure_footer): use failure_footer_text*.
* t/tap-doc2.sh: change grep.
* t/testsuite-summary-color.sh: likewise.
* t/testsuite-summary-count-many.sh: likewise.
* t/testsuite-summary-count.sh: likewise.
* t/testsuite-summary-reference-log.sh: likewise.
diff --git a/NEWS b/NEWS
index 153cb7733..964d82e8c 100644
--- a/NEWS
+++ b/NEWS
@@ -44,6 +44,10 @@ New in 1.17:
   - The missing script also supports autoreconf, autogen, and perl.
     (https://lists.gnu.org/archive/html/automake-patches/2015-08/msg00000.html)

+  - test-suite.log now contains basic system information, and the
+    console message about bug reporting on failure has a bit more detail.
+    (bug#68746)
+
 * Bugs fixed

   - Generated file timestamp checks handle filesystems with sub-second
diff --git a/lib/am/check.am b/lib/am/check.am
index d210172a6..b6fa3aaad 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -246,6 +246,7 @@ am--force-recheck:
 ## Exists only to be overridden.  See bug#11745.
 AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING)'

+## emacs-page
 $(TEST_SUITE_LOG): $(TEST_LOGS)
 	@$(am__set_TESTS_bases); \
 ## Helper shell function, tells whether a path refers to an existing,
@@ -360,7 +361,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  result_count $$1 "ERROR:" $$error "$$mgn"; \
 	}; \
 ##
-## Write some basic system info in the log. 
+## Function to write some basic system info in the log. 
 	output_system_information () \
 	{ \
 ## Omit the hostname for privacy.  In practice it's a single word?
@@ -372,6 +373,16 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	    cat /etc/issue;                        \
 	  fi;                                      \
 	}; \
+##
+## Function to output the "please report bugs" message.
+	please_report () \
+	{ \
+echo "Some test(s) failed.  Please report this to $(PACKAGE_BUGREPORT),";    \
+echo "together with the test-suite.log file (gzipped) and your system";      \
+echo "information.  Thanks.";                                                \
+	}; \
+##
+## emacs-page
 ## Write "global" testsuite log.
 	{								\
 	  echo "$(PACKAGE_STRING): $(subdir)/$(TEST_SUITE_LOG)" |	\
@@ -379,8 +390,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  create_testsuite_report --no-color;				\
 	  output_system_information;                                    \
 	  echo;								\
-## What is this line for? It seems to be intended for machine parsing,
-## but I can't find any other reference to it. --karl, 27jan24.
+## What is this line for? It's a Sphinx reStructuredText directive that
+## would insert a table of contents that goes two levels deep on headings.
+## But why is it in test-suite.log? No other references to it are evident.
+## Please inform [email-masked].
 	  echo ".. contents:: :depth: 2";				\
 	  echo;								\
 	  for b in $$bases; do echo $$b; done				\
@@ -403,9 +416,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	create_testsuite_report --maybe-color;				\
 	echo "$$col$$br$$std";						\
 	if $$success; then :; else					\
-	  echo "$${col}See $(subdir)/$(TEST_SUITE_LOG)$${std}";		\
+	  echo "$${col}See $(subdir)/$(TEST_SUITE_LOG) for debugging.$${std}";\
 	  if test -n "$(PACKAGE_BUGREPORT)"; then			\
-	    echo "$${col}Please report to $(PACKAGE_BUGREPORT)$${std}";	\
+	    please_report | sed -e "s/^/$${col}/" -e s/'$$'/"$${std}"/; \
 	  fi;								\
 	  echo "$$col$$br$$std";					\
 	fi;								\
diff --git a/t/ax/tap-summary-aux.sh b/t/ax/tap-summary-aux.sh
index d1e2888c3..a709f1ef3 100644
--- a/t/ax/tap-summary-aux.sh
+++ b/t/ax/tap-summary-aux.sh
@@ -87,8 +87,17 @@ if test $use_colors = yes; then
   mgn="$esc[0;35m"
   brg="$esc[1m"
   std="$esc[m"
+  failure_footer="\
+${red}${br}${std}
+$failure_footer_text_colorized
+${red}${br}${std}"
+
 else
   red= grn= lgn= blu= mgn= brg= std=
+  failure_footer="\
+${red}${br}${std}
+$failure_footer_text
+${red}${br}${std}"
 fi

   success_header="\
@@ -101,12 +110,6 @@ ${grn}${br}${std}"
   failure_header="\
 ${red}${br}${std}
 ${red}Testsuite summary for GNU AutoTAP 5.12${std}
-${red}${br}${std}"
-
-  failure_footer="\
-${red}${br}${std}
-${red}See ./test-suite.log${std}
-${red}Please report to [email-masked]${std}
 ${red}${br}${std}"

 $ACLOCAL
diff --git a/t/ax/test-defs.in b/t/ax/test-defs.in
index 8ea842355..c1ead418e 100644
--- a/t/ax/test-defs.in
+++ b/t/ax/test-defs.in
@@ -215,6 +215,20 @@ sleep="sleep $MTIME_RESOLUTION"
 # in UTC; see <https://www.gnu.org/gnu/initial-announcement.html>.
 old_timestamp=198309271735.59

+# We need this in multiple tests and the wording is subject to change.
+# Don't include the ${br} lines since br is not defined at this point.
+#
+failure_footer_text="\
+See ./test-suite.log for debugging.
+Some test(s) failed.  Please report this to [email-masked],
+together with the test-suite.log file (gzipped) and your system
+information.  Thanks."
+#
+# And with color.
+red='[0;31m'
+std='[m';
+failure_footer_text_colorized="`echo \"$failure_footer_text\" | sed -e "s/^/${red}/" -e s/'$'/"${std}"/`"
+
 # Make our wrapper script (or installed scripts, if running under
 # "installcheck") accessible by default.  And avoid uselessly
 # extending $PATH multiple times if this file is sourced multiple times.
diff --git a/t/tap-doc2.sh b/t/tap-doc2.sh
index fdca66ad8..54c69b631 100644
--- a/t/tap-doc2.sh
+++ b/t/tap-doc2.sh
@@ -102,7 +102,7 @@ cat exp
 cat got
 diff exp got

-grep '^Please report to bug-automake@gnu\.org$' stdout
+grep 'Please report this to bug-automake@gnu\.org' stdout

 run_make -O check \
   TESTS='foo.test baz.test' \
diff --git a/t/testsuite-summary-color.sh b/t/testsuite-summary-color.sh
index 7b6254f61..ffbe5c158 100644
--- a/t/testsuite-summary-color.sh
+++ b/t/testsuite-summary-color.sh
@@ -46,8 +46,7 @@ ${red}${br}${std}"

 failure_footer="\
 ${red}${br}${std}
-${red}See ./test-suite.log${std}
-${red}Please report to [email-masked]${std}
+$failure_footer_text_colorized
 ${red}${br}${std}"

 do_check '' <<END
diff --git a/t/testsuite-summary-count-many.sh b/t/testsuite-summary-count-many.sh
index 2560ef82c..3cb5a500c 100644
--- a/t/testsuite-summary-count-many.sh
+++ b/t/testsuite-summary-count-many.sh
@@ -34,7 +34,7 @@ ${br}"

 footer="\
 ${br}
-See ./test-suite.log
+See ./test-suite.log for debugging.
 ${br}"

 echo AC_OUTPUT >> configure.ac
diff --git a/t/testsuite-summary-count.sh b/t/testsuite-summary-count.sh
index c8f15389d..9cd2fa58e 100644
--- a/t/testsuite-summary-count.sh
+++ b/t/testsuite-summary-count.sh
@@ -32,8 +32,7 @@ success_footer=${br}

 failure_footer="\
 ${br}
-See ./test-suite.log
-Please report to [email-masked]
+$failure_footer_text
 ${br}"

 # Corner cases.
diff --git a/t/testsuite-summary-reference-log.sh b/t/testsuite-summary-reference-log.sh
index 64cc4eb6c..f27e6fc39 100644
--- a/t/testsuite-summary-reference-log.sh
+++ b/t/testsuite-summary-reference-log.sh
@@ -46,11 +46,11 @@ cd build
 ../configure

 run_make -O -e FAIL check
-grep '^See \./my_test_suite\.log$' stdout
+grep '^See \./my_test_suite\.log for debugging\.$' stdout

 mkdir bar
 run_make -O -e FAIL TEST_SUITE_LOG=bar/bar.log check
-grep '^See \./bar/bar\.log$' stdout
+grep '^See \./bar/bar\.log for debugging\.$' stdout

 cd ..

@@ -70,13 +70,13 @@ $AUTOMAKE

 ./configure
 run_make -O -e FAIL check
-grep '^See sub/test-suite\.log$' stdout
+grep '^See sub/test-suite\.log for debugging\.$' stdout
 cd sub
 run_make -O -e FAIL check
-grep '^See sub/test-suite\.log$' stdout
+grep '^See sub/test-suite\.log for debugging\.$' stdout
 cd ..

 run_make -O -e FAIL TEST_SUITE_LOG=foo.log check
-grep '^See sub/foo\.log$' stdout
+grep '^See sub/foo\.log for debugging\.$' stdout

 :

compile finished at Wed Jan 31 10:56:25 2024