GNU bug report logs - #11855
Enabling the 'color-tests' option by default

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 11855 in the body.
You can then email your comments to 11855 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: bug-automake <at> gnu.org
Subject: Enabling the 'color-tests' option by default
Date: Wed, 04 Jul 2012 18:29:06 +0200

Severity: wishlist

Hello automakers.

Anyone has an idea about why the 'color-tests' option should be not
enabled by default (apart from the obvious historical reasons, that
is)?

Even when this option is on, the developer is free to disable testsuite
coloring on a per-makefile basis (by adding "AM_COLOR_TESTS = no" to the
Makefile.am) or on a whole-project basis (by AC_SUBST'ing AM_COLOR_TESTS
to "no" in configure.ac).  And the best thing about these approaches is
that they still allow the user to request the testsuite output to be
colorized, if he really wants to:

    # With GNU make:
    make AM_COLOR_TESTS=yes check
    # With non-GNU make:
    make AM_COLOR_TESTS=yes AM_MAKEFLAGS=AM_COLOR_TESTS=yes check

So I think we should make the 'color-tests' option enabled by default (while
keeping it around as a no-op, for backward-compatibility), and document the
two tricks above for developers who (for one reason or another) don't want
testsuite colorization enabled by default.

WDYT?

Regards,
  Stefano

Message #8 received at 11855 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: 11855 <at> debbugs.gnu.org
Cc: automake-patches <at> gnu.org
Subject: [PATCH 0/2] Enable colorized testsuite output by default
Date: Sun, 22 Jul 2012 17:31:25 +0200

Reference: <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11855>

The actual implementation is done by the first patch.  The second on
is just a minor testsuite cleanup "while we are at it".

I intend to push this series by tomorrow.

Regards,
  Stefano

Stefano Lattarini (2):
  check: support colorized testsuite output by default
  tests: better name for a couple of tests

 NEWS                             |    7 +++++++
 automake.in                      |    1 -
 doc/automake.texi                |   37 +++++++++++++++---------------------
 lib/am/check.am                  |    5 +----
 t/ax/tap-summary-aux.sh          |    1 -
 t/ax/testsuite-summary-checks.sh |    4 +---
 t/color-tests-opt.sh             |   39 ++++++++++++++++++++++++++++++++++++++
 t/{color.sh => color-tests.sh}   |    1 -
 t/{color2.sh => color-tests2.sh} |   18 ++++++++++++++++--
 t/list-of-tests.mk               |    5 +++--
 t/tap-realtime.sh                |    2 ++
 11 files changed, 84 insertions(+), 36 deletions(-)
 create mode 100755 t/color-tests-opt.sh
 rename t/{color.sh => color-tests.sh} (99%)
 rename t/{color2.sh => color-tests2.sh} (91%)

-- 
1.7.10.4

Message #11 received at 11855 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: 11855 <at> debbugs.gnu.org
Cc: automake-patches <at> gnu.org
Subject: [PATCH 1/2] check: support colorized testsuite output by default
Date: Sun, 22 Jul 2012 17:31:26 +0200

Fixes automake bug#11855.

With this change, we make the 'color-tests' option enabled by default.
That option remains still silently accepted as a no-op, for backward
compatibility.

The developer of a package is still free to disable testsuite coloring on
a per-makefile basis (by adding "AM_COLOR_TESTS = no" to the Makefile.am)
or on a whole-project basis (by AC_SUBST'ing AM_COLOR_TESTS to "no" in
configure.ac).  But now the user will be able to request the testsuite
output to be colorized, if he really wants to:

    # With GNU make:
    make AM_COLOR_TESTS=yes check
    # With non-GNU make:
    make AM_COLOR_TESTS=yes AM_MAKEFLAGS=AM_COLOR_TESTS=yes check

* NEWS: Update.
* doc/automake.texi: Updated, and some related minor reformatting
and rewording.
* automake.in (handle_tests): No need to pass the transform '%COLOR%'
when processing "check.am".
* lib/am/parallel-tests.am: Remove use of the '%?COLOR%' transform;
just act as if it were unconditionally TRUE.
* t/color.sh: No need to specify 'color-tests' in 'AUTOMAKE_OPTIONS'
nor in 'AM_INIT_AUTOMAKE'.
* t/ax/testsuite-summary-checks.sh: Likewise.
* t/ax/tap-summary-aux.sh: Likewise.
* t/color2.sh: Likewise.  Also, ensure that colorized testsuite output
can be disabled by default by calling "AC_SUBST([AM_COLOR_TESTS], [no])".
* t/tap-realtime.sh: Define 'AM_COLOR_TESTS' to "no" in Makefile.am, to
avoid spurious colorization of the output due to the use of the 'expect'
program.
* t/color-tests-opt.sh: New test, check that the 'color-tests' option is
still recognized as a no-op.
* t/list-of-tests.mk: Add the new test.

Signed-off-by: Stefano Lattarini <stefano.lattarini <at> gmail.com>
---
 NEWS                             |    7 +++++++
 automake.in                      |    1 -
 doc/automake.texi                |   37 +++++++++++++++---------------------
 lib/am/check.am                  |    5 +----
 t/ax/tap-summary-aux.sh          |    1 -
 t/ax/testsuite-summary-checks.sh |    4 +---
 t/color-tests-opt.sh             |   39 ++++++++++++++++++++++++++++++++++++++
 t/color.sh                       |    1 -
 t/color2.sh                      |   18 ++++++++++++++++--
 t/list-of-tests.mk               |    1 +
 t/tap-realtime.sh                |    2 ++
 11 files changed, 82 insertions(+), 34 deletions(-)
 create mode 100755 t/color-tests-opt.sh

diff --git a/NEWS b/NEWS
index 6c39a7d..3d7b2bf 100644
--- a/NEWS
+++ b/NEWS
@@ -51,6 +51,13 @@ New in 1.13:
     testsuite harness will still be available through the use of the
     'serial-tests' option (introduced in Automake 1.12).
 
+  - The 'color-tests' option is now unconditionally activated by default.
+    In particular, this means that testsuite output is now colorized by
+    default if the attached terminal seems to support ANSI escapes, and
+    that the user can force output colorization by setting the variable
+    AM_COLOR_TESTS to "always".  The 'color-tests' is still recognized
+    for backward-compatibility, although it's a handled as a no-op now.
+
 * Silent rules support:
 
   - Support for silent rules is now always active in Automake-generated
diff --git a/automake.in b/automake.in
index 15d545d..f2a6da5 100644
--- a/automake.in
+++ b/automake.in
@@ -4866,7 +4866,6 @@ sub handle_tests
       push (@check_tests, 'check-TESTS');
       my $check_deps = "@check";
       $output_rules .= &file_contents ('check', new Automake::Location,
-				       COLOR => !! option 'color-tests',
 				       SERIAL_TESTS => !! option 'serial-tests',
                                        CHECK_DEPS => $check_deps);
 
diff --git a/doc/automake.texi b/doc/automake.texi
index 9ef7e9f..251c26b 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8843,18 +8843,18 @@ run.
 @anchor{Simple tests and color-tests}
 @vindex AM_COLOR_TESTS
 @cindex Colorized testsuite output
-If the Automake option @code{color-tests} is used (@pxref{Options})
-and standard output is connected to a capable terminal, then the test
-results and the summary are colored appropriately.  The user can disable
-colored output by setting the @command{make} variable
-@samp{AM_COLOR_TESTS=no}, or force colored output even without a connecting
-terminal with @samp{AM_COLOR_TESTS=always}.  It's also worth noting that
-some @command{make} implementations, when used in parallel mode, have
-slightly different semantics (@pxref{Parallel make,,, autoconf,
-The Autoconf Manual}), which can break the automatic detection of a
-connection to a capable terminal.  If this is the case, you'll have to
-resort to the use of @samp{AM_COLOR_TESTS=always} in order to have the
-testsuite output colorized.
+If the standard output is connected to a capable terminal, then the test
+results and the summary are colored appropriately.  The developer and the
+user can disable colored output by setting the @command{make} variable
+@samp{AM_COLOR_TESTS=no}; the user can in addition force colored output
+even without a connecting terminal with @samp{AM_COLOR_TESTS=always}.  
+It's also worth noting that some @command{make} implementations,
+when used in parallel mode, have slightly different semantics
+(@pxref{Parallel make,,, autoconf, The Autoconf Manual}), which can
+break the automatic detection of a connection to a capable terminal.
+If this is the case, the user will have to resort to the use of
+@samp{AM_COLOR_TESTS=always} in order to have the testsuite output
+colorized.
 
 Test programs that need data files should look for them in @code{srcdir}
 (which is both a make variable and an environment variable made available
@@ -9279,9 +9279,9 @@ definition of generic and extension-specific @code{LOG_COMPILER} and
 @end itemize
 
 @noindent
-On the other hand, the exact semantics of how (and if)
-@option{color-tests}, @code{XFAIL_TESTS}, and hard errors are supported
-and handled is left to the individual test drivers.
+On the other hand, the exact semantics of how (and if) testsuite output
+colorization, @code{XFAIL_TESTS}, and hard errors are supported and
+handled is left to the individual test drivers.
 
 @c TODO: We should really add a working example in the doc/ directory,
 @c TODO: and reference if from here.
@@ -9973,13 +9973,6 @@ implies options @option{readme-alpha} and @option{check-news}.
 Cause @samp{make dist} to fail unless the current version number appears
 in the first few lines of the @file{NEWS} file.
 
-@item @option{color-tests}
-@cindex Option, @option{color-tests}
-@opindex color-tests
-Cause output of the serial and parallel test harnesses (see @ref{Simple
-Tests}) and of properly-written custom test drivers (@pxref{Custom Test
-Drivers}) to be colorized on capable terminals.
-
 @item @option{dejagnu}
 @cindex Option, @option{dejagnu}
 @opindex dejagnu
diff --git a/lib/am/check.am b/lib/am/check.am
index 0bbe2ee..c1e8aac 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -17,7 +17,7 @@
 am__tty_colors_dummy = \
   mgn= red= grn= lgn= blu= brg= std=; \
   am__color_tests=no
-if %?COLOR%
+
 am__tty_colors = { \
   $(am__tty_colors_dummy); \
   if test "X$(AM_COLOR_TESTS)" = Xno; then \
@@ -40,9 +40,6 @@ am__tty_colors = { \
     std=''; \
   fi; \
 }
-else !%?COLOR%
-am__tty_colors = $(am__tty_colors_dummy)
-endif !%?COLOR%
 
 .PHONY: check-TESTS
 
diff --git a/t/ax/tap-summary-aux.sh b/t/ax/tap-summary-aux.sh
index 25fedfe..451020c 100644
--- a/t/ax/tap-summary-aux.sh
+++ b/t/ax/tap-summary-aux.sh
@@ -89,7 +89,6 @@ if test $use_colors = yes; then
   mgn="$esc[0;35m"
   brg="$esc[1m"
   std="$esc[m"
-  echo AUTOMAKE_OPTIONS = color-tests >> Makefile.am
 else
   red= grn= lgn= blu= mgn= brg= std=
 fi
diff --git a/t/ax/testsuite-summary-checks.sh b/t/ax/testsuite-summary-checks.sh
index 9137c22..6eafdc9 100644
--- a/t/ax/testsuite-summary-checks.sh
+++ b/t/ax/testsuite-summary-checks.sh
@@ -28,10 +28,8 @@ case $use_colors in
     # Forced colorization should take place also with non-ANSI
     # terminals; hence this setting.
     TERM=dumb; export TERM
-    am_opts='color-tests'
     ;;
   no)
-    am_opts=''
     ;;
   *)
     fatal_ "invalid use_colors='$use_colors'";;
@@ -39,7 +37,7 @@ esac
 
 cat > configure.ac <<END
 AC_INIT([GNU AutoFoo], [7.1], [bug-automake <at> gnu.org])
-AM_INIT_AUTOMAKE([$am_opts])
+AM_INIT_AUTOMAKE
 AC_CONFIG_FILES([Makefile])
 AC_OUTPUT
 END
diff --git a/t/color-tests-opt.sh b/t/color-tests-opt.sh
new file mode 100755
index 0000000..87d88c9
--- /dev/null
+++ b/t/color-tests-opt.sh
@@ -0,0 +1,39 @@
+#! /bin/sh
+# Copyright (C) 2007-2012 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the 'color-tests' option, now active by default, is
+# nonetheless still silently accepted, for backward compatibility.
+
+. ./defs || exit 1
+
+cat >configure.ac <<END
+AC_INIT([$me], [1.0])
+AM_INIT_AUTOMAKE([color-tests])
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+END
+
+cat >Makefile.am <<END
+AUTOMAKE_OPTIONS = color-tests
+TESTS = foo.test
+END
+
+: > test-driver
+
+$ACLOCAL
+$AUTOMAKE
+
+:
diff --git a/t/color.sh b/t/color.sh
index 61158b1..fafa403 100755
--- a/t/color.sh
+++ b/t/color.sh
@@ -34,7 +34,6 @@ AC_OUTPUT
 END
 
 cat >Makefile.am <<'END'
-AUTOMAKE_OPTIONS = color-tests
 TESTS = $(check_SCRIPTS)
 check_SCRIPTS = pass fail skip xpass xfail error
 XFAIL_TESTS = xpass xfail
diff --git a/t/color2.sh b/t/color2.sh
index e79ab27..65ca6c7 100755
--- a/t/color2.sh
+++ b/t/color2.sh
@@ -58,12 +58,14 @@ rm -f expect-check Makefile
 
 # Do the tests.
 
-cat >>configure.ac <<END
+cat >>configure.ac << 'END'
+if $testsuite_colorized; then :; else
+  AC_SUBST([AM_COLOR_TESTS], [no])
+fi
 AC_OUTPUT
 END
 
 cat >Makefile.am <<'END'
-AUTOMAKE_OPTIONS = color-tests
 TESTS = $(check_SCRIPTS)
 check_SCRIPTS = pass fail skip xpass xfail error
 XFAIL_TESTS = xpass xfail
@@ -182,6 +184,18 @@ for vpath in false :; do
   cat stdout
   test_no_color
 
+  $srcdir/configure testsuite_colorized=false
+
+  TERM=ansi MAKE=$MAKE expect -f $srcdir/expect-make >stdout \
+    || { cat stdout; exit 1; }
+  cat stdout
+  test_no_color
+
+  TERM=ansi MAKE="env AM_COLOR_TESTS=always $MAKE" \
+    expect -f $srcdir/expect-make >stdout || { cat stdout; exit 1; }
+  cat stdout
+  test_color
+
   $MAKE distclean
   cd $srcdir
 
diff --git a/t/list-of-tests.mk b/t/list-of-tests.mk
index 2a28992..91f32cc 100644
--- a/t/list-of-tests.mk
+++ b/t/list-of-tests.mk
@@ -229,6 +229,7 @@ t/colon6.sh \
 t/colon7.sh \
 t/color.sh \
 t/color2.sh \
+t/color-tests-opt.sh \
 t/comment.sh \
 t/comment2.sh \
 t/comment3.sh \
diff --git a/t/tap-realtime.sh b/t/tap-realtime.sh
index ecd3bef..97b035c 100755
--- a/t/tap-realtime.sh
+++ b/t/tap-realtime.sh
@@ -53,7 +53,9 @@ fi
 
 cat > Makefile.am << 'END'
 TESTS = all.test
+AM_COLOR_TESTS= no
 END
+
 . "$am_testauxdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
 
 cat > all.test <<'END'
-- 
1.7.10.4

Message #14 received at 11855 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: 11855 <at> debbugs.gnu.org
Cc: automake-patches <at> gnu.org
Subject: [PATCH 2/2] tests: better name for a couple of tests
Date: Sun, 22 Jul 2012 17:31:27 +0200

* t/color.sh: Rename ...
* t/color-tests.sh: ... like this.
* t/color2.sh: Rename ...
* t/color-tests2.sh: ... like this.
* t/list-of-tests.mk: Adjust.

Signed-off-by: Stefano Lattarini <stefano.lattarini <at> gmail.com>
---
 t/{color.sh => color-tests.sh}   |    0
 t/{color2.sh => color-tests2.sh} |    0
 t/list-of-tests.mk               |    4 ++--
 3 files changed, 2 insertions(+), 2 deletions(-)
 rename t/{color.sh => color-tests.sh} (100%)
 rename t/{color2.sh => color-tests2.sh} (100%)

diff --git a/t/color.sh b/t/color-tests.sh
similarity index 100%
rename from t/color.sh
rename to t/color-tests.sh
diff --git a/t/color2.sh b/t/color-tests2.sh
similarity index 100%
rename from t/color2.sh
rename to t/color-tests2.sh
diff --git a/t/list-of-tests.mk b/t/list-of-tests.mk
index 91f32cc..9ab8f28 100644
--- a/t/list-of-tests.mk
+++ b/t/list-of-tests.mk
@@ -227,8 +227,8 @@ t/colon4.sh \
 t/colon5.sh \
 t/colon6.sh \
 t/colon7.sh \
-t/color.sh \
-t/color2.sh \
+t/color-tests.sh \
+t/color-tests2.sh \
 t/color-tests-opt.sh \
 t/comment.sh \
 t/comment2.sh \
-- 
1.7.10.4

Message #17 received at 11855 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: 11855 <at> debbugs.gnu.org
Cc: control <at> debbugs.gnu.org, automake-patches <at> gnu.org
Subject: Re: [PATCH 0/2] Enable colorized testsuite output by default
Date: Tue, 24 Jul 2012 19:48:05 +0200

tags 11855 + patch
close 11855
thanks

On 07/22/2012 05:31 PM, Stefano Lattarini wrote:
> Reference: <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11855>
> 
> The actual implementation is done by the first patch.  The second on
> is just a minor testsuite cleanup "while we are at it".
> 
> I intend to push this series by tomorrow.
> 
> Regards,
>   Stefano
> 
> Stefano Lattarini (2):
>   check: support colorized testsuite output by default
>   tests: better name for a couple of tests
> 
>  NEWS                             |    7 +++++++
>  automake.in                      |    1 -
>  doc/automake.texi                |   37 +++++++++++++++---------------------
>  lib/am/check.am                  |    5 +----
>  t/ax/tap-summary-aux.sh          |    1 -
>  t/ax/testsuite-summary-checks.sh |    4 +---
>  t/color-tests-opt.sh             |   39 ++++++++++++++++++++++++++++++++++++++
>  t/{color.sh => color-tests.sh}   |    1 -
>  t/{color2.sh => color-tests2.sh} |   18 ++++++++++++++++--
>  t/list-of-tests.mk               |    5 +++--
>  t/tap-realtime.sh                |    2 ++
>  11 files changed, 84 insertions(+), 36 deletions(-)
>  create mode 100755 t/color-tests-opt.sh
>  rename t/{color.sh => color-tests.sh} (99%)
>  rename t/{color2.sh => color-tests2.sh} (91%)
> 
Pushed.  I'm thus closing this bug report.

Regards,
  Stefano

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.