GNU bug report logs - #7868
parallel-tests and command-line length limit issue

Previous Next

To reply to this bug, email your comments to 7868 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>
To: bug-automake <at> gnu.org
Subject: splitting up test suites
Date: Wed, 19 Jan 2011 23:22:19 +0100

The testsuite is too large for MSYS.

We've finally reached the point where we have more than 1000
tests, $(TESTS) expands to 15k characters, and where 'make check' will
not work at all any more on MSYS, because it cannot spawn sh any more,
presumably in 'make check TESTS="..."'.  (MSYS make doesn't export
macros to the environment of spawned processes even without .NOEXPORT,
presumably otherwise lots of Makefiles would be really unusable here.)
This also clears up the spurious failure of sed a few days ago.


Here's a preliminary plan for multiple testsuites per Makefile.am.
It would be nice if this worked somehow:

# These are all specified by the user:
TEST_SUITE_LOGS = suite1.log suite2.log suite3.log ..
TEST_EXTENSIONS = .test ...
# these undergo $(EXEEXT) autoexpansion internally:
suite1_log_SOURCES = all.test aclocal7.test ...
suite2_log_SOURCES = suffix.test ...

# These are then produced by automake:
TEST_SUITE_HTMLS = $(TEST_SUITE_LOGS:.log=.html)
suite1_log_LOGS = $(am__helper_var:.test=.log)

mostlyclean-generic:
	-test -z "$(suite1_log_LOGS)" || rm -f $(suite1_log_LOGS)
	-test -z "$(suite2_log_LOGS)" || rm -f $(suite2_log_LOGS)


In the <suite>_SOURCES, $(EXEEXT) transformation should take place
(unless no-exeext is given, of course), just as is currently done for
TESTS and *_PROGRAMS.  Hmm, alternatively we could also require all
<suite>_SOURCES to be listed in $(TESTS), that would allow reuse of
this variable as well, at the cost of some specification redundance.

Open questions: how to produce nice results, both on stdout and in suite
log files.  One way is to merge all logs into a final TEST_SUITE_LOG
(that way we could also reuse that variable).  Another is to try to even
hide the summaries of the individual suites, iff a final suite is being
made.

Internally, it would be nice if the register_language could be exploited
(but this is not a requirement for this, as some features with tests are
distinctly different from other languages).

Cheers,
Ralf

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: bug-automake <at> gnu.org
Cc: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>, 7868 <at> debbugs.gnu.org
Subject: Re: bug#7868: splitting up test suites
Date: Thu, 20 Jan 2011 12:38:17 +0100

Hello Ralf.

On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> The testsuite is too large for MSYS.
>
Ouch.

> We've finally reached the point where we have more than 1000
> tests, $(TESTS) expands to 15k characters, and where 'make check' will
> not work at all any more on MSYS, because it cannot spawn sh any more,
> presumably in 'make check TESTS="..."'.  (MSYS make doesn't export
> macros to the environment of spawned processes even without .NOEXPORT,
> presumably otherwise lots of Makefiles would be really unusable here.)
> This also clears up the spurious failure of sed a few days ago.
>  
> Here's a preliminary plan for multiple testsuites per Makefile.am.
>
Hmmm... while this feature might be worth having even indipendently
from the issue at hand (but see below for small nits), I still think
that in the long run it would be nicer to transparently work around
such command-line length issues in the test driver, if possible.  Do
you think your patch "parallel-tests: avoid command-line length limit
issue" (from commit v1.11-191-g24e3b4e) could be resurrected in some
way?

> It would be nice if this worked somehow:
> 
> # These are all specified by the user:
> TEST_SUITE_LOGS = suite1.log suite2.log suite3.log ..
> TEST_EXTENSIONS = .test ...
> # these undergo $(EXEEXT) autoexpansion internally:
> suite1_log_SOURCES = all.test aclocal7.test ...
> suite2_log_SOURCES = suffix.test ...
>
What about using `TESTS' instead of `SOURCES' in these two last variables?
That would seems more natural to me, especially considering the API of the
current parallel-tests driver.

OTOH, would that maybe make the implementation more difficult?

> # These are then produced by automake:
> TEST_SUITE_HTMLS = $(TEST_SUITE_LOGS:.log=.html)
> suite1_log_LOGS = $(am__helper_var:.test=.log)
>
> mostlyclean-generic:
> 	-test -z "$(suite1_log_LOGS)" || rm -f $(suite1_log_LOGS)
> 	-test -z "$(suite2_log_LOGS)" || rm -f $(suite2_log_LOGS)
>
 
I think it would also be nice to generate separate check/recheck targets
for each testsuite; for example, "make check-suite1" could run all the
tests in $(suite1_log_SOURCES), and "make recheck-suite2" could re-run
all the tests in $(suite2_log_SOURCES) that failed (or xfailed) in the
previous run.

Hmm... No, wait, it would be even nicer to allow the user choose which
testsuite(s) to run by resetting the $(TEST_SUITE_LOGS) variable:

  make check TEST_SUITE_LOGS='suite1.log suite2.log'

Which makes me think that, perhaps, a variable like $(TEST_SUITES) would
be preferable:

  make check TEST_SUITES='suite1 suite2'

but then the API should be changed to something like:

 # These are all specified by the user:
 TEST_SUITES = suite1 suite2 suite3 ..
 TEST_EXTENSIONS = .test ...
 # these undergo $(EXEEXT) autoexpansion internally:
 suite1_TESTS = all.test aclocal7.test ...
 suite2_TESTS = suffix.test ...

 # These are then produced by automake:
 TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
 TEST_SUITE_HTMLS = $(TEST_SUITES:=.html)
 suite1_LOGS = $(am__helper_var:.test=.log)

> In the <suite>_SOURCES, $(EXEEXT) transformation should take place
> (unless no-exeext is given, of course), just as is currently done for
> TESTS and *_PROGRAMS.  Hmm, alternatively we could also require all
> <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of
> this variable as well, at the cost of some specification redundance.
>

> Open questions: how to produce nice results, both on stdout and in suite
> log files.  One way is to merge all logs into a final TEST_SUITE_LOG
> (that way we could also reuse that variable).  Another is to try to even
> hide the summaries of the individual suites, iff a final suite is being
> made.
>
These both sound sensible.  Another problem is how to avoid that, in a
parallel make run, the summary from a testsuite gets mixed and garbled
with the output/summary from another testsuite, as in e.g.:

 suite1_log_SOURCES = all.test aclocal7.test
 suite2_log_SOURCES = suffix.test yacc-dist-nobuild.test

 $ make check TEST_SUITE_LOGS='suite1.log suite2.log'
 XFAIL: all.test
 PASS: aclocal7.test
 ====================================================
 PASS: suffix.test
 All 2 tests behaved as expected (1 expected failure)
 FAIL: yacc-dist-nobuild.test
 =====================================
 1 of 2 tests failed
 ====================================================
 See tests/suite2.log
 Please report to bug-automake <at> gnu.org
 =====================================

(yuck!), where we'd want at least:

 $ make check TEST_SUITE_LOGS='suite1.log suite2.log'
 XFAIL: all.test
 PASS: suffix.test
 PASS: aclocal7.test
 ====================================================
 All 2 tests behaved as expected (1 expected failure)
 ====================================================
 FAIL: yacc-dist-nobuild.test
 =====================================
 1 of 2 tests failed
 See tests/suite2.log
 Please report to bug-automake <at> gnu.org
 =====================================

or better again:

 $ make check TEST_SUITE_LOGS='suite1.log suite2.log'
 XFAIL: all.test
 PASS: suffix.test
 PASS: aclocal7.test
 FAIL: yacc-dist-nobuild.test

 ==========================================================
 suite1: All tests behaved as expected (1 expected failure)
 ==========================================================

 =====================================
 suite2: 1 of 2 tests failed
 See tests/suite2.log
 Please report to bug-automake <at> gnu.org
 =====================================

> Internally, it would be nice if the register_language could be exploited
> (but this is not a requirement for this, as some features with tests are
> distinctly different from other languages).
> 
> Cheers,
> Ralf
> 

Thanks for tackling this!

Regards,
   Stefano

Message #14 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: automake-patches <at> gnu.org
Cc: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>, 7868 <at> debbugs.gnu.org
Subject: [RFC] parallel-tests: avoid command-line length limit issue (was: Re:
	bug#7868: splitting up test suites)
Date: Thu, 20 Jan 2011 18:11:00 +0100

[Message part 1 (text/plain, inline)]

On Thursday 20 January 2011, Stefano Lattarini wrote:
> Hello Ralf.
> 
> On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> > The testsuite is too large for MSYS.
> >
> Ouch.
> 
> > We've finally reached the point where we have more than 1000
> > tests, $(TESTS) expands to 15k characters, and where 'make check' will
> > not work at all any more on MSYS, because it cannot spawn sh any more,
> > presumably in 'make check TESTS="..."'.  (MSYS make doesn't export
> > macros to the environment of spawned processes even without .NOEXPORT,
> > presumably otherwise lots of Makefiles would be really unusable here.)
> > This also clears up the spurious failure of sed a few days ago.
> >  
> > Here's a preliminary plan for multiple testsuites per Makefile.am.
> >
> Hmmm... while this feature might be worth having even indipendently
> from the issue at hand (but see below for small nits), I still think
> that in the long run it would be nicer to transparently work around
> such command-line length issues in the test driver, if possible.  Do
> you think your patch "parallel-tests: avoid command-line length limit
> issue" (from commit v1.11-191-g24e3b4e) could be resurrected in some
> way?
>
OK, I've done my homework and come up with the attached patch.

It's not yet very well polished, and certainly needs more testsuite
exposure [1] on various systems before being "ok to apply", but it
seems quite promising to me.

 [1] I'm posting it now anyway because I'm out of time for today.
     Sorry.

Also, I'd appreciate if anyone could test it on MSYS (and Cygwin?),
since I don't have access to those systems.

Thanks,
  Stefano

[0001-parallel-tests-avoid-command-line-length-limit-issue.patch (text/x-patch, inline)]

From f6f4dc5d2e6e3d174f696409fa5c07e207d377a4 Mon Sep 17 00:00:00 2001
From: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>
Date: Tue, 7 Sep 2010 04:38:08 +0200
Subject: [PATCH] parallel-tests: avoid command-line length limit issue.

* automake.in (handle_tests): New argument $makefile, new
substitution %MAKEFILE%.
(generate_makefile): Adjust.
* lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Use a
temporary makefile to sanitize TEST_LOGS value passed to the
recursive $(MAKE) invocation, to avoid exceeding the command
line limit on w32 (MSYS).  Extend comments.
* tests/parallel-tests-linewrap.test: New test.
* tests/parallel-tests-cleanup.test: Likewise.
* tests/parallel-tests-gnumakefile.test: Likewise.
* tests/parallel-tests-long-cmdline.test: Likewise.
* tests/Makefile.am (TESTS): Updated.
* NEWS: Update.

Report by Bob Friesenhahn.
---
 ChangeLog                             |   19 +++++++
 NEWS                                  |    3 +
 automake.in                           |   11 +++-
 lib/Automake/tests/Makefile.in        |   21 ++++++--
 lib/am/check.am                       |   35 +++++++++++--
 tests/Makefile.am                     |    3 +
 tests/Makefile.in                     |   24 +++++++--
 tests/parallel-tests-cleanup.test     |   91 +++++++++++++++++++++++++++++++++
 tests/parallel-tests-gnumakefile.test |   49 ++++++++++++++++++
 tests/parallel-tests-linewrap.test    |   63 +++++++++++++++++++++++
 10 files changed, 301 insertions(+), 18 deletions(-)
 create mode 100755 tests/parallel-tests-cleanup.test
 create mode 100755 tests/parallel-tests-gnumakefile.test
 create mode 100755 tests/parallel-tests-linewrap.test

diff --git a/ChangeLog b/ChangeLog
index 31ff009..4a0e7e1 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,22 @@
+2011-01-20  Stefano Lattarini  <stefano.lattarini <at> gmail.com>
+	    Ralf Wildenhues  <Ralf.Wildenhues <at> gmx.de>
+
+	parallel-tests: avoid command-line length limit issue.
+	* automake.in (handle_tests): New argument $makefile, new
+	substitution %MAKEFILE%.
+	(generate_makefile): Adjust.
+	* lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Use a
+	temporary makefile to sanitize TEST_LOGS value passed to the
+	recursive $(MAKE) invocation, to avoid exceeding the command
+	line limit on w32 (MSYS).  Extend comments.
+	* tests/parallel-tests-linewrap.test: New test.
+	* tests/parallel-tests-cleanup.test: Likewise.
+	* tests/parallel-tests-gnumakefile.test: Likewise.
+	* tests/parallel-tests-long-cmdline.test: Likewise.
+	* tests/Makefile.am (TESTS): Updated.
+	* NEWS: Update.
+	Report by Bob Friesenhahn.
+
 2011-01-19  Stefano Lattarini  <stefano.lattarini <at> gmail.com>
 	    Ralf Wildenhues  <Ralf.Wildenhues <at> gmx.de>
 
diff --git a/NEWS b/NEWS
index b5cb6e9..c5f28b5 100644
--- a/NEWS
+++ b/NEWS
@@ -22,6 +22,9 @@ Bugs fixed in 1.11.0a:
   - The AM_COND_IF macro also works if the shell expression for the conditional
     is no longer valid for the condition.
 
+  - The `parallel-tests' driver works around a problem with command-line
+    length limits with `make check' on w32 (MSYS).
+
 * Long standing bugs:
 
   - On Darwin 9, `pythondir' and `pyexecdir' pointed below `/Library/Python'
diff --git a/automake.in b/automake.in
index d56fbf7..bd9f453 100755
--- a/automake.in
+++ b/automake.in
@@ -4919,9 +4919,13 @@ sub handle_tests_dejagnu
 }
 
 
+# handle_tests ($MAKEFILE)
+# ------------------------
 # Handle TESTS variable and other checks.
-sub handle_tests
+sub handle_tests ($)
 {
+  my ($makefile) = @_;
+
   if (option 'dejagnu')
     {
       &handle_tests_dejagnu;
@@ -4940,7 +4944,8 @@ sub handle_tests
       push (@check_tests, 'check-TESTS');
       $output_rules .= &file_contents ('check', new Automake::Location,
 				       COLOR => !! option 'color-tests',
-				       PARALLEL_TESTS => !! option 'parallel-tests');
+				       PARALLEL_TESTS => !! option 'parallel-tests',
+				       MAKEFILE => basename $makefile);
 
       # Tests that are known programs should have $(EXEEXT) appended.
       # For matching purposes, we need to adjust XFAIL_TESTS as well.
@@ -8220,7 +8225,7 @@ sub generate_makefile ($$)
   handle_tags;
   handle_minor_options;
   # Must come after handle_programs so that %known_programs is up-to-date.
-  handle_tests;
+  handle_tests ($makefile);
 
   # This must come after most other rules.
   handle_dist;
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index b4940db..d3846aa 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -403,11 +403,22 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 check-TESTS:
 	@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
 	@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
-	@list='$(TEST_LOGS)';						\
-	list=`for f in $$list; do					\
-	  test .log = $$f || echo $$f;					\
-	done | tr '\012\015' '  '`;					\
-	$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$list"
+	@am__tmpdir=am-tmp$$$$; 					\
+	am__mkfile=$$am__tmpdir/Makefile;				\
+	am__trap='rm -rf $$am__tmpdir; (exit $$st); exit $$st'; 	\
+	trap "st=129; $$am__trap" 1; trap "st=130; $$am__trap" 2;	\
+	trap "st=141; $$am__trap" 13; trap "st=143; $$am__trap" 15; 	\
+	st=0; 								\
+	mkdir $$am__tmpdir						\
+	  && { echo "TEST_LOGS = \\";					\
+		list='$(TEST_LOGS)'; for f in $$list; do		\
+		   test .log = $$f || echo "$$f \\";			\
+		done;							\
+	     } | sed '$$s/ *\\$$//' > $$am__mkfile			\
+	  && sed '/^TEST_LOGS =/d' Makefile >> $$am__mkfile		\
+	  && $(MAKE) -f $$am__mkfile $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) 	\
+	  || st=$$?; 							\
+	rm -rf $$am__tmpdir; exit $$st
 
 .log.html:
 	@list='$(RST2HTML) $$RST2HTML rst2html rst2html.py';		\
diff --git a/lib/am/check.am b/lib/am/check.am
index 5728081..837f1b4 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -236,11 +236,36 @@ check-TESTS:
 ## cannot use `$?' to compute the set of lazily rerun tests, lest
 ## we rely on .PHONY to work portably.
 	@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
-	@list='$(TEST_LOGS)';						\
-	list=`for f in $$list; do					\
-	  test .log = $$f || echo $$f;					\
-	done | tr '\012\015' '  '`;					\
-	$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$list"
+## We'll need a temporary file (see comments below for why), and the safer
+## way to create it portably is to use a temporary directory (as the mkdir
+## utility should be truly atomic everywhere).
+	@am__tmpdir=am-tmp$$$$; 					\
+	am__mkfile=$$am__tmpdir/Makefile;				\
+	am__trap='rm -rf $$am__tmpdir; (exit $$st); exit $$st'; 	\
+	trap "st=129; $$am__trap" 1; trap "st=130; $$am__trap" 2;	\
+	trap "st=141; $$am__trap" 13; trap "st=143; $$am__trap" 15; 	\
+	st=0; 								\
+	mkdir $$am__tmpdir						\
+## Yes, this is hacky, but we cannot simply override the $(TEST_LOGS)
+## definition by appending to Makefile, since at that point it's original
+## value has been already used in a dependency declaration, i.e.
+##   $(TEST_SUITE_LOG): $(TEST_LOGS)
+## (see above in this same *.am fragment).
+	  && { echo "TEST_LOGS = \\";					\
+		list='$(TEST_LOGS)'; for f in $$list; do		\
+## A bug in GNU make 3.80 can lead to bare `.log' occurrences.
+## Strip them out.
+		   test .log = $$f || echo "$$f \\";			\
+		done;							\
+	     } | sed '$$s/ *\\$$//' > $$am__mkfile			\
+	  && sed '/^TEST_LOGS =/d' %MAKEFILE% >> $$am__mkfile		\
+## It would have been nice to be able to use something like:
+##   $(POST_PROCESSING_COMMAND) Makefile.in | $(MAKE) -f -
+## instead of a temporary file here, but unfortunately that doesn't
+## work with at least Solaris 10 dmake.
+	  && $(MAKE) -f $$am__mkfile $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) 	\
+	  || st=$$?; 							\
+	rm -rf $$am__tmpdir; exit $$st
 
 AM_RECURSIVE_TARGETS += check
 
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 713dd92..f263764 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -567,6 +567,9 @@ parallel-tests7.test \
 parallel-tests8.test \
 parallel-tests9.test \
 parallel-tests10.test \
+parallel-tests-cleanup.test \
+parallel-tests-gnumakefile.test \
+parallel-tests-linewrap.test \
 parallel-tests-unreadable-log.test \
 parse.test \
 percent.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 45adb04..7491f05 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -834,6 +834,9 @@ parallel-tests7.test \
 parallel-tests8.test \
 parallel-tests9.test \
 parallel-tests10.test \
+parallel-tests-cleanup.test \
+parallel-tests-gnumakefile.test \
+parallel-tests-linewrap.test \
 parallel-tests-unreadable-log.test \
 parse.test \
 percent.test \
@@ -1222,11 +1225,22 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 check-TESTS:
 	@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
 	@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
-	@list='$(TEST_LOGS)';						\
-	list=`for f in $$list; do					\
-	  test .log = $$f || echo $$f;					\
-	done | tr '\012\015' '  '`;					\
-	$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$list"
+	@am__tmpdir=am-tmp$$$$; 					\
+	am__mkfile=$$am__tmpdir/Makefile;				\
+	am__trap='rm -rf $$am__tmpdir; (exit $$st); exit $$st'; 	\
+	trap "st=129; $$am__trap" 1; trap "st=130; $$am__trap" 2;	\
+	trap "st=141; $$am__trap" 13; trap "st=143; $$am__trap" 15; 	\
+	st=0; 								\
+	mkdir $$am__tmpdir						\
+	  && { echo "TEST_LOGS = \\";					\
+		list='$(TEST_LOGS)'; for f in $$list; do		\
+		   test .log = $$f || echo "$$f \\";			\
+		done;							\
+	     } | sed '$$s/ *\\$$//' > $$am__mkfile			\
+	  && sed '/^TEST_LOGS =/d' Makefile >> $$am__mkfile		\
+	  && $(MAKE) -f $$am__mkfile $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) 	\
+	  || st=$$?; 							\
+	rm -rf $$am__tmpdir; exit $$st
 
 .log.html:
 	@list='$(RST2HTML) $$RST2HTML rst2html rst2html.py';		\
diff --git a/tests/parallel-tests-cleanup.test b/tests/parallel-tests-cleanup.test
new file mode 100755
index 0000000..26e9d5c
--- /dev/null
+++ b/tests/parallel-tests-cleanup.test
@@ -0,0 +1,91 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the temporary files/directories used by the Makefile
+# post-processing code in the parallel-tests testsuite driver are
+# duly cleaned up on success, on failure, and when well-known signals
+# are received.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS = .sh
+TESTS = foo.sh
+SH_LOG_COMPILER = sh
+EXTRA_DIST = foo.sh
+END
+
+check_filelist ()
+{
+  expect_filelist=$1
+  got_filelist=`ls`
+  if test x"$expect_filelist" != x"$got_filelist"; then
+    # Display the differences in a more user-friendly way.
+    # Useful for debugging and failure analysis.
+    set +x # Don't be overly verbose.
+    for f in $expect_filelist; do echo $f; done > exp
+    for f in $got_filelist; do echo $f; done > got
+    set -x
+    diff exp got || :
+    return 1
+  else
+    return 0
+  fi
+}
+
+$ACLOCAL
+$AUTOMAKE -a
+$AUTOCONF
+
+./configure
+
+aborted=`(ls && echo foo.sh ) | sort`
+completed=`(echo "$aborted" && echo test-suite.log && echo foo.log) | sort`
+
+echo 'exit 0' > foo.sh
+$MAKE check
+check_filelist "$completed"
+$MAKE clean
+
+echo 'exit 1' > foo.sh
+$MAKE check && Exit 1
+check_filelist "$completed"
+$MAKE clean
+
+echo 'sleep 10' > foo.sh
+
+# Yes, all the "sleeps" below sucks, but here is better to play "dumb
+# and safer" than having stray I/O or leaving zombies around.
+for signum in  1 2 13 15; do 
+  failed=false
+  $MAKE check &
+  kill -$signum $!
+  check_filelist "$aborted" || failed=:
+  sleep 12 # Wait for all the children to complete.
+  $failed && Exit 1
+  $MAKE clean
+done
+
+$MAKE distcheck
+
+:
diff --git a/tests/parallel-tests-gnumakefile.test b/tests/parallel-tests-gnumakefile.test
new file mode 100755
index 0000000..60d6fd7
--- /dev/null
+++ b/tests/parallel-tests-gnumakefile.test
@@ -0,0 +1,49 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that recursive make calls triggered by parallel-tests testsuite
+# driver work also if the makefile is named GNUmakefile.
+
+required=GNUmake
+parallel_tests=yes
+. ./defs || Exit 1
+set -e
+
+cat configure.in - > t << 'END'
+AC_OUTPUT
+END
+sed '/^AC_CONFIG_FILES/s|Makefile|GNUmakefile|' t > configure.in
+rm -f t
+
+cat > GNUmakefile.am <<'END'
+TESTS = foo.test
+END
+
+cat > foo.test <<'END'
+#! /bin/sh
+exit 0
+END
+chmod +x foo.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+$MAKE check
+
+:
diff --git a/tests/parallel-tests-linewrap.test b/tests/parallel-tests-linewrap.test
new file mode 100755
index 0000000..b3969f0
--- /dev/null
+++ b/tests/parallel-tests-linewrap.test
@@ -0,0 +1,63 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the definition of TEST_LOGS is not broken on multiple lines,
+# even if the list of tests, the list of test extensions, the test names
+# and the test extensions are all very long.
+# We need to be sure of this in order for the Makefile post-processing
+# code in the parallel-tests testsuite driver to work.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+# Avoid forks if possible
+i=1
+if (i=$(($i+1)) && test $i -eq 2); then
+  incr_i() { i=$(($i+1)); }
+else
+  incr_i() { i=`expr $i + 1`; }
+fi
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS =
+TESTS =
+END
+
+lc_ext=a_very_very_very_long_test_extension_indeed_it_is_slightly_longer_than_81_characters
+uc_ext=A_VERY_VERY_VERY_LONG_TEST_EXTENSION_INDEED_IT_IS_SLIGHTLY_LONGER_THAN_81_CHARACTERS
+
+while test $i -lt 200; do
+  echo TEST_EXTENSIONS += .${lc_ext}_${i}
+  echo ${uc_ext}_${i}_LOG_COMPILER = sh
+  echo TESTS += wow-this-definitely-is-a-very-long-name-for-a-dummy-testcase.${lc_ext}_${i}
+  incr_i
+done >> Makefile.am
+
+$ACLOCAL
+$AUTOMAKE -a
+
+# For debugging.
+$FGREP 'TEST_LOGS' Makefile.in
+
+grep '^TEST_LOGS *=.*\\$' Makefile.in && Exit 1
+
+:
-- 
1.7.2.3

Message #17 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 7868 <at> debbugs.gnu.org
Subject: Re: bug#7868: splitting up test suites
Date: Thu, 20 Jan 2011 20:41:10 +0100

Hi Stefano,

* Stefano Lattarini wrote on Thu, Jan 20, 2011 at 12:38:17PM CET:
> On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> > We've finally reached the point where we have more than 1000
> > tests, $(TESTS) expands to 15k characters, and where 'make check' will
> > not work at all any more on MSYS, because it cannot spawn sh any more,
> > presumably in 'make check TESTS="..."'.  (MSYS make doesn't export
> > macros to the environment of spawned processes even without .NOEXPORT,
> > presumably otherwise lots of Makefiles would be really unusable here.)
> > This also clears up the spurious failure of sed a few days ago.
> >  
> > Here's a preliminary plan for multiple testsuites per Makefile.am.
> >
> Hmmm... while this feature might be worth having even indipendently
> from the issue at hand (but see below for small nits), I still think
> that in the long run it would be nicer to transparently work around
> such command-line length issues in the test driver, if possible.

No, that is not possible with portable make.

> Do
> you think your patch "parallel-tests: avoid command-line length limit
> issue" (from commit v1.11-191-g24e3b4e) could be resurrected in some
> way?

No.  It is fundamentally flawed.  Here's why: while it may fix things at
the level of recursion below that, eventually the rule commands which
were changed in that patch will fail to execute, because they pose too
long a command line for the shell.

There simply is no way to get things to scale except by using GNU make
specifics, or splitting long lists of files up manually.  Well, an
external list of files to be read in could work, but then we don't have
make semantics easily available.

In that light, I'm sorry you already wrote the patch you did, because I
don't see how it can improve things significantly.  I didn't try the
patch or look in detail, please ping me if you still want me to do that.

> > It would be nice if this worked somehow:
> > 
> > # These are all specified by the user:
> > TEST_SUITE_LOGS = suite1.log suite2.log suite3.log ..
> > TEST_EXTENSIONS = .test ...
> > # these undergo $(EXEEXT) autoexpansion internally:
> > suite1_log_SOURCES = all.test aclocal7.test ...
> > suite2_log_SOURCES = suffix.test ...
> >
> What about using `TESTS' instead of `SOURCES' in these two last variables?
> That would seems more natural to me, especially considering the API of the
> current parallel-tests driver.

First off, it would make sense to review the several discussions we had
about this back when parallel-tests was implemented (more precisely,
ported from Akim's GNU make-specific implementation).  See e.g.,
http://thread.gmane.org/gmane.comp.sysutils.automake.patches/3225
http://thread.gmane.org/gmane.comp.sysutils.automake.general/8063

The idea was to treat tests as we treat compiled languages:
.c files are sources for programs but get turned into .o files.
Here, .test files are sources for testsuite logs, but the intermediates
are the per-test .log files.  And there's maybe the unification of
suite*.log's to a final log on top of that.

If we can handle the latter via automake's register_language machinery,
then putting tests in *_SOURCES variables would be fairly natural.

OTOH, you are right in that elsewhere, _SOURCES are usually meant to be
final, nonderived files, which doesn't fit the picture here.  So maybe
suite1_log_TESTS is more appropriate after all.

> OTOH, would that maybe make the implementation more difficult?

Not if it's like the current implementation, no.

> I think it would also be nice to generate separate check/recheck targets
> for each testsuite; for example, "make check-suite1" could run all the
> tests in $(suite1_log_SOURCES), and "make recheck-suite2" could re-run
> all the tests in $(suite2_log_SOURCES) that failed (or xfailed) in the
> previous run.
> 
> Hmm... No, wait, it would be even nicer to allow the user choose which
> testsuite(s) to run by resetting the $(TEST_SUITE_LOGS) variable:
> 
>   make check TEST_SUITE_LOGS='suite1.log suite2.log'
> 
> Which makes me think that, perhaps, a variable like $(TEST_SUITES) would
> be preferable:
> 
>   make check TEST_SUITES='suite1 suite2'
> 
> but then the API should be changed to something like:
> 
>  # These are all specified by the user:
>  TEST_SUITES = suite1 suite2 suite3 ..
>  TEST_EXTENSIONS = .test ...
>  # these undergo $(EXEEXT) autoexpansion internally:
>  suite1_TESTS = all.test aclocal7.test ...
>  suite2_TESTS = suffix.test ...
> 
>  # These are then produced by automake:
>  TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
>  TEST_SUITE_HTMLS = $(TEST_SUITES:=.html)
>  suite1_LOGS = $(am__helper_var:.test=.log)

Unfortunately, this is error-prone, because some GNU make versions
expand
  empty =
  TEST_SUITES = foo $(empty)
  TEST_SUITE_LOGS = $(TEST_SUITES:=.log)

into `foo.log .log' rather than `foo.log '.  This happens in practice
if `empty' is set only under some Automake conditional, as in
  if COND
  TESTS += bar
  endif

That is the reason the check-TESTS rule is so ugly (and recursive) in
the first place.  I really would like to avoid more instances of this
wart; so specifying files without extension is Not Good(TM).

Separate check/recheck targets are of course nice, but IMVHO optional
for the first iteration.

> > In the <suite>_SOURCES, $(EXEEXT) transformation should take place
> > (unless no-exeext is given, of course), just as is currently done for
> > TESTS and *_PROGRAMS.  Hmm, alternatively we could also require all
> > <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of
> > this variable as well, at the cost of some specification redundance.
> >
> 
> > Open questions: how to produce nice results, both on stdout and in suite
> > log files.  One way is to merge all logs into a final TEST_SUITE_LOG
> > (that way we could also reuse that variable).  Another is to try to even
> > hide the summaries of the individual suites, iff a final suite is being
> > made.
> >
> These both sound sensible.  Another problem is how to avoid that, in a
> parallel make run, the summary from a testsuite gets mixed and garbled
> with the output/summary from another testsuite, as in e.g.:

Yes.  We definitely want to be able to run tests from different
testsuites in parallel.  So some form of partial summary hiding is
prudent; I hope we can achieve that without yet another make
indirection, but I'm not optimistic.

Thanks for the feedback,
Ralf

Message #20 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>
Cc: 7868 <at> debbugs.gnu.org
Subject: Re: bug#7868: splitting up test suites
Date: Thu, 20 Jan 2011 22:19:24 +0100

[Message part 1 (text/plain, inline)]

On Thursday 20 January 2011, Ralf Wildenhues wrote:
> Hi Stefano,
> 
> * Stefano Lattarini wrote on Thu, Jan 20, 2011 at 12:38:17PM CET:
> > On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> > > We've finally reached the point where we have more than 1000
> > > tests, $(TESTS) expands to 15k characters, and where 'make check' will
> > > not work at all any more on MSYS, because it cannot spawn sh any more,
> > > presumably in 'make check TESTS="..."'.  (MSYS make doesn't export
> > > macros to the environment of spawned processes even without .NOEXPORT,
> > > presumably otherwise lots of Makefiles would be really unusable here.)
> > > This also clears up the spurious failure of sed a few days ago.
> > >  
> > > Here's a preliminary plan for multiple testsuites per Makefile.am.
> > >
> > Hmmm... while this feature might be worth having even indipendently
> > from the issue at hand (but see below for small nits), I still think
> > that in the long run it would be nicer to transparently work around
> > such command-line length issues in the test driver, if possible.
> 
> No, that is not possible with portable make.
>
Yes :-(  I've found that out while trying to write a a proper test
'parallel-tests-long-cmdline.test' (attached, just for reference).

Relevant excerpt of the log (on Linux 2.6.30 with Bash 4.1 and GNU
make 3.81):

 + automake-1.11 --foreign -Werror -Wall -a
 + ./configure
 checking for a BSD-compatible install... /usr/bin/install -c
 checking whether build environment is sane... yes
 checking for a thread-safe mkdir -p... /bin/mkdir -p
 checking for gawk... gawk
 checking whether make sets $(MAKE)... yes
 configure: creating ./config.status
 config.status: creating Makefile
 + make check
 make[1]: execvp: /bin/sh: Argument list too long
 make[1]: *** [check-TESTS] Error 127
 make: *** [check-am] Error 2

But this makes me think.  If I substitute the `$MAKE check' call
in my test with a `$MAKE dist' call, I get:

  + make dist
  { test ! -d "parallel-tests-long-cmdline-1.0" || \
    { find "parallel-tests-long-cmdline-1.0" -type d ! -perm -200 -exec chmod u+w {} ';' \
      && rm -fr "parallel-tests-long-cmdline-1.0"; }; }
  test -d "parallel-tests-long-cmdline-1.0" || mkdir "parallel-tests-long-cmdline-1.0"
  make: execvp: /bin/sh: Argument list too long
  make: *** [distdir] Error 127

which means that, with the current implementation of the `dist' target,
even breaking the testsuite in two or more testsuites *in the same
directory* won't help at "make dist" time.  Or am I missing something?

> > Do you think your patch "parallel-tests: avoid command-line length
> > limit issue" (from commit v1.11-191-g24e3b4e) could be resurrected
> > in some way?
> 
> No.  It is fundamentally flawed.  Here's why: while it may fix things at
> the level of recursion below that, eventually the rule commands which
> were changed in that patch will fail to execute, because they pose too
> long a command line for the shell.
>
Exactly :-(

> There simply is no way to get things to scale except by using GNU make
> specifics, or splitting long lists of files up manually.  Well, an
> external list of files to be read in could work,
>
I thought about that too ..

> but then we don't have make semantics easily available.
>
... and I thought about this too.  But maybe allowing the user to say,
e.g.:

 TESTS = test1 test2 test3
 EXTRA_TESTS_LIST = @file-with-list-of-test

might still be worth after all; if he prefers to lose some make semantics
in order not to have to split the testsuite, then he should be allowed to
do so.  It remains to see if that's doable (preferably in an easy way).

> In that light, I'm sorry you already wrote the patch you did, because I
> don't see how it can improve things significantly.  I didn't try the
> patch or look in detail, please ping me if you still want me to do that.
>
No need to: the approach is really fundamentally flawed (as you said).

> > > It would be nice if this worked somehow:
> > > 
> > > # These are all specified by the user:
> > > TEST_SUITE_LOGS = suite1.log suite2.log suite3.log ..
> > > TEST_EXTENSIONS = .test ...
> > > # these undergo $(EXEEXT) autoexpansion internally:
> > > suite1_log_SOURCES = all.test aclocal7.test ...
> > > suite2_log_SOURCES = suffix.test ...
> > >
> > What about using `TESTS' instead of `SOURCES' in these two last variables?
> > That would seems more natural to me, especially considering the API of the
> > current parallel-tests driver.
> 
> First off, it would make sense to review the several discussions we had
> about this back when parallel-tests was implemented (more precisely,
> ported from Akim's GNU make-specific implementation).  See e.g.,
> http://thread.gmane.org/gmane.comp.sysutils.automake.patches/3225
> http://thread.gmane.org/gmane.comp.sysutils.automake.general/8063
>
Thanks for the pointers, I'll take a look.

> The idea was to treat tests as we treat compiled languages:
> .c files are sources for programs but get turned into .o files.
> Here, .test files are sources for testsuite logs, but the intermediates
> are the per-test .log files.  And there's maybe the unification of
> suite*.log's to a final log on top of that.
>
I haven't thought in thie terms before -- nice abstraction!  But see
below.

> If we can handle the latter via automake's register_language machinery,
> then putting tests in *_SOURCES variables would be fairly natural.
> 
> OTOH, you are right in that elsewhere, _SOURCES are usually meant to be
> final, nonderived files,
>
That's not exactly true, as automke allows (as it IMHO should)
generated files to be places in *_SOURCES.  But automake also needs
to know the contents of a foo_SOURCES variable statically [1] at
Makefile.in's generation time, while it wants to allow the user to
specify *at make runtime* the list of TESTS to run.

 [1] OK, it's smart enough to resolve variable indirections and
     conditionals, but won't allow @var@ substitution to be placed
     in a *_SOURCES variable (by design).

> which doesn't fit the picture here.  So maybe suite1_log_TESTS
> is more appropriate after all.

> > OTOH, would that maybe make the implementation more difficult?
> 
> Not if it's like the current implementation, no.
> 
> > I think it would also be nice to generate separate check/recheck targets
> > for each testsuite; for example, "make check-suite1" could run all the
> > tests in $(suite1_log_SOURCES), and "make recheck-suite2" could re-run
> > all the tests in $(suite2_log_SOURCES) that failed (or xfailed) in the
> > previous run.
> > 
> > Hmm... No, wait, it would be even nicer to allow the user choose which
> > testsuite(s) to run by resetting the $(TEST_SUITE_LOGS) variable:
> > 
> >   make check TEST_SUITE_LOGS='suite1.log suite2.log'
> > 
> > Which makes me think that, perhaps, a variable like $(TEST_SUITES) would
> > be preferable:
> > 
> >   make check TEST_SUITES='suite1 suite2'
> > 
> > but then the API should be changed to something like:
> > 
> >  # These are all specified by the user:
> >  TEST_SUITES = suite1 suite2 suite3 ..
> >  TEST_EXTENSIONS = .test ...
> >  # these undergo $(EXEEXT) autoexpansion internally:
> >  suite1_TESTS = all.test aclocal7.test ...
> >  suite2_TESTS = suffix.test ...
> > 
> >  # These are then produced by automake:
> >  TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> >  TEST_SUITE_HTMLS = $(TEST_SUITES:=.html)
> >  suite1_LOGS = $(am__helper_var:.test=.log)
> 
> Unfortunately, this is error-prone, because some GNU make versions
> expand
>   empty =
>   TEST_SUITES = foo $(empty)
>   TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> 
> into `foo.log .log' rather than `foo.log '.  This happens in practice
> if `empty' is set only under some Automake conditional, as in
>   if COND
>   TESTS += bar
>   endif
>
And I guess that specifying dummy `.log:' (and maybe `.html:' etc.)
rules won't work either, right? :-(

> That is the reason the check-TESTS rule is so ugly (and recursive) in
> the first place.  I really would like to avoid more instances of this
> wart; so specifying files without extension is Not Good(TM).
>
OK, noted (and these considerations could IMHO end up somewhere in the
manual).

> Separate check/recheck targets are of course nice, but IMVHO optional
> for the first iteration.
>
I agree; the only important thing is to devise a design that will allow
them to be easily added in later refinings.

> > > In the <suite>_SOURCES, $(EXEEXT) transformation should take place
> > > (unless no-exeext is given, of course), just as is currently done for
> > > TESTS and *_PROGRAMS.  Hmm, alternatively we could also require all
> > > <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of
> > > this variable as well, at the cost of some specification redundance.
> > >
> > 
> > > Open questions: how to produce nice results, both on stdout and in suite
> > > log files.  One way is to merge all logs into a final TEST_SUITE_LOG
> > > (that way we could also reuse that variable).  Another is to try to even
> > > hide the summaries of the individual suites, iff a final suite is being
> > > made.
> > >
> > These both sound sensible.  Another problem is how to avoid that, in a
> > parallel make run, the summary from a testsuite gets mixed and garbled
> > with the output/summary from another testsuite, as in e.g.:
> 
> Yes.  We definitely want to be able to run tests from different
> testsuites in parallel.  So some form of partial summary hiding is
> prudent; I hope we can achieve that without yet another make
> indirection, but I'm not optimistic.
> 
> Thanks for the feedback,
> Ralf
> 

Regards,
  Stefano

[parallel-tests-long-cmdline.test (application/x-shellscript, inline)]

Message #23 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 7868 <at> debbugs.gnu.org
Subject: Re: bug#7868: splitting up test suites
Date: Sat, 22 Jan 2011 11:06:48 +0100

Hi Stefano,

* Stefano Lattarini wrote on Thu, Jan 20, 2011 at 10:19:24PM CET:
> On Thursday 20 January 2011, Ralf Wildenhues wrote:
> > * Stefano Lattarini wrote on Thu, Jan 20, 2011 at 12:38:17PM CET:
> > > Hmmm... while this feature might be worth having even indipendently
> > > from the issue at hand (but see below for small nits), I still think
> > > that in the long run it would be nicer to transparently work around
> > > such command-line length issues in the test driver, if possible.
> > 
> > No, that is not possible with portable make.
> >
> Yes :-(  I've found that out while trying to write a a proper test
> 'parallel-tests-long-cmdline.test' (attached, just for reference).
[...]

> But this makes me think.  If I substitute the `$MAKE check' call
> in my test with a `$MAKE dist' call, I get:
> 
>   + make dist
>   { test ! -d "parallel-tests-long-cmdline-1.0" || \
>     { find "parallel-tests-long-cmdline-1.0" -type d ! -perm -200 -exec chmod u+w {} ';' \
>       && rm -fr "parallel-tests-long-cmdline-1.0"; }; }
>   test -d "parallel-tests-long-cmdline-1.0" || mkdir "parallel-tests-long-cmdline-1.0"
>   make: execvp: /bin/sh: Argument list too long
>   make: *** [distdir] Error 127
> 
> which means that, with the current implementation of the `dist' target,
> even breaking the testsuite in two or more testsuites *in the same
> directory* won't help at "make dist" time.  Or am I missing something?

No, you are correct.  This was similarly reported even for GNU/Linux by
Xan Lopez on the automake list over a year ago (and probably before that
too).

I'm not too worried about dist however, for a couple of reasons:

dist is not usually required for downloading, building, testing, and
installing software.  As such, even if it is GCS-mandated, it is not
quite as big a problem when it doesn't work everywhere; but of course
it needs to work on developer machines.  Furthermore, it is almost
always possible for the developer to restructure the package so that
line lengths are not exceeded, usually by introducing (more) Makefile
recursion.  See the `Length Limitations' node in the manual.


That said, I would like to address distdir in the long run as well.
I see two possible strategies:

As far as possible, automake can try to split the copying part of
distdir into several targets that copy parts of the files.  This isn't
as easy as it sounds, because usually it's one of the variables SOURCES,
EXTRA_DIST, etc. alone that exceeds the limit.  But the splitting can be
done at automake time.  (Note that if @substitutions@ are involved, then
a single expansion can currently not exceed the line length limit by
itself, that would already break at configure or config.status time.)

Second, and much simpler: when we have gnu-make support, we can simply
use GNU make-specific code to avoid running into limits.  This can be
done transparently, and since in practice virtually all package
maintainers at least have access to GNU make, it should be sufficient
in practice.

I have a half-done patch for the second scheme.  I hope to post all the
gnu-make stuff soonish, sorry for the delay.

> But maybe allowing the user to say, e.g.:
> 
>  TESTS = test1 test2 test3
>  EXTRA_TESTS_LIST = @file-with-list-of-test
> 
> might still be worth after all; if he prefers to lose some make semantics
> in order not to have to split the testsuite, then he should be allowed to
> do so.  It remains to see if that's doable (preferably in an easy way).

Yes, it may be worth it.  But if we can avoid it, all the better.
I prefer not breaking our usual semantic scheme.

> > If we can handle the latter via automake's register_language machinery,
> > then putting tests in *_SOURCES variables would be fairly natural.
> > 
> > OTOH, you are right in that elsewhere, _SOURCES are usually meant to be
> > final, nonderived files,
> >
> That's not exactly true, as automke allows (as it IMHO should)
> generated files to be places in *_SOURCES.  But automake also needs
> to know the contents of a foo_SOURCES variable statically [1] at
> Makefile.in's generation time, while it wants to allow the user to
> specify *at make runtime* the list of TESTS to run.

Yes, that is one of the other complications/semantic differences:
users usually cope with not being able to override *_OBJECTS.
We considered declaring TESTS an internal detail too, but I just
find it too helpful to give up this API.

>  [1] OK, it's smart enough to resolve variable indirections and
>      conditionals, but won't allow @var@ substitution to be placed
>      in a *_SOURCES variable (by design).

FWIW, I regard that a necessary limitation, not a design feature.

> > >  # These are all specified by the user:
> > >  TEST_SUITES = suite1 suite2 suite3 ..
[...]
> > >  # These are then produced by automake:
> > >  TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> > >  TEST_SUITE_HTMLS = $(TEST_SUITES:=.html)
> > >  suite1_LOGS = $(am__helper_var:.test=.log)
> > 
> > Unfortunately, this is error-prone, because some GNU make versions
> > expand
> >   empty =
> >   TEST_SUITES = foo $(empty)
> >   TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> > 
> > into `foo.log .log' rather than `foo.log '.  This happens in practice
> > if `empty' is set only under some Automake conditional, as in
> >   if COND
> >   TESTS += bar
> >   endif
> >
> And I guess that specifying dummy `.log:' (and maybe `.html:' etc.)
> rules won't work either, right? :-(

Nope.

> > That is the reason the check-TESTS rule is so ugly (and recursive) in
> > the first place.  I really would like to avoid more instances of this
> > wart; so specifying files without extension is Not Good(TM).
> >
> OK, noted (and these considerations could IMHO end up somewhere in the
> manual).

Good suggestion.  They should be mentioned in the portability section of
autoconf.texi if they aren't already, and as ## comments in check.am.
Let me see about a quick patch.

> > Separate check/recheck targets are of course nice, but IMVHO optional
> > for the first iteration.
> >
> I agree; the only important thing is to devise a design that will allow
> them to be easily added in later refinings.

Don't worry.

Cheers,
Ralf

Message #28 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Cc: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>, 7868 <at> debbugs.gnu.org,
	automake-patches <at> gnu.org
Subject: Re: parallel-tests: avoid command-line length limit issue.
Date: Thu, 29 Dec 2011 00:26:15 +0100

[Message part 1 (text/plain, inline)]

On 12/28/2011 08:27 PM, Bob Friesenhahn wrote:
> Sorry for top-posting.  I don't want to lose any text of the original mail.
> 
> I totally forgot that I had applied Ralf's patch to my 1.11.1 install. Now
> that I have updated GraphicsMagick to using Automake 1.11.2, the patch is no
> longer present and so 'make check' on MinGW/MSYS leads to this:
> 
> /usr/bin/csmake  check-TESTS check-local
> csmake[2]: Entering directory `/home/bfriesen/mingw/GM-16-static'
> csmake[3]: Entering directory `/home/bfriesen/mingw/GM-16-static'
> csmake[3]: execvp: /bin/sh: Invalid argument
> csmake[3]: *** [tests/constitute_char_bgr.log] Error 127
> csmake[3]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake[2]: *** [check-TESTS] Error 2
> csmake[2]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake[1]: *** [check-am] Error 2
> csmake[1]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake: *** [check] Error 2
> 
> Today I lost more hair because I had totally forgotten about this issue and
> the problem was first noticed after making some changes to configure.ac.
> 
> Hopefully this problem can be resolved before Automake 1.12 is released.
> 
Let's start by exposing the problem once and for all.

What about the attached patch?  Tested on Debian unstable, NetBSD 5.1,
Cygwin 1.5.25 and Solaris 10, and it seems to correctly expose the problem
on all those systems.

Thanks,
  Stefano

[0001-coverage-expose-automake-bug-7868.patch (text/x-diff, attachment)]

Message #31 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Cc: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>, 7868 <at> debbugs.gnu.org,
	automake-patches <at> gnu.org
Subject: Re: parallel-tests: avoid command-line length limit issue.
Date: Fri, 30 Dec 2011 13:20:15 +0100

[Message part 1 (text/plain, inline)]

On 12/29/2011 12:26 AM, Stefano Lattarini wrote:
> On 12/28/2011 08:27 PM, Bob Friesenhahn wrote:
>> Sorry for top-posting.  I don't want to lose any text of the original mail.
>>
>> I totally forgot that I had applied Ralf's patch to my 1.11.1 install. Now
>> that I have updated GraphicsMagick to using Automake 1.11.2, the patch is no
>> longer present and so 'make check' on MinGW/MSYS leads to this:
>>
>> /usr/bin/csmake  check-TESTS check-local
>> csmake[2]: Entering directory `/home/bfriesen/mingw/GM-16-static'
>> csmake[3]: Entering directory `/home/bfriesen/mingw/GM-16-static'
>> csmake[3]: execvp: /bin/sh: Invalid argument
>> csmake[3]: *** [tests/constitute_char_bgr.log] Error 127
>> csmake[3]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
>> csmake[2]: *** [check-TESTS] Error 2
>> csmake[2]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
>> csmake[1]: *** [check-am] Error 2
>> csmake[1]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
>> csmake: *** [check] Error 2
>>
>> Today I lost more hair because I had totally forgotten about this issue and
>> the problem was first noticed after making some changes to configure.ac.
>>
>> Hopefully this problem can be resolved before Automake 1.12 is released.
>>
> Let's start by exposing the problem once and for all.
> 
> [SNIP]
>
OK, I hope I've finally managed to partially fix this incredibly annoying bug
-- I say "partially" because the fix is sadly *for GNU make*.  Even more sadly,
the solution is pretty hacky and somewhat brittle.   Still, it should cause no
regression with the non-GNU makes (as they will continue to use the old
implementation), and I have verified that it works with at least all the GNU
make versions >= 3.78.

I'm posting the patches here for early feedback (note that the second one still
needs a commit message and possibly some polishing), with the hope that someone
can suggest a less hacky solution.

Regards,
  Stefano

[0001-coverage-expose-automake-bug-7868.patch (text/x-diff, attachment)]

[0002-fix-the-bug-for-GNU-make.patch (text/x-diff, attachment)]

Message #34 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 7868 <at> debbugs.gnu.org, automake-patches <at> gnu.org
Subject: Re: parallel-tests: avoid command-line length limit issue.
Date: Fri, 30 Dec 2011 10:10:09 -0600 (CST)

On Fri, 30 Dec 2011, Stefano Lattarini wrote:
>>
> OK, I hope I've finally managed to partially fix this incredibly annoying bug
> -- I say "partially" because the fix is sadly *for GNU make*.  Even more sadly,
> the solution is pretty hacky and somewhat brittle.   Still, it should cause no
> regression with the non-GNU makes (as they will continue to use the old
> implementation), and I have verified that it works with at least all the GNU
> make versions >= 3.78.

I see that there is a new test which tests for GNU make.  How solid is 
it?  For example, the GNU make I use under MinGW is called 'csmake' 
(for "Case Sensitive make") because the default MinGW GNU 
case-insensitive make is horrendously slow, taking a couple of minutes 
before it does anything at all.  It seems that someone decided that 
MinGW make needs to be case-insensitive and it operates via 
brute-force, trying every case permutation.  If a network is involved, 
this becomes frightfully slow.  As a result, whever I would normally 
type 'make' I type 'csmake' instead.

Bob
-- 
Bob Friesenhahn
bfriesen <at> simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Message #37 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Cc: 7868 <at> debbugs.gnu.org, automake-patches <at> gnu.org
Subject: Re: bug#7868: parallel-tests: avoid command-line length limit issue.
Date: Fri, 30 Dec 2011 17:51:38 +0100

Hi Bob.

On 12/30/2011 05:10 PM, Bob Friesenhahn wrote:
> On Fri, 30 Dec 2011, Stefano Lattarini wrote:
>>>
>> OK, I hope I've finally managed to partially fix this incredibly annoying bug
>> -- I say "partially" because the fix is sadly *for GNU make*.  Even more sadly,
>> the solution is pretty hacky and somewhat brittle.   Still, it should cause no
>> regression with the non-GNU makes (as they will continue to use the old
>> implementation), and I have verified that it works with at least all the GNU
>> make versions >= 3.78.
> 
> I see that there is a new test which tests for GNU make.  How solid is it?
>
Pretty solid IMHO.  In fact, it correctly exposes the problem at hand in the
following setups:

  * Solaris 10 with /usr/ccs/bin/make
  * Solaris 10 with GNU make 3.82
  * Debian unstable with GNU make 3.81, freebsd-make (8.x) and
    pmake (port of NetBSDmake)
  * NetBSD 5.1 with the system make

On Cygwin 1.5, it exposes a probably-related problem (a segfault in make).

> For example, the GNU make I use under MinGW is called 'csmake' (for
> "Case Sensitive make") because the default MinGW GNU case-insensitive
> make is horrendously slow... [SNIP] ... As a result, whenever I would
> normally type 'make' I type 'csmake' instead.
>
Then, to be 100% correct, you'll have to call configure as:

  ./configure MAKE=csmake

Still, if you only have GNU make installed on you system, the correct
code paths in the generated Makefile will be correctly activated even
if you don't specify the $MAKE override at configure time, so you will
later be able to safely run the generated Makefiles with other versions
of GNU make.

Regards,
  Stefano

Message #44 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Peter Rosin <peda <at> lysator.liu.se>
Cc: automake <at> gnu.org, 7868 <at> debbugs.gnu.org, automake-patches <at> gnu.org,
	Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Subject: Re: automake 1.11.3 check-TESTS and command line length
Date: Mon, 27 Feb 2012 20:52:58 +0100

[re-adding the relevant automake bug in CC]

Hi Peter, thanks for chiming in.

On 02/27/2012 12:15 PM, Peter Rosin wrote:
>
> I *think* the environment and the command line shares space (approx 64kB,
> I repeat *think* here, I don't know the details off the top of my head, Cygwin
> isn't affected since it uses the cygwin DLL to communicate this stuff between
> cygwin processes using normal ipc mechanisms).  If I'm right, reducing the
> command line length for a process might sufficiently elevate the limit, even
> if it isn't eliminated.
> 
So, basically, on MSYS, a:

  $ make TESTS="..."

invocation reduces the size available for the command lines of the make
recipes, because the value of TESTS that gets exported in the environment
eats away space that could be used by those command lines?  Oh joy ...

Anyway, if we now reintroduce Ralf's patch, we will have a good explanation
of why it actually works.

Thanks,
  Stefano

Message #47 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: Peter Rosin <peda <at> lysator.liu.se>, 7868 <at> debbugs.gnu.org,
	automake-patches <at> gnu.org, automake <at> gnu.org
Subject: Re: automake 1.11.3 check-TESTS and command line length
Date: Mon, 27 Feb 2012 13:05:25 -0700

[Message part 1 (text/plain, inline)]

On 02/27/2012 12:52 PM, Stefano Lattarini wrote:
> [re-adding the relevant automake bug in CC]
> 
> Hi Peter, thanks for chiming in.
> 
> On 02/27/2012 12:15 PM, Peter Rosin wrote:
>>
>> I *think* the environment and the command line shares space (approx 64kB,

I can confirm this, based on my testing of Windows process spawning.

> So, basically, on MSYS, a:
> 
>   $ make TESTS="..."
> 
> invocation reduces the size available for the command lines of the make
> recipes, because the value of TESTS that gets exported in the environment
> eats away space that could be used by those command lines?  Oh joy ...

Yes.  Unlike on *ix systems, where argv and environ are independent
entities, Windows makes you deal with both limits at the same time, so
increasing the size of environ reduces the size permitted in argv.

-- 
Eric Blake   eblake <at> redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #50 received at 7868 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: 7868 <at> debbugs.gnu.org
Subject: Re: bug#7868: automake 1.11.3 check-TESTS and command line length
Date: Sat, 14 Nov 2020 19:20:11 -0700

It seems something more needs to be done with this bug, but I don't know
what. I hope someone with more energy, expertise, and/or time can look
into it and find a resolution.

Marking it as severity normal (can't be that important after 8 years)
and with the "help" tag.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.