Subject: bug#7868: parallel-tests: avoid command-line length limit issue.

From: Stefano Lattarini <[email-masked]>
Date: Thu, 29 Dec 2011 00:26:15 +0100

On 12/28/2011 08:27 PM, Bob Friesenhahn wrote:
> Sorry for top-posting.  I don't want to lose any text of the original mail.
> 
> I totally forgot that I had applied Ralf's patch to my 1.11.1 install. Now
> that I have updated GraphicsMagick to using Automake 1.11.2, the patch is no
> longer present and so 'make check' on MinGW/MSYS leads to this:
> 
> /usr/bin/csmake  check-TESTS check-local
> csmake[2]: Entering directory `/home/bfriesen/mingw/GM-16-static'
> csmake[3]: Entering directory `/home/bfriesen/mingw/GM-16-static'
> csmake[3]: execvp: /bin/sh: Invalid argument
> csmake[3]: *** [tests/constitute_char_bgr.log] Error 127
> csmake[3]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake[2]: *** [check-TESTS] Error 2
> csmake[2]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake[1]: *** [check-am] Error 2
> csmake[1]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake: *** [check] Error 2
> 
> Today I lost more hair because I had totally forgotten about this issue and
> the problem was first noticed after making some changes to configure.ac.
> 
> Hopefully this problem can be resolved before Automake 1.12 is released.
> 
Let's start by exposing the problem once and for all.

What about the attached patch?  Tested on Debian unstable, NetBSD 5.1,
Cygwin 1.5.25 and Solaris 10, and it seems to correctly expose the problem
on all those systems.

Thanks,
  Stefano
>From 426d0ba21120eacf296eb0a6c81902d2b24ffa60 Mon Sep 17 00:00:00 2001
Message-Id: <426d0ba21[email-masked]>
From: Stefano Lattarini <[email-masked]>
Date: Wed, 28 Dec 2011 22:37:44 +0100
Subject: [PATCH] coverage: expose automake bug#7868

Expose the command-line length limit issue that can affect the
Automake-generated parallel-tests harness, especially on systems
where this limit is smaller (e.g., MinGW/MSYS).

Suggestion by Bob Friesenhahn.

* tests/parallel-tests-many.test: New test.  We have verified that
it actually exposes the bug#7868, as it passes when we opportunely
reduce the number of test cases in $(TESTS).  Checked on NetBSD 5.1,
Debian unstable, Solaris 10 and Cygwin 1.5.
* tests/Makefile.am (TESTS, XFAIL_TESTS): Add it.
---
 tests/Makefile.am              |    2 +
 tests/parallel-tests-many.test |   92 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+), 0 deletions(-)
 create mode 100755 tests/parallel-tests-many.test

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 6ce7c53..e14d57e 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -26,6 +26,7 @@ gcj6.test \
 java-nobase.test \
 pr8365-remake-timing.test \
 lex-subobj-nodep.test \
+parallel-tests-many.test \
 remake-am-pr10111.test \
 remake-m4-pr10111.test \
 vala-vpath.test \
@@ -648,6 +649,7 @@ parallel-tests-log-override-1.test \
 parallel-tests-log-override-2.test \
 parallel-tests-log-override-recheck.test \
 parallel-tests-log-compiler-example.test \
+parallel-tests-many.test \
 test-extensions.test \
 test-extensions-cond.test \
 parse.test \
diff --git a/tests/parallel-tests-many.test b/tests/parallel-tests-many.test
new file mode 100755
index 0000000..f0b4c85
--- /dev/null
+++ b/tests/parallel-tests-many.test
@@ -0,0 +1,92 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the parallel-tests harness does not hit errors due to
+# an exceeded command line length when there are many tests.
+# For automake bug#7868.  This test is currently expected to fail.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+basename="wow-this-is-a-very-long-name-for-a-simple-dummy-test-case"
+basename=$basename-$basename
+count=10000
+
+cat > Makefile.am <<'END'
+## Sanity check that the $(TESTS) is going to exceed the system
+## command line length.  We can't use echo here, since it might
+## be a built-in, so use our dummy program.
+this-will-fail:
+## Extra quoting and indirections below are required to ensure the
+## various make implementations (e.g, GNU make or Sun Distributed Make)
+## will truly spawn a shell to execute this command, instead of relying
+## on optimizations that might mask the "Argument list too long" error
+## we expect.
+	@sh=sh && "$sh" -c : $(TESTS)
+TEST_LOG_COMPILER = true
+## We will shave *a lot* of automake execution time by extending
+## this by hand, later.
+TESTS = @my_tests@
+END
+
+cat >> configure.in << 'END'
+AC_SUBST([my_tests], ['$(my_tests)'])
+AM_SUBST_NOTMAKE([my_tests])
+AC_OUTPUT
+END
+
+awk "
+  BEGIN {
+    for (i = 1; i <= $count; i = i + 1)
+       print \"$basename-\" i \".test\"
+  }
+" > list-of-tests
+
+$ACLOCAL && $AUTOCONF && $AUTOMAKE -a || Exit 99
+
+echo 'my_tests = \' >> Makefile.in
+sed 's/$/  \\/' list-of-tests >> Makefile.in
+echo '  $(empty)' >> Makefile.in
+# So that we won't have to create a ton of dummy test cases.
+echo '$(TESTS):' >> Makefile.in
+
+tail -n 15 Makefile.in || : # For debugging.
+
+./configure || Exit 99
+
+env TESTS="$basename-1.test" $MAKE -e check && rm -f *.log || Exit 99
+
+if $MAKE this-will-fail; then
+  framework_failure_ "system has a too-high limit on command line length"
+fi
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+grep "All $count tests" stdout
+
+grep "^PASS: $basename" stdout > grp
+ls -1 | grep '\.log' | $EGREP -v '^(config|test-suite)\.log$' > lst
+
+sed 20q lst # For debugging.
+sed 20q grp # Likewise.
+
+test `cat <grp | wc -l` -eq $count
+test `cat <lst | wc -l` -eq $count
+
+:
-- 
1.7.7.3