GNU bug report logs - #9245
FreeBSD make in concurrent mode report spurious success in automake-generated tests harness

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 9245 in the body.
You can then email your comments to 9245 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: bug-automake <at> gnu.org
Subject: FreeBSD make in concurrent mode report spurious success in
	automake-generated tests harness
Date: Fri, 5 Aug 2011 11:08:20 +0200

Here is a brief illustration of the issue:

  $ cat > Makefile.am <<'END'
  AUTOMAKE_OPTIONS = foreign parallel-tests -Wall -Werror
  TEST_LOG_COMPILER = false
  TESTS = foo.test bar.test
  END
  $ touch foo.test bar.test
  $ autoreconf -vi && ./configure
  ...

  $ freebsd-make -j2 check; echo %% STATUS = $? %%
  freebsd-make  check-TESTS
  FAIL: foo.test
  FAIL: bar.test
  ====================
  2 of 2 tests failed
  See ./test-suite.log
  ====================
  %% STATUS = 0 %%

It works as expected with GNU make and NetBSD make though:

 $ gmake -j2 check; echo %% STATUS = $? %%
  gmake  check-TESTS
  gmake[1]: Entering directory `/tmp/...'
  gmake[2]: Entering directory `/tmp/...'
  FAIL: foo.test
  FAIL: bar.test
  ====================
  2 of 2 tests failed
  See ./test-suite.log
  ====================
  gmake[2]: *** [test-suite.log] Error 1
  gmake[2]: Leaving directory `/tmp/...'
  gmake[1]: *** [check-TESTS] Error 2
  gmake[1]: Leaving directory `/tmp/...'
  gmake: *** [check-am] Error 2
  %% STATUS = 2 %%


  $ netbsd-make -j2 check; echo %% STATUS = $? %%
  --- check-am ---
  netbsd-make  check-TESTS
  --- check-TESTS ---
  --- foo.log ---
  --- bar.log ---
  FAIL: bar.test
  --- foo.log ---
  FAIL: foo.test
  --- test-suite.log ---
  ====================
  2 of 2 tests failed
  See ./test-suite.log
  ====================
  *** [test-suite.log] Error code 1
  1 error
  netbsd-make: stopped in /tmp/...
  *** [check-TESTS] Error code 2
  1 error
  netbsd-make: stopped in /tmp/...
  *** [check-am] Error code 2
  1 error
  netbsd-make: stopped in /tmp/...
  %% STATUS = 2 %%

The problem is present both in automake 1.11.1 and in the developement
version from the 'test-protocols' branch.

Regards,
  Stefano

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: bug-automake <at> gnu.org
Cc: 9245 <at> debbugs.gnu.org
Subject: Re: bug#9245: FreeBSD make in concurrent mode report spurious success
	in automake-generated tests harness
Date: Fri, 12 Aug 2011 18:32:57 +0200

[Message part 1 (text/plain, inline)]

On Friday 05 August 2011, Stefano Lattarini wrote:
> Here is a brief illustration of the issue:
> 
>   $ cat > Makefile.am <<'END'
>   AUTOMAKE_OPTIONS = foreign parallel-tests -Wall -Werror
>   TEST_LOG_COMPILER = false
>   TESTS = foo.test bar.test
>   END
>   $ touch foo.test bar.test
>   $ autoreconf -vi && ./configure
>   ...
> 
>   $ freebsd-make -j2 check; echo %% STATUS = $? %%
>   freebsd-make  check-TESTS
>   FAIL: foo.test
>   FAIL: bar.test
>   ====================
>   2 of 2 tests failed
>   See ./test-suite.log
>   ====================
>   %% STATUS = 0 %%
> 
> It works as expected with GNU make and NetBSD make though:
> 
>  $ gmake -j2 check; echo %% STATUS = $? %%
>   gmake  check-TESTS
>   gmake[1]: Entering directory `/tmp/...'
>   gmake[2]: Entering directory `/tmp/...'
>   FAIL: foo.test
>   FAIL: bar.test
>   ====================
>   2 of 2 tests failed
>   See ./test-suite.log
>   ====================
>   gmake[2]: *** [test-suite.log] Error 1
>   gmake[2]: Leaving directory `/tmp/...'
>   gmake[1]: *** [check-TESTS] Error 2
>   gmake[1]: Leaving directory `/tmp/...'
>   gmake: *** [check-am] Error 2
>   %% STATUS = 2 %%
> 
> 
>   $ netbsd-make -j2 check; echo %% STATUS = $? %%
>   --- check-am ---
>   netbsd-make  check-TESTS
>   --- check-TESTS ---
>   --- foo.log ---
>   --- bar.log ---
>   FAIL: bar.test
>   --- foo.log ---
>   FAIL: foo.test
>   --- test-suite.log ---
>   ====================
>   2 of 2 tests failed
>   See ./test-suite.log
>   ====================
>   *** [test-suite.log] Error code 1
>   1 error
>   netbsd-make: stopped in /tmp/...
>   *** [check-TESTS] Error code 2
>   1 error
>   netbsd-make: stopped in /tmp/...
>   *** [check-am] Error code 2
>   1 error
>   netbsd-make: stopped in /tmp/...
>   %% STATUS = 2 %%
> 
> The problem is present both in automake 1.11.1 and in the developement
> version from the 'test-protocols' branch.
> 
OK, the attached hacky patch seems to fix the bug.  I have no idea why
exactly it is so yet, so we might want to wait to apply the patch until
we have fully understood the reasons of the original failure, and why
the proposed patch seems to fix it.

Regards,
  Stefano

[0001-parallel-tests-no-more-spurious-successes-for-FreeBS.patch (text/x-patch, inline)]

From e7568ee0cfaccc5c9998275a5d93643a49ea533d Mon Sep 17 00:00:00 2001
Message-Id: <e7568ee0cfaccc5c9998275a5d93643a49ea533d.1313166621.git.stefano.lattarini <at> gmail.com>
From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Date: Fri, 12 Aug 2011 18:29:28 +0200
Subject: [PATCH] parallel-tests: no more spurious successes for FreeBSD make

Work around a bug of FreeBSD make bug that was causing the
automake-generated "check" target to complete with success
even if some tests failed; this happened only when FreeBSD
make was run in concurrent mode (as in, e.g., "make -j2
check").  The bug is not present in NetBSD make.

This change fixes automake bug#9245.

* lib/am/check.am [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Use a
more "safe" (and apparently redundant) idiom to exit with error,
so that the non-zero exit status is picked up also by FreeBSD
make when it's running in concurrent mode.
* NEWS: Update.
* tests/parallel-tests-concurrency-bug9245.test: New test.
* tests/Makefile.am (TESTS): Update.
---
 ChangeLog                                     |   17 ++++++++
 NEWS                                          |    4 +-
 lib/Automake/tests/Makefile.in                |    2 +-
 lib/am/check.am                               |    4 +-
 tests/Makefile.am                             |    1 +
 tests/Makefile.in                             |    3 +-
 tests/parallel-tests-concurrency-bug9245.test |   55 +++++++++++++++++++++++++
 7 files changed, 82 insertions(+), 4 deletions(-)
 create mode 100755 tests/parallel-tests-concurrency-bug9245.test

diff --git a/ChangeLog b/ChangeLog
index af2556f..f7a3565 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,20 @@
+2011-08-12  Stefano Lattarini  <stefano.lattarini <at> gmail.com>
+
+	parallel-tests: no more spurious successes for FreeBSD make
+	Work around a bug of FreeBSD make bug that was causing the
+	automake-generated "check" target to complete with success
+	even if some tests failed; this happened only when FreeBSD
+	make was run in concurrent mode (as in, e.g., "make -j2
+	check").  The bug is not present in NetBSD make.
+	This change fixes automake bug#9245.
+	* lib/am/check.am [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Use a
+	more "safe" (and apparently redundant) idiom to exit with error,
+	so that the non-zero exit status is picked up also by FreeBSD
+	make when it's running in concurrent mode.
+	* NEWS: Update.
+	* tests/parallel-tests-concurrency-bug9245.test: New test.
+	* tests/Makefile.am (TESTS): Update.
+
 2011-08-08  Stefano Lattarini  <stefano.lattarini <at> gmail.com>
 
 	test defs: more environment cleanup
diff --git a/NEWS b/NEWS
index 29c552b..5086f75 100644
--- a/NEWS
+++ b/NEWS
@@ -23,7 +23,9 @@ Bugs fixed in 1.11.0a:
 * Bugs introduced by 1.11:
 
   - The `parallel-tests' test driver works around a GNU make 3.80 bug with
-    trailing white space in the test list (`TESTS = foo $(EMPTY)').
+    trailing white space in the test list (`TESTS = foo $(EMPTY)'), and
+    does not report spurious successes when used with concurrent FreeBSD
+    make (e.g., "make check -j3").
 
   - The `silent-rules' option now also silences all compile rules if dependency
     tracking is disabled.  Also, when `silent-rules' is not used, the output from
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 2f553ed..1cb438c 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -405,7 +405,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  col="$$red";							\
 	fi;								\
 	echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";		\
-	$$exit
+	$$exit || (exit 1); exit 1
 
 # Run all the tests.
 check-TESTS:
diff --git a/lib/am/check.am b/lib/am/check.am
index 0b54312..75f8bba 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -224,7 +224,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  col="$$red";							\
 	fi;								\
 	echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";		\
-	$$exit
+## The apparently redundant and repeated "exit 1" works around a bug in
+## FreeBSD make (present only when running in concurrent mode).
+	$$exit || (exit 1); exit 1
 
 RECHECK_LOGS = $(TEST_LOGS)
 
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b545dda..a1f7818 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -605,6 +605,7 @@ parallel-tests-log-override-1.test \
 parallel-tests-log-override-2.test \
 parallel-tests-log-override-recheck.test \
 parallel-tests-log-compiler-example.test \
+parallel-tests-concurrency-bug9245.test \
 parse.test \
 percent.test \
 percent2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index fe44a40..7bb6586 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -882,6 +882,7 @@ parallel-tests-log-override-1.test \
 parallel-tests-log-override-2.test \
 parallel-tests-log-override-recheck.test \
 parallel-tests-log-compiler-example.test \
+parallel-tests-concurrency-bug9245.test \
 parse.test \
 percent.test \
 percent2.test \
@@ -1274,7 +1275,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  col="$$red";							\
 	fi;								\
 	echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";		\
-	$$exit
+	$$exit || (exit 1); exit 1
 
 # Run all the tests.
 check-TESTS:
diff --git a/tests/parallel-tests-concurrency-bug9245.test b/tests/parallel-tests-concurrency-bug9245.test
new file mode 100755
index 0000000..78bd98d
--- /dev/null
+++ b/tests/parallel-tests-concurrency-bug9245.test
@@ -0,0 +1,55 @@
+#! /bin/sh
+# Copyright (C) 2009, 2010, 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Look for a bug where FreeBSD make in concurrent mode reported success
+# even when the Automake-generated parallel testsuite harness failed.
+# See automake bug#9245.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_COMPILER = false
+TESTS = foo.test bar.test
+END
+
+: > foo.test
+: > bar.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+# Some make implementations don't grok the `-j' option.
+$MAKE -j1 || Exit 77
+
+for j in '' -j1 -j2; do
+  $MAKE $j check && Exit 1
+  $MAKE $j recheck && Exit 1
+  TESTS=foo.test $MAKE $j -e check && Exit 1
+  TEST_LOGS=foo.log $MAKE $j -e check && Exit 1
+  rm -f test-suite.log
+  $MAKE $j test-suite.log && Exit 1
+  test -f test-suite.log
+done
+
+:
-- 
1.7.2.3

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: bug-automake <at> gnu.org
Cc: bug-autoconf <at> gnu.org, 9245 <at> debbugs.gnu.org
Subject: Re: bug#9245: FreeBSD make in concurrent mode report spurious success
	in automake-generated tests harness
Date: Fri, 12 Aug 2011 20:49:06 +0200

[Message part 1 (text/plain, inline)]

[CC:ing bug-autoconf, about a FreeBSD make bug]

On Friday 12 August 2011, Stefano Lattarini wrote:
> OK, the attached hacky patch seems to fix the bug.  I have no idea why
> exactly it is so yet, so we might want to wait to apply the patch until
> we have fully understood the reasons of the original failure, and why
> the proposed patch seems to fix it.
> 
I have managed to reproduce the FreeBSD make bug with this one liner:

 $ echo 'all: ; @set +e; false' | make -j2 -f-

Notice that all of `-j', `@' and `set +e' are required to trigger the bug:

 $ echo 'all: ; set +e; false' | make -j2 -f-
 set +e; false
 *** Error code 1
 1 error

 $ echo 'all: ; @false' | make -j2 -f-
 *** Error code 1
 1 error

 $ echo 'all: ; @set +e; false' | make  -f-
 *** Error code 1
 Stop in /tmp.

A well-placed use of ps might give an hint about the problem:

 $ echo 'all: ; @set +e; ps -o args $$$$' | make -f- -j2
 COMMAND
 sh -ev

That `-v' flag seems truly fishy ...

System information:

 $ uname -rsm
 FreeBSD 8.2-RELEASE i386
 $ echo 'all: ; @echo $(MAKE_VERSION)' | make -f- 
 5200408120

This issue should IMHO be probably documented in the autoconf manual, and
reported to the FreeBSD developers.  I'll try to do that soonish (couple
of days) if nobody beats me.

Regards,
  Stefano

[Message part 2 (text/html, inline)]

Message #20 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: bug-automake <at> gnu.org
Cc: bug-autoconf <at> gnu.org, 9245 <at> debbugs.gnu.org
Subject: Re: bug#9245: FreeBSD make in concurrent mode report spurious success
	in automake-generated tests harness
Date: Fri, 12 Aug 2011 23:28:21 +0200

[Message part 1 (text/plain, inline)]

On Friday 12 August 2011, Stefano Lattarini wrote:
> [CC:ing bug-autoconf, about a FreeBSD make bug]
>
> [SNIP]
>
> This issue should IMHO be probably reported to the FreeBSD developers
> 
Done, see freebsd pr `bin/159730':

 <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>

Regards,
  Stefano

[Message part 2 (text/html, inline)]

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: bug-automake <at> gnu.org
Cc: 9245 <at> debbugs.gnu.org
Subject: Re: bug#9245: FreeBSD make in concurrent mode report spurious success
	in automake-generated tests harness
Date: Tue, 16 Aug 2011 18:04:38 +0200

[Message part 1 (text/plain, inline)]

On Friday 12 August 2011, Stefano Lattarini wrote:
> OK, the attached hacky patch seems to fix the bug.
>
LOL no, there was an embarassingly stupid error in my patch:

 diff --git a/lib/am/check.am b/lib/am/check.am
 index 0b54312..75f8bba 100644
 --- a/lib/am/check.am
 +++ b/lib/am/check.am
 @@ -224,7 +224,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
           col="$$red";                                                  \
         fi;                                                             \
         echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";         \
 -       $$exit
 +## The apparently redundant and repeated "exit 1" works around a bug in
 +## FreeBSD make (present only when running in concurrent mode).
 +       $$exit || (exit 1); exit 1
 
This obviously causes the `check' target to exit unconditionally with a
non-zero exit status.  Hardly a fix for anything, sigh.

The "proper fix" is even easier BTW: just use "$$exit || exit 1" instead.

The updated patch should now work (and I've tested it properly this time).
This new patch hasn't been derived by "cargo-cult guessing", but from the
explanation of the relevant FreeBSD make bug provided by Jilles Tjoelker:
 <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>

Now that's something to be documented in the Autoconf manual I'd say ...

> I have no idea why
> exactly it is so yet, so we might want to wait to apply the patch until
> we have fully understood the reasons of the original failure,
> 
Glad I've waited this time!  And I will allow another couple of days for
comments and suggestions before pushing.

Regards,
  Stefano

[0001-parallel-tests-no-more-spurious-successes-for-FreeBS.patch (text/x-patch, inline)]

From 3dfdeba20496399036825e321830e6079949e2ce Mon Sep 17 00:00:00 2001
Message-Id: <3dfdeba20496399036825e321830e6079949e2ce.1313510614.git.stefano.lattarini <at> gmail.com>
From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Date: Fri, 12 Aug 2011 18:29:28 +0200
Subject: [PATCH] parallel-tests: no more spurious successes for FreeBSD make

Work around a bug of FreeBSD make bug that was causing the
automake-generated "check" target to complete with success
even if some tests failed; this happened only when FreeBSD
make was run in concurrent mode (as in, e.g., "make -j2
check").  The bug is not present in NetBSD make.

This change fixes automake bug#9245:
 <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>

See also FreeBSD PR bin/159730:
 <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>

* lib/am/check.am [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Use a
more "safe" (and apparently redundant) idiom to exit with error,
so that the non-zero exit status is picked up also by FreeBSD
make when it's running in concurrent mode.
* NEWS: Update.
* tests/check-concurrency-bug9245.test: New test.
* tests/Makefile.am (TESTS): Update.
---
 ChangeLog                            |   20 +++++++++++
 NEWS                                 |    4 ++-
 lib/am/check.am                      |    7 +++-
 tests/Makefile.am                    |    1 +
 tests/Makefile.in                    |    4 ++-
 tests/check-concurrency-bug9245.test |   62 ++++++++++++++++++++++++++++++++++
 6 files changed, 95 insertions(+), 3 deletions(-)
 create mode 100755 tests/check-concurrency-bug9245.test

diff --git a/ChangeLog b/ChangeLog
index af2556f..9be4bbf 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,23 @@
+2011-08-16  Stefano Lattarini  <stefano.lattarini <at> gmail.com>
+
+	parallel-tests: no more spurious successes for FreeBSD make
+	Work around a bug of FreeBSD make bug that was causing the
+	automake-generated "check" target to complete with success
+	even if some tests failed; this happened only when FreeBSD
+	make was run in concurrent mode (as in, e.g., "make -j2
+	check").  The bug is not present in NetBSD make.
+	This change fixes automake bug#9245:
+	 <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
+	See also FreeBSD PR bin/159730:
+	 <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
+	* lib/am/check.am [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Use a
+	more "safe" (and apparently redundant) idiom to exit with error,
+	so that the non-zero exit status is picked up also by FreeBSD
+	make when it's running in concurrent mode.
+	* NEWS: Update.
+	* tests/check-concurrency-bug9245.test: New test.
+	* tests/Makefile.am (TESTS): Update.
+
 2011-08-08  Stefano Lattarini  <stefano.lattarini <at> gmail.com>
 
 	test defs: more environment cleanup
diff --git a/NEWS b/NEWS
index 29c552b..5086f75 100644
--- a/NEWS
+++ b/NEWS
@@ -23,7 +23,9 @@ Bugs fixed in 1.11.0a:
 * Bugs introduced by 1.11:
 
   - The `parallel-tests' test driver works around a GNU make 3.80 bug with
-    trailing white space in the test list (`TESTS = foo $(EMPTY)').
+    trailing white space in the test list (`TESTS = foo $(EMPTY)'), and
+    does not report spurious successes when used with concurrent FreeBSD
+    make (e.g., "make check -j3").
 
   - The `silent-rules' option now also silences all compile rules if dependency
     tracking is disabled.  Also, when `silent-rules' is not used, the output from
diff --git a/lib/am/check.am b/lib/am/check.am
index 0b54312..3d0188d 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -224,7 +224,12 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  col="$$red";							\
 	fi;								\
 	echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";		\
-	$$exit
+## The use of "exit 1" below is required to work around a FreeBSD make bug
+## (present only when running in concurrent mode).  See automake bug#9245:
+##  <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
+## and FreeBSD PR bin/159730:
+##  <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>.
+	$$exit || exit 1
 
 RECHECK_LOGS = $(TEST_LOGS)
 
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b545dda..c2a55fd 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -160,6 +160,7 @@ check11.test \
 check-exported-srcdir.test \
 check-tests-in-builddir.test \
 check-tests_environment.test \
+check-concurrency-bug9245.test \
 tests-environment-backcompat.test \
 checkall.test \
 clean.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index fe44a40..5a3259e 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -291,6 +291,7 @@ yacc-dist-nobuild-subdir.test \
 txinfo5.test
 
 parallel_tests = \
+check-concurrency-bug9245-p.test \
 check-exported-srcdir-p.test \
 check-tests-in-builddir-p.test \
 check-tests_environment-p.test \
@@ -437,6 +438,7 @@ check11.test \
 check-exported-srcdir.test \
 check-tests-in-builddir.test \
 check-tests_environment.test \
+check-concurrency-bug9245.test \
 tests-environment-backcompat.test \
 checkall.test \
 clean.test \
@@ -1274,7 +1276,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  col="$$red";							\
 	fi;								\
 	echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";		\
-	$$exit
+	$$exit || exit 1
 
 # Run all the tests.
 check-TESTS:
diff --git a/tests/check-concurrency-bug9245.test b/tests/check-concurrency-bug9245.test
new file mode 100755
index 0000000..e5dd153
--- /dev/null
+++ b/tests/check-concurrency-bug9245.test
@@ -0,0 +1,62 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Look for a bug where FreeBSD make in concurrent mode reported success
+# even when the Automake-generated parallel testsuite harness failed.
+# See automake bug#9245.
+
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = foo.test bar.test
+END
+
+cat > foo.test <<'END'
+#!/bin/sh
+exit 1
+END
+chmod a+x foo.test
+
+cp foo.test bar.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+# Some make implementations don't grok the `-j' option.
+$MAKE -j1 || Exit 77
+
+for j in '' -j1 -j2; do
+  $MAKE $j check && Exit 1
+  TESTS=foo.test $MAKE $j -e check && Exit 1
+  if test x"$parallel_tests" = x"yes"; then
+    $MAKE $j recheck && Exit 1
+    TEST_LOGS=foo.log $MAKE $j -e check && Exit 1
+    rm -f test-suite.log
+    $MAKE $j test-suite.log && Exit 1
+    test -f test-suite.log || Exit 1
+  else
+    : # For shells with buggy 'set -e'.
+  fi
+done
+
+:
-- 
1.7.2.3

Message #32 received at 9245 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: Autoconf <Autoconf <at> gnu.org>, 9245 <at> debbugs.gnu.org
Subject: Re: bug#9245: FreeBSD make in concurrent mode report spurious success
	in	automake-generated tests harness
Date: Tue, 16 Aug 2011 10:18:51 -0600

On 08/16/2011 10:04 AM, Stefano Lattarini wrote:
> The "proper fix" is even easier BTW: just use "$$exit || exit 1" instead.
>
> The updated patch should now work (and I've tested it properly this time).
> This new patch hasn't been derived by "cargo-cult guessing", but from the
> explanation of the relevant FreeBSD make bug provided by Jilles Tjoelker:
>   <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
>
> Now that's something to be documented in the Autoconf manual I'd say ...

Indeed.  So it all boils down to BSD make rewriting rules (but only for 
make -j) to be roughly:

sh -c 'set -v; your rules; set +v'

and losing your exit status (not to mention sharing shell status between 
commands that are supposed to be in independent shells), when it really 
should have been rewriting as something more complex:

sh -c '(your rule 1); (your rule 2); st=$?; set +v; exit $st'

And your workaround of an explicit exit rather than relying on $? at the 
end of your 'set +e' invocation is indeed the proper way to force the 
parallel make bug to be avoided, by exiting prior to any post-commands 
injected by the BSD make rewriting going on that corrupt $?.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Message #35 received at 9245 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: Autoconf <Autoconf <at> gnu.org>, 9245 <at> debbugs.gnu.org
Subject: Re: bug#9245: FreeBSD make in concurrent mode report spurious success
	in automake-generated tests harness
Date: Tue, 16 Aug 2011 18:28:12 +0200

Hi Eric.

On Tuesday 16 August 2011, Eric Blake wrote:
> On 08/16/2011 10:04 AM, Stefano Lattarini wrote:
> > The "proper fix" is even easier BTW: just use "$$exit || exit 1" instead.
> >
> > The updated patch should now work (and I've tested it properly this time).
> > This new patch hasn't been derived by "cargo-cult guessing", but from the
> > explanation of the relevant FreeBSD make bug provided by Jilles Tjoelker:
> >   <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
> >
> > Now that's something to be documented in the Autoconf manual I'd say ...
> 
> Indeed.  So it all boils down to BSD make rewriting rules (but only for 
> make -j) to be roughly:
> 
> sh -c 'set -v; your rules; set +v'
> 
> and losing your exit status (not to mention sharing shell status between 
> commands that are supposed to be in independent shells), when it really 
> should have been rewriting as something more complex:
> 
> sh -c '(your rule 1); (your rule 2); st=$?; set +v; exit $st'
> 
> And your workaround of an explicit exit rather than relying on $? at the 
> end of your 'set +e' invocation is indeed the proper way to force the 
> parallel make bug to be avoided, by exiting prior to any post-commands 
> injected by the BSD make rewriting going on that corrupt $?.
> 
I'll have a "draft patch" read soonish.  There is ample room for improvements,
but I'll post it here anyway since it can benefit from early feedback.

Regards,
  Stefano

Message #42 received at 9245-done <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: 9245-done <at> debbugs.gnu.org
Subject: Re: bug#9245: FreeBSD make in concurrent mode report spurious success
	in automake-generated tests harness
Date: Thu, 18 Aug 2011 21:42:32 +0200

On Tuesday 16 August 2011, Stefano Lattarini wrote:
> On Friday 12 August 2011, Stefano Lattarini wrote:
> > OK, the attached hacky patch seems to fix the bug.
> >
> LOL no, there was an embarassingly stupid error in my patch:
> 
>  diff --git a/lib/am/check.am b/lib/am/check.am
>  index 0b54312..75f8bba 100644
>  --- a/lib/am/check.am
>  +++ b/lib/am/check.am
>  @@ -224,7 +224,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
>            col="$$red";                                                  \
>          fi;                                                             \
>          echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";         \
>  -       $$exit
>  +## The apparently redundant and repeated "exit 1" works around a bug in
>  +## FreeBSD make (present only when running in concurrent mode).
>  +       $$exit || (exit 1); exit 1
>  
> This obviously causes the `check' target to exit unconditionally with a
> non-zero exit status.  Hardly a fix for anything, sigh.
> 
> The "proper fix" is even easier BTW: just use "$$exit || exit 1" instead.
> 
> The updated patch should now work (and I've tested it properly this time).
> This new patch hasn't been derived by "cargo-cult guessing", but from the
> explanation of the relevant FreeBSD make bug provided by Jilles Tjoelker:
>  <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
> 
> Now that's something to be documented in the Autoconf manual I'd say ...
>
BTW, done with commit v2.68-84-g96cdf32 "docs: other issues with parallel BSD
make"; see:
 <http://lists.gnu.org/archive/html/autoconf-patches/2011-08/msg00013.html>

> > I have no idea why
> > exactly it is so yet, so we might want to wait to apply the patch until
> > we have fully understood the reasons of the original failure,
> > 
> Glad I've waited this time!  And I will allow another couple of days for
> comments and suggestions before pushing.
> 
> Regards,
>   Stefano
> 
I've pushed the patch now, with the additional squash-in below.  I'm closing
this bug report.

Regards,
  Stefano

-*-*-

diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 2f553ed..17a2002 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -405,7 +405,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
          col="$$red";                                                  \
        fi;                                                             \
        echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";         \
-       $$exit
+       $$exit || exit 1
 
 # Run all the tests.
 check-TESTS:

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.