Subject: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness
From: Stefano Lattarini <[email-masked]>
To: [email-masked]
Date: Fri, 5 Aug 2011 11:08:20 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Here is a brief illustration of the issue:

  $ cat > Makefile.am <<'END'
  AUTOMAKE_OPTIONS = foreign parallel-tests -Wall -Werror
  TEST_LOG_COMPILER = false
  TESTS = foo.test bar.test
  END
  $ touch foo.test bar.test
  $ autoreconf -vi && ./configure
  ...

  $ freebsd-make -j2 check; echo %% STATUS = $? %%
  freebsd-make  check-TESTS
  FAIL: foo.test
  FAIL: bar.test
  ====================
  2 of 2 tests failed
  See ./test-suite.log
  ====================
  %% STATUS = 0 %%

It works as expected with GNU make and NetBSD make though:

 $ gmake -j2 check; echo %% STATUS = $? %%
  gmake  check-TESTS
  gmake[1]: Entering directory `/tmp/...'
  gmake[2]: Entering directory `/tmp/...'
  FAIL: foo.test
  FAIL: bar.test
  ====================
  2 of 2 tests failed
  See ./test-suite.log
  ====================
  gmake[2]: *** [test-suite.log] Error 1
  gmake[2]: Leaving directory `/tmp/...'
  gmake[1]: *** [check-TESTS] Error 2
  gmake[1]: Leaving directory `/tmp/...'
  gmake: *** [check-am] Error 2
  %% STATUS = 2 %%

  $ netbsd-make -j2 check; echo %% STATUS = $? %%
  --- check-am ---
  netbsd-make  check-TESTS
  --- check-TESTS ---
  --- foo.log ---
  --- bar.log ---
  FAIL: bar.test
  --- foo.log ---
  FAIL: foo.test
  --- test-suite.log ---
  ====================
  2 of 2 tests failed
  See ./test-suite.log
  ====================
  *** [test-suite.log] Error code 1
  1 error
  netbsd-make: stopped in /tmp/...
  *** [check-TESTS] Error code 2
  1 error
  netbsd-make: stopped in /tmp/...
  *** [check-am] Error code 2
  1 error
  netbsd-make: stopped in /tmp/...
  %% STATUS = 2 %%

The problem is present both in automake 1.11.1 and in the developement
version from the 'test-protocols' branch.

Regards,
  Stefano