GNU bug report logs - #11898
automake 1.12.2: self-check-exit.tap fails on Mac OS X 10.7.4

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 11898 in the body.
You can then email your comments to 11898 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Max Horn <max <at> quendi.de>
To: bug-automake <at> gnu.org
Subject: automake 1.12.2: self-check-exit.tap fails on Mac OS X 10.7.4
Date: Tue, 10 Jul 2012 16:26:57 +0200

[Message part 1 (text/plain, inline)]

Hi there,

there is another test failure I am experiencing on Mac OS X 10.7.4, but this time I have no idea what's wrong (in particular, no idea if this is a problem with my particular system; with OS Xl or with automake). Anyway, here are the fails:

PASS: t/self-check-exit.tap 31 - command not found
PASS: t/self-check-exit.tap 32 - permission denied
FAIL: t/self-check-exit.tap 33 - syntax error 1
FAIL: t/self-check-exit.tap 34 - syntax error 2


Attached is the log the test produced on my system. I have a hard time making any sense out of tap files, and also this .log file, so I hope somebody who knows more about that stuff might see right away what is wrong.

Cheers,
Max

[self-check-exit.log (application/octet-stream, attachment)]

Message #8 received at 11898 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Max Horn <max <at> quendi.de>
Cc: 11898 <at> debbugs.gnu.org
Subject: Re: bug#11898: automake 1.12.2: self-check-exit.tap fails on Mac
	OS X 10.7.4
Date: Tue, 10 Jul 2012 16:54:00 +0200

severity 11898 minor
thanks

On 07/10/2012 04:26 PM, Max Horn wrote:
> Hi there,
> 
> there is another test failure I am experiencing on Mac OS X 10.7.4,
> but this time I have no idea what's wrong (in particular, no idea
> if this is a problem with my particular system; with OS X or with
> automake). Anyway, here are the fails:
> 
> PASS: t/self-check-exit.tap 31 - command not found
> PASS: t/self-check-exit.tap 32 - permission denied
> FAIL: t/self-check-exit.tap 33 - syntax error 1
> FAIL: t/self-check-exit.tap 34 - syntax error 2
>
Yeah, I'm seeing something similar on Solaris 10 when the /usr/xpg4/bin/sh
shell is used.  What shell is your /bin/sh exactly (Bash, Zsh, something
else)?

Anyway, this failure shows that we cannot trust that our exit trap will
be able to portably detect tests that fails due to syntax error in them
-- they will be reported as passed instead :-/

So I think we should add a check-local recipe (or something like that)
that verifies, using the "-n" option od the shell, that our test cases
are syntactically valid.  Then we can get rid of the two spuriously
failing tests in self-check-exit.tap.

> 
> Attached is the log the test produced on my system. I have a hard time
> making any sense out of tap files, and also this .log file, so I hope
> somebody who knows more about that stuff might see right away what is
> wrong.
> 
> Cheers,
> Max

Thanks,
  Stefano

Message #13 received at 11898 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Max Horn <postbox <at> quendi.de>
Cc: 11898 <at> debbugs.gnu.org
Subject: Re: bug#11898: automake 1.12.2: self-check-exit.tap fails on Mac
	OS X 10.7.4
Date: Tue, 10 Jul 2012 17:58:41 +0200

Max, could you keep the mailing list in the loop in your future replies,
so that they will be registered in the bug tracker?  Thanks.

For this time, I'm quoting the relevant part of your reply, so no need to
re-send.

On 07/10/2012 05:44 PM, Max Horn wrote:
> 
> On 10.07.2012, at 16:54, Stefano Lattarini wrote:
> 
>> On 07/10/2012 04:26 PM, Max Horn wrote:
>>
>>> PASS: t/self-check-exit.tap 31 - command not found
>>> PASS: t/self-check-exit.tap 32 - permission denied
>>> FAIL: t/self-check-exit.tap 33 - syntax error 1
>>> FAIL: t/self-check-exit.tap 34 - syntax error 2
>>>
>> Yeah, I'm seeing something similar on Solaris 10 when the /usr/xpg4/bin/sh
>> shell is used.  What shell is your /bin/sh exactly (Bash, Zsh, something
>> else)?
> 
> $ /bin/sh --version
> GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin11)
> Copyright (C) 2007 Free Software Foundation, Inc.
>
Yes, I can reproduce the issue:

  $ cat > foo.sh << 'END'
  trap 'exit $?' 0
  set -e
  true ( true )
  END

  $ bash-3.2 foo.sh; echo status = $?
  foo.sh: line 3: syntax error near unexpected token `true'
  status = 0

  $ bash-3.0 foo.sh; echo status = $?
  foo.sh: line 3: syntax error near unexpected token `true'
  status = 0

  $ bash-4.1 foo.sh ; echo status = $?
  foo.sh: line 3: syntax error near unexpected token `true'
  status = 2

  # On Solaris 10:
  $ /usr/xpg4/bin/sh foo.sh; echo status = $?
  foo.sh[3]: syntax error at line 3 : `(' unexpected
  status = 0

>
> This is on Mac OS X 10.7.4.
> 
> Max
> 

Regards,
  Stefano

Message #16 received at 11898 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: automake-patches <at> gnu.org
Cc: 11898 <at> debbugs.gnu.org
Subject: [PATCH] tests: verify the shell test scripts are syntactically valid
Date: Wed, 11 Jul 2012 11:30:35 +0200

Fixes automake bug#11898.

This measure of extra safety is mostly motivated by the fact that some
shells (at least some versions of Bash in the 3.x release series, one
of which severs as /bin/sh on Mac OS X 10.7, as well as Bash 4.0 and the
/usr/xpg4/bin/sh shell from Solaris 10) erroneously exit with exit status
0 upon encountering a syntax error, if an exit trap is sett (as it is in
our test scripts).

* Makefile.am (check-tests-syntax): New, check that the shell test
scripts listed in $(TESTS) are syntactically correct.
(.PHONY, check-local): Depend on it.
* t/self-check-exit.tap : Remove checks verifying that a script exits
with non-zero status upon encountering a syntax error; as explained
above, we can't depend on that.

Signed-off-by: Stefano Lattarini <stefano.lattarini <at> gmail.com>
---

 I plan to push this by this evening or tomorrow.

 Regards,
   Stefano

 Makefile.am           |   35 +++++++++++++++++++++++++++++++++++
 t/self-check-exit.tap |    8 +-------
 2 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index 2764481..5aa543e 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -426,6 +426,41 @@ check-no-repeated-test-name:
 check-local: check-no-repeated-test-name
 .PHONY: check-no-repeated-test-name
 
+# Check that our test cases are syntactically correct.
+# See automake bug#11898.
+check-tests-syntax:
+	@st=0; \
+	err () { echo "$@: $$*" >&2; st=1; }; \
+## The user might do something like "make check TESTS=t/foo" or
+## "make check TESTS_LOGS=t/foo.log" and expect (say) the test
+## 't/foo.sh' to be run; this has worked well until today, and
+## we want to continue supporting this use case.
+	bases=`for log in : $(TEST_LOGS); do echo $$log; done \
+	  | sed -e '/^:$$/d' -e 's/\.log$$//'`; \
+	for bas in $$bases; do \
+	  for suf in sh tap pl; do \
+	    tst=$$bas.$$suf; \
+## Emulate VPATH search.
+	    if test -f $$tst; then \
+	      break; \
+	    elif test -f $(srcdir)/$$tst; then \
+	      tst=$(srcdir)/$$tst; \
+	      break; \
+	    else \
+	      tst=''; \
+	    fi; \
+	  done; \
+	  test -n "$$tst" || err "couldn't find test '$$bas'"; \
+## Don't check that perl tests are valid shell scripts!
+	  test $$suf = pl && continue; \
+	  $(AM_V_P) && echo " $(AM_TEST_RUNNER_SHELL) -n $$tst"; \
+	  $(AM_TEST_RUNNER_SHELL) -n "$$tst" \
+	    || err "test '$$tst' syntactically invalid"; \
+	done; \
+	exit $$st
+check-local: check-tests-syntax
+.PHONY: check-tests-syntax
+
 ## Checking the list of tests.
 test_subdirs = t t/pm t/perf
 include $(srcdir)/t/CheckListOfTests.am
diff --git a/t/self-check-exit.tap b/t/self-check-exit.tap
index 0fbc9d8..19cf27a 100755
--- a/t/self-check-exit.tap
+++ b/t/self-check-exit.tap
@@ -22,7 +22,7 @@
 am_create_testdir=no
 . ./defs || exit 99
 
-plan_ 34
+plan_ 32
 
 # This test becomes more cumbersome if we keep the 'errexit' shell flag
 # set.  And removing it is no big deal, as this test is a TAP-based one,
@@ -79,10 +79,4 @@ test -f Makefile && test ! -x Makefile || \
 $AM_TEST_RUNNER_SHELL -c "$init ./Makefile; :" "$dummy_test_script"
 command_ok_ "permission denied" test $? -gt 0
 
-: Syntax errors in the test code.
-$AM_TEST_RUNNER_SHELL -c "$init if :; then" "$dummy_test_script"
-command_ok_ "syntax error 1" test $? -gt 0
-$AM_TEST_RUNNER_SHELL -c "$init true ( true )" "$dummy_test_script"
-command_ok_ "syntax error 2" test $? -gt 0
-
 :
-- 
1.7.9.5

Message #19 received at 11898 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: automake-patches <at> gnu.org
Cc: 11898 <at> debbugs.gnu.org
Subject: Re: [PATCH] tests: verify the shell test scripts are syntactically
	valid
Date: Fri, 13 Jul 2012 14:49:00 +0200

tags 11898 + patch
close 11898
thanks

On 07/11/2012 11:30 AM, Stefano Lattarini wrote:
> [PATCH] tests: verify the shell test scripts are syntactically valid
> 
> This measure of extra safety is mostly motivated by the fact that some
> shells (at least some versions of Bash in the 3.x release series, one
> of which severs
>
Oops, s/severs/serves/ here.  Fixed before pushing.

> as /bin/sh on Mac OS X 10.7, as well as Bash 4.0 and the
> /usr/xpg4/bin/sh shell from Solaris 10) erroneously exit with exit status
> 0 upon encountering a syntax error, if an exit trap is sett (as it is in
> our test scripts).
> 
> * Makefile.am (check-tests-syntax): New, check that the shell test
> scripts listed in $(TESTS) are syntactically correct.
> (.PHONY, check-local): Depend on it.
> * t/self-check-exit.tap : Remove checks verifying that a script exits
> with non-zero status upon encountering a syntax error; as explained
> above, we can't depend on that.
>
Pushed now.  I'm thus closing this bug report.

Thanks,
  Stefano

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.