GNU bug report logs - #72384
srfi-64: test-end should not clear fail list

Previous Next

Full log

View this message in rfc822 format

From: Taylan Kammer <taylan.kammer <at> gmail.com>
To: Tomas Volf <~@wolfsden.cz>, 72384 <at> debbugs.gnu.org
Subject: bug#72384: srfi-64: test-end should not clear fail list
Date: Wed, 2 Oct 2024 01:38:33 +0200

[Message part 1 (text/plain, inline)]

On 30.07.2024 21:52, Tomas Volf wrote:
> Hello,
>
> I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.
>
> Reading the specification for test-expect-fail I do not see a mandate to clear
> expect-fail list on test-end.  test-skip does have such provision, but it is
> lacking in the test-expect-fail.  Therefore I think current behavior is wrong:
>
>     (use-modules (srfi srfi-64))
>
>     (test-begin "x")
>
>     (test-begin "group1")
>     (test-expect-fail "test-a")
>     (test-assert "test-a" #t)
>     (pk (test-result-kind))
>     (test-end "group1")
>     (test-assert "test-a" #t)
>     (pk (test-result-kind))
>
>     (test-end)
>
> Leading to:
>
>     ;;; (xpass)
>
>     ;;; (pass)
>
> Have a nice day
> Tomas Volf
>
>
I'm inclined to see this as an error/omission in the spec itself. It makes sense for test-end to clear the expected-fail list, just like it clears the skip list. If it didn't, it might cause one to accidentally mark tests as "expected failure" that weren't meant to be marked as such. Consider the following:

(test-group "group1"
  (test-expect-fail "test-a")
  (test-assert "test-a" #t))

(test-group "group2"
  (test-assert "test-a" #t))

Since test-group is equivalent to a pair of test-begin/test-end calls, this code would be very "deceptive" if the implicit test-end didn't clear the expected-fail list. After all, the two groups look completely disjoint, and one wouldn't expect any state from the former to implicitly bleed into the latter. Also, if I'm not mistaken, there's not even a way to clear the expected-fail list explicitly.

I actually have some "real-world" code that uses repetitive names in a test suite within different groups, so this isn't just a theoretical issue either:

    https://codeberg.org/taylan/scheme-bytestructures/src/branch/master/run-tests.body.scm

Notice how often the names "ref" and "set" are used.

So, I think the behavior of the reference implementation is correct/desirable here. My implementation of SRFI-64 does the same.

- Taylan

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.