GNU bug report logs - #72384
srfi-64: test-end should not clear fail list

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 72384 in the body.
You can then email your comments to 72384 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tomas Volf <~@wolfsden.cz>
To: bug-guile <at> gnu.org
Subject: srfi-64: test-end should not clear fail list
Date: Tue, 30 Jul 2024 21:52:00 +0200

Hello,

I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.

Reading the specification for test-expect-fail I do not see a mandate to clear
expect-fail list on test-end.  test-skip does have such provision, but it is
lacking in the test-expect-fail.  Therefore I think current behavior is wrong:

    (use-modules (srfi srfi-64))

    (test-begin "x")

    (test-begin "group1")
    (test-expect-fail "test-a")
    (test-assert "test-a" #t)
    (pk (test-result-kind))
    (test-end "group1")
    (test-assert "test-a" #t)
    (pk (test-result-kind))

    (test-end)

Leading to:

    ;;; (xpass)

    ;;; (pass)

Have a nice day
Tomas Volf

Message #8 received at 72384 <at> debbugs.gnu.org (full text, mbox):

From: Taylan Kammer <taylan.kammer <at> gmail.com>
To: Tomas Volf <~@wolfsden.cz>, 72384 <at> debbugs.gnu.org
Subject: Re: bug#72384: srfi-64: test-end should not clear fail list
Date: Wed, 2 Oct 2024 01:38:33 +0200

[Message part 1 (text/plain, inline)]

On 30.07.2024 21:52, Tomas Volf wrote:
> Hello,
>
> I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.
>
> Reading the specification for test-expect-fail I do not see a mandate to clear
> expect-fail list on test-end.  test-skip does have such provision, but it is
> lacking in the test-expect-fail.  Therefore I think current behavior is wrong:
>
>     (use-modules (srfi srfi-64))
>
>     (test-begin "x")
>
>     (test-begin "group1")
>     (test-expect-fail "test-a")
>     (test-assert "test-a" #t)
>     (pk (test-result-kind))
>     (test-end "group1")
>     (test-assert "test-a" #t)
>     (pk (test-result-kind))
>
>     (test-end)
>
> Leading to:
>
>     ;;; (xpass)
>
>     ;;; (pass)
>
> Have a nice day
> Tomas Volf
>
>
I'm inclined to see this as an error/omission in the spec itself. It makes sense for test-end to clear the expected-fail list, just like it clears the skip list. If it didn't, it might cause one to accidentally mark tests as "expected failure" that weren't meant to be marked as such. Consider the following:

(test-group "group1"
  (test-expect-fail "test-a")
  (test-assert "test-a" #t))

(test-group "group2"
  (test-assert "test-a" #t))

Since test-group is equivalent to a pair of test-begin/test-end calls, this code would be very "deceptive" if the implicit test-end didn't clear the expected-fail list. After all, the two groups look completely disjoint, and one wouldn't expect any state from the former to implicitly bleed into the latter. Also, if I'm not mistaken, there's not even a way to clear the expected-fail list explicitly.

I actually have some "real-world" code that uses repetitive names in a test suite within different groups, so this isn't just a theoretical issue either:

    https://codeberg.org/taylan/scheme-bytestructures/src/branch/master/run-tests.body.scm

Notice how often the names "ref" and "set" are used.

So, I think the behavior of the reference implementation is correct/desirable here. My implementation of SRFI-64 does the same.

- Taylan

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.