GNU bug report logs - #72371
srfi-64: test marked for skip and as expected failure has wrong result-kind in on-test-begin-function

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 72371 in the body.
You can then email your comments to 72371 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tomas Volf <~@wolfsden.cz>
To: bug-guile <at> gnu.org
Subject: srfi-64: test marked for skip and as expected failure has wrong
 result-kind in on-test-begin-function
Date: Tue, 30 Jul 2024 21:51:54 +0200

Hello,

I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.

The specification says the following regarding the test-result-kind:

> If we've started on a new test, but don't have a result yet, then the result
> kind is 'xfail if the test is expected to fail, 'skip if the test is supposed
> to be skipped, or #f otherwise.

Thus I believe that following should print `xfail':

    (use-modules (srfi srfi-64))
    (test-begin "x")

    (test-runner-on-test-begin! (test-runner-current)
      (λ (runner)
        (pk (test-result-kind))))

    (test-skip 1)
    (test-expect-fail 1)
    (test-assert #t)

    (test-end)

However it does not:

    ;;; (skip)

Have a nice day
Tomas Volf

Message #8 received at 72371 <at> debbugs.gnu.org (full text, mbox):

From: Taylan Kammer <taylan.kammer <at> gmail.com>
To: Tomas Volf <~@wolfsden.cz>, 72371 <at> debbugs.gnu.org
Subject: Re: bug#72371: srfi-64: test marked for skip and as expected failure
 has wrong result-kind in on-test-begin-function
Date: Tue, 1 Oct 2024 23:45:49 +0200

On 30.07.2024 21:51, Tomas Volf wrote:
> Hello,
>
> I think I found a bug in (srfi srfi-64) module shipped with GNU Guile.
>
> The specification says the following regarding the test-result-kind:
>
>> If we've started on a new test, but don't have a result yet, then the result
>> kind is 'xfail if the test is expected to fail, 'skip if the test is supposed
>> to be skipped, or #f otherwise.
> Thus I believe that following should print `xfail':
>
>     (use-modules (srfi srfi-64))
>     (test-begin "x")
>
>     (test-runner-on-test-begin! (test-runner-current)
>       (λ (runner)
>         (pk (test-result-kind))))
>
>     (test-skip 1)
>     (test-expect-fail 1)
>     (test-assert #t)
>
>     (test-end)
>
> However it does not:
>
>     ;;; (skip)
>
> Have a nice day
> Tomas Volf
>
I think this is a case where the spec didn't actually consider what should happen if skip and expect-fail are combined. Otherwise, I would expect to see a more explicit description of what should happen in such cases.

In other words, I think the English description of what's supposed to happen, that you've quoted, is *not* intended to be read like procedural pseudo-code: "If expected to fail, return 'xfail; if supposed to be skipped, return 'skip." The reference implementation does it the exact other way around, in a rather straightforward manner (two consecutive clasuses of a cond expression), so I don't think it's a bug.

Intuitively, I also think it makes the most sense to treat skipping as a higher priority. While an xfail test is still executed, a skipped test is not executed at all, which is a more significant change in the test suite's behavior and should be honored IMO. If I've marked a test to be skipped, it could be because executing it currently leads to a crash or an infinite loop, so it would be important to skip it even if it's marked as xfail.

So, I think the observed behavior is probably best, and intended. Opinions welcome.

- Taylan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.