GNU bug report logs - #23521
control test "PASS" exit status on a per-test basis

Previous Next

Full log

Message #8 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Lirzin <mthl <at> gnu.org>
To: Reuben Thomas <rrt <at> sc3d.org>
Cc: 23521 <at> debbugs.gnu.org
Subject: Re: bug#23521: XFAIL
Date: Thu, 19 May 2016 01:04:28 +0200

Hi,

Reuben Thomas <rrt <at> sc3d.org> writes:

> The documentation says: "It's not uncommon, especially during early
> development stages, that some tests fail for known reasons, and that
> the developer doesn't want
> to tackle these failures immediately (this is especially true when the
> failing tests deal with corner cases)."
>
> Another common use for "expected failure" is to write tests to check
> that error conditions arise as expected, for example, by checking that
> a program raises an error when given invalid input.

I agree that XFAIL can be ambiguous, however I think this usage is not
desirable.  It gives an additional opposite meaning to XFAIL symbol
which makes it even more confusing.

> If that's a reasonable use of automake's test harness, perhaps the
> documentation could reflect that, e.g. by adding:
>
> "Another use for XFAIL is to mark tests that are supposed to fail, for
> example, to check that a program raises an error when given invalid
> input."
>
> It is often easier to write expected-to-fail tests this way (so that
> they can all look the same), rather than have to have, for example, an
> extra driver that converts expected errors into success codes for the
> automake test harness.

What do you mean precisely by “an extra driver”?

Thanks

-- 
Mathieu Lirzin

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.