GNU bug report logs - #23521
control test "PASS" exit status on a per-test basis

Previous Next

To reply to this bug, email your comments to 23521 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Reuben Thomas <rrt <at> sc3d.org>
To: bug-automake <bug-automake <at> gnu.org>
Subject: XFAIL
Date: Thu, 12 May 2016 11:01:11 +0100

[Message part 1 (text/plain, inline)]

The documentation says: "It's not uncommon, especially during early
development stages, that some tests fail for known reasons, and that the
developer doesn't want
to tackle these failures immediately (this is especially true when the
failing tests deal with corner cases)."

Another common use for "expected failure" is to write tests to check that
error conditions arise as expected, for example, by checking that a program
raises an error when given invalid input.

If that's a reasonable use of automake's test harness, perhaps the
documentation could reflect that, e.g. by adding:

"Another use for XFAIL is to mark tests that are supposed to fail, for
example, to check that a program raises an error when given invalid input."

It is often easier to write expected-to-fail tests this way (so that they
can all look the same), rather than have to have, for example, an extra
driver that converts expected errors into success codes for the automake
test harness.

-- 
http://rrt.sc3d.org

[Message part 2 (text/html, inline)]

Message #8 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Lirzin <mthl <at> gnu.org>
To: Reuben Thomas <rrt <at> sc3d.org>
Cc: 23521 <at> debbugs.gnu.org
Subject: Re: bug#23521: XFAIL
Date: Thu, 19 May 2016 01:04:28 +0200

Hi,

Reuben Thomas <rrt <at> sc3d.org> writes:

> The documentation says: "It's not uncommon, especially during early
> development stages, that some tests fail for known reasons, and that
> the developer doesn't want
> to tackle these failures immediately (this is especially true when the
> failing tests deal with corner cases)."
>
> Another common use for "expected failure" is to write tests to check
> that error conditions arise as expected, for example, by checking that
> a program raises an error when given invalid input.

I agree that XFAIL can be ambiguous, however I think this usage is not
desirable.  It gives an additional opposite meaning to XFAIL symbol
which makes it even more confusing.

> If that's a reasonable use of automake's test harness, perhaps the
> documentation could reflect that, e.g. by adding:
>
> "Another use for XFAIL is to mark tests that are supposed to fail, for
> example, to check that a program raises an error when given invalid
> input."
>
> It is often easier to write expected-to-fail tests this way (so that
> they can all look the same), rather than have to have, for example, an
> extra driver that converts expected errors into success codes for the
> automake test harness.

What do you mean precisely by “an extra driver”?

Thanks

-- 
Mathieu Lirzin

Message #11 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Peter Johansson <trojkan <at> gmail.com>
To: Mathieu Lirzin <mthl <at> gnu.org>, Reuben Thomas <rrt <at> sc3d.org>
Cc: 23521 <at> debbugs.gnu.org
Subject: Re: bug#23521: XFAIL
Date: Thu, 19 May 2016 09:55:51 +1000

On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:
>> Another common use for "expected failure" is to write tests to check
>> >that error conditions arise as expected, for example, by checking that
>> >a program raises an error when given invalid input.
> I agree that XFAIL can be ambiguous, however I think this usage is not
> desirable.  It gives an additional opposite meaning to XFAIL symbol
> which makes it even more confusing.
>
I agree. When I wanna tests that a program fails with incorrect input, I 
prefer writing a tests that calls the program, check that it fails (exit 
1 or whatever is expected), and perhaps even parse the error message, 
and if it looks like I expect exit 0 aka PASS.

Cheers,
Peter

Message #14 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Reuben Thomas <rrt <at> sc3d.org>
To: Mathieu Lirzin <mthl <at> gnu.org>
Cc: 23521 <at> debbugs.gnu.org
Subject: Re: bug#23521: XFAIL
Date: Thu, 19 May 2016 10:15:33 +0100

[Message part 1 (text/plain, inline)]

On 19 May 2016 at 00:04, Mathieu Lirzin <mthl <at> gnu.org> wrote:

> > It is often easier to write expected-to-fail tests this way (so that
> > they can all look the same), rather than have to have, for example, an
> > extra driver that converts expected errors into success codes for the
> > automake test harness.
>
> What do you mean precisely by “an extra driver”?
>

​A custom test driver.​

-- 
http://rrt.sc3d.org

[Message part 2 (text/html, inline)]

Message #17 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Reuben Thomas <rrt <at> sc3d.org>
To: Peter Johansson <trojkan <at> gmail.com>
Cc: 23521 <at> debbugs.gnu.org, Mathieu Lirzin <mthl <at> gnu.org>
Subject: Re: bug#23521: XFAIL
Date: Thu, 19 May 2016 10:16:28 +0100

[Message part 1 (text/plain, inline)]

On 19 May 2016 at 00:55, Peter Johansson <trojkan <at> gmail.com> wrote:

>
>
> On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:
>
>> Another common use for "expected failure" is to write tests to check
>>> >that error conditions arise as expected, for example, by checking that
>>> >a program raises an error when given invalid input.
>>>
>> I agree that XFAIL can be ambiguous, however I think this usage is not
>> desirable.  It gives an additional opposite meaning to XFAIL symbol
>> which makes it even more confusing.
>>
>> I agree. When I wanna tests that a program fails with incorrect input, I
> prefer writing a tests that calls the program, check that it fails (exit 1
> or whatever is expected), and perhaps even parse the error message, and if
> it looks like I expect exit 0 aka PASS.
>

​Thanks. I shall continue with my "deviant" usage for now, but if that is
not considered normal, I understand that you won't want to document it.

-- 
http://rrt.sc3d.org

[Message part 2 (text/html, inline)]

Message #20 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Gavin Smith <gavinsmith0123 <at> gmail.com>
To: Mathieu Lirzin <mthl <at> gnu.org>
Cc: 23521 <at> debbugs.gnu.org, Reuben Thomas <rrt <at> sc3d.org>
Subject: Re: bug#23521: XFAIL
Date: Fri, 20 May 2016 15:58:02 +0100

On 19 May 2016 at 00:04, Mathieu Lirzin <mthl <at> gnu.org> wrote:
>> It is often easier to write expected-to-fail tests this way (so that
>> they can all look the same), rather than have to have, for example, an
>> extra driver that converts expected errors into success codes for the
>> automake test harness.
>
> What do you mean precisely by “an extra driver”?

This would be a reference to a "custom test driver".
https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support

Message #23 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Reuben Thomas <rrt <at> sc3d.org>
To: Gavin Smith <gavinsmith0123 <at> gmail.com>
Cc: 23521 <at> debbugs.gnu.org, Mathieu Lirzin <mthl <at> gnu.org>
Subject: Re: bug#23521: XFAIL
Date: Fri, 20 May 2016 16:03:59 +0100

[Message part 1 (text/plain, inline)]

On 20 May 2016 at 15:58, Gavin Smith <gavinsmith0123 <at> gmail.com> wrote:

> On 19 May 2016 at 00:04, Mathieu Lirzin <mthl <at> gnu.org> wrote:
> >> It is often easier to write expected-to-fail tests this way (so that
> >> they can all look the same), rather than have to have, for example, an
> >> extra driver that converts expected errors into success codes for the
> >> automake test harness.
> >
> > What do you mean precisely by “an extra driver”?
>
> This would be a reference to a "custom test driver".
>
> https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support
>

​Thanks, that's what I meant.

-- 
http://rrt.sc3d.org

[Message part 2 (text/html, inline)]

Message #26 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Lirzin <mthl <at> gnu.org>
To: Reuben Thomas <rrt <at> sc3d.org>
Cc: 23521 <at> debbugs.gnu.org
Subject: Re: bug#23521: XFAIL
Date: Fri, 20 May 2016 17:49:54 +0200

Reuben Thomas <rrt <at> sc3d.org> writes:

> On 19 May 2016 at 00:04, Mathieu Lirzin <mthl <at> gnu.org> wrote:
>
>     > It is often easier to write expected-to-fail tests this way (so
>     that
>     > they can all look the same), rather than have to have, for
>     example, an
>     > extra driver that converts expected errors into success codes
>     for the
>     > automake test harness.
>     
>     What do you mean precisely by “an extra driver”?
>     
>
> ​A custom test driver.​

OK, I wasn't sure.  Indeed a custom test driver seems a bit heavy just
checking failures.  IMO the solution Peter proposed is nice and simple.

-- 
Mathieu Lirzin

Message #29 received at 23521 <at> debbugs.gnu.org (full text, mbox):

From: Reuben Thomas <rrt <at> sc3d.org>
To: Mathieu Lirzin <mthl <at> gnu.org>
Cc: 23521 <at> debbugs.gnu.org
Subject: Re: bug#23521: XFAIL
Date: Fri, 20 May 2016 17:00:30 +0100

[Message part 1 (text/plain, inline)]

On 20 May 2016 at 16:49, Mathieu Lirzin <mthl <at> gnu.org> wrote:

> Reuben Thomas <rrt <at> sc3d.org> writes:
>
> > On 19 May 2016 at 00:04, Mathieu Lirzin <mthl <at> gnu.org> wrote:
> >
> >     > It is often easier to write expected-to-fail tests this way (so
> >     that
> >     > they can all look the same), rather than have to have, for
> >     example, an
> >     > extra driver that converts expected errors into success codes
> >     for the
> >     > automake test harness.
> >
> >     What do you mean precisely by “an extra driver”?
> >
> >
> > ​A custom test driver.​
>
> OK, I wasn't sure.  Indeed a custom test driver seems a bit heavy just
> checking failures.  IMO the solution Peter proposed is nice and simple.
>

​What Peter proposed is essentially a custom test driver: I would not
expect to duplicate the logic to check the return code &c. in each test
expected to fail; rather, I would put it in a custom test driver that would
handle expected fails and mark them as passes. (My expected fails are all
of the same type, i.e. a non-zero exit code. It might additionally be
useful, as Peter suggests, to check that an expected error message is
produced.)

-- 
http://rrt.sc3d.org

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.