GNU bug report logs - #9803
Add ERT option to skip test

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 9803 in the body.
You can then email your comments to 9803 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #3 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: submit <at> debbugs.gnu.org
Subject: Add ERT option to skip test
Date: Wed, 19 Oct 2011 23:42:45 -0400

Package: emacs
Version: 24.0.90
Severity: wishlist

I think it would be nice if ert had the ability to skip tests.
Eg, a :skip argument that works the same way as :expected-result.
This would be useful eg when a test relies on external executable that
might not be installed on the system running the tests. You can get the
same result by using :expected-result, but :skip might be nicer in such
cases.

Message #6 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: Add ERT option to skip test
Date: Thu, 04 Jul 2013 20:40:10 +0200

Glenn Morris <rgm <at> gnu.org> writes:

> I think it would be nice if ert had the ability to skip tests.
> Eg, a :skip argument that works the same way as :expected-result.
> This would be useful eg when a test relies on external executable that
> might not be installed on the system running the tests. You can get the
> same result by using :expected-result, but :skip might be nicer in such
> cases.

Skipping tests is already possible. Instead of writing

(ert-deftest vc-bzr-test-bug9726 ()
  "Test for http://debbugs.gnu.org/9726 ."
  :expected-result (if (executable-find vc-bzr-program) :passed :failed)
  (should (executable-find vc-bzr-program))
  ...

you could do

(when (executable-find vc-bzr-program)
  (ert-deftest vc-bzr-test-bug9726 ()
    "Test for http://debbugs.gnu.org/9726 ."
    ...

The only drawback I see is that such skipped tests are not listed in the
output of an ert run.

Best regards, Michael.

Message #9 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Fri, 18 Oct 2013 15:37:52 +0200

[Message part 1 (text/plain, inline)]

Glenn Morris <rgm <at> gnu.org> writes:

Hi Glenn,

> I think it would be nice if ert had the ability to skip tests.
> Eg, a :skip argument that works the same way as :expected-result.
> This would be useful eg when a test relies on external executable that
> might not be installed on the system running the tests. You can get the
> same result by using :expected-result, but :skip might be nicer in such
> cases.

I have written a new macro, which should do the job. It is called
`skip-if' and works like `should' and companions. You pass a form as
argument, and when it returns non-nil the test is skipped.

Test summary is showing skipped tests.

Could you, please, check the appended patch, whether it fits your needs?

Best regards, Michael.

[ert.el.patch (text/x-diff, attachment)]

Message #12 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Fri, 18 Oct 2013 21:02:45 -0400

Michael Albinus wrote:

>> I think it would be nice if ert had the ability to skip tests.
>> Eg, a :skip argument that works the same way as :expected-result.
[...]
> I have written a new macro, which should do the job. It is called
> `skip-if' and works like `should' and companions. You pass a form as
> argument, and when it returns non-nil the test is skipped.

Thank you, looks nice.
Your method is not quite how I imagined it working, but maybe your way is
better, I haven't thought about it much... With your approach, it seems
like I have to specify the skip condition twice? Eg I have to write:

(ert-deftest foo-test ()
  "Test for foo."
  :expected-result (if (executable-find "foo") :passed :skipped)
  (skip-if (not (executable-find "foo")))
  t ; in a real use case, some test using "foo" here
)

rather than:

(ert-deftest foo-test ()
  "Test for foo."
  :skip (not (executable-find "foo"))
  t)


I think `skip-if' should have an ert- prefix. (I know `should' doesn't,
but I think it, err, should as well. But too late for that one now.)


Also, ert-run-tests-batch-and-exit seems to need updating:

  Running 1 tests (2013-10-18 17:49:11-0700)
    skipped  1/1  foo-test

  Ran 1 tests, 0 results as expected (2013-10-18 17:49:11-0700)

I don't think "0 results as expected" is appropriate.

Eg automake uses a summary like this:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11745

Message #15 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Glenn Morris <rgm <at> gnu.org>
Cc: Michael Albinus <michael.albinus <at> gmx.de>, 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Fri, 18 Oct 2013 22:12:24 -0400

> I think `skip-if' should have an ert- prefix. (I know `should' doesn't,
> but I think it, err, should as well. But too late for that one now.)

Rather than add a prefix to `should' we could/should make it so it's
only defined within an ert-deftest (like the `go' macro in `cl-tagbody').


        Stefan

Message #18 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Sat, 19 Oct 2013 08:44:14 +0200

Glenn Morris <rgm <at> gnu.org> writes:

> Thank you, looks nice.
> Your method is not quite how I imagined it working, but maybe your way is
> better, I haven't thought about it much... With your approach, it seems
> like I have to specify the skip condition twice? Eg I have to write:
>
> (ert-deftest foo-test ()
>   "Test for foo."
>   :expected-result (if (executable-find "foo") :passed :skipped)
>   (skip-if (not (executable-find "foo")))
>   t ; in a real use case, some test using "foo" here
> )
>
> rather than:
>
> (ert-deftest foo-test ()
>   "Test for foo."
>   :skip (not (executable-find "foo"))
>   t)

No. The idea is that skip-if works whatever you have defined in
:expected-result. It simply ignores :expected-result, when it finds a
non-nil form.

> Also, ert-run-tests-batch-and-exit seems to need updating:
>
>   Running 1 tests (2013-10-18 17:49:11-0700)
>     skipped  1/1  foo-test
>
>   Ran 1 tests, 0 results as expected (2013-10-18 17:49:11-0700)
>
> I don't think "0 results as expected" is appropriate.

Will check. I've tested the interactive call so far.

Best regards, Michael.

Message #21 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Glenn Morris <rgm <at> gnu.org>, 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Sat, 19 Oct 2013 08:44:56 +0200

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>> I think `skip-if' should have an ert- prefix. (I know `should' doesn't,
>> but I think it, err, should as well. But too late for that one now.)
>
> Rather than add a prefix to `should' we could/should make it so it's
> only defined within an ert-deftest (like the `go' macro in `cl-tagbody').

Yep. Will do.

>         Stefan

Best regards, Michael.

Message #24 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Sat, 19 Oct 2013 22:02:30 -0400

Michael Albinus wrote:

> No. The idea is that skip-if works whatever you have defined in
> :expected-result. It simply ignores :expected-result, when it finds a
> non-nil form.

For some reason I thought I had to specify :expected-result :skipped as
well, but I can no longer reproduce whatever led me to think that.

Message #27 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Sun, 20 Oct 2013 16:09:07 +0200

[Message part 1 (text/plain, inline)]

Glenn Morris <rgm <at> gnu.org> writes:

Hi Glenn,

> I think `skip-if' should have an ert- prefix. (I know `should' doesn't,
> but I think it, err, should as well. But too late for that one now.)

I've tried to change it as proposed by Stefan, but I'm too stupid to
manage all this sophisticated cl-* stuff :-(

> Also, ert-run-tests-batch-and-exit seems to need updating:
>
>   Running 1 tests (2013-10-18 17:49:11-0700)
>     skipped  1/1  foo-test
>
>   Ran 1 tests, 0 results as expected (2013-10-18 17:49:11-0700)
>
> I don't think "0 results as expected" is appropriate.
>
> Eg automake uses a summary like this:
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=11745

Such a verbose summary is returned when you run the tests
interactively. I've reworked the patch in order to give also valid
statistics in batch mode.

Best regards, Michael.

[123.patch (text/x-patch, attachment)]

Message #30 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Mon, 21 Oct 2013 17:08:31 +0200

[Message part 1 (text/plain, inline)]

Michael Albinus <michael.albinus <at> gmx.de> writes:

Hi Glenn,

>> I think `skip-if' should have an ert- prefix. (I know `should' doesn't,
>> but I think it, err, should as well. But too late for that one now.)
>
> I've tried to change it as proposed by Stefan, but I'm too stupid to
> manage all this sophisticated cl-* stuff :-(

Well, I've made a change to call (fset 'skip-if 'ert--skip-if) in
ert--run-test-internal. After running the test, it is reverted by
(unintern 'skip-if nil).

Therefore, `skip-if' is visible only inside tests defined with
`ert-deftest'. If this is acceptable, I could apply this change also for
`should', `should-not' and `should-error'.

Do you (and Stefan) agree?

Best regards, Michael.

[123.patch (text/x-diff, attachment)]

Message #33 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: Glenn Morris <rgm <at> gnu.org>, 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Mon, 21 Oct 2013 12:53:14 -0400

> Well, I've made a change to call (fset 'skip-if 'ert--skip-if) in
> ert--run-test-internal. After running the test, it is reverted by
> (unintern 'skip-if nil).

Yuck!

> Therefore, `skip-if' is visible only inside tests defined with
> `ert-deftest'.

Not "inside" but "during".  I.e. it can still affect any code run during
the tests.

> If this is acceptable, I could apply this change also for
> `should', `should-not' and `should-error'.

No, better leave them as always-global.

> I've tried to change it as proposed by Stefan, but I'm too stupid to
> manage all this sophisticated cl-* stuff :-(

What have you tried and how did it fail?


        Stefan

Message #36 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Glenn Morris <rgm <at> gnu.org>, 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Mon, 21 Oct 2013 19:02:00 +0200

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>> I've tried to change it as proposed by Stefan, but I'm too stupid to
>> manage all this sophisticated cl-* stuff :-(
>
> What have you tried and how did it fail?

I've tried a kind of `letf'; this failed because it replaces only
function calls of functions which exist already. Does not work, because
`skip-if' shall not exist.

Then I did traverse `remaining' in `ert--parse-keys-and-body', replacing all
`skip-if' occurences in the car's by `ert--skip-if'. This did work
somehow, but only if `skip-if' is called on the top-level of the body of
an ert test. Parsing the whole structure might be possible, but I don't
know a respective function/macro, and writing an own recursive parsing
seems to expensive to me.

>         Stefan

Best regards, Michael.

Message #39 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: Glenn Morris <rgm <at> gnu.org>, 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Mon, 21 Oct 2013 15:23:22 -0400

> I've tried a kind of `letf'; this failed because it replaces only
> function calls of functions which exist already. Does not work, because
> `skip-if' shall not exist.

You want to use either `cl-flet' or `cl-macrolet', depending on whether
you want to define skip-if as a local function or a local macro.


        Stefan

Message #42 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Glenn Morris <rgm <at> gnu.org>, 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Tue, 22 Oct 2013 10:23:50 +0200

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

> You want to use either `cl-flet' or `cl-macrolet', depending on whether
> you want to define skip-if as a local function or a local macro.

`cl-macrolet' does it, thanks! I will also handle `should', `should-not'
and `should-error' this way.

Meanwhile I'm thinking, we should introduce `skip-unless' instead of
`skip-if'. This would allow to catch errors, like this:

  (skip-unless (ignore-errors (a-test-which-might-raise-a-signal)))

We could even move `ignore-errors' into `skip-unless'.

Opinions?

>         Stefan

Best regards, Michael.

Message #45 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>,  9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Wed, 23 Oct 2013 14:21:59 +0200

Michael Albinus <michael.albinus <at> gmx.de> writes:

> Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
>
>> You want to use either `cl-flet' or `cl-macrolet', depending on whether
>> you want to define skip-if as a local function or a local macro.
>
> `cl-macrolet' does it, thanks! I will also handle `should', `should-not'
> and `should-error' this way.

Well, it works only for tests which use `should', `should-not' and
`should-error' in the body of `ert-deftest'. Unfortunately, there are
several tests which use helper functions, calling `should' and
friends. For those tests it doesn't work this way.

Since this is not related to the `skip-unless' check, I will commit just
a patch introducing this macro. Changing the scope of `should',
`should-not' and `should-error' could be fixed later.

>>         Stefan

Best regards, Michael.

Message #48 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Thu, 24 Oct 2013 09:46:56 +0200

Hi,

I've committed the final patch to the trunk.

Best regards, Michael.

Message #51 received at 9803 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 9803 <at> debbugs.gnu.org
Subject: Re: bug#9803: [PATCH] Add ERT option to skip test
Date: Thu, 24 Oct 2013 04:00:22 -0400

Michael Albinus wrote:

> I've committed the final patch to the trunk.

Fantastic, thank you.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.