GNU bug report logs - #42883
Selecting a subset of tests in recursive make causes unnecessary failures

Previous Next

Full log

Message #11 received at 42883 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Washburn <development <at> efficientek.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: development <at> efficientek.com, 42883 <at> debbugs.gnu.org
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
Date: Fri, 21 Aug 2020 03:07:21 -0500

Hi Karl,

On Wed, 19 Aug 2020 19:17:13 -0600
Karl Berry <karl <at> freefriends.org> wrote:

> Hi Glenn,
> 
>     I noticed that when specifying a subset of tests to run via the
> TESTS environment variable, I could have all tests complete
> successfully
> 
> Thanks for the report. That does seem suboptimal, but on the other
> hand, subsets of tests are a complication. Does it work to override
> SUBDIRS along with TESTS? As in (thanks to Jim):
>   env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=

Yes this does work in the special case of grub, thanks for the
suggestion. I think at a minimum this behavior should be documented
with the test subsets documentation and this work around mentioned.
Ideally, the subset tests feature design should be updated to play nice
with makefile recursion.  Setting SUBDIRS= is just disabling the
makefile recusion.

If I had a test in grub-core/unittest1 and I wanted to run just the
test subset of unittest1 and grub_cmd_echo, how would I do that such
that make returns success?  If I run `TESTS="unittest1 grub_cmd_echo"
make -e check', make will fail even if both test individually succeed
because the recursive make check in grub-core will try to run
grub_cmd_echo (in addition to unittest1) and fail.

Another issue is that if there is a test X in the root makefile and a
test X in SUBDIR, how does one choose to run the X in SUBDIR and not the
X in root?  Right now TESTS=X will run both tests.  Sure one could do
`TESTS=X make -e -C SUBDIR check', but then again you're just bypassing
the makefile recursion.

And thanks Karl for helping to maintain this venerable and essential,
yet not super glamorous project.

Glenn

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.