GNU bug report logs - #12620
Parallel tests vs fast tests (and beyond)

Previous Next

To reply to this bug, email your comments to 12620 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Reuben Thomas <rrt <at> sc3d.org>
To: bug-automake <bug-automake <at> gnu.org>
Subject: Parallel tests vs fast tests (and beyond)
Date: Wed, 10 Oct 2012 20:53:46 +0100

With the recent work on parallel tests in automake I thought it was
time to give them a spin, so I did, for the "zee" branch of GNU Zile.
This has about 100 tests, the total wall clock time being around 8s on
my 2.5GHz 4-core Sandy Bridge machine, with the following target:

 check-local: $(builddir)/src/zee
	echo $(TESTS) | $(LUA_ENV) $(LUA_TESTS_ENVIRONMENT) xargs $(RUNLUATESTS)

So, I rewrote it as:

LOG_COMPILER = $(LUA_ENV) $(LUA_TESTS_ENVIRONMENT) $(RUNLUATESTS)

and lo! it ran in 2/3 of the time. Not bad for a few minutes' work.
But then I looked closer, and noticed that although the wall-clock
time had gone down, the user+system time had almost doubled. Oops: I
was only winning because I had four cores. Developers using
single-core or dual-core machines might well lose out with parallel
tests.

The test harness is a Lua script, so the parallel tests were starting
an extra Lua interpreter per test, so  this is not a huge surprise,
but it is a pity.

In the end, I found I got a much bigger speed up (down to under 3s), with:

check-local: $(builddir)/src/zee
	NPROC=`nproc`; \
	echo $(LUA_TESTS) | $(LUA_ENV) $(LUA_TESTS_ENVIRONMENT) xargs
--max-procs=$$NPROC --max-args=$$(( `echo $(LUA_TESTS) | wc -w` /
$$NPROC + 1 )) $(RUNLUATESTS) > /dev/null

That would be relatively straightforward to make portable (or rather,
bail out if non-GNU xargs is used, but it's still much clumsier than
parallel-tests, and of course uses the old serial-tests. (I'm working
with automake 1.11.6, in case it matters.) The implication of the 1.12
release notes is that serial-tests will be dropped at some point
post-1.13.

This whole problem is an instance of a more general problem, of which
another example is parallel make: for best performance, it should
probably batch up calls to gcc, for example, so that multiple source
files are compiled by each invocation.

I can imagine a view that we're rapidly increasing the number of
cores, so in fact there's little point working on these intermediate
solutions, but it seems to me that it's not just less-well-off
developers who could benefit, but also compile farms.

Comments? Solutions I've missed?

-- 
http://rrt.sc3d.org

Message #8 received at 12620 <at> debbugs.gnu.org (full text, mbox):

From: Jack Kelly <jack <at> jackkelly.name>
To: Reuben Thomas <rrt <at> sc3d.org>
Cc: 12620 <at> debbugs.gnu.org
Subject: Re: bug#12620: Parallel tests vs fast tests (and beyond)
Date: Fri, 12 Oct 2012 08:12:59 +1100

On Thu, Oct 11, 2012 at 6:53 AM, Reuben Thomas <rrt <at> sc3d.org> wrote:
> This whole problem is an instance of a more general problem, of which
> another example is parallel make: for best performance, it should
> probably batch up calls to gcc, for example, so that multiple source
> files are compiled by each invocation.

We had a discussion along these lines when refactoring elisp
compilation: in the past it was all done in one big batch. Now it's
done with an emacs invocation per .el file. The result of that
discussion was that while you slow down single core machines, the new
way of doing things fits make's model much better: instead of fiddling
around with stamp files, the Makefile just has a suffix rule.

IMHO, your point is valid, but to do it safely would require a tool
smarter than make.

-- Jack

Message #11 received at 12620 <at> debbugs.gnu.org (full text, mbox):

From: Reuben Thomas <rrt <at> sc3d.org>
To: Jack Kelly <jack <at> jackkelly.name>
Cc: 12620 <at> debbugs.gnu.org
Subject: Re: bug#12620: Parallel tests vs fast tests (and beyond)
Date: Thu, 11 Oct 2012 22:40:16 +0100

On 11 October 2012 22:12, Jack Kelly <jack <at> jackkelly.name> wrote:
>
> We had a discussion along these lines when refactoring elisp
> compilation: in the past it was all done in one big batch. Now it's
> done with an emacs invocation per .el file. The result of that
> discussion was that while you slow down single core machines, the new
> way of doing things fits make's model much better: instead of fiddling
> around with stamp files, the Makefile just has a suffix rule.

That's very interesting, of course that's the way make works naturally
for C programs, and I never thought of changing it (although I read
about how much faster for compiling several files it is, but it only
ever occurred to me that it would be nice to batch up compilations for
building from source, not for development where typically you may only
need to recompile one file).

-- 
http://rrt.sc3d.org

Message #14 received at 12620 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: 12620 <at> debbugs.gnu.org
Cc: Reuben Thomas <rrt <at> sc3d.org>
Subject: Re: Parallel tests vs fast tests (and beyond)
Date: Thu, 26 Dec 2013 17:46:48 +0100

severity 12620 wishlist
stop

Reference:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12620>

Marking this as a "wishlist" rather than a bug, since the current
implementation of parallel tests is working correctly and is mostly
"good enough", and your proposal is thus geared towards improving an
already decent situation rather than fixing errors or misbehaviors
(this doesn't mean I'm denying or belittling the merits of your
proposal, of course).

Just one nit:

Reuben Thomas wrote:
>
> The implication of the 1.12 release notes is that serial-tests
> will be dropped at some point post-1.13.
>
I've been convinced that any idea I had in such a direction was
misguided; the old serial harness is now marked merely as
"discouraged", but there no plans to deprecate pr remove it in
the foreseeable future.

Thanks,
  Stefano

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.