GNU bug report logs - #36630
[PATCH] guix: parallelize building the manual-database

Previous Next

Full log

View this message in rfc822 format

From: Ludovic Courtès <ludo <at> gnu.org>
To: Arne Babenhauserheide <arne_bab <at> web.de>
Cc: 36630 <at> debbugs.gnu.org
Subject: [bug#36630] [PATCH] guix: parallelize building the manual-database
Date: Mon, 15 Jul 2019 18:12:27 +0200

Hi Arne,

Arne Babenhauserheide <arne_bab <at> web.de> skribis:

> * guix/profiles.scm (manual-database): par-map over the entries.  This
> distributes the load roughly equally over all cores and avoids blocking on
> I/O.  The order of the entries stays the same since write-mandb-database sorts
> them.

I would think the whole process is largely I/O-bound.  Did you try
measuring differences?

I picked the manual-database derivation returned for:

  guix environment --ad-hoc jupyter python-ipython python-ipykernel -n

(It has 3,046 entries.)

On a SSD and with a hot cache, on my 4-core laptop, I get 74s with
‘master’, and 53s with this patch.

However, it will definitely not scale linearly, so we should probably
cap at 2 or 4 threads.  WDYT?

Another issue with the patch is that the [n/total] counter does not grow
monotically now: it might temporally go backwards.  Consequently, at
-v1, users will see a progress bar that hesitates and occasionally goes
backward, which isn’t great.

This would need to fix it with a mutex-protected global counter.

All in all, I’m not sure this is worth the complexity.

WDYT?

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.