GNU bug report logs - #36630
[PATCH] guix: parallelize building the manual-database

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: arne_bab <at> web.de
Subject: bug#36630: closed (Re: [bug#36630] [PATCH] guix: parallelize
 building the manual-database)
Date: Tue, 31 Mar 2020 13:03:02 +0000

[Message part 1 (text/plain, inline)]

Your bug report

#36630: [PATCH] guix: parallelize building the manual-database

which was filed against the guix-patches package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to 36630 <at> debbugs.gnu.org.

-- 
36630: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=36630
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Ludovic Courtès <ludo <at> gnu.org>
To: Arne Babenhauserheide <arne_bab <at> web.de>
Cc: 36630-done <at> debbugs.gnu.org
Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
Date: Tue, 31 Mar 2020 15:02:47 +0200

Hi Arne,

Arne Babenhauserheide <arne_bab <at> web.de> skribis:

> I now reduced the thread count to exactly 2 (to avoid running into
> resource troubles; I hope that two should be safe) and added a mutex for
> status messages to ensure that writes don’t overlap.

It’s been 9 months but I finally committed a slightly modified variant
as ef4b5f2fed3ca13a0e15a821ba7e561cd4395aa6.  It turns out that the
mutex was unnecessary as ports are thread-safe.

As noted in the log, I see a 36% speedup on my SSD laptop with 4 cores
(slightly less with 2 cores).  It’s not great, but still an improvement!

Thanks,
Ludo’.

[Message part 3 (message/rfc822, inline)]

From: Arne Babenhauserheide <arne_bab <at> web.de>
To: guix-patches <at> gnu.org
Cc: Arne Babenhauserheide <arne_bab <at> web.de>
Subject: [PATCH] guix: parallelize building the manual-database
Date: Fri, 12 Jul 2019 23:42:45 +0200

* guix/profiles.scm (manual-database): par-map over the entries.  This
distributes the load roughly equally over all cores and avoids blocking on
I/O.  The order of the entries stays the same since write-mandb-database sorts
them.
---
 guix/profiles.scm | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index f5c863945c..374f0f8a90 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -1312,15 +1312,11 @@ the entries in MANIFEST."
         #~(begin
             (use-modules (guix man-db)
                          (guix build utils)
+                         (ice-9 threads)
                          (srfi srfi-1)
                          (srfi srfi-19))

-            (define (compute-entries)
-              ;; This is the most expensive part (I/O and CPU, due to
-              ;; decompression), so report progress as we traverse INPUTS.
-              (let* ((inputs '#$(manifest-inputs manifest))
-                     (total  (length inputs)))
-                (append-map (lambda (directory count)
+            (define (compute-entry directory count total)
                               (format #t "\r[~3d/~3d] building list of \
 man-db entries..."
                                       count total)
@@ -1330,8 +1326,16 @@ man-db entries..."
                                 (if (directory-exists? man)
                                     (mandb-entries man)
                                     '())))
-                            inputs
-                            (iota total 1))))
+
+            (define (compute-entries)
+              ;; This is the most expensive part (I/O and CPU, due to
+              ;; decompression), so report progress as we traverse INPUTS.
+              (let* ((inputs '#$(manifest-inputs manifest))
+                     (total  (length inputs)))
+                (apply append (par-map compute-entry
+                                       inputs
+                                       (iota total 1)
+                                       (make-list total total)))))

             (define man-directory
               (string-append #$output "/share/man"))
--
2.22.0

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.