Subject: bug#36630: closed (Re: [bug#36630] [PATCH] guix: parallelize building the manual-database)

From: [email-masked] (GNU bug Tracking System)
Date: Tue, 31 Mar 2020 13:03:02 +0000

Your bug report

#36630: [PATCH] guix: parallelize building the manual-database

which was filed against the guix-patches package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
36630: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=36630
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
From: Ludovic Courtès <[email-masked]>
To: Arne Babenhauserheide <[email-masked]>
Cc: [email-masked]
Date: Tue, 31 Mar 2020 15:02:47 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi Arne,

Arne Babenhauserheide <[email-masked]> skribis:

> I now reduced the thread count to exactly 2 (to avoid running into
> resource troubles; I hope that two should be safe) and added a mutex for
> status messages to ensure that writes don’t overlap.

It’s been 9 months but I finally committed a slightly modified variant
as ef4b5f2fed3ca13a0e15a821ba7e561cd4395aa6.  It turns out that the
mutex was unnecessary as ports are thread-safe.

As noted in the log, I see a 36% speedup on my SSD laptop with 4 cores
(slightly less with 2 cores).  It’s not great, but still an improvement!

Thanks,
Ludo’.

Subject: [PATCH] guix: parallelize building the manual-database
From: Arne Babenhauserheide <[email-masked]>
To: [email-masked]
Cc: Arne Babenhauserheide <[email-masked]>
Date: Fri, 12 Jul 2019 23:42:45 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

* guix/profiles.scm (manual-database): par-map over the entries.  This
distributes the load roughly equally over all cores and avoids blocking on
I/O.  The order of the entries stays the same since write-mandb-database sorts
them.
---
 guix/profiles.scm | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index f5c863945c..374f0f8a90 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -1312,15 +1312,11 @@ the entries in MANIFEST."
         #~(begin
             (use-modules (guix man-db)
                          (guix build utils)
+                         (ice-9 threads)
                          (srfi srfi-1)
                          (srfi srfi-19))

-            (define (compute-entries)
-              ;; This is the most expensive part (I/O and CPU, due to
-              ;; decompression), so report progress as we traverse INPUTS.
-              (let* ((inputs '#$(manifest-inputs manifest))
-                     (total  (length inputs)))
-                (append-map (lambda (directory count)
+            (define (compute-entry directory count total)
                               (format #t "\r[~3d/~3d] building list of \
 man-db entries..."
                                       count total)
@@ -1330,8 +1326,16 @@ man-db entries..."
                                 (if (directory-exists? man)
                                     (mandb-entries man)
                                     '())))
-                            inputs
-                            (iota total 1))))
+
+            (define (compute-entries)
+              ;; This is the most expensive part (I/O and CPU, due to
+              ;; decompression), so report progress as we traverse INPUTS.
+              (let* ((inputs '#$(manifest-inputs manifest))
+                     (total  (length inputs)))
+                (apply append (par-map compute-entry
+                                       inputs
+                                       (iota total 1)
+                                       (make-list total total)))))

             (define man-directory
               (string-append #$output "/share/man"))
--
2.22.0