GNU bug report logs - #34982
[PATCH] guile-build-system: Support building in parallel.

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Christopher Baines <mail <at> cbaines.net>
Cc: tracker <at> debbugs.gnu.org
Subject: bug#34982: closed ([PATCH] guile-build-system: Support building
 in parallel.)
Date: Fri, 19 Apr 2019 10:24:03 +0000

[Message part 1 (text/plain, inline)]

Your message dated Fri, 19 Apr 2019 08:43:39 +0100
with message-id <87y346tp78.fsf <at> cbaines.net>
and subject line Re: [bug#34982] [PATCH] guile-build-system: Support building in parallel.
has caused the debbugs.gnu.org bug report #34982,
regarding [PATCH] guile-build-system: Support building in parallel.
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
34982: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=34982
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Christopher Baines <mail <at> cbaines.net>
To: guix-patches <at> gnu.org
Subject: [PATCH] guile-build-system: Support building in parallel.
Date: Sun, 24 Mar 2019 21:23:45 +0000

* guix/build/guile-build-system.scm (build): Use n-par-for-each, instead of
for-each, to use multiple cores if available.
---
 guix/build/guile-build-system.scm | 43 +++++++++++++++++++------------
 1 file changed, 26 insertions(+), 17 deletions(-)

diff --git a/guix/build/guile-build-system.scm b/guix/build/guile-build-system.scm
index 0bed049436..a5741081bf 100644
--- a/guix/build/guile-build-system.scm
+++ b/guix/build/guile-build-system.scm
@@ -23,6 +23,7 @@
   #:use-module (ice-9 match)
   #:use-module (ice-9 popen)
   #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 threads)
   #:use-module (guix build utils)
   #:export (target-guile-effective-version
             %standard-phases
@@ -101,24 +102,32 @@ Return #false if it cannot be determined."
                            (match (getenv "GUILE_LOAD_COMPILED_PATH")
                              (#f "")
                              (path (string-append ":" path)))))
-    (for-each (lambda (file)
-                (let* ((go (string-append go-dir
-                                          (file-sans-extension file)
-                                          ".go")))
-                  ;; Install source module.
-                  (install-file (string-append source-directory "/" file)
-                                (string-append module-dir
-                                               "/" (dirname file)))
+    (n-par-for-each
+     (parallel-job-count)
+     (lambda (file)
+       (catch #t
+         (lambda ()
+           (let* ((go (string-append go-dir
+                                     (file-sans-extension file)
+                                     ".go")))
+             ;; Install source module.
+             (install-file (string-append source-directory "/" file)
+                           (string-append module-dir
+                                          "/" (dirname file)))
 
-                  ;; Install and compile module.
-                  (apply invoke guild "compile" "-L" source-directory
-                         "-o" go
-                         (string-append source-directory "/" file)
-                         flags)))
-
-              ;; Arrange to strip SOURCE-DIRECTORY from file names.
-              (with-directory-excursion source-directory
-                (find-files "." scheme-file-regexp)))
+             ;; Install and compile module.
+             (apply invoke guild "compile" "-L" source-directory
+                    "-o" go
+                    (string-append source-directory "/" file)
+                    flags)))
+         (lambda (key . args)
+           ;; Since ports are not thread-safe as of Guile 2.0, reopen stderr.
+           (let ((port (fdopen 2 "w0")))
+             (print-exception port #f key args)
+             (primitive-exit 1)))))
+     ;; Arrange to strip SOURCE-DIRECTORY from file names.
+     (with-directory-excursion source-directory
+       (find-files "." scheme-file-regexp)))
     #t))
 
 (define* (install-documentation #:key outputs
-- 
2.20.1

[Message part 3 (message/rfc822, inline)]

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 34982-done <at> debbugs.gnu.org
Subject: Re: [bug#34982] [PATCH] guile-build-system: Support building in
 parallel.
Date: Fri, 19 Apr 2019 08:43:39 +0100

[Message part 4 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Christopher Baines <mail <at> cbaines.net> skribis:
>
>> * guix/build/guile-build-system.scm (build): Use invoke-each, instead of
>> for-each, to use multiple cores if available.
>> (invoke-each, report-build-process): New procedures.
>
> [...]
>
>> +  (define (fork-and-run-command command)
>> +    (match (primitive-fork)
>> +      (0
>> +       (apply execlp command))
>> +      (pid
>> +       #t)))
>
> To be on the safe side, you should probably wrap the ‘execlp’ call like
> this:
>
>   (dynamic-wind
>     (const #t)
>     (lambda ()
>       (apply execlp command))
>     (lambda ()
>       (primitive-exit 127)))
>
> This ensures that the child process exits immediately if something goes
> wrong (e.g., ‘execlp’ raises an exception because the executable could
> not be found.)
>
> Otherwise LGTM, thank you!

Great, I've added in dynamic-wind, made some minor tweaks to the output,
and pushed this as 3fdb9a375f1cee7dd302349a9527437df20b3f61.

Thanks for taking a look :)

Chris

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.