GNU bug report logs - #65456
[PATCH 0/2] Split guix build into more steps for 32bit hosts.

Previous Next

Full log

View this message in rfc822 format

From: Janneke Nieuwenhuizen <janneke <at> gnu.org>
To: 65456 <at> debbugs.gnu.org
Cc: Christopher Baines <guix <at> cbaines.net>, Josselin Poiret <dev <at> jpoiret.xyz>, Ludovic Courtès <ludo <at> gnu.org>, Mathieu Othacehe <othacehe <at> gnu.org>, Ricardo Wurmus <rekado <at> elephly.net>, Simon Tournier <zimon.toutoune <at> gmail.com>, Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps.
Date: Tue, 22 Aug 2023 19:49:01 +0200

Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks
when building on 32bit.  Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
"gnu/packages" into 26 chunks.
---
 guix/self.scm | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..151fa9f452 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017-2023 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2020 Martin Becze <mjbecze <at> riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files
                             '((guix build compile)
                               (guix build utils)))
       #~(begin
-          (use-modules (srfi srfi-26)
+          (use-modules (srfi srfi-1)
+                       (srfi srfi-26)
+                       (srfi srfi-71)
                        (ice-9 match)
                        (ice-9 format)
+                       (ice-9 regex)
                        (ice-9 threads)
                        (guix build compile)
                        (guix build utils))
@@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files
             (force-output))
 
           (define (process-directory directory files output)
-            ;; Hide compilation warnings.
-            (parameterize ((current-warning-port (%make-void-port "w")))
-              (compile-files directory #$output files
-                             #:workers (parallel-job-count)
-                             #:report-load report-load
-                             #:report-compilation report-compilation)))
+            ;; Split gnu/packages in 26 chunks to avoid OOM errors
+            (let* ((chunks (map (compose
+                                 (cute partition <> files)
+                                 (lambda (regex)
+                                   (cute string-match regex <>))
+                                 (cute string-append "^gnu/packages/" <>)
+                                 (cute make-string 1 <>)
+                                 integer->char
+                                 (cute + (char->integer #\a) <>))
+                                (iota 26)))
+                   (chunks (filter pair? chunks)))
+              (for-each
+               (lambda (chunck)
+                 (parameterize ((current-warning-port (%make-void-port "w")))
+                   ;; Hide compilation warnings.
+                   (compile-files directory #$output chunck
+                                  #:workers (parallel-job-count)
+                                  #:report-load report-load
+                                  #:report-compilation report-compilation))
+                 (gc))
+               chunks)))
 
           (setvbuf (current-output-port) 'line)
           (setvbuf (current-error-port) 'line)
-- 
2.41.0

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.