GNU bug report logs - #65456
[PATCH 0/2] Split guix build into more steps for 32bit hosts.

Previous Next

Full log

Message #41 received at 65456 <at> debbugs.gnu.org (full text, mbox):

From: Janneke Nieuwenhuizen <janneke <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Josselin Poiret <dev <at> jpoiret.xyz>, Tobias Geerinckx-Rice <me <at> tobias.gr>,
 Simon Tournier <zimon.toutoune <at> gmail.com>, Mathieu Othacehe <othacehe <at> gnu.org>,
 65456 <at> debbugs.gnu.org, Ricardo Wurmus <rekado <at> elephly.net>,
 Christopher Baines <guix <at> cbaines.net>
Subject: Re: bug#65456: [PATCH v4] self: Build directories in chunks of max
 25 files at a time.
Date: Wed, 23 Aug 2023 11:41:31 +0200

[Message part 1 (text/plain, inline)]

Janneke Nieuwenhuizen writes:

Hi!

> Ludovic Courtès writes:
>> Janneke Nieuwenhuizen <janneke <at> gnu.org> skribis:
>>
[..]

> it still has the "gnu/packages" directory name hardcoded here.  (I even
> believe this was kind-of intentional, meaning not to split-up builds of
> other directories...).  The result is, of course, that partition always
> produces an empty result for any '("guix/foo.scm") files!
>
> It is fixed by doing
>
>              (cute string-append "^" directory "/" <>)

Well, almost...In my test script it worked, but directory is "." in the
actual self.scm.

So with v3 no .go files get built at all...and then a naive `guix build
hello' seems to work!; `guix shell' for example fails.  Comparing the
list of files from `find /gnu/store/...-guix-20230823.08 -follow' is
also helpful.

I realised that there's no need to use the name-based splitting hack
that Makefile.am uses and changed the creation of chunks to simply use a
maximum chunk length of 25 files.

New version attached.

Greetings,
Janneke

[v4-0001-self-Build-directories-in-chunks-of-max-25-files-.patch (text/x-patch, inline)]

From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke <at> gnu.org>
From: Janneke Nieuwenhuizen <janneke <at> gnu.org>
Date: Thu, 22 Jun 2023 08:30:25 +0200
Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
 time.

Similar to split build of make-go in Makefile.am, this breaks-up building
directories into chunks of max 25 files.  Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
directories into chunks of max 25 files.
---
 guix/self.scm | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..fc2dfd8131 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017-2023 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2020 Martin Becze <mjbecze <at> riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1210,7 +1211,8 @@ (define* (compiled-modules name module-tree module-files
                             '((guix build compile)
                               (guix build utils)))
       #~(begin
-          (use-modules (srfi srfi-26)
+          (use-modules (srfi srfi-1)
+                       (srfi srfi-26)
                        (ice-9 match)
                        (ice-9 format)
                        (ice-9 threads)
@@ -1244,12 +1246,23 @@ (define* (compiled-modules name module-tree module-files
             (force-output))
 
           (define (process-directory directory files output)
-            ;; Hide compilation warnings.
-            (parameterize ((current-warning-port (%make-void-port "w")))
-              (compile-files directory #$output files
-                             #:workers (parallel-job-count)
-                             #:report-load report-load
-                             #:report-compilation report-compilation)))
+            (let* ((size 25)                      ;compile max 25 files a time
+                   (chunks (unfold
+                            (lambda (seed) (< (length seed) size)) ;p
+                            (cute take <> size)                    ;f
+                            (cute drop <> size)                    ;g
+                            files                                  ;seed
+                            list)))                                ;tail
+              (for-each
+               (lambda (chunck)
+                 ;; Hide compilation warnings.
+                 (parameterize ((current-warning-port (%make-void-port "w")))
+                   (compile-files directory output chunck
+                                  #:workers (parallel-job-count)
+                                  #:report-load report-load
+                                  #:report-compilation report-compilation))
+                 (gc))
+               chunks)))
 
           (setvbuf (current-output-port) 'line)
           (setvbuf (current-error-port) 'line)

base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea
-- 
2.41.0

[Message part 3 (text/plain, inline)]

-- 
Janneke Nieuwenhuizen <janneke <at> gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.