From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 17:19:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: 65456@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-To: bug-guix@gnu.org X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by submit@debbugs.gnu.org id=B.169272469831808 (code B ref -1); Tue, 22 Aug 2023 17:19:02 +0000 Received: (at submit) by debbugs.gnu.org; 22 Aug 2023 17:18:18 +0000 Received: from localhost ([127.0.0.1]:60402 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV1F-0008Gx-S4 for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:18:18 -0400 Received: from lists.gnu.org ([2001:470:142::17]:37640) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV1C-0008Gj-70 for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:18:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYV13-000130-HS for bug-guix@gnu.org; Tue, 22 Aug 2023 13:18:05 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYV12-0005UE-RR; Tue, 22 Aug 2023 13:18:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=MXGP6JnLn55nxGu/RKVZvRSzD14tw//JK6nmQvWbg9I=; b=WjXuKUaiepc7MV tDUtENMXkMHbiRmQFNtusOAjJzKCVtYcMbAGjaqrRceB9nNQuEo9rJLAbgm+/dsgl3XuU5kq7byLM FxnE2gNSC7vmOBvn4+zsJ04utBdDENMrhoDzB8F7SjRwFHogYR/mhuToUbLFXUhbmtlQNtQMLP7IG gGJW9tJyZzg4qNaxQ40Wcmg8Fwk5T2QkbyfFjXAkRlipvXgpbgMUJHzHzHTez4O3OTzs1tz6N342B HIHQm8A5bxyt4lWl+ayf+tANwxVTpIDOUr/QjQG/8DMzc8rnF2Y7eawxiJYSeo5HIEnZdFI39CaoJ jvH27LwYBD2NjbaWL+7g==; From: Janneke Nieuwenhuizen Date: Tue, 22 Aug 2023 19:17:59 +0200 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi! Initially writter for the Hurd (see hurd-team , ) This afternoon on IRC, Maxim confirmed my suspicion that this could be 32bit issue, rather than a Hurd problem (, ). I didn't submit these for the Hurd yet, as I didn't really feel comfortable with the 26-way split. I tried a 5-way split and that still gave OOM errors on the Hurd and I didn't feel like hand-coding a 10-way split, so yeah. Also, the 5-way split in the Makefile.am will produce percentages over 100 which suggests a typo...but I couldn't spot it. Lastly, these patches feel like a workaround for Guile / libgc memory management? Otoh, being able to build Guix on 32bit hosts is kinda nice too... So yeah. Greetings, Janneke Janneke Nieuwenhuizen (2): build: Build gnu/packages/*.go in five steps. self: Build gnu/packages/*.go in 26 steps on 32bit. Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++------- guix/self.scm | 31 ++++++++++++++++++++++++-- 2 files changed, 83 insertions(+), 10 deletions(-) base-commit: c655231b72ac28b5a433069fcf86a835c9c83691 -- 2.41.0 From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 17:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: 65456@debbugs.gnu.org Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169272481232034 (code B ref 65456); Tue, 22 Aug 2023 17:21:02 +0000 Received: (at 65456) by debbugs.gnu.org; 22 Aug 2023 17:20:12 +0000 Received: from localhost ([127.0.0.1]:60415 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV35-0008Ka-Dd for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:20:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52470) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV33-0008KE-66 for 65456@debbugs.gnu.org; Tue, 22 Aug 2023 13:20:09 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYV2h-0005s8-31; Tue, 22 Aug 2023 13:19:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=Oe8hQC6MMmoGvrSOR461hETpHKGf413EvruExd8ICiM=; b=apcykOGWKk9UgY4AblEo HwvHx1k6K/Rdg20E45BeRUkcLsEbfBsZrXSE5sMyjBsBpqp0d2hRM6NgE8SBXmNn73EWd4MgJ/Wun TJzlPMdzPaB8wpBKyXrkQwxtdyW4ULGG+8U/yAkCYvbIN3vp7ZJ3oid0vdpVEriClyIBh5OaELtj/ x/gQ8jS1+r3SxXgoTi3v+GQGDujYI2Ms8EOxpoi/lWhL+sepFPLH400P7fWlegnyO68fjASqK0nwh UzyU2HoklzjYs4KKFTyfclJm1r3a4u38MmQG4pYRxjUMjQOrb3l05hNjmSwzkTDuc/Iu/JpFYeSFT BR2zqNFQzI+iNg==; From: Janneke Nieuwenhuizen Date: Tue, 22 Aug 2023 19:19:41 +0200 Message-ID: <606351ff2416c9086638a1c379b84ba811b384fa.1692723764.git.janneke@gnu.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) This breaks-up packages into five chunks of ~150,000 lines, allowing guix build --target=i586-pc-gnu from an x86 host. This is a followup to 1aa7ee52c6c520c2dbbdb06f1381466e9fd96294. * Makefile.am (first_half): Rename to... (first_quart): ...this, and also split into... (second_quart): ...this. (third_quart, MODULES_PACKAGES3, MODULES_PACKAGE4): New variables. (make-packages3-go, make-packages4-go): New targets. (make-packages-go): Add them. --- Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 54 insertions(+), 8 deletions(-) diff --git a/Makefile.am b/Makefile.am index 738532f839..7c8c215f86 100644 --- a/Makefile.am +++ b/Makefile.am @@ -776,19 +776,45 @@ endef # in . Each 'eval' call below creates a # 'make-*-go' phony target that builds the corresponding subset. -first_half := \ - gnu/packages/a% gnu/packages/b% gnu/packages/c% gnu/packages/d% \ - gnu/packages/e% gnu/packages/f% gnu/packages/g% gnu/packages/h% \ - gnu/packages/i% gnu/packages/j% gnu/packages/k% gnu/packages/l% +first_fifth := \ + gnu/packages/a% gnu/packages/b% gnu/packages/c% + +second_fifth := \ + gnu/packages/d% gnu/packages/e% gnu/packages/f% + +third_fifth := \ + gnu/packages/g% gnu/packages/h% gnu/packages/i% + +fourth_fifth := \ + gnu/packages/j% gnu/packages/k% gnu/packages/l% gnu/packages/m% \ + gnu/packages/n% gnu/packages/o% MODULES_CORE := guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULES))) -MODULES_PACKAGES1 := $(filter $(first_half),$(MODULES)) -MODULES_PACKAGES2 := $(filter-out $(first_half),$(filter gnu/packages/%,$(MODULES))) -MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) +MODULES_PACKAGES1 := $(filter $(first_fifth),$(MODULES)) +MODULES_PACKAGES2 := $(filter $(second_fifth),$(MODULES)) +MODULES_PACKAGES3 := $(filter $(third_fifth),$(MODULES)) +MODULES_PACKAGES4 := $(filter $(fourth_fifth),$(MODULES)) +MODULES_PACKAGES5 := $(filter-out $(first_fifth) $(second_fifth) \ + $(third_fifth) $(fourth_fifth), \ + $(filter gnu/packages/%,$(MODULES))) +MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3) $(MODULES_PACKAGES4) \ + $(MODULES_PACKAGES5) MODULES_SYSTEM := gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES))) MODULES_CLI := $(filter guix/scripts/%,$(MODULES)) MODULES_PO := guix/build/po.scm +print-p1: + @echo $(MODULES_PACKAGES1) +print-p2: + @echo $(MODULES_PACKAGES2) +print-p3: + @echo $(MODULES_PACKAGES3) +print-p4: + @echo $(MODULES_PACKAGES4) +print-p5: + @echo $(MODULES_PACKAGES5) + $(eval $(call guile-compilation-rule,make-core-go, \ $(MODULES_CORE) guix/config.scm $(dist_noinst_DATA), \ 0)) @@ -804,7 +830,27 @@ $(eval $(call guile-compilation-rule,make-packages2-go, \ $(words $(MODULES_CORE) $(MODULES_PACKAGES1)))) .PHONY: make-packages2-go -make-packages-go: make-packages1-go make-packages2-go +$(eval $(call guile-compilation-rule,make-packages3-go, \ + $(MODULES_PACKAGES3) make-core-go make-packages1-go make-packages2-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)))) +.PHONY: make-packages3-go + +$(eval $(call guile-compilation-rule,make-packages4-go, \ + $(MODULES_PACKAGES4) make-core-go make-packages1-go make-packages2-go \ + make-packages3-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3)))) +.PHONY: make-packages4-go + +$(eval $(call guile-compilation-rule,make-packages5-go, \ + $(MODULES_PACKAGES5) make-core-go make-packages1-go make-packages2-go \ + make-packages3-go make-packages4-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3) $(MODULES_PACKAGES4)))) +.PHONY: make-packages5-go + +make-packages-go: make-packages1-go make-packages2-go \ + make-packages3-go make-packages4-go make-packages5-go .PHONY: make-packages-go $(eval $(call guile-compilation-rule,make-system-go, \ -- 2.41.0 From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 17:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: 65456@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169272481232040 (code B ref 65456); Tue, 22 Aug 2023 17:21:02 +0000 Received: (at 65456) by debbugs.gnu.org; 22 Aug 2023 17:20:12 +0000 Received: from localhost ([127.0.0.1]:60417 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV35-0008Kd-TI for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:20:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52456) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV33-0008KD-5U for 65456@debbugs.gnu.org; Tue, 22 Aug 2023 13:20:09 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYV2h-0005sE-Uu; Tue, 22 Aug 2023 13:20:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=Ep45t5a10PI/VqpixTSiANIULKoSrJL8Z8rztjhRCGI=; b=LAm+pVOFS6AMc//MiBoD rI6boW0oqTXbmL/mu4llUlmPhV+IlMP8H+z7NnTTSRzGXo3K9yEA6TrLNn5/wRaUSebyiaaX+tQzn yiog1PNlaS/rVHZkJczp6TFR+NXEwGwVC4D2A/S0q43DfqTGXkFbVQFhLh/af9daE2t0LRMEeQy5T b0UyPn/aPrXEFQr5XXL+iHaOnJo1Ek7Lr04/Q9hJZI14nFAP5F/JJsGih1a4HqZeMd85xsOcD01Vi 071d1KN1HZtUex2b0E1o+FfdqFBfbFyu5P8zzEeQqNw4FAejK03Gjtg473tbwp4ToUosTBqwre6Hz GcJY2Wdc6gzljQ==; From: Janneke Nieuwenhuizen Date: Tue, 22 Aug 2023 19:19:42 +0200 Message-ID: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Similar to the Makefile.am change, this breaks-up packages into 26 chunks when building on 32bit. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define. Use it when building on a "i586" or "i686" cpu. --- guix/self.scm | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..474715fe7a 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017-2023 Ludovic Courtès ;;; Copyright © 2020 Martin Becze +;;; Copyright © 2023 Janneke Nieuwenhuizen ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) + (srfi srfi-71) (ice-9 match) (ice-9 format) + (ice-9 regex) (ice-9 threads) (guix build compile) (guix build utils)) @@ -1251,6 +1255,26 @@ (define* (compiled-modules name module-tree module-files #:report-load report-load #:report-compilation report-compilation))) + (define (process-directory/32bit directory files output) + (let* ((chunks (map (compose + (cute partition <> files) + (lambda (regex) + (cute string-match regex <>)) + (cute string-append "^gnu/packages/" <>) + (cute make-string 1 <>) + integer->char + (cute + (char->integer #\a) <>)) + (iota 26))) + (chunks (filter pair? chunks))) + (for-each + (lambda (chunck) + (compile-files directory #$output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation) + (gc)) + chunks))) + (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) @@ -1277,7 +1301,10 @@ (define* (compiled-modules name module-tree module-files (mkdir #$output) (chdir #+module-tree) - (process-directory "." '#+module-files #$output) + (match (string-split %host-type #\-) + (((or "i586" "i686") rest ...) + (process-directory/32bit "." '#+module-files #$output)) + (_ (process-directory "." '#+module-files #$output))) (newline)))) (computed-file name build -- 2.41.0 From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 1/2] build: Build gnu/packages/*.go in five steps. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 17:34:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: 65456@debbugs.gnu.org Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169272561910934 (code B ref 65456); Tue, 22 Aug 2023 17:34:02 +0000 Received: (at 65456) by debbugs.gnu.org; 22 Aug 2023 17:33:39 +0000 Received: from localhost ([127.0.0.1]:60436 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYVG6-0002qI-MQ for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:33:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40610) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYVG2-0002q1-Rj for 65456@debbugs.gnu.org; Tue, 22 Aug 2023 13:33:36 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYVFt-0000VI-ME; Tue, 22 Aug 2023 13:33:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=/ao2ZZVj9PSF8ZrKmfV6H1Ki2chn+/6562SQa7VZiUA=; b=PahdipKBa2QJWAEowINb LAW194DlvTPiy0Z4vHxy0QDNbhJG0tMmHcUhsl5Q5EZ/HnMkIDZYLuHaEXw9nF26jU7qFb83OEmPj H9ZgLls7SEfD4Nm4ybIVRrYmIW7+QkPT7npFr73ZkUfkkOEeNtI+MlT9axJ0Ltw6qgOHCLp7mxWmb mgOtyexY8v7SyRsFy6YD+3iPsgngp+u6lFYBgSwsSaOs9xVC1NjnGSdHQaXifzHgdl7XPowTT5p2G hb5bSUiYRKcuEemWyx4dC+IS6ZbdNeoE7uwCNS+i20BSy8r9MAPua5Z7B5a9+EWVa1Dd5LnFB/Ezb CLMPG28ohTltUg==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <606351ff2416c9086638a1c379b84ba811b384fa.1692723764.git.janneke@gnu.org> X-Url: http://AvatarAcademy.nl Date: Tue, 22 Aug 2023 19:33:22 +0200 In-Reply-To: <606351ff2416c9086638a1c379b84ba811b384fa.1692723764.git.janneke@gnu.org> (Janneke Nieuwenhuizen's message of "Tue, 22 Aug 2023 19:19:41 +0200") Message-ID: <87sf8b7zj1.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Janneke Nieuwenhuizen writes: > * Makefile.am (first_half): Rename to... [..] > +print-p1: > + @echo $(MODULES_PACKAGES1) > +print-p2: > + @echo $(MODULES_PACKAGES2) > +print-p3: > + @echo $(MODULES_PACKAGES3) > +print-p4: > + @echo $(MODULES_PACKAGES4) > +print-p5: > + @echo $(MODULES_PACKAGES5) > + ...I mean to remove this debugging code, but it might come in handy for review. --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH v2 1/2] build: Build gnu/packages/*.go in five steps. References: In-Reply-To: Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 17:50:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: 65456@debbugs.gnu.org Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169272655712635 (code B ref 65456); Tue, 22 Aug 2023 17:50:02 +0000 Received: (at 65456) by debbugs.gnu.org; 22 Aug 2023 17:49:17 +0000 Received: from localhost ([127.0.0.1]:60451 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYVVF-0003Hi-9E for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:49:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44868) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYVVC-0003HO-PE for 65456@debbugs.gnu.org; Tue, 22 Aug 2023 13:49:15 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYVV4-0003Zv-2m; Tue, 22 Aug 2023 13:49:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=crJIX5iyfoD2VVd6xSyQqHpc+dJlMcKzSKM4+/EoUUo=; b=NCf3gE1cVC6GKU xSTh+L7j3Ot1xRfuVxOefxbSekpikO7gTgG18GEmrBmRxtF5zyoVBaGDXLHtjqeAUKRrWOgxwwoT5 8ieJOo7YwX4HDu0LSHZ8P0LlE+zM4rrN6qTe83tZBEtihQPWXP1VMmsMoWDV2i1SjPb0Zo5R0MwDH lEQwCHO942CV1oK1krI8Qzyv3TtLCRB9rwrzbv2D4VrNXbQ3OnmED+3wkgmQtRd+1eij0LbWQXgQL GIF3u0e5T+fxvVGZTkk3Mb591BShdo4GtY1TkxDpdSZcbc45bc3kSkmOzwNE9KV/Na7q9V1YgPfqb 6qTKn6MfDAPQVgo/TWgw==; From: Janneke Nieuwenhuizen Date: Tue, 22 Aug 2023 19:49:00 +0200 Message-ID: <99867a1bd9869a0fb0f3beacaaec9cec9090cb1e.1692726496.git.janneke@gnu.org> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) This breaks-up packages into five chunks of ~150,000 lines, allowing guix build --target=i586-pc-gnu from an x86 host. This is a followup to 1aa7ee52c6c520c2dbbdb06f1381466e9fd96294. * Makefile.am (first_half): Rename to... (first_quart): ...this, and also split into... (second_quart): ...this. (third_quart, MODULES_PACKAGES3, MODULES_PACKAGE4): New variables. (make-packages3-go, make-packages4-go): New targets. (make-packages-go): Add them. --- Makefile.am | 51 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 8 deletions(-) diff --git a/Makefile.am b/Makefile.am index 738532f839..922913355c 100644 --- a/Makefile.am +++ b/Makefile.am @@ -776,15 +776,30 @@ endef # in . Each 'eval' call below creates a # 'make-*-go' phony target that builds the corresponding subset. -first_half := \ - gnu/packages/a% gnu/packages/b% gnu/packages/c% gnu/packages/d% \ - gnu/packages/e% gnu/packages/f% gnu/packages/g% gnu/packages/h% \ - gnu/packages/i% gnu/packages/j% gnu/packages/k% gnu/packages/l% +first_fifth := \ + gnu/packages/a% gnu/packages/b% gnu/packages/c% + +second_fifth := \ + gnu/packages/d% gnu/packages/e% gnu/packages/f% + +third_fifth := \ + gnu/packages/g% gnu/packages/h% gnu/packages/i% + +fourth_fifth := \ + gnu/packages/j% gnu/packages/k% gnu/packages/l% gnu/packages/m% \ + gnu/packages/n% gnu/packages/o% MODULES_CORE := guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULES))) -MODULES_PACKAGES1 := $(filter $(first_half),$(MODULES)) -MODULES_PACKAGES2 := $(filter-out $(first_half),$(filter gnu/packages/%,$(MODULES))) -MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) +MODULES_PACKAGES1 := $(filter $(first_fifth),$(MODULES)) +MODULES_PACKAGES2 := $(filter $(second_fifth),$(MODULES)) +MODULES_PACKAGES3 := $(filter $(third_fifth),$(MODULES)) +MODULES_PACKAGES4 := $(filter $(fourth_fifth),$(MODULES)) +MODULES_PACKAGES5 := $(filter-out $(first_fifth) $(second_fifth) \ + $(third_fifth) $(fourth_fifth), \ + $(filter gnu/packages/%,$(MODULES))) +MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3) $(MODULES_PACKAGES4) \ + $(MODULES_PACKAGES5) MODULES_SYSTEM := gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES))) MODULES_CLI := $(filter guix/scripts/%,$(MODULES)) MODULES_PO := guix/build/po.scm @@ -804,7 +819,27 @@ $(eval $(call guile-compilation-rule,make-packages2-go, \ $(words $(MODULES_CORE) $(MODULES_PACKAGES1)))) .PHONY: make-packages2-go -make-packages-go: make-packages1-go make-packages2-go +$(eval $(call guile-compilation-rule,make-packages3-go, \ + $(MODULES_PACKAGES3) make-core-go make-packages1-go make-packages2-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)))) +.PHONY: make-packages3-go + +$(eval $(call guile-compilation-rule,make-packages4-go, \ + $(MODULES_PACKAGES4) make-core-go make-packages1-go make-packages2-go \ + make-packages3-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3)))) +.PHONY: make-packages4-go + +$(eval $(call guile-compilation-rule,make-packages5-go, \ + $(MODULES_PACKAGES5) make-core-go make-packages1-go make-packages2-go \ + make-packages3-go make-packages4-go, \ + $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \ + $(MODULES_PACKAGES3) $(MODULES_PACKAGES4)))) +.PHONY: make-packages5-go + +make-packages-go: make-packages1-go make-packages2-go \ + make-packages3-go make-packages4-go make-packages5-go .PHONY: make-packages-go $(eval $(call guile-compilation-rule,make-system-go, \ base-commit: c655231b72ac28b5a433069fcf86a835c9c83691 -- 2.41.0 From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 17:50:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: 65456@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169272655812642 (code B ref 65456); Tue, 22 Aug 2023 17:50:02 +0000 Received: (at 65456) by debbugs.gnu.org; 22 Aug 2023 17:49:18 +0000 Received: from localhost ([127.0.0.1]:60453 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYVVF-0003Hk-OG for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:49:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44882) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYVVD-0003HQ-LO for 65456@debbugs.gnu.org; Tue, 22 Aug 2023 13:49:16 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYVV5-0003a3-9D; Tue, 22 Aug 2023 13:49:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=JZJ/G73ZQRytXVucXZeflULlfHeFU82PKTgmSzUyVCA=; b=UDsu5ICy2s0gipw7m9Of Vt5rv4Zo82ezcENlKkcdeZVQVy1w9tNQPLNa6nG2+2F6TP4DnhT8T0puL9g+umylXonkomTlVScy/ zzpUmZSZZv8p6kHw9TF2M410dRe/ecwlVevORx3bbrGOOFrKbmam6TO1Trt1+jvXvLb6QRWEopxyT BmWHYCdqBu2pi8VaAMGAbQtEJTj7xsm5CD91npPw0CLt9B74KNLS9xifdBSV8G7S4vI8NUPjlufIH YlEctc6yoGPzyhEvGFN5fjHRWy2BerX1jpHKfqG/EOEdBOOcUbK2gSw/ycwnlkDDdAQQGhnJ4r7BC CXy5TDP9dRFoSg==; From: Janneke Nieuwenhuizen Date: Tue, 22 Aug 2023 19:49:01 +0200 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: <99867a1bd9869a0fb0f3beacaaec9cec9090cb1e.1692726496.git.janneke@gnu.org> References: <99867a1bd9869a0fb0f3beacaaec9cec9090cb1e.1692726496.git.janneke@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks when building on 32bit. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory]: Split building of "gnu/packages" into 26 chunks. --- guix/self.scm | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..151fa9f452 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017-2023 Ludovic Courtès ;;; Copyright © 2020 Martin Becze +;;; Copyright © 2023 Janneke Nieuwenhuizen ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) + (srfi srfi-71) (ice-9 match) (ice-9 format) + (ice-9 regex) (ice-9 threads) (guix build compile) (guix build utils)) @@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files (force-output)) (define (process-directory directory files output) - ;; Hide compilation warnings. - (parameterize ((current-warning-port (%make-void-port "w"))) - (compile-files directory #$output files - #:workers (parallel-job-count) - #:report-load report-load - #:report-compilation report-compilation))) + ;; Split gnu/packages in 26 chunks to avoid OOM errors + (let* ((chunks (map (compose + (cute partition <> files) + (lambda (regex) + (cute string-match regex <>)) + (cute string-append "^gnu/packages/" <>) + (cute make-string 1 <>) + integer->char + (cute + (char->integer #\a) <>)) + (iota 26))) + (chunks (filter pair? chunks))) + (for-each + (lambda (chunck) + (parameterize ((current-warning-port (%make-void-port "w"))) + ;; Hide compilation warnings. + (compile-files directory #$output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation)) + (gc)) + chunks))) (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) -- 2.41.0 From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 17:56:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: 65456@debbugs.gnu.org Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169272692613231 (code B ref 65456); Tue, 22 Aug 2023 17:56:02 +0000 Received: (at 65456) by debbugs.gnu.org; 22 Aug 2023 17:55:26 +0000 Received: from localhost ([127.0.0.1]:60468 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYVbB-0003RL-OK for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:55:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51594) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYVb8-0003R3-LZ for 65456@debbugs.gnu.org; Tue, 22 Aug 2023 13:55:24 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYVb0-00053m-AM; Tue, 22 Aug 2023 13:55:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=2c21sYI23CqXMIKrUpQoo/8BI9YoVLLsGevjceTWuRM=; b=qTVmO/K3+4eSXerOBCUr d82Q+K4le3ueyxG/EsjCSeMDP2qO42eE7MMgsgmg3E21HUcryznKnXi145lGu58ucYXUkmoec8Yim ZlWbXPYXxevR1CQwMULykgDcN9eyBHTwCFp0AckyVRPjKvuKc2jfygy152g4zCh3e4/vBUWBHX/nc POFaxT7huIG3/7NQyFdif9ivYiRx5YRLW1//UN5AF6pt2Op1dqxmXmVAuHBkJTJGC6oWBcKBsyoAf qWVNS+re8FjygXURXbbQ+Ze1Op1Pj4t/hpEOG+/tvzu8nZSQpUxZ2qIgdJGj7BCk1DaDHB958ZlWu feR9VbMiBcMq/g==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <99867a1bd9869a0fb0f3beacaaec9cec9090cb1e.1692726496.git.janneke@gnu.org> X-Url: http://AvatarAcademy.nl Date: Tue, 22 Aug 2023 19:55:10 +0200 In-Reply-To: (Janneke Nieuwenhuizen's message of "Tue, 22 Aug 2023 19:49:01 +0200") Message-ID: <87o7iz7yip.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Janneke Nieuwenhuizen writes: > diff --git a/guix/self.scm b/guix/self.scm > index 81a36e007f..151fa9f452 100644 [..] > + (parameterize ((current-warning-port (%make-void-port "= w"))) meant to move this comment > + ;; Hide compilation warnings. above the parameterize; fixed locally Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com From unknown Fri Aug 15 20:27:28 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Janneke Nieuwenhuizen Subject: bug#65456: closed (Re: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps.) Message-ID: References: <87jztm98qy.fsf@gnu.org> X-Gnu-PR-Message: they-closed 65456 X-Gnu-PR-Package: guix X-Gnu-PR-Keywords: patch Reply-To: 65456@debbugs.gnu.org Date: Tue, 22 Aug 2023 19:30:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1692732602-22830-1" This is a multi-part message in MIME format... ------------=_1692732602-22830-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. which was filed against the guix package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 65456@debbugs.gnu.org. --=20 65456: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D65456 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1692732602-22830-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 65456-done) by debbugs.gnu.org; 22 Aug 2023 19:29:11 +0000 Received: from localhost ([127.0.0.1]:60497 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYX3u-0005uc-Jg for submit@debbugs.gnu.org; Tue, 22 Aug 2023 15:29:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46192) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYX3r-0005uH-DN for 65456-done@debbugs.gnu.org; Tue, 22 Aug 2023 15:29:09 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYX3i-0003Es-Mq; Tue, 22 Aug 2023 15:28:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=wpqCs2PJCD7o6Q26O+Myjr1FCi8GrJQyzxOdJjisNcg=; b=NyzR6BCt9/0lclWdvui+ SDEEDpwiTQyWQoC8lG/mOShF4FhMFFinIuHcloIdYcC1Uw6AJtygpqdTI8xGSF1WrnPrKMHWMewz1 X/c7DLVPskvHndLlPNfLodsWLsEdytetXM/NtWkAXOdgGPoJzkKwvWy2jK17ihzLrW5mysq57xUO2 7skIZkaj1WyQVHr6c+pCQK0uFB7EifQ8v/crJDHfVxndNKnZ74MQa3iXNxuud0Cf8Jzp80zKbgd9r /p14WUj8V8lujvNqvBwFIr15k/cCbQA+EqIn02Kc5iKYOd9tOiSuqdKUTH3/BlVkdyFwAYZ0XGut9 ddZ4IiSftecTPA==; From: Janneke Nieuwenhuizen To: 65456-done@debbugs.gnu.org Subject: Re: [PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps. Organization: AvatarAcademy.nl References: <99867a1bd9869a0fb0f3beacaaec9cec9090cb1e.1692726496.git.janneke@gnu.org> <87o7iz7yip.fsf@gnu.org> X-Url: http://AvatarAcademy.nl Date: Tue, 22 Aug 2023 21:28:53 +0200 In-Reply-To: <87o7iz7yip.fsf@gnu.org> (Janneke Nieuwenhuizen's message of "Tue, 22 Aug 2023 19:55:10 +0200") Message-ID: <87jztm98qy.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 65456-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Janneke Nieuwenhuizen writes: After a LGTM from Ludo and discussing with Maxim on IRC, pushed to master as 5898b2e8a3dbf7797e83b39a2783c5b543015725 Maxim suggested () we could maybe use Guile in make to split building .go files in Makefile.am algorithmically like we do in guix/self.scm. Greetings, Janneke --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com ------------=_1692732602-22830-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 22 Aug 2023 17:18:18 +0000 Received: from localhost ([127.0.0.1]:60402 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV1F-0008Gx-S4 for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:18:18 -0400 Received: from lists.gnu.org ([2001:470:142::17]:37640) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV1C-0008Gj-70 for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:18:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYV13-000130-HS for bug-guix@gnu.org; Tue, 22 Aug 2023 13:18:05 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYV12-0005UE-RR; Tue, 22 Aug 2023 13:18:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=MXGP6JnLn55nxGu/RKVZvRSzD14tw//JK6nmQvWbg9I=; b=WjXuKUaiepc7MV tDUtENMXkMHbiRmQFNtusOAjJzKCVtYcMbAGjaqrRceB9nNQuEo9rJLAbgm+/dsgl3XuU5kq7byLM FxnE2gNSC7vmOBvn4+zsJ04utBdDENMrhoDzB8F7SjRwFHogYR/mhuToUbLFXUhbmtlQNtQMLP7IG gGJW9tJyZzg4qNaxQ40Wcmg8Fwk5T2QkbyfFjXAkRlipvXgpbgMUJHzHzHTez4O3OTzs1tz6N342B HIHQm8A5bxyt4lWl+ayf+tANwxVTpIDOUr/QjQG/8DMzc8rnF2Y7eawxiJYSeo5HIEnZdFI39CaoJ jvH27LwYBD2NjbaWL+7g==; From: Janneke Nieuwenhuizen To: bug-guix@gnu.org Subject: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Date: Tue, 22 Aug 2023 19:17:59 +0200 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi! Initially writter for the Hurd (see hurd-team , ) This afternoon on IRC, Maxim confirmed my suspicion that this could be 32bit issue, rather than a Hurd problem (, ). I didn't submit these for the Hurd yet, as I didn't really feel comfortable with the 26-way split. I tried a 5-way split and that still gave OOM errors on the Hurd and I didn't feel like hand-coding a 10-way split, so yeah. Also, the 5-way split in the Makefile.am will produce percentages over 100 which suggests a typo...but I couldn't spot it. Lastly, these patches feel like a workaround for Guile / libgc memory management? Otoh, being able to build Guix on 32bit hosts is kinda nice too... So yeah. Greetings, Janneke Janneke Nieuwenhuizen (2): build: Build gnu/packages/*.go in five steps. self: Build gnu/packages/*.go in 26 steps on 32bit. Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++------- guix/self.scm | 31 ++++++++++++++++++++++++-- 2 files changed, 83 insertions(+), 10 deletions(-) base-commit: c655231b72ac28b5a433069fcf86a835c9c83691 -- 2.41.0 ------------=_1692732602-22830-1-- From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 22 17:48:41 2023 Received: (at control) by debbugs.gnu.org; 22 Aug 2023 21:48:41 +0000 Received: from localhost ([127.0.0.1]:60552 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYZEu-0001GA-Mn for submit@debbugs.gnu.org; Tue, 22 Aug 2023 17:48:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37882) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYZEs-0001FY-CY for control@debbugs.gnu.org; Tue, 22 Aug 2023 17:48:39 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYZEh-00037o-Lc for control@debbugs.gnu.org; Tue, 22 Aug 2023 17:48:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to: references; bh=O2ygIAA1snG2z/8kB/CpKpOe41rXWConyTsBSY1QrOg=; b=GpIAcuWq98DFw3 F3SCwhHKxit2psqdSeSzYCnOvqFa/FdcBygLSfZgZulgvTjD8Q5hO5O2I3mY46Gsio66YsxnOvOFw BH+qUKDLdrpacaiWbgPTFLckcZKPEKFuQ4Bz5mSU2h02i0HLkmwhINwkWTJ+3y3Jvk2HUhllt0gBc DoNrFxMWjt+TvqNE96KJ1YWDr44AqA9ZayX+v5sxPxBcTdwUf8wwZ1U6rDUElWaQlrz5zC5YKLbO4 4l3b67zfZhb5dtrrwHAubp9qGlF4fv7/JVDEv5q+KvcHY+JtFTbefaStRSnKAcUD91PGBvR0BsmOQ /Dcu8hdjVOwVdnYnFskQ==; Date: Tue, 22 Aug 2023 23:48:24 +0200 Message-Id: <87ttsq3g0n.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #65456 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) reopen 65456 tags 65456 - fixed patch quit From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 21:53:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Janneke Nieuwenhuizen Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , 65456@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.16927411375187 (code B ref 65456); Tue, 22 Aug 2023 21:53:01 +0000 Received: (at 65456) by debbugs.gnu.org; 22 Aug 2023 21:52:17 +0000 Received: from localhost ([127.0.0.1]:60557 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYZIO-0001Lb-9p for submit@debbugs.gnu.org; Tue, 22 Aug 2023 17:52:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36712) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYZIL-0001LK-U1 for 65456@debbugs.gnu.org; Tue, 22 Aug 2023 17:52:14 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYZI4-0004Yl-41; Tue, 22 Aug 2023 17:52:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=xA6xKZd0R8odnqBJIEAbgIub4oudzEMvBekUZ7/8lGg=; b=hdqJg/oVb6pSHWqfMM3G H5mz5ppAjTVz7/MpNei7xMnxcmSWdpzyWjAgVsS0i82mBhCAZdb+p6gHZguUV1Z/u2ZSmQrZNR8Ph XxZYTUXIshpGAmhIhnQ9rqCz4wuG37hy9kKcevrpcwW7lZukGACvxE3thDl08QFlJl9E3GzOfyS4j Xu+XnMvsnCDdVCkXfKInfJd5s+CGX2qtwCJ3pw35hilPA2u9N8Y2yGzks91oaGfWGIOFVEoBAR91a YCcHLF+haRQeMhHH5w34U/Nql/KjO2USuNHS9tIDmjmlCVMYAOWSnKUeiwl868r0ptmM1yyPsq3fP TEq/ltVLhW9hoQ==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> Date: Tue, 22 Aug 2023 23:51:53 +0200 In-Reply-To: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> (Janneke Nieuwenhuizen's message of "Tue, 22 Aug 2023 19:19:42 +0200") Message-ID: <87pm3e3fuu.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Janneke Nieuwenhuizen skribis: > Similar to the Makefile.am change, this breaks-up packages into 26 chunks > when building on 32bit. Also force garbage collection. > > * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner de= fine. > Use it when building on a "i586" or "i686" cpu. .go files of a Guix built with this new (guix self) would not be found: --8<---------------cut here---------------start------------->8--- newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-modu= le-union/share/guile/site/3.0/guix/ui.scm", {st_mode=3DS_IFREG|0444, st_siz= e=3D95284, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-modu= le-union/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) =3D -1 E= NOENT (No such file or directory) newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0= .9/lib/guile/3.0/ccache/guix/ui.go", 0x7ffc4bf07640, 0) =3D -1 ENOENT (No s= uch file or directory) newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0= .9/lib/guile/3.0/ccache/guix/ui.go", 0x7ffc4bf07640, 0) =3D -1 ENOENT (No s= uch file or directory) newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0= .9/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) =3D -1 ENOENT = (No such file or directory) newfstatat(AT_FDCWD, "/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/l= ib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) =3D -1 ENOENT (No = such file or directory) newfstatat(AT_FDCWD, "/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/s= hare/guile/site/3.0/guix/ui.go", 0x7ffc4bf07640, 0) =3D -1 ENOENT (No such = file or directory) newfstatat(AT_FDCWD, "/home/ludo/.guix-home/profile/lib/guile/3.0/site-ccac= he/guix/ui.go", 0x7ffc4bf07640, 0) =3D -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/home/ludo/.guix-home/profile/lib/guile/3.0/site-ccac= he/guix/ui.go", 0x7ffc4bf07640, 0) =3D -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "/run/current-system/profile/lib/guile/3.0/site-ccache= /guix/ui.go", {st_mode=3DS_IFREG|0444, st_size=3D530053, ...}, 0) =3D 0 --8<---------------cut here---------------end--------------->8--- In fact, guix/*.go is entirely missing, it seems: $ --8<---------------cut here---------------start------------->8--- ls /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/= 3.0/site-ccache/guix ls: cannot access '/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-= union/lib/guile/3.0/site-ccache/guix': No such file or directory $ ls /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guil= e/3.0/site-ccache/gnu packages/ --8<---------------cut here---------------end--------------->8--- With the patch, without the patch: --8<---------------cut here---------------start------------->8--- $ find -L /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union -na= me \*.go | wc -l 719 $ find -L /gnu/store/4dm864f7iif1vg9sz355gdqkgb7air98-guix-module-union -n= ame \*.go | wc -l 1188 --8<---------------cut here---------------end--------------->8--- The problem would then manifest like this: --8<---------------cut here---------------start------------->8--- $ /gnu/store/57x6wrf78147lif1ss0ck2z3a0323b9l-guix-20230822.21/bin/guix bui= ld hello error: license:arphic-1999: unbound variable hint: Did you forget a `use-modules' form? error: googletest: unbound variable hint: Did you forget a `use-modules' form? [=E2=80=A6] Throw to key `unbound-variable' with args `("resolve-interface" "no binding= `~A' in module ~A" (shared-mime-info (gnu packages gnome)) #f)'. Backtrace: In guix/store.scm: 659:37 19 (thunk) 1298:8 18 (call-with-build-handler # =E2=80=A6) In guix/scripts/build.scm: 584:2 17 (_) In srfi/srfi-1.scm: 673:15 16 (append-map _ _ . _) 586:17 15 (map1 ((argument . "hello") (build-mode . 0) (graft? . #t) (su= bstitutes? . #t) # =E2=80=A6)) In guix/scripts/build.scm: 604:31 14 (_ _) In gnu/packages.scm: 485:2 13 (%find-package "hello" "hello" #f) 365:6 12 (find-best-packages-by-name _ _) 295:56 11 (_ "hello" _) In unknown file: 10 (force #>) In gnu/packages.scm: 242:33 9 (fold-packages # # =E2=80=A6) In guix/discovery.scm: 158:11 8 (all-modules _ #:warn _) In srfi/srfi-1.scm: 460:18 7 (fold # _ ((. #))) In guix/discovery.scm: 148:19 6 (_ _ ()) 115:5 5 (scheme-modules _ _ #:warn _) In srfi/srfi-1.scm: 691:23 4 (filter-map # _ . _) In guix/discovery.scm: 123:24 3 (_ . _) In guix/ui.scm: 343:2 2 (report-unbound-variable-error _ #:frame _) In ice-9/boot-9.scm: 1685:16 1 (raise-exception _ #:continuable? _) 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: Throw to key `match-error' with args `("match" "no matching pattern" (unbou= nd-variable "resolve-interface" "no binding `~A' in module ~A" (shared-mime= -info (gnu packages gnome)) #f))'. --8<---------------cut here---------------end--------------->8--- This is because an old (guix licenses) module would be loaded. I=E2=80=99m reverting for now. Note that (guix self) is very sensitive, so we should test it thoroughly. Here I used =E2=80=98make as-derivation=E2=80=99 and ran the r= esulting =E2=80=98guix=E2=80=99 command. Ludo=E2=80=99. From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 23 Aug 2023 06:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , 65456@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169277140112467 (code B ref 65456); Wed, 23 Aug 2023 06:17:01 +0000 Received: (at 65456) by debbugs.gnu.org; 23 Aug 2023 06:16:41 +0000 Received: from localhost ([127.0.0.1]:60860 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYhAW-0003Ey-Vr for submit@debbugs.gnu.org; Wed, 23 Aug 2023 02:16:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36138) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYhAU-0003Ek-1X for 65456@debbugs.gnu.org; Wed, 23 Aug 2023 02:16:39 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYhAG-0003HI-DT; Wed, 23 Aug 2023 02:16:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=Lu+M30ibGWuuhdcTXqERwEHNqAt551tbsd2Iitsrng0=; b=NrHtHujE4duvQVRO8EzP kko03ynLFbepvSw+6wGKiDKTnd+ZD6U7kPB/Wp0v0g7avhxOO4yM3P/8sH5fYBVjbnTx2K1CnahAE WevaPBVePqvcQz2fYfPX3oq+kw9wG6Y415DwKAOOMcleWC++OiF6InOBG58REKLNAx4/VpYXk4mbT 4JTiBeMnYFcA88T34j4tXUH7/sua+Bm1kwkeJpPYyAVB8TNS7/LsnoZWIFa2xot7/qoNBqaUsvT3k ETeCRGfK0nP1j+6uSjLy0xn6Q/D8LlffNu1KjP9XcAvEmy64mxT352HivhyrGBfMEwfOmLQX6PLuY Zm95yaPSzbkXZA==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> X-Url: http://AvatarAcademy.nl Date: Wed, 23 Aug 2023 08:16:18 +0200 In-Reply-To: <87pm3e3fuu.fsf_-_@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Tue, 22 Aug 2023 23:51:53 +0200") Message-ID: <871qfu8erx.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: Hi! > Janneke Nieuwenhuizen skribis: > >> Similar to the Makefile.am change, this breaks-up packages into 26 chunks >> when building on 32bit. Also force garbage collection. >> >> * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner d= efine. >> Use it when building on a "i586" or "i686" cpu. > > .go files of a Guix built with this new (guix self) would not be found: > > newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-mo= dule-union/share/guile/site/3.0/guix/ui.scm", {st_mode=3DS_IFREG|0444, st_s= ize=3D95284, ...}, 0) =3D 0 Hmm, how did I miss this? > In fact, guix/*.go is entirely missing, it seems: Right...that makes sense now that I look at the code again. Wait...I probably even experienced this breakage after running `guix pull' on the Hurd but failed to notice its cause, the missing guix/*.go, and ascribed it to "something" being broken on Hurd. So the good news is that we'll most probably have guix pull work on the Hurd after fixing this! --8<---------------cut here---------------start------------->8--- (cute partition <> files) (lambda (regex) (cute string-match regex <>)) (cute string-append "^gnu/packages/" <>) --8<---------------cut here---------------end--------------->8--- it still has the "gnu/packages" directory name hardcoded here. (I even believe this was kind-of intentional, meaning not to split-up builds of other directories...). The result is, of course, that partition always produces an empty result for any '("guix/foo.scm") files! It is fixed by doing --8<---------------cut here---------------start------------->8--- (cute string-append "^" directory "/" <>) --8<---------------cut here---------------end--------------->8--- i.e., also splitting guix/ 26 ways. > I=E2=80=99m reverting for now. Thank you for looking into and taking care of this mess I made, sorry! > Note that (guix self) is very sensitive, so we should test it > thoroughly. Using the attached patch, I ran --8<---------------cut here---------------start------------->8--- $ guix pull --branch=3Dmaster --url=3D$PWD [..] 08:11:02 janneke@drakenpad:~/src/guix/master=20 $ guix describe Git checkout: repository: /home/janneke/src/guix/master/ branch: master commit: a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 08:11:04 janneke@drakenpad:~/src/guix/master=20 $ guix build hello /gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1 --8<---------------cut here---------------end--------------->8--- > Here I used =E2=80=98make as-derivation=E2=80=99 and ran the resulting > =E2=80=98guix=E2=80=99 command. Ah, nice that guix works too for me too now --8<---------------cut here---------------start------------->8--- $ make as-derivation [..] /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06 08:13:15 janneke@drakenpad:~/src/guix/master [env] $ /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06/bin/guix bui= ld hello /gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1 --8<---------------cut here---------------end--------------->8--- Greetings, Janneke --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=v3-0001-self-Build-guix-and-gnu-packages-directories-in-2.patch Content-Transfer-Encoding: quoted-printable >From a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 Mon Sep 17 00:00:00 2001 Message-ID: From: Janneke Nieuwenhuizen Date: Thu, 22 Jun 2023 08:30:25 +0200 Subject: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chun= ks when building on 32bit. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory]: Split building of directories into 26 chunks. --- guix/self.scm | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..e5a85eaad6 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2017-2023 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2020 Martin Becze +;;; Copyright =C2=A9 2023 Janneke Nieuwenhuizen ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-f= iles '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) + (srfi srfi-71) (ice-9 match) (ice-9 format) + (ice-9 regex) (ice-9 threads) (guix build compile) (guix build utils)) @@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-= files (force-output)) =20 (define (process-directory directory files output) - ;; Hide compilation warnings. - (parameterize ((current-warning-port (%make-void-port "w"))) - (compile-files directory #$output files - #:workers (parallel-job-count) - #:report-load report-load - #:report-compilation report-compilation))) + ;; Split gnu/packages in 26 chunks to avoid OOM errors + (let* ((chunks (map (compose + (cute partition <> files) + (lambda (regex) + (cute string-match regex <>)) + (cute string-append "^" directory "/" <>) + (cute make-string 1 <>) + integer->char + (cute + (char->integer #\a) <>)) + (iota 26))) + (chunks (filter pair? chunks))) + (for-each + (lambda (chunck) + ;; Hide compilation warnings. + (parameterize ((current-warning-port (%make-void-port "w"= ))) + (compile-files directory #$output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation)) + (gc)) + chunks))) =20 (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea --=20 2.41.0 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com --=-=-=-- From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 23 Aug 2023 09:42:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , 65456@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169278371311271 (code B ref 65456); Wed, 23 Aug 2023 09:42:01 +0000 Received: (at 65456) by debbugs.gnu.org; 23 Aug 2023 09:41:53 +0000 Received: from localhost ([127.0.0.1]:32838 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYkN7-0002vj-4r for submit@debbugs.gnu.org; Wed, 23 Aug 2023 05:41:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37276) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYkN1-0002vQ-Uj for 65456@debbugs.gnu.org; Wed, 23 Aug 2023 05:41:52 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYkMr-0000by-1N; Wed, 23 Aug 2023 05:41:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=0wx6mti9qOpQZNEM1pj4fFtWEPAIy49Ewl/5ebj5m98=; b=YjLjlrt0k9ohEEg3Hc5V p9M33cXYFQCJ0HiBs7a2Fkm//JrU6P5mMxnSNaOlBJ8FmWDIjYOo31LmV+QL3cXGI9hZEgsDMu7u3 VyQVEbfyzUQ6h0Vrvq2TpzSDDrbZD88DUNLLqfDb7PNzj6cj5ptfg8t1tcc1Q84vwPtOhqjziMQI1 eiWXXAlwFq4UOq7NK3JWPPfmgUgN///23Bm1z3kMAoeZFILycb1xczcBIS+umFeMnlEBIR0OiIV9X DJkxQrPt44kKo3AzySzT9WmJTarsviCpJod+Lk9wVoaQ1B9ENrpyhQVwBQp5Raip/t++qTt3iLxDu OH1LUNogOLi/Iw==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> X-Url: http://AvatarAcademy.nl Date: Wed, 23 Aug 2023 11:41:31 +0200 In-Reply-To: <871qfu8erx.fsf_-_@gnu.org> (Janneke Nieuwenhuizen's message of "Wed, 23 Aug 2023 08:16:18 +0200") Message-ID: <87h6oq6qpg.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Janneke Nieuwenhuizen writes: Hi! > Ludovic Court=C3=A8s writes: >> Janneke Nieuwenhuizen skribis: >> [..] > it still has the "gnu/packages" directory name hardcoded here. (I even > believe this was kind-of intentional, meaning not to split-up builds of > other directories...). The result is, of course, that partition always > produces an empty result for any '("guix/foo.scm") files! > > It is fixed by doing > > (cute string-append "^" directory "/" <>) Well, almost...In my test script it worked, but directory is "." in the actual self.scm. So with v3 no .go files get built at all...and then a naive `guix build hello' seems to work!; `guix shell' for example fails. Comparing the list of files from `find /gnu/store/...-guix-20230823.08 -follow' is also helpful. I realised that there's no need to use the name-based splitting hack that Makefile.am uses and changed the creation of chunks to simply use a maximum chunk length of 25 files. New version attached. Greetings, Janneke --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=v4-0001-self-Build-directories-in-chunks-of-max-25-files-.patch Content-Transfer-Encoding: quoted-printable >From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 Message-ID: From: Janneke Nieuwenhuizen Date: Thu, 22 Jun 2023 08:30:25 +0200 Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a time. Similar to split build of make-go in Makefile.am, this breaks-up building directories into chunks of max 25 files. Also force garbage collection. * guix/self.scm (compiled-modules)[process-directory]: Split building of directories into chunks of max 25 files. --- guix/self.scm | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 81a36e007f..fc2dfd8131 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2017-2023 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2020 Martin Becze +;;; Copyright =C2=A9 2023 Janneke Nieuwenhuizen ;;; ;;; This file is part of GNU Guix. ;;; @@ -1210,7 +1211,8 @@ (define* (compiled-modules name module-tree module-fi= les '((guix build compile) (guix build utils))) #~(begin - (use-modules (srfi srfi-26) + (use-modules (srfi srfi-1) + (srfi srfi-26) (ice-9 match) (ice-9 format) (ice-9 threads) @@ -1244,12 +1246,23 @@ (define* (compiled-modules name module-tree module-= files (force-output)) =20 (define (process-directory directory files output) - ;; Hide compilation warnings. - (parameterize ((current-warning-port (%make-void-port "w"))) - (compile-files directory #$output files - #:workers (parallel-job-count) - #:report-load report-load - #:report-compilation report-compilation))) + (let* ((size 25) ;compile max 25 files a = time + (chunks (unfold + (lambda (seed) (< (length seed) size)) ;p + (cute take <> size) ;f + (cute drop <> size) ;g + files ;seed + list))) ;tail + (for-each + (lambda (chunck) + ;; Hide compilation warnings. + (parameterize ((current-warning-port (%make-void-port "w"= ))) + (compile-files directory output chunck + #:workers (parallel-job-count) + #:report-load report-load + #:report-compilation report-compilation)) + (gc)) + chunks))) =20 (setvbuf (current-output-port) 'line) (setvbuf (current-error-port) 'line) base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea --=20 2.41.0 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com --=-=-=-- From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 24 Aug 2023 05:34:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , 65456@debbugs.gnu.org Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169285523626611 (code B ref 65456); Thu, 24 Aug 2023 05:34:01 +0000 Received: (at 65456) by debbugs.gnu.org; 24 Aug 2023 05:33:56 +0000 Received: from localhost ([127.0.0.1]:35904 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qZ2yh-0006v9-OA for submit@debbugs.gnu.org; Thu, 24 Aug 2023 01:33:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42228) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qZ2ye-0006uo-NO for 65456@debbugs.gnu.org; Thu, 24 Aug 2023 01:33:53 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qZ2yV-0004fX-6q; Thu, 24 Aug 2023 01:33:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=zuoOwkXGWatcxxhXrNGEM3ksQhEgfrBdLSa1qxQSJFY=; b=dFYJeRg3bnEzdcBgfds4 eWkta1lRZIJi3E+69sMyUogTwLr50h6rxAHURMDIVrb2uwSjyjTagtXaGpsWUTtucWeoR2AzhANhY jy6NxbU247CfdfTUcAXFp5ixxEM+BcGLQobqP5xU53Hv75Mg4z6THHtiU+SPGfmjHe3Cxkcgfi2bV 6UE31JoipCbLdVnqBAk14OQq2UOrCmILYzTKKw1FYOu7ENRk3KfOMijBHDEPwJWFH9vZC9HQgTZ3Z /7QkwnifvwWCOdWoMSVlf9SeSjXDyTPaDZaEL6p+frEkqn8m9O05uJaQSzUEJQcE/UksbwHbvJztJ 7sCbDU2VCwdOvA==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> X-Url: http://AvatarAcademy.nl Date: Thu, 24 Aug 2023 07:33:38 +0200 In-Reply-To: <871qfu8erx.fsf_-_@gnu.org> (Janneke Nieuwenhuizen's message of "Wed, 23 Aug 2023 08:16:18 +0200") Message-ID: <87zg2h57il.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Janneke Nieuwenhuizen writes: Hi! > Ludovic Court=C3=A8s writes: > > Hi! > >> Janneke Nieuwenhuizen skribis: >> [..] >> In fact, guix/*.go is entirely missing, it seems: > > Right...that makes sense now that I look at the code again. > > Wait...I probably even experienced this breakage after running `guix > pull' on the Hurd but failed to notice its cause, the missing guix/*.go, > and ascribed it to "something" being broken on Hurd. So the good news > is that we'll most probably have guix pull work on the Hurd after fixing > this! So, some good news at last; I can confirm that using v4 of this patch we now have "guix pull", pulling from a local git directory, now fully working on the Hurd! (When pulling from a git url, I get "Illegal instruction" while receiving objects.) See log below. Greetings, Janneke /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=3Dhttps://gitla= b.com/janneke/guix --branch=3Dwip-hurd Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/= guix'... guix pull: error: Git error: the SSL certificate is invalid /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=3Dhttp://gitlab= .com/janneke/guix --branch=3Dwip-hurd Updating channel 'guix' from Git repository at 'http://gitlab.com/janneke/g= uix'... guix pull: error: Git error: the SSL certificate is invalid /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix install nss-certs guix install: warning: Consider running 'guix pull' followed by 'guix package -u' to get up-to-date packages and security updates. The following package will be installed: nss-certs 3.88.1 substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 10= 0.0% The following derivations will be built: /gnu/store/z1jyx8lyhgr1gykiky5wjh5mncwvp6ls-profile.drv /gnu/store/p3kkmm8ap3z68irnzl6dsq9clkzrg9cj-nss-certs-3.88.1.drv /gnu/store/cjkisarl0gcwrvgcc3hklz63x7dz6ji2-nss-3.88.1.tar.xz.drv /gnu/store/iwsyc1bvgn9x7qg0x3an5g2q23a6l2xk-certdata2pem-0.0.0.drv 71.6 MB will be downloaded certdata2pem.c 2KiB 23KiB/s 00:00 [######= ############] 100.0% nss-3.88.1.tar.gz 68.3MiB 2.1MiB/s 00:32 [######= ############] 100.0% building /gnu/store/iwsyc1bvgn9x7qg0x3an5g2q23a6l2xk-certdata2pem-0.0.0.drv= ... building /gnu/store/cjkisarl0gcwrvgcc3hklz63x7dz6ji2-nss-3.88.1.tar.xz.drv.= .. building /gnu/store/p3kkmm8ap3z68irnzl6dsq9clkzrg9cj-nss-certs-3.88.1.drv... building CA certificate bundle... listing Emacs sub-directories... building fonts directory... building directory of Info manuals... building profile with 1 package... killing process 174: Invalid argument killing process 175: Invalid argument /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=3Dhttps://gitla= b.com/janneke/guix --branch=3Dwip-hurd Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/= guix'... guix pull: error: Git error: the SSL certificate is invalid /ssh:childhurd1:/root/src/guix/wip-hurd/ #$ bash -login root@guixydevel ~/src/guix/wip-hurd#=20 root@guixydevel ~/src/guix/wip-hurd# guix pull --url=3Dhttps://gitlab.com/j= anneke/guix --branch=3Dwip-hurd Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/= guix'... receiving objects 49% [################################## = ]Illegal instruction (core dumped) root@guixydevel ~/src/guix/wip-hurd# guix pull --url=3D$PWD --branch=3Dwip-= hurd Updating channel 'guix' from Git repository at '/root/src/guix/wip-hurd'... guix pull: error: Git error: cannot locate remote-tracking branch 'origin/k= eyring' root@guixydevel ~/src/guix/wip-hurd# git branch keyring origin/keyring branch 'keyring' set up to track 'origin/keyring'. root@guixydevel ~/src/guix/wip-hurd# guix pull --url=3D$PWD --branch=3Dwip-= hurd Updating channel 'guix' from Git repository at '/root/src/guix/wip-hurd'... Authenticating channel 'guix', commits 9edb3f6 to beb2704 (26 new commits).= .. guix pull: warning: pulled channel 'guix' from a mirror of https://git.sava= nnah.gnu.org/git/guix.git, which might be stale Building from this channel: guix /root/src/guix/wip-hurd beb2704 substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 10= 0.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 10= 0.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 10= 0.0% building /gnu/store/67p25fiinjvm1h0ga8bays6d0kyy0dp4-config.scm.drv... building /gnu/store/iy189p9rd62ymac23l83h6vz86sx775y-git.scm.drv... building /gnu/store/w5ld3vgpsd7im25zqjrk7rxpw3j6km3j-hash.scm.drv... building /gnu/store/151iwm03mirbpbgsrrmxnpr97bj5w1ps-module-import.drv... building /gnu/store/sdnlhfdqrg0kkh33adg4dcb15qnx4205-module-import.drv... building /gnu/store/s835yp2frsy9mf4ha7q0b3mrg4mc78sc-module-import-compiled= .drv... building /gnu/store/sc3ik9br6k1fzkczg51qr54d9ilkl268-module-import-compiled= .drv... building /gnu/store/yn9hi0kmyygk29s79fa44nqh0wqip82n-compute-guix-derivatio= n.drv... substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% bash 389KiB/s 00:02 = | 907KiB transferred dbus-1.14.0 1.2MiB/s 00:00 = | 355KiB transferred guile-json-3.5.0 5.9MiB/s 00:00= | 61KiB transferre| guix-1.4.0-12.329069a-checkout 2.2MiB/s 00:09 |= 20.3MiB transferred libdaemon-0.14 +inf.0TiB/s 00:00= | 39KiB transferred libevent-2.1.12 167KiB/s 00:03 = | 463KiB transferred mkdir 155KiB/s 00:03 = | 432KiB transferred avahi-0.8 229KiB/s 00:02 = | 466KiB transferred tar 3.2MiB/s 00:00 = | 840KiB transferred guile-avahi-0.4.1 26KiB/s 00:03= | 73KiB transferred xz 670KiB/s 00:01 = | 529KiB transferred building /gnu/store/nr3r518p14rj0672q37c320fkd00r01y-guix-daemon-1.4.0-12.3= 29069a.drv... \ 'build' phaseild-log 286 52 CXX nix/nix-daemon/guix_daemon-guix-daemon.o | 'build' phase47 CXX nix/libstore/libstore_a-pathlocks.o / 'make-dynamic-linker-cache' phas/ killing process 283: Invalid argument killing process 284: Invalid argument substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 10= 0.0% The following derivations will be built: /gnu/store/k6z2f845lqzkz2h4q60w089rzl7p7ay7-profile.drv /gnu/store/k3rsfdspx1bqicpsv6vn6crw9lxqpwba-guix-beb270468.drv /gnu/store/8wyidapyb2qibc1qsizm6r64fz2xjcw9-guix-manual.drv /gnu/store/q2a7nzpg6i5qrh7w1mzzbp12lgwd0f75-guix-translated-texinfo.drv /gnu/store/9fcfgcbswidhr8snjf7zqfyn4znahnmf-guix-beb270468-modules.drv /gnu/store/2kdrxg03299x7n0gb3zrlwvrdgna3gi7-guix-packages-base-modules.drv /gnu/store/vz2mic633i2biqn8v4b4znkl8mxdyasy-guix-packages-base.drv /gnu/store/0hxl8ck28afjvx5q6ga0sgpfywbchizw-guix-core.drv /gnu/store/n8z3r1vp95lrzbbq17h04j087cm6hmv2-guix-core-source.drv /gnu/store/va8ncxqnbmljnxf5vn0rv62mz6lkyk0m-config.scm.drv /gnu/store/v7mn3gb4jbh0qq5yy161rckw4vww6gbv-guix-extra.drv /gnu/store/5qhkmkxwap8fykf5y1fwxxq9kz2yw2x2-guix-core-modules.drv /gnu/store/fwjkn1cf3yjxg8znddbri1f560g60rb2-guix-cli-core-modules.drv /gnu/store/x2h3sb1vpf2nf58byyxfgl79b4pyckn9-guix-cli-core.drv /gnu/store/l07fiikl45adg3iv6ff4p0f3qbf6splw-guix-system-modules.drv /gnu/store/gzgizh4wmbzdhdc9gmx9q9dz2jkxrdcd-guix-system.drv /gnu/store/i8qmsx9myyykv8s2n6f3mrg20nclb43k-guix-packages.drv /gnu/store/ljphi1irng8cc3vnqpfcfxy0rq1i4nv8-guix-config-modules.drv /gnu/store/f4n6bxkzb1r06j1r600ivdll3cbhcvxv-guix-config.drv /gnu/store/hq8s053m2bi184xr5wnymwi56hx2820k-guix-config-source.drv /gnu/store/k6mz0x5yv13dsmhg0247zj19yb5ckjp0-config.scm.drv /gnu/store/lqnmqbi39g7rxmfzs31ricahanaqmbbl-guix-packages-modules.drv /gnu/store/rcfi26j7jyfnmc9cwp2cznacmm40viii-guix-extra-modules.drv /gnu/store/sx0lrfjii302l72y2rzry1g1pn9x7awh-guix-system-tests-modules.drv /gnu/store/43qri996hp9lmjk3dbqp19w2j7s0ixf2-guix-system-tests.drv /gnu/store/z2wrv851s184asfp90zccmbzi37xbfmz-guix-cli.drv /gnu/store/650wrs5h0css6baqq2zyj4cyn46dlcyx-guix-home.drv /gnu/store/vvby7fdf2500fy7v6vl7jp6xcvd0psik-guix-cli-modules.drv /gnu/store/xj31wyyq7i234bgss9dbagcci8siny5s-guix-home-modules.drv /gnu/store/bjb1s22wxm427aig9kwygc7298nkzfjj-guix-daemon.drv /gnu/store/kfnmjag44n6d6giwkmlckf5gqnxx2wid-guix-command.drv /gnu/store/abh52n6ddp3i9w2hj75vqlgx44ynfwv5-guix-locale-guix-packages.drv /gnu/store/p9kdrf7vxihq5gl85wmlpqrz3pd16naq-guix-locale-guix.drv /gnu/store/shgbmdpa189mk7kza5fi6x1dmh0d9d0z-guix-module-union.drv /gnu/store/jl04xar2gc4c0aybslgcnzg2b8ir551p-guix-misc.drv /gnu/store/66fa0pvf2c24kl3dvhh4n4f8j4s0kzhi-guix-daemon.cil.drv /gnu/store/50fk0z57vprxh13pp000alz6200lm9r6-profile.drv /gnu/store/ij3cnmqmcfg8jd81j1k87xski8cifp2h-inferior-script.scm.drv 5 items will be downloaded module-import-compiled 800KiB/s 00:00= | 64KiB transferred guile-wrapper 160KiB/s 00:0= 0 | 3KiB transferred module-import-compiled 541KiB/s 00:00= | 81KiB transferred module-import-compiled 860KiB/s 00:00 = | 138KiB transferred module-import-compiled 328KiB/s 00:00= | 10KiB transferred building /gnu/store/k6mz0x5yv13dsmhg0247zj19yb5ckjp0-config.scm.drv... building /gnu/store/va8ncxqnbmljnxf5vn0rv62mz6lkyk0m-config.scm.drv... building /gnu/store/hq8s053m2bi184xr5wnymwi56hx2820k-guix-config-source.drv= ... building /gnu/store/n8z3r1vp95lrzbbq17h04j087cm6hmv2-guix-core-source.drv... building /gnu/store/f4n6bxkzb1r06j1r600ivdll3cbhcvxv-guix-config.drv... building /gnu/store/0hxl8ck28afjvx5q6ga0sgpfywbchizw-guix-core.drv... building /gnu/store/ljphi1irng8cc3vnqpfcfxy0rq1i4nv8-guix-config-modules.dr= v... building /gnu/store/5qhkmkxwap8fykf5y1fwxxq9kz2yw2x2-guix-core-modules.drv.= .. building /gnu/store/66fa0pvf2c24kl3dvhh4n4f8j4s0kzhi-guix-daemon.cil.drv... building /gnu/store/v7mn3gb4jbh0qq5yy161rckw4vww6gbv-guix-extra.drv... building /gnu/store/abh52n6ddp3i9w2hj75vqlgx44ynfwv5-guix-locale-guix-packa= ges.drv... building /gnu/store/rcfi26j7jyfnmc9cwp2cznacmm40viii-guix-extra-modules.drv= ... building /gnu/store/p9kdrf7vxihq5gl85wmlpqrz3pd16naq-guix-locale-guix.drv... building /gnu/store/jl04xar2gc4c0aybslgcnzg2b8ir551p-guix-misc.drv... building /gnu/store/vz2mic633i2biqn8v4b4znkl8mxdyasy-guix-packages-base.drv= ... building /gnu/store/q2a7nzpg6i5qrh7w1mzzbp12lgwd0f75-guix-translated-texinf= o.drv... building /gnu/store/x2h3sb1vpf2nf58byyxfgl79b4pyckn9-guix-cli-core.drv... building /gnu/store/8wyidapyb2qibc1qsizm6r64fz2xjcw9-guix-manual.drv... building /gnu/store/fwjkn1cf3yjxg8znddbri1f560g60rb2-guix-cli-core-modules.= drv... building /gnu/store/2kdrxg03299x7n0gb3zrlwvrdgna3gi7-guix-packages-base-mod= ules.drv... building /gnu/store/i8qmsx9myyykv8s2n6f3mrg20nclb43k-guix-packages.drv... building /gnu/store/lqnmqbi39g7rxmfzs31ricahanaqmbbl-guix-packages-modules.= drv... building /gnu/store/gzgizh4wmbzdhdc9gmx9q9dz2jkxrdcd-guix-system.drv... building /gnu/store/650wrs5h0css6baqq2zyj4cyn46dlcyx-guix-home.drv... building /gnu/store/l07fiikl45adg3iv6ff4p0f3qbf6splw-guix-system-modules.dr= v... building /gnu/store/z2wrv851s184asfp90zccmbzi37xbfmz-guix-cli.drv... building /gnu/store/xj31wyyq7i234bgss9dbagcci8siny5s-guix-home-modules.drv.= .. building /gnu/store/vvby7fdf2500fy7v6vl7jp6xcvd0psik-guix-cli-modules.drv... building /gnu/store/43qri996hp9lmjk3dbqp19w2j7s0ixf2-guix-system-tests.drv.= .. building /gnu/store/sx0lrfjii302l72y2rzry1g1pn9x7awh-guix-system-tests-modu= les.drv... building /gnu/store/9fcfgcbswidhr8snjf7zqfyn4znahnmf-guix-beb270468-modules= .drv... building /gnu/store/shgbmdpa189mk7kza5fi6x1dmh0d9d0z-guix-module-union.drv.= .. building /gnu/store/kfnmjag44n6d6giwkmlckf5gqnxx2wid-guix-command.drv... building /gnu/store/bjb1s22wxm427aig9kwygc7298nkzfjj-guix-daemon.drv... building /gnu/store/k3rsfdspx1bqicpsv6vn6crw9lxqpwba-guix-beb270468.drv... building CA certificate bundle... listing Emacs sub-directories... building fonts directory... building directory of Info manuals... building profile with 1 package... building /gnu/store/ij3cnmqmcfg8jd81j1k87xski8cifp2h-inferior-script.scm.dr= v... building package cache... building profile with 1 package... hint: Consider setting the necessary environment variables by running: GUIX_PROFILE=3D"/root/.config/guix/current" . "$GUIX_PROFILE/etc/profile" Alternately, see `guix package --search-paths -p "/root/.config/guix/curren= t"'. hint: After setting `PATH', run `hash guix' to make sure your shell refers = to `/root/.config/guix/current/bin/guix'. root@guixydevel ~/src/guix/wip-hurd# GUIX_PROFILE=3D"/root/.config/gui= x/current" . "$GUIX_PROFILE/etc/profile" ~/src/guix/wip-hurd# type -p guix /root/.config/guix/current/bin/guix root@guixydevel ~/src/guix/wip-hurd# guix --version guix (GNU Guix) beb27046836d60974390f6a940167e8b5def1066 Copyright (C) 2023 the Guix authors License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. root@guixydevel ~/src/guix/wip-hurd# guix describe Generation 1 Aug 24 2023 01:34:21 (current) guix beb2704 repository URL: /root/src/guix/wip-hurd branch: wip-hurd commit: beb27046836d60974390f6a940167e8b5def1066 root@guixydevel ~/src/guix/wip-hurd# guix shell -D guix hint: Consider passing the `--check' option once to make sure your shell do= es not clobber environment variables. root@guixydevel ~/src/guix/wip-hurd [env]# guix build hello substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0% The following file will be downloaded: /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1 substituting /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1... downloading from http://dezyne.org:8181/nar/zstd/cqnh7hih2n42mfx3x9yy11i3jz= yn6gqj-hello-2.12.1 ... hello-2.12.1 297KiB/s 00:00= | 74KiB transferred /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1 root@guixydevel ~/src/guix/wip-hurd [env]#=20 --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps. Resent-From: "Dr. Arne Babenhauserheide" Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 24 Aug 2023 06:46:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Janneke Nieuwenhuizen Cc: ludo@gnu.org, 65456@debbugs.gnu.org, dev@jpoiret.xyz X-Debbugs-Original-Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 65456@debbugs.gnu.org, bug-guix@gnu.org, Josselin Poiret Received: via spool by submit@debbugs.gnu.org id=B.16928595291562 (code B ref -1); Thu, 24 Aug 2023 06:46:01 +0000 Received: (at submit) by debbugs.gnu.org; 24 Aug 2023 06:45:29 +0000 Received: from localhost ([127.0.0.1]:35988 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qZ45x-0000P8-Fe for submit@debbugs.gnu.org; Thu, 24 Aug 2023 02:45:29 -0400 Received: from lists.gnu.org ([2001:470:142::17]:36820) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qZ45v-0000Ot-DY for submit@debbugs.gnu.org; Thu, 24 Aug 2023 02:45:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qZ45k-0001UZ-Gc for bug-guix@gnu.org; Thu, 24 Aug 2023 02:45:16 -0400 Received: from mout.web.de ([212.227.15.14]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qZ45h-0002ce-JF; Thu, 24 Aug 2023 02:45:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1692859499; x=1693464299; i=arne_bab@web.de; bh=5jgroDApdYyZilf1+iR6I8qQODxATOwdM5ket07Ba2I=; h=X-UI-Sender-Class:References:From:To:Cc:Subject:Date:In-reply-to; b=WVXT8OWErIs89CnXjGKt0Gl3tRG09p8c5blJeMm7WCtoQS5hwC0Ea+O9eg5ptyaXCtcOeEO //NOclJOj4IYAHeLElzZRKvIZ6XiLZldduiIsFHsETy6sJc8lSOGE/3Y3vUAkJjImRn0+kFzd if/7HSqSWiB4g9J8eQBeDnRBV8gc/SMJWoFbuouToQgzDNeCqi7JWbFbmoHMLQzOMYSXjDEfB dKyGPYo40A6VaZNn5CEHTiQppEtM1VdzKHMYrLnDbZJAbnG4yacCwGYQrG0HQO4QlDdApbU+F Q3LP8g7Zq9PhTrYkS2uk498ZLFkzo32PIuGVeRUpbhkPuioaaaQg== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Received: from fluss ([84.165.27.117]) by smtp.web.de (mrweb006 [213.165.67.108]) with ESMTPSA (Nemesis) id 1MYLmo-1qCOjJ1PbW-00VfAg; Thu, 24 Aug 2023 08:44:59 +0200 References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> <87zg2h57il.fsf@gnu.org> User-agent: mu4e 1.10.5; emacs 29.0.92 From: "Dr. Arne Babenhauserheide" Date: Thu, 24 Aug 2023 08:44:26 +0200 In-reply-to: <87zg2h57il.fsf@gnu.org> Message-ID: <87zg2hvt05.fsf@web.de> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Provags-ID: V03:K1:ItbLYmPzrYXC9X8GVKU1tx2WP6SlNgofUkWw1U9W97/xR2G8htE AQyRx/0nw6WMRYXOyl1irQRyHd93Q4g2NRSVD/v2gbc8GISTm5LX9UM//XGGipa7jlVirIT 0du/Smt6gCkNozqqvW/pORFheEUOv0nH0O24wfabtOTya8b9K2KowOtBVyri5vLSVVI7HL0 SgZWZHvQtelxsH8dkJfEQ== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:8RoeHI8NgDo=;hVDSwm3svziAFJP3cvMbsVG0g4V bNNMIRM9l7fB9kkgKiUqPTDevdciotNiG6RkXH2mJPefYuz9JRd6ogr/qXAJs/yKqxbaH0KE0 wCBZYhGWws+uIwvhTz5501WckSUTOziYI1hI3Ltja8q65vPmgEzU/wtE4zoZA/YzEniBV/agG 1RHiy/0+BprQ7FlgIyR9i7bHNKxwpZTh4ddT9KkxGuoenRsIYnePE+dS8NcYLpMYE4O5/2w// vugc/js2WxXTb7UmujKOnefcHBVVJazc0jd7lC/Gum2cY5xFIyoehuknV60WVKWvUEFaibISW 75L+QcCzP78ma8l5cx3ZcmCXnhj4PKdc00P2nFAc1n5FB5Za4JEsmZsb8e8T1qgRhsz8SKxP9 ohDnIma4CEYzwhntsIo9GunT+3B83QRD1Iadj6ABW0b6ZWy5jsPPJFUltwZx9gf0LdWPZ4qoU OMXdZczW35L6JtGNP1hj8P9LYfcytCcNMX1FfpnHLI6FdsRHUmv1ijzmB6CnQNuhYHEE02/OQ 0TsxUlK8o5ObNyfZVfw/l/DFhln5I8zzzPb/1XGNQlsA4ZWW0/6gt8SDu3dxSCtl9bY0k/ILk T8hLD/2WRl4sAeR2MLT/uNC1v+3jg0l7mMVvJDRSC2AaydjFiWm5LxV2v6vb9f1TLhwlfeaC1 /9jodkVPZMC3rxo0JcP6acxNzTcAsrtNa6WJc45NSd1QFbI3p4nMq8+rB9IVkj3MeDdnTwOMG IC7SAElFPWrcjkj9uuJIerWkuB5b4Z4WazJN2emYW2jZoz72fK6lkWxkb546Sgwb37buZXUla O7w16nufBWDae42uBD0A0j4txSlmI/UXuBbXQaXhfm3HkSYT/7PlG8Vp2JY3ZNFQYU7YZh5R5 23WYHACGltjo1RrHsnM0Zs1MvUm+4yfwFzazY+E0q8yHxlk6B7aHG4vWcM5uEQBz5eZH7NnCu +FRKTw== Received-SPF: pass client-ip=212.227.15.14; envelope-from=arne_bab@web.de; helo=mout.web.de X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 1.0 (+) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Janneke Nieuwenhuizen writes: > So, some good news at last; I can confirm that using v4 of this patch we > now have "guix pull", pulling from a local git directory, now fully > working on the Hurd! That=E2=80=99s awesome! Thank you for your work! Best wishes, Arne =2D-=20 Unpolitisch sein hei=C3=9Ft politisch sein, ohne es zu merken. draketo.de --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJEBAEBCAAuFiEE801qEjXQSQPNItXAE++NRSQDw+sFAmTm/GoQHGFybmVfYmFi QHdlYi5kZQAKCRAT741FJAPD6ziZEACyFGL3aNHfDk39Q5gClsX0Obv0uhQqxELa SLy1+cSYOp9edrK+n3KNFLO4FGhK7cvvv1AVV+ExefCW080zRL5RskoOK34nAbRb n8DHj9htOmVqZylSFZXkBqUBNlxMBSnbMFIcx8HnAiqI7Bju7Bu9cKZP8rXmlNQ+ XryzaueTaJIppsGd6lYHBZUrQsP6JgcofnSa/1qpSMNDAX5WUxVbZV1Z73VZWvZJ eJcYiI76JVf9wOLuHq7gZrvbehp/NKmqm66qMyLxZ8yRUehwwdnItShyKfKgYi4q ssQv1MWYlm4Q5kv+sTY1EyHdhw703D4m518KIcBhG9zAefz8v62oaI0qC2Oa7mIs Wm80Gsrc4OMkCGW1vHFlZv1hKqgYk1LfJ7mEJvV3nRgyRxQCLeguvfNT+J1EoLg6 S+7Mq3GkAvnSUs3MsfJwmyXD/IjC2WsNDmgppA84xZ2F+g3rGJzHZ8U2zJfMfeAK XLy5TRm5A9FMHn+Dc379cfD/9ikdi0y1FkLcfCOiJ5weQsxCMcmYCOal9RkJm/+Q 9V2+w1KjOPANpwGIhenWzV0zxNhDBHlj96Lu5xEZJH3weJXM4qMg5GdOep7czMlq U+FNrQJnbMFML7foA9LJFITWC23L2hYacOP4X7JUOSqCH4G4JXyURd2Ef3XIO5Yd /ZLXcZ5tlIjEBAEBCAAuFiEE3Si95tmHXKvOSosd3M8NswvBBUgFAmTm/GoQHGFy bmVfYmFiQHdlYi5kZQAKCRDczw2zC8EFSGFrA/9sN2IEACGKnYVJTn6YsCYyI5XQ gOrB+f+Y6mUWJMiyGxbftAgXMyYunuchN5Y7JOz/C9Qq9zUn+gW7tA3Fc3c2DsfS /EI19S0yfP59L633+7FX5WxR6wrBK6DFNvn/55YMhHzqGbwIkiBUc/4VTbp+iFO/ 7PQMJ0bCdbnCZhBeAw== =idoo -----END PGP SIGNATURE----- --=-=-=-- From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 24 Aug 2023 14:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Janneke Nieuwenhuizen Cc: Josselin Poiret , 65456@debbugs.gnu.org, Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.169288818231611 (code B ref 65456); Thu, 24 Aug 2023 14:44:02 +0000 Received: (at 65456) by debbugs.gnu.org; 24 Aug 2023 14:43:02 +0000 Received: from localhost ([127.0.0.1]:38372 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qZBY5-0008Da-V0 for submit@debbugs.gnu.org; Thu, 24 Aug 2023 10:43:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59298) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qZBY4-0008DJ-Pb for 65456@debbugs.gnu.org; Thu, 24 Aug 2023 10:43:01 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qZBXt-0002pX-7b; Thu, 24 Aug 2023 10:42:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=EX22l2OaN++J/SLX8onjGIXWXbfdoEt0jw2uW6vgA4s=; b=nUhHZsiFxRBaOU04sWVO Jne3nfvHz/W+5KZgM0eZwYt3jdmX1wCBFv+VXOA2xzjOYWn7im2SWARKABUJ5uO6SXvqJAsSzzqge SBKjHPeHNbfkIzpiQn7HL1v0Qvj2BHQNvaCFKnpBRlfL4nAaoXN2WcbIBsVRnfugAuK10W81j2Dwy lvFINPR+NjisPQ72yEU5Fwk5++LVRx7QkB8fw33qObrGRxkbcAql6P1k40+qel68kld7CvPcEWM0c B9T6Hz8wCboWYEKiEql9RSOYgClwckNo5lW3/G3w7TFhS5HeuXooLkJOyUX5m3/TngVxpi9QEKU/Z Owl5xcTp6QNhbw==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> <87h6oq6qpg.fsf_-_@gnu.org> Date: Thu, 24 Aug 2023 16:42:32 +0200 In-Reply-To: <87h6oq6qpg.fsf_-_@gnu.org> (Janneke Nieuwenhuizen's message of "Wed, 23 Aug 2023 11:41:31 +0200") Message-ID: <87zg2gy013.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Janneke Nieuwenhuizen skribis: >>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 > Message-ID: > From: Janneke Nieuwenhuizen > Date: Thu, 22 Jun 2023 08:30:25 +0200 > Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a > time. > > Similar to split build of make-go in Makefile.am, this breaks-up building > directories into chunks of max 25 files. Also force garbage collection. The big difference with =E2=80=98make-go=E2=80=99 is that =E2=80=98make-go= =E2=80=99 spawns a new process for each chunk of files: each process starts with an empty heap, which is not the case here as we reuse the same process. However, (guix self) is already splitting gnu/packages/*.scm in two pieces: =E2=80=98guix-packages-base=E2=80=99 and =E2=80=98guix-packages=E2= =80=99. The former is the closure of (gnu packages base), and the latter contains the remaining files. Unfortunately this is uneven: --8<---------------cut here---------------start------------->8--- $ readlink -f $(type -P guix) /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep = packages-base /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base /gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source /gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules $ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base -typ= e f |wc -l 361 $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep = packages$ /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1= avx-guix-locale-guix-packages | wc -l 64 $ guix describe Generation 271 Aug 20 2023 23:48:59 (current) guix a0f5885 repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05 --8<---------------cut here---------------end--------------->8--- Maxime Devos looked into this a while back: https://issues.guix.gnu.org/54539 > * guix/self.scm (compiled-modules)[process-directory]: Split building of > directories into chunks of max 25 files. [...] > (define (process-directory directory files output) > - ;; Hide compilation warnings. > - (parameterize ((current-warning-port (%make-void-port "w"))) > - (compile-files directory #$output files > - #:workers (parallel-job-count) > - #:report-load report-load > - #:report-compilation report-compilation))) > + (let* ((size 25) ;compile max 25 files = a time > + (chunks (unfold > + (lambda (seed) (< (length seed) size)) ;p > + (cute take <> size) ;f > + (cute drop <> size) ;g > + files ;seed > + list))) ;tail > + (for-each > + (lambda (chunck) s/chunck/chunk/ Can you confirm that this reduces memory usage observably? One way to check that would be to print (gc-stats) from =E2=80=98process-directory=E2= =80=99, with and without the change. Could you give it a try? Intuitively, I don=E2=80=99t see why it would eat less memory; maybe peak m= emory usage is lower because we do less at once? Also, I think we should remove the explicit (gc) call: it should not be necessary, and if we depend on that, something=E2=80=99s wrong. Anyhow, thanks for tackling this issue! Ludo=E2=80=99. From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 01 Sep 2023 12:50:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , 65456@debbugs.gnu.org, Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , Janneke Nieuwenhuizen Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.16935725427766 (code B ref 65456); Fri, 01 Sep 2023 12:50:02 +0000 Received: (at 65456) by debbugs.gnu.org; 1 Sep 2023 12:49:02 +0000 Received: from localhost ([127.0.0.1]:60033 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qc3a9-00020z-Vs for submit@debbugs.gnu.org; Fri, 01 Sep 2023 08:49:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38356) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qc3a7-00020h-KZ for 65456@debbugs.gnu.org; Fri, 01 Sep 2023 08:49:00 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qc3Zr-0004wp-Mj; Fri, 01 Sep 2023 08:48:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=kiJ9+wIc58mvfXQZTng7YzBPInHkqy0QH+XPMFT7f9o=; b=pwuX5n4dXAg+0H6ti7N/ uM1GEx4AIEg4jn+oRTXIkLcqX1sUUVITSNAB7yeks4Z8eCI9AejV/jVMROKO0ct6biq+6epkfs1R6 hPwZ2NFHtJXGdHtm7PNcGs9OTP7SZuKF6AH7c1P+WXFr8CTm5eqR+VAcoZbSRiNJpumprFNLBCunc 2je0yoWD/Dnt0iI6r6JrKPlaOaTzlRbku/yNB9UU+mx6NbiBRiQEbCclVVTK7xy5K4pY/A0yWeT92 2vJxtw38jKqLcUTAC4o2EsQ1moFPX09HpRBXh3XoFvBr3NgE7dr9kAxGpvZZwI0B5hhwbWV7PLFTc X6R9udkelxfa6A==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> <87h6oq6qpg.fsf_-_@gnu.org> <87zg2gy013.fsf_-_@gnu.org> X-Url: http://AvatarAcademy.nl Date: Fri, 01 Sep 2023 14:48:38 +0200 In-Reply-To: <87zg2gy013.fsf_-_@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Thu, 24 Aug 2023 16:42:32 +0200") Message-ID: <87h6oeys7t.fsf@verum.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Ludovic Court=C3=A8s writes: Hello! > Janneke Nieuwenhuizen skribis: > >>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 >> Message-ID: >> From: Janneke Nieuwenhuizen >> Date: Thu, 22 Jun 2023 08:30:25 +0200 >> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at= a >> time. >> >> Similar to split build of make-go in Makefile.am, this breaks-up building >> directories into chunks of max 25 files. Also force garbage collection. > > The big difference with =E2=80=98make-go=E2=80=99 is that =E2=80=98make-g= o=E2=80=99 spawns a new process > for each chunk of files: each process starts with an empty heap, which > is not the case here as we reuse the same process. Right. > However, (guix self) is already splitting gnu/packages/*.scm in two > pieces: =E2=80=98guix-packages-base=E2=80=99 and =E2=80=98guix-packages= =E2=80=99. The former is the > closure of (gnu packages base), and the latter contains the remaining > files. Unfortunately this is uneven: Okay... > $ readlink -f $(type -P guix) > /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command > $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|gre= p packages-base > /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base > /gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source > /gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules > $ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base -t= ype f |wc -l > 361 > $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|gre= p packages$ > /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages > ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9= f1avx-guix-locale-guix-packages | wc -l > 64 > $ guix describe > Generation 271 Aug 20 2023 23:48:59 (current) > guix a0f5885 > repository URL: https://git.savannah.gnu.org/git/guix.git > branch: master > commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05 > > Maxime Devos looked into this a while back: > > https://issues.guix.gnu.org/54539 Oh my.... >> * guix/self.scm (compiled-modules)[process-directory]: Split building of >> directories into chunks of max 25 files. >> + (for-each >> + (lambda (chunck) > > s/chunck/chunk/ Oops, fixed. > Can you confirm that this reduces memory usage observably? One way to > check that would be to print (gc-stats) from =E2=80=98process-directory= =E2=80=99, with > and without the change. Could you give it a try? What a good and seemingly simple question. After a week of instrumentation and testing, my answer can only be: I tried, and maybe. (see below). > Intuitively, I don=E2=80=99t see why it would eat less memory; maybe peak= memory > usage is lower because we do less at once? Okay... > Also, I think we should remove the explicit (gc) call: it should not be > necessary, and if we depend on that, something=E2=80=99s wrong. > Anyhow, thanks for tackling this issue! Hehe. You've probably seen Josselin's recent GraphML backend effort that might really help to address this? I'm afraid this patch can maybe only postpone what really needs to be done... There is gc-stats output from a successful `guix pull' or `make as-derivation' on Guix/Hurd, that I can show you, and I've tried more than 20 times; it always fails (OOM, hang, spontaneous reset, ...). Below is a typical output of gc-stats on the Hurd for building self.scm, when heap-size peaks (using the the max 25 files patch): --8<---------------cut here---------------start------------->8--- ((gc-time-taken . 1530) (heap-size . 2,625,474,560) (heap-free-size . 1127989248) (heap-total-allocated . 1337029496) (heap-allocated-since-gc . 28728) (protected-objects . 28) (gc-times . 324)) --8<---------------cut here---------------end--------------->8--- notice that it's *much* bigger (more than twice) than my findings on linux-64 below. I have no idea why this is of what it might mean... So I turned to Guix GNU/Linux to get some gc-stat measurements. What you see below is the maximum head-size at any point (I also have heap-total-allocated but I think that's irrelevant? and initially didn't use a script that measured the time). --8<---------------cut here---------------start------------->8--- * guix/self.scm: Vanilla, not chunked; print gc-stats. ((gc-time-taken . 27319485051) (heap-size . 1,360,330,752) (heap-free-size . 285,696,000) (heap-total-allocated . 74,067,590,944) (heap-allocated-since-gc . 186,250,144) (protected-objects . 28) (gc-times . 464)) real 24m36.643s * guix/self.scm: Split building of directories into 26 chunks; print gc-sta= ts. (heap-size . 1,131,298,816) * guix/self.scm: Split building of directories into 26 chunks; no gc; print= gc-stats. (heap-size . 1,121,116,160) * guix/self.scm: Chunks of 25 files; run gc; print gc-stats. (heap-size . 1,066,725,376) * guix/self.scm: Chunks of 50 files; no gc; print gc-stats. (heap-size . 1,299,230,720) real 26m40.708s * guix/self.scm: Chunks of 25 files; no gc; print gc-stats. (heap-size . 1,024,045,056) ; 1st run real 28m4.451s * guix/self.scm: Chunks of 10 files; no gc; print gc-stats. (heap-size . 1,077,895,168) real 30m14.049s --8<---------------cut here---------------end--------------->8--- ...strangely enough, if we assume that these statistics translate to the Hurd, using chunks of max 25 files seems to be a sort of sweet spot? 25% less peak memory (~300MB), "only" 12% (3"45') slower... though not great for GNU/Linux users... I have produced a handful of successful `guix pull's (from a local checked-out worktree) using the 26-way split and chunks of max-25 files patches, but sadly also many more attempts failed. Initially, when creating this patch series, I was convinced this fixed building on the Hurd, but I'm much less enthusiastic now. So I still have a slight preference for using the latest max-25-files patch, but I'm sorry to say that I cannot back it up with tangible data. All in all a rather discouraging week with much effort spent for little gain. Hopefully Josselin can do some of his magic here :) Greetings, Janneke --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com From unknown Fri Aug 15 20:27:28 2025 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 16 Sep 2023 15:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , 65456@debbugs.gnu.org, Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.16948774113518 (code B ref 65456); Sat, 16 Sep 2023 15:17:01 +0000 Received: (at 65456) by debbugs.gnu.org; 16 Sep 2023 15:16:51 +0000 Received: from localhost ([127.0.0.1]:48278 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qhX2R-0000uf-9s for submit@debbugs.gnu.org; Sat, 16 Sep 2023 11:16:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50042) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qhX2O-0000uP-Ea for 65456@debbugs.gnu.org; Sat, 16 Sep 2023 11:16:50 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qhX29-0005AV-9w; Sat, 16 Sep 2023 11:16:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=66Fm+6ivg6wgunEtF2D7GBV5QewktvOHAv2DVmSKan4=; b=EBja5e0TtBzWdF2e2Bfp K0iGcr2AB9q1Wyp+wC4h+G3iu39eI60zk/sT8OQrXiUP8N+GzXSHhwMaDbDsrzOY9qW13w+bcTVjV ddqBpe6sNusNLpEI912OBagdsddHug63apaki301QfJsi78mHaJarEOhz5bUQxK/Aesefl/HWF10F skW4UKYdXymrcgYR5BG2SHuZLiWowiDb5LTopEv8B0+k9ePxy30q8oGMJtbNxcPZTiBRipE8k8WSH vAagyFt+Xv9KyIWiAebZSCk6LtgC7M/LkIyyrJRHPm7rgKNn7RkXpMG1T1bx2sevKaWRFdLA0PfRS bOJXg4ncLPs94Q==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> <87h6oq6qpg.fsf_-_@gnu.org> <87zg2gy013.fsf_-_@gnu.org> <87h6oeys7t.fsf@verum.com> X-Url: http://AvatarAcademy.nl Date: Sat, 16 Sep 2023 17:16:28 +0200 In-Reply-To: <87h6oeys7t.fsf@verum.com> (Janneke Nieuwenhuizen's message of "Fri, 01 Sep 2023 14:48:38 +0200") Message-ID: <87led6ywqb.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Janneke Nieuwenhuizen writes: Hi! > Ludovic Court=C3=A8s writes: > > Hello! > >> Janneke Nieuwenhuizen skribis: >> >>>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 [..] >> Anyhow, thanks for tackling this issue! > > Hehe. You've probably seen Josselin's recent GraphML backend effort > that might really help to address this? I'm afraid this patch can maybe > only postpone what really needs to be done... > > There is gc-stats output from a successful `guix pull' or `make > as-derivation' on Guix/Hurd, that I can show you, and I've tried more > than 20 times; it always fails (OOM, hang, spontaneous reset, ...). > > Below is a typical output of gc-stats on the Hurd for building self.scm, > when heap-size peaks (using the the max 25 files patch): > > ((gc-time-taken . 1530) > (heap-size . 2,625,474,560) > (heap-free-size . 1127989248) > (heap-total-allocated . 1337029496) > (heap-allocated-since-gc . 28728) > (protected-objects . 28) > (gc-times . 324)) > > > notice that it's *much* bigger (more than twice) than my findings on > linux-64 below. I have no idea why this is of what it might mean... > > So I turned to Guix GNU/Linux to get some gc-stat measurements. What > you see below is the maximum head-size at any point (I also have > heap-total-allocated but I think that's irrelevant? and initially didn't > use a script that measured the time). > > * guix/self.scm: Vanilla, not chunked; print gc-stats. > ((gc-time-taken . 27319485051) > (heap-size . 1,360,330,752) > (heap-free-size . 285,696,000) > (heap-total-allocated . 74,067,590,944) > (heap-allocated-since-gc . 186,250,144) > (protected-objects . 28) > (gc-times . 464)) > real 24m36.643s > > * guix/self.scm: Split building of directories into 26 chunks; print gc-s= tats. > (heap-size . 1,131,298,816) > > * guix/self.scm: Split building of directories into 26 chunks; no gc; pri= nt gc-stats. > (heap-size . 1,121,116,160) > > * guix/self.scm: Chunks of 25 files; run gc; print gc-stats. > (heap-size . 1,066,725,376) > > * guix/self.scm: Chunks of 50 files; no gc; print gc-stats. > (heap-size . 1,299,230,720) > real 26m40.708s > > * guix/self.scm: Chunks of 25 files; no gc; print gc-stats. > (heap-size . 1,024,045,056) ; 1st run > real 28m4.451s > > * guix/self.scm: Chunks of 10 files; no gc; print gc-stats. > (heap-size . 1,077,895,168) > real 30m14.049s > > ...strangely enough, if we assume that these statistics translate to the > Hurd, using chunks of max 25 files seems to be a sort of sweet spot? > 25% less peak memory (~300MB), "only" 12% (3"45') slower... though not > great for GNU/Linux users... > > I have produced a handful of successful `guix pull's (from a local > checked-out worktree) using the 26-way split and chunks of max-25 files > patches, but sadly also many more attempts failed. Initially, when > creating this patch series, I was convinced this fixed building on the > Hurd, but I'm much less enthusiastic now. > > So I still have a slight preference for using the latest max-25-files > patch, but I'm sorry to say that I cannot back it up with tangible data. > All in all a rather discouraging week with much effort spent for little > gain. Hopefully Josselin can do some of his magic here :) Anyway, I've finally --8<---------------cut here---------------start------------->8--- --8<---------------cut here---------------end--------------->8--- managed to get all remaining necessary dependencies built to run `guix system reconfigure' on the Hurd*. The only patch that we need that's not on master yet is this one. I'm still rooting for Josselin's graphml cycle analysis and/or lechner Felix's bespoke make rewrite will render this hack obsolete real soon now... If there are no objections I'll go forward and install this patch by the end of the weekend. Greetings, Janneke *) using the hurd-team branch, grub install still fails, apparently because "someone" imagines /dev/wd0 to be /dev/hd0, but the new system derivation is being enabled. --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com From unknown Fri Aug 15 20:27:28 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Janneke Nieuwenhuizen Subject: bug#65456: closed (Re: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.) Message-ID: References: <87led4m6ak.fsf@gnu.org> X-Gnu-PR-Message: they-closed 65456 X-Gnu-PR-Package: guix Reply-To: 65456@debbugs.gnu.org Date: Mon, 18 Sep 2023 04:54:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1695012842-4384-1" This is a multi-part message in MIME format... ------------=_1695012842-4384-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. which was filed against the guix package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 65456@debbugs.gnu.org. --=20 65456: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D65456 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1695012842-4384-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 65456-done) by debbugs.gnu.org; 18 Sep 2023 04:53:15 +0000 Received: from localhost ([127.0.0.1]:51847 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qi6G3-00017c-1Y for submit@debbugs.gnu.org; Mon, 18 Sep 2023 00:53:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38398) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qi6G0-00017A-H1 for 65456-done@debbugs.gnu.org; Mon, 18 Sep 2023 00:53:14 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qi6Fl-0004Aj-PX; Mon, 18 Sep 2023 00:52:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=JMz9nrAGyNO10MvkL0zwpAmrN1tnku52wP1Q7r8CP04=; b=BJUorqVhQQLUlHgKdDyT H2JuQkRBx35dR6yRGeGD7Q8JTmPEQqxk69cvxTaIFXbeVAoZlov3lCZ1CILGkAzLomxmxMU2LZBfa DCFog5MXt+Zsc59p24JWoynh4IhKcCwdlkL14DjAOzILXpqIxgV/42l1J28RKyJppFIdD/nKXKiKZ MHLN0UAzgm4tUDKo8u1jlqT2A4XXyYpH0aEh32jJvdE4HoR/Cp02CFLI9NXNez40v9rvh+D2AXY0X lPMRPnLeZAJR6UVvHQYEAguXHApgANnSiwKUOWmGgI8UGOyzasavHI7cR0DoG3AXlC3rsKvAfMkOV zIJBeSwbkiRnEw==; From: Janneke Nieuwenhuizen To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Organization: AvatarAcademy.nl References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> <87h6oq6qpg.fsf_-_@gnu.org> <87zg2gy013.fsf_-_@gnu.org> <87h6oeys7t.fsf@verum.com> <87led6ywqb.fsf@gnu.org> X-Url: http://AvatarAcademy.nl Date: Mon, 18 Sep 2023 06:52:51 +0200 In-Reply-To: <87led6ywqb.fsf@gnu.org> (Janneke Nieuwenhuizen's message of "Sat, 16 Sep 2023 17:16:28 +0200") Message-ID: <87led4m6ak.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 65456-done Cc: Josselin Poiret , Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , 65456-done@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Janneke Nieuwenhuizen writes: Hi! > Janneke Nieuwenhuizen writes: > >> Ludovic Court=C3=A8s writes: > If there are no objections I'll go forward and install this patch by the > end of the weekend. So, I've been running guix pull from hurd-team on the Hurd all day. Starting with the max-25 files split without (gc), it failed 5 consecutive times building guix-packages-base. Because the initial patch worked pretty well and had the (gc) call (and yes, also used the 26-way split), I re-added (gc), and now the fourth attempt succeeded. After `guix-packages-base' built, `guix-packages' then built right away. I seem to remember guix-packages was the main problem when I wrote the first version of this patch some four months ago, but that one split the list of packages 26-way of course. That suggests that the (combined) compilation of certain files is especially problematic, or that it has since become more problematic (). On the other hand, we may be able to use more than 4GiB of memory on the Hurd some time soon (and after we upgrade and compile using --enable-pae) . Pushed to master as 658de25e990a56265e2398a3737e9cf1eb57af68 Greetings, Janneke --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com ------------=_1695012842-4384-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 22 Aug 2023 17:18:18 +0000 Received: from localhost ([127.0.0.1]:60402 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV1F-0008Gx-S4 for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:18:18 -0400 Received: from lists.gnu.org ([2001:470:142::17]:37640) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYV1C-0008Gj-70 for submit@debbugs.gnu.org; Tue, 22 Aug 2023 13:18:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYV13-000130-HS for bug-guix@gnu.org; Tue, 22 Aug 2023 13:18:05 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYV12-0005UE-RR; Tue, 22 Aug 2023 13:18:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=MXGP6JnLn55nxGu/RKVZvRSzD14tw//JK6nmQvWbg9I=; b=WjXuKUaiepc7MV tDUtENMXkMHbiRmQFNtusOAjJzKCVtYcMbAGjaqrRceB9nNQuEo9rJLAbgm+/dsgl3XuU5kq7byLM FxnE2gNSC7vmOBvn4+zsJ04utBdDENMrhoDzB8F7SjRwFHogYR/mhuToUbLFXUhbmtlQNtQMLP7IG gGJW9tJyZzg4qNaxQ40Wcmg8Fwk5T2QkbyfFjXAkRlipvXgpbgMUJHzHzHTez4O3OTzs1tz6N342B HIHQm8A5bxyt4lWl+ayf+tANwxVTpIDOUr/QjQG/8DMzc8rnF2Y7eawxiJYSeo5HIEnZdFI39CaoJ jvH27LwYBD2NjbaWL+7g==; From: Janneke Nieuwenhuizen To: bug-guix@gnu.org Subject: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Date: Tue, 22 Aug 2023 19:17:59 +0200 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi! Initially writter for the Hurd (see hurd-team , ) This afternoon on IRC, Maxim confirmed my suspicion that this could be 32bit issue, rather than a Hurd problem (, ). I didn't submit these for the Hurd yet, as I didn't really feel comfortable with the 26-way split. I tried a 5-way split and that still gave OOM errors on the Hurd and I didn't feel like hand-coding a 10-way split, so yeah. Also, the 5-way split in the Makefile.am will produce percentages over 100 which suggests a typo...but I couldn't spot it. Lastly, these patches feel like a workaround for Guile / libgc memory management? Otoh, being able to build Guix on 32bit hosts is kinda nice too... So yeah. Greetings, Janneke Janneke Nieuwenhuizen (2): build: Build gnu/packages/*.go in five steps. self: Build gnu/packages/*.go in 26 steps on 32bit. Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++------- guix/self.scm | 31 ++++++++++++++++++++++++-- 2 files changed, 83 insertions(+), 10 deletions(-) base-commit: c655231b72ac28b5a433069fcf86a835c9c83691 -- 2.41.0 ------------=_1695012842-4384-1--