GNU bug report logs - #74654
[PATCH 0/5] Optimize 'all-packages'; add ungrafting manifest

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 74654 in the body.
You can then email your comments to 74654 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 0/5] Optimize 'all-packages'; add ungrafting manifest
Date: Mon,  2 Dec 2024 17:51:53 +0100

Hello Guix!

This is a followup to <https://issues.guix.gnu.org/74542>:

  • Optimizing ‘all-packages’ significantly (speeds up
    ‘guix build -P’ and ‘guix refresh -l’);

  • Moving manifests to ‘etc/manifests’ (we’ll have to
    update Cuirass specs that refer to these files);

  • Adding an “ungrafting” package that rebuilds everything
    without grafts, just like what we’d do manually.

The goal is to have that ungrafting manifest under CI, such
that anytime the corresponding jobset is green, we can ungraft
everything with the guarantee that binaries are available.

Thoughts?

Ludo’.

Ludovic Courtès (5):
  etc: Move manifests to a separate directory.
  packages: Optimize ‘all-packages’.
  build-system/cargo: Simplify ‘crate-closure’.
  guix build: Last argument of ‘dependents’ is optional.
  maint: Add ungrafting manifest.

 Makefile.am                                   | 21 ++++----
 .../disarchive.scm}                           |  2 +-
 etc/{hurd-manifest.scm => manifests/hurd.scm} |  0
 .../kernels.scm}                              |  0
 .../release.scm}                              |  0
 .../source.scm}                               |  0
 etc/{ => manifests}/system-tests.scm          |  0
 .../time-travel.scm}                          |  0
 etc/manifests/ungraft.scm                     | 49 +++++++++++++++++++
 .../upgrade.scm}                              |  0
 gnu/packages.scm                              | 30 +++++++-----
 guix/build-system/cargo.scm                   | 12 ++---
 guix/scripts/build.scm                        |  4 +-
 13 files changed, 87 insertions(+), 31 deletions(-)
 rename etc/{disarchive-manifest.scm => manifests/disarchive.scm} (99%)
 rename etc/{hurd-manifest.scm => manifests/hurd.scm} (100%)
 rename etc/{kernels-manifest.scm => manifests/kernels.scm} (100%)
 rename etc/{release-manifest.scm => manifests/release.scm} (100%)
 rename etc/{source-manifest.scm => manifests/source.scm} (100%)
 rename etc/{ => manifests}/system-tests.scm (100%)
 rename etc/{time-travel-manifest.scm => manifests/time-travel.scm} (100%)
 create mode 100644 etc/manifests/ungraft.scm
 rename etc/{upgrade-manifest.scm => manifests/upgrade.scm} (100%)


base-commit: 522732d5c15e44fc9e061f36a41f7129edfee66f
-- 
2.46.0

Message #11 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 74654 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 1/5] etc: Move manifests to a separate directory.
Date: Mon,  2 Dec 2024 17:53:26 +0100

* etc/disarchive-manifest.scm, etc/hurd-manifest.scm,
etc/kernels-manifest.scm, etc/release-manifest.scm,
etc/source-manifest.scm, etc/system-tests.scm,
etc/time-travel-manifest.scm, etc/upgrade-manifest.scm: Move to…
* etc/manifests: … here, and drop “-manifest” from file name.
* Makefile.am (EXTRA_DIST, assert-binaries-available, check-system):
Adjust accordingly.

Change-Id: Iedee3d0cdd42e72ef8bbf654ea5d3b47dca95874
---
 Makefile.am                                   | 20 +++++++++----------
 .../disarchive.scm}                           |  2 +-
 etc/{hurd-manifest.scm => manifests/hurd.scm} |  0
 .../kernels.scm}                              |  0
 .../release.scm}                              |  0
 .../source.scm}                               |  0
 etc/{ => manifests}/system-tests.scm          |  0
 .../time-travel.scm}                          |  0
 .../upgrade.scm}                              |  0
 9 files changed, 11 insertions(+), 11 deletions(-)
 rename etc/{disarchive-manifest.scm => manifests/disarchive.scm} (99%)
 rename etc/{hurd-manifest.scm => manifests/hurd.scm} (100%)
 rename etc/{kernels-manifest.scm => manifests/kernels.scm} (100%)
 rename etc/{release-manifest.scm => manifests/release.scm} (100%)
 rename etc/{source-manifest.scm => manifests/source.scm} (100%)
 rename etc/{ => manifests}/system-tests.scm (100%)
 rename etc/{time-travel-manifest.scm => manifests/time-travel.scm} (100%)
 rename etc/{upgrade-manifest.scm => manifests/upgrade.scm} (100%)

diff --git a/Makefile.am b/Makefile.am
index 0cff32c607..8c763c68aa 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -686,7 +686,7 @@ endif !CAN_RUN_TESTS
 
 check-system: $(GOBJECTS)
 	$(AM_V_at)$(top_builddir)/pre-inst-env			\
-	  guix build -m $(top_srcdir)/etc/system-tests.scm -K
+	  guix build -m $(top_srcdir)/etc/manifests/system-tests.scm -K
 
 # Public keys used to sign substitutes.
 dist_pkgdata_DATA =				\
@@ -733,17 +733,17 @@ EXTRA_DIST +=						\
   build-aux/update-guix-package.scm			\
   build-aux/xgettext.scm				\
   doc/build.scm						\
-  etc/disarchive-manifest.scm				\
   etc/guix-install.sh					\
   etc/historical-authorizations				\
   etc/news.scm						\
-  etc/hurd-manifest.scm					\
-  etc/kernels-manifest.scm				\
-  etc/release-manifest.scm				\
-  etc/source-manifest.scm				\
-  etc/system-tests.scm					\
-  etc/time-travel-manifest.scm				\
-  etc/upgrade-manifest.scm				\
+  etc/manifests/disarchive.scm				\
+  etc/manifests/hurd.scm				\
+  etc/manifests/kernels.scm				\
+  etc/manifests/release.scm				\
+  etc/manifests/source.scm				\
+  etc/manifests/system-tests.scm			\
+  etc/manifests/time-travel.scm				\
+  etc/manifests/upgrade.scm				\
   scripts/guix.in					\
   tests/cve-sample.json					\
   tests/keys/civodul.pub				\
@@ -1194,7 +1194,7 @@ assert-no-store-file-names:
 # server so that '--display-missing' doesn't print two lists.
 assert-binaries-available: $(GOBJECTS)
 	$(AM_V_at)$(top_builddir)/pre-inst-env				\
-	  guix weather -m "$(top_srcdir)/etc/release-manifest.scm"	\
+	  guix weather -m "$(top_srcdir)/etc/manifests/release.scm"	\
 	                --substitute-urls="https://ci.guix.gnu.org"	\
 	                --display-missing
 
diff --git a/etc/disarchive-manifest.scm b/etc/manifests/disarchive.scm
similarity index 99%
rename from etc/disarchive-manifest.scm
rename to etc/manifests/disarchive.scm
index 3dbfa356df..a7f71414b6 100644
--- a/etc/disarchive-manifest.scm
+++ b/etc/manifests/disarchive.scm
@@ -24,7 +24,7 @@
              (guix base16)
              (gnu packages))
 
-(include "source-manifest.scm")
+(include "source.scm")
 
 (define (tarball-origin? origin)
   (match (origin-actual-file-name origin)
diff --git a/etc/hurd-manifest.scm b/etc/manifests/hurd.scm
similarity index 100%
rename from etc/hurd-manifest.scm
rename to etc/manifests/hurd.scm
diff --git a/etc/kernels-manifest.scm b/etc/manifests/kernels.scm
similarity index 100%
rename from etc/kernels-manifest.scm
rename to etc/manifests/kernels.scm
diff --git a/etc/release-manifest.scm b/etc/manifests/release.scm
similarity index 100%
rename from etc/release-manifest.scm
rename to etc/manifests/release.scm
diff --git a/etc/source-manifest.scm b/etc/manifests/source.scm
similarity index 100%
rename from etc/source-manifest.scm
rename to etc/manifests/source.scm
diff --git a/etc/system-tests.scm b/etc/manifests/system-tests.scm
similarity index 100%
rename from etc/system-tests.scm
rename to etc/manifests/system-tests.scm
diff --git a/etc/time-travel-manifest.scm b/etc/manifests/time-travel.scm
similarity index 100%
rename from etc/time-travel-manifest.scm
rename to etc/manifests/time-travel.scm
diff --git a/etc/upgrade-manifest.scm b/etc/manifests/upgrade.scm
similarity index 100%
rename from etc/upgrade-manifest.scm
rename to etc/manifests/upgrade.scm
-- 
2.46.0

Message #14 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 74654 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 2/5] packages: Optimize ‘all-packages’.
Date: Mon,  2 Dec 2024 17:53:27 +0100

On my laptop, wall-clock time for (all-packages) goes from 27s to 1s.

* gnu/packages.scm (all-packages): Use a hash table to remember visited
packages instead of calling ‘delete-duplicates’ on the final list.

Change-Id: I4aae804656b56ef2095993e91f0572a5891f419f
---
 gnu/packages.scm | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)

diff --git a/gnu/packages.scm b/gnu/packages.scm
index 1af3b8d440..bdd5d21940 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -258,18 +258,26 @@ (define all-packages
   (mlambda ()
     "Return the list of all public packages, including replacements and hidden
 packages, excluding superseded packages."
-    (delete-duplicates
-     (fold-packages (lambda (package result)
-                      (match (package-replacement package)
-                        ((? package? replacement)
-                         (cons* replacement package result))
-                        (#f
-                         (cons package result))))
-                    '()
+    ;; Note: 'fold-packages' never traverses the same package twice but
+    ;; replacements break that (they may or may not be visible to
+    ;; 'fold-packages'), hence this hash table to track visited packages.
+    (define visited (make-hash-table))
 
-                    ;; Dismiss deprecated packages but keep hidden packages.
-                    #:select? (negate package-superseded))
-     eq?)))
+    (fold-packages (lambda (package result)
+                     (if (hashq-ref visited package)
+                         result
+                         (begin
+                           (hashq-set! visited package #t)
+                           (match (package-replacement package)
+                             ((? package? replacement)
+                              (hashq-set! visited replacement #t)
+                              (cons* replacement package result))
+                             (#f
+                              (cons package result))))))
+                   '()
+
+                   ;; Dismiss deprecated packages but keep hidden packages.
+                   #:select? (negate package-superseded))))
 
 (define %package-cache-file
   ;; Location of the package cache.
-- 
2.46.0

Message #17 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 74654 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 3/5] build-system/cargo: Simplify ‘crate-closure’.
Date: Mon,  2 Dec 2024 17:53:28 +0100

* guix/build-system/cargo.scm (crate-closure): Remove ‘first?’.

Change-Id: Ica1a661eb422a882ae3a1a54819b2dd7d68449de
---
 guix/build-system/cargo.scm | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm
index 658a2e525e..0e9a4b1d23 100644
--- a/guix/build-system/cargo.scm
+++ b/guix/build-system/cargo.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2013, 2014, 2015, 2016, 2019, 2021 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2013-2016, 2019, 2021, 2024 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2013 Andreas Enge <andreas <at> enge.fr>
 ;;; Copyright © 2013 Nikita Karetnikov <nikita <at> karetnikov.org>
 ;;; Copyright © 2016 David Craven <david <at> craven.ch>
@@ -227,24 +227,22 @@ (define (crate-closure inputs)
   (let loop ((inputs     inputs)
              (result     '())
              (propagated '())
-             (first?     #t)
              (seen       vlist-null))
     (match inputs
       (()
        (if (null? propagated)
            (reverse result)
-           (loop (reverse (concatenate propagated)) result '() #f seen)))
+           (loop (reverse (concatenate propagated)) result '() seen)))
       (((and input (label (? package? package))) rest ...)
-       (if (and (not first?) (seen? seen package))
-           (loop rest result propagated first? seen)
+       (if (seen? seen package)
+           (loop rest result propagated seen)
            (loop rest
                  (cons input result)
                  (cons (package-cargo-inputs package)
                        propagated)
-                 first?
                  (vhash-consq package package seen))))
       ((input rest ...)
-       (loop rest (cons input result) propagated first? seen)))))
+       (loop rest (cons input result) propagated seen)))))
 
 (define (expand-crate-sources cargo-inputs cargo-development-inputs)
   "Extract all transitive sources for CARGO-INPUTS and CARGO-DEVELOPMENT-INPUTS
-- 
2.46.0

Message #20 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 74654 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 4/5] guix build: Last argument of ‘dependents’ is optional.
Date: Mon,  2 Dec 2024 17:53:29 +0100

* guix/scripts/build.scm (dependents): Make ‘max-depth’ optional.
Improve docstring.

Change-Id: Ibeffcf15297ecba3d4000b3adb4e3bc7853f6957
---
 guix/scripts/build.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 0080f1fe58..f0a637a2ef 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -564,8 +564,8 @@ (define %options
                  %standard-cross-build-options
                  %standard-native-build-options)))
 
-(define (dependents store packages max-depth)
-  "List all the things that would need to be rebuilt if PACKAGES are changed."
+(define* (dependents store packages #:optional (max-depth +inf.0))
+  "Return the list of dependents of all of PACKAGES up to distance MAX-DEPTH."
   ;; Using %BAG-NODE-TYPE is more accurate than using %PACKAGE-NODE-TYPE
   ;; because it includes implicit dependencies.
   (define (get-dependents packages edges)
-- 
2.46.0

Message #23 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 74654 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 5/5] maint: Add ungrafting manifest.
Date: Mon,  2 Dec 2024 17:53:30 +0100

* etc/manifests/ungraft.scm: New file.
* Makefile.am (EXTRA_DIST): Add it.

Change-Id: If4aa7649721b93d3297ecd967d5db4a6b71ec6eb
---
 Makefile.am               |  1 +
 etc/manifests/ungraft.scm | 49 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+)
 create mode 100644 etc/manifests/ungraft.scm

diff --git a/Makefile.am b/Makefile.am
index 8c763c68aa..d0d195636b 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -743,6 +743,7 @@ EXTRA_DIST +=						\
   etc/manifests/source.scm				\
   etc/manifests/system-tests.scm			\
   etc/manifests/time-travel.scm				\
+  etc/manifests/ungraft.scm				\
   etc/manifests/upgrade.scm				\
   scripts/guix.in					\
   tests/cve-sample.json					\
diff --git a/etc/manifests/ungraft.scm b/etc/manifests/ungraft.scm
new file mode 100644
index 0000000000..3e42b98ece
--- /dev/null
+++ b/etc/manifests/ungraft.scm
@@ -0,0 +1,49 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Ludovic Courtès <ludo <at> gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;; This manifest "ungrafts" all the currently grafted packages and returns
+;; said packages and all their dependents.
+
+(use-modules (guix diagnostics)
+             (guix i18n)
+             (guix packages)
+             (guix profiles)
+             (guix store)
+             ((guix scripts build) #:select (dependents))
+             ((gnu packages) #:select (all-packages))
+             (srfi srfi-1))
+
+(define (grafted-packages)
+  (info (G_ "enumerating grafted packages...~%"))
+  (let ((result (filter package-replacement (all-packages))))
+    (info (G_ "found ~d grafted packages:~{ ~a~}~%")
+          (length result) (map package-full-name result))
+    result))
+
+(manifest
+ (with-store store
+   (let* ((grafted (grafted-packages))
+          (ungraft-all (package-input-rewriting
+                        (map (lambda (package)
+                               `(,package . ,(package-replacement package)))
+                             grafted))))
+     (map (lambda (package)
+            (manifest-entry
+              (inherit (package->manifest-entry (ungraft-all package)))
+              (name (string-append (package-name package) "-ungrafted"))))
+          (dependents store grafted)))))
-- 
2.46.0

Message #26 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 74654 <at> debbugs.gnu.org
Subject: Re: [bug#74654] [PATCH 3/5] build-system/cargo: Simplify ‘crate-closure’.
Date: Mon, 2 Dec 2024 21:11:56 +0200

[Message part 1 (text/plain, inline)]

I test applied this on master and then went to build my rusty-packages
manifest. The time on './pre-inst-env guix build -m ... -n' went from 35
seconds to 20 seconds and the derivations didn't change.

On Mon, Dec 02, 2024 at 05:53:28PM +0100, Ludovic Courtès wrote:
> * guix/build-system/cargo.scm (crate-closure): Remove ‘first?’.
> 
> Change-Id: Ica1a661eb422a882ae3a1a54819b2dd7d68449de
> ---
>  guix/build-system/cargo.scm | 12 +++++-------
>  1 file changed, 5 insertions(+), 7 deletions(-)
> 
> diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm
> index 658a2e525e..0e9a4b1d23 100644
> --- a/guix/build-system/cargo.scm
> +++ b/guix/build-system/cargo.scm
> @@ -1,5 +1,5 @@
>  ;;; GNU Guix --- Functional package management for GNU
> -;;; Copyright © 2013, 2014, 2015, 2016, 2019, 2021 Ludovic Courtès <ludo <at> gnu.org>
> +;;; Copyright © 2013-2016, 2019, 2021, 2024 Ludovic Courtès <ludo <at> gnu.org>
>  ;;; Copyright © 2013 Andreas Enge <andreas <at> enge.fr>
>  ;;; Copyright © 2013 Nikita Karetnikov <nikita <at> karetnikov.org>
>  ;;; Copyright © 2016 David Craven <david <at> craven.ch>
> @@ -227,24 +227,22 @@ (define (crate-closure inputs)
>    (let loop ((inputs     inputs)
>               (result     '())
>               (propagated '())
> -             (first?     #t)
>               (seen       vlist-null))
>      (match inputs
>        (()
>         (if (null? propagated)
>             (reverse result)
> -           (loop (reverse (concatenate propagated)) result '() #f seen)))
> +           (loop (reverse (concatenate propagated)) result '() seen)))
>        (((and input (label (? package? package))) rest ...)
> -       (if (and (not first?) (seen? seen package))
> -           (loop rest result propagated first? seen)
> +       (if (seen? seen package)
> +           (loop rest result propagated seen)
>             (loop rest
>                   (cons input result)
>                   (cons (package-cargo-inputs package)
>                         propagated)
> -                 first?
>                   (vhash-consq package package seen))))
>        ((input rest ...)
> -       (loop rest (cons input result) propagated first? seen)))))
> +       (loop rest (cons input result) propagated seen)))))
>  
>  (define (expand-crate-sources cargo-inputs cargo-development-inputs)
>    "Extract all transitive sources for CARGO-INPUTS and CARGO-DEVELOPMENT-INPUTS
> -- 
> 2.46.0
> 
> 
> 

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 74654 <at> debbugs.gnu.org
Subject: Re: [bug#74654] [PATCH 3/5] build-system/cargo: Simplify
 ‘crate-closure’.
Date: Mon, 02 Dec 2024 23:24:58 +0100

Efraim Flashner <efraim <at> flashner.co.il> skribis:

> I test applied this on master and then went to build my rusty-packages
> manifest. The time on './pre-inst-env guix build -m ... -n' went from 35
> seconds to 20 seconds and the derivations didn't change.

Thanks for testing!  I didn’t expect this to have an impact on
performance, interesting.

Ludo’.

Message #32 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: David Elsing <david.elsing <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 74654 <at> debbugs.gnu.org
Subject: Re: [PATCH 2/5] packages: Optimize ‘all-packages’.
Date: Mon, 02 Dec 2024 23:47:48 +0000

Hi,

Ludovic Courtès <ludo <at> gnu.org> writes:

> On my laptop, wall-clock time for (all-packages) goes from 27s to 1s.
>
> * gnu/packages.scm (all-packages): Use a hash table to remember visited
> packages instead of calling ‘delete-duplicates’ on the final list.

Thanks for fixing this, commit 4b5dae8 also made
`guix graph -t reverse-package` much slower, because `delete-duplicates`
had not been used in the old `all-packages` procedure before. I wanted
to make a patch, but then saw your series already. :)

Cheers,
David

Message #35 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Z572 <zhengjunjie <at> iscas.ac.cn>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 74654 <at> debbugs.gnu.org
Subject: Re: [bug#74654] [PATCH 1/5] etc: Move manifests to a separate
 directory.
Date: Tue, 03 Dec 2024 11:05:15 +0800

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> * etc/disarchive-manifest.scm, etc/hurd-manifest.scm,
> etc/kernels-manifest.scm, etc/release-manifest.scm,
> etc/source-manifest.scm, etc/system-tests.scm,
> etc/time-travel-manifest.scm, etc/upgrade-manifest.scm: Move to…
> * etc/manifests: … here, and drop “-manifest” from file name.
> * Makefile.am (EXTRA_DIST, assert-binaries-available, check-system):
> Adjust accordingly.

Once these files are moved, it may be necessary to mention them in
etc/news.scm to inform others that they need to update their cuirass
specs.

>
> Change-Id: Iedee3d0cdd42e72ef8bbf654ea5d3b47dca95874
> ---
>  Makefile.am                                   | 20 +++++++++----------
>  .../disarchive.scm}                           |  2 +-
>  etc/{hurd-manifest.scm => manifests/hurd.scm} |  0
>  .../kernels.scm}                              |  0
>  .../release.scm}                              |  0
>  .../source.scm}                               |  0
>  etc/{ => manifests}/system-tests.scm          |  0
>  .../time-travel.scm}                          |  0
>  .../upgrade.scm}                              |  0
>  9 files changed, 11 insertions(+), 11 deletions(-)
>  rename etc/{disarchive-manifest.scm => manifests/disarchive.scm} (99%)
>  rename etc/{hurd-manifest.scm => manifests/hurd.scm} (100%)
>  rename etc/{kernels-manifest.scm => manifests/kernels.scm} (100%)
>  rename etc/{release-manifest.scm => manifests/release.scm} (100%)
>  rename etc/{source-manifest.scm => manifests/source.scm} (100%)
>  rename etc/{ => manifests}/system-tests.scm (100%)
>  rename etc/{time-travel-manifest.scm => manifests/time-travel.scm} (100%)
>  rename etc/{upgrade-manifest.scm => manifests/upgrade.scm} (100%)
>
> diff --git a/Makefile.am b/Makefile.am
> index 0cff32c607..8c763c68aa 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -686,7 +686,7 @@ endif !CAN_RUN_TESTS
>  
>  check-system: $(GOBJECTS)
>  	$(AM_V_at)$(top_builddir)/pre-inst-env			\
> -	  guix build -m $(top_srcdir)/etc/system-tests.scm -K
> +	  guix build -m $(top_srcdir)/etc/manifests/system-tests.scm -K
>  
>  # Public keys used to sign substitutes.
>  dist_pkgdata_DATA =				\
> @@ -733,17 +733,17 @@ EXTRA_DIST +=						\
>    build-aux/update-guix-package.scm			\
>    build-aux/xgettext.scm				\
>    doc/build.scm						\
> -  etc/disarchive-manifest.scm				\
>    etc/guix-install.sh					\
>    etc/historical-authorizations				\
>    etc/news.scm						\
> -  etc/hurd-manifest.scm					\
> -  etc/kernels-manifest.scm				\
> -  etc/release-manifest.scm				\
> -  etc/source-manifest.scm				\
> -  etc/system-tests.scm					\
> -  etc/time-travel-manifest.scm				\
> -  etc/upgrade-manifest.scm				\
> +  etc/manifests/disarchive.scm				\
> +  etc/manifests/hurd.scm				\
> +  etc/manifests/kernels.scm				\
> +  etc/manifests/release.scm				\
> +  etc/manifests/source.scm				\
> +  etc/manifests/system-tests.scm			\
> +  etc/manifests/time-travel.scm				\
> +  etc/manifests/upgrade.scm				\
>    scripts/guix.in					\
>    tests/cve-sample.json					\
>    tests/keys/civodul.pub				\
> @@ -1194,7 +1194,7 @@ assert-no-store-file-names:
>  # server so that '--display-missing' doesn't print two lists.
>  assert-binaries-available: $(GOBJECTS)
>  	$(AM_V_at)$(top_builddir)/pre-inst-env				\
> -	  guix weather -m "$(top_srcdir)/etc/release-manifest.scm"	\
> +	  guix weather -m "$(top_srcdir)/etc/manifests/release.scm"	\
>  	                --substitute-urls="https://ci.guix.gnu.org"	\
>  	                --display-missing
>  
> diff --git a/etc/disarchive-manifest.scm b/etc/manifests/disarchive.scm
> similarity index 99%
> rename from etc/disarchive-manifest.scm
> rename to etc/manifests/disarchive.scm
> index 3dbfa356df..a7f71414b6 100644
> --- a/etc/disarchive-manifest.scm
> +++ b/etc/manifests/disarchive.scm
> @@ -24,7 +24,7 @@
>               (guix base16)
>               (gnu packages))
>  
> -(include "source-manifest.scm")
> +(include "source.scm")
>  
>  (define (tarball-origin? origin)
>    (match (origin-actual-file-name origin)
> diff --git a/etc/hurd-manifest.scm b/etc/manifests/hurd.scm
> similarity index 100%
> rename from etc/hurd-manifest.scm
> rename to etc/manifests/hurd.scm
> diff --git a/etc/kernels-manifest.scm b/etc/manifests/kernels.scm
> similarity index 100%
> rename from etc/kernels-manifest.scm
> rename to etc/manifests/kernels.scm
> diff --git a/etc/release-manifest.scm b/etc/manifests/release.scm
> similarity index 100%
> rename from etc/release-manifest.scm
> rename to etc/manifests/release.scm
> diff --git a/etc/source-manifest.scm b/etc/manifests/source.scm
> similarity index 100%
> rename from etc/source-manifest.scm
> rename to etc/manifests/source.scm
> diff --git a/etc/system-tests.scm b/etc/manifests/system-tests.scm
> similarity index 100%
> rename from etc/system-tests.scm
> rename to etc/manifests/system-tests.scm
> diff --git a/etc/time-travel-manifest.scm b/etc/manifests/time-travel.scm
> similarity index 100%
> rename from etc/time-travel-manifest.scm
> rename to etc/manifests/time-travel.scm
> diff --git a/etc/upgrade-manifest.scm b/etc/manifests/upgrade.scm
> similarity index 100%
> rename from etc/upgrade-manifest.scm
> rename to etc/manifests/upgrade.scm

[signature.asc (application/pgp-signature, inline)]

Message #38 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Z572 <zhengjunjie <at> iscas.ac.cn>
Cc: 74654 <at> debbugs.gnu.org
Subject: Re: [bug#74654] [PATCH 1/5] etc: Move manifests to a separate
 directory.
Date: Wed, 04 Dec 2024 17:32:36 +0100

Hi,

Z572 <zhengjunjie <at> iscas.ac.cn> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> * etc/disarchive-manifest.scm, etc/hurd-manifest.scm,
>> etc/kernels-manifest.scm, etc/release-manifest.scm,
>> etc/source-manifest.scm, etc/system-tests.scm,
>> etc/time-travel-manifest.scm, etc/upgrade-manifest.scm: Move to…
>> * etc/manifests: … here, and drop “-manifest” from file name.
>> * Makefile.am (EXTRA_DIST, assert-binaries-available, check-system):
>> Adjust accordingly.
>
> Once these files are moved, it may be necessary to mention them in
> etc/news.scm to inform others that they need to update their cuirass
> specs.

Do you think they are used outside the Guix project?  They’re not
documented and not meant for general consumption.

Ludo’.

Message #41 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: David Elsing <david.elsing <at> posteo.net>
Cc: 74654 <at> debbugs.gnu.org
Subject: Re: [bug#74654] [PATCH 2/5] packages: Optimize
 ‘all-packages’.
Date: Wed, 04 Dec 2024 17:34:01 +0100

Hello,

David Elsing <david.elsing <at> posteo.net> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> On my laptop, wall-clock time for (all-packages) goes from 27s to 1s.
>>
>> * gnu/packages.scm (all-packages): Use a hash table to remember visited
>> packages instead of calling ‘delete-duplicates’ on the final list.
>
> Thanks for fixing this, commit 4b5dae8 also made
> `guix graph -t reverse-package` much slower, because `delete-duplicates`
> had not been used in the old `all-packages` procedure before. I wanted
> to make a patch, but then saw your series already. :)

Yes, my bad!  I tried to take the common denominator of all the
‘all-packages’ procedures, and here’s what happens.  :-)

Ludo’.

Message #44 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>, 74654 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: [bug#74654] [PATCH 2/5] packages: Optimize
 ‘all-packages’.
Date: Tue, 10 Dec 2024 15:05:57 +0100

Hi,

On Mon, 02 Dec 2024 at 17:53, Ludovic Courtès <ludo <at> gnu.org> wrote:

> -    (delete-duplicates
> -     (fold-packages (lambda (package result)
> -                      (match (package-replacement package)
> -                        ((? package? replacement)
> -                         (cons* replacement package result))
> -                        (#f
> -                         (cons package result))))
> -                    '()
> +    ;; Note: 'fold-packages' never traverses the same package twice but
> +    ;; replacements break that (they may or may not be visible to
> +    ;; 'fold-packages'), hence this hash table to track visited packages.
> +    (define visited (make-hash-table))
>  
> -                    ;; Dismiss deprecated packages but keep hidden packages.
> -                    #:select? (negate package-superseded))
> -     eq?)))
> +    (fold-packages (lambda (package result)
> +                     (if (hashq-ref visited package)
> +                         result
> +                         (begin
> +                           (hashq-set! visited package #t)
> +                           (match (package-replacement package)
> +                             ((? package? replacement)
> +                              (hashq-set! visited replacement #t)
> +                              (cons* replacement package result))
> +                             (#f
> +                              (cons package result))))))
> +                   '()
> +
> +                   ;; Dismiss deprecated packages but keep hidden packages.
> +                   #:select? (negate package-superseded))))

Oh!  This is a better implementation, indeed. :-)

LGTM.

Cheers,
simon

Message #47 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 74654 <at> debbugs.gnu.org
Subject: Re: [bug#74654] [PATCH 0/5] Optimize 'all-packages'; add ungrafting
 manifest
Date: Fri, 13 Dec 2024 00:06:36 +0100

Ludovic Courtès <ludo <at> gnu.org> skribis:

>   etc: Move manifests to a separate directory.
>   packages: Optimize ‘all-packages’.
>   build-system/cargo: Simplify ‘crate-closure’.
>   guix build: Last argument of ‘dependents’ is optional.
>   maint: Add ungrafting manifest.

Pushed as 5e2daf4b10cdbb7b5b3a7b1a49b0dfdba4346748.

Ludo'.

Message #50 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 74654 <at> debbugs.gnu.org
Subject: Re: [bug#74654] [PATCH 0/5] Optimize 'all-packages'; add ungrafting
 manifest
Date: Fri, 13 Dec 2024 00:08:20 +0100

Ludovic Courtès <ludo <at> gnu.org> skribis:

>   • Adding an “ungrafting” package that rebuilds everything
>     without grafts, just like what we’d do manually.
>
> The goal is to have that ungrafting manifest under CI, such
> that anytime the corresponding jobset is green, we can ungraft
> everything with the guarantee that binaries are available.

I’ve set up a jobset, currently building for i686-linux and x86_64-linux
only:

  https://ci.guix.gnu.org/jobset/ungrafting

Evaluation is still in progress right now.  There are no pending builds
for these architectures (but 100k builds for aarch64-linux…).

Ludo’.

Message #53 received at 74654 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 74654 <at> debbugs.gnu.org
Subject: Re: [bug#74654] [PATCH 0/5] Optimize 'all-packages'; add ungrafting
 manifest
Date: Sat, 14 Dec 2024 00:24:20 +0100

Ludovic Courtès <ludo <at> gnu.org> skribis:

> I’ve set up a jobset, currently building for i686-linux and x86_64-linux
> only:
>
>   https://ci.guix.gnu.org/jobset/ungrafting
>
> Evaluation is still in progress right now.  There are no pending builds
> for these architectures (but 100k builds for aarch64-linux…).

So far it’s a failure, first because of a mistake in how “total
ungrafting” is computed in the manifest (I’ll propose a fix soon), but
more importantly because of references to grafted packages in places
other than package inputs, specifically in ‘rust-ring-0.16-sources’ &
co.: these references escape package rewriting, and thus the
computational ungrafting process doesn’t yield the same result as
manually ungrafting the source.

To be continued…

(I’ve disabled the jobset for now.)

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.