From unknown Fri Aug 15 16:19:52 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#77708 <77708@debbugs.gnu.org> To: bug#77708 <77708@debbugs.gnu.org> Subject: Status: [PATCH] gexp: =?UTF-8?Q?=E2=80=98with-parameters=E2=80=98?= is respected by caches. Reply-To: bug#77708 <77708@debbugs.gnu.org> Date: Fri, 15 Aug 2025 23:19:52 +0000 retitle 77708 [PATCH] gexp: =E2=80=98with-parameters=E2=80=98 is respected = by caches. reassign 77708 guix-patches submitter 77708 David Elsing severity 77708 normal tag 77708 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Thu Apr 10 10:49:37 2025 Received: (at submit) by debbugs.gnu.org; 10 Apr 2025 14:49:37 +0000 Received: from localhost ([127.0.0.1]:46257 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1u2tDk-0001ty-9l for submit@debbugs.gnu.org; Thu, 10 Apr 2025 10:49:37 -0400 Received: from lists.gnu.org ([2001:470:142::17]:55902) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1u2tDh-0001tX-5e for submit@debbugs.gnu.org; Thu, 10 Apr 2025 10:49:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1u2tDN-0001K3-Sv for guix-patches@gnu.org; Thu, 10 Apr 2025 10:49:20 -0400 Received: from mout02.posteo.de ([185.67.36.66]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1u2tDL-000145-1c for guix-patches@gnu.org; Thu, 10 Apr 2025 10:49:13 -0400 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 816FA240101 for ; Thu, 10 Apr 2025 16:49:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1744296547; bh=yGQGCKctIOGjlZMiRfvrJ5j4FOcjHHaHIdMWI6PKZvM=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type: Content-Transfer-Encoding:From; b=TyV++57UVLhdCmBGKK45+fo3U75YWXuP1e0HSeO/1Z4hoo6rHuVck4k8bIovGHCFk hvtWr6HDechZale22gKt4a238TOMdoEVbJDlORdcTOHijKFnmpxHPrMAeuOMl8AMgD oDn1x3IEcVLNRfPuiaiTkDafu5/BzXmKc3MtCynFthyZ7lLArZuncsg8Ot6Y6n1hiI l4xdU3fCbAwuYqWJpcP2VjR9eOfNGhOenDAcmaLNNJTWcpclEmodg73//fUGUgJipF PCp7IjBIW1FVbgsKvq589RoZjuNA7ILN7WAmG5XHMae3iHh5izhbF2zQcCZfIB7mIF Kks+e/YVxtZvg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4ZYN4L6rMzz9rxM; Thu, 10 Apr 2025 16:49:06 +0200 (CEST) From: David Elsing To: guix-patches@gnu.org Subject: [PATCH] =?UTF-8?q?gexp:=20=E2=80=98with-parameters=E2=80=98=20is?= =?UTF-8?q?=20respected=20by=20caches.?= Date: Thu, 10 Apr 2025 14:46:54 +0000 Message-ID: <633ad28c062af23019c7ce7e172cec811065be18.1744296414.git.david.elsing@posteo.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Debbugs-Cc: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, zimon.toutoune@gmail.com, me@tobias.gr Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=185.67.36.66; envelope-from=david.elsing@posteo.net; helo=mout02.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit Cc: David Elsing X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) * guix/gexp.scm (lower-object, lower+expand-object): Use (%parameterized-counter) as additional cache key. (%parameterized-counter): New parameter. (%parameterized-counter-next-value): New variable. (%parameterized-counters): New variable. (add-parameterized-counter): New procedure. (compile-parameterized): Add %parameterized-counter to parameters. * guix/packages.scm (cache!): Use ‘hash-set!‘ instead of ‘hashq-set!‘. Use `(,(scm->pointer package) . ,(%parameterized-counter)) as key. (cached, package->derivation, package->cross-derivation): Use (%parameterized-counter) as additional cache key. * tests/gexp.scm ("with-parameters for custom parameter"): New test. --- As noted by Ludo' [1], several objects dependent on packages (such as derivations or grafts) are cached by the package and do not take parameters (apart from %current-system, %current-target-system and %graft?) into account. To fix that, my idea was to introduce an additional parameter `%parameterized-counter', which uniquely identifies a set of parameters and values in the object and which is used as additional key by the caches. To prevent a collision, the parameters and values are stored in a hash table, which keeps them alive forever. Would it be preferable to use something like a cryptographic hash instead? For `cache!' in (guix packages), I used `(,(scm->pointer package) . ,(%parameterized-counter)) as key together with hash-set! and hash-ref instead of hashq-set! and hashq-ref. Is that OK? [1] https://issues.guix.gnu.org/75879 guix/gexp.scm | 48 +++++++++++++++++++++++++++++++++++++++-------- guix/packages.scm | 22 +++++++++++----------- tests/gexp.scm | 31 ++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+), 19 deletions(-) diff --git a/guix/gexp.scm b/guix/gexp.scm index 8dd746eee0..11e3b5968f 100644 --- a/guix/gexp.scm +++ b/guix/gexp.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2019, 2020 Mathieu Othacehe ;;; Copyright © 2020 Maxim Cournoyer ;;; Copyright © 2021, 2022 Maxime Devos +;;; Copyright © 2025 David Elsing ;;; ;;; This file is part of GNU Guix. ;;; @@ -32,6 +33,7 @@ (define-module (guix gexp) #:use-module (srfi srfi-1) #:use-module (srfi srfi-9) #:use-module (srfi srfi-9 gnu) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -94,6 +96,7 @@ (define-module (guix gexp) with-parameters parameterized? + %parameterized-counter load-path-expression gexp-modules @@ -302,7 +305,7 @@ (define* (lower-object obj (not (derivation? lowered))) (loop lowered) (return lowered))) - obj + obj (%parameterized-counter) system target graft?))))))) (define* (lower+expand-object obj @@ -321,7 +324,7 @@ (define* (lower+expand-object obj (lowered (if (derivation? obj) (return obj) (mcached (lower obj system target) - obj + obj (%parameterized-counter) system target graft?)))) ;; LOWER might return something that needs to be further ;; lowered. @@ -731,13 +734,40 @@ (define-syntax-rule (with-parameters ((param value) ...) body ...) (lambda () body ...))) +;; Counter which uniquely identifies specific parameters and values used for +;; . +(define %parameterized-counter + (make-parameter #f)) + +(define %parameterized-counter-next-value 0) + +(define %parameterized-counters (make-hash-table)) + +;; Add %parameterized-counter to PARAMETERS and its value, +;; which depends on PARAMETERS and VALUES, to PARAMETER-VALUES. +(define (add-parameterized-counter parameters parameter-values) + (let* ((key `(,parameters . ,parameter-values)) + (counter + (match (hash-ref %parameterized-counters key) + (#f + (let ((val %parameterized-counter-next-value)) + (hash-set! %parameterized-counters key val) + (set! %parameterized-counter-next-value (+ val 1)) + val)) + (counter counter)))) + (values + (cons %parameterized-counter parameters) + (cons counter parameter-values)))) + (define-gexp-compiler compile-parameterized compiler => (lambda (parameterized system target) (match (parameterized-bindings parameterized) (((parameters values) ...) - (let ((thunk (parameterized-thunk parameterized)) - (values (map (lambda (thunk) (thunk)) values))) + (let*-values (((parameters values) + (add-parameterized-counter + parameters (map (lambda (thunk) (thunk)) values))) + ((thunk) (parameterized-thunk parameterized))) ;; Install the PARAMETERS for the store monad. (state-with-parameters parameters values ;; Install the PARAMETERS for the dynamic extent of THUNK. @@ -762,11 +792,13 @@ (define-gexp-compiler compile-parameterized expander => (lambda (parameterized lowered output) (match (parameterized-bindings parameterized) (((parameters values) ...) - (let ((fluids (map parameter-fluid parameters)) - (thunk (parameterized-thunk parameterized))) + (let*-values (((parameters values) + (add-parameterized-counter + parameters (map (lambda (thunk) (thunk)) values))) + ((thunk) (parameterized-thunk parameterized))) ;; Install the PARAMETERS for the dynamic extent of THUNK. - (with-fluids* fluids - (map (lambda (thunk) (thunk)) values) + (with-fluids* (map parameter-fluid parameters) + values (lambda () (match (thunk) ((? struct? base) diff --git a/guix/packages.scm b/guix/packages.scm index 18ab23e0aa..1ee456ced2 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -11,7 +11,7 @@ ;;; Copyright © 2022 jgart ;;; Copyright © 2023 Simon Tournier ;;; Copyright © 2024 Janneke Nieuwenhuizen -;;; Copyright © 2024 David Elsing +;;; Copyright © 2024, 2025 David Elsing ;;; ;;; This file is part of GNU Guix. ;;; @@ -57,6 +57,7 @@ (define-module (guix packages) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) #:use-module (srfi srfi-71) + #:use-module (system foreign) #:use-module (rnrs bytevectors) #:use-module (web uri) #:autoload (texinfo) (texi-fragment->stexi) @@ -1689,13 +1690,12 @@ (define (cache! cache package system thunk) SYSTEM." ;; FIXME: This memoization should be associated with the open store, because ;; otherwise it breaks when switching to a different store. - (let ((result (thunk))) - ;; Use `hashq-set!' instead of `hash-set!' because `hash' returns the - ;; same value for all structs (as of Guile 2.0.6), and because pointer - ;; equality is sufficient in practice. - (hashq-set! cache package - `((,system . ,result) - ,@(or (hashq-ref cache package) '()))) + (let ((result (thunk)) + (key `(,(scm->pointer package) . ,(%parameterized-counter)))) + (hash-set! cache key + `((,system . ,result) + ,@(or (hash-ref cache key) + '()))) result)) (define-syntax cached @@ -1828,7 +1828,7 @@ (define (input-graft system) (with-parameters ((%current-system system)) replacement)) (replacement-output output)))) - package output system) + package output (%parameterized-counter) system) (return #f)))) (_ (return #f))))) @@ -2068,7 +2068,7 @@ (define* (package->derivation package #:system system #:guile guile))))) (return drv))) - package system #f graft?)) + package (%parameterized-counter) system #f graft?)) (define* (package->cross-derivation package target #:optional (system (%current-system)) @@ -2091,7 +2091,7 @@ (define* (package->cross-derivation package target #:system system #:guile guile))))) (return drv))) - package system target graft?)) + package (%parameterized-counter) system target graft?)) (define* (package-output store package #:optional (output "out") (system (%current-system))) diff --git a/tests/gexp.scm b/tests/gexp.scm index 00bb729e76..91819806d0 100644 --- a/tests/gexp.scm +++ b/tests/gexp.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014-2025 Ludovic Courtès ;;; Copyright © 2021-2022 Maxime Devos +;;; Copyright © 2025 David Elsing ;;; ;;; This file is part of GNU Guix. ;;; @@ -487,6 +488,36 @@ (define (match-input thing) (return (and (eq? drv0 result0) (eq? drv1 result1))))) +(test-assertm "with-parameters for custom parameter" + (mlet* %store-monad + ((%param -> (make-parameter "A")) + (pkg -> (package + (name "testp") + (version "0") + (source #f) + (build-system trivial-build-system) + (arguments + (list + #:builder + #~(let ((port (open-file (string-append #$output) "w"))) + (display (string-append #$(%param) "\n") port) + (close-port port)))) + (home-page #f) + (synopsis #f) + (description #f) + (license #f))) + (obj1 -> (with-parameters ((%param "B")) pkg)) + (obj2 -> (with-parameters ((%param "C")) pkg)) + (result0 (package->derivation pkg)) + (result1 (lower-object obj1)) + (result2 (lower-object obj2)) + (result3 (lower-object pkg))) + (return (and (not + (or (eq? result0 result1) + (eq? result0 result2) + (eq? result1 result2))) + (eq? result0 result3))))) + (test-assert "with-parameters + file-append" (let* ((system (match (%current-system) ("aarch64-linux" "x86_64-linux") -- 2.48.1 From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 02 05:59:43 2025 Received: (at 77708) by debbugs.gnu.org; 2 Jun 2025 09:59:43 +0000 Received: from localhost ([127.0.0.1]:48554 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uM1xG-0006eW-8O for submit@debbugs.gnu.org; Mon, 02 Jun 2025 05:59:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51072) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uM1xD-0006dD-3o for 77708@debbugs.gnu.org; Mon, 02 Jun 2025 05:59:40 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uM1x5-0002eB-Lo; Mon, 02 Jun 2025 05:59:31 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: David Elsing Subject: Re: [bug#77708] [PATCH] gexp: =?utf-8?Q?=E2=80=98with-parameters?= =?utf-8?Q?=E2=80=98?= is respected by caches. In-Reply-To: <633ad28c062af23019c7ce7e172cec811065be18.1744296414.git.david.elsing@posteo.net> (David Elsing's message of "Thu, 10 Apr 2025 14:46:54 +0000") References: <633ad28c062af23019c7ce7e172cec811065be18.1744296414.git.david.elsing@posteo.net> Date: Mon, 02 Jun 2025 10:45:06 +0200 Message-ID: <87o6v6r0q5.fsf@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 77708 Cc: dev@jpoiret.xyz, zimon.toutoune@gmail.com, othacehe@gnu.org, ludo@gnu.org, me@tobias.gr, 77708@debbugs.gnu.org, guix@cbaines.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi David, Apologies for the delay. David Elsing writes: > As noted by Ludo' [1], several objects dependent on packages > (such as derivations or grafts) are cached by the package and do not > take parameters (apart from %current-system, %current-target-system and > %graft?) into account. To fix that, my idea was to introduce an > additional parameter `%parameterized-counter', which uniquely identifies > a set of parameters and values in the object and which > is used as additional key by the caches. > > To prevent a collision, the parameters and values are stored in a hash ta= ble, > which keeps them alive forever. Would it be preferable to use something l= ike a > cryptographic hash instead? > > For `cache!' in (guix packages), I used > `(,(scm->pointer package) . ,(%parameterized-counter)) as key together wi= th > hash-set! and hash-ref instead of hashq-set! and hashq-ref. Is that OK? > > [1] https://issues.guix.gnu.org/75879 Overall I have two main concerns: the added complexity and the run-time overhead. I would be tempted to just not pretend supporting for arbitrary parameters than to pay this price. WDYT? > @@ -302,7 +305,7 @@ (define* (lower-object obj > (not (derivation? lowered))) > (loop lowered) > (return lowered))) > - obj > + obj (%parameterized-counter) > system target graft?))))))) [...] > +;; Counter which uniquely identifies specific parameters and values used= for > +;; . > +(define %parameterized-counter > + (make-parameter #f)) The =E2=80=9Ccounter=E2=80=9D really determines the dynamic scope of parame= ters in fact, right? Should it be renamed to =E2=80=9Cparameter scope=E2=80=9D or simila= r? > +(define %parameterized-counter-next-value 0) > + > +(define %parameterized-counters (make-hash-table)) This is very much like the regular dynamic environment (info "(guile) Fluids and Dynamic States"). > +;; Add %parameterized-counter to PARAMETERS and its value, > +;; which depends on PARAMETERS and VALUES, to PARAMETER-VALUES. > +(define (add-parameterized-counter parameters parameter-values) > + (let* ((key `(,parameters . ,parameter-values)) > + (counter > + (match (hash-ref %parameterized-counters key) Note that this is quite expensive: hashing traverses part of KEY, and then all of it must be traversed for comparison; in practice there would probably be few parameters though. > @@ -1689,13 +1690,12 @@ (define (cache! cache package system thunk) > SYSTEM." > ;; FIXME: This memoization should be associated with the open store, b= ecause > ;; otherwise it breaks when switching to a different store. > - (let ((result (thunk))) > - ;; Use `hashq-set!' instead of `hash-set!' because `hash' returns the > - ;; same value for all structs (as of Guile 2.0.6), and because point= er > - ;; equality is sufficient in practice. > - (hashq-set! cache package > - `((,system . ,result) > - ,@(or (hashq-ref cache package) '()))) > + (let ((result (thunk)) > + (key `(,(scm->pointer package) . ,(%parameterized-counter)))) > + (hash-set! cache key > + `((,system . ,result) > + ,@(or (hash-ref cache key) > + '()))) The =E2=80=98scm->pointer=E2=80=99 trick prevents the expensive s= tructure traversal, but I=E2=80=99m unsure about the performance of switching to =E2= =80=98equal?=E2=80=99 instead of =E2=80=98eq?=E2=80=99. Does the =E2=80=98scm->pointer=E2=80=99 trick work with weak-key hash table= s like =E2=80=98%bag-cache=E2=80=99? I think it does because =E2=80=98scm->pointe= r=E2=80=99 records the link between the pointer object and the package object in a weak hash table. But it=E2=80=99s still quite bumpy. > @@ -2068,7 +2068,7 @@ (define* (package->derivation package > #:system system > #:guile guile))))) > (return drv))) > - package system #f graft?)) > + package (%parameterized-counter) system #f graft?)) I wonder about the performance impact of the extra lookup key. Thanks for working on this! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 24 13:38:36 2025 Received: (at 77708) by debbugs.gnu.org; 24 Jun 2025 17:38:37 +0000 Received: from localhost ([127.0.0.1]:47586 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uU7bQ-0002Bd-AP for submit@debbugs.gnu.org; Tue, 24 Jun 2025 13:38:36 -0400 Received: from mout01.posteo.de ([185.67.36.65]:38895) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uU7bM-0002Au-QJ for 77708@debbugs.gnu.org; Tue, 24 Jun 2025 13:38:34 -0400 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id D70EA240028 for <77708@debbugs.gnu.org>; Tue, 24 Jun 2025 19:38:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=posteo.net; s=1984.ea087b; t=1750786706; bh=NfZF7v9Uo09Kq+RdO1xYk/hMN2TLumsNi6/uCOM2fFs=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type: Content-Transfer-Encoding:From; b=U1YrR5m6VIqJPvJOc1k8X3Q/wNuQTdQ8HU5lcjhszlkJMY8kQODvWKIAsCtJb1scb zN2tiHa2LDOLCpxt+ycyhUJi03/XUFh6KFY2mufNAV9VVhJXLffo2deaUR06HP/LXf V4UHY/MwKKhm9wbtDz+odFIfq9fjQR8R94VvOqAniHsTb33PynDFypbUncudSFDXhR r+zgOGAENmWTn9XIPVqiP8bV+gPNf2VJlYLpKfTmdG2N4pCGo3F47wYPm/G0z76ulE SycbUckY9UTcVpJyiNZHepqzjy4tuAP/MABdLZFSjleikaHensiORapWlsnji/yEIC HWK0U1zUC5QxzIY9x0rRV6cugROJQRDXYDTNrT044uqx7WXziMP8zARuYNMAo8Zbwp 33/0MqD4weorbun4k8R/XOHmnLHsUvx5bD/bmujDuwavsxcOAolevwrxHeu3Mhx7Ca /rJRqlGWjxRORcJOoEKa9KKzDPEeVS77upI9dFD0zWRk0G7VdqG Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4bRXH56V5kz6twP; Tue, 24 Jun 2025 19:38:25 +0200 (CEST) From: David Elsing To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#77708] [PATCH] gexp: =?utf-8?Q?=E2=80=98with-parameters?= =?utf-8?Q?=E2=80=98?= is respected by caches. In-Reply-To: <87o6v6r0q5.fsf@inria.fr> References: <633ad28c062af23019c7ce7e172cec811065be18.1744296414.git.david.elsing@posteo.net> <87o6v6r0q5.fsf@inria.fr> Date: Tue, 24 Jun 2025 17:38:25 +0000 Message-ID: <86ms9xca5q.fsf@posteo.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 77708 Cc: dev@jpoiret.xyz, zimon.toutoune@gmail.com, othacehe@gnu.org, ludo@gnu.org, me@tobias.gr, 77708@debbugs.gnu.org, guix@cbaines.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Ludo', Ludovic Court=C3=A8s writes: > Apologies for the delay. No problem, thanks for looking at the patch! > Overall I have two main concerns: the added complexity and the run-time > overhead. I would be tempted to just not pretend supporting > for arbitrary parameters than to pay this price. > > WDYT? I think parameterized packages would be very useful. Something like the previous approach [1] would of course be nicer, but I didn't find any information why it was not continued. From what I can tell, `with-parameters' is now working except for the caches, so in my opinion it makes sense to fix that. > The =E2=80=9Ccounter=E2=80=9D really determines the dynamic scope of para= meters in fact, > right? Should it be renamed to =E2=80=9Cparameter scope=E2=80=9D or simi= lar? I only defined it to avoid frequently recalculating the hash of the dynamic parameters, so the caches only need to include the single integer in the key. >> +(define %parameterized-counter-next-value 0) >> + >> +(define %parameterized-counters (make-hash-table)) > > This is very much like the regular dynamic environment (info "(guile) > Fluids and Dynamic States"). Yes, this just assigns a unique integer to each unique set of parameters. >> +;; Add %parameterized-counter to PARAMETERS and its value, >> +;; which depends on PARAMETERS and VALUES, to PARAMETER-VALUES. >> +(define (add-parameterized-counter parameters parameter-values) >> + (let* ((key `(,parameters . ,parameter-values)) >> + (counter >> + (match (hash-ref %parameterized-counters key) > > Note that this is quite expensive: hashing traverses part of KEY, and > then all of it must be traversed for comparison; in practice there would > probably be few parameters though. Do you think using a cryptographic hash (to avoid collisions) would be better? I think we need to look at the values of the parameters and not just at object identity in case an object was modified or for objects like strings, right? > The =E2=80=98scm->pointer=E2=80=99 trick prevents the expensive = structure > traversal, but I=E2=80=99m unsure about the performance of switching to = =E2=80=98equal?=E2=80=99 > instead of =E2=80=98eq?=E2=80=99. > I wonder about the performance impact of the extra lookup key. I did some benchmarking with 10 runs of 'guix build --no-grafts libreoffice= -d': - Previously: 3.684 s =C2=B1 0.019 s - Using `scm->pointer' and `hash-set' and `hash-ref' in `cache!': 3.712 s =C2=B1 0.027 s - Using an additional key in `package->derivation': 3.697 s =C2=B1 0.028 s - With the full path applied: 3.726 s =C2=B1 0.032 s So there indeed seems to be a perfomance impact of the order of ~1% in this case. > Does the =E2=80=98scm->pointer=E2=80=99 trick work with weak-key hash tab= les like > =E2=80=98%bag-cache=E2=80=99? I think it does because =E2=80=98scm->poin= ter=E2=80=99 records the link > between the pointer object and the package object in a weak hash table. > But it=E2=80=99s still quite bumpy. Oh that's a good point, is chaining weak hash tables a problem? I couldn't think of another way to add the integer describing the parameters to the key though. Also problematic is the `%parameterized-counter' hash table, as it keeps the keys (i.e. all parameters passed to `with-parameters') alive forever, so I think it should also be a weak hash table, right? If the parameters are dropped and later constructed again, the derivation would just be recalculated, like for `%bag-cache'. Cheers, David [1] https://guix.gnu.org/en/blog/2023/parameterized-packages-for-gnu-guix/ From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 26 02:58:45 2025 Received: (at 77708) by debbugs.gnu.org; 26 Jun 2025 06:58:45 +0000 Received: from localhost ([127.0.0.1]:47521 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uUgZI-0004Au-N4 for submit@debbugs.gnu.org; Thu, 26 Jun 2025 02:58:45 -0400 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:59207) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uUgZE-0004Ae-Vu for 77708@debbugs.gnu.org; Thu, 26 Jun 2025 02:58:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version:content-transfer-encoding; bh=oy8gRNGpiWRceoFs7xvibsKNSz+/c5sVmZf244CzDus=; b=gZzxJ38vyeJf3hrAdNR4tVLMpcSSRLomTX10TAMEGIjj/G3OsRNOnFlo B97rjNGnaaFYi9oC+lNSVTOZOFSecXRDyUVq25OlUpUR1JT71FutYBZjx mEJtBnSg1tYVDwDWLEI+EQXs7kjoTm+Y9iSn+3HcXTF4NI4lnLlQMiQI1 E=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=ludovic.courtes@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.16,267,1744063200"; d="scan'208";a="120010022" Received: from unknown (HELO ribbon) ([193.50.110.182]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Jun 2025 08:58:34 +0200 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: David Elsing Subject: Re: [bug#77708] [PATCH] gexp: =?utf-8?Q?=E2=80=98with-parameters?= =?utf-8?Q?=E2=80=98?= is respected by caches. In-Reply-To: <86ms9xca5q.fsf@posteo.net> (David Elsing's message of "Tue, 24 Jun 2025 17:38:25 +0000") Organization: Inria References: <633ad28c062af23019c7ce7e172cec811065be18.1744296414.git.david.elsing@posteo.net> <87o6v6r0q5.fsf@inria.fr> <86ms9xca5q.fsf@posteo.net> User-Agent: mu4e 1.12.11; emacs 30.1 X-URL: https://people.bordeaux.inria.fr/lcourtes/ X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu X-Revolutionary-Date: Octidi 8 Messidor an 233 de la =?utf-8?Q?R=C3=A9volu?= =?utf-8?Q?tion=2C?= jour de =?utf-8?Q?l'=C3=89chalotte?= Date: Thu, 26 Jun 2025 08:56:51 +0200 Message-ID: <87bjqbvvm4.fsf@inria.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 77708 Cc: dev@jpoiret.xyz, zimon.toutoune@gmail.com, othacehe@gnu.org, ludo@gnu.org, me@tobias.gr, 77708@debbugs.gnu.org, guix@cbaines.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, Quick note=E2=80=A6 David Elsing writes: > Ludovic Court=C3=A8s writes: [...] >> Overall I have two main concerns: the added complexity and the run-time >> overhead. I would be tempted to just not pretend supporting >> for arbitrary parameters than to pay this price. >> >> WDYT? > > I think parameterized packages would be very useful. Something like the > previous approach [1] would of course be nicer, but I didn't find any > information why it was not continued. Oh, I see. The GSoC project you=E2=80=99re referring to was very nice but = IMO went a bit too far in terms of what it provides; but the main factor for it being discontinued is that the intern walked away after the internship I believe, and nobody picked it up. FWIW, I was advocating for something simpler and that would try to reduce the risks of combinatorial explosion=C2=B9. At any rate, I don=E2=80=99t think parameters (in the sense of SRFI-39) can= be a useful building block for parameterized packages (in the sense of Gentoo USE flags.) I think parameterized packages need to be implemented with some declarative interface as was proposed during that GSoC project and that earlier message I mentioned. I hope that makes sense! Thanks, Ludo=E2=80=99. =C2=B9 https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00312.html From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 06 12:15:22 2025 Received: (at 77708-done) by debbugs.gnu.org; 6 Jul 2025 16:15:22 +0000 Received: from localhost ([127.0.0.1]:56885 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uYS1R-0006v2-HO for submit@debbugs.gnu.org; Sun, 06 Jul 2025 12:15:21 -0400 Received: from mout02.posteo.de ([185.67.36.66]:45825) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uYS1O-0006pc-Cl for 77708-done@debbugs.gnu.org; Sun, 06 Jul 2025 12:15:19 -0400 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id D1C1A240104 for <77708-done@debbugs.gnu.org>; Sun, 6 Jul 2025 18:15:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=posteo.net; s=1984.ea087b; t=1751818511; bh=epGZr8F3Cduz1kFr08bClKBfIG+V+JVWdyEBKm13KFc=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type: Content-Transfer-Encoding:From; b=aUjn4A9xyvIbz+PgGxUXyYREx/XTjdH8UrBZDdai/UU745yuuD4/U2taxWGWETWHc n+KwbAEJ5sRJO8uVjGj8oRxuZLVgJCKEDPBJ02wLqILq+N2x9osIokwP31uKnVZPWM +4Q/4wRKSf6SssDZMrsCajMBpH38kpHNAg57OtRvCyqFt/1RQgM3YVmcg+k4ifsFsK VWW7jx4SDfZkvUPvxMWN2SoTSvpNKQGFHZt2VdwNJdPb8XHaLUaEY+KxfDAREbn6eE OMeR9STvNsBvIkQgnh+nQ90qaS/omnsBuSQtMBgh1Sk2eVgBrgZ6lEupjVHFfnHd31 JL7+F3WSmNh2+z6K8Jjkj692Ld7cdCRpbb0yGhQc+MkHq7rx3t1ASdOFDa10lLO/xg 63GK7It0mq28mNN1uzumtxbvRjGB7PjfRRLpDW38EwN7ZEOBegaqVnfCEwGfwQQCsP OKJkS4pnTFgI2IzPscdiRNq3e2m1+i4Nkkw55vqxFDaqpZxyp4c Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4bZssV4qWnz9rxD; Sun, 6 Jul 2025 18:15:10 +0200 (CEST) From: David Elsing To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#77708] [PATCH] gexp: =?utf-8?Q?=E2=80=98with-parameters?= =?utf-8?Q?=E2=80=98?= is respected by caches. In-Reply-To: <87bjqbvvm4.fsf@inria.fr> References: <633ad28c062af23019c7ce7e172cec811065be18.1744296414.git.david.elsing@posteo.net> <87o6v6r0q5.fsf@inria.fr> <86ms9xca5q.fsf@posteo.net> <87bjqbvvm4.fsf@inria.fr> Date: Sun, 06 Jul 2025 16:15:01 +0000 Message-ID: <865xg55m9m.fsf@posteo.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -3.3 (---) X-Debbugs-Envelope-To: 77708-done Cc: dev@jpoiret.xyz, 77708-done@debbugs.gnu.org, zimon.toutoune@gmail.com, othacehe@gnu.org, ludo@gnu.org, me@tobias.gr, guix@cbaines.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.3 (----) Hi, Ludovic Court=C3=A8s writes: > Oh, I see. The GSoC project you=E2=80=99re referring to was very nice bu= t IMO > went a bit too far in terms of what it provides; but the main factor for > it being discontinued is that the intern walked away after the > internship I believe, and nobody picked it up. FWIW, I was advocating > for something simpler and that would try to reduce the risks of > combinatorial explosion=C2=B9. Ah ok, that makes sense. Would it be possible for multiple packages to share the same parameter, so that it can be set for all dependencies of a package? I guess it should be doable with `package-mapping'. I'm also not sure the full Cartesial product of all possible parameters should always need to be supported, as it might or some combinations don't make sense, so I think there should be a way to specify the supported combinations (which isn't the case for both proposals IIUC). > At any rate, I don=E2=80=99t think parameters (in the sense of SRFI-39) c= an be a > useful building block for parameterized packages (in the sense of Gentoo > USE flags.) I think parameterized packages need to be implemented with > some declarative interface as was proposed during that GSoC project and > that earlier message I mentioned. Yes I agree that using SRFI-39 parameters is not very suitable for parameterized packages. My idea was just to fix `with-parameters' until parameterized packages are implemented properly, but it makes sense if you don't want to support and later deprecate this. Maybe `with-parameters' and `' should be renamed if they only support `%graft?', `%current-system' and `%current-target-system'? Thanks, David From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 11 03:35:17 2025 Received: (at 77708-done) by debbugs.gnu.org; 11 Jul 2025 07:35:17 +0000 Received: from localhost ([127.0.0.1]:34314 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ua8Hs-00067i-JZ for submit@debbugs.gnu.org; Fri, 11 Jul 2025 03:35:16 -0400 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]:3687) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1ua8Ho-00062V-Rd for 77708-done@debbugs.gnu.org; Fri, 11 Jul 2025 03:35:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version:content-transfer-encoding; bh=i6McPwU3JRRrVHF0X3tHEimI5aDyD4D33DPx6wgHMGU=; b=j0aElaUeoNyPXqF41OYgYfPi7VzYLpyAZTjQFs7WjSa1vFRdmaVATrHq rhY6dNB46nX+lwZGmOV/5lDZb4QfbTWDsacmTI+tV1gvZlcaXED6mxWSa /W/VNKtVRpd5cwAJkL7oV8aVCul55l2HNSInZ0Je5Bt4tLzLiA12/RmZS g=; X-CSE-ConnectionGUID: trKrN4uqQzKlgrruxWwM3A== X-CSE-MsgGUID: dinJYgrGTxO8Jxy9M5fw6g== Authentication-Results: mail2-relais-roc.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=ludovic.courtes@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.16,303,1744063200"; d="scan'208";a="231296168" Received: from unknown (HELO ribbon) ([193.50.110.182]) by mail2-relais-roc.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jul 2025 09:35:06 +0200 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: David Elsing Subject: Parameterized packages In-Reply-To: <865xg55m9m.fsf@posteo.net> (David Elsing's message of "Sun, 06 Jul 2025 16:15:01 +0000") Organization: Inria References: <633ad28c062af23019c7ce7e172cec811065be18.1744296414.git.david.elsing@posteo.net> <87o6v6r0q5.fsf@inria.fr> <86ms9xca5q.fsf@posteo.net> <87bjqbvvm4.fsf@inria.fr> <865xg55m9m.fsf@posteo.net> User-Agent: mu4e 1.12.11; emacs 30.1 X-URL: https://people.bordeaux.inria.fr/lcourtes/ X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu X-Revolutionary-Date: Tridi 23 Messidor an 233 de la =?utf-8?Q?R=C3=A9volu?= =?utf-8?Q?tion=2C?= jour du Haricot Date: Fri, 11 Jul 2025 09:29:24 +0200 Message-ID: <875xfzchij.fsf@inria.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 77708-done Cc: dev@jpoiret.xyz, 77708-done@debbugs.gnu.org, zimon.toutoune@gmail.com, othacehe@gnu.org, ludo@gnu.org, me@tobias.gr, guix@cbaines.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, David Elsing writes: > Ludovic Court=C3=A8s writes: > >> Oh, I see. The GSoC project you=E2=80=99re referring to was very nice b= ut IMO >> went a bit too far in terms of what it provides; but the main factor for >> it being discontinued is that the intern walked away after the >> internship I believe, and nobody picked it up. FWIW, I was advocating >> for something simpler and that would try to reduce the risks of >> combinatorial explosion=C2=B9. > > Ah ok, that makes sense. Would it be possible for multiple packages to > share the same parameter, so that it can be set for all dependencies of > a package? I guess it should be doable with `package-mapping'. > I'm also not sure the full Cartesial product of all possible parameters > should always need to be supported, as it might or some combinations > don't make sense, so I think there should be a way to specify the > supported combinations (which isn't the case for both proposals IIUC). Yes, what I was proposing was to have both global and per-package parameters. Global parameters would be things like =E2=80=98x11-support?= =E2=80=99, whose value would be the same for all the affected packages (either all of them are built with X11 support or none of them is). > Yes I agree that using SRFI-39 parameters is not very suitable for > parameterized packages. My idea was just to fix `with-parameters' until > parameterized packages are implemented properly, but it makes sense if > you don't want to support and later deprecate this. > Maybe `with-parameters' and `' should be renamed if they > only support `%graft?', `%current-system' and `%current-target-system'? Yes, but I can=E2=80=99t think of a better name, and renaming would entail deprecation etc. Naming is hard. :-) Thanks, Ludo=E2=80=99. From unknown Fri Aug 15 16:19:52 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 08 Aug 2025 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator