From unknown Sat Jun 21 10:27:01 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#22990 <22990@debbugs.gnu.org> To: bug#22990 <22990@debbugs.gnu.org> Subject: Status: Grafts leads to inefficient substitute info retrieval Reply-To: bug#22990 <22990@debbugs.gnu.org> Date: Sat, 21 Jun 2025 17:27:01 +0000 retitle 22990 Grafts leads to inefficient substitute info retrieval reassign 22990 guix submitter 22990 ludo@gnu.org (Ludovic Court=C3=A8s) severity 22990 important thanks From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 11 11:53:21 2016 Received: (at submit) by debbugs.gnu.org; 11 Mar 2016 16:53:21 +0000 Received: from localhost ([127.0.0.1]:45348 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aeQJV-0001gI-17 for submit@debbugs.gnu.org; Fri, 11 Mar 2016 11:53:21 -0500 Received: from eggs.gnu.org ([208.118.235.92]:38696) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aeQJS-0001g6-GT for submit@debbugs.gnu.org; Fri, 11 Mar 2016 11:53:18 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aeQJK-0001xV-Lf for submit@debbugs.gnu.org; Fri, 11 Mar 2016 11:53:13 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:43762) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeQJK-0001xJ-IS for submit@debbugs.gnu.org; Fri, 11 Mar 2016 11:53:10 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57899) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeQJE-0003BY-PZ for bug-guix@gnu.org; Fri, 11 Mar 2016 11:53:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aeQJC-0001sH-3V for bug-guix@gnu.org; Fri, 11 Mar 2016 11:53:04 -0500 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:48889) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeQJC-0001sD-0d for bug-guix@gnu.org; Fri, 11 Mar 2016 11:53:02 -0500 Received: from pluto.bordeaux.inria.fr ([193.50.110.57]:49892 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1aeQJB-0005Gm-Eo for bug-guix@gnu.org; Fri, 11 Mar 2016 11:53:01 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: bug-guix@gnu.org Subject: Grafts leads to inefficient substitute info retrieval X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 22 =?utf-8?Q?Vent=C3=B4se?= an 224 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x3D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Fri, 11 Mar 2016 17:52:59 +0100 Message-ID: <8737rxx8gk.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) As of right now (v0.9.0-2007-g66a30a3), =E2=80=98graft-derivation=E2=80=99 = works either by: 1. Fetching substitute info about the things being built so that it can determine its references, which in turns allows it to determine whether they need to be grafted. 2. Building stuff, as a last resort, so that it can determine its references. Case #1 is hopefully going to be the most common. The problem with #1 is that when building a profile, we do one =E2=80=98package-derivation=E2=80=99 call for each package in the profile, = which translates in one =E2=80=98graft-derivation=E2=80=99 call for each relevant= package=C2=B9, which translates into one =E2=80=98references/substitutes=E2=80=99 call for= each. Concretely, what this means is this: $ guix package -u substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'.= .. 100.0% [=E2=80=A6] The following files would be downloaded: Each of the initial =E2=80=9Cupdating list=E2=80=9D message corresponds to = an HTTP request for a single narinfo file, which can take around 1 second. Instead, the ideal thing would be to fetch the narinfo files for all the relevant packages at once; that way, we=E2=80=99d spawn =E2=80=98guix subst= itute=E2=80=99 only once, and it would benefit from HTTP pipelining (one round-trip instead of N.) To achieve this, I=E2=80=99m thinking of extending gexp code such that gexp compilers can return a list of applicable grafts. The =E2=80=98package=E2= =80=99 compiler would do #:graft? #f and instead let =E2=80=98gexp->derivation=E2= =80=99 call =E2=80=98graft-derivation=E2=80=99. I=E2=80=99ll give it a try and report back. Ludo=E2=80=99. =C2=B9 A package is =E2=80=9Crelevant=E2=80=9D if =E2=80=98package-grafts= =E2=80=99 returns a non-empty list. From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 12 04:23:25 2016 Received: (at 22990) by debbugs.gnu.org; 12 Mar 2016 09:23:25 +0000 Received: from localhost ([127.0.0.1]:45716 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aefld-0000Lw-Iv for submit@debbugs.gnu.org; Sat, 12 Mar 2016 04:23:25 -0500 Received: from mail-lb0-f172.google.com ([209.85.217.172]:34464) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aeflb-0000Ld-Mt for 22990@debbugs.gnu.org; Sat, 12 Mar 2016 04:23:24 -0500 Received: by mail-lb0-f172.google.com with SMTP id xr8so179102683lbb.1 for <22990@debbugs.gnu.org>; Sat, 12 Mar 2016 01:23:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=4WuO3BqpwsW0S32YaYjqP8MLksyEX1YyUIDb5fQNzXE=; b=hzCONfhQtsN6FUnaH1eBEol0cGpcUAYYEcyEbtKKrmwICwo5Y3WWXa7dp+ir4jhkJe gY+CPEgKebmG6dMhXWU//9e6hHy0RoNLrRFYm6BuMNJNOzoJB2o/Z5cy0YxUfMLwWyHr syIN8Y7GJGKsXJvtpnVinOJa0+F4GMwc4egYN7Zs5b9lUBJzy4e2T27BKzJrrkne+JQP 3bXCYMzz9NpBuTBQufoD5Esw061cLxsAHNjW3kbgzU76QIAGJmmAl3dLiXQgJF9iSdqi qMWP/Pg4TZHPYyWzI15m6oE1PnTxvySGoD0ZXBjX8kYcDQ6l0oSuMfeGNeqFuFzDMyvg n0QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=4WuO3BqpwsW0S32YaYjqP8MLksyEX1YyUIDb5fQNzXE=; b=Y4mX+H8XBdtHMhYdbK+odlcWMx8Ukl7BtEhVuBp+BeoAwtDL9Sg2kqICYL0h/jBbpT 7ZwbtmdmmiXjZDUC+95Gwz/wmJBXontl0d6xPdfqAmYWRJE85fype548jlwX7imKsUQs Gl8mHyK0CuAxGrpVMnsLeUdNl+enpVYTC24xrFK8O1qKTOmfUKWHC8F871elUa4qcoVg i3RKNXdvciA+YbXRxGZrVH1ZsPyU5fUfBoCZMrGCMdSNf1vhoSjmAHhTiGvZHyglcWng +/NsQ/KOgRvVT2H1zWxFl3wMgJH+BhIk8prL/lxPC0oNM7WLzzi49OpAWBaLSeUf/byA WhGQ== X-Gm-Message-State: AD7BkJLpKw5/dQyguHeeKt2VEXbsynRJnuxRlub2k55OC/prSBju8PImpWqaycNpP7qg6g== X-Received: by 10.25.88.69 with SMTP id m66mr3615553lfb.137.1457774597995; Sat, 12 Mar 2016 01:23:17 -0800 (PST) Received: from leviafan ([217.107.192.146]) by smtp.gmail.com with ESMTPSA id f6sm1965935lbd.4.2016.03.12.01.23.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 12 Mar 2016 01:23:17 -0800 (PST) From: Alex Kost To: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> Date: Sat, 12 Mar 2016 12:23:35 +0300 In-Reply-To: <8737rxx8gk.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 11 Mar 2016 17:52:59 +0100") Message-ID: <87a8m4123s.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 22990 Cc: 22990@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) Ludovic Court=C3=A8s (2016-03-11 19:52 +0300) wrote: > As of right now (v0.9.0-2007-g66a30a3), =E2=80=98graft-derivation=E2=80= =99 works either by: > > 1. Fetching substitute info about the things being built so that it > can determine its references, which in turns allows it to determine > whether they need to be grafted. > > 2. Building stuff, as a last resort, so that it can determine its > references. I noticed that #1 is happening even with --no-substitutes option. Is it intended? I've tried this: $ guix build --dry-run --no-substitutes mutt substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100= .0% substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100= .0% substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100= .0% substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100= .0% substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100= .0% The following derivations would be built: /gnu/store/a2w22xlmfwkgwx4vw11dxc6zrdmww435-mutt-1.5.24.drv ... --=20 Alex From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 13 08:11:52 2016 Received: (at 22990) by debbugs.gnu.org; 13 Mar 2016 12:11:52 +0000 Received: from localhost ([127.0.0.1]:46660 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1af4sC-0001ie-0V for submit@debbugs.gnu.org; Sun, 13 Mar 2016 08:11:52 -0400 Received: from eggs.gnu.org ([208.118.235.92]:53736) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1af4sA-0001iS-3d for 22990@debbugs.gnu.org; Sun, 13 Mar 2016 08:11:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1af4s0-0001kE-0E for 22990@debbugs.gnu.org; Sun, 13 Mar 2016 08:11:44 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36331) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1af4rz-0001kA-Td; Sun, 13 Mar 2016 08:11:39 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:60896 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1af4ry-0003dP-TK; Sun, 13 Mar 2016 08:11:39 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Alex Kost Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> <87a8m4123s.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 24 =?utf-8?Q?Vent=C3=B4se?= an 224 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x3D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Sun, 13 Mar 2016 13:11:36 +0100 In-Reply-To: <87a8m4123s.fsf@gmail.com> (Alex Kost's message of "Sat, 12 Mar 2016 12:23:35 +0300") Message-ID: <874mcazifb.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22990 Cc: 22990@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Alex Kost skribis: > Ludovic Court=C3=A8s (2016-03-11 19:52 +0300) wrote: > >> As of right now (v0.9.0-2007-g66a30a3), =E2=80=98graft-derivation=E2=80= =99 works either by: >> >> 1. Fetching substitute info about the things being built so that it >> can determine its references, which in turns allows it to determine >> whether they need to be grafted. >> >> 2. Building stuff, as a last resort, so that it can determine its >> references. > > I noticed that #1 is happening even with --no-substitutes option. Is it > intended? Not really, but I see this is because =E2=80=98substitutable-path-info=E2= =80=99 (called from =E2=80=98references/substitutes=E2=80=99, called from =E2=80=98graft-d= erivation=E2=80=99) works regardless of whether substitutes are enabled: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> ,use(guix) scheme@(guile-user)> (define s (open-connection)) scheme@(guile-user)> (set-build-options s #:use-substitutes? #f) $2 =3D #t scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1faz= v6smn-mutt-1.5.24") $3 =3D #f scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyyg= xz8zq0r1ca1fazv6smn-mutt-1.5.24")) $4 =3D (#< path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6sm= n-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1= .5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.= 4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/= d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z= 4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fa= zv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.= 0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store= /sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77= qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-b= ash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21"= ) dl-size: 0 nar-size: 6661016>) --8<---------------cut here---------------end--------------->8--- However, substitutes are not downloaded, so in this regard --no-substitutes is honored. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 15 04:24:55 2016 Received: (at 22990) by debbugs.gnu.org; 15 Mar 2016 08:24:55 +0000 Received: from localhost ([127.0.0.1]:48481 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1afkHf-00023R-Js for submit@debbugs.gnu.org; Tue, 15 Mar 2016 04:24:55 -0400 Received: from eggs.gnu.org ([208.118.235.92]:34251) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1afkHd-00023E-Qp for 22990@debbugs.gnu.org; Tue, 15 Mar 2016 04:24:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1afkHV-0005NB-Fe for 22990@debbugs.gnu.org; Tue, 15 Mar 2016 04:24:48 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:57703) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1afkHV-0005N7-C6 for 22990@debbugs.gnu.org; Tue, 15 Mar 2016 04:24:45 -0400 Received: from pluto.bordeaux.inria.fr ([193.50.110.57]:56398 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1afkHU-0005Ix-Qi for 22990@debbugs.gnu.org; Tue, 15 Mar 2016 04:24:45 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: 22990@debbugs.gnu.org Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> Date: Tue, 15 Mar 2016 09:24:42 +0100 In-Reply-To: <8737rxx8gk.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 11 Mar 2016 17:52:59 +0100") Message-ID: <87mvq0tagl.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22990 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > Concretely, what this means is this: > > $ guix package -u > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > [=E2=80=A6] > The following files would be downloaded: This is somewhat mitigated by commits f09aea1b58b3ef961d3cc712f116fe4617bc8f90 and 264fdedb408ba3620d1e361de6c77e7925025301. In addition, 026ca50fa4c46a8e280cd51621bbec76b12c0757 slightly improves =E2=80=98guix substitute=E2=80=99 performance. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 15 14:50:08 2016 Received: (at 22990) by debbugs.gnu.org; 15 Mar 2016 18:50:08 +0000 Received: from localhost ([127.0.0.1]:49372 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1afu2i-00020X-I7 for submit@debbugs.gnu.org; Tue, 15 Mar 2016 14:50:08 -0400 Received: from world.peace.net ([50.252.239.5]:39761 ident=hope0) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1afu2g-00020P-Rk for 22990@debbugs.gnu.org; Tue, 15 Mar 2016 14:50:07 -0400 Received: from [10.1.10.78] (helo=jojen) by world.peace.net with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1afu2Z-0004re-0v; Tue, 15 Mar 2016 14:49:59 -0400 From: Mark H Weaver To: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> <87a8m4123s.fsf@gmail.com> <874mcazifb.fsf@gnu.org> Date: Tue, 15 Mar 2016 14:49:55 -0400 In-Reply-To: <874mcazifb.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Sun, 13 Mar 2016 13:11:36 +0100") Message-ID: <8737rrmv8s.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.0.92 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 22990 Cc: Alex Kost , 22990@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Alex Kost skribis: > >> Ludovic Court=C3=A8s (2016-03-11 19:52 +0300) wrote: >> >>> As of right now (v0.9.0-2007-g66a30a3), =E2=80=98graft-derivation=E2=80= =99 works either by: >>> >>> 1. Fetching substitute info about the things being built so that it >>> can determine its references, which in turns allows it to determine >>> whether they need to be grafted. >>> >>> 2. Building stuff, as a last resort, so that it can determine its >>> references. >> >> I noticed that #1 is happening even with --no-substitutes option. Is it >> intended? > > Not really, but I see this is because =E2=80=98substitutable-path-info=E2= =80=99 (called > from =E2=80=98references/substitutes=E2=80=99, called from =E2=80=98graft= -derivation=E2=80=99) works > regardless of whether substitutes are enabled: > > scheme@(guile-user)> ,use(guix) > scheme@(guile-user)> (define s (open-connection)) > scheme@(guile-user)> (set-build-options s #:use-substitutes? #f) > $2 =3D #t > scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1f= azv6smn-mutt-1.5.24") > $3 =3D #f > scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiy= ygxz8zq0r1ca1fazv6smn-mutt-1.5.24")) > $4 =3D (#< path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6= smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt= -1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-= 2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/stor= e/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa= 4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1= fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-= 1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/sto= re/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac= 77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp= -bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.2= 1") dl-size: 0 nar-size: 6661016>) Is the information from the substitute server authenticated by checking hydra's signature against the list of keys in /etc/guix/acls? The reason I ask is that if the set of runtime dependencies received is incomplete, it could lead to incorrect grafting, namely that references to compromised libraries could be retained. > However, substitutes are not downloaded, so in this regard > --no-substitutes is honored. It depends on the intent of --no-substitutes. If the intent is to avoid trusting the substitute server, then by relying on the accuracy of the runtime dependency data from Hydra, we are failing to honor that intent. That said, I think it's okay to document that --no-substitutes alone is not sufficient to avoid trusting a substitute server, and that the proper way to accomplish that is to make sure its key is not in /etc/guix/acls. What do you think? Thanks, Mark From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 15 17:50:19 2016 Received: (at 22990) by debbugs.gnu.org; 15 Mar 2016 21:50:19 +0000 Received: from localhost ([127.0.0.1]:49490 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1afwr5-0006QK-7y for submit@debbugs.gnu.org; Tue, 15 Mar 2016 17:50:19 -0400 Received: from eggs.gnu.org ([208.118.235.92]:48155) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1afwr3-0006Q7-L9 for 22990@debbugs.gnu.org; Tue, 15 Mar 2016 17:50:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1afwqv-0002Qd-Cz for 22990@debbugs.gnu.org; Tue, 15 Mar 2016 17:50:12 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:44015) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1afwqv-0002QZ-9c; Tue, 15 Mar 2016 17:50:09 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:39208 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1afwqu-0003s7-IN; Tue, 15 Mar 2016 17:50:09 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mark H Weaver Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> <87a8m4123s.fsf@gmail.com> <874mcazifb.fsf@gnu.org> <8737rrmv8s.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 26 =?utf-8?Q?Vent=C3=B4se?= an 224 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x3D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Tue, 15 Mar 2016 22:50:05 +0100 In-Reply-To: <8737rrmv8s.fsf@netris.org> (Mark H. Weaver's message of "Tue, 15 Mar 2016 14:49:55 -0400") Message-ID: <87twk7qulu.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 22990 Cc: Alex Kost , 22990@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Mark H Weaver skribis: > ludo@gnu.org (Ludovic Court=C3=A8s) writes: > >> Alex Kost skribis: >> >>> Ludovic Court=C3=A8s (2016-03-11 19:52 +0300) wrote: >>> >>>> As of right now (v0.9.0-2007-g66a30a3), =E2=80=98graft-derivation=E2= =80=99 works either by: >>>> >>>> 1. Fetching substitute info about the things being built so that it >>>> can determine its references, which in turns allows it to determi= ne >>>> whether they need to be grafted. >>>> >>>> 2. Building stuff, as a last resort, so that it can determine its >>>> references. >>> >>> I noticed that #1 is happening even with --no-substitutes option. Is it >>> intended? >> >> Not really, but I see this is because =E2=80=98substitutable-path-info= =E2=80=99 (called >> from =E2=80=98references/substitutes=E2=80=99, called from =E2=80=98graf= t-derivation=E2=80=99) works >> regardless of whether substitutes are enabled: >> >> scheme@(guile-user)> ,use(guix) >> scheme@(guile-user)> (define s (open-connection)) >> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f) >> $2 =3D #t >> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1= fazv6smn-mutt-1.5.24") >> $3 =3D #f >> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpii= yygxz8zq0r1ca1fazv6smn-mutt-1.5.24")) >> $4 =3D (#< path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv= 6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mut= t-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan= -2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/sto= re/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxags= a4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca= 1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl= -1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/st= ore/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wa= c77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0p= p-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.= 21") dl-size: 0 nar-size: 6661016>) > > Is the information from the substitute server authenticated by checking > hydra's signature against the list of keys in /etc/guix/acls? Yes. >> However, substitutes are not downloaded, so in this regard >> --no-substitutes is honored. > > It depends on the intent of --no-substitutes. If the intent is to avoid > trusting the substitute server, then by relying on the accuracy of the > runtime dependency data from Hydra, we are failing to honor that intent. > > That said, I think it's okay to document that --no-substitutes alone is > not sufficient to avoid trusting a substitute server, and that the > proper way to accomplish that is to make sure its key is not in > /etc/guix/acls. The sysadmin gets to choose which principals are trusted; unprivileged users can only shrink this set. The weird thing is that in this case, passing --substitute-urls=3D'' on the client side would effectively disable substitutes entirely. > What do you think? We could augment the doc for --no-substitutes, I guess we should first document that grafting relies on server-provided info. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Dec 09 18:25:17 2016 Received: (at control) by debbugs.gnu.org; 9 Dec 2016 23:25:17 +0000 Received: from localhost ([127.0.0.1]:36367 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cFUXV-0006XD-6X for submit@debbugs.gnu.org; Fri, 09 Dec 2016 18:25:17 -0500 Received: from eggs.gnu.org ([208.118.235.92]:59640) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cFU8i-0004Ph-Ft for control@debbugs.gnu.org; Fri, 09 Dec 2016 17:59:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cFS5O-0004mp-Fc for control@debbugs.gnu.org; Fri, 09 Dec 2016 15:48:10 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=5.0 tests=BAYES_40,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:45698) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cFS5O-0004ml-De for control@debbugs.gnu.org; Fri, 09 Dec 2016 15:48:06 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:36550 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cFS5N-0001EC-P2 for control@debbugs.gnu.org; Fri, 09 Dec 2016 15:48:06 -0500 Date: Fri, 09 Dec 2016 21:48:04 +0100 Message-Id: <87shpw7rsr.fsf@gnu.org> To: control@debbugs.gnu.org From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: control message for bug #25137 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -8.0 (--------) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -8.0 (--------) merge 25137 22990 From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 08 16:44:54 2017 Received: (at 22990) by debbugs.gnu.org; 8 Jan 2017 21:44:54 +0000 Received: from localhost ([127.0.0.1]:47176 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQLGo-0005QX-Ij for submit@debbugs.gnu.org; Sun, 08 Jan 2017 16:44:54 -0500 Received: from eggs.gnu.org ([208.118.235.92]:54108) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQLGn-0005QL-BP for 22990@debbugs.gnu.org; Sun, 08 Jan 2017 16:44:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cQLGe-0007mm-9i for 22990@debbugs.gnu.org; Sun, 08 Jan 2017 16:44:48 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:33368) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cQLGe-0007mY-65 for 22990@debbugs.gnu.org; Sun, 08 Jan 2017 16:44:44 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:47630 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cQLGd-0003OK-HJ for 22990@debbugs.gnu.org; Sun, 08 Jan 2017 16:44:43 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: 22990@debbugs.gnu.org Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> Date: Sun, 08 Jan 2017 22:44:41 +0100 In-Reply-To: <8737rxx8gk.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 11 Mar 2016 17:52:59 +0100") Message-ID: <87k2a5rzue.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -8.2 (--------) X-Debbugs-Envelope-To: 22990 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -8.2 (--------) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > Concretely, what this means is this: > > $ guix package -u > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > [=E2=80=A6] > The following files would be downloaded: > > Each of the initial =E2=80=9Cupdating list=E2=80=9D message corresponds t= o an HTTP > request for a single narinfo file, which can take around 1 second. > > Instead, the ideal thing would be to fetch the narinfo files for all the > relevant packages at once; that way, we=E2=80=99d spawn =E2=80=98guix sub= stitute=E2=80=99 only > once, and it would benefit from HTTP pipelining (one round-trip instead > of N.) > > To achieve this, I=E2=80=99m thinking of extending gexp code such that ge= xp > compilers can return a list of applicable grafts. The =E2=80=98package= =E2=80=99 > compiler would do #:graft? #f and instead let =E2=80=98gexp->derivation= =E2=80=99 call > =E2=80=98graft-derivation=E2=80=99. The =E2=80=98wip-gexp-grafts=E2=80=99 branch does that. Namely, it=E2=80= =99s possible to know what grafts would apply to a gexp derivation build, so that one can first build ungrafted, and then apply the grafts to the results. So for a profile, we=E2=80=99d first build the profile as is, and only then would we graft it. Right now the tip of this branch is a hack such that =E2=80=98guix package= =E2=80=99: 1. Builds the original (ungrafted) derivation of the profile; 2. Manually calls =E2=80=98graft-derivation=E2=80=99 on that, passing it = the list of applicable grafts. Conceptually it=E2=80=99s what we want to do, but the drawback is that the caller (here =E2=80=98guix package=E2=80=99) goes through a lot of hops to = get the list of grafts and to apply it. I think we should instead have a way to annotate a derivation with a list of grafts, as well as a procedure to build that derivations in two phases (first the original derivation, then the grafts). To be continued=E2=80=A6 Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 09 17:55:51 2017 Received: (at 22990) by debbugs.gnu.org; 9 Jan 2017 22:55:51 +0000 Received: from localhost ([127.0.0.1]:48249 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQir1-0002Pd-IO for submit@debbugs.gnu.org; Mon, 09 Jan 2017 17:55:51 -0500 Received: from eggs.gnu.org ([208.118.235.92]:59576) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQiqz-0002PP-8l for 22990@debbugs.gnu.org; Mon, 09 Jan 2017 17:55:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cQiqq-0004p0-Sf for 22990@debbugs.gnu.org; Mon, 09 Jan 2017 17:55:44 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:50048) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cQiqq-0004oq-PZ for 22990@debbugs.gnu.org; Mon, 09 Jan 2017 17:55:40 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:34442 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cQiqq-0005El-3O for 22990@debbugs.gnu.org; Mon, 09 Jan 2017 17:55:40 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: 22990@debbugs.gnu.org Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> <87k2a5rzue.fsf@gnu.org> Date: Mon, 09 Jan 2017 23:55:38 +0100 In-Reply-To: <87k2a5rzue.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Sun, 08 Jan 2017 22:44:41 +0100") Message-ID: <877f633kt1.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -8.2 (--------) X-Debbugs-Envelope-To: 22990 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -8.2 (--------) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > ludo@gnu.org (Ludovic Court=C3=A8s) skribis: [...] >> To achieve this, I=E2=80=99m thinking of extending gexp code such that g= exp >> compilers can return a list of applicable grafts. The =E2=80=98package= =E2=80=99 >> compiler would do #:graft? #f and instead let =E2=80=98gexp->derivation= =E2=80=99 call >> =E2=80=98graft-derivation=E2=80=99. > > The =E2=80=98wip-gexp-grafts=E2=80=99 branch does that. Namely, it=E2=80= =99s possible to know > what grafts would apply to a gexp derivation build, so that one can > first build ungrafted, and then apply the grafts to the results. > So for a profile, we=E2=80=99d first build the profile as is, and only th= en > would we graft it. > > Right now the tip of this branch is a hack such that =E2=80=98guix packag= e=E2=80=99: > > 1. Builds the original (ungrafted) derivation of the profile; > > 2. Manually calls =E2=80=98graft-derivation=E2=80=99 on that, passing i= t the list of > applicable grafts. > > Conceptually it=E2=80=99s what we want to do, but the drawback is that the > caller (here =E2=80=98guix package=E2=80=99) goes through a lot of hops t= o get the list > of grafts and to apply it. > > I think we should instead have a way to annotate a derivation with a > list of grafts, as well as a procedure to build that derivations in two > phases (first the original derivation, then the grafts). The current iteration introduces =E2=80=9Cbuild continuation=E2=80=9D: a de= rivation can be annotated with a continuation, and the =E2=80=98build-things=E2=80=99 pr= ocedure will loop over continuations and return the final results (that only works for derivations directly passed as an argument to =E2=80=98build-things=E2= =80=99.) Example (where Guile 2.0.12 replaced by 2.0.13): --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix package -p foo -i guile-json guile-ssh gdb The following packages will be installed: guile-json 0.5.0 /gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-= 0.5.0 guile-ssh 0.10.2 /gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0= .10.2 gdb 7.12 /gnu/store/d2a4lmycc13ssdf47a9h410knlfqqq41-gdb-7.12 applying 6 grafts to /gnu/store/s7w6r9ih52kdzcl7kprf2rq48s69zh98-profile 3 packages in profile The following environment variable definitions may be needed: export PATH=3D"foo/bin${PATH:+:}$PATH" $ guix gc -R foo | grep guile /gnu/store/7fisf4frrgsjzmknjbab1dal23wxrp8d-guile-2.0.13 /gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2 /gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0 --8<---------------cut here---------------end--------------->8--- That seems to be a good model. Now we must make sure that it works also for =E2=80=98guix build foo=E2=80= =99 and =E2=80=98guix system build=E2=80=99. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 11 04:54:58 2017 Received: (at 22990) by debbugs.gnu.org; 11 Jan 2017 09:54:58 +0000 Received: from localhost ([127.0.0.1]:49391 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cRFcP-0001OL-Us for submit@debbugs.gnu.org; Wed, 11 Jan 2017 04:54:58 -0500 Received: from eggs.gnu.org ([208.118.235.92]:60301) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cRFcO-0001O6-Qv for 22990@debbugs.gnu.org; Wed, 11 Jan 2017 04:54:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cRFcF-0003Rf-Sp for 22990@debbugs.gnu.org; Wed, 11 Jan 2017 04:54:51 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:47715) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cRFcF-0003Ra-Pw for 22990@debbugs.gnu.org; Wed, 11 Jan 2017 04:54:47 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:37620 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cRFcF-0003tS-56 for 22990@debbugs.gnu.org; Wed, 11 Jan 2017 04:54:47 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: 22990@debbugs.gnu.org Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> <87k2a5rzue.fsf@gnu.org> <877f633kt1.fsf@gnu.org> Date: Wed, 11 Jan 2017 10:54:42 +0100 In-Reply-To: <877f633kt1.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 09 Jan 2017 23:55:38 +0100") Message-ID: <874m15yl99.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -8.2 (--------) X-Debbugs-Envelope-To: 22990 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -8.2 (--------) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > >> ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > > [...] > >>> To achieve this, I=E2=80=99m thinking of extending gexp code such that = gexp >>> compilers can return a list of applicable grafts. The =E2=80=98package= =E2=80=99 >>> compiler would do #:graft? #f and instead let =E2=80=98gexp->derivation= =E2=80=99 call >>> =E2=80=98graft-derivation=E2=80=99. >> >> The =E2=80=98wip-gexp-grafts=E2=80=99 branch does that. Namely, it=E2= =80=99s possible to know >> what grafts would apply to a gexp derivation build, so that one can >> first build ungrafted, and then apply the grafts to the results. >> So for a profile, we=E2=80=99d first build the profile as is, and only t= hen >> would we graft it. >> >> Right now the tip of this branch is a hack such that =E2=80=98guix packa= ge=E2=80=99: >> >> 1. Builds the original (ungrafted) derivation of the profile; >> >> 2. Manually calls =E2=80=98graft-derivation=E2=80=99 on that, passing = it the list of >> applicable grafts. >> >> Conceptually it=E2=80=99s what we want to do, but the drawback is that t= he >> caller (here =E2=80=98guix package=E2=80=99) goes through a lot of hops = to get the list >> of grafts and to apply it. >> >> I think we should instead have a way to annotate a derivation with a >> list of grafts, as well as a procedure to build that derivations in two >> phases (first the original derivation, then the grafts). > > The current iteration introduces =E2=80=9Cbuild continuation=E2=80=9D: a = derivation can > be annotated with a continuation, and the =E2=80=98build-things=E2=80=99 = procedure will > loop over continuations and return the final results (that only works > for derivations directly passed as an argument to =E2=80=98build-things= =E2=80=99.) On second thought, the whole idea of applying grafts on the final result (instead of applying grafts at each step like we do now) doesn=E2=80=99t fl= y. It works well for things like a profile or the system derivation, but breaks for less trivial things. For example, if you=E2=80=99re building a VM image or a binary tarball, you really need to graft packages early on; trying to graft the VM image or binary tarball wouldn=E2=80=99t have the desired effect. I=E2=80=99ve pushed =E2=80=98wip-gexp-grafts=E2=80=99 again but it seems to= be a dead end. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 11 05:51:36 2017 Received: (at 22990) by debbugs.gnu.org; 11 Jan 2017 10:51:36 +0000 Received: from localhost ([127.0.0.1]:49435 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cRGVE-0002lG-Ai for submit@debbugs.gnu.org; Wed, 11 Jan 2017 05:51:36 -0500 Received: from mail-qt0-f169.google.com ([209.85.216.169]:34444) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cRGVD-0002l1-5t for 22990@debbugs.gnu.org; Wed, 11 Jan 2017 05:51:35 -0500 Received: by mail-qt0-f169.google.com with SMTP id l7so142575291qtd.1 for <22990@debbugs.gnu.org>; Wed, 11 Jan 2017 02:51:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=craven.ch; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=fnR3BzWXg9zH+oz7DlIxF0nDKjqa9biKzR3J9MGA87c=; b=F/mMGKwa1YZR+mKT5PPquYNLky3fMTI4cfR4VFJ5zBWNbtzEPfCpugYMOkk1jKn2AW lWY7uscaNjbQhiOJfORBeRCFMMeN5e1NwzseEJ8/o57D54isbXz5W1iYIOwKCzMU12Ko qIFkbXQbBX2YMPqZd5AKF5fouWegQ/gya1Zk8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=fnR3BzWXg9zH+oz7DlIxF0nDKjqa9biKzR3J9MGA87c=; b=gjxboqGPvXK5PvP3UHmSaN6VpyPZA4mQ8CawBe0c8UcXcAbH2frjGMNnbciVY+HsG4 4b/YCxD76/pI2SDSLk8V1oU0/o4PQAuKLFKB10huxqiisabuq6JWndriqwCqPKSEki2q mzxiP+alZS7XqHU0ZBO1h9SEs5UpMhU1moRVeN4wB1KcYsuNHPVEY5TliGXt43+D9knO jRtN5j9RSWCT9N8xz2JjKR3JGzYIxcBhJIif+dGEEbMH4mIh3kR46nSdyXAIKK0/iIK8 NfzcO7Zk9+RhMvzXSpCdYHq8zfSFBbODJ4zGWVjm7FXbqG9ZRPQ4K+kdjXxC5aGz9t/F HZjg== X-Gm-Message-State: AIkVDXJ5PziZOzyq9ZC7JW8VRkzGpFu2otCCokl7Bnm5wmZwX+ekaPHmRPk39lUnh4ruu/1LKGt/v6ePY+BU1g== X-Received: by 10.237.59.186 with SMTP id r55mr6862998qte.22.1484131889543; Wed, 11 Jan 2017 02:51:29 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.133.38 with HTTP; Wed, 11 Jan 2017 02:51:29 -0800 (PST) X-Originating-IP: [151.248.146.13] In-Reply-To: <874m15yl99.fsf@gnu.org> References: <8737rxx8gk.fsf@gnu.org> <87k2a5rzue.fsf@gnu.org> <877f633kt1.fsf@gnu.org> <874m15yl99.fsf@gnu.org> From: David Craven Date: Wed, 11 Jan 2017 11:51:29 +0100 Message-ID: Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 22990 Cc: 22990@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) > On second thought, the whole idea of applying grafts on the final result > (instead of applying grafts at each step like we do now) doesn=E2=80=99t = fly. > It works well for things like a profile or the system derivation, but > breaks for less trivial things. > For example, if you=E2=80=99re building a VM image or a binary tarball, y= ou > really need to graft packages early on; trying to graft the VM image or > binary tarball wouldn=E2=80=99t have the desired effect. Isn't a system derivation or a profile derivation an intermediate step to these derivations? Can't there be a flag or something called #:already-grafted? #t? From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 11 16:19:33 2017 Received: (at 22990) by debbugs.gnu.org; 11 Jan 2017 21:19:33 +0000 Received: from localhost ([127.0.0.1]:50226 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cRQIv-00047s-Dd for submit@debbugs.gnu.org; Wed, 11 Jan 2017 16:19:33 -0500 Received: from eggs.gnu.org ([208.118.235.92]:49280) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cRQIt-00047e-2K for 22990@debbugs.gnu.org; Wed, 11 Jan 2017 16:19:31 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cRQIk-0000ft-2Y for 22990@debbugs.gnu.org; Wed, 11 Jan 2017 16:19:26 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:56705) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cRQIj-0000fo-Up; Wed, 11 Jan 2017 16:19:21 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:38774 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cRQIj-0003BI-Cg; Wed, 11 Jan 2017 16:19:21 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: David Craven Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> <87k2a5rzue.fsf@gnu.org> <877f633kt1.fsf@gnu.org> <874m15yl99.fsf@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 22 =?utf-8?Q?Niv=C3=B4se?= an 225 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Wed, 11 Jan 2017 22:19:19 +0100 In-Reply-To: (David Craven's message of "Wed, 11 Jan 2017 11:51:29 +0100") Message-ID: <87k2a1wazs.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -8.2 (--------) X-Debbugs-Envelope-To: 22990 Cc: 22990@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -8.2 (--------) David Craven skribis: >> On second thought, the whole idea of applying grafts on the final result >> (instead of applying grafts at each step like we do now) doesn=E2=80=99t= fly. >> It works well for things like a profile or the system derivation, but >> breaks for less trivial things. > >> For example, if you=E2=80=99re building a VM image or a binary tarball, = you >> really need to graft packages early on; trying to graft the VM image or >> binary tarball wouldn=E2=80=99t have the desired effect. > > Isn't a system derivation or a profile derivation an intermediate step > to these derivations? Yes, you=E2=80=99re right. However the implementation I had come up with r= elied on =E2=80=9Cbuild continuations=E2=80=9D, which only worked for the =E2=80= =9Ctop-level=E2=80=9D derivations (those you pass to =E2=80=98build-derivations=E2=80=99.) > Can't there be a flag or something called #:already-grafted? #t? Yeah, we need something like that. I need to chew a bit more on this to find a nice way to achieve that. Thanks for your feedback, I feel less lonely now. :-) Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat May 27 06:04:01 2017 Received: (at control) by debbugs.gnu.org; 27 May 2017 10:04:01 +0000 Received: from localhost ([127.0.0.1]:39910 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dEYZl-000391-9B for submit@debbugs.gnu.org; Sat, 27 May 2017 06:04:01 -0400 Received: from eggs.gnu.org ([208.118.235.92]:53728) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dEYZk-00038o-It for control@debbugs.gnu.org; Sat, 27 May 2017 06:04:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dEYZc-0007tG-Ao for control@debbugs.gnu.org; Sat, 27 May 2017 06:03:55 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:58345) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dEYZc-0007tC-74 for control@debbugs.gnu.org; Sat, 27 May 2017 06:03:52 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:55280 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dEYZb-0007C9-KC for control@debbugs.gnu.org; Sat, 27 May 2017 06:03:52 -0400 Date: Sat, 27 May 2017 12:03:48 +0200 Message-Id: <87tw46vcor.fsf@gnu.org> To: control@debbugs.gnu.org From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: control message for bug #22990 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) severity 22990 important From ludo@gnu.org Sun Mar 29 09:41:32 2020 Received: (at 22990-done) by debbugs.gnu.org; 30 Mar 2020 02:35:38 +0000 Received: from eggs.gnu.org ([209.51.188.92]:43540) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jIYBv-00022g-LX for 22990-done@debbugs.gnu.org; Sun, 29 Mar 2020 09:41:32 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51035) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1jIYBq-0008D6-G0 for 22990-done@debbugs.gnu.org; Sun, 29 Mar 2020 09:41:26 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49626 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jIYBp-00023G-R7 for 22990-done@debbugs.gnu.org; Sun, 29 Mar 2020 09:41:26 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 22990-done@debbugs.gnu.org Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval References: <8737rxx8gk.fsf@gnu.org> Date: Sun, 29 Mar 2020 15:41:24 +0200 In-Reply-To: <8737rxx8gk.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 11 Mar 2016 17:52:59 +0100") Message-ID: <87tv271dob.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 22990-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Mar 2020 13:41:32 -0000 Hi younger self! ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > As of right now (v0.9.0-2007-g66a30a3), =E2=80=98graft-derivation=E2=80= =99 works either by: > > 1. Fetching substitute info about the things being built so that it > can determine its references, which in turns allows it to determine > whether they need to be grafted. > > 2. Building stuff, as a last resort, so that it can determine its > references. > > Case #1 is hopefully going to be the most common. > > The problem with #1 is that when building a profile, we do one > =E2=80=98package-derivation=E2=80=99 call for each package in the profile= , which > translates in one =E2=80=98graft-derivation=E2=80=99 call for each releva= nt package=C2=B9, > which translates into one =E2=80=98references/substitutes=E2=80=99 call f= or each. > > Concretely, what this means is this: > > $ guix package -u > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > substitute: updating list of substitutes from 'http://mirror.guixsd.org= '... 100.0% > [=E2=80=A6] > The following files would be downloaded: > > Each of the initial =E2=80=9Cupdating list=E2=80=9D message corresponds t= o an HTTP > request for a single narinfo file, which can take around 1 second. This is finally fixed with commit 710854304b1ab29332edcb76f3de532e0724c197, as discussed earlier this week=C2=B9. \o/ The solution is simpler and more orthogonal than what was envisioned earlier in this thread, which is nice. I also potentially addresses other uses of =E2=80=9Cdynamic dependencies=E2= =80=9D elsewhere in the code base, such as =E2=80=98remote-eval=E2=80=99 calls in = =E2=80=98guix deploy=E2=80=99. Feedback welcome! Ludo=E2=80=99. =C2=B9 https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00337.html From unknown Sat Jun 21 10:27:01 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 27 Apr 2020 11:24:08 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator