From unknown Sat Jun 14 18:05:47 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#48435 <48435@debbugs.gnu.org> To: bug#48435 <48435@debbugs.gnu.org> Subject: Status: [PATCH] Start enabling substitutes from bayfront. Reply-To: bug#48435 <48435@debbugs.gnu.org> Date: Sun, 15 Jun 2025 01:05:47 +0000 retitle 48435 [PATCH] Start enabling substitutes from bayfront. reassign 48435 guix-patches submitter 48435 Christopher Baines severity 48435 normal tag 48435 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Sat May 15 06:08:42 2021 Received: (at submit) by debbugs.gnu.org; 15 May 2021 10:08:42 +0000 Received: from localhost ([127.0.0.1]:47648 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhrDo-0007Y0-HR for submit@debbugs.gnu.org; Sat, 15 May 2021 06:08:42 -0400 Received: from lists.gnu.org ([209.51.188.17]:58790) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhrDj-0007Xp-Bf for submit@debbugs.gnu.org; Sat, 15 May 2021 06:08:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34606) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhrDj-00040A-44 for guix-patches@gnu.org; Sat, 15 May 2021 06:08:31 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:39657) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhrDd-0000SJ-Pl for guix-patches@gnu.org; Sat, 15 May 2021 06:08:30 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 14FC927BC78 for ; Sat, 15 May 2021 11:08:23 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 15048380 for ; Sat, 15 May 2021 10:08:22 +0000 (UTC) From: Christopher Baines To: guix-patches@gnu.org Subject: [PATCH] Start enabling substitutes from bayfront. Date: Sat, 15 May 2021 11:08:22 +0100 Message-Id: <20210515100822.11784-1-mail@cbaines.net> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) In addition to substitutes from ci.guix.gnu.org. There are more changes that can be made in the future, but these changes seem like a good start. * etc/substitutes/bayfront.guix.gnu.org.pub: New file. * Makefile.am (dist_pkgdata_DATA): Add it. * config-daemon.ac (guix_substitute_urls): Add https://bayfront.guix.gnu.org. * guix/scripts/substitute.scm (%default-substitute-urls): Add http://bayfront.guix.gnu.org. * guix/store.scm (%default-substitute-urls): Add bayfront.guix.gnu.org. --- Makefile.am | 3 ++- config-daemon.ac | 2 +- etc/substitutes/bayfront.guix.gnu.org.pub | 6 ++++++ guix/scripts/substitute.scm | 3 ++- guix/store.scm | 3 ++- 5 files changed, 13 insertions(+), 4 deletions(-) create mode 100644 etc/substitutes/bayfront.guix.gnu.org.pub diff --git a/Makefile.am b/Makefile.am index 4cbef5a6e6..bad17e59fd 100644 --- a/Makefile.am +++ b/Makefile.am @@ -584,7 +584,8 @@ check-system: $(GOBJECTS) dist_pkgdata_DATA = \ etc/substitutes/berlin.guix.gnu.org.pub \ etc/substitutes/ci.guix.gnu.org.pub \ - etc/substitutes/ci.guix.info.pub + etc/substitutes/ci.guix.info.pub \ + etc/substitutes/bayfront.guix.gnu.org.pub # Bash completion file. dist_bashcompletion_DATA = etc/completion/bash/guix \ diff --git a/config-daemon.ac b/config-daemon.ac index 85caee269b..ee2ffbff42 100644 --- a/config-daemon.ac +++ b/config-daemon.ac @@ -117,7 +117,7 @@ if test "x$guix_build_daemon" = "xyes"; then dnl Determine the appropriate default list of substitute URLs (GnuTLS dnl is required so we can default to 'https'.) - guix_substitute_urls="https://ci.guix.gnu.org" + guix_substitute_urls="https://ci.guix.gnu.org https://bayfront.guix.gnu.org" AC_MSG_CHECKING([for default substitute URLs]) AC_MSG_RESULT([$guix_substitute_urls]) diff --git a/etc/substitutes/bayfront.guix.gnu.org.pub b/etc/substitutes/bayfront.guix.gnu.org.pub new file mode 100644 index 0000000000..0092301e5a --- /dev/null +++ b/etc/substitutes/bayfront.guix.gnu.org.pub @@ -0,0 +1,6 @@ +(public-key + (ecc + (curve Ed25519) + (q #7D602902D3A2DBB83F8A0FB98602A754C5493B0B778C8D1DD4E0F41DE14DE34F#) + ) + ) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 8e4eae00b3..2fa2b1e219 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -638,7 +638,8 @@ found." (#f ;; This can only happen when this script is not invoked by the ;; daemon. - '("http://ci.guix.gnu.org")))) + '("http://ci.guix.gnu.org" + "http://bayfront.guix.gnu.org")))) ;; In order to prevent using large number of discovered local substitute ;; servers, limit the local substitute urls list size. diff --git a/guix/store.scm b/guix/store.scm index 9d706ae590..8fca37283f 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -783,7 +783,8 @@ encoding conversion errors." (map (if (false-if-exception (resolve-interface '(gnutls))) (cut string-append "https://" <>) (cut string-append "http://" <>)) - '("ci.guix.gnu.org"))) + '("ci.guix.gnu.org" + "bayfront.guix.gnu.org"))) (define (current-user-name) "Return the name of the calling user." -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sat May 15 07:01:20 2021 Received: (at 48435) by debbugs.gnu.org; 15 May 2021 11:01:20 +0000 Received: from localhost ([127.0.0.1]:47699 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhs2q-0002wM-4W for submit@debbugs.gnu.org; Sat, 15 May 2021 07:01:20 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47338) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhs2m-0002w3-Be for 48435@debbugs.gnu.org; Sat, 15 May 2021 07:01:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40620) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhs2g-0000RE-E6; Sat, 15 May 2021 07:01:10 -0400 Received: from [2a01:e34:ed27:e500:a880:d241:574d:84e1] (port=41396 helo=meije) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhs2f-0006Lt-S4; Sat, 15 May 2021 07:01:10 -0400 From: Mathieu Othacehe To: Christopher Baines Subject: Re: [bug#48435] [PATCH] Start enabling substitutes from bayfront. References: <20210515100822.11784-1-mail@cbaines.net> Date: Sat, 15 May 2021 13:01:07 +0200 In-Reply-To: <20210515100822.11784-1-mail@cbaines.net> (Christopher Baines's message of "Sat, 15 May 2021 11:08:22 +0100") Message-ID: <874kf4jm6k.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 48435 Cc: 48435@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello Chris, > + guix_substitute_urls="https://ci.guix.gnu.org https://bayfront.guix.gnu.org" What is the rationale behind adding a new substitution server? I feel like having two substitute servers will make things more complex in term of maintenance. Having both servers compute the same set of substitutes is also not great from an energetic and resource saving point of view. Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Sat May 15 08:20:58 2021 Received: (at 48435) by debbugs.gnu.org; 15 May 2021 12:20:58 +0000 Received: from localhost ([127.0.0.1]:47815 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhtHu-0007Jk-6a for submit@debbugs.gnu.org; Sat, 15 May 2021 08:20:58 -0400 Received: from mira.cbaines.net ([212.71.252.8]:37436) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhtHs-0007Jb-0r for 48435@debbugs.gnu.org; Sat, 15 May 2021 08:20:56 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 1E52E27BC78; Sat, 15 May 2021 13:20:55 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 7939a586; Sat, 15 May 2021 12:20:54 +0000 (UTC) References: <20210515100822.11784-1-mail@cbaines.net> <874kf4jm6k.fsf@gnu.org> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines To: Mathieu Othacehe Subject: Re: [bug#48435] [PATCH] Start enabling substitutes from bayfront. In-reply-to: <874kf4jm6k.fsf@gnu.org> Date: Sat, 15 May 2021 13:20:49 +0100 Message-ID: <87sg2ochni.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 48435 Cc: 48435@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Mathieu Othacehe writes: > Hello Chris, > >> + guix_substitute_urls=3D"https://ci.guix.gnu.org https://bayfront.guix= .gnu.org" > > What is the rationale behind adding a new substitution server? I feel > like having two substitute servers will make things more complex in term > of maintenance. > > Having both servers compute the same set of substitutes is also not > great from an energetic and resource saving point of view. Hey, I should have probably written a cover letter, but this patch is me starting to try and work out the changes involved in getting substitutes from bayfront to general Guix users, but the discussion has been happening in this thread [1]. 1: https://lists.gnu.org/archive/html/guix-devel/2021-05/msg00241.html Bayfront has been around for a while, and has been serving substitutes, although I'm not sure it's provided much value to users in that time. I hope that can change with switching to using the Guix Build Coordinator though, that happened around a month ago, and it's slowly building things and catching up. I guess there's a greater need to maintain it if starts getting used by more users, so I do think the maintenance involved is something to consider. Personally, I see the arguments for having multiple substitute servers getting stronger over time. Multiple independent substitute servers would provide more reliability than a single source, as well as enabling things like K of N trust in substitutes [2]. 2: https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00179.html In terms of energy and resources, currently there are 5 machines in use, most of which were mostly idle before being put to use building things for substitutes. While having them build things does use more power than having them idle, I think the value provided, even if that's providing exactly the same bytes as ci.guix.gnu.org, is worth the cost, for the reasons I give above. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCfvKFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XerPw//S8ti4XHBYxhCMY0vDLuM16HkdrLtgt82 zwQbOCg6DbrKPLlL5LItz+myxKzOuV3a0yubOKg+kQ0AaUO9+ANKUNBi0WSAKgon 6vlxo7l7RI1+HnRpixuKnUOAUdR8YEtx9cD9ttlhATx+FKYTFzwa1Sh8ZpSFmOUL bWNcAgSzZymqOoM9H/xyaR4QzO3n4V8vVK390WCi2Z9oq/h8NVgr6PA4EW6WFOFX G28WmmQ1qVPtoYx3m95mb4ixw0AGDo+qkTYhCGf1hIUIM8Yoa8E4a70hYQw4rzFa FygjG37pY4nJhtI+qUK+n8yDfgm2JAw555cPhgghkqOQJtW0arogvqE1oKNM19JR po4ObOH6bd/X9oq+fEgbJttEzSHjaKotFe70odURuwoYGLhKNX71kxyLgUbbBazE RBM5T9rNPlzeb9CrFD63MK6pJFt+tkGRfQRvP7V6Nqr+jcS+vCSmESg6phWkJZtq E3LbSK28wJlOZzsrFGoMIQF8Cli7MUTPqc1SPUPS/KcMhTh3EHfuQjw5mexCWZOE lou3Qq6/t1As9mywZt66LUTjJ0HHtsAcyLqFgpUJrOr0fIB3yB2nUcQ9GEN853ma kknqiTDr/aiUKZyHSjm2y8O8W1QSXBzYsx2mYqkNMUbG90lSu4bWdSV8JlsZa9Ru H9bWzE6kY9I= =AHnt -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue May 18 15:45:58 2021 Received: (at 48435) by debbugs.gnu.org; 18 May 2021 19:45:58 +0000 Received: from localhost ([127.0.0.1]:57357 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lj5fC-0007ma-1u for submit@debbugs.gnu.org; Tue, 18 May 2021 15:45:58 -0400 Received: from mira.cbaines.net ([212.71.252.8]:41074) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lj5f9-0007mR-B2 for 48435@debbugs.gnu.org; Tue, 18 May 2021 15:45:56 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 1035027BC78; Tue, 18 May 2021 20:45:54 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 20355396; Tue, 18 May 2021 19:45:53 +0000 (UTC) References: <878s4ye116.fsf@cbaines.net> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines To: guix-devel@gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= , Andreas Enge , 48435@debbugs.gnu.org Subject: Re: Bringing substitutes from the Guix Build Coordinator to users In-reply-to: <878s4ye116.fsf@cbaines.net> Date: Tue, 18 May 2021 20:45:50 +0100 Message-ID: <87lf8bbzbl.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 48435 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Christopher Baines writes: > Is there still a path to bring some of these benefits to users, and if > so, what things need doing? It's been a few weeks now, so to summarise, I think only one path emerged, and that is to get substitutes from bayfront to users. Bayfront was already running the Guix Build Coordinator (although only for the last month), and it's now caught up to the point where I'm seeing similar or better substitute availability percentages for x86_64-linux (and powerpc64le-linux) when compared to ci.guix.gnu.org. It's also building i686-linux and aarch64-linux things, but they're still catching up. Obviously just having the substitutes doesn't magically get them to users, so I've started looking in to the changes to start making that happen. Adding the signing key and changing the defaults in a few places seems like a good step forward [1]. 1: https://issues.guix.gnu.org/48435 I want to push on with this within the next couple of weeks, mostly so I can shift focus to Outreachy and the security related tooling work, but also because I still think this will be a good step forward in terms of substitute availability for users. It's been over a year now since implementation started, so it would be good to actually make a positive difference. There's a few issues still on my mind. Even though the substitute availability percentages are good when compared to ci.guix.gnu.org, as bayfront has much less compute power connected, it might not keep up as well when big sets of changes are merged. I think that's just an argument for using the build coordinator on berlin and the connected machines though. The other thing in comparison to ci.guix.gnu.org is that bayfront only has ~4TB of storage rather than ~37TB, and given that currently none of the generated nars are deleted, that will need thinking about in a few months to avoid running out of space. I've had some plans around this for a while [1], but they still require implementing. 1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html Apart from merging the changes in [1], I guess a blog post might be useful. Have I missed anything? Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCkGW5fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XesURAAnDB83dBydNri1IFhkMUDhRsr9nyDaEeM TLv3Uo9t3ntpLNp3EBcGTdofUM2TP3XyIK2BpvtjCBbkzgdviWm3+KUW04FNB8rn Ri2EWyTxCU3XSZ+jZsWEUpPvyv8W4npP9sG46f9+6d/1DIApgcJaNUTW4VYXjcOJ P4h109WlRssjuHhzX7EZ3+az5pFrk3ZZwJaJxDj7+zHuFoMsnfnTOpS8PfN0718Q hJC+KU0sXLn7Mp+OSzGuceSaTo+V9iGq2BpMpnmaPxPl5W89w0O3YnUniPxY79YK WoX5nk43h40ZDdkhzj2qQ44yzn3T1DLvYWzHslvh1Qd+g6FWT6WbaXgo0GgI6E8C rdnDrgTb8W8vKGxsM9ptHNOiuHk75InMgaU3DasOkw+vEIYrTaM09oUb9MtJXYmp TRKGTbQ34tc647n90Vdn0e/6fwpt67H3LlKuE/Jw45hoXjpYKFNLoa6yhJUFu12q 9RwcabKLuY8A/2gVvJT6R+X1wOoUPS6AQF7MUdKkU1/pjTw/iFbOxWMoLmQ3OmZ5 c+iv1IbPcyyb/9U7dYQFHMozcxIUV4Z3KXRD0Cr7zXiffXI60ZS46YXEblmLDkKM 1qCQQoUFtKdK5CUQrfI1TB5t8nbjKHqbC5LEmpbXFOVEDLEI2WxLjU/QHQAPppXq oEueR3Pj+oE= =5y08 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue May 18 17:24:48 2021 Received: (at 48435) by debbugs.gnu.org; 18 May 2021 21:24:48 +0000 Received: from localhost ([127.0.0.1]:57478 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lj7Cq-00065f-1a for submit@debbugs.gnu.org; Tue, 18 May 2021 17:24:48 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36826) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lj7Cm-00065Q-7W for 48435@debbugs.gnu.org; Tue, 18 May 2021 17:24:46 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52638) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lj7Cg-0000yl-Ou; Tue, 18 May 2021 17:24:38 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=37496 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lj7Cg-00020k-Eu; Tue, 18 May 2021 17:24:38 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: Bringing substitutes from the Guix Build Coordinator to users References: <878s4ye116.fsf@cbaines.net> <87lf8bbzbl.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 29 =?utf-8?Q?Flor=C3=A9al?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 18 May 2021 23:24:36 +0200 In-Reply-To: <87lf8bbzbl.fsf@cbaines.net> (Christopher Baines's message of "Tue, 18 May 2021 20:45:50 +0100") Message-ID: <87wnrv68h7.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 48435 Cc: guix-devel@gnu.org, Andreas Enge , 48435@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello! Christopher Baines skribis: > Christopher Baines writes: > >> Is there still a path to bring some of these benefits to users, and if >> so, what things need doing? [...] > Obviously just having the substitutes doesn't magically get them to > users, so I've started looking in to the changes to start making that > happen. Adding the signing key and changing the defaults in a few places > seems like a good step forward [1]. > > 1: https://issues.guix.gnu.org/48435 > > I want to push on with this within the next couple of weeks, mostly so I > can shift focus to Outreachy and the security related tooling work, but > also because I still think this will be a good step forward in terms of > substitute availability for users. It's been over a year now since > implementation started, so it would be good to actually make a positive > difference. I=E2=80=99m fine with distributing an extra signing key alongside that of ci.guix.gnu.org. I=E2=80=99m unsure about having two substitute URLs by default since it add= s a bit of overhead, though that overhead is only upon cache misses (I have that setup on my laptop actually). It=E2=80=99s also a one-way change: people are likely to keep the defaults =E2=80=9Cforever=E2=80=9D. So we can=E2=80=99t just =E2=80=9Cexperiment=E2= =80=9D and change our mind later. That means we should at least have a DNS entry that=E2=80=99s not tied to a particular machine, like ci2.guix.gnu.org or whatever. WDYT? Now, what would be nice is to have a second build farm with the K-out-of-N policy you mention in mind. > There's a few issues still on my mind. Even though the substitute > availability percentages are good when compared to ci.guix.gnu.org, as > bayfront has much less compute power connected, it might not keep up as > well when big sets of changes are merged. I think that's just an > argument for using the build coordinator on berlin and the connected > machines though. As much as I=E2=80=99d have preferred a single solution in this area, fueli= ng competition between the Coordinator and Cuirass and their access to official infrastructure doesn=E2=80=99t seem like a viable path to me. I think the primary value in having a second build farm would be reproducibility and doing away with the single point of failure. Overall substitute coverage probably wouldn=E2=80=99t change much. I agree with Mathieu that maintaining it has a cost, but maybe we can try. I realize I=E2=80=99m asking questions rather than providing answers, which= may be because I don=E2=80=99t see a clear path ahead. :-) Thanks! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue May 18 18:30:00 2021 Received: (at 48435) by debbugs.gnu.org; 18 May 2021 22:30:00 +0000 Received: from localhost ([127.0.0.1]:57543 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lj8Dw-0007ct-48 for submit@debbugs.gnu.org; Tue, 18 May 2021 18:30:00 -0400 Received: from mira.cbaines.net ([212.71.252.8]:41476) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lj8Ds-0007ci-2Z for 48435@debbugs.gnu.org; Tue, 18 May 2021 18:29:58 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 0E1B927BC78; Tue, 18 May 2021 23:29:55 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 424a449e; Tue, 18 May 2021 22:29:54 +0000 (UTC) References: <878s4ye116.fsf@cbaines.net> <87lf8bbzbl.fsf@cbaines.net> <87wnrv68h7.fsf@gnu.org> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Bringing substitutes from the Guix Build Coordinator to users In-reply-to: <87wnrv68h7.fsf@gnu.org> Date: Tue, 18 May 2021 23:29:52 +0100 Message-ID: <87im3fbrq7.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 48435 Cc: guix-devel@gnu.org, Andreas Enge , 48435@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hello! > > Christopher Baines skribis: > >> Christopher Baines writes: >> >>> Is there still a path to bring some of these benefits to users, and if >>> so, what things need doing? > > [...] > >> Obviously just having the substitutes doesn't magically get them to >> users, so I've started looking in to the changes to start making that >> happen. Adding the signing key and changing the defaults in a few places >> seems like a good step forward [1]. >> >> 1: https://issues.guix.gnu.org/48435 >> >> I want to push on with this within the next couple of weeks, mostly so I >> can shift focus to Outreachy and the security related tooling work, but >> also because I still think this will be a good step forward in terms of >> substitute availability for users. It's been over a year now since >> implementation started, so it would be good to actually make a positive >> difference. > > I=E2=80=99m fine with distributing an extra signing key alongside that of > ci.guix.gnu.org. Great. > I=E2=80=99m unsure about having two substitute URLs by default since it a= dds a > bit of overhead, though that overhead is only upon cache misses (I have > that setup on my laptop actually). All of this work has been built on the assumption that it's possible to do better in providing substitutes, and anecdotally from the data I've seen over the last year, that should be possible, even with the limited hardware (compared to ci.guix.gnu.org) connected to bayfront. So yes, that's a valid concern, but if all the addition of bayfront does is make users wait a little longer because of cache misses, it's a sign that the whole endeavour is not working out. > It=E2=80=99s also a one-way change: people are likely to keep the defaults > =E2=80=9Cforever=E2=80=9D. So we can=E2=80=99t just =E2=80=9Cexperiment= =E2=80=9D and change our mind later. > That means we should at least have a DNS entry that=E2=80=99s not tied to= a > particular machine, like ci2.guix.gnu.org or whatever. That sounds sensible. On the specific name, given this is just about substitutes, and at least in my opinion has nothing to do with continuous integration, maybe picking just another word would avoid thinking too much, it could be bordeaux, or hippo, or anything really. As you say, stability and not being tied to a particular machine is the important thing. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCkP+BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdMWQ/8DxpIzlA5N/TgdeqrcgcvPDVNwZ1Oqk3b qwI2nL/rb2gM7HXZf+HC1q8YfrNZHctxPBoTqUYm8PnXb24yTdJRElcx1pW2Fjkq +TwZLII4SYrxJzVdUSYHtb1DMkJrAu7ern+WOi44zD51zAF8qSjR6joP2dv7XjJl M6TW4X8DWRl4GwRpF1VGvFk2h4Zx4qTJgg7T3oOw++AJ1A4oOWrIOe5Jphepfrgv TBTUSRKrBLqHeZ8ilQTBzeq7lVmRT1htrNC+Y3D8r0gAwcKs4+fxQY0Nr8yWQvil HsttpbLvjxopeLpHTRAyBdYXGtlw9zctNIzHLEJyHb9hfrjeEWNQ0gp2MAtqOUlE 2VHw8YPzQrLmZJzi0dPtYsiALTBnExa0DYoZPpwv1uNGN8aurA4YdQeRhOpVlD6u cMteS5Rj8xU16yM0z021iSsGgc1dCFlPXg6Kh7M+eFnchyzu5QXB8frnOWwSCjtd 0F71LUcAL60BHS+DCPT5APxTWOCo1Bzdq+wOB6Tl62QarbYY1PJgYatq7tkE+awo 7AwuulAb3+u1D8o2fEwdtxVP29gx9VCSJLkagEaj7r9I4SMQ1bN57LM8gQTivXA5 72V7xz0Pg8zsmhgc9rVozkZjilUPeV6v2b5uKd2EnfE87o1+7HPimADqEbLKBPqw /AjA5zIZYW8= =XHRM -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed May 19 02:54:19 2021 Received: (at 48435) by debbugs.gnu.org; 19 May 2021 06:54:19 +0000 Received: from localhost ([127.0.0.1]:57837 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljG5z-0002tT-Dx for submit@debbugs.gnu.org; Wed, 19 May 2021 02:54:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38828) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljG5x-0002tH-Ux for 48435@debbugs.gnu.org; Wed, 19 May 2021 02:54:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40024) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ljG5s-0006yi-98; Wed, 19 May 2021 02:54:12 -0400 Received: from [2a01:e0a:19b:d9a0:45b5:a14a:5c75:5737] (port=53846 helo=meije) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ljG5r-0008N7-TM; Wed, 19 May 2021 02:54:12 -0400 From: Mathieu Othacehe To: Christopher Baines Subject: Re: Bringing substitutes from the Guix Build Coordinator to users References: <878s4ye116.fsf@cbaines.net> <87lf8bbzbl.fsf@cbaines.net> <87wnrv68h7.fsf@gnu.org> <87im3fbrq7.fsf@cbaines.net> Date: Wed, 19 May 2021 08:54:10 +0200 In-Reply-To: <87im3fbrq7.fsf@cbaines.net> (Christopher Baines's message of "Tue, 18 May 2021 23:29:52 +0100") Message-ID: <87eee3tdrh.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 48435 Cc: guix-devel@gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= , 48435@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hey Chris, > That sounds sensible. On the specific name, given this is just about > substitutes, and at least in my opinion has nothing to do with > continuous integration, maybe picking just another word would avoid > thinking too much, it could be bordeaux, or hippo, or anything > really. As you say, stability and not being tied to a particular machine > is the important thing. The substitutes coverage is one indicator to take into account but there are many others. For instance, the evaluation speed, the failed evaluation count, the average evaluation builds completion time, the availability of the connected build machines between other things. Deploying a solution that builds substitutes is fine, but as soon as it is deployed and accessible to all Guix users, the system administrators will have to monitor it and maintain it in the long run. Having two heterogeneous build infrastructures on two sets of machines, providing different metrics will make the update and maintenance of those machines harder. I hear your point about K-out-of-N policy and it also makes sense to me. However, we should maybe consider doing it using two similar infrastructures. Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Wed May 19 03:58:03 2021 Received: (at 48435) by debbugs.gnu.org; 19 May 2021 07:58:03 +0000 Received: from localhost ([127.0.0.1]:57928 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljH5f-0006cr-KY for submit@debbugs.gnu.org; Wed, 19 May 2021 03:58:03 -0400 Received: from mira.cbaines.net ([212.71.252.8]:41576) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljH5d-0006cP-2b for 48435@debbugs.gnu.org; Wed, 19 May 2021 03:58:02 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 13D2127BC78; Wed, 19 May 2021 08:58:00 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 51fb0d00; Wed, 19 May 2021 07:57:58 +0000 (UTC) References: <878s4ye116.fsf@cbaines.net> <87lf8bbzbl.fsf@cbaines.net> <87wnrv68h7.fsf@gnu.org> <87im3fbrq7.fsf@cbaines.net> <87eee3tdrh.fsf@gnu.org> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines To: Mathieu Othacehe Subject: Re: Bringing substitutes from the Guix Build Coordinator to users In-reply-to: <87eee3tdrh.fsf@gnu.org> Date: Wed, 19 May 2021 08:57:56 +0100 Message-ID: <87fsyjb1ff.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 48435 Cc: guix-devel@gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= , 48435@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Mathieu Othacehe writes: > Hey Chris, > >> That sounds sensible. On the specific name, given this is just about >> substitutes, and at least in my opinion has nothing to do with >> continuous integration, maybe picking just another word would avoid >> thinking too much, it could be bordeaux, or hippo, or anything >> really. As you say, stability and not being tied to a particular machine >> is the important thing. > > The substitutes coverage is one indicator to take into account but there > are many others. For instance, the evaluation speed, the failed > evaluation count, the average evaluation builds completion time, the > availability of the connected build machines between other things. Indeed, and I'm aware that the Guix Data Service, which performs a similar function to the evaluations in Cuirass, is much slower. > Deploying a solution that builds substitutes is fine, but as soon as it > is deployed and accessible to all Guix users, the system administrators > will have to monitor it and maintain it in the long run. > > Having two heterogeneous build infrastructures on two sets of machines, > providing different metrics will make the update and maintenance of > those machines harder. > > I hear your point about K-out-of-N policy and it also makes sense to > me. However, we should maybe consider doing it using two similar > infrastructures. Indeed. The reality though is that two different approaches have been in development now for a little over a year, and this is a reflection of that. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCkxQRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XeEURAApmgklmUDbZqn5sm5oIsaVHhdSg/frkNu n7FhpwMLN3FdADtVfQeDWqM23PwoWBWmkAA5h1xu8mxUF7kw3ND3pN4uWt5XZ948 OeR5Z4Wx6TAn9eQb5OUjWAqmhrHkKD1+W2TXONF77FRTcT7W9P5zuBYAk/bd3mOb zQB8Nw3j7fva9L2STTWHPrPHZCoDKLWTcVVOguMkT/uSZMV9/DswWmmPJ6qmfu+k ACPzfXr91scPiXLVAEOnweBE/ITwRGUw90v3QuP4vfWp/nkwLyRfdrcladNxNZwr Dq4NECglfcxj6zNsXLFiK7qru6tBtVFO7ZPvhiR91AFCh6IbXTu1WfcGXfDa4WKS fpEqB4Yzx9JV7/V5M/xrzodcxrdlB0q07CGrpqxXW8cxxOOKf0Wb/myWgUi+TLbD 7tEMd8Kp/5lMmsuRHrSM/rREqCGzS9DsvBR6s1BQjLzWnKZ8aE5rS7g8s0y2Wi5e T9NXu8/GLLYsZbI555+xbpmGM/zTKSLbbZ8u9HWlZtgeKJGq1trtxThDFnE67L2R /zEH6rxo8Wndl44aY4Yi32Nn9ZGGFgV7QHJSkOp46h8UA863xLMIyt30360t1Miy sJXpS7HbXED1slUtM8cKVBzIXmSYjjpedNel1OPAvqq4vDEgbKR0YaUFDGweWb66 337znit9PGo= =eS5v -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 07 07:07:40 2021 Received: (at 48435) by debbugs.gnu.org; 7 Jun 2021 11:07:40 +0000 Received: from localhost ([127.0.0.1]:54543 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lqD6a-0001v9-9E for submit@debbugs.gnu.org; Mon, 07 Jun 2021 07:07:40 -0400 Received: from mira.cbaines.net ([212.71.252.8]:37262) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lqD6Y-0001v0-Fc for 48435@debbugs.gnu.org; Mon, 07 Jun 2021 07:07:39 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 1BBBE27BC78 for <48435@debbugs.gnu.org>; Mon, 7 Jun 2021 12:07:37 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id cff6cefd for <48435@debbugs.gnu.org>; Mon, 7 Jun 2021 11:07:36 +0000 (UTC) From: Christopher Baines To: 48435@debbugs.gnu.org Subject: [PATCH v2] Start enabling substitutes from bordeaux.guix.gnu.org. Date: Mon, 7 Jun 2021 12:07:36 +0100 Message-Id: <20210607110736.23894-1-mail@cbaines.net> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 48435 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) In addition to substitutes from ci.guix.gnu.org. There are more changes that can be made in the future, but these changes seem like a good start. * config-daemon.ac (guix_substitute_urls): Add https://bordeaux.guix.gnu.org. * guix/scripts/substitute.scm (%default-substitute-urls): Add http://bordeaux.guix.gnu.org. * guix/store.scm (%default-substitute-urls): Add bordeaux.guix.gnu.org. --- config-daemon.ac | 2 +- guix/scripts/substitute.scm | 3 ++- guix/store.scm | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/config-daemon.ac b/config-daemon.ac index 85caee269b..5ddc740600 100644 --- a/config-daemon.ac +++ b/config-daemon.ac @@ -117,7 +117,7 @@ if test "x$guix_build_daemon" = "xyes"; then dnl Determine the appropriate default list of substitute URLs (GnuTLS dnl is required so we can default to 'https'.) - guix_substitute_urls="https://ci.guix.gnu.org" + guix_substitute_urls="https://ci.guix.gnu.org https://bordeaux.guix.gnu.org" AC_MSG_CHECKING([for default substitute URLs]) AC_MSG_RESULT([$guix_substitute_urls]) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 44448ff3e9..3ea1c73e10 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -643,7 +643,8 @@ found." (#f ;; This can only happen when this script is not invoked by the ;; daemon. - '("http://ci.guix.gnu.org")))) + '("http://ci.guix.gnu.org" + "http://bordeaux.guix.gnu.org")))) ;; In order to prevent using large number of discovered local substitute ;; servers, limit the local substitute urls list size. diff --git a/guix/store.scm b/guix/store.scm index cf5d5eeccc..eaf422b615 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -775,7 +775,8 @@ encoding conversion errors." (map (if (false-if-exception (resolve-interface '(gnutls))) (cut string-append "https://" <>) (cut string-append "http://" <>)) - '("ci.guix.gnu.org"))) + '("ci.guix.gnu.org" + "bordeaux.guix.gnu.org"))) (define (current-user-name) "Return the name of the calling user." -- 2.31.1 From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 07 10:53:20 2021 Received: (at 48435) by debbugs.gnu.org; 7 Jun 2021 14:53:20 +0000 Received: from localhost ([127.0.0.1]:56424 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lqGcu-0001ix-HK for submit@debbugs.gnu.org; Mon, 07 Jun 2021 10:53:20 -0400 Received: from mira.cbaines.net ([212.71.252.8]:37468) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lqGcp-0001ik-CC for 48435@debbugs.gnu.org; Mon, 07 Jun 2021 10:53:15 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 686A727BC78; Mon, 7 Jun 2021 15:53:10 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 155aa5de; Mon, 7 Jun 2021 14:53:09 +0000 (UTC) References: <878s4ye116.fsf@cbaines.net> <87lf8bbzbl.fsf@cbaines.net> User-agent: mu4e 1.4.15; emacs 27.2 From: Christopher Baines To: guix-devel@gnu.org Subject: Re: Bringing substitutes from the Guix Build Coordinator to users In-reply-to: <87lf8bbzbl.fsf@cbaines.net> Date: Mon, 07 Jun 2021 15:53:07 +0100 Message-ID: <87k0n5u3os.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 48435 Cc: 48435@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Christopher Baines writes: > Christopher Baines writes: > >> Is there still a path to bring some of these benefits to users, and if >> so, what things need doing? > > It's been a few weeks now, so to summarise, I think only one path > emerged, and that is to get substitutes from bayfront to users. More weeks have past, it's taking me longer to get things sorted out that I'd like, but things are still moving forward. > Bayfront was already running the Guix Build Coordinator (although only > for the last month), and it's now caught up to the point where I'm > seeing similar or better substitute availability percentages for > x86_64-linux (and powerpc64le-linux) when compared to > ci.guix.gnu.org. It's also building i686-linux and aarch64-linux things, > but they're still catching up. Substitute availability for x86_64-linux and i686-linux should be roughly comparable to ci.guix.gnu.org. powerpc64le-linux substitute availability is OK, and aarch64-linux might even be doing better than ci.guix.gnu.org somehow. Other things like armhf-linux and i586-gnu are still very much works in progress. > Obviously just having the substitutes doesn't magically get them to > users, so I've started looking in to the changes to start making that > happen. Adding the signing key and changing the defaults in a few places > seems like a good step forward [1]. > > 1: https://issues.guix.gnu.org/48435 I've gone ahead and put the key in to the Guix git repository [1] and sent an updated patch for changing various bits of configuration [2]. 1: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=26499816a973b3aab9aaf8e13b909d0bde4e2dd5 2: https://issues.guix.gnu.org/48435#8 I think the patch still needs a bit more work, mostly to update the docs. I'll try to work out what needs tweaking in the docs and send a v3 ASAP. In terms of what to initially change, I'm still not sure if there's something that needs updating that I'm currently missing, or something that I'm updating that can be done later. > Apart from merging the changes in [1], I guess a blog post might be > useful. Have I missed anything? I'll start another thread on guix-devel to solicit feedback about substitutes from bordeaux.guix.gnu.org, I'm not sure what specifically about, but peoples observations might be helpful when writing a blog post about this. I'll also try to start drafting a blog post. What else needs doing to actually get these substitutes to users? Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmC+MtNfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xfr7hAAoRub/BRuMGD1Ifvn1kHryDWqA3l/pjpJ rXnBxr3+GklX0l9lCcmMD9BzKaMPMvFWQwjOBoBHfq+b0Z95spSVvVWEJNe3nAAX gOlV2AAltNDMLShKB7gt/NmCZk0rBFk8fW2o0WjCA5LOERV0401XCr0lPUkFTdxj vE8skmo482v1KWZm/Vxb9MLYQ45WLQS3rCByCxbmKpnE5bgCx818woKKyTzPReTj HKpnDIwM6TJg2gqs8gk2GoDdMKLIHmJZC7t+Wz1eFZcZgytJ0hDKEAhdLSK9D5Er Wieb8vfSXI7D0vYUN6/ivdldrMdRaseB9iFIV7fJWeI2hH539xgEQm5Hc0EHbklj i02rTc+cAcHuBl3tnEz8R5V++DDkym9qs1XGdb5a2jU+3jQ7oG7B7pD1ZkCmJvR9 /lsJA0qUMSF9GIcwFjPNK2M5yUt+Zp+3Ndy2pmqcUz4nKVNBYjRh/3U3kgzzIu0y kdIIpaoHjvEutd6OJPJ8iNbB5y57U10PFoPLO9TOPE3xHYqthFEnPiozmA9Yqwdk 4gqDIpaR7K13ME2rReHdULvZVhImyN4VAotVhUsmKc7a56O5bxvOXuh257PQ7X+0 jrmVl+SXo7/GA+nv+QZnMmfGmZhc3/HdV92GKmSKSyONAIYDBKh/jf7C8YqJhSWL rYhmygqRlQc= =vvnQ -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 08 08:14:18 2021 Received: (at 48435) by debbugs.gnu.org; 8 Jun 2021 12:14:18 +0000 Received: from localhost ([127.0.0.1]:57613 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lqacb-0005A2-IQ for submit@debbugs.gnu.org; Tue, 08 Jun 2021 08:14:18 -0400 Received: from mira.cbaines.net ([212.71.252.8]:38338) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lqacW-00059p-Jj for 48435@debbugs.gnu.org; Tue, 08 Jun 2021 08:14:16 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 899E527BC78 for <48435@debbugs.gnu.org>; Tue, 8 Jun 2021 13:14:11 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id b273bb73 for <48435@debbugs.gnu.org>; Tue, 8 Jun 2021 12:14:11 +0000 (UTC) From: Christopher Baines To: 48435@debbugs.gnu.org Subject: [PATCH v3] Start enabling substitutes from bordeaux.guix.gnu.org. Date: Tue, 8 Jun 2021 13:14:11 +0100 Message-Id: <20210608121411.4819-1-mail@cbaines.net> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 48435 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) In addition to substitutes from ci.guix.gnu.org. There are more changes that can be made in the future, but these changes seem like a good start. * config-daemon.ac (guix_substitute_urls): Add https://bordeaux.guix.gnu.org. * guix/scripts/substitute.scm (%default-substitute-urls): Add http://bordeaux.guix.gnu.org. * guix/store.scm (%default-substitute-urls): Add bordeaux.guix.gnu.org. * doc/guix.texi: Adjust accordingly. * doc/contributing.texi: Adjust accordingly. --- config-daemon.ac | 2 +- doc/contributing.texi | 8 +- doc/guix.texi | 142 +++++++++++++++++++----------------- guix/scripts/substitute.scm | 3 +- guix/store.scm | 3 +- 5 files changed, 86 insertions(+), 72 deletions(-) diff --git a/config-daemon.ac b/config-daemon.ac index 85caee269b..5ddc740600 100644 --- a/config-daemon.ac +++ b/config-daemon.ac @@ -117,7 +117,7 @@ if test "x$guix_build_daemon" = "xyes"; then dnl Determine the appropriate default list of substitute URLs (GnuTLS dnl is required so we can default to 'https'.) - guix_substitute_urls="https://ci.guix.gnu.org" + guix_substitute_urls="https://ci.guix.gnu.org https://bordeaux.guix.gnu.org" AC_MSG_CHECKING([for default substitute URLs]) AC_MSG_RESULT([$guix_substitute_urls]) diff --git a/doc/contributing.texi b/doc/contributing.texi index d67e632520..1086bb9fd4 100644 --- a/doc/contributing.texi +++ b/doc/contributing.texi @@ -376,12 +376,12 @@ Once your package builds correctly, please send us a patch (@pxref{Submitting Patches}). Well, if you need help, we will be happy to help you too. Once the patch is committed in the Guix repository, the new package automatically gets built on the supported platforms by -@url{@value{SUBSTITUTE-URL}, our continuous integration system}. +@url{https://@value{SUBSTITUTE-SERVER-1}, our continuous integration system}. @cindex substituter Users can obtain the new package definition simply by running @command{guix pull} (@pxref{Invoking guix pull}). When -@code{@value{SUBSTITUTE-SERVER}} is done building the package, installing the +@code{@value{SUBSTITUTE-SERVER-1}} is done building the package, installing the package automatically downloads binaries from there (@pxref{Substitutes}). The only place where human intervention is needed is to review and apply the patch. @@ -1107,7 +1107,7 @@ changes). This branch is intended to be merged in @code{master} every until late in its development process. @end table -All these branches are @uref{@value{SUBSTITUTE-URL}, +All these branches are @uref{https://@value{SUBSTITUTE-SERVER-1}, tracked by our build farm} and merged into @code{master} once everything has been successfully built. This allows us to fix issues before they hit users, and to reduce the window during which pre-built @@ -1141,7 +1141,7 @@ as timestamps or randomly-generated output in the build result. Another option is to use @command{guix challenge} (@pxref{Invoking guix challenge}). You may run it once the package has been committed and -built by @code{@value{SUBSTITUTE-SERVER}} to check whether it obtains the same +built by @code{@value{SUBSTITUTE-SERVER-1}} to check whether it obtains the same result as you did. Better yet: Find another machine that can build it and run @command{guix publish}. Since the remote build machine is likely different from yours, this can catch non-determinism issues diff --git a/doc/guix.texi b/doc/guix.texi index 59b4ac11b4..a917d4092f 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -17,8 +17,9 @@ @set BASE-URL https://ftp.gnu.org/gnu/guix @c The official substitute server used by default. -@set SUBSTITUTE-SERVER ci.guix.gnu.org -@set SUBSTITUTE-URL https://@value{SUBSTITUTE-SERVER} +@set SUBSTITUTE-SERVER-1 ci.guix.gnu.org +@set SUBSTITUTE-SERVER-2 bordeaux.guix.gnu.org +@set SUBSTITUTE-URLS https://@value{SUBSTITUTE-SERVER-1} https://@value{SUBSTITUTE-SERVER-2} @copying Copyright @copyright{} 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès@* @@ -229,7 +230,7 @@ Package Management Substitutes -* Official Substitute Server:: One particular source of substitutes. +* Official Substitute Servers:: One particular source of substitutes. * Substitute Server Authorization:: How to enable or disable substitutes. * Getting Substitutes from Other Servers:: Substitute diversity. * Substitute Authentication:: How Guix verifies substitutes. @@ -780,12 +781,15 @@ Info search path). @item @cindex substitutes, authorization thereof -To use substitutes from @code{@value{SUBSTITUTE-SERVER}} or one of its mirrors -(@pxref{Substitutes}), authorize them: +To use substitutes from @code{@value{SUBSTITUTE-SERVER-1}}, +@code{@value{SUBSTITUTE-SERVER-2}} or a mirror (@pxref{Substitutes}), +authorize them: @example # guix archive --authorize < \ - ~root/.config/guix/current/share/guix/@value{SUBSTITUTE-SERVER}.pub + ~root/.config/guix/current/share/guix/@value{SUBSTITUTE-SERVER-1}.pub +# guix archive --authorize < \ + ~root/.config/guix/current/share/guix/@value{SUBSTITUTE-SERVER-2}.pub @end example @quotation Note @@ -1547,7 +1551,7 @@ remote procedure call (@pxref{The Store}). @item --substitute-urls=@var{urls} Consider @var{urls} the default whitespace-separated list of substitute source URLs. When this option is omitted, -@indicateurl{https://@value{SUBSTITUTE-SERVER}} is used. +@indicateurl{@value{SUBSTITUTE-URLS}} is used. This means that substitutes may be downloaded from @var{urls}, as long as they are signed by a trusted signature (@pxref{Substitutes}). @@ -3685,7 +3689,7 @@ pre-built package binaries, but source tarballs, for instance, which also result from derivation builds, can be available as substitutes. @menu -* Official Substitute Server:: One particular source of substitutes. +* Official Substitute Servers:: One particular source of substitutes. * Substitute Server Authorization:: How to enable or disable substitutes. * Getting Substitutes from Other Servers:: Substitute diversity. * Substitute Authentication:: How Guix verifies substitutes. @@ -3694,14 +3698,15 @@ also result from derivation builds, can be available as substitutes. * On Trusting Binaries:: How can you trust that binary blob? @end menu -@node Official Substitute Server -@subsection Official Substitute Server +@node Official Substitute Servers +@subsection Official Substitute Servers @cindex build farm -The @code{@value{SUBSTITUTE-SERVER}} server is a front-end to an official build farm -that builds packages from Guix continuously for some -architectures, and makes them available as substitutes. This is the -default source of substitutes; it can be overridden by passing the +@code{@value{SUBSTITUTE-SERVER-1}} and +@code{@value{SUBSTITUTE-SERVER-2}} are both front-ends to official build +farms that build packages from Guix continuously for some architectures, +and make them available as substitutes. These are the default source of +substitutes; which can be overridden by passing the @option{--substitute-urls} option either to @command{guix-daemon} (@pxref{daemon-substitute-urls,, @code{guix-daemon --substitute-urls}}) or to client tools such as @command{guix package} @@ -3714,7 +3719,7 @@ using HTTP makes all communications visible to an eavesdropper, who could use the information gathered to determine, for instance, whether your system has unpatched security vulnerabilities. -Substitutes from the official build farm are enabled by default when +Substitutes from the official build farms are enabled by default when using Guix System (@pxref{GNU Distribution}). However, they are disabled by default when using Guix on a foreign distribution, unless you have explicitly enabled them via one of the recommended @@ -3730,27 +3735,28 @@ other substitute server. @cindex substitutes, authorization thereof @cindex access control list (ACL), for substitutes @cindex ACL (access control list), for substitutes -To allow Guix to download substitutes from @code{@value{SUBSTITUTE-SERVER}} or a -mirror thereof, you -must add its public key to the access control list (ACL) of archive +To allow Guix to download substitutes from @code{@value{SUBSTITUTE-SERVER-1}}, @code{@value{SUBSTITUTE-SERVER-2}} or a mirror, you +must add the releavnt public key to the access control list (ACL) of archive imports, using the @command{guix archive} command (@pxref{Invoking guix -archive}). Doing so implies that you trust @code{@value{SUBSTITUTE-SERVER}} to not +archive}). Doing so implies that you trust the substitute server to not be compromised and to serve genuine substitutes. @quotation Note If you are using Guix System, you can skip this section: Guix System -authorizes substitutes from @code{@value{SUBSTITUTE-SERVER}} by default. +authorizes substitutes from @code{@value{SUBSTITUTE-SERVER-1}} and +@code{@value{SUBSTITUTE-SERVER-2}} by default. @end quotation -The public key for @code{@value{SUBSTITUTE-SERVER}} is installed along with Guix, in -@code{@var{prefix}/share/guix/@value{SUBSTITUTE-SERVER}.pub}, where @var{prefix} is -the installation prefix of Guix. If you installed Guix from source, -make sure you checked the GPG signature of +The public keys for each of the project maintained substitute servers +are installed along with Guix, in @code{@var{prefix}/share/guix/}, where +@var{prefix} is the installation prefix of Guix. If you installed Guix +from source, make sure you checked the GPG signature of @file{guix-@value{VERSION}.tar.gz}, which contains this public key file. Then, you can run something like this: @example -# guix archive --authorize < @var{prefix}/share/guix/@value{SUBSTITUTE-SERVER}.pub +# guix archive --authorize < @var{prefix}/share/guix/@value{SUBSTITUTE-SERVER-1}.pub +# guix archive --authorize < @var{prefix}/share/guix/@value{SUBSTITUTE-SERVER-2}.pub @end example Once this is in place, the output of a command like @code{guix build} @@ -3782,8 +3788,8 @@ $ guix build emacs --dry-run @noindent The text changed from ``The following derivations would be built'' to ``112.3 MB would be downloaded''. This indicates that substitutes from -@code{@value{SUBSTITUTE-SERVER}} are usable and will be downloaded, when -possible, for future builds. +the configured substitute servers are usable and will be downloaded, +when possible, for future builds. @cindex substitutes, how to disable The substitute mechanism can be disabled globally by running @@ -3817,8 +3823,9 @@ its configuration and add the URLs and substitute keys that you want As an example, suppose you want to fetch substitutes from @code{guix.example.org} and to authorize the signing key of that server, -in addition to the default @code{@value{SUBSTITUTE-SERVER}}. The -resulting operating system configuration will look something like: +in addition to the default @code{@value{SUBSTITUTE-SERVER-1}} and +@code{@value{SUBSTITUTE-SERVER-2}}. The resulting operating system +configuration will look something like: @lisp (operating-system @@ -3862,7 +3869,7 @@ line and list the URLs of interest (@pxref{daemon-substitute-urls, @code{guix-daemon --substitute-urls}}): @example -@dots{} --substitute-urls='https://guix.example.org https://@value{SUBSTITUTE-SERVER}' +@dots{} --substitute-urls='https://guix.example.org @value{SUBSTITUTE-URLS}' @end example @item @@ -3885,10 +3892,12 @@ Again this assumes @file{key.pub} contains the public key that @end enumerate Now you're all set! Substitutes will be preferably taken from -@code{https://guix.example.org}, using @code{@value{SUBSTITUTE-SERVER}} -as a fallback. Of course you can list as many substitute servers as you -like, with the caveat that substitute lookup can be slowed down if too -many servers need to be contacted. +@code{https://guix.example.org}, using +@code{@value{SUBSTITUTE-SERVER-1}} then +@code{@value{SUBSTITUTE-SERVER-2}} as fallback options. Of course you +can list as many substitute servers as you like, with the caveat that +substitute lookup can be slowed down if too many servers need to be +contacted. Note that there are also situations where one may want to add the URL of a substitute server @emph{without} authorizing its key. @@ -3976,12 +3985,12 @@ by a server. Today, each individual's control over their own computing is at the mercy of institutions, corporations, and groups with enough power and determination to subvert the computing infrastructure and exploit its -weaknesses. While using @code{@value{SUBSTITUTE-SERVER}} substitutes can be -convenient, we encourage users to also build on their own, or even run -their own build farm, such that @code{@value{SUBSTITUTE-SERVER}} is less of an -interesting target. One way to help is by publishing the software you -build using @command{guix publish} so that others have one more choice -of server to download substitutes from (@pxref{Invoking guix publish}). +weaknesses. While using substitutes can be convenient, we encourage +users to also build on their own, or even run their own build farm, such +that the project run substitute servers are less of an interesting +target. One way to help is by publishing the software you build using +@command{guix publish} so that others have one more choice of server to +download substitutes from (@pxref{Invoking guix publish}). Guix has the foundations to maximize build reproducibility (@pxref{Features}). In most cases, independent builds of a given @@ -4945,11 +4954,11 @@ Read a single-item archive as served by substitute servers low-level operation needed in only very narrow use cases; see below. For example, the following command extracts the substitute for Emacs -served by @code{@value{SUBSTITUTE-SERVER}} to @file{/tmp/emacs}: +served by @code{@value{SUBSTITUTE-SERVER-1}} to @file{/tmp/emacs}: @example $ wget -O - \ - https://@value{SUBSTITUTE-SERVER}/nar/gzip/@dots{}-emacs-24.5 \ + https://@value{SUBSTITUTE-SERVER-1}/nar/gzip/@dots{}-emacs-24.5 \ | gunzip | guix archive -x /tmp/emacs @end example @@ -4971,7 +4980,7 @@ this example: @example $ wget -O - \ - https://@value{SUBSTITUTE-SERVER}/nar/lzip/@dots{}-emacs-26.3 \ + https://@value{SUBSTITUTE-SERVER-1}/nar/lzip/@dots{}-emacs-26.3 \ | lzip -d | guix archive -t @end example @@ -10905,7 +10914,7 @@ but you are actually on an @code{x86_64} machine: @example $ guix build --log-file gdb -s aarch64-linux -https://@value{SUBSTITUTE-SERVER}/log/@dots{}-gdb-7.10 +https://@value{SUBSTITUTE-SERVER-1}/log/@dots{}-gdb-7.10 @end example You can freely access a huge library of build logs! @@ -12558,7 +12567,7 @@ When @command{guix publish} runs, it spawns an HTTP server which allows anyone with network access to obtain substitutes from it. This means that any machine running Guix can also act as if it were a build farm, since the HTTP interface is compatible with Cuirass, the software behind -the @code{@value{SUBSTITUTE-SERVER}} build farm. +the @code{@value{SUBSTITUTE-SERVER-1}} build farm. For security, each substitute is signed, allowing recipients to check their authenticity and integrity (@pxref{Substitutes}). Because @@ -12847,12 +12856,12 @@ any given store item. The command output looks like this: @smallexample -$ guix challenge --substitute-urls="https://@value{SUBSTITUTE-SERVER} https://guix.example.org" -updating list of substitutes from 'https://@value{SUBSTITUTE-SERVER}'... 100.0% +$ guix challenge --substitute-urls="https://@value{SUBSTITUTE-SERVER-1} https://guix.example.org" +updating list of substitutes from 'https://@value{SUBSTITUTE-SERVER-1}'... 100.0% updating list of substitutes from 'https://guix.example.org'... 100.0% /gnu/store/@dots{}-openssl-1.0.2d contents differ: local hash: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q - https://@value{SUBSTITUTE-SERVER}/nar/@dots{}-openssl-1.0.2d: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q + https://@value{SUBSTITUTE-SERVER-1}/nar/@dots{}-openssl-1.0.2d: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q https://guix.example.org/nar/@dots{}-openssl-1.0.2d: 1zy4fmaaqcnjrzzajkdn3f5gmjk754b43qkq47llbyak9z0qjyim differing files: /lib/libcrypto.so.1.1 @@ -12860,14 +12869,14 @@ updating list of substitutes from 'https://guix.example.org'... 100.0% /gnu/store/@dots{}-git-2.5.0 contents differ: local hash: 00p3bmryhjxrhpn2gxs2fy0a15lnip05l97205pgbk5ra395hyha - https://@value{SUBSTITUTE-SERVER}/nar/@dots{}-git-2.5.0: 069nb85bv4d4a6slrwjdy8v1cn4cwspm3kdbmyb81d6zckj3nq9f + https://@value{SUBSTITUTE-SERVER-1}/nar/@dots{}-git-2.5.0: 069nb85bv4d4a6slrwjdy8v1cn4cwspm3kdbmyb81d6zckj3nq9f https://guix.example.org/nar/@dots{}-git-2.5.0: 0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73 differing file: /libexec/git-core/git-fsck /gnu/store/@dots{}-pius-2.1.1 contents differ: local hash: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax - https://@value{SUBSTITUTE-SERVER}/nar/@dots{}-pius-2.1.1: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax + https://@value{SUBSTITUTE-SERVER-1}/nar/@dots{}-pius-2.1.1: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax https://guix.example.org/nar/@dots{}-pius-2.1.1: 1cy25x1a4fzq5rk0pmvc8xhwyffnqz95h2bpvqsz2mpvlbccy0gs differing file: /share/man/man1/pius.1.gz @@ -12889,7 +12898,7 @@ the servers obtained a result different from the local build. @cindex non-determinism, in package builds As an example, @code{guix.example.org} always gets a different answer. -Conversely, @code{@value{SUBSTITUTE-SERVER}} agrees with local builds, except in the +Conversely, @code{@value{SUBSTITUTE-SERVER-1}} agrees with local builds, except in the case of Git. This might indicate that the build process of Git is non-deterministic, meaning that its output varies as a function of various things that Guix does not fully control, in spite of building @@ -12905,7 +12914,7 @@ to run: @example guix challenge git \ --diff=diffoscope \ - --substitute-urls="https://@value{SUBSTITUTE-SERVER} https://guix.example.org" + --substitute-urls="https://@value{SUBSTITUTE-SERVER-1} https://guix.example.org" @end example This automatically invokes @command{diffoscope}, which displays detailed @@ -12915,14 +12924,14 @@ Alternatively, we can do something along these lines (@pxref{Invoking guix archive}): @example -$ wget -q -O - https://@value{SUBSTITUTE-SERVER}/nar/lzip/@dots{}-git-2.5.0 \ +$ wget -q -O - https://@value{SUBSTITUTE-SERVER-1}/nar/lzip/@dots{}-git-2.5.0 \ | lzip -d | guix archive -x /tmp/git $ diff -ur --no-dereference /gnu/store/@dots{}-git.2.5.0 /tmp/git @end example This command shows the difference between the files resulting from the local build, and the files resulting from the build on -@code{@value{SUBSTITUTE-SERVER}} (@pxref{Overview, Comparing and Merging Files,, +@code{@value{SUBSTITUTE-SERVER-1}} (@pxref{Overview, Comparing and Merging Files,, diffutils, Comparing and Merging Files}). The @command{diff} command works great for text files. When binary files differ, a better option is @uref{https://diffoscope.org/, Diffoscope}, a tool that helps @@ -12937,7 +12946,7 @@ In the meantime, @command{guix challenge} is one tool to help address the problem. If you are writing packages for Guix, you are encouraged to check -whether @code{@value{SUBSTITUTE-SERVER}} and other substitute servers obtain the +whether @code{@value{SUBSTITUTE-SERVER-1}} and other substitute servers obtain the same build result as you did with: @example @@ -13218,14 +13227,14 @@ on @var{a} and @var{a} has no substitutes, only @var{a} is listed, even though @var{b} usually lacks substitutes as well. The result looks like this: @example -$ guix weather --substitute-urls=@value{SUBSTITUTE-URL} -c 10 +$ guix weather --substitute-urls=@value{SUBSTITUTE-URLS} -c 10 computing 8,983 package derivations for x86_64-linux... -looking for 9,343 store items on @value{SUBSTITUTE-URL}... -updating substitutes from '@value{SUBSTITUTE-URL}'... 100.0% -@value{SUBSTITUTE-URL} +looking for 9,343 store items on @value{SUBSTITUTE-URLS}... +updating substitutes from '@value{SUBSTITUTE-URLS}'... 100.0% +@value{SUBSTITUTE-URLS} 64.7% substitutes available (6,047 out of 9,343) @dots{} -2502 packages are missing from '@value{SUBSTITUTE-URL}' for 'x86_64-linux', among which: +2502 packages are missing from '@value{SUBSTITUTE-URLS}' for 'x86_64-linux', among which: 58 kcoreaddons@@5.49.0 /gnu/store/@dots{}-kcoreaddons-5.49.0 46 qgpgme@@1.11.1 /gnu/store/@dots{}-qgpgme-1.11.1 37 perl-http-cookiejar@@0.008 /gnu/store/@dots{}-perl-http-cookiejar-0.008 @@ -13234,7 +13243,7 @@ updating substitutes from '@value{SUBSTITUTE-URL}'... 100.0% What this example shows is that @code{kcoreaddons} and presumably the 58 packages that depend on it have no substitutes at -@code{@value{SUBSTITUTE-SERVER}}; likewise for @code{qgpgme} and the 46 +@code{@value{SUBSTITUTE-SERVER-1}}; likewise for @code{qgpgme} and the 46 packages that depend on it. If you are a Guix developer, or if you are taking care of this build farm, @@ -15441,7 +15450,9 @@ Number of build user accounts to create. @item @code{authorize-key?} (default: @code{#t}) @cindex substitutes, authorization thereof Whether to authorize the substitute keys listed in -@code{authorized-keys}---by default that of @code{@value{SUBSTITUTE-SERVER}} +@code{authorized-keys}---by default that of +@code{@value{SUBSTITUTE-SERVER-1}} and +@code{@value{SUBSTITUTE-SERVER-2}} (@pxref{Substitutes}). When @code{authorize-key?} is true, @file{/etc/guix/acl} cannot be @@ -15462,8 +15473,9 @@ allowed for in-place modifications to @file{/etc/guix/acl}. @item @code{authorized-keys} (default: @code{%default-authorized-guix-keys}) The list of authorized key files for archive imports, as a list of string-valued gexps (@pxref{Invoking guix archive}). By default, it -contains that of @code{@value{SUBSTITUTE-SERVER}} (@pxref{Substitutes}). -See @code{substitute-urls} below for an example on how to change it. +contains that of @code{@value{SUBSTITUTE-SERVER-1}} and +@code{@value{SUBSTITUTE-SERVER-2}} (@pxref{Substitutes}). See +@code{substitute-urls} below for an example on how to change it. @item @code{use-substitutes?} (default: @code{#t}) Whether to use substitutes. @@ -15472,7 +15484,7 @@ Whether to use substitutes. The list of URLs where to look for substitutes by default. Suppose you would like to fetch substitutes from @code{guix.example.org} -in addition to @code{@value{SUBSTITUTE-SERVER}}. You will need to do +in addition to @code{@value{SUBSTITUTE-SERVER-1}}. You will need to do two things: (1) add @code{guix.example.org} to @code{substitute-urls}, and (2) authorize its signing key, having done appropriate checks (@pxref{Substitute Server Authorization}). The configuration below does diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 44448ff3e9..3ea1c73e10 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -643,7 +643,8 @@ found." (#f ;; This can only happen when this script is not invoked by the ;; daemon. - '("http://ci.guix.gnu.org")))) + '("http://ci.guix.gnu.org" + "http://bordeaux.guix.gnu.org")))) ;; In order to prevent using large number of discovered local substitute ;; servers, limit the local substitute urls list size. diff --git a/guix/store.scm b/guix/store.scm index b761264ac0..1ab2b08b47 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -787,7 +787,8 @@ encoding conversion errors." (map (if (false-if-exception (resolve-interface '(gnutls))) (cut string-append "https://" <>) (cut string-append "http://" <>)) - '("ci.guix.gnu.org"))) + '("ci.guix.gnu.org" + "bordeaux.guix.gnu.org"))) (define (current-user-name) "Return the name of the calling user." -- 2.31.1 From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 18 07:06:23 2021 Received: (at 48435-done) by debbugs.gnu.org; 18 Jun 2021 11:06:23 +0000 Received: from localhost ([127.0.0.1]:55910 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1luCKN-0006wr-AH for submit@debbugs.gnu.org; Fri, 18 Jun 2021 07:06:23 -0400 Received: from mira.cbaines.net ([212.71.252.8]:52102) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1luCKJ-0006wd-SC for 48435-done@debbugs.gnu.org; Fri, 18 Jun 2021 07:06:21 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 810F827BC78 for <48435-done@debbugs.gnu.org>; Fri, 18 Jun 2021 12:06:18 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id d981e161 for <48435-done@debbugs.gnu.org>; Fri, 18 Jun 2021 11:06:18 +0000 (UTC) References: <20210515100822.11784-1-mail@cbaines.net> <20210608121411.4819-1-mail@cbaines.net> User-agent: mu4e 1.4.15; emacs 27.2 From: Christopher Baines To: 48435-done@debbugs.gnu.org Subject: Re: [bug#48435] [PATCH v3] Start enabling substitutes from bordeaux.guix.gnu.org. In-reply-to: <20210608121411.4819-1-mail@cbaines.net> Date: Fri, 18 Jun 2021 12:06:15 +0100 Message-ID: <87wnqr8mag.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 48435-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Christopher Baines writes: > In addition to substitutes from ci.guix.gnu.org. There are more changes that > can be made in the future, but these changes seem like a good start. > > * config-daemon.ac (guix_substitute_urls): Add https://bordeaux.guix.gnu.org. > * guix/scripts/substitute.scm (%default-substitute-urls): Add > http://bordeaux.guix.gnu.org. > * guix/store.scm (%default-substitute-urls): Add bordeaux.guix.gnu.org. > * doc/guix.texi: Adjust accordingly. > * doc/contributing.texi: Adjust accordingly. Pushed as 4985a4272497bf9ba87a2190353d915da9b55906. I'll update the guix package shortly. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmDMfidfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xcu6g/8Cb0+NfWrldNkYtA7ylAOtyP7RGrQbVQw /bZJK8EBTqexc5TCC8ZffKjTq+Dx0TaIYrQJ+d+D3Jo5dUk//eqSefKXTXHsv76s HyTmC65kipYV0B0UzQU5DCf1KoxwOTgHck8bb7sZWO7tta3HmV1CegqrljUXtnpr FrWfkIV8cSdMzspKz1Ki/XvfLh0l6t/UdNv8Jbx/LJxAKczhwCq8sIndQ59k7dcX uZXRhL8jUmFGiFG9I0vD6JtPGG45CaqImVZbMr6dI9SAHljvlmlwl5Ub5nQb8y95 Qs6s5/A2KPpd4P0Q7JoC42V/9/bpnjdly54BfQfzmx2KKhUSucEjBl6+1hDu4QBD F+ak1lkF4lfucE7/okVUCGy5a6P2tugtABIb/r1h52ftIfDjgXbrv30QB9ZTz+8u vxPW0TFI27pzggWn7ottgPaEhKibIsmgBTDnsbbxloXiBDdcCJFylGyjUBG8OubQ iob+eHFrqVRXGu+z0sxVEnL7jdUdSqcVBjLSK0mxopNJddZU+twHHSRfHHQx1F5m wTTDbYHnQVemOcAsAu393nQCIqjjkbJ5tnZZr+bypVsvm83lZkhoZ0l8kX3X8K50 TSwFGCFM16MjIaJPen5n53lURHooWRrg45ZuhnzcRQg6Ti+/SeuBnuGgiXtvseYm GXFi+G0TbvI= =SyKg -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 18:05:47 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 16 Jul 2021 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator