From unknown Sat Jul 26 16:38:32 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#45174 <45174@debbugs.gnu.org> To: bug#45174 <45174@debbugs.gnu.org> Subject: Status: =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?substitute=E2=80=99_?= =?UTF-8?Q?doesn=E2=80=99t?= handle HTTP redirects Reply-To: bug#45174 <45174@debbugs.gnu.org> Date: Sat, 26 Jul 2025 23:38:32 +0000 retitle 45174 =E2=80=98guix substitute=E2=80=99 doesn=E2=80=99t handle HTTP= redirects reassign 45174 guix submitter 45174 Ludovic Court=C3=A8s severity 45174 important thanks From debbugs-submit-bounces@debbugs.gnu.org Fri Dec 11 04:50:48 2020 Received: (at submit) by debbugs.gnu.org; 11 Dec 2020 09:50:48 +0000 Received: from localhost ([127.0.0.1]:40010 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1knf4Z-0007DL-Sj for submit@debbugs.gnu.org; Fri, 11 Dec 2020 04:50:48 -0500 Received: from lists.gnu.org ([209.51.188.17]:55152) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1knf4W-0007DE-JY for submit@debbugs.gnu.org; Fri, 11 Dec 2020 04:50:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59328) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1knf4W-0001HW-BV for bug-guix@gnu.org; Fri, 11 Dec 2020 04:50:44 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:61517) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1knf4R-0005if-QG for bug-guix@gnu.org; Fri, 11 Dec 2020 04:50:43 -0500 X-IronPort-AV: E=Sophos;i="5.78,411,1599516000"; d="scan'208";a="367320768" Received: from 91-160-117-201.subs.proxad.net (HELO ribbon) ([91.160.117.201]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2020 10:50:36 +0100 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: bug-guix@gnu.org Subject: =?utf-8?Q?=E2=80=98guix_substitute=E2=80=99_doesn=E2=80=99t?= handle HTTP redirects X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 21 Frimaire an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 11 Dec 2020 10:50:36 +0100 Message-ID: <87a6uk8ydv.fsf@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=192.134.164.104; envelope-from=ludovic.courtes@inria.fr; helo=mail3-relais-sop.national.inria.fr X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hi! As reported by mange on #guix, it seems that =E2=80=98guix substitute=E2=80= =99 does not follow redirects: --8<---------------cut here---------------start------------->8--- $ guix weather icecat emacs --substitute-urls=3Dhttps://ci.guix.gnu.org computing 2 package derivations for x86_64-linux... looking for 2 store items on https://ci.guix.gnu.org... updating substitutes from 'https://ci.guix.gnu.org'... 100.0% https://ci.guix.gnu.org 100.0% substitutes available (2 out of 2) at least 201.3 MiB of nars (compressed) 293.5 MiB on disk (uncompressed) 0.084 seconds per request (0.2 seconds in total) 11.9 requests per second at least 1,000 queued builds x86_64-linux: 368 (36.8%) i686-linux: 556 (55.6%) armhf-linux: 6 (.6%) aarch64-linux: 69 (6.9%) i586-gnu: 1 (.1%) build rate: 154.41 builds per hour i686-linux: 40.77 builds per hour x86_64-linux: 37.06 builds per hour armhf-linux: 40.30 builds per hour aarch64-linux: 35.83 builds per hour i586-gnu: 0.55 builds per hour $ guix weather icecat emacs --substitute-urls=3Dhttps://ci.guix.info computing 2 package derivations for x86_64-linux... looking for 2 store items on https://ci.guix.info... updating substitutes from 'https://ci.guix.info'... 100.0% https://ci.guix.info 0.0% substitutes available (0 out of 2) unknown substitute sizes 0.0 MiB on disk (uncompressed) 0.069 seconds per request (0.1 seconds in total) 14.4 requests per second ni sekvas la redirektigon al 'https://ci.guix.gnu.org/api/queue'... 'https://ci.guix.gnu.org/api/queue' returned 500 ("Internal Server Error") --8<---------------cut here---------------end--------------->8--- This might explain things like . (*.guix.info were turned into HTTP redirects a few days ago, see .) Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Dec 11 09:15:40 2020 Received: (at control) by debbugs.gnu.org; 11 Dec 2020 14:15:40 +0000 Received: from localhost ([127.0.0.1]:40616 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1knjCu-0003a6-8W for submit@debbugs.gnu.org; Fri, 11 Dec 2020 09:15:40 -0500 Received: from eggs.gnu.org ([209.51.188.92]:44122) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1knjCq-0003Zk-VU for control@debbugs.gnu.org; Fri, 11 Dec 2020 09:15:39 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:46574) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1knjCl-0005hU-Qe for control@debbugs.gnu.org; Fri, 11 Dec 2020 09:15:31 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=35386 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1knjCj-0002EV-L1 for control@debbugs.gnu.org; Fri, 11 Dec 2020 09:15:31 -0500 Date: Fri, 11 Dec 2020 15:15:27 +0100 Message-Id: <87lfe477k0.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #45174 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) severity 45174 important quit From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 12 12:12:14 2021 Received: (at 45174) by debbugs.gnu.org; 12 Jan 2021 17:12:14 +0000 Received: from localhost ([127.0.0.1]:32983 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kzNDJ-0007WE-CB for submit@debbugs.gnu.org; Tue, 12 Jan 2021 12:12:13 -0500 Received: from lepiller.eu ([89.234.186.109]:60112) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kzNDG-0007W0-SJ for 45174@debbugs.gnu.org; Tue, 12 Jan 2021 12:12:12 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id 5f912d75; Tue, 12 Jan 2021 17:12:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type; s=dkim; bh=uAki5ARhQPt5ahRW/Hm/4mXufjFcxVQZ/WNKXW q+cmc=; b=blP+zTLZb5UvzrVdGUR1B6ICdkOWnZ7FPMm33thBdglKBcAN0Lt5zB Se497yScO2v6vH+LZoKQtc7+9ue4ornrPL0GrDxec1LIa8emIUaRJ65g4OQkzPqX jVrGbd/z3weAlapBgoeCOOyTQ7pKG/IllLB+9DcjY7cm6mUtdI2sUSXTIQGKfdWZ dl3RGlX+qEmYbFoVWfIfSHn0ot71+30ArnJIVhLckQHJgP03hNJLDKg8jkJt0oZk oL4NAK+MA0sWQW3tzjrmSHkgjq3p2J9+0rjGoQAOpNULTB8NOAzoPfaQSkzM4IyR 8tvQdGDZ5+oufbPMLtYWw7Fg1jP056LQ== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id 00eab565 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Tue, 12 Jan 2021 17:12:04 +0000 (UTC) Date: Tue, 12 Jan 2021 18:11:55 +0100 From: Julien Lepiller To: Ludovic =?UTF-8?B?Q291cnTDqHM=?= Subject: Re: bug#45174: =?UTF-8?B?4oCYZ3VpeCBzdWJzdGl0dXRl4oCZIGRvZXNu4oCZ?= =?UTF-8?B?dA==?= handle HTTP redirects Message-ID: <20210112181155.05b7992d@tachikoma.lepiller.eu> In-Reply-To: <87a6uk8ydv.fsf@inria.fr> References: <87a6uk8ydv.fsf@inria.fr> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/HMfGdeue67HATiQgj_SBCh/" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45174 Cc: 45174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --MP_/HMfGdeue67HATiQgj_SBCh/ Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Le Fri, 11 Dec 2020 10:50:36 +0100, Ludovic Court=C3=A8s a =C3=A9crit : > Hi! >=20 > As reported by mange on #guix, it seems that =E2=80=98guix substitute=E2= =80=99 does > not follow redirects: >=20 > --8<---------------cut here---------------start------------->8--- > $ guix weather icecat emacs --substitute-urls=3Dhttps://ci.guix.gnu.org > computing 2 package derivations for x86_64-linux... > looking for 2 store items on https://ci.guix.gnu.org... > updating substitutes from 'https://ci.guix.gnu.org'... 100.0% > https://ci.guix.gnu.org > 100.0% substitutes available (2 out of 2) > at least 201.3 MiB of nars (compressed) > 293.5 MiB on disk (uncompressed) > 0.084 seconds per request (0.2 seconds in total) > 11.9 requests per second >=20 > at least 1,000 queued builds > x86_64-linux: 368 (36.8%) > i686-linux: 556 (55.6%) > armhf-linux: 6 (.6%) > aarch64-linux: 69 (6.9%) > i586-gnu: 1 (.1%) > build rate: 154.41 builds per hour > i686-linux: 40.77 builds per hour > x86_64-linux: 37.06 builds per hour > armhf-linux: 40.30 builds per hour > aarch64-linux: 35.83 builds per hour > i586-gnu: 0.55 builds per hour > $ guix weather icecat emacs --substitute-urls=3Dhttps://ci.guix.info > computing 2 package derivations for x86_64-linux... > looking for 2 store items on https://ci.guix.info... > updating substitutes from 'https://ci.guix.info'... 100.0% > https://ci.guix.info > 0.0% substitutes available (0 out of 2) > unknown substitute sizes > 0.0 MiB on disk (uncompressed) > 0.069 seconds per request (0.1 seconds in total) > 14.4 requests per second > ni sekvas la redirektigon al 'https://ci.guix.gnu.org/api/queue'... > 'https://ci.guix.gnu.org/api/queue' returned 500 ("Internal Server > Error") --8<---------------cut > here---------------end--------------->8--- >=20 > This might explain things like . >=20 > (*.guix.info were turned into HTTP redirects a few days ago, see > .) >=20 > Ludo=E2=80=99. >=20 >=20 >=20 Here is a patch to fix that issue. Since ci.guix.info now returns 200, it's difficult to test the patch. I created a location block on my website to redirect lepiller.eu/*.narinfo -> ci.guix.gnu.org/*.narinfo. Here's the result: $ guix weather icecat emacs --substitute-urls=3Dhttps://lepiller.eu calcul de 2 d=C3=A9rivations de paquets pour x86_64-linux=E2=80=A6 recherche de 2 =C3=A9l=C3=A9ments du d=C3=A9p=C3=B4t sur https://lepiller.e= u... mise =C3=A0 jour des substituts depuis =C2=AB=C2=A0https://lepiller.eu=C2= =A0=C2=BB... 100.0 % https://lepiller.eu 0.0 % des substituts sont disponibles (0 sur 2) taille des substituts inconnue 0,0 Mo sur le disque (d=C3=A9compress=C3=A9) 0,207 secondes par requ=C3=AAte (0,4 secondes en tout) 4,8 requ=C3=AAtes par seconde $ ./pre-inst-env guix weather icecat emacs --substitute-urls=3Dhttps://lepiller.eu computing 2 package derivations for x86_64-linux... looking for 2 store items on https://lepiller.eu... updating substitutes from 'https://lepiller.eu'... 100.0% https://lepiller.eu 100.0% substitutes available (2 out of 2) at least 201,3 MiB of nars (compressed) 293,5 MiB on disk (uncompressed) 0,525 seconds per request (1,1 seconds in total) 1,9 requests per second (note that I didn't redirect the ci API on my server, so the rest of the weather command fails with a backtrace, but it's unrelated to this patch and this issue). --MP_/HMfGdeue67HATiQgj_SBCh/ Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0001-substitute-Follow-narinfo-redirections.patch >From f20e01f2a8df538519660772a7431b53d650d64f Mon Sep 17 00:00:00 2001 From: Julien Lepiller Date: Tue, 12 Jan 2021 18:07:25 +0100 Subject: [PATCH] substitute: Follow narinfo redirections. * guix/scripts/substitute.scm (fetch-narinfos): Follow redirections. --- guix/scripts/substitute.scm | 38 +++++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 10 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index e53de8c304..790168091e 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -663,18 +663,36 @@ port to it, or, if connection failed, print a warning and return #f. Pass (len (response-content-length response)) (cache (response-cache-control response)) (ttl (and cache (assoc-ref cache 'max-age)))) - (update-progress!) ;; Make sure to read no more than LEN bytes since subsequent bytes may ;; belong to the next response. - (if (= code 200) ; hit - (let ((narinfo (read-narinfo port url #:size len))) - (if (string=? (dirname (narinfo-path narinfo)) - (%store-prefix)) - (begin - (cache-narinfo! url (narinfo-path narinfo) narinfo ttl) - (cons narinfo result)) - result)) + (case code + ((200) ; hit + (update-progress!) + (let ((narinfo (read-narinfo port url #:size len))) + (if (string=? (dirname (narinfo-path narinfo)) + (%store-prefix)) + (begin + (cache-narinfo! url (narinfo-path narinfo) narinfo ttl) + (cons narinfo result)) + result))) + ((301 302 303 307 308) ; redirect + (let* ((uri (response-location response)) + (new-request (build-request + uri #:headers '((User-Agent . "GNU Guile"))))) + (if len + (get-bytevector-n port len) + (read-to-eof port)) + (append + (http-multiple-get uri + handle-narinfo-response '() + (list new-request) + #:open-connection + open-connection-for-uri/cached + #:verify-certificate? #f) + result))) + (else + (update-progress!) (let* ((path (uri-path (request-uri request))) (hash-part (basename (string-drop-right path 8)))) ;drop ".narinfo" @@ -685,7 +703,7 @@ port to it, or, if connection failed, print a warning and return #f. Pass (if (or (= 404 code) (= 202 code)) ttl %narinfo-transient-error-ttl)) - result)))) + result))))) (define (do-fetch uri) (case (and=> uri uri-scheme) -- 2.29.2 --MP_/HMfGdeue67HATiQgj_SBCh/-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 12 16:34:05 2021 Received: (at 45174) by debbugs.gnu.org; 12 Jan 2021 21:34:05 +0000 Received: from localhost ([127.0.0.1]:33476 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kzRIj-000456-9l for submit@debbugs.gnu.org; Tue, 12 Jan 2021 16:34:05 -0500 Received: from world.peace.net ([64.112.178.59]:39494) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kzRIg-00044Z-34 for 45174@debbugs.gnu.org; Tue, 12 Jan 2021 16:34:04 -0500 Received: from mhw by world.peace.net with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kzRIX-0006RT-Dr; Tue, 12 Jan 2021 16:33:53 -0500 From: Mark H Weaver To: Julien Lepiller , Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#45174: =?utf-8?Q?=E2=80=98guix_substitute=E2=80=99_doesn?= =?utf-8?Q?=E2=80=99t?= handle HTTP redirects In-Reply-To: <20210112181155.05b7992d@tachikoma.lepiller.eu> References: <87a6uk8ydv.fsf@inria.fr> <20210112181155.05b7992d@tachikoma.lepiller.eu> Date: Tue, 12 Jan 2021 16:32:39 -0500 Message-ID: <87pn294z8t.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45174 Cc: 45174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Julien, Julien Lepiller writes: > Here is a patch to fix that issue. Since ci.guix.info now returns 200, > it's difficult to test the patch. [...] [...] > From f20e01f2a8df538519660772a7431b53d650d64f Mon Sep 17 00:00:00 2001 > From: Julien Lepiller > Date: Tue, 12 Jan 2021 18:07:25 +0100 > Subject: [PATCH] substitute: Follow narinfo redirections. > > * guix/scripts/substitute.scm (fetch-narinfos): Follow redirections. > --- > guix/scripts/substitute.scm | 38 +++++++++++++++++++++++++++---------- > 1 file changed, 28 insertions(+), 10 deletions(-) > > diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm > index e53de8c304..790168091e 100755 > --- a/guix/scripts/substitute.scm > +++ b/guix/scripts/substitute.scm > @@ -663,18 +663,36 @@ port to it, or, if connection failed, print a warning and return #f. Pass [...] > + ((301 302 303 307 308) ; redirect > + (let* ((uri (response-location response)) > + (new-request (build-request > + uri #:headers '((User-Agent . "GNU Guile"))))) > + (if len > + (get-bytevector-n port len) > + (read-to-eof port)) > + (append > + (http-multiple-get uri > + handle-narinfo-response '() > + (list new-request) > + #:open-connection > + open-connection-for-uri/cached > + #:verify-certificate? #f) > + result))) Granted, it's been almost six years since I first implemented proper HTTP redirects for Guix, but as I vaguely recall the URI in the response may be a relative URI or have some missing components, so in the general case it must be interpreted relative to the previous URI in accordance with RFC 3986 section 5.2. A proper implementation should use 'resolve-uri-reference' from (guix build download). Here's the original commit that added that function, and used it to fix HTTP redirection support in (guix http-client): https://git.savannah.gnu.org/cgit/guix.git/commit/?id=04dec194d8e460831ec0695a944d9c7313affea2 Also, keep in mind that multiple redirects may occur, so a proper implementation requires some kind of loop. I haven't looked closely enough at your code above to know whether that case is handled correctly. See the relevant code in (guix http-client) for hints. Anyway, thanks for working on it! Regards, Mark