From unknown Mon Aug 18 14:20:27 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#45323 <45323@debbugs.gnu.org> To: bug#45323 <45323@debbugs.gnu.org> Subject: Status: [PATCH] substitute: Reuse connections for '--query'. Reply-To: bug#45323 <45323@debbugs.gnu.org> Date: Mon, 18 Aug 2025 21:20:27 +0000 retitle 45323 [PATCH] substitute: Reuse connections for '--query'. reassign 45323 guix-patches submitter 45323 Ludovic Court=C3=A8s severity 45323 normal tag 45323 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Sat Dec 19 09:50:04 2020 Received: (at submit) by debbugs.gnu.org; 19 Dec 2020 14:50:04 +0000 Received: from localhost ([127.0.0.1]:41097 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqdYZ-0001on-Fx for submit@debbugs.gnu.org; Sat, 19 Dec 2020 09:50:04 -0500 Received: from lists.gnu.org ([209.51.188.17]:47080) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqdYW-0001oJ-2E for submit@debbugs.gnu.org; Sat, 19 Dec 2020 09:50:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55652) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kqdYV-0006LY-N6 for guix-patches@gnu.org; Sat, 19 Dec 2020 09:49:59 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:56516) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kqdYU-0000Nj-Pr; Sat, 19 Dec 2020 09:49:58 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51302 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kqdYU-0002Hd-A9; Sat, 19 Dec 2020 09:49:58 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH] substitute: Reuse connections for '--query'. Date: Sat, 19 Dec 2020 15:49:52 +0100 Message-Id: <20201219144952.2725-1-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) This significantly speeds up things like substituting the closure of a .drv. This is a followup to 5ff521452b9ec2aae9ed8e4bb7bdc250a581f203. * guix/scripts/substitute.scm (http-multiple-get): Add #:open-connection and #:keep-alive? and honor them. (open-connection-for-uri/maybe): Use 'open-connection-for-uri/cached' instead of 'guix:open-connection-for-uri'. Call 'http-multiple-get' within 'call-with-cached-connection'. (open-connection-for-uri/cached): Add #:timeout and #:verify-certificate? and honor them. (call-with-cached-connection): Add 'open-connection' parameter and honor it. --- guix/scripts/substitute.scm | 97 ++++++++++++++++++++++--------------- 1 file changed, 59 insertions(+), 38 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 38702d0c4b..8084c89ae5 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -514,12 +514,18 @@ return its MAX-LENGTH first elements and its tail." (define* (http-multiple-get base-uri proc seed requests #:key port (verify-certificate? #t) + (open-connection guix:open-connection-for-uri) + (keep-alive? #t) (batch-size 1000)) "Send all of REQUESTS to the server at BASE-URI. Call PROC for each response, passing it the request object, the response, a port from which to read the response body, and the previous result, starting with SEED, à la -'fold'. Return the final result. When PORT is specified, use it as the -initial connection on which HTTP requests are sent." +'fold'. Return the final result. + +When PORT is specified, use it as the initial connection on which HTTP +requests are sent; otherwise call OPEN-CONNECTION to open a new connection for +a URI. When KEEP-ALIVE? is false, close the connection port before +returning." (let connect ((port port) (requests requests) (result seed)) @@ -528,10 +534,9 @@ initial connection on which HTTP requests are sent." ;; (format (current-error-port) "connecting (~a requests left)..." ;; (length requests)) - (let ((p (or port (guix:open-connection-for-uri - base-uri - #:verify-certificate? - verify-certificate?)))) + (let ((p (or port (open-connection base-uri + #:verify-certificate? + verify-certificate?)))) ;; For HTTPS, P is not a file port and does not support 'setvbuf'. (when (file-port? p) (setvbuf p 'block (expt 2 16))) @@ -556,7 +561,8 @@ initial connection on which HTTP requests are sent." (() (match (drop requests processed) (() - (close-port p) + (unless keep-alive? + (close-port p)) (reverse result)) (remainder (connect p remainder result)))) @@ -598,18 +604,18 @@ if file doesn't exist, and the narinfo otherwise." (define* (open-connection-for-uri/maybe uri #:key - (verify-certificate? #f) + fresh? (time %fetch-timeout)) - "Open a connection to URI and return a port to it, or, if connection failed, -print a warning and return #f." + "Open a connection to URI via 'open-connection-for-uri/cached' and return a +port to it, or, if connection failed, print a warning and return #f. Pass +#:fresh? to 'open-connection-for-uri/cached'." (define host (uri-host uri)) (catch #t (lambda () - (guix:open-connection-for-uri uri - #:verify-certificate? verify-certificate? - #:timeout time)) + (open-connection-for-uri/cached uri #:timeout time + #:fresh? fresh?)) (match-lambda* (('getaddrinfo-error error) (unless (hash-ref %unreachable-hosts host) @@ -683,23 +689,26 @@ print a warning and return #f." (define (do-fetch uri) (case (and=> uri uri-scheme) ((http https) - (let ((requests (map (cut narinfo-request url <>) paths))) - (match (open-connection-for-uri/maybe uri) - (#f - '()) - (port - (update-progress!) - ;; Note: Do not check HTTPS server certificates to avoid depending - ;; on the X.509 PKI. We can do it because we authenticate - ;; narinfos, which provides a much stronger guarantee. - (let ((result (http-multiple-get uri - handle-narinfo-response '() - requests - #:verify-certificate? #f - #:port port))) - (close-port port) - (newline (current-error-port)) - result))))) + ;; Note: Do not check HTTPS server certificates to avoid depending + ;; on the X.509 PKI. We can do it because we authenticate + ;; narinfos, which provides a much stronger guarantee. + (let* ((requests (map (cut narinfo-request url <>) paths)) + (result (call-with-cached-connection uri + (lambda (port) + (if port + (begin + (update-progress!) + (http-multiple-get uri + handle-narinfo-response '() + requests + #:open-connection + open-connection-for-uri/cached + #:verify-certificate? #f + #:port port)) + '())) + open-connection-for-uri/maybe))) + (newline (current-error-port)) + result)) ((file #f) (let* ((base (string-append (uri-path uri) "/")) (files (map (compose (cut string-append base <> ".narinfo") @@ -990,10 +999,14 @@ the URI, its compression method (a string), and the compressed file size." (define open-connection-for-uri/cached (let ((cache '())) - (lambda* (uri #:key fresh?) + (lambda* (uri #:key fresh? timeout verify-certificate?) "Return a connection for URI, possibly reusing a cached connection. -When FRESH? is true, delete any cached connections for URI and open a new -one. Return #f if URI's scheme is 'file' or #f." +When FRESH? is true, delete any cached connections for URI and open a new one. +Return #f if URI's scheme is 'file' or #f. + +When true, TIMEOUT is the maximum number of milliseconds to wait for +connection establishment. When VERIFY-CERTIFICATE? is true, verify HTTPS +server certificates." (define host (uri-host uri)) (define scheme (uri-scheme uri)) (define key (list host scheme (uri-port uri))) @@ -1005,7 +1018,9 @@ one. Return #f if URI's scheme is 'file' or #f." ;; CACHE, if any. (let-values (((socket) (guix:open-connection-for-uri - uri #:verify-certificate? #f)) + uri + #:verify-certificate? verify-certificate? + #:timeout timeout)) ((new-cache evicted) (at-most (- %max-cached-connections 1) cache))) (for-each (match-lambda @@ -1019,14 +1034,19 @@ one. Return #f if URI's scheme is 'file' or #f." (begin (false-if-exception (close-port socket)) (set! cache (alist-delete key cache)) - (open-connection-for-uri/cached uri)) + (open-connection-for-uri/cached uri #:timeout timeout + #:verify-certificate? + verify-certificate?)) (begin ;; Drain input left from the previous use. (drain-input socket) socket)))))))) -(define (call-with-cached-connection uri proc) - (let ((port (open-connection-for-uri/cached uri))) +(define* (call-with-cached-connection uri proc + #:optional + (open-connection + open-connection-for-uri/cached)) + (let ((port (open-connection uri))) (catch #t (lambda () (proc port)) @@ -1038,7 +1058,7 @@ one. Return #f if URI's scheme is 'file' or #f." (if (or (and (eq? key 'system-error) (= EPIPE (system-error-errno `(,key ,@args)))) (memq key '(bad-response bad-header bad-header-component))) - (proc (open-connection-for-uri/cached uri #:fresh? #t)) + (proc (open-connection uri #:fresh? #t)) (apply throw key args)))))) (define-syntax-rule (with-cached-connection uri port exp ...) @@ -1341,6 +1361,7 @@ default value." ;;; Local Variables: ;;; eval: (put 'with-timeout 'scheme-indent-function 1) ;;; eval: (put 'with-cached-connection 'scheme-indent-function 2) +;;; eval: (put 'call-with-cached-connection 'scheme-indent-function 1) ;;; End: ;;; substitute.scm ends here -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 23 10:06:32 2020 Received: (at 45323-done) by debbugs.gnu.org; 23 Dec 2020 15:06:32 +0000 Received: from localhost ([127.0.0.1]:53809 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ks5ih-0002qx-N3 for submit@debbugs.gnu.org; Wed, 23 Dec 2020 10:06:32 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51848) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ks5ig-0002qm-JX for 45323-done@debbugs.gnu.org; Wed, 23 Dec 2020 10:06:30 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51347) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ks5iV-0001XW-U9 for 45323-done@debbugs.gnu.org; Wed, 23 Dec 2020 10:06:21 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=46056 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ks5iR-0005PM-Uh for 45323-done@debbugs.gnu.org; Wed, 23 Dec 2020 10:06:16 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 45323-done@debbugs.gnu.org Subject: Re: [bug#45323] [PATCH] substitute: Reuse connections for '--query'. References: <20201219144952.2725-1-ludo@gnu.org> Date: Wed, 23 Dec 2020 16:06:14 +0100 In-Reply-To: <20201219144952.2725-1-ludo@gnu.org> ("Ludovic =?utf-8?Q?Cour?= =?utf-8?Q?t=C3=A8s=22's?= message of "Sat, 19 Dec 2020 15:49:52 +0100") Message-ID: <87o8ikwoih.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45323-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Ludovic Court=C3=A8s skribis: > This significantly speeds up things like substituting the closure of a > .drv. This is a followup to 5ff521452b9ec2aae9ed8e4bb7bdc250a581f203. > > * guix/scripts/substitute.scm (http-multiple-get): Add #:open-connection > and #:keep-alive? and honor them. > (open-connection-for-uri/maybe): Use 'open-connection-for-uri/cached' > instead of 'guix:open-connection-for-uri'. Call 'http-multiple-get' > within 'call-with-cached-connection'. > (open-connection-for-uri/cached): Add #:timeout and #:verify-certificate? > and honor them. > (call-with-cached-connection): Add 'open-connection' parameter and > honor it. > --- > guix/scripts/substitute.scm | 97 ++++++++++++++++++++++--------------- > 1 file changed, 59 insertions(+), 38 deletions(-) Pushed as be5a75ebb5988b87b2392e2113f6590f353dd6cd! You can check the effect by running =E2=80=98guix build XYZ.drv=E2=80=99, w= here XYZ.drv is not available locally yet. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 24 06:06:50 2020 Received: (at 45323) by debbugs.gnu.org; 24 Dec 2020 11:06:50 +0000 Received: from localhost ([127.0.0.1]:54834 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ksOSI-0000OU-0a for submit@debbugs.gnu.org; Thu, 24 Dec 2020 06:06:50 -0500 Received: from mira.cbaines.net ([212.71.252.8]:52674) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ksOSD-0000OJ-AU for 45323@debbugs.gnu.org; Thu, 24 Dec 2020 06:06:48 -0500 Received: from localhost (188.29.98.108.threembb.co.uk [188.29.98.108]) by mira.cbaines.net (Postfix) with ESMTPSA id 5618127BC05; Thu, 24 Dec 2020 11:06:44 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 706c6008; Thu, 24 Dec 2020 11:06:42 +0000 (UTC) References: <20201219144952.2725-1-ludo@gnu.org> <87o8ikwoih.fsf@gnu.org> User-agent: mu4e 1.4.13; emacs 27.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#45323: [PATCH] substitute: Reuse connections for '--query'. In-reply-to: <87o8ikwoih.fsf@gnu.org> Date: Thu, 24 Dec 2020 11:06:39 +0000 Message-ID: <871rffbgzk.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45323 Cc: 45323@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Ludovic Court=C3=A8s skribis: > >> This significantly speeds up things like substituting the closure of a >> .drv. This is a followup to 5ff521452b9ec2aae9ed8e4bb7bdc250a581f203. >> >> * guix/scripts/substitute.scm (http-multiple-get): Add #:open-connection >> and #:keep-alive? and honor them. >> (open-connection-for-uri/maybe): Use 'open-connection-for-uri/cached' >> instead of 'guix:open-connection-for-uri'. Call 'http-multiple-get' >> within 'call-with-cached-connection'. >> (open-connection-for-uri/cached): Add #:timeout and #:verify-certificate? >> and honor them. >> (call-with-cached-connection): Add 'open-connection' parameter and >> honor it. >> --- >> guix/scripts/substitute.scm | 97 ++++++++++++++++++++++--------------- >> 1 file changed, 59 insertions(+), 38 deletions(-) > > Pushed as be5a75ebb5988b87b2392e2113f6590f353dd6cd! > > You can check the effect by running =E2=80=98guix build XYZ.drv=E2=80=99,= where XYZ.drv > is not available locally yet. Hey, I did do some testing of this, and didn't spot any issues, but I think it might be causing some issues when things go wrong. The Guix Build Coordinator uses code from this script, and I'm sometimes seeing exceptions like [1] when running with these changes. This is when calling lookup-narinfos. 1: #<&compound-exception components: (#<&error> #<&irritants irritants: (# write_to_session_record_port)> #<&exception-with-kind-and-args kin= d: gnutls-error args: (# write_to_session_record_port)>)>, When this happens, things seem to get stuck and retrying calling lookup-narinfos leads to the same exception. I'm guessing this might be happening because the broken connection is being cached and reused. Any ideas? Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl/kdj9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfkoxAArD6tfSwRpQ3GBdRNImgsseI1/kfjKZPb qaJr7BHyrkS/uqbbx3nwp5mN5NfvyI13Jqx/mHy1mblm79n22d61p7YFXP+eLxEZ utNiwdOLh8/xvNfo+YPPnap69IksGxQBm03e8XOi7tqyhMeWrkxuxwHLHBnDitXt JTZdk88xOsaMHT0QjBExMh7/JSq3wgrEENaP+BGuiBjDI+Fg4fmRKGRO2HKKV1H2 Y+Cb5ElSoHgyx2M1yEpWLlXTdo/3NAoPJi8/Iv2MtT2Rqi81M0ZmB6dN9D3C0BoT rSHav6RnIfXOBBNoKepf5fyFu3aTWfTtUJy3pL2WB08pOBv9hLWcALajZWYGazIZ XmoRbvxr73sBZFpl7JovyXVovAeX09memFz5Loxb++wSZfcoWvonGStx734drkNN C4Qd9P4cApxTzj1pyU03eLXqdvBHi7Wx3/YyZv1rad4NLeZwIOYYc1UVgcQfQ11b LP20LqDD3O+UkTgf50Q7Fi+fw+kL1U8Ifp4H6D9PRW9IK1mU14gtbLpbDOv+FPK2 M/fJ6DtN4y7LJWBVjAwbaAjCOKRBfR5KxnuxrKoFEOFnWickKxByCB8RxoouPi11 nJTbaoqeamrWG5x3IKvGKCb1nGjaGfajRJVfoCi1ExYIms+4AIbgGvLGVaIgNlSX coR0FNCOt7I= =D5uU -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Dec 27 09:57:53 2020 Received: (at 45323) by debbugs.gnu.org; 27 Dec 2020 14:57:53 +0000 Received: from localhost ([127.0.0.1]:33228 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ktXUX-00034R-A8 for submit@debbugs.gnu.org; Sun, 27 Dec 2020 09:57:53 -0500 Received: from eggs.gnu.org ([209.51.188.92]:36286) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ktXUV-00034E-RW for 45323@debbugs.gnu.org; Sun, 27 Dec 2020 09:57:52 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50018) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ktXUQ-0001Lv-EI; Sun, 27 Dec 2020 09:57:46 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51538 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ktXUO-000739-7E; Sun, 27 Dec 2020 09:57:45 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#45323: [PATCH] substitute: Reuse connections for '--query'. References: <20201219144952.2725-1-ludo@gnu.org> <87o8ikwoih.fsf@gnu.org> <871rffbgzk.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 7 =?utf-8?Q?Niv=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 27 Dec 2020 15:57:42 +0100 In-Reply-To: <871rffbgzk.fsf@cbaines.net> (Christopher Baines's message of "Thu, 24 Dec 2020 11:06:39 +0000") Message-ID: <87tus7thy1.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45323 Cc: 45323@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi! Christopher Baines skribis: > The Guix Build Coordinator uses code from this script, and I'm sometimes > seeing exceptions like [1] when running with these changes. This is when > calling lookup-narinfos. > > 1: > #<&compound-exception components: (#<&error> #<&irritants irritants: (# reason.> write_to_session_record_port)> #<&exception-with-kind-and-args k= ind: gnutls-error args: (# write_to_session_record_port)>)>, > > When this happens, things seem to get stuck and retrying calling > lookup-narinfos leads to the same exception. I'm guessing this might be > happening because the broken connection is being cached and reused. Ah, that looks like another thing that might break. Does the patch below help? Thanks for reporting it, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 8084c89ae5..e53de8c304 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -43,6 +43,7 @@ (open-connection-for-uri . guix:open-connection-for-uri) store-path-abbreviation byte-count->string)) + #:autoload (gnutls) (error/invalid-session) #:use-module (guix progress) #:use-module ((guix build syscalls) #:select (set-thread-name)) @@ -1054,9 +1055,12 @@ server certificates." ;; If PORT was cached and the server closed the connection in the ;; meantime, we get EPIPE. In that case, open a fresh connection and ;; retry. We might also get 'bad-response or a similar exception from - ;; (web response) later on, once we've sent the request. + ;; (web response) later on, once we've sent the request, or a + ;; ERROR/INVALID-SESSION from GnuTLS. (if (or (and (eq? key 'system-error) (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session)) (memq key '(bad-response bad-header bad-header-component))) (proc (open-connection uri #:fresh? #t)) (apply throw key args)))))) --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 30 17:56:02 2020 Received: (at 45323) by debbugs.gnu.org; 30 Dec 2020 22:56:02 +0000 Received: from localhost ([127.0.0.1]:51441 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kukNu-0001T9-FG for submit@debbugs.gnu.org; Wed, 30 Dec 2020 17:56:02 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43628) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kukNs-0001SR-H6 for 45323@debbugs.gnu.org; Wed, 30 Dec 2020 17:56:01 -0500 Received: from localhost (92.41.186.20.threembb.co.uk [92.41.186.20]) by mira.cbaines.net (Postfix) with ESMTPSA id 5A48927BC05; Wed, 30 Dec 2020 22:55:59 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 52604a6b; Wed, 30 Dec 2020 22:55:56 +0000 (UTC) References: <20201219144952.2725-1-ludo@gnu.org> <87o8ikwoih.fsf@gnu.org> <871rffbgzk.fsf@cbaines.net> <87tus7thy1.fsf@gnu.org> User-agent: mu4e 1.4.13; emacs 27.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#45323: [PATCH] substitute: Reuse connections for '--query'. In-reply-to: <87tus7thy1.fsf@gnu.org> Date: Wed, 30 Dec 2020 22:55:56 +0000 Message-ID: <87zh1u9a4j.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45323 Cc: 45323@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi! > > Christopher Baines skribis: > >> The Guix Build Coordinator uses code from this script, and I'm sometimes >> seeing exceptions like [1] when running with these changes. This is when >> calling lookup-narinfos. >> >> 1: >> #<&compound-exception components: (#<&error> #<&irritants irritants: (#<= gnutls-error-enum The specified session has been invalidated for some >> reason.> write_to_session_record_port)> #<&exception-with-kind-and-args = kind: gnutls-error args: (# write_to_session_record_port)>)>, >> >> When this happens, things seem to get stuck and retrying calling >> lookup-narinfos leads to the same exception. I'm guessing this might be >> happening because the broken connection is being cached and reused. > > Ah, that looks like another thing that might break. Does the patch > below help? I've tried using it, and I haven't spotted any problems yet, so I believe so. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl/tBXxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfN5A/+L5ttW8BPV9j4RDe2b8l7BmVEv9g+HdFh EXMEReI6tcSHIoeT1VHLjq9EGpluiCGTNavKBS4o5Nz9i7sXfXIloBCqzuXLzY0i V1BuCfChulLmgTQSar89V4eCfk6C+EAkw/Lq5vHVmDjtQT63mUJ8fnSWNvOw59Tq M9tGa8UPgu6h4urygVqkkjSxGihOWqvILCkcKssyhXhpSei6tPu//IzZ0A42af3+ lRTZdhlbhKBvOedCPP/P4VEGX+d4HtSacSFJ6Q/y5Quznk/oacqy7SD4ZaT9QQSC Qx2IiTxVaF2GdvLw2+SU4qvtEjfIO1bsWfg8qtPWjYULyF02oMPLzIFOE35DaAWY aZngSvqbpoUF9O4jne/sIpaKYapcxDo6Yw+oRpE+hO6ki7unTiSP/VRDM2fWtfBy j18tOuhRBaP+cA9rDgK3UKOSi5RFFTAXVWd4MuGX31RIDaIBt5IPBnPaGeRjjXa3 jFVKLIJvHYc5+0UdmmX9aTGqeYkP9vpmVgmFrzWm8bQ02FmJ+IupsWiPazvs4ijt 5sp/ttdAH60CTr3ORWP36mHYzcP7HcrhqYNdEUAr7lXUWnwDS8GVd2blY2xIjllA c3Qr0kv7/+P4oAZuqBMxtYfqkmj1gXVeuOjFVU97eqP15EQLgTXKeOZOEzv/Tbv4 kl8jNCGP6zg= =AOHt -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 04 05:55:50 2021 Received: (at 45323) by debbugs.gnu.org; 4 Jan 2021 10:55:50 +0000 Received: from localhost ([127.0.0.1]:56038 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kwNWg-00050Y-Av for submit@debbugs.gnu.org; Mon, 04 Jan 2021 05:55:50 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33672) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kwNWf-00050L-5a for 45323@debbugs.gnu.org; Mon, 04 Jan 2021 05:55:49 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:60473) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kwNWZ-0002zM-RJ; Mon, 04 Jan 2021 05:55:43 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=35682 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kwNWX-00078r-Vl; Mon, 04 Jan 2021 05:55:42 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#45323: [PATCH] substitute: Reuse connections for '--query'. References: <20201219144952.2725-1-ludo@gnu.org> <87o8ikwoih.fsf@gnu.org> <871rffbgzk.fsf@cbaines.net> <87tus7thy1.fsf@gnu.org> <87zh1u9a4j.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 15 =?utf-8?Q?Niv=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 04 Jan 2021 11:55:40 +0100 In-Reply-To: <87zh1u9a4j.fsf@cbaines.net> (Christopher Baines's message of "Wed, 30 Dec 2020 22:55:56 +0000") Message-ID: <87turxvumr.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45323 Cc: 45323@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Christopher Baines skribis: > Ludovic Court=C3=A8s writes: [...] >>> #<&compound-exception components: (#<&error> #<&irritants irritants: (#= >> reason.> write_to_session_record_port)> #<&exception-with-kind-and-args= kind: gnutls-error args: (# write_to_session_record_port)>)>, >>> >>> When this happens, things seem to get stuck and retrying calling >>> lookup-narinfos leads to the same exception. I'm guessing this might be >>> happening because the broken connection is being cached and reused. >> >> Ah, that looks like another thing that might break. Does the patch >> below help? > > I've tried using it, and I haven't spotted any problems yet, so I > believe so. Pushed as 9158020d7853b6e7925802e0d0a082801c680e8f, thanks! Ludo=E2=80=99. From unknown Mon Aug 18 14:20:27 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 01 Feb 2021 12:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator