From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 15 15:21:57 2021 Received: (at submit) by debbugs.gnu.org; 15 Mar 2021 19:21:57 +0000 Received: from localhost ([127.0.0.1]:37209 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLsmq-0006yN-S4 for submit@debbugs.gnu.org; Mon, 15 Mar 2021 15:21:57 -0400 Received: from lists.gnu.org ([209.51.188.17]:50388) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLsmo-0006yF-FR for submit@debbugs.gnu.org; Mon, 15 Mar 2021 15:21:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58256) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lLsml-0000NJ-Jg for guix-patches@gnu.org; Mon, 15 Mar 2021 15:21:54 -0400 Received: from mira.cbaines.net ([212.71.252.8]:53620) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLsme-0008W0-Lq for guix-patches@gnu.org; Mon, 15 Mar 2021 15:21:51 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 2E6D727BC52 for ; Mon, 15 Mar 2021 19:21:43 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 75122fc1 for ; Mon, 15 Mar 2021 19:21:42 +0000 (UTC) User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines To: guix-patches@gnu.org Subject: [PATCH 0/2] substitute: Handle closing connections to substitute servers. Date: Mon, 15 Mar 2021 19:21:41 +0000 Message-ID: <87y2eodxyy.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) --=-=-= Content-Type: text/plain Christopher Baines (2): guix: Alter http-fetch to return the response. substitute: Handle closing connections to substitute servers. guix/build/download-nar.scm | 5 +++-- guix/build/download.scm | 9 ++++++--- guix/http-client.scm | 12 ++++++------ guix/scripts/substitute.scm | 31 ++++++++++++++++++++++++------- 4 files changed, 39 insertions(+), 18 deletions(-) --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBPs8VfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xef2A/+Lm+zmZgKgVlBPQz+dN6k5H9MnTXQNU3m iPPZ0BeaY9d8ZTg1zSnwLeF9kRQsm3Z/VlU0wmwv56AO3RS2U6q8iXzk+V6kmui/ 0a+c84k+TsIkFcyEJmQlGXXvcoGZpGi2BHMo4fdAsKdKkTW3nIk+fHfc0fJ31wuO skHZF5Q6KRIGyURtym9w2NB4fBjoesjJqsZ7Q55OXgViGITfDqCGprdlF1cxdGvt bNNCoNj26bOzlT7D10rW9IFZWJrG+6CGxfM0+d2LaW30viO+P8RPxLs5UfkcGsTE 41/1VCXUoCBAR2/GRoPGiZsYNIznLuV+/hqD+vNLh1nIctybX0cld74uttSROImp 4RiJy4YBdjnYu4e//rzui0jIU7QfX/iVSCYtgt6ymmsNVNfklzU0CEgrBHln8Dha RRWVGY2R4SWN57K2p5CSjDQiQzH37IP+N1WRKeebCR8Brub7mcA3d5kFprvz/BS3 2AvrNUkVnxBw2D4Vk30QB2OMDofIfxLBJmNsZbPsG87BUjaspOB8kPvzSl00RnBr WFICIPwEa5vAFv/N6fOQrkNCH2eNbK3R/2LEcLeNWTRq7UiXu1zL7kNtfjyZ4sXZ C/zE+xMXGkCcLD9QeubMCuW3usRJAQgdNra6pl71FCE1VxyjC30/JBOTjBlXmHPG ICJyXcsD3QQ= =SskM -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 15 15:24:52 2021 Received: (at 47174) by debbugs.gnu.org; 15 Mar 2021 19:24:52 +0000 Received: from localhost ([127.0.0.1]:37215 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLspg-00073F-Bn for submit@debbugs.gnu.org; Mon, 15 Mar 2021 15:24:52 -0400 Received: from mira.cbaines.net ([212.71.252.8]:35640) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLspe-000731-CP for 47174@debbugs.gnu.org; Mon, 15 Mar 2021 15:24:50 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 6C29B27BC54 for <47174@debbugs.gnu.org>; Mon, 15 Mar 2021 19:24:49 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 1dfa5663 for <47174@debbugs.gnu.org>; Mon, 15 Mar 2021 19:24:49 +0000 (UTC) From: Christopher Baines To: 47174@debbugs.gnu.org Subject: [PATCH 2/2] substitute: Handle closing connections to substitute servers. Date: Mon, 15 Mar 2021 19:24:49 +0000 Message-Id: <20210315192449.16248-2-mail@cbaines.net> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210315192449.16248-1-mail@cbaines.net> References: <20210315192449.16248-1-mail@cbaines.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) When reusing a HTTP connection to fetch multiple nars, and the remote server signals that the connection should be closed. * guix/scripts/substitute.scm (process-substitution): Close connections to substitute servers when a Connection: close header is specified in the response. --- guix/scripts/substitute.scm | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index cb79ea6927..deb6fbdaa2 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -406,7 +406,9 @@ the current output port." (case (uri-scheme uri) ((file) (let ((port (open-file (uri-path uri) "r0b"))) - (values port (stat:size (stat port))))) + (values port + (stat:size (stat port)) + (const #t)))) ; no cleanup to do ((http https) (guard (c ((http-get-error? c) (leave (G_ "download from '~a' failed: ~a, ~s~%") @@ -434,7 +436,12 @@ the current output port." #:buffered? #f #:verify-certificate? #f))) (values raw - (response-content-length response)))))))) + (response-content-length response) + (match (assq 'connection (response-headers response)) + (('connection 'close) + (lambda () + (close-port (response-port response)))) + (_ (const #t)))))))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) @@ -449,7 +456,7 @@ the current output port." (format (current-error-port) (G_ "Downloading ~a...~%") (uri->string uri))) - (let*-values (((raw download-size) + (let*-values (((raw download-size post-fetch-cleanup) ;; 'guix publish' without '--cache' doesn't specify a ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. (fetch uri)) @@ -493,6 +500,10 @@ the current output port." ;; Wait for the reporter to finish. (every (compose zero? cdr waitpid) pids) + ;; Do post-fetch cleanup, maybe closing the HTTP connection if HTTP is + ;; being used, and the connection should be closed + (post-fetch-cleanup) + ;; Skip a line after what 'progress-reporter/file' printed, and another ;; one to visually separate substitutions. (display "\n\n" (current-error-port)) -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 15 15:24:53 2021 Received: (at 47174) by debbugs.gnu.org; 15 Mar 2021 19:24:53 +0000 Received: from localhost ([127.0.0.1]:37217 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLspg-00073H-Mh for submit@debbugs.gnu.org; Mon, 15 Mar 2021 15:24:53 -0400 Received: from mira.cbaines.net ([212.71.252.8]:35638) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLspe-000730-CM for 47174@debbugs.gnu.org; Mon, 15 Mar 2021 15:24:50 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 5FFE527BC52 for <47174@debbugs.gnu.org>; Mon, 15 Mar 2021 19:24:49 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 55fff41d for <47174@debbugs.gnu.org>; Mon, 15 Mar 2021 19:24:49 +0000 (UTC) From: Christopher Baines To: 47174@debbugs.gnu.org Subject: [PATCH 1/2] guix: Alter http-fetch to return the response. Date: Mon, 15 Mar 2021 19:24:48 +0000 Message-Id: <20210315192449.16248-1-mail@cbaines.net> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Rather than just the port and response-content-length. I'm looking at using the response headers within the substitute script to work out when to close the connection. * guix/http-client.scm (http-fetch): Return the response as the second value, rather than the response-content-length. * guix/build/download-nar.scm (download-nar): Adapt accordingly. * guix/build/download.scm (url-fetch): Adapt accordingly. * guix/scripts/substitute.scm (process-substitution): Adapt accordingly. --- guix/build/download-nar.scm | 5 +++-- guix/build/download.scm | 9 ++++++--- guix/http-client.scm | 12 ++++++------ guix/scripts/substitute.scm | 16 +++++++++++----- 4 files changed, 26 insertions(+), 16 deletions(-) diff --git a/guix/build/download-nar.scm b/guix/build/download-nar.scm index 867f3c10bb..fbb5d37c0a 100644 --- a/guix/build/download-nar.scm +++ b/guix/build/download-nar.scm @@ -23,6 +23,7 @@ #:autoload (zlib) (call-with-gzip-input-port) #:use-module (guix progress) #:use-module (web uri) + #:use-module (web response) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (ice-9 format) @@ -101,7 +102,7 @@ success, #f otherwise." ((url rest ...) (format #t "Trying content-addressed mirror at ~a...~%" (uri-host (string->uri url))) - (let-values (((port size) + (let-values (((port resp) (catch #t (lambda () (http-fetch (string->uri url))) @@ -109,7 +110,7 @@ success, #f otherwise." (values #f #f))))) (if (not port) (loop rest) - (begin + (let ((size (response-content-length resp))) (if size (format #t "Downloading from ~a (~,2h MiB)...~%" url (/ size (expt 2 20.))) diff --git a/guix/build/download.scm b/guix/build/download.scm index f24a1e20df..437184b9cb 100644 --- a/guix/build/download.scm +++ b/guix/build/download.scm @@ -21,6 +21,7 @@ (define-module (guix build download) #:use-module (web uri) #:use-module (web http) + #:use-module (web response) #:use-module ((web client) #:hide (open-socket-for-uri)) #:use-module (web response) #:use-module (guix base64) @@ -647,7 +648,7 @@ otherwise simply ignore them." (case (uri-scheme uri) ((http https) (false-if-exception* - (let-values (((port size) + (let-values (((port resp) (http-fetch uri #:verify-certificate? verify-certificate? #:timeout timeout))) @@ -657,9 +658,11 @@ otherwise simply ignore them." #:buffer-size %http-receive-buffer-size #:reporter (if print-build-trace? (progress-reporter/trace - file (uri->string uri) size) + file (uri->string uri) + (response-content-length resp)) (progress-reporter/file - (uri-abbreviation uri) size))) + (uri-abbreviation uri) + (response-content-length resp)))) (newline))) file))) ((ftp) diff --git a/guix/http-client.scm b/guix/http-client.scm index 2d7458a56e..47076d41f6 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -80,11 +80,11 @@ (verify-certificate? #t) (headers '((user-agent . "GNU Guile"))) timeout) - "Return an input port containing the data at URI, and the expected number of -bytes available or #f. If TEXT? is true, the data at URI is considered to be -textual. Follow any HTTP redirection. When BUFFERED? is #f, return an -unbuffered port, suitable for use in `filtered-port'. HEADERS is an alist of -extra HTTP headers. + "Return an input port containing the data at URI, and the HTTP response from +the server. If TEXT? is true, the data at URI is considered to be textual. +Follow any HTTP redirection. When BUFFERED? is #f, return an unbuffered port, +suitable for use in `filtered-port'. HEADERS is an alist of extra HTTP +headers. When KEEP-ALIVE? is true, the connection is marked as 'keep-alive' and PORT is not closed upon completion. @@ -120,7 +120,7 @@ Raise an '&http-get-error' condition if downloading fails." (response-code resp))) (case code ((200) - (values data (response-content-length resp))) + (values data resp)) ((301 ; moved permanently 302 ; found (redirection) 303 ; see other diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 6892aa999b..cb79ea6927 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -60,6 +60,7 @@ #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) #:use-module (web uri) + #:use-module (web response) #:use-module (guix http-client) #:export (%allow-unauthenticated-substitutes? %error-to-file-descriptor-4? @@ -424,11 +425,16 @@ the current output port." (call-with-connection-error-handling uri (lambda () - (http-fetch uri #:text? #f - #:open-connection open-connection-for-uri/cached - #:keep-alive? #t - #:buffered? #f - #:verify-certificate? #f)))))) + (let-values (((raw response) + (http-fetch + uri + #:text? #f + #:open-connection open-connection-for-uri/cached + #:keep-alive? #t + #:buffered? #f + #:verify-certificate? #f))) + (values raw + (response-content-length response)))))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 15 16:36:11 2021 Received: (at 47174) by debbugs.gnu.org; 15 Mar 2021 20:36:11 +0000 Received: from localhost ([127.0.0.1]:37306 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLtwh-0000Kr-BG for submit@debbugs.gnu.org; Mon, 15 Mar 2021 16:36:11 -0400 Received: from eggs.gnu.org ([209.51.188.92]:35428) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLtwf-0000Ke-2s for 47174@debbugs.gnu.org; Mon, 15 Mar 2021 16:36:09 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51905) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLtwZ-0000FY-64; Mon, 15 Mar 2021 16:36:03 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=46674 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLtwY-0007JT-Bt; Mon, 15 Mar 2021 16:36:02 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#47174: [PATCH 0/2] substitute: Handle closing connections to substitute servers. References: <20210315192449.16248-1-mail@cbaines.net> <20210315192449.16248-2-mail@cbaines.net> Date: Mon, 15 Mar 2021 21:36:01 +0100 In-Reply-To: <20210315192449.16248-2-mail@cbaines.net> (Christopher Baines's message of "Mon, 15 Mar 2021 19:24:49 +0000") Message-ID: <874khccfym.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47174 Cc: 47174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Christopher Baines skribis: > When reusing a HTTP connection to fetch multiple nars, and the remote ser= ver > signals that the connection should be closed. > > * guix/scripts/substitute.scm (process-substitution): Close connections to > substitute servers when a Connection: close header is specified in the > response. In the context of , honoring =E2=80=9CConnection: close=E2=80=9D isn=E2=80=99t enough. We need to handl= e the case where the server didn=E2=80=99t express the intent to close the connection but eventu= ally closed it after some time. Does that make sense? Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 15 16:42:38 2021 Received: (at 47174) by debbugs.gnu.org; 15 Mar 2021 20:42:38 +0000 Received: from localhost ([127.0.0.1]:37322 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLu2v-0000V4-VT for submit@debbugs.gnu.org; Mon, 15 Mar 2021 16:42:38 -0400 Received: from mira.cbaines.net ([212.71.252.8]:36892) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLu2t-0000Uv-Nw for 47174@debbugs.gnu.org; Mon, 15 Mar 2021 16:42:36 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 2097327BC52; Mon, 15 Mar 2021 20:42:35 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 71a5997a; Mon, 15 Mar 2021 20:42:34 +0000 (UTC) References: <20210315192449.16248-1-mail@cbaines.net> <20210315192449.16248-2-mail@cbaines.net> <874khccfym.fsf_-_@gnu.org> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#47174: [PATCH 0/2] substitute: Handle closing connections to substitute servers. In-reply-to: <874khccfym.fsf_-_@gnu.org> Date: Mon, 15 Mar 2021 20:42:34 +0000 Message-ID: <87pn00du85.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 Cc: 47174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> When reusing a HTTP connection to fetch multiple nars, and the remote se= rver >> signals that the connection should be closed. >> >> * guix/scripts/substitute.scm (process-substitution): Close connections = to >> substitute servers when a Connection: close header is specified in the >> response. > > In the context of , honoring > =E2=80=9CConnection: close=E2=80=9D isn=E2=80=99t enough. We need to han= dle the case where the > server didn=E2=80=99t express the intent to close the connection but even= tually > closed it after some time. > > Does that make sense? Yeah, of course, this was something I was thinking about in addition to the changes in [1]. 1: https://issues.guix.gnu.org/47160 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBPxrpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xd0Jg//cyAQJF65Ezu0jNXzIoiUOwNZ5xc2mfsO Rnbn/G3E+LiZnjzEBXRLLXobWuVON8jIwKcrmroVB+BIJbbEzTLCSu4hahmT/0SE osbOr0hC9e0a9XcVTRxj0w9raRfyAhtvOl7IANH/Nl+SfM9ZXU1BOve5VRGPnBwZ 5QCLfh/m0naWoSErs8KTtz4zfFPJPS8K/3F7euiH6wzbgv27LK7e9z1Le4MnmbAJ p47/Qt9i/mBPMYXkCs1uN0ciylC59u12lUPPLAyUk/5J+VBRyGCdy1rxRTVfogRv k7PtVOPvkjSwwYWIRF1YAU/EGlM3IurHj/EnbBLAKM8Nuaa9V27puwzs44TCWCz5 weArBWRuusiYQuCXHoJSb2x14MXWADZacTslEnbVqCqmnsq8osqq8COvySJHg+HU RSKWCXU+7g/ePqEjIfr4Wt/slBrawvl4eP6IYf4Gv8xAV5XeIQNdGBe8ZeqcBK6O yB/R4rd9bnikZwHWTWiximsNdXoSoHCGthi5D+1jFrHOjM/u3FAD/7KOLUjaRcL8 12PVpMFfz2XY0T2obZa/DtZxLKadjAxJZ03L6uyLbC+diNMlyj/hiy3iJ4SRonRp 8A0siXnnPWcHb4yYbufKFs8Fhj9kA3oQsAPnwvkdZGq6bQkD1gZ80msDS7r7jRBn EOtl2pIqcQY= =PQKS -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun May 16 18:11:24 2021 Received: (at 47174) by debbugs.gnu.org; 16 May 2021 22:11:24 +0000 Received: from localhost ([127.0.0.1]:51628 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liOyq-0002Fs-2w for submit@debbugs.gnu.org; Sun, 16 May 2021 18:11:24 -0400 Received: from mira.cbaines.net ([212.71.252.8]:38896) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liOyo-0002Fe-Ih for 47174@debbugs.gnu.org; Sun, 16 May 2021 18:11:22 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id C995427BC81 for <47174@debbugs.gnu.org>; Sun, 16 May 2021 23:11:21 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 29660e40 for <47174@debbugs.gnu.org>; Sun, 16 May 2021 22:11:21 +0000 (UTC) From: Christopher Baines To: 47174@debbugs.gnu.org Subject: [PATCH v2 2/2] substitute: Handle closing connections to substitute servers. Date: Sun, 16 May 2021 23:11:21 +0100 Message-Id: <20210516221121.16705-2-mail@cbaines.net> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210516221121.16705-1-mail@cbaines.net> References: <20210516221121.16705-1-mail@cbaines.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) When reusing a HTTP connection to fetch multiple nars, and the remote server signals that the connection should be closed. * guix/scripts/substitute.scm (process-substitution): Close connections to substitute servers when a Connection: close header is specified in the response. --- guix/scripts/substitute.scm | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 96f425eaa0..208b8f1273 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -464,7 +464,9 @@ PORT." (case (uri-scheme uri) ((file) (let ((port (open-file (uri-path uri) "r0b"))) - (values port (stat:size (stat port))))) + (values port + (stat:size (stat port)) + (const #t)))) ; no cleanup to do ((http https) (guard (c ((http-get-error? c) (leave (G_ "download from '~a' failed: ~a, ~s~%") @@ -487,7 +489,12 @@ PORT." #:keep-alive? #t #:buffered? #f))) (values raw - (response-content-length response))))))) + (response-content-length response) + (match (assq 'connection (response-headers response)) + (('connection 'close) + (lambda () + (close-port port))) + (_ (const #t))))))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) @@ -504,7 +511,7 @@ PORT." (format (current-error-port) (G_ "Downloading ~a...~%") (uri->string uri))) - (let*-values (((raw download-size) + (let*-values (((raw download-size post-fetch-cleanup) ;; 'guix publish' without '--cache' doesn't specify a ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. (fetch uri)) @@ -565,6 +572,10 @@ PORT." ;; Wait for the reporter to finish. (every (compose zero? cdr waitpid) pids) + ;; Do post-fetch cleanup, maybe closing the HTTP connection if HTTP is + ;; being used, and the connection should be closed + (post-fetch-cleanup) + ;; Skip a line after what 'progress-reporter/file' printed, and another ;; one to visually separate substitutions. When PRINT-BUILD-TRACE? is ;; true, leave it up to (guix status) to prettify things. -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sun May 16 18:11:24 2021 Received: (at 47174) by debbugs.gnu.org; 16 May 2021 22:11:24 +0000 Received: from localhost ([127.0.0.1]:51630 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liOyq-0002Fu-BD for submit@debbugs.gnu.org; Sun, 16 May 2021 18:11:24 -0400 Received: from mira.cbaines.net ([212.71.252.8]:38894) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liOyo-0002Ff-Ih for 47174@debbugs.gnu.org; Sun, 16 May 2021 18:11:23 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id C0B0627BC78 for <47174@debbugs.gnu.org>; Sun, 16 May 2021 23:11:21 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 309302e4 for <47174@debbugs.gnu.org>; Sun, 16 May 2021 22:11:21 +0000 (UTC) From: Christopher Baines To: 47174@debbugs.gnu.org Subject: [PATCH v2 1/2] guix: Alter http-fetch to return the response. Date: Sun, 16 May 2021 23:11:20 +0100 Message-Id: <20210516221121.16705-1-mail@cbaines.net> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Rather than just the port and response-content-length. I'm looking at using the response headers within the substitute script to work out when to close the connection. * guix/http-client.scm (http-fetch): Return the response as the second value, rather than the response-content-length. * guix/build/download-nar.scm (download-nar): Adapt accordingly. * guix/build/download.scm (url-fetch): Adapt accordingly. * guix/scripts/substitute.scm (process-substitution): Adapt accordingly. --- guix/build/download-nar.scm | 5 +++-- guix/build/download.scm | 9 ++++++--- guix/http-client.scm | 12 ++++++------ guix/scripts/substitute.scm | 12 ++++++++---- 4 files changed, 23 insertions(+), 15 deletions(-) diff --git a/guix/build/download-nar.scm b/guix/build/download-nar.scm index 867f3c10bb..fbb5d37c0a 100644 --- a/guix/build/download-nar.scm +++ b/guix/build/download-nar.scm @@ -23,6 +23,7 @@ #:autoload (zlib) (call-with-gzip-input-port) #:use-module (guix progress) #:use-module (web uri) + #:use-module (web response) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (ice-9 format) @@ -101,7 +102,7 @@ success, #f otherwise." ((url rest ...) (format #t "Trying content-addressed mirror at ~a...~%" (uri-host (string->uri url))) - (let-values (((port size) + (let-values (((port resp) (catch #t (lambda () (http-fetch (string->uri url))) @@ -109,7 +110,7 @@ success, #f otherwise." (values #f #f))))) (if (not port) (loop rest) - (begin + (let ((size (response-content-length resp))) (if size (format #t "Downloading from ~a (~,2h MiB)...~%" url (/ size (expt 2 20.))) diff --git a/guix/build/download.scm b/guix/build/download.scm index b14db42352..d2006cc1fd 100644 --- a/guix/build/download.scm +++ b/guix/build/download.scm @@ -22,6 +22,7 @@ (define-module (guix build download) #:use-module (web uri) #:use-module (web http) + #:use-module (web response) #:use-module ((web client) #:hide (open-socket-for-uri)) #:use-module (web response) #:use-module (guix base64) @@ -706,7 +707,7 @@ otherwise simply ignore them." (case (uri-scheme uri) ((http https) (false-if-exception* - (let-values (((port size) + (let-values (((port resp) (http-fetch uri #:verify-certificate? verify-certificate? #:timeout timeout))) @@ -716,9 +717,11 @@ otherwise simply ignore them." #:buffer-size %http-receive-buffer-size #:reporter (if print-build-trace? (progress-reporter/trace - file (uri->string uri) size) + file (uri->string uri) + (response-content-length resp)) (progress-reporter/file - (uri-abbreviation uri) size))) + (uri-abbreviation uri) + (response-content-length resp)))) (newline))) file))) ((ftp) diff --git a/guix/http-client.scm b/guix/http-client.scm index 10bc278023..189535079b 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -81,11 +81,11 @@ (headers '((user-agent . "GNU Guile"))) (log-port (current-error-port)) timeout) - "Return an input port containing the data at URI, and the expected number of -bytes available or #f. If TEXT? is true, the data at URI is considered to be -textual. Follow any HTTP redirection. When BUFFERED? is #f, return an -unbuffered port, suitable for use in `filtered-port'. HEADERS is an alist of -extra HTTP headers. + "Return an input port containing the data at URI, and the HTTP response from +the server. If TEXT? is true, the data at URI is considered to be textual. +Follow any HTTP redirection. When BUFFERED? is #f, return an unbuffered port, +suitable for use in `filtered-port'. HEADERS is an alist of extra HTTP +headers. When KEEP-ALIVE? is true, the connection is marked as 'keep-alive' and PORT is not closed upon completion. @@ -123,7 +123,7 @@ Raise an '&http-get-error' condition if downloading fails." (response-code resp))) (case code ((200) - (values data (response-content-length resp))) + (values data resp)) ((301 ; moved permanently 302 ; found (redirection) 303 ; see other diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 8e4eae00b3..96f425eaa0 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -61,6 +61,7 @@ #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) #:use-module (web uri) + #:use-module (web response) #:use-module (guix http-client) #:export (%allow-unauthenticated-substitutes? %reply-file-descriptor @@ -480,10 +481,13 @@ PORT." (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) (with-cached-connection uri port - (http-fetch uri #:text? #f - #:port port - #:keep-alive? #t - #:buffered? #f))))) + (let-values (((raw response) + (http-fetch uri #:text? #f + #:port port + #:keep-alive? #t + #:buffered? #f))) + (values raw + (response-content-length response))))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Mon May 17 10:44:27 2021 Received: (at 47174) by debbugs.gnu.org; 17 May 2021 14:44:28 +0000 Received: from localhost ([127.0.0.1]:53382 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lieTr-0005kg-M4 for submit@debbugs.gnu.org; Mon, 17 May 2021 10:44:27 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38988) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lieTo-0005kO-UB for 47174@debbugs.gnu.org; Mon, 17 May 2021 10:44:26 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:44900) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lieTj-0000ET-7e; Mon, 17 May 2021 10:44:19 -0400 Received: from [109.190.253.11] (port=55366 helo=meije) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lieTi-0001Ww-Dn; Mon, 17 May 2021 10:44:19 -0400 From: Mathieu Othacehe To: Christopher Baines Subject: Re: [bug#47174] [PATCH v2 1/2] guix: Alter http-fetch to return the response. References: <87y2eodxyy.fsf@cbaines.net> <20210516221121.16705-1-mail@cbaines.net> Date: Mon, 17 May 2021 16:44:15 +0200 In-Reply-To: <20210516221121.16705-1-mail@cbaines.net> (Christopher Baines's message of "Sun, 16 May 2021 23:11:20 +0100") Message-ID: <87o8d9a08w.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 47174 Cc: 47174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello Chis, > * guix/http-client.scm (http-fetch): Return the response as the second value, > rather than the response-content-length. I think there is a missing adaptation in the call-with-nar procedure of the (guix scripts challenge) module. Otherwise, looks fine! Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Mon May 17 10:46:38 2021 Received: (at 47174) by debbugs.gnu.org; 17 May 2021 14:46:38 +0000 Received: from localhost ([127.0.0.1]:53386 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lieVy-00077G-3d for submit@debbugs.gnu.org; Mon, 17 May 2021 10:46:38 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39606) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lieVv-00072R-UB for 47174@debbugs.gnu.org; Mon, 17 May 2021 10:46:36 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:45020) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lieVq-0001kn-Jb; Mon, 17 May 2021 10:46:30 -0400 Received: from [109.190.253.11] (port=55380 helo=meije) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lieVp-0001sr-Qs; Mon, 17 May 2021 10:46:30 -0400 From: Mathieu Othacehe To: Christopher Baines Subject: Re: [bug#47174] [PATCH v2 2/2] substitute: Handle closing connections to substitute servers. References: <20210516221121.16705-1-mail@cbaines.net> <20210516221121.16705-2-mail@cbaines.net> Date: Mon, 17 May 2021 16:46:27 +0200 In-Reply-To: <20210516221121.16705-2-mail@cbaines.net> (Christopher Baines's message of "Sun, 16 May 2021 23:11:21 +0100") Message-ID: <87k0nxa058.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 47174 Cc: 47174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > + (match (assq 'connection (response-headers response)) > + (('connection 'close) > + (lambda () > + (close-port port))) You could maybe factorize it in a close-connection? procedure. Out of curiosity, when does the remote server asks for connection closing? Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Thu May 20 06:59:12 2021 Received: (at 47174) by debbugs.gnu.org; 20 May 2021 10:59:12 +0000 Received: from localhost ([127.0.0.1]:33196 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljgOW-0007yx-8j for submit@debbugs.gnu.org; Thu, 20 May 2021 06:59:12 -0400 Received: from mira.cbaines.net ([212.71.252.8]:42918) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljgOU-0007yp-FY for 47174@debbugs.gnu.org; Thu, 20 May 2021 06:59:11 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 5669927BC78; Thu, 20 May 2021 11:59:09 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 4388d8f3; Thu, 20 May 2021 10:59:08 +0000 (UTC) References: <20210516221121.16705-1-mail@cbaines.net> <20210516221121.16705-2-mail@cbaines.net> <87k0nxa058.fsf@gnu.org> User-agent: mu4e 1.4.15; emacs 27.2 From: Christopher Baines To: Mathieu Othacehe Subject: Re: [bug#47174] [PATCH v2 2/2] substitute: Handle closing connections to substitute servers. In-reply-to: <87k0nxa058.fsf@gnu.org> Date: Thu, 20 May 2021 11:59:06 +0100 Message-ID: <87r1i1hds5.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 Cc: 47174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Mathieu Othacehe writes: >> + (match (assq 'connection (response-headers response)) >> + (('connection 'close) >> + (lambda () >> + (close-port port))) > > You could maybe factorize it in a close-connection? procedure. Out of > curiosity, when does the remote server asks for connection closing? A server can at any time ask for the connection to be closed. With NGinx for example, by default, it'll close connections after 1000 requests: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCmQPpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfDnhAAtkDEx29yB3n4MYI0C9kqZ117qAKVpILo ySQS0RrnKKpIJ2wcVgwFIfTesJPYkXvOmBxX08ZvDdlz9Ts0GAlJMkab2NuLT84U STmzxvN/nGaIvnZx6ToZNcQQFFsSTliUdt/fH48UrPnDmi4wQCQfSfo+oW8yl/e2 d81YtWTulnsVu/suJ6yz3s9kU7y/usEJ1DPS1iEA/Eu1Vs+rBN/4yHDsJ9aDsXiC MC+si2YdLrTZ7/hmevLYzdTePyJV6b+z+nJ8LPWH3Huca0yEXI9eTqeVFy7fTbPs dxFkw15RUbb7IEK7k25rJNB8n4tg16+z8eMeCgK+oFt9bXjNgWKCSedO8Fv6t0Kl nT2bTBXzRnb8Rul7Yy5UglEaOq6MjQSQkljX7ze1WxFs+cPEXhjhqnjFbKphqiAD TTpIloTAvKUM3d36K9+/JTs0hDnmpluw3zNRGMpC2agaKZVcv8VtKsOncFXb+/rG 6seAWiLLYmCAu6DVmeKgSEiSlRP0xsnMfvxS0YjNghrX22rHunpdQXfov1MZK5Q8 qkSvjUvyQqA8jlXtrq8wuPJov7IMI2mvPW0O1pOjwf7Hxvmju1g062+moXPVVj9S Oiq+9KXDJbW+2AK7M1MEeMo7/N8CDptEjKpe9QaIy6j+0ywh4pHGQvEPAcxrRoDf bvQAobACPQs= =9qvw -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu May 20 07:12:23 2021 Received: (at 47174) by debbugs.gnu.org; 20 May 2021 11:12:24 +0000 Received: from localhost ([127.0.0.1]:33209 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljgbH-0008Lp-Lt for submit@debbugs.gnu.org; Thu, 20 May 2021 07:12:23 -0400 Received: from mira.cbaines.net ([212.71.252.8]:42934) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljgbF-0008Lh-TE for 47174@debbugs.gnu.org; Thu, 20 May 2021 07:12:22 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 113DC27BC78; Thu, 20 May 2021 12:12:21 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id af0b4aa4; Thu, 20 May 2021 11:12:21 +0000 (UTC) References: <87y2eodxyy.fsf@cbaines.net> <20210516221121.16705-1-mail@cbaines.net> <87o8d9a08w.fsf@gnu.org> User-agent: mu4e 1.4.15; emacs 27.2 From: Christopher Baines To: Mathieu Othacehe Subject: Re: [bug#47174] [PATCH v2 1/2] guix: Alter http-fetch to return the response. In-reply-to: <87o8d9a08w.fsf@gnu.org> Date: Thu, 20 May 2021 12:12:18 +0100 Message-ID: <87o8d5hd65.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 Cc: 47174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Mathieu Othacehe writes: > Hello Chis, > >> * guix/http-client.scm (http-fetch): Return the response as the second value, >> rather than the response-content-length. > > I think there is a missing adaptation in the call-with-nar procedure of > the (guix scripts challenge) module. Indeed, I've fixed that and I'll send a v3 series. > Otherwise, looks fine! Great, I'll try and do some testing of this at some point, as I haven't done any testing yet. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCmRBJfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xcg1w//UD6w0Y1z/KBTDVyFbEqhK7cmeVy204ij LMlY11N8vCrFJ94gQAt9n7xT58vt0o/03WOYrgNk8sMGjE+PFdMx0P43+z93lpxA XoSP5893Nunx4bLhYGKynWOUY/jSZlPIfQxU2lyWhrGOAHbBksagLtRYDfMq++3+ idqABYhM/rdZFA4ZzcH8COWmsLmQfXG76mMYE77x6IXvk5eC1OrxpXBoDJ+E1vya eUQQPOCc01aTUdiW+uq6QCiOjDN5DY1Owrs8C0jH1K033viff2/YcGdoJ3uLSjQQ NQdxXtS4QHW2Sll85MT9dRiiZv+7rRtJWuTdimE6EeWfxREYms4U10lz0CPhYBAw 8X/NJ+5zbBhPQsaOLUO7NXxVOo/iCuunWUIMPYrgUhS1caKFwWvzXc1zuK5Dhgr/ qLXX1q7lVfEqOpstndFw2/Z/V+z+3WLoJ0DMa39F+ozW+IeKDgr8Z0RJ68WW4uDe 1elLoFIn604lduUGhhMqi/ktMeDoEv/T3UaNKU364uj6t+CQvkPlbFQNz4l7dqpT /4I1fkFRmy48fE34EZsyCVj+NclptdsV1C+2dOLdcO18hkCSViNpDW5hEFwAjHzz vmsnorm19JQHU91k8hqEHyqCO9XHrCLsuqyea5niDhfuM5LADtJW/JRHC59MdLuX oN+GIb/Of40= =/BdU -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu May 20 08:04:17 2021 Received: (at 47174) by debbugs.gnu.org; 20 May 2021 12:04:18 +0000 Received: from localhost ([127.0.0.1]:33259 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljhPV-0003Rp-Hr for submit@debbugs.gnu.org; Thu, 20 May 2021 08:04:17 -0400 Received: from mira.cbaines.net ([212.71.252.8]:42970) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljhPT-0003RZ-AH for 47174@debbugs.gnu.org; Thu, 20 May 2021 08:04:16 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 44D0127BC78 for <47174@debbugs.gnu.org>; Thu, 20 May 2021 13:04:14 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id e84199e6 for <47174@debbugs.gnu.org>; Thu, 20 May 2021 12:04:13 +0000 (UTC) From: Christopher Baines To: 47174@debbugs.gnu.org Subject: [PATCH v3 1/2] guix: Alter http-fetch to return the response. Date: Thu, 20 May 2021 13:04:12 +0100 Message-Id: <20210520120413.21644-1-mail@cbaines.net> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Rather than just the port and response-content-length. I'm looking at using the response headers within the substitute script to work out when to close the connection. * guix/http-client.scm (http-fetch): Return the response as the second value, rather than the response-content-length. * guix/build/download-nar.scm (download-nar): Adapt accordingly. * guix/build/download.scm (url-fetch): Adapt accordingly. * guix/scripts/substitute.scm (process-substitution): Adapt accordingly. --- guix/build/download-nar.scm | 5 +++-- guix/build/download.scm | 9 ++++++--- guix/http-client.scm | 12 ++++++------ guix/scripts/challenge.scm | 6 ++++-- guix/scripts/substitute.scm | 12 ++++++++---- 5 files changed, 27 insertions(+), 17 deletions(-) diff --git a/guix/build/download-nar.scm b/guix/build/download-nar.scm index 867f3c10bb..fbb5d37c0a 100644 --- a/guix/build/download-nar.scm +++ b/guix/build/download-nar.scm @@ -23,6 +23,7 @@ #:autoload (zlib) (call-with-gzip-input-port) #:use-module (guix progress) #:use-module (web uri) + #:use-module (web response) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (ice-9 format) @@ -101,7 +102,7 @@ success, #f otherwise." ((url rest ...) (format #t "Trying content-addressed mirror at ~a...~%" (uri-host (string->uri url))) - (let-values (((port size) + (let-values (((port resp) (catch #t (lambda () (http-fetch (string->uri url))) @@ -109,7 +110,7 @@ success, #f otherwise." (values #f #f))))) (if (not port) (loop rest) - (begin + (let ((size (response-content-length resp))) (if size (format #t "Downloading from ~a (~,2h MiB)...~%" url (/ size (expt 2 20.))) diff --git a/guix/build/download.scm b/guix/build/download.scm index b14db42352..d2006cc1fd 100644 --- a/guix/build/download.scm +++ b/guix/build/download.scm @@ -22,6 +22,7 @@ (define-module (guix build download) #:use-module (web uri) #:use-module (web http) + #:use-module (web response) #:use-module ((web client) #:hide (open-socket-for-uri)) #:use-module (web response) #:use-module (guix base64) @@ -706,7 +707,7 @@ otherwise simply ignore them." (case (uri-scheme uri) ((http https) (false-if-exception* - (let-values (((port size) + (let-values (((port resp) (http-fetch uri #:verify-certificate? verify-certificate? #:timeout timeout))) @@ -716,9 +717,11 @@ otherwise simply ignore them." #:buffer-size %http-receive-buffer-size #:reporter (if print-build-trace? (progress-reporter/trace - file (uri->string uri) size) + file (uri->string uri) + (response-content-length resp)) (progress-reporter/file - (uri-abbreviation uri) size))) + (uri-abbreviation uri) + (response-content-length resp)))) (newline))) file))) ((ftp) diff --git a/guix/http-client.scm b/guix/http-client.scm index 10bc278023..189535079b 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -81,11 +81,11 @@ (headers '((user-agent . "GNU Guile"))) (log-port (current-error-port)) timeout) - "Return an input port containing the data at URI, and the expected number of -bytes available or #f. If TEXT? is true, the data at URI is considered to be -textual. Follow any HTTP redirection. When BUFFERED? is #f, return an -unbuffered port, suitable for use in `filtered-port'. HEADERS is an alist of -extra HTTP headers. + "Return an input port containing the data at URI, and the HTTP response from +the server. If TEXT? is true, the data at URI is considered to be textual. +Follow any HTTP redirection. When BUFFERED? is #f, return an unbuffered port, +suitable for use in `filtered-port'. HEADERS is an alist of extra HTTP +headers. When KEEP-ALIVE? is true, the connection is marked as 'keep-alive' and PORT is not closed upon completion. @@ -123,7 +123,7 @@ Raise an '&http-get-error' condition if downloading fails." (response-code resp))) (case code ((200) - (values data (response-content-length resp))) + (values data resp)) ((301 ; moved permanently 302 ; found (redirection) 303 ; see other diff --git a/guix/scripts/challenge.scm b/guix/scripts/challenge.scm index 69c2781abb..73103a061b 100644 --- a/guix/scripts/challenge.scm +++ b/guix/scripts/challenge.scm @@ -253,12 +253,14 @@ taken since we do not import the archives." NARINFO." (let*-values (((uri compression size) (narinfo-best-uri narinfo)) - ((port actual-size) + ((port response) (http-fetch uri))) (define reporter (progress-reporter/file (narinfo-path narinfo) (and size - (max size (or actual-size 0))) ;defensive + (max size (or + (response-content-length response) + 0))) ;defensive #:abbreviation (const (uri-host uri)))) (define result diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 8e4eae00b3..96f425eaa0 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -61,6 +61,7 @@ #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) #:use-module (web uri) + #:use-module (web response) #:use-module (guix http-client) #:export (%allow-unauthenticated-substitutes? %reply-file-descriptor @@ -480,10 +481,13 @@ PORT." (uri->string uri)) (warning (G_ "try `--no-substitutes' if the problem persists~%"))) (with-cached-connection uri port - (http-fetch uri #:text? #f - #:port port - #:keep-alive? #t - #:buffered? #f))))) + (let-values (((raw response) + (http-fetch uri #:text? #f + #:port port + #:keep-alive? #t + #:buffered? #f))) + (values raw + (response-content-length response))))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) -- 2.31.1 From debbugs-submit-bounces@debbugs.gnu.org Thu May 20 08:04:18 2021 Received: (at 47174) by debbugs.gnu.org; 20 May 2021 12:04:18 +0000 Received: from localhost ([127.0.0.1]:33261 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljhPV-0003Rr-V9 for submit@debbugs.gnu.org; Thu, 20 May 2021 08:04:18 -0400 Received: from mira.cbaines.net ([212.71.252.8]:42972) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ljhPT-0003Ra-AJ for 47174@debbugs.gnu.org; Thu, 20 May 2021 08:04:16 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 50D4F27BC81 for <47174@debbugs.gnu.org>; Thu, 20 May 2021 13:04:14 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id f0eb79bb for <47174@debbugs.gnu.org>; Thu, 20 May 2021 12:04:13 +0000 (UTC) From: Christopher Baines To: 47174@debbugs.gnu.org Subject: [PATCH v3 2/2] substitute: Handle closing connections to substitute servers. Date: Thu, 20 May 2021 13:04:13 +0100 Message-Id: <20210520120413.21644-2-mail@cbaines.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210520120413.21644-1-mail@cbaines.net> References: <20210520120413.21644-1-mail@cbaines.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 47174 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) When reusing a HTTP connection to fetch multiple nars, and the remote server signals that the connection should be closed. * guix/scripts/substitute.scm (process-substitution): Close connections to substitute servers when a Connection: close header is specified in the response. --- guix/scripts/substitute.scm | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 96f425eaa0..208b8f1273 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -464,7 +464,9 @@ PORT." (case (uri-scheme uri) ((file) (let ((port (open-file (uri-path uri) "r0b"))) - (values port (stat:size (stat port))))) + (values port + (stat:size (stat port)) + (const #t)))) ; no cleanup to do ((http https) (guard (c ((http-get-error? c) (leave (G_ "download from '~a' failed: ~a, ~s~%") @@ -487,7 +489,12 @@ PORT." #:keep-alive? #t #:buffered? #f))) (values raw - (response-content-length response))))))) + (response-content-length response) + (match (assq 'connection (response-headers response)) + (('connection 'close) + (lambda () + (close-port port))) + (_ (const #t))))))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) @@ -504,7 +511,7 @@ PORT." (format (current-error-port) (G_ "Downloading ~a...~%") (uri->string uri))) - (let*-values (((raw download-size) + (let*-values (((raw download-size post-fetch-cleanup) ;; 'guix publish' without '--cache' doesn't specify a ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. (fetch uri)) @@ -565,6 +572,10 @@ PORT." ;; Wait for the reporter to finish. (every (compose zero? cdr waitpid) pids) + ;; Do post-fetch cleanup, maybe closing the HTTP connection if HTTP is + ;; being used, and the connection should be closed + (post-fetch-cleanup) + ;; Skip a line after what 'progress-reporter/file' printed, and another ;; one to visually separate substitutions. When PRINT-BUILD-TRACE? is ;; true, leave it up to (guix status) to prettify things. -- 2.31.1 From debbugs-submit-bounces@debbugs.gnu.org Sat May 29 17:41:59 2021 Received: (at 47174) by debbugs.gnu.org; 29 May 2021 21:41:59 +0000 Received: from localhost ([127.0.0.1]:57843 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ln6iV-0004G1-6n for submit@debbugs.gnu.org; Sat, 29 May 2021 17:41:59 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40002) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ln6iT-0004Fo-70 for 47174@debbugs.gnu.org; Sat, 29 May 2021 17:41:58 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:55828) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ln6iN-0007vS-Oj; Sat, 29 May 2021 17:41:51 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41664 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ln6iN-0005Qq-Gj; Sat, 29 May 2021 17:41:51 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#47174: [PATCH 0/2] substitute: Handle closing connections to substitute servers. References: <87y2eodxyy.fsf@cbaines.net> <20210520120413.21644-1-mail@cbaines.net> Date: Sat, 29 May 2021 23:41:49 +0200 In-Reply-To: <20210520120413.21644-1-mail@cbaines.net> (Christopher Baines's message of "Thu, 20 May 2021 13:04:12 +0100") Message-ID: <87zgwd43qa.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 47174 Cc: 47174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Chris, Christopher Baines skribis: > Rather than just the port and response-content-length. I'm looking at us= ing > the response headers within the substitute script to work out when to clo= se > the connection. > > * guix/http-client.scm (http-fetch): Return the response as the second va= lue, > rather than the response-content-length. > * guix/build/download-nar.scm (download-nar): Adapt accordingly. > * guix/build/download.scm (url-fetch): Adapt accordingly. > * guix/scripts/substitute.scm (process-substitution): Adapt accordingly. Nitpick: use =E2=80=9Chttp-client:=E2=80=9D rather than =E2=80=9Cguix:=E2= =80=9D as the subject line. > + (let-values (((port resp) Conventionally we=E2=80=99d spell it out: =E2=80=98response=E2=80=99. Otherwise LGTM. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat May 29 17:46:35 2021 Received: (at 47174) by debbugs.gnu.org; 29 May 2021 21:46:36 +0000 Received: from localhost ([127.0.0.1]:57860 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ln6mx-0004OE-LF for submit@debbugs.gnu.org; Sat, 29 May 2021 17:46:35 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40692) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ln6mw-0004O1-1l for 47174@debbugs.gnu.org; Sat, 29 May 2021 17:46:34 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:55930) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ln6mq-0002ca-Bi; Sat, 29 May 2021 17:46:28 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41668 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ln6mq-0003UE-1U; Sat, 29 May 2021 17:46:28 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#47174: [PATCH 0/2] substitute: Handle closing connections to substitute servers. References: <20210520120413.21644-1-mail@cbaines.net> <20210520120413.21644-2-mail@cbaines.net> Date: Sat, 29 May 2021 23:46:26 +0200 In-Reply-To: <20210520120413.21644-2-mail@cbaines.net> (Christopher Baines's message of "Thu, 20 May 2021 13:04:13 +0100") Message-ID: <87tuml43il.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 47174 Cc: 47174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Christopher Baines skribis: > When reusing a HTTP connection to fetch multiple nars, and the remote ser= ver > signals that the connection should be closed. Incomplete sentence? > * guix/scripts/substitute.scm (process-substitution): Close connections to > substitute servers when a Connection: close header is specified in the > response. > --- > guix/scripts/substitute.scm | 17 ++++++++++++++--- > 1 file changed, 14 insertions(+), 3 deletions(-) > > diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm > index 96f425eaa0..208b8f1273 100755 > --- a/guix/scripts/substitute.scm > +++ b/guix/scripts/substitute.scm > @@ -464,7 +464,9 @@ PORT." > (case (uri-scheme uri) > ((file) > (let ((port (open-file (uri-path uri) "r0b"))) > - (values port (stat:size (stat port))))) > + (values port > + (stat:size (stat port)) > + (const #t)))) ; no cleanup to do > ((http https) > (guard (c ((http-get-error? c) > (leave (G_ "download from '~a' failed: ~a, ~s~%") > @@ -487,7 +489,12 @@ PORT." > #:keep-alive? #t > #:buffered? #f))) > (values raw > - (response-content-length response))))))) > + (response-content-length response) > + (match (assq 'connection (response-headers respon= se)) > + (('connection 'close) > + (lambda () > + (close-port port))) > + (_ (const #t))))))))) > (else > (leave (G_ "unsupported substitute URI scheme: ~a~%") > (uri->string uri))))) > @@ -504,7 +511,7 @@ PORT." > (format (current-error-port) > (G_ "Downloading ~a...~%") (uri->string uri))) >=20=20 > - (let*-values (((raw download-size) > + (let*-values (((raw download-size post-fetch-cleanup) > ;; 'guix publish' without '--cache' doesn't specify a > ;; Content-Length, so DOWNLOAD-SIZE is #f in this cas= e. > (fetch uri)) > @@ -565,6 +572,10 @@ PORT." > ;; Wait for the reporter to finish. > (every (compose zero? cdr waitpid) pids) >=20=20 > + ;; Do post-fetch cleanup, maybe closing the HTTP connection if HTT= P is > + ;; being used, and the connection should be closed > + (post-fetch-cleanup) How about returning a Boolean as the third value, =E2=80=98close?=E2=80=99,= indicating whether the port should be closed upon completion? That seems marginally clearer to me that the post-cleanup thunk. Otherwise LGTM, thanks! Ludo=E2=80=99.