From unknown Tue Jun 24 01:38:35 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47897] [PATCH] substitutes: Don't cache negative lookups or transient errors. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 19 Apr 2021 18:41:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 47897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47897@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.161885765632392 (code B ref -1); Mon, 19 Apr 2021 18:41:02 +0000 Received: (at submit) by debbugs.gnu.org; 19 Apr 2021 18:40:56 +0000 Received: from localhost ([127.0.0.1]:52064 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lYYpM-0008QO-K3 for submit@debbugs.gnu.org; Mon, 19 Apr 2021 14:40:56 -0400 Received: from lists.gnu.org ([209.51.188.17]:40328) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lYYpH-0008QB-48 for submit@debbugs.gnu.org; Mon, 19 Apr 2021 14:40:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40064) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYYpD-0003xE-E8 for guix-patches@gnu.org; Mon, 19 Apr 2021 14:40:50 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:59795) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lYYp9-00026t-2y for guix-patches@gnu.org; Mon, 19 Apr 2021 14:40:47 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 92E0A27BC6C for ; Mon, 19 Apr 2021 19:40:37 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id b370b44f for ; Mon, 19 Apr 2021 18:40:36 +0000 (UTC) From: Christopher Baines Date: Mon, 19 Apr 2021 19:40:36 +0100 Message-Id: <20210419184036.18632-1-mail@cbaines.net> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) Caching these responses can delay users getting substitutes, since they'll have to wait at least as long as the cache TTL to fetch the substitute, even if it's available sooner. * guix/substitutes.scm (%narinfo-negative-ttl, %narinfo-transient-error-ttl): Remove variables. (cache-narinfo!): Adjust to not use the negative TTL. (fetch-narinfos): Don't cache non 200 status code responses. --- guix/substitutes.scm | 22 +++------------------- 1 file changed, 3 insertions(+), 19 deletions(-) diff --git a/guix/substitutes.scm b/guix/substitutes.scm index 08f8c24efd..55daa6c964 100644 --- a/guix/substitutes.scm +++ b/guix/substitutes.scm @@ -70,14 +70,6 @@ ;; 'Cache-Control' response header. (* 36 3600)) -(define %narinfo-negative-ttl - ;; Likewise, but for negative lookups---i.e., cached lookup failures (404). - (* 1 3600)) - -(define %narinfo-transient-error-ttl - ;; Likewise, but for transient errors such as 504 ("Gateway timeout"). - (* 10 60)) - (define %narinfo-cache-directory ;; A local cache of narinfos, to avoid going to the network. Most of the ;; time, 'guix substitute' is called by guix-daemon as root and stores its @@ -104,8 +96,7 @@ entry is stored in a sub-directory specific to CACHE-URL." (define (cache-narinfo! cache-url path narinfo ttl) "Cache locally NARNIFO for PATH, which originates from CACHE-URL, with the -given TTL (a number of seconds or #f). NARINFO may be #f, in which case it -indicates that PATH is unavailable at CACHE-URL." +given TTL (a number of seconds or #f)." (define now (current-time time-monotonic)) @@ -113,8 +104,7 @@ indicates that PATH is unavailable at CACHE-URL." `(narinfo (version 2) (cache-uri ,cache-uri) (date ,(time-second now)) - (ttl ,(or ttl - (if narinfo %narinfo-ttl %narinfo-negative-ttl))) + (ttl ,(or ttl %narinfo-ttl)) (value ,(and=> narinfo narinfo->string)))) (let ((file (narinfo-cache-file cache-url path))) @@ -214,16 +204,10 @@ if file doesn't exist, and the narinfo otherwise." (cache-narinfo! url (narinfo-path narinfo) narinfo ttl) (cons narinfo result)) result)) - (let* ((path (uri-path (request-uri request))) - (hash-part (basename - (string-drop-right path 8)))) ;drop ".narinfo" + (begin (if len (get-bytevector-n port len) (read-to-eof port)) - (cache-narinfo! url (hash-part->path hash-part) #f - (if (or (= 404 code) (= 202 code)) - ttl - %narinfo-transient-error-ttl)) result)))) (define (do-fetch uri) -- 2.30.1 From unknown Tue Jun 24 01:38:35 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47897] Narinfo negative and transient error caching Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 22 Apr 2021 22:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: guix-devel@gnu.org, 47897@debbugs.gnu.org Received: via spool by 47897-submit@debbugs.gnu.org id=B47897.16191295056058 (code B ref 47897); Thu, 22 Apr 2021 22:12:01 +0000 Received: (at 47897) by debbugs.gnu.org; 22 Apr 2021 22:11:45 +0000 Received: from localhost ([127.0.0.1]:35754 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZhY0-0001Ze-O1 for submit@debbugs.gnu.org; Thu, 22 Apr 2021 18:11:45 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47432) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZhXz-0001ZS-8e for 47897@debbugs.gnu.org; Thu, 22 Apr 2021 18:11:43 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:57992) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZhXu-0003Rr-1h; Thu, 22 Apr 2021 18:11:38 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=38408 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lZhXp-0004Jg-Rp; Thu, 22 Apr 2021 18:11:37 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87mtvhnsn6.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 3 =?UTF-8?Q?Flor=C3=A9al?= an 229 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 23 Apr 2021 00:11:31 +0200 In-Reply-To: <87mtvhnsn6.fsf@cbaines.net> (Christopher Baines's message of "Fri, 05 Mar 2021 22:27:09 +0000") Message-ID: <874kfyufzg.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! (=E2=80=9CSorry for the long delay=E2=80=9D is officially my motto at this = point.) Christopher Baines skribis: > This has been on my mind for a while, as I wonder what effect it has on > users fetching substitues. > > The narinfo caching as I understand it works as follows: > > Default success TTL =3D> 36 hours > Negative TTL =3D> 1 hour > Transient error TTL =3D> 10 minutes > > I'm ignoring the success TTL, I'm just interested in the negative and > transient error values. Negative means that when a server says it > doesn't have an output, that response will be cached for an > hour. Transient errors are for other HTTP response codes, like 504. You=E2=80=99re looking at the default TTLs, which are not the actual TTLs. Specifically, servers can include a =E2=80=98Cache-Control=E2=80=99 header = in their reply specifying the TTL of their choice, and =E2=80=98guix substitute=E2= =80=99 honors that: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/substitutes.scm#n200 https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/publish.scm#= n371 =E2=80=98guix publish=E2=80=99 returns 404 with a TTL of 5mn when the reque= sted item is in store but needs to be =E2=80=9Cbaked=E2=80=9D. However, =E2=80=98guix publish=E2=80=99 does not set =E2=80=98Cache-Control= =E2=80=99 when the request item is not in store. In that case, clients use =E2=80=98%narinfo-negative= -ttl=E2=80=99 (1h). > I had a look through the Git history, caching negative lookups has been > a thing for a while. Caching transient errors was added, but I couldn't > see why. Transient error caching was most likely added in the days of hydra.gnu.org, that VM that was extremely slow. When overloaded, you=E2=80= =99d get 500 or similar, and at that point it was safer for clients to wait and come back later, possibly much later. :-) > Personally I don't see a reason to keep either behaviours? The main arguments for these negative TTLs are: 1. Reducing server load: if the server doesn=E2=80=99t have libreoffice, = don=E2=80=99t come back asking every 10s, it=E2=80=99s prolly useless. You could ea= sily have =E2=80=9CGET storms=E2=80=9D for libreoffice if clients don=E2=80= =99t restrain themselves. 2. Improving client performance: don=E2=80=99t GET things that are likely= to fail. Now, the penalty it imposes is annoying. I=E2=80=99ve sometimes found myse= lf working around it, too (because I knew the server was going to have the store item sooner than 1h). Rather than removing it entirely, I can think of these options: 1. Reduce the default negative timeouts. 2. Add an option to =E2=80=98guix publish=E2=80=99 (and to the Coordinato= r?) so they send a =E2=80=98Cache-Control=E2=80=99 header with the chosen TTL on 4= 04. That way, if the server operator doesn=E2=80=99t mind extra load, they can = run =E2=80=9Cguix publish --negative-ttl=3D0=E2=80=9D. WDYT? Does that make any sense? Ludo=E2=80=99. From unknown Tue Jun 24 01:38:35 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47897] Narinfo negative and transient error caching Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 22 Apr 2021 22:15:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: guix-devel@gnu.org, 47897@debbugs.gnu.org Received: via spool by 47897-submit@debbugs.gnu.org id=B47897.16191296586297 (code B ref 47897); Thu, 22 Apr 2021 22:15:02 +0000 Received: (at 47897) by debbugs.gnu.org; 22 Apr 2021 22:14:18 +0000 Received: from localhost ([127.0.0.1]:35758 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZhaU-0001dU-8k for submit@debbugs.gnu.org; Thu, 22 Apr 2021 18:14:18 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47860) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZhaS-0001dG-Ve for 47897@debbugs.gnu.org; Thu, 22 Apr 2021 18:14:17 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58041) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZhaN-0004wf-Ny; Thu, 22 Apr 2021 18:14:11 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=38410 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lZhaN-0004Yw-7q; Thu, 22 Apr 2021 18:14:11 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87mtvhnsn6.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 4 =?UTF-8?Q?Flor=C3=A9al?= an 229 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 23 Apr 2021 00:14:09 +0200 In-Reply-To: <87mtvhnsn6.fsf@cbaines.net> (Christopher Baines's message of "Fri, 05 Mar 2021 22:27:09 +0000") Message-ID: <87tunyt1am.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) BTW, one thing that would be interesting too is to return 404 with a long =E2=80=98Cache-Control=E2=80=99 validity when the requested store item= is among the cached failures. We could also add an extra response header to explicitly communicate that the store item is known to fail to build. Ludo=E2=80=99. From unknown Tue Jun 24 01:38:35 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47897] Narinfo negative and transient error caching Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 22 Apr 2021 23:15:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org, 47897@debbugs.gnu.org Received: via spool by 47897-submit@debbugs.gnu.org id=B47897.161913327111612 (code B ref 47897); Thu, 22 Apr 2021 23:15:01 +0000 Received: (at 47897) by debbugs.gnu.org; 22 Apr 2021 23:14:31 +0000 Received: from localhost ([127.0.0.1]:35791 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZiWl-00031D-32 for submit@debbugs.gnu.org; Thu, 22 Apr 2021 19:14:31 -0400 Received: from mira.cbaines.net ([212.71.252.8]:37462) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZiWi-000315-Sw for 47897@debbugs.gnu.org; Thu, 22 Apr 2021 19:14:30 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 0603627BC7C; Fri, 23 Apr 2021 00:14:28 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id c7eb486c; Thu, 22 Apr 2021 23:14:27 +0000 (UTC) References: <87mtvhnsn6.fsf@cbaines.net> <874kfyufzg.fsf@gnu.org> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines In-reply-to: <874kfyufzg.fsf@gnu.org> Date: Fri, 23 Apr 2021 00:14:24 +0100 Message-ID: <87wnsthpyn.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi! > > (=E2=80=9CSorry for the long delay=E2=80=9D is officially my motto at thi= s point.) > > Christopher Baines skribis: > >> This has been on my mind for a while, as I wonder what effect it has on >> users fetching substitues. >> >> The narinfo caching as I understand it works as follows: >> >> Default success TTL =3D> 36 hours >> Negative TTL =3D> 1 hour >> Transient error TTL =3D> 10 minutes >> >> I'm ignoring the success TTL, I'm just interested in the negative and >> transient error values. Negative means that when a server says it >> doesn't have an output, that response will be cached for an >> hour. Transient errors are for other HTTP response codes, like 504. > > You=E2=80=99re looking at the default TTLs, which are not the actual TTLs. > Specifically, servers can include a =E2=80=98Cache-Control=E2=80=99 heade= r in their > reply specifying the TTL of their choice, and =E2=80=98guix substitute=E2= =80=99 honors > that: > > https://git.savannah.gnu.org/cgit/guix.git/tree/guix/substitutes.scm#n2= 00 > https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/publish.sc= m#n371 > > =E2=80=98guix publish=E2=80=99 returns 404 with a TTL of 5mn when the req= uested item is > in store but needs to be =E2=80=9Cbaked=E2=80=9D. > > However, =E2=80=98guix publish=E2=80=99 does not set =E2=80=98Cache-Contr= ol=E2=80=99 when the request > item is not in store. In that case, clients use =E2=80=98%narinfo-negati= ve-ttl=E2=80=99 > (1h). You're right that the negative ttl is just a default, so it's possible to override the default behaviour in the success and negative lookup cases, but I don't believe the Cache-Control header is used for transient errors. >> I had a look through the Git history, caching negative lookups has been >> a thing for a while. Caching transient errors was added, but I couldn't >> see why. > > Transient error caching was most likely added in the days of > hydra.gnu.org, that VM that was extremely slow. When overloaded, you=E2= =80=99d > get 500 or similar, and at that point it was safer for clients to wait > and come back later, possibly much later. :-) > >> Personally I don't see a reason to keep either behaviours? > > The main arguments for these negative TTLs are: > > 1. Reducing server load: if the server doesn=E2=80=99t have libreoffice= , don=E2=80=99t > come back asking every 10s, it=E2=80=99s prolly useless. You could = easily > have =E2=80=9CGET storms=E2=80=9D for libreoffice if clients don=E2= =80=99t restrain > themselves. > > 2. Improving client performance: don=E2=80=99t GET things that are like= ly to > fail. As you say, for the negative TTL, the question here is really what's the best default value, if a server isn't specifying one. Given that most narinfo requests precede a build for that thing if the response is negative, I have my doubts about those two arguments above. This is assuming the most common case is users asking guix to install and upgrade things. If a user gets a negative response, they'll just build it instead and not check for that narinfo again. Even if they cancel that build when they realise they don't want to build libreoffice, they'll wait a bit anyway before retrying. > Now, the penalty it imposes is annoying. I=E2=80=99ve sometimes found my= self > working around it, too (because I knew the server was going to have the > store item sooner than 1h). > > Rather than removing it entirely, I can think of these options: > > 1. Reduce the default negative timeouts. I think reducing it is good, as you say, it's possible to override the default from the server side. Just in case someone wants caching behaviour, it might be worth keeping that functionality at least. > 2. Add an option to =E2=80=98guix publish=E2=80=99 (and to the Coordina= tor?) so they > send a =E2=80=98Cache-Control=E2=80=99 header with the chosen TTL on= 404. That > way, if the server operator doesn=E2=80=99t mind extra load, they ca= n run > =E2=80=9Cguix publish --negative-ttl=3D0=E2=80=9D. That sounds sensible. The Guix Build Coordinator doesn't do any serving, that's left to something else like nginx. For the deployments I maintain though, I don't think I'm setting the relevant headers, but I'll look at changing that. Going back to the %narinfo-transient-error-ttl, if I'm correct in saying that it's not possible to override that, maybe that should also use the relevant header value if set? --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCCA1BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XettQ//Ytdi5P4uajikIBpDpdEehCyYayl3awpY /KimsScWS6QbULGBPCHFZx92glZYjM3osEZWoBrUApVa5cODKSZ61/R1CzA+rGUO 2ed2Drqvr5OJPzZnf1chojFHqnM3AHpbffJpcqyF3mTZqAhQPD6nW1l6MITpOpLI BG0Af/qZYm4/dKgsJNeTGLqcGiGkew/zne9yzMNwMfirOoaDKgoycJTrJ3PdCCfT MBpkE7UgFMuYagIVqpNnHk+RDnZFf3Ayb23b/Kcrbeo3xcGyDwy4H4raf1uw67G9 xXzTM5WnhlpRsZqM3MKO9aNf6s7PN3x6KiyhmuulwArCzVyI6bF8jTlzuPEE/PSJ 5RrlGQRzxCUQ/mjfL95Xi2/nGWpF8X0I3H3IPdntBjj2lqfyQMww9mna3Byqldwj CEuDwvBFeiwNuNRJaXhQf6jZXPBsZ6MjWdkZNGtyYi+3Zy+43ww8Xvl8RHvKHcuI qSxC4TWAIfXiB8ZmWWPGxVnGVNIGgVl+0N/UB4+AXDuJuXtKGhkY0k2ZZdq3LQ37 /X14mJTYDpLnkvZEvOJzfTpJnL61Q5YBNdRmz0iTlpBMPymZQtHYKbdOtDFoXg5X AmMa+39Mv2+Ep6rMH0c9CPlnmNz97qELsivQ4VmUJohIl5wYGnsWwt1kKPHrA1bq vvGRj5HahDE= =s2Q0 -----END PGP SIGNATURE----- --=-=-=-- From unknown Tue Jun 24 01:38:35 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47897] [PATCH 2/2] substitutes: Reduce negative TTLs. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 11 May 2021 13:09:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47897@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 47897-submit@debbugs.gnu.org id=B47897.162073853915090 (code B ref 47897); Tue, 11 May 2021 13:09:02 +0000 Received: (at 47897) by debbugs.gnu.org; 11 May 2021 13:08:59 +0000 Received: from localhost ([127.0.0.1]:35350 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lgS8B-0003vI-AY for submit@debbugs.gnu.org; Tue, 11 May 2021 09:08:59 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42912) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lgS89-0003uz-Fr for 47897@debbugs.gnu.org; Tue, 11 May 2021 09:08:58 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48408) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lgS84-00062z-Bl; Tue, 11 May 2021 09:08:52 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=57486 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lgS83-0004e7-Ra; Tue, 11 May 2021 09:08:52 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Tue, 11 May 2021 15:08:42 +0200 Message-Id: <20210511130842.32381-2-ludo@gnu.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210511130842.32381-1-ludo@gnu.org> References: <87wnsthpyn.fsf@cbaines.net> <20210511130842.32381-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/substitutes.scm (%narinfo-negative-ttl): Change to 15mn. (%narinfo-transient-error-ttl): Halve. --- guix/substitutes.scm | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/guix/substitutes.scm b/guix/substitutes.scm index 08f8c24efd..c1b036e42c 100644 --- a/guix/substitutes.scm +++ b/guix/substitutes.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2014 Nikita Karetnikov ;;; Copyright © 2018 Kyle Meyer ;;; Copyright © 2020 Christopher Baines @@ -72,11 +72,11 @@ (define %narinfo-negative-ttl ;; Likewise, but for negative lookups---i.e., cached lookup failures (404). - (* 1 3600)) + (* 15 60)) (define %narinfo-transient-error-ttl ;; Likewise, but for transient errors such as 504 ("Gateway timeout"). - (* 10 60)) + (* 5 60)) (define %narinfo-cache-directory ;; A local cache of narinfos, to avoid going to the network. Most of the -- 2.31.1 From unknown Tue Jun 24 01:38:35 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47897] [PATCH 1/2] publish: Add '--negative-ttl'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 11 May 2021 13:09:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47897@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 47897-submit@debbugs.gnu.org id=B47897.162073854015098 (code B ref 47897); Tue, 11 May 2021 13:09:02 +0000 Received: (at 47897) by debbugs.gnu.org; 11 May 2021 13:09:00 +0000 Received: from localhost ([127.0.0.1]:35352 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lgS8B-0003vK-Kz for submit@debbugs.gnu.org; Tue, 11 May 2021 09:09:00 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42908) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lgS89-0003uy-FN for 47897@debbugs.gnu.org; Tue, 11 May 2021 09:08:58 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48406) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lgS83-000625-Ic; Tue, 11 May 2021 09:08:51 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=57486 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lgS83-0004e7-5o; Tue, 11 May 2021 09:08:51 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Tue, 11 May 2021 15:08:41 +0200 Message-Id: <20210511130842.32381-1-ludo@gnu.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <87wnsthpyn.fsf@cbaines.net> References: <87wnsthpyn.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/scripts/publish.scm (show-help, %options): Add '--negative-ttl'. (render-narinfo, render-narinfo/cached, make-request-handler): Add #:negative-ttl and honor it. (run-publish-server): Add #:narinfo-negative-ttl and honor it. (guix-publish): Honor '--negative-ttl'. * tests/publish.scm ("negative TTL", "no negative TTL"): New tests. --- doc/guix.texi | 10 ++++++++++ guix/scripts/publish.scm | 30 ++++++++++++++++++++++-------- tests/publish.scm | 32 +++++++++++++++++++++++++++++++- 3 files changed, 63 insertions(+), 9 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 0947b9f028..a34b2fca1e 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -12727,6 +12727,16 @@ Additionally, when @option{--cache} is used, cached entries that have not been accessed for @var{ttl} and that no longer have a corresponding item in the store, may be deleted. +@item --negative-ttl=@var{ttl} +Similarly produce @code{Cache-Control} HTTP headers to advertise the +time-to-live (TTL) of @emph{negative} lookups---missing store items, for +which the HTTP 404 code is returned. By default, no negative TTL is +advertised. + +This parameter can help adjust server load and substitute latency by +instructing cooperating clients to be more or less patient when a store +item is missing. + @item --cache-bypass-threshold=@var{size} When used in conjunction with @option{--cache}, store items smaller than @var{size} are immediately available, even when they are not yet in diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 39bb224cad..ef6fa5f074 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 David Thompson ;;; Copyright © 2020 by Amar M. Singh -;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2020 Maxim Cournoyer ;;; Copyright © 2021 Simon Tournier ;;; @@ -101,6 +101,8 @@ Publish ~a over HTTP.\n") %store-directory) --workers=N use N workers to bake items")) (display (G_ " --ttl=TTL announce narinfos can be cached for TTL seconds")) + (display (G_ " + --negative-ttl=TTL announce missing narinfos can be cached for TTL seconds")) (display (G_ " --nar-path=PATH use PATH as the prefix for nar URLs")) (display (G_ " @@ -224,6 +226,13 @@ usage." (leave (G_ "~a: invalid duration~%") arg)) (alist-cons 'narinfo-ttl (time-second duration) result)))) + (option '("negative-ttl") #t #f + (lambda (opt name arg result) + (let ((duration (string->duration arg))) + (unless duration + (leave (G_ "~a: invalid duration~%") arg)) + (alist-cons 'narinfo-negative-ttl (time-second duration) + result)))) (option '("nar-path") #t #f (lambda (opt name arg result) (alist-cons 'nar-path arg result))) @@ -390,14 +399,14 @@ References: ~a~%" (define* (render-narinfo store request hash #:key ttl (compressions (list %no-compression)) - (nar-path "nar")) + (nar-path "nar") negative-ttl) "Render metadata for the store path corresponding to HASH. If TTL is true, advertise it as the maximum validity period (in seconds) via the 'Cache-Control' header. This allows 'guix substitute' to cache it for an appropriate duration. NAR-PATH specifies the prefix for nar URLs." (let ((store-path (hash-part->path store hash))) (if (string-null? store-path) - (not-found request #:phrase "") + (not-found request #:phrase "" #:ttl negative-ttl) (values `((content-type . (application/x-nix-narinfo)) ,@(if ttl `((cache-control (max-age . ,ttl))) @@ -512,7 +521,7 @@ interpreted as the basename of a store item." (define* (render-narinfo/cached store request hash #:key ttl (compressions (list %no-compression)) - (nar-path "nar") + (nar-path "nar") negative-ttl cache pool) "Respond to the narinfo request for REQUEST. If the narinfo is available in CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo @@ -536,7 +545,7 @@ requested using POOL." #:compression (first compressions))))) (cond ((string-null? item) - (not-found request)) + (not-found request #:ttl negative-ttl)) ((file-exists? cached) ;; Narinfo is in cache, send it. (values `((content-type . (application/x-nix-narinfo)) @@ -584,7 +593,7 @@ requested using POOL." #:phrase "We're baking it" #:ttl 300))) ;should be available within 5m (else - (not-found request #:phrase ""))))) + (not-found request #:phrase "" #:ttl negative-ttl))))) (define (compress-nar cache item compression) "Save in directory CACHE the nar for ITEM compressed with COMPRESSION." @@ -974,7 +983,7 @@ methods, return the applicable compression." (define* (make-request-handler store #:key cache pool - narinfo-ttl + narinfo-ttl narinfo-negative-ttl (nar-path "nar") (compressions (list %no-compression))) (define compression-type? @@ -1006,10 +1015,12 @@ methods, return the applicable compression." #:cache cache #:pool pool #:ttl narinfo-ttl + #:negative-ttl narinfo-negative-ttl #:nar-path nar-path #:compressions compressions) (render-narinfo store request hash #:ttl narinfo-ttl + #:negative-ttl narinfo-negative-ttl #:nar-path nar-path #:compressions compressions))) ;; /nar/file/NAME/sha256/HASH @@ -1068,7 +1079,7 @@ methods, return the applicable compression." #:key advertise? port (compressions (list %no-compression)) - (nar-path "nar") narinfo-ttl + (nar-path "nar") narinfo-ttl narinfo-negative-ttl cache pool) (when advertise? (let ((name (service-name))) @@ -1084,6 +1095,7 @@ methods, return the applicable compression." #:pool pool #:nar-path nar-path #:narinfo-ttl narinfo-ttl + #:narinfo-negative-ttl narinfo-negative-ttl #:compressions compressions) concurrent-http-server `(#:socket ,socket))) @@ -1127,6 +1139,7 @@ methods, return the applicable compression." (user (assoc-ref opts 'user)) (port (assoc-ref opts 'port)) (ttl (assoc-ref opts 'narinfo-ttl)) + (negative-ttl (assoc-ref opts 'narinfo-negative-ttl)) (compressions (match (filter-map (match-lambda (('compression . compression) compression) @@ -1192,6 +1205,7 @@ consider using the '--user' option!~%"))) "publish worker")) #:nar-path nar-path #:compressions compressions + #:narinfo-negative-ttl negative-ttl #:narinfo-ttl ttl)))))) ;;; Local Variables: diff --git a/tests/publish.scm b/tests/publish.scm index 3e67c435ac..c3d086995a 100644 --- a/tests/publish.scm +++ b/tests/publish.scm @@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 David Thompson ;;; Copyright © 2020 by Amar M. Singh -;;; Copyright © 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -700,6 +700,36 @@ References: ~%" (= (response-content-length response) (stat:size (stat log))) (first (response-content-type response)))))) +(test-equal "negative TTL" + `(404 42) + + (call-with-temporary-directory + (lambda (cache) + (let ((thread (with-separate-output-ports + (call-with-new-thread + (lambda () + (guix-publish "--port=6786" "-C0" + "--negative-ttl=42s")))))) + (wait-until-ready 6786) + + (let* ((base "http://localhost:6786/") + (url (string-append base (make-string 32 #\z) + ".narinfo")) + (response (http-get url))) + (list (response-code response) + (match (assq-ref (response-headers response) 'cache-control) + ((('max-age . ttl)) ttl) + (_ #f)))))))) + +(test-equal "no negative TTL" + `(404 #f) + (let* ((uri (publish-uri + (string-append "/" (make-string 32 #\z) + ".narinfo"))) + (response (http-get uri))) + (list (response-code response) + (assq-ref (response-headers response) 'cache-control)))) + (test-equal "/log/NAME not found" 404 (let ((uri (publish-uri "/log/does-not-exist"))) -- 2.31.1 From unknown Tue Jun 24 01:38:35 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47897] [PATCH] substitutes: Don't cache negative lookups or transient errors. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 11 May 2021 13:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: guix-devel@gnu.org, 47897@debbugs.gnu.org Received: via spool by 47897-submit@debbugs.gnu.org id=B47897.162073857815179 (code B ref 47897); Tue, 11 May 2021 13:10:02 +0000 Received: (at 47897) by debbugs.gnu.org; 11 May 2021 13:09:38 +0000 Received: from localhost ([127.0.0.1]:35357 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lgS8o-0003wk-B1 for submit@debbugs.gnu.org; Tue, 11 May 2021 09:09:38 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43092) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lgS8n-0003wZ-NX for 47897@debbugs.gnu.org; Tue, 11 May 2021 09:09:37 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48422) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lgS8i-0006Zb-1z; Tue, 11 May 2021 09:09:32 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=57488 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lgS8a-0000pp-Rs; Tue, 11 May 2021 09:09:29 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87mtvhnsn6.fsf@cbaines.net> <874kfyufzg.fsf@gnu.org> <87wnsthpyn.fsf@cbaines.net> Date: Tue, 11 May 2021 15:09:23 +0200 In-Reply-To: <87wnsthpyn.fsf@cbaines.net> (Christopher Baines's message of "Fri, 23 Apr 2021 00:14:24 +0100") Message-ID: <87tun9zabg.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Christopher Baines skribis: >> Now, the penalty it imposes is annoying. I=E2=80=99ve sometimes found m= yself >> working around it, too (because I knew the server was going to have the >> store item sooner than 1h). >> >> Rather than removing it entirely, I can think of these options: >> >> 1. Reduce the default negative timeouts. > > I think reducing it is good, as you say, it's possible to override the > default from the server side. Just in case someone wants caching > behaviour, it might be worth keeping that functionality at least. OK, let=E2=80=99s do that. >> 2. Add an option to =E2=80=98guix publish=E2=80=99 (and to the Coordin= ator?) so they >> send a =E2=80=98Cache-Control=E2=80=99 header with the chosen TTL o= n 404. That >> way, if the server operator doesn=E2=80=99t mind extra load, they c= an run >> =E2=80=9Cguix publish --negative-ttl=3D0=E2=80=9D. > > That sounds sensible. The Guix Build Coordinator doesn't do any serving, > that's left to something else like nginx. For the deployments I maintain > though, I don't think I'm setting the relevant headers, but I'll look at > changing that. Cool. > Going back to the %narinfo-transient-error-ttl, if I'm correct in saying > that it's not possible to override that, maybe that should also use the > relevant header value if set? Correct, =E2=80=98%narinfo-transient-error-ttl=E2=80=99 cannot be overridde= n. We can halve it if you think that=E2=80=99s useful, thought when that happens, it = means something=E2=80=99s wrong with the server (returning 500 or similar). I=E2=80=99ve sent patches to address this, lemme know what you think! Thanks, Ludo=E2=80=99. From unknown Tue Jun 24 01:38:35 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47897] [PATCH] substitutes: Don't cache negative lookups or transient errors. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 14 May 2021 07:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47897 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org, 47897@debbugs.gnu.org Received: via spool by 47897-submit@debbugs.gnu.org id=B47897.162097747628535 (code B ref 47897); Fri, 14 May 2021 07:32:02 +0000 Received: (at 47897) by debbugs.gnu.org; 14 May 2021 07:31:16 +0000 Received: from localhost ([127.0.0.1]:44854 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhSI0-0007QB-7p for submit@debbugs.gnu.org; Fri, 14 May 2021 03:31:16 -0400 Received: from mira.cbaines.net ([212.71.252.8]:35948) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhSHx-0007Q1-Us for 47897@debbugs.gnu.org; Fri, 14 May 2021 03:31:14 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 188AE27BC78; Fri, 14 May 2021 08:31:13 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id be91b3db; Fri, 14 May 2021 07:31:12 +0000 (UTC) References: <87mtvhnsn6.fsf@cbaines.net> <874kfyufzg.fsf@gnu.org> <87wnsthpyn.fsf@cbaines.net> <87tun9zabg.fsf_-_@gnu.org> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines In-reply-to: <87tun9zabg.fsf_-_@gnu.org> Date: Fri, 14 May 2021 08:31:09 +0100 Message-ID: <874kf5epqa.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi, > > Christopher Baines skribis: > >>> Now, the penalty it imposes is annoying. I=E2=80=99ve sometimes found = myself >>> working around it, too (because I knew the server was going to have the >>> store item sooner than 1h). >>> >>> Rather than removing it entirely, I can think of these options: >>> >>> 1. Reduce the default negative timeouts. >> >> I think reducing it is good, as you say, it's possible to override the >> default from the server side. Just in case someone wants caching >> behaviour, it might be worth keeping that functionality at least. > > OK, let=E2=80=99s do that. > >>> 2. Add an option to =E2=80=98guix publish=E2=80=99 (and to the Coordi= nator?) so they >>> send a =E2=80=98Cache-Control=E2=80=99 header with the chosen TTL = on 404. That >>> way, if the server operator doesn=E2=80=99t mind extra load, they = can run >>> =E2=80=9Cguix publish --negative-ttl=3D0=E2=80=9D. >> >> That sounds sensible. The Guix Build Coordinator doesn't do any serving, >> that's left to something else like nginx. For the deployments I maintain >> though, I don't think I'm setting the relevant headers, but I'll look at >> changing that. > > Cool. > >> Going back to the %narinfo-transient-error-ttl, if I'm correct in saying >> that it's not possible to override that, maybe that should also use the >> relevant header value if set? > > Correct, =E2=80=98%narinfo-transient-error-ttl=E2=80=99 cannot be overrid= den. We can > halve it if you think that=E2=80=99s useful, thought when that happens, i= t means > something=E2=80=99s wrong with the server (returning 500 or similar). > > I=E2=80=99ve sent patches to address this, lemme know what you think! The patches you've sent look good. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCeJz5fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XeYRA/+IpdxSoaVxtfa2ScTBNvlH99k0xVw3RBN FGVOvxa+bZimKwQjUzbT/tQz5xZqOhHbD6AeCvuYy0ilfktZqaEpaC2Abcy/n/KU ayWvUpaAMoE6VAGgn0TuD91WIWEPdrgibp/wCTVsezTIR6F8z6XMeyErSKztkdNp kKF8d86RxaCw/sVprO3o6iirsfX+ZI76WEW8P25VXhIqK8hy1H1ZcPQ8XPPKtL+k AnOfycEq+M3Q64NTnLfRHq7qet0juU/ttuszfgRktekXVWjAYVhvZ+3OXt+UqLcN UotpeOEFioCIJx8nwYJYOOVK4/JbUgwx2MvecX7UbnscrW42Tp6WFHy7hL2OmcPT Mkgzl+Y7OZ+cnRSxsXnO8CzutTUPHQzJq2i9ZWmctE7hsnr+hZ8vyWGc3NITZDAm xdY64g8FB4cWV2a0JKA4lAaxZZEu1SqNvKfLRobj7uevphk+GusnRGMcdaVjvRzK zrc4gEcL7cLsEikruSz9y0/0SAqAxUVov/PkOU0BRPWLOeUToEa5h6ypKhDlfpoF 5rgMbSTvXkHksKRBrX08uI6AqlOJrswQt+elhkdW3DRJzC50gPx74ipWlqL20YsG m4iAivyvR78ho+3/e3txAXWFvFClwqME8nw1f7GMzB7bxXQng/dmHDL/05YUPalI emBhOobfsAk= =Ym9E -----END PGP SIGNATURE----- --=-=-=-- From unknown Tue Jun 24 01:38:35 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Christopher Baines Subject: bug#47897: closed (Re: bug#47897: [PATCH] substitutes: Don't cache negative lookups or transient errors.) Message-ID: References: <87im3ifjs6.fsf_-_@gnu.org> <20210419184036.18632-1-mail@cbaines.net> X-Gnu-PR-Message: they-closed 47897 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 47897@debbugs.gnu.org Date: Sun, 16 May 2021 21:32:01 +0000 Content-Type: multipart/mixed; boundary="----------=_1621200721-5087-1" This is a multi-part message in MIME format... ------------=_1621200721-5087-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #47897: [PATCH] substitutes: Don't cache negative lookups or transient erro= rs. which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 47897@debbugs.gnu.org. --=20 47897: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D47897 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1621200721-5087-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 47897-done) by debbugs.gnu.org; 16 May 2021 21:31:15 +0000 Received: from localhost ([127.0.0.1]:51596 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liOLz-0001J4-68 for submit@debbugs.gnu.org; Sun, 16 May 2021 17:31:15 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38892) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liOLw-0001Ip-Eg for 47897-done@debbugs.gnu.org; Sun, 16 May 2021 17:31:13 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37030) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1liOLr-000299-84; Sun, 16 May 2021 17:31:07 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=59190 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1liOLq-00036J-Qe; Sun, 16 May 2021 17:31:07 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#47897: [PATCH] substitutes: Don't cache negative lookups or transient errors. References: <87mtvhnsn6.fsf@cbaines.net> <874kfyufzg.fsf@gnu.org> <87wnsthpyn.fsf@cbaines.net> <87tun9zabg.fsf_-_@gnu.org> <874kf5epqa.fsf@cbaines.net> Date: Sun, 16 May 2021 23:31:05 +0200 In-Reply-To: <874kf5epqa.fsf@cbaines.net> (Christopher Baines's message of "Fri, 14 May 2021 08:31:09 +0100") Message-ID: <87im3ifjs6.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 47897-done Cc: guix-devel@gnu.org, 47897-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Christopher Baines skribis: > The patches you've sent look good. Pushed as 938ffcbb0589adc07dc12c79eda3e1e2bb9e7cf8 (I was generous and lowered =E2=80=98%narinfo-negative-ttl=E2=80=99 to 10mn :-)). Thanks, Ludo=E2=80=99. ------------=_1621200721-5087-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 19 Apr 2021 18:40:56 +0000 Received: from localhost ([127.0.0.1]:52064 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lYYpM-0008QO-K3 for submit@debbugs.gnu.org; Mon, 19 Apr 2021 14:40:56 -0400 Received: from lists.gnu.org ([209.51.188.17]:40328) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lYYpH-0008QB-48 for submit@debbugs.gnu.org; Mon, 19 Apr 2021 14:40:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40064) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYYpD-0003xE-E8 for guix-patches@gnu.org; Mon, 19 Apr 2021 14:40:50 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:59795) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lYYp9-00026t-2y for guix-patches@gnu.org; Mon, 19 Apr 2021 14:40:47 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 92E0A27BC6C for ; Mon, 19 Apr 2021 19:40:37 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id b370b44f for ; Mon, 19 Apr 2021 18:40:36 +0000 (UTC) From: Christopher Baines To: guix-patches@gnu.org Subject: [PATCH] substitutes: Don't cache negative lookups or transient errors. Date: Mon, 19 Apr 2021 19:40:36 +0100 Message-Id: <20210419184036.18632-1-mail@cbaines.net> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) Caching these responses can delay users getting substitutes, since they'll have to wait at least as long as the cache TTL to fetch the substitute, even if it's available sooner. * guix/substitutes.scm (%narinfo-negative-ttl, %narinfo-transient-error-ttl): Remove variables. (cache-narinfo!): Adjust to not use the negative TTL. (fetch-narinfos): Don't cache non 200 status code responses. --- guix/substitutes.scm | 22 +++------------------- 1 file changed, 3 insertions(+), 19 deletions(-) diff --git a/guix/substitutes.scm b/guix/substitutes.scm index 08f8c24efd..55daa6c964 100644 --- a/guix/substitutes.scm +++ b/guix/substitutes.scm @@ -70,14 +70,6 @@ ;; 'Cache-Control' response header. (* 36 3600)) -(define %narinfo-negative-ttl - ;; Likewise, but for negative lookups---i.e., cached lookup failures (404). - (* 1 3600)) - -(define %narinfo-transient-error-ttl - ;; Likewise, but for transient errors such as 504 ("Gateway timeout"). - (* 10 60)) - (define %narinfo-cache-directory ;; A local cache of narinfos, to avoid going to the network. Most of the ;; time, 'guix substitute' is called by guix-daemon as root and stores its @@ -104,8 +96,7 @@ entry is stored in a sub-directory specific to CACHE-URL." (define (cache-narinfo! cache-url path narinfo ttl) "Cache locally NARNIFO for PATH, which originates from CACHE-URL, with the -given TTL (a number of seconds or #f). NARINFO may be #f, in which case it -indicates that PATH is unavailable at CACHE-URL." +given TTL (a number of seconds or #f)." (define now (current-time time-monotonic)) @@ -113,8 +104,7 @@ indicates that PATH is unavailable at CACHE-URL." `(narinfo (version 2) (cache-uri ,cache-uri) (date ,(time-second now)) - (ttl ,(or ttl - (if narinfo %narinfo-ttl %narinfo-negative-ttl))) + (ttl ,(or ttl %narinfo-ttl)) (value ,(and=> narinfo narinfo->string)))) (let ((file (narinfo-cache-file cache-url path))) @@ -214,16 +204,10 @@ if file doesn't exist, and the narinfo otherwise." (cache-narinfo! url (narinfo-path narinfo) narinfo ttl) (cons narinfo result)) result)) - (let* ((path (uri-path (request-uri request))) - (hash-part (basename - (string-drop-right path 8)))) ;drop ".narinfo" + (begin (if len (get-bytevector-n port len) (read-to-eof port)) - (cache-narinfo! url (hash-part->path hash-part) #f - (if (or (= 404 code) (= 202 code)) - ttl - %narinfo-transient-error-ttl)) result)))) (define (do-fetch uri) -- 2.30.1 ------------=_1621200721-5087-1--