GNU bug report logs - #75782
[PATCH cuirass] remote-server: Disable url-fetch* fallback when triggering substitutes.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75782 in the body.
You can then email your comments to 75782 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: vicvbcun <guix <at> ikherbers.com>
To: guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH cuirass] remote-server: Disable url-fetch* fallback when
 triggering substitutes.
Date: Thu, 23 Jan 2025 12:36:10 +0100

If an output is larger than the configured cache bypass threshold and the
substitute hasn't already been baked, guix publish returns a 404 response and
schedules baking.  This error code causes `url-fetch*' to instead try fetching
the narinfo from the Internet Archive.  Suppress this fallback behaviour as
the sole purpose is triggering the baking of substitutes.

* src/cuirass/scripts/remote-server.scm (trigger-substitutes-baking): Bind
`%download-methods' to the empty list befor calling `url-fetch*'.
---
Hello Guix!

I've stumbled upon this as somehow (presumably due to the way my server is
connected to the IPv4 internet) the remote server would block indefinitely
reading a reply from web.archive.org, needing to be restarted.  This patch
fixes (or at least hides) this behaviour.  Anyway, since there is no point
to the fallback in the first place, I think this merits a patch regardless
of that.

 src/cuirass/scripts/remote-server.scm | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/cuirass/scripts/remote-server.scm b/src/cuirass/scripts/remote-server.scm
index caeb8ed..fb99551 100644
--- a/src/cuirass/scripts/remote-server.scm
+++ b/src/cuirass/scripts/remote-server.scm
@@ -45,7 +45,8 @@
   #:use-module ((guix build utils) #:select (mkdir-p
                                              strip-store-file-name
                                              call-with-temporary-output-file))
-  #:autoload   (guix build download) (url-fetch)
+  #:autoload   (guix build download) (%download-methods
+                                      url-fetch)
   #:autoload   (gcrypt pk-crypto) (read-file-sexp)
   #:use-module (simple-zmq)
   #:use-module (srfi srfi-1)
@@ -255,7 +256,13 @@ Start a remote build server.\n") (%program-name))
     (log-debug "Bake: ~a" narinfo-url)
     (call-with-temporary-output-file
      (lambda (tmp-file port)
-       (url-fetch* narinfo-url tmp-file)))))
+       (parameterize
+           ;; XXX: guix publish returns a 404 response when the output is
+           ;; large and has not already been baked.  By default, this causes
+           ;; `url-fetch*' to fall back to the Internet Archive.  Disable
+           ;; this.
+           ((%download-methods '()))
+         (url-fetch* narinfo-url tmp-file))))))
 
 (define (add-to-store drv outputs url)
   "Add the OUTPUTS that are available from the substitute server at URL to the

base-commit: 6b499a74999fc05e19e349e2c8ca8e7f417147f5
-- 
2.47.1

Message #8 received at 75782 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: vicvbcun <guix <at> ikherbers.com>
Cc: 75782 <at> debbugs.gnu.org
Subject: Re: bug#75782: [PATCH cuirass] remote-server: Disable url-fetch*
 fallback when triggering substitutes.
Date: Thu, 13 Feb 2025 16:09:20 +0100

Hi,

vicvbcun <guix <at> ikherbers.com> skribis:

> If an output is larger than the configured cache bypass threshold and the
> substitute hasn't already been baked, guix publish returns a 404 response and
> schedules baking.  This error code causes `url-fetch*' to instead try fetching
> the narinfo from the Internet Archive.  Suppress this fallback behaviour as
> the sole purpose is triggering the baking of substitutes.
>
> * src/cuirass/scripts/remote-server.scm (trigger-substitutes-baking): Bind
> `%download-methods' to the empty list befor calling `url-fetch*'.

Good catch!

> +       (parameterize
> +           ;; XXX: guix publish returns a 404 response when the output is
> +           ;; large and has not already been baked.  By default, this causes
> +           ;; `url-fetch*' to fall back to the Internet Archive.  Disable
> +           ;; this.
> +           ((%download-methods '()))
> +         (url-fetch* narinfo-url tmp-file))))))

Shouldn’t it be '(upstream) rather than '()?

(I’m looking at guix/build/download.scm.)

Thanks,
Ludo’.

Message #11 received at 75782 <at> debbugs.gnu.org (full text, mbox):

From: vicvbcun <guix <at> ikherbers.com>
To: 75782 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH cuirass v2] remote-server: Disable url-fetch* fallback when
 triggering substitutes.
Date: Thu, 13 Feb 2025 17:17:07 +0100

If an output is larger than the configured cache bypass threshold and the
substitute hasn't already been baked, guix publish returns a 404 response and
schedules baking.  This error code causes `url-fetch*' to instead try fetching
the narinfo from the Internet Archive.  Suppress this fallback behaviour as
the sole purpose is triggering the baking of substitutes.

* src/cuirass/scripts/remote-server.scm (trigger-substitutes-baking): Disable
all download methods except "upstream" before calling `url-fetch*'.
---

Changes since v1:
  - bind `%download-methods' to '(upstream) instead of the empty list

 src/cuirass/scripts/remote-server.scm | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/cuirass/scripts/remote-server.scm b/src/cuirass/scripts/remote-server.scm
index caeb8ed..90a3db7 100644
--- a/src/cuirass/scripts/remote-server.scm
+++ b/src/cuirass/scripts/remote-server.scm
@@ -45,7 +45,8 @@
   #:use-module ((guix build utils) #:select (mkdir-p
                                              strip-store-file-name
                                              call-with-temporary-output-file))
-  #:autoload   (guix build download) (url-fetch)
+  #:autoload   (guix build download) (%download-methods
+                                      url-fetch)
   #:autoload   (gcrypt pk-crypto) (read-file-sexp)
   #:use-module (simple-zmq)
   #:use-module (srfi srfi-1)
@@ -255,7 +256,13 @@ Start a remote build server.\n") (%program-name))
     (log-debug "Bake: ~a" narinfo-url)
     (call-with-temporary-output-file
      (lambda (tmp-file port)
-       (url-fetch* narinfo-url tmp-file)))))
+       (parameterize
+           ;; XXX: guix publish returns a 404 response when the output is
+           ;; large and has not already been baked.  By default, this causes
+           ;; `url-fetch*' to fall back to the Internet Archive.  Disable
+           ;; this.
+           ((%download-methods '(upstream)))
+         (url-fetch* narinfo-url tmp-file))))))
 
 (define (add-to-store drv outputs url)
   "Add the OUTPUTS that are available from the substitute server at URL to the

base-commit: 6b499a74999fc05e19e349e2c8ca8e7f417147f5
-- 
2.48.1

Message #14 received at 75782 <at> debbugs.gnu.org (full text, mbox):

From: vicvbcun <guix <at> ikherbers.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 75782 <at> debbugs.gnu.org
Subject: Re: [bug#75782] [PATCH cuirass] remote-server: Disable url-fetch*
 fallback when triggering substitutes.
Date: Thu, 13 Feb 2025 17:25:24 +0100

Hello,

On 2025-02-13T16:09:20+0100, Ludovic Courtès wrote:
> [...]
>> +       (parameterize
>> +           ;; XXX: guix publish returns a 404 response when the output is
>> +           ;; large and has not already been baked.  By default, this causes
>> +           ;; `url-fetch*' to fall back to the Internet Archive.  Disable
>> +           ;; this.
>> +           ((%download-methods '()))
>> +         (url-fetch* narinfo-url tmp-file))))))
>
>Shouldn’t it be '(upstream) rather than '()?
>
>(I’m looking at guix/build/download.scm.)
Of course ….  I guess I was so happy it didn't block that I forgot to 
verify it actually still does its job :).

I have sent a v2 but haven't tested it yet because of missing 
aarch64-linux substitutes.

vicvbcun

Message #17 received at 75782 <at> debbugs.gnu.org (full text, mbox):

From: vicvbcun <guix <at> ikherbers.com>
To: Ludovic Courtès <ludo <at> gnu.org>,
	75782 <at> debbugs.gnu.org
Subject: Re: [bug#75782] [PATCH cuirass] remote-server: Disable url-fetch*
 fallback when triggering substitutes.
Date: Thu, 13 Feb 2025 22:09:50 +0100

Hello,

On 2025-02-13T17:25:24+0100, vicvbcun wrote:
>Hello,
>
>On 2025-02-13T16:09:20+0100, Ludovic Courtès wrote:
>>[...]
>>>+       (parameterize
>>>+           ;; XXX: guix publish returns a 404 response when the output is
>>>+           ;; large and has not already been baked.  By default, this causes
>>>+           ;; `url-fetch*' to fall back to the Internet Archive.  Disable
>>>+           ;; this.
>>>+           ((%download-methods '()))
>>>+         (url-fetch* narinfo-url tmp-file))))))
>>
>>Shouldn’t it be '(upstream) rather than '()?
>>
>>(I’m looking at guix/build/download.scm.)
>Of course ….  I guess I was so happy it didn't block that I forgot to 
>verify it actually still does its job :).
>
>I have sent a v2 but haven't tested it yet because of missing 
>aarch64-linux substitutes.

I have replaced the cuirass-remote-server shepherd service with a 
transient service running the v2 patch.  It did trigger baking and 
didn't block when served a 404 respone.

vicvbcun

Message #22 received at 75782-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: vicvbcun <guix <at> ikherbers.com>
Cc: 75782-done <at> debbugs.gnu.org
Subject: Re: bug#75782: [PATCH cuirass] remote-server: Disable url-fetch*
 fallback when triggering substitutes.
Date: Fri, 14 Feb 2025 17:44:45 +0100

vicvbcun <guix <at> ikherbers.com> skribis:

> If an output is larger than the configured cache bypass threshold and the
> substitute hasn't already been baked, guix publish returns a 404 response and
> schedules baking.  This error code causes `url-fetch*' to instead try fetching
> the narinfo from the Internet Archive.  Suppress this fallback behaviour as
> the sole purpose is triggering the baking of substitutes.
>
> * src/cuirass/scripts/remote-server.scm (trigger-substitutes-baking): Disable
> all download methods except "upstream" before calling `url-fetch*'.

Pushed, thanks!

Ludo'.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.