Subject: [bug#76003] [PATCH] distribute NARs through IPFS.

From: Justin Veilleux <[email-masked]>
Date: Sat, 01 Feb 2025 23:36:18 -0500

Hello everyone.

I think Distributed substitute distribution has been on many people's
radar for a time. However, from what I understood of the discussion at
https://issues.guix.gnu.org/33899 (which has been stagnant for many
years), actually achieving this is complicated because of the lack of a
good way to encode into IPFS store items in a way that correctly
deduplicates identical files (giving them the same hash) and works well
with the rest of the IPFS ecosystem. Solving this problem is hard and
until a spec for UnixFSv2 is created and implemented in the main IPFS
implementations, making perfect use of all the IPFS features will be
harder and require a lot of effort.

However, I would really like to have distributed substitutes, even if it
is done through a less-than optimal mechanism, and I feel like (I might be
wrong) that is a common sentiment.

The attached patch series adds to the `guix publish` daemon an option
`--ipfs-api` which, when present, makes it publish through ipfs each
compressed nar file it creates, and adds a corresponding entry in the
narinfo through the URL: ipfs://Qm... property.

on the `guix substitute` side, an `ipfs-fetch` function is also added to
download nars through a local gateway. This means that the new URL:
ipfs:Qm... fields can now be exploited as easily as the regular
nar/lzip/... urls.

These changes don't exploit every advantage of IPFS delivery
(deduplication, "true decentralization", etc), but enable a non-trivial
use-case: many machines on a local network can now distribute amongst
themselves substitutes (thus reducing the total needed bandwidth)
transparently.

WDYT?

>From 07bf18b20faa86375d2eb80a49a6a6b36ad294f9 Mon Sep 17 00:00:00 2001
Message-ID: <07b[email-masked]>
From: terramorpha <[email-masked]>
Date: Sat, 1 Feb 2025 20:13:47 -0500
Subject: [PATCH 1/4] gnu: substitute: Add support for downloading nars over
 ipfs.

Change-Id: Ia8e7779a65ba07e980586793dc159e06a9da0a66
---
 guix/build/download.scm     | 26 ++++++++++++++++++++++++++
 guix/ipfs.scm               |  4 +++-
 guix/scripts/substitute.scm | 15 ++++++++++++++-
 3 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/guix/build/download.scm b/guix/build/download.scm
index 74b7486b7b..7e59aecd3d 100644
--- a/guix/build/download.scm
+++ b/guix/build/download.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2015 Mark H Weaver <[email-masked]>
 ;;; Copyright © 2017 Tobias Geerinckx-Rice <[email-masked]>
 ;;; Copyright © 2021 Timothy Sample <[email-masked]>
+;;; Copyright © 2025 Justin Veilleux <[email-masked]>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -29,6 +30,7 @@ (define-module (guix build download)
   #:use-module (guix build utils)
   #:use-module (guix progress)
   #:use-module (guix memoization)
+  #:use-module (guix ipfs)
   #:use-module (rnrs io ports)
   #:use-module (rnrs bytevectors)
   #:use-module (srfi srfi-1)
@@ -46,6 +48,7 @@ (define-module (guix build download)
             open-socket-for-uri
             open-connection-for-uri
             http-fetch
+            ipfs-fetch
             %x509-certificate-directory
             close-connection
             resolve-uri-reference
@@ -614,6 +617,29 @@ (define* (http-fetch uri #:key timeout (verify-certificate? #t))
        (error "download failed" (uri->string uri)
               code (response-reason-phrase resp))))))
 
+(define (ipfs-fetch uri)
+  "Return an input port containing the data at URI (which is expected to be
+ipfs://hash), and the expected number of bytes available or #f."
+  (unless (eq? (uri-scheme uri) 'ipfs)
+    (error "scheme is not ipfs"))
+
+  (let ((hash (uri-host uri)))
+    (let-values (((response port)
+                  (http-post (string-append (%ipfs-base-url)
+                                            "/api/v0/cat?arg=" hash)
+                             #:streaming? #t
+                             ;; Always pass "Connection: close".
+                             #:keep-alive? #f
+                             #:headers `((connection close)))))
+      (let ((content-length-header (assoc-ref (response-headers response)
+                                              'x-content-length)))
+       (case (response-code response)
+         ((200)
+          (values port (and content-length-header (string->number content-length-header))))
+         (else
+          (error "ipfs download failed" (uri->string uri)
+                 (response-code response) (response-reason-phrase response))))))))
+
 
 (define-syntax-rule (false-if-exception* body ...)
   "Like `false-if-exception', but print the exception on the error port."
diff --git a/guix/ipfs.scm b/guix/ipfs.scm
index 3c25f2a499..43e7dfb2f2 100644
--- a/guix/ipfs.scm
+++ b/guix/ipfs.scm
@@ -49,7 +49,9 @@ (define-module (guix ipfs)
 
 (define %ipfs-base-url
   ;; URL of the IPFS gateway.
-  (make-parameter "http://localhost:5001"))
+  (make-parameter
+   (or (getenv "GUIX_IPFS_API")
+       "http://localhost:5001")))
 
 (define* (call url decode #:optional (method http-post)
                #:key body (false-if-404? #t) (headers '()))
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index ab18104e0a..66b6911515 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2014 Nikita Karetnikov <[email-masked]>
 ;;; Copyright © 2018 Kyle Meyer <[email-masked]>
 ;;; Copyright © 2020 Christopher Baines <[email-masked]>
+;;; Copyright © 2025 Justin Veilleux <[email-masked]>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -62,11 +63,13 @@ (define-module (guix scripts substitute)
   #:use-module (srfi srfi-71)
   #:use-module (web uri)
   #:use-module (guix http-client)
+  #:use-module (guix ipfs)
   #:export (%allow-unauthenticated-substitutes?
             %reply-file-descriptor
 
             substitute-urls
-            guix-substitute))
+            guix-substitute
+            download-nar))
 
 ;;; Comment:
 ;;;
@@ -504,6 +507,16 @@ (define* (download-nar narinfo destination
                          #:port port
                          #:keep-alive? #t
                          #:buffered? #f))))
+      ((ipfs)
+         (let ((hash (uri-host uri)))
+           (if fetch-timeout
+               (with-timeout %fetch-timeout
+                 (begin
+                   (warning (G_ "while fetching ~a: ipfs is somewhat slow~%")
+                            (uri->string uri))
+                   (warning (G_ "try `--no-substitutes' if the problem persists~%")))
+                 (ipfs-fetch uri))
+               (ipfs-fetch uri))))
       (else
        (raise
         (formatted-message

base-commit: 0550c4191fae3b3b00b8bb9faba8a2fe2ad0d188
-- 
2.47.1

>From efb9fc07bc797553da8b900ac6b0593119420a8c Mon Sep 17 00:00:00 2001
Message-ID: <efb[email-masked]>
In-Reply-To: <07b[email-masked]>
References: <07b[email-masked]>
From: terramorpha <[email-masked]>
Date: Sat, 1 Feb 2025 22:06:11 -0500
Subject: [PATCH 2/4] gnu: publish: Add support for distributing nars through
 ipfs.

Change-Id: I0702114aeb91f19a89c6093095c39d21607373fc
---
 guix/scripts/publish.scm | 68 ++++++++++++++++++++++++++++++++--------
 1 file changed, 55 insertions(+), 13 deletions(-)

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index a000c559a7..425c880e83 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2020 Maxim Cournoyer <[email-masked]>
 ;;; Copyright © 2021 Simon Tournier <[email-masked]>
 ;;; Copyright © 2021, 2022 Mathieu Othacehe <[email-masked]>
+;;; Copyright © 2025 Justin Veilleux <[email-masked]>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -66,6 +67,7 @@ (define-module (guix scripts publish)
   #:use-module ((guix build utils)
                 #:select (dump-port mkdir-p find-files))
   #:use-module ((guix build syscalls) #:select (set-thread-name))
+  #:use-module ((guix ipfs) #:prefix ipfs:)
   #:export (%default-gzip-compression
 
             %public-key
@@ -93,6 +95,8 @@ (define (show-help)
                          compress archives with METHOD at LEVEL"))
   (display (G_ "
   -c, --cache=DIRECTORY  cache published items to DIRECTORY"))
+  (display (G_ "
+      --ipfs-api=URL     if present, publish nar to ipfs through the api at URL"))
   (display (G_ "
       --cache-bypass-threshold=SIZE
                          serve store items below SIZE even when not cached"))
@@ -245,7 +249,10 @@ (define %options
                 (lambda (opt name arg result)
                   ;; If port unspecified, use default Guile REPL port.
                   (let ((port (and arg (string->number* arg))))
-                    (alist-cons 'repl (or port 37146) result))))))
+                    (alist-cons 'repl (or port 37146) result))))
+        (option '("ipfs-api") #t #f
+                (lambda (opt name arg result)
+                  (alist-cons 'ipfs-api arg result)))))
 
 (define %default-options
   `((port . 8080)
@@ -303,7 +310,8 @@ (define* (store-item->recutils store-item
                                #:key
                                (nar-path "nar")
                                (compression %no-compression)
-                               file-size)
+                               file-size
+                               ipfs-hashes)
   "Return the 'Compression' and 'URL' fields of the narinfo for STORE-ITEM,
 with COMPRESSION, starting at NAR-PATH."
   (let ((url (encode-and-join-uri-path
@@ -319,7 +327,11 @@ (define* (store-item->recutils store-item
 
 (define* (narinfo-string store store-path
                          #:key (compressions (list %no-compression))
-                         (nar-path "nar") (file-sizes '()))
+                         (nar-path "nar")
+                         ;; ((compression-method . file-size) ...)
+                         (file-sizes '())
+                         ;; ((compression-method . hash) ...)
+                         (ipfs-hashes '()))
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
@@ -363,12 +375,26 @@ (define* (narinfo-string store store-path
             ;; narinfo without having to resign them.
             (map (lambda (compression)
                    (let ((size (assoc-ref file-sizes
-                                          compression)))
-                     (store-item->recutils store-path
-                                           #:file-size size
-                                           #:nar-path nar-path
-                                           #:compression
-                                           compression)))
+                                          compression))
+                         (ipfs-hash (assoc-ref ipfs-hashes
+                                               compression)))
+                     (string-append
+                      (store-item->recutils store-path
+                                            #:file-size size
+                                            #:nar-path nar-path
+                                            #:compression
+                                            compression)
+
+                      (if ipfs-hash
+                          (format #f
+                                "URL: ~a
+Compression: ~a
+FileSize: ~a
+"
+                                (string-append "ipfs://" ipfs-hash)
+                                (compression-type compression)
+                                size)
+                          ""))))
                  compressions))))
 
 ;; Custom header to indicate that baking is in progress.
@@ -645,8 +671,21 @@ (define* (bake-narinfo+nar cache item
            (cons compression (stat:size stat)))))
 
   (let ((compression (actual-compressions item compressions)))
-
-    (for-each (cut compress-nar cache item <>) compressions)
+    (define ipfs-hashes '())
+
+    (for-each
+     (lambda (compression)
+       (compress-nar cache item compression)
+       (when (ipfs:%ipfs-base-url)
+         (let* ((place (nar-cache-file cache item #:compression compression))
+                ;; In the future, we might want to use the api equivalent of
+                ;; the (experimental) --nocopy flag, which "interns" the file
+                ;; without duplicating it in the ipfs store.
+                (ipfs-thing (ipfs:add-file place))
+                (hash (ipfs:content-hash ipfs-thing)))
+           (set! ipfs-hashes (alist-cons compression hash
+                                         ipfs-hashes)))))
+         compressions)
 
     (match compressions
       ((main others ...)
@@ -662,7 +701,8 @@ (define* (bake-narinfo+nar cache item
                  (display (narinfo-string store item
                                           #:nar-path nar-path
                                           #:compressions compressions
-                                          #:file-sizes sizes)
+                                          #:file-sizes sizes
+                                          #:ipfs-hashes ipfs-hashes)
                           port)))
 
              ;; Make the cached narinfo world-readable, contrary to what
@@ -1277,7 +1317,8 @@ (define-command (guix-publish . args)
            ;; Read the key right away so that (1) we fail early on if we can't
            ;; access them, and (2) we can then drop privileges.
            (public-key  (read-file-sexp (assoc-ref opts 'public-key-file)))
-           (private-key (read-file-sexp (assoc-ref opts 'private-key-file))))
+           (private-key (read-file-sexp (assoc-ref opts 'private-key-file)))
+           (ipfs-api (assoc-ref opts 'ipfs-api)))
 
       (when user
         ;; Now that we've read the key material and opened the socket, we can
@@ -1290,6 +1331,7 @@ (define-command (guix-publish . args)
 
       (parameterize ((%public-key public-key)
                      (%private-key private-key)
+                     (ipfs:%ipfs-base-url ipfs-api)
                      (cache-bypass-threshold
                       (or (assoc-ref opts 'cache-bypass-threshold)
                           (cache-bypass-threshold))))
-- 
2.47.1

>From 80934014c6f603347996f90f2b915ebdb6b224e7 Mon Sep 17 00:00:00 2001
Message-ID: <809[email-masked]>
In-Reply-To: <07b[email-masked]>
References: <07b[email-masked]>
From: terramorpha <[email-masked]>
Date: Sat, 1 Feb 2025 22:16:29 -0500
Subject: [PATCH 3/4] doc: publish: Add documentation for the --ipfs-api
 argument.

---
 doc/guix.texi | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index bb5f29277f..fe1b3c08d9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -104,7 +104,7 @@
 Copyright @copyright{} 2022 Aleksandr Vityazev@*
 Copyright @copyright{} 2022 Philip M@sup{c}Grath@*
 Copyright @copyright{} 2022 Karl Hallsby@*
-Copyright @copyright{} 2022 Justin Veilleux@*
+Copyright @copyright{} 2022, 2025 Justin Veilleux@*
 Copyright @copyright{} 2022 Reily Siegel@*
 Copyright @copyright{} 2022 Simon Streit@*
 Copyright @copyright{} 2022 (@*
@@ -16198,6 +16198,10 @@ Invoking guix publish
 service discovery (DNS-SD), currently @i{via} Guile-Avahi (@pxref{Top,,,
 guile-avahi, Using Avahi in Guile Scheme Programs}).
 
+When the @option{--ipfs-api=URL} argument is passed and caching is
+enabled, the server will also make its substitutes available through the
+@uref{https://ipfs.io,IPFS} daemon available at the given url.
+
 The general syntax is:
 
 @example
-- 
2.47.1

>From f71fb156b6b217cea0a181db13d88ccce69f0b5d Mon Sep 17 00:00:00 2001
Message-ID: <f71[email-masked]>
In-Reply-To: <07b[email-masked]>
References: <07b[email-masked]>
From: terramorpha <[email-masked]>
Date: Sat, 1 Feb 2025 22:41:28 -0500
Subject: [PATCH 4/4] services: guix-publish: Expose and document the
 --ipfs-api argument.

Change-Id: Iecabcc79b11a263fd77b25e51ce9d02fcb3aeddc
---
 doc/guix.texi         | 4 ++++
 gnu/services/base.scm | 7 ++++++-
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index fe1b3c08d9..ccd7b7d73f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20352,6 +20352,10 @@ Base Services
 When true, advertise the service on the local network @i{via} the DNS-SD
 protocol, using Avahi.
 
+@item @code{ipfs-api} (default: @code{#f})
+The url to an IPFS node api, when present (and @code{cache} has a
+value), publish the compressed NARs on the ipfs network.
+
 This allows neighboring Guix devices with discovery on (see
 @code{guix-configuration} above) to discover this @command{guix publish}
 instance and to automatically download substitutes from it.
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 7331c030d7..f8c586d189 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -2252,6 +2252,8 @@ (define-record-type* <guix-publish-configuration>
            (default "localhost"))
   (advertise? guix-publish-advertise?             ;boolean
               (default #f))
+  (ipfs-api guix-publish-ipfs-api                 ;string to a url
+            (default #f))
   (compression       guix-publish-configuration-compression
                      (thunked)
                      (default (default-compression this-record)))
@@ -2291,7 +2293,7 @@ (define (guix-publish-shepherd-service config)
 
   (match-record config <guix-publish-configuration>
     (guix port host nar-path cache workers ttl negative-ttl
-          cache-bypass-threshold advertise?)
+          cache-bypass-threshold advertise? ipfs-api)
     (let ((command #~(list #$(file-append guix "/bin/guix")
                            "publish" "-u" "guix-publish"
                            "-p" #$(number->string port)
@@ -2301,6 +2303,9 @@ (define (guix-publish-shepherd-service config)
                            #$@(if advertise?
                                   #~("--advertise")
                                   #~())
+                           #$@(if ipfs-api
+                                  #~((string-append "--ipfs-api=" ipfs-api))
+                                  #~())
                            #$@(if workers
                                   #~((string-append "--workers="
                                                     #$(number->string
-- 
2.47.1