GNU bug report logs - #76003
[PATCH] distribute NARs through IPFS.

Previous Next

Full log

View this message in rfc822 format

From: Justin Veilleux <terramorpha <at> cock.li>
To: 76003 <at> debbugs.gnu.org, ludo <at> gnu.org
Subject: [bug#76003] [PATCH] distribute NARs through IPFS.
Date: Sat, 01 Feb 2025 23:36:18 -0500

[Message part 1 (text/plain, inline)]

Hello everyone.

I think Distributed substitute distribution has been on many people's
radar for a time. However, from what I understood of the discussion at
https://issues.guix.gnu.org/33899 (which has been stagnant for many
years), actually achieving this is complicated because of the lack of a
good way to encode into IPFS store items in a way that correctly
deduplicates identical files (giving them the same hash) and works well
with the rest of the IPFS ecosystem. Solving this problem is hard and
until a spec for UnixFSv2 is created and implemented in the main IPFS
implementations, making perfect use of all the IPFS features will be
harder and require a lot of effort.

However, I would really like to have distributed substitutes, even if it
is done through a less-than optimal mechanism, and I feel like (I might be
wrong) that is a common sentiment.

The attached patch series adds to the `guix publish` daemon an option
`--ipfs-api` which, when present, makes it publish through ipfs each
compressed nar file it creates, and adds a corresponding entry in the
narinfo through the URL: ipfs://Qm... property.

on the `guix substitute` side, an `ipfs-fetch` function is also added to
download nars through a local gateway. This means that the new URL:
ipfs:Qm... fields can now be exploited as easily as the regular
nar/lzip/... urls.

These changes don't exploit every advantage of IPFS delivery
(deduplication, "true decentralization", etc), but enable a non-trivial
use-case: many machines on a local network can now distribute amongst
themselves substitutes (thus reducing the total needed bandwidth)
transparently.

WDYT?

[0001-gnu-substitute-Add-support-for-downloading-nars-over.patch (text/x-patch, attachment)]

[0002-gnu-publish-Add-support-for-distributing-nars-throug.patch (text/x-patch, attachment)]

[0003-doc-publish-Add-documentation-for-the-ipfs-api-argum.patch (text/x-patch, attachment)]

[0004-services-guix-publish-Expose-and-document-the-ipfs-a.patch (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.