GNU bug report logs - #46800
[PATCH] Allow defining multiple substituters

Previous Next

Full log

View this message in rfc822 format

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: 46800 <at> debbugs.gnu.org
Subject: [bug#46800] [PATCH] Allow defining multiple substituters
Date: Fri, 12 Mar 2021 18:37:53 +0100

Hi Maxime,

Maxime Devos <maximedevos <at> telenet.be> skribis:

> On Tue, 2021-03-02 at 21:37 +0100, Ludovic Courtès wrote:

[...]

>> As discussed on IRC, the daemon used to have support for multiple
>> substituters, but as a built-in C++ interface, which I removed in
>> f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f.
>
> Was there any particular reason this support was removed, beyond
> moving from C++ to Scheme and the absence of any alternative substituters?

These were the main reasons, yes.

>> The Scheme interface you propose is of course nicer :-), but I’m still
>> not sure it’s necessary.  For example, in the IPFS prototype at
>> <https://issues.guix.gnu.org/33899>;, IPFS support goes hand in hand with
>> HTTP support: narinfos are retrieved over HTTP and nars can be retrieved
>> over IPFS, or HTTP.
>
> About X going hand-in-hand with Y:
>
> Note that fetching narinfos, or fetching the nar itself are separated
> A method can support both procedures, or just one of them (or none,
> but that's rather useless.)
>
> Users (well, the system administrator) can choose multiple methods, which
> will be each fetch narinfos after each other & combine the
> results into
> one large list (or maybe some other data structure, I don't recall the
> details), and each substituter will be asked to produce
> a nar until a substituter
> succeeds or all have said "sorry, I don't have that nar".

OK.

> (That's different from C++ interface for multiple substituters I think, where
> the methods are only tried sequentialy, they aren't combined.)
>
> In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are
> enabled, in that order: "--substitute-methods=ipfs http".  The IPFS substituter
> won't be able to produce any narinfos by itself, but that's no problem as
> the HTTP substituter can find some.  Then, the IPFS substituter will be asked
> first to download a substitute, as it's first in the "--substitute-methods" list.
>
> And what if the narinfo doesn't have a IPFS URI, as the substitute server doesn't
> support that?  Then "guix substitute" automatically fall-backs to HTTP.
>
> Summary: some substitution methods can't do everything on their own, but that's ok,
> as "guix substitute" will just ask them to try what they can and will see if some
> combination of methods works.

Alright.

> About ‘not sure it's necessary’: there presumably will be a GNUnet substituter
> at some point.  I suppose it would be possible to define all substitute methods
> in (guix scripts substitute), but then you would still end up with a procedure
> that tries all methods (e.g. in wip-ipfs-substitutes, process-substitution has
> an "if" structures that tries downloading via IPFS with fall-back to HTTP; this
> would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-code?))

I guess considerations that are more important to me (and to users, I
suppose) now than a few years back are maintainability and robustness.

Concretely, I wouldn’t want Guix to offer out of the box 4 methods, 3 of
which perform poorly or are downright buggy.  I think it would be more
fruitful if, as a project, we would focus on one or two methods or
method combinations that we have battle-tested, perform well, and a nice
long-term maintenance story, and so on.

[...]

>>  we’d rather let them choose a policy that
>> can automatically pick the “best” method, dynamically adjusting choices.
>
> Who's the user here?
> (a) the system administrator, who configuring the daemon to use a certain
>     list of substituters and defines a default list of substitute uris.
> (b) the ‘user’, that doesn't directly have the capability to modify
>     the system's guix daemon (or possibly an administrator that wants to
>     to test some things out without the possibility of accidentally messing
>     up the ‘real’ system).

I think (b) should be possible, just like users can pass
‘--substitute-urls’.

[...]

> About *automatically* dynamically adjusting choices: would be nice, but how is
> this supposed to work?  Any ideas?  The only thing I could think of is a
> allowing the user to choose which narinfo to use (e.g. from the list of found
> narinfos try to choose a narinfo that has an IPFS URI).

I think it’ll have to be fine-tuned once we have several stable
substitute methods.  After all, we have yet to figure out how to choose
between zstd and lzip for the current substitution mechanism; the
tradeoffs when very different methods are in use may be more complex!

>> All in all, I would prefer to wait until there’s a clear need for this
>> abstraction.
>
> See above responses.

I don’t think my concerns are really addressed :-), but at the same time
I think we need a playground for these things so we can actually grow
new substitute methods like those you’ve been looking at.  Hmmm tricky!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.