GNU bug report logs - #47336
Disarchive as a fallback for downloads

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Timothy Sample <samplet <at> ngyro.com>
Subject: bug#47336: closed (Re: bug#47336: Disarchive as a fallback for
 downloads)
Date: Thu, 29 Apr 2021 17:25:02 +0000

[Message part 1 (text/plain, inline)]

Your bug report

#47336: Disarchive as a fallback for downloads

which was filed against the guix-patches package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to 47336 <at> debbugs.gnu.org.

-- 
47336: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=47336
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Timothy Sample <samplet <at> ngyro.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 47336-done <at> debbugs.gnu.org
Subject: Re: bug#47336: Disarchive as a fallback for downloads
Date: Thu, 29 Apr 2021 13:24:09 -0400

Hello,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Timothy Sample <samplet <at> ngyro.com> skribis:
>
>> And broke “guix pull”!!  (I somehow fooled myself into thinking that I
>> had already tested with “guix pull --url=...” locally.)  I reverted the
>> offending commit.
>
> You can test with ‘guix pull’ (you need to make sure to specify the
> right file:// URL *and* branch), or you can run “make as-derivation”.

I will definitely be more careful with this in the future.

>> [...]  Does this approach look okay?
>
> That’s one possibility.
>
> The patch below takes another approach.  I think it aesthetically
> slightly more pleasant because we don’t have to play ‘resolve-module’
> tricks for obscure reasons.  WDYT?

This is exactly what I was hoping for, but I couldn’t quite connect all
the dots in “build-self.scm”.  Thanks!

> (It also fixes a format string argument mismatch.)

Good catch!

I’ve pushed the updated patch and am closing the issue.  :)


-- Tim

[Message part 3 (message/rfc822, inline)]

From: Timothy Sample <samplet <at> ngyro.com>
To: bug-guix <at> gnu.org
Subject: Disarchive as a fallback for downloads
Date: Tue, 23 Mar 2021 00:42:12 -0400

Hello,

This patch series adds Disarchive assembly (backed by SWH lookup) as a
fallback for downloads.

To try it, make sure you are running the daemon in an environment with
Disarchive available:

    $ ./pre-inst-env guix environment --ad-hoc guile disarchive
    # ./pre-inst-env guix-daemon --build-users-group=guixbuild

Don’t forget to stop your existing Guix Daemon.  :)

You also need to make sure that regular downloads are unavailable.  I do
this by adjusting the “try” loop at the end of “url-fetch” in
“guix/build/download.scm”.  I replace the usual list of URLs with ‘()’:

    (let try ((uri (append uri content-addressed-uris)))
      (match '() ; uri
        ...))

Now you can ask Guix for a recent .tar.gz source package:

    $ ./pre-inst-env guix build --no-substitutes -S python-httpretty

You should see:

    Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz
    Assembling the directory httpretty-1.0.5
    Downloading from Software Heritage...
    7903d608efc89c14afb4d692a3721156e31a43e2/
    7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/
    7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING
    [...]
    Checking httpretty-1.0.5 digest... ok
    Assembling the tarball httpretty-1.0.5.tar
    Checking httpretty-1.0.5.tar digest... ok
    Assembling the Gzip file httpretty-1.0.5.tar.gz
    Checking httpretty-1.0.5.tar.gz digest... ok
    Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz
    successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpretty-1.0.5.tar.gz.drv

There’s lots to talk about though....

First, it looks up the metadata on my server.  This is fine for a demo,
but not what we want forever.  The patch series supports adding several
mirrors for looking up the metadata.  In the past, we talked about
putting everything on one or a few of the big Git hosting platforms like
GitHub or Gitlab.  That way, it would be easily picked up by SWH and
archived “forever”.  Right now, I have Cuirass set up to build the
metadata, and a little script that moves it from the build server to my
Web server.  It would be simple enough to adjust that script to push it
to a remote Git repo.  (Of course, the next step is to move this setup
to Guix infrastructure.)  Thoughts?

On the code level, there were two things I couldn’t figure out for
myself.

I made the mirror list just simple strings.  AIUI, the client and the
daemon have to agree about the format of the mirror list.  Given that
running old daemons is common, changing the format is difficult.  Is it
worth it to copy the more flexible interface used by the content
addressed mirrors?  If yes, do I have to do the same ‘module-autoload!’
dance to use ‘bytevector->base16-string’?  :)  (I probably would have
just copied it, but that part confused me a bit.)

I imported some modules from “guix/build/download.scm” (well, just
“base16” and “swh”).  It feels weird to use a bunch of host-side modules
from what’s nominally a “guix/build” module.  This is okay because
“guix/build/download.scm” is not /really/ build-side code.  It’s more
like daemon (-ish) code that just happens to live in “guix/build”, which
is why importing host-side modules is OK... right?

Hopefully everything else is more-or-less fine.  :)


-- Tim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.