GNU bug report logs - #69058
'vault-fetch' does not follow redirects, preventing downloads from SWH

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 69058 in the body.
You can then email your comments to 69058 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: Failure when fallback to Disarchive+SWH
Date: Sun, 11 Feb 2024 18:21:29 +0100

Hi,

Here an example where something is wrong:

 + Diarsarchive specificiation is ok.
   <https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152>
 + SWH has it
   <https://archive.softwareheritage.org/browse/directory/35e81cfbd7ca261e2ddebf372f38a4d1baa2f931>

But something is failing on Guix side.

--8<---------------cut here---------------start------------->8---
$ guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv
/gnu/store/58m53a8ygffj4l5kpqhyxby1ylhri9dr-xscreensaver-6.04.tar.gz

$ guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv --check
The following derivation will be built:
  /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv
building /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv...

Starting download of /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz

[...]

Trying to use Disarchive to assemble /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz...
Retrieving Disarchive spec from https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152 ...
Assembling the directory xscreensaver-6.04
Downloading /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz from Software Heritage...
SWH: object swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 could not be fetched from the vault
Could not resolve directory reference
failed to download "/gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz" from "https://www.jwz.org/xscreensaver/xscreensaver-6.04.tar.gz"
builder for `/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv' failed to produce output path `/gnu/store/58m53a8ygffj4l5kpqhyxby1ylhri9dr-xscreensaver-6.04.tar.gz'
build of /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv failed
View build log at '/var/log/guix/drvs/mx/sfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv.gz'.
guix build: error: build of `/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv' failed
--8<---------------cut here---------------end--------------->8---


Let’s do it manually:

 1. fetch content and specification
 2. assemble
 3. compare the assembled with the one from content-addressed-mirrors


1.

--8<---------------cut here---------------start------------->8---
$ wget -O /tmp/spec https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152
/tmp/spec               [    <=>             ]   3,25M   512KB/s    in 6,6s    

$ wget -O /tmp/content https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931/raw
/tmp/content        100%[===================>]  24,86M   530KB/s    in 49s     
--8<---------------cut here---------------end--------------->8---


2.

--8<---------------cut here---------------start------------->8---
$ tar xf content
$ disarchive assemble swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 spec > xscreensaver-6.04.tar.gz
Assembling the directory xscreensaver-6.04
Checking xscreensaver-6.04 digest... ok
Assembling the tarball xscreensaver-6.04.tar
Checking xscreensaver-6.04.tar digest... ok
Assembling the Gzip file xscreensaver-6.04.tar.gz
Checking xscreensaver-6.04.tar.gz digest... ok
Writing result to output port
--8<---------------cut here---------------end--------------->8---


3.

--8<---------------cut here---------------start------------->8---
$ disarchive assemble swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 spec 2>/dev/null | guix hash -
0lmiyvp3qs2gngd53f191jmlizs9l04i2gnrqbn96mqckyr18w3q

$ guix hash $(guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv)
0lmiyvp3qs2gngd53f191jmlizs9l04i2gnrqbn96mqckyr18w3q
--8<---------------cut here---------------end--------------->8---

Cheers,
simon

Message #8 received at 69058 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: 69058 <at> debbugs.gnu.org
Subject: Re: bug#69058: Failure when fallback to Disarchive+SWH
Date: Sun, 11 Feb 2024 19:11:02 +0100

Hi,

Another example using Git.  It is from v1.1.0.

--8<---------------cut here---------------start------------->8---
$ guix time-machine -q --commit=v1.1.0 -- build -S redkite
guile: warning: failed to install locale
The following derivation will be built:
   /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv
building /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin:/gnu/store/jh17p4sns7dvbizwz58gdh953qpic144-tar-1.32/bin'
Initialized empty Git repository in /gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout/.git/
fatal: could not read Username for 'https://gitlab.com': No such device or address
Failed to do a shallow fetch; retrying a full fetch...
fatal: could not read Username for 'https://gitlab.com': No such device or address
git-fetch: '/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
SWH: directory d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
builder for `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed to produce output path `/gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout'
build of /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv failed
View build log at '/var/log/guix/drvs/x4/hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv.gz'.
guix build: error: build of `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed
--8<---------------cut here---------------end--------------->8---


From its definition the package reads:

--8<---------------cut here---------------start------------->8---
(define-public redkite
[...]
       (sha256
        (base32
         "1747w1kg8y9jbl11xi018d85dm38xk7843pz26sh0k5fdv87a10q"))))
--8<---------------cut here---------------end--------------->8---

Then if I manually download it:

--8<---------------cut here---------------start------------->8---
$ wget -O /tmp/content https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728/raw

[...]
/tmp/content        100%[===================>] 129,38K   555KB/s    in 0,2s    
2024-02-11 19:00:33 (555 KB/s) - ‘/tmp/content’ saved [132488/132488]

$ tar xf content
--8<---------------cut here---------------end--------------->8---

And we get the expected value:

--8<---------------cut here---------------start------------->8---
$ guix hash -rx swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728
1747w1kg8y9jbl11xi018d85dm38xk7843pz26sh0k5fdv87a10q
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

Message #11 received at 69058 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: 69058 <at> debbugs.gnu.org
Subject: Re: bug#69058: Failure when fallback to Disarchive+SWH
Date: Mon, 12 Feb 2024 11:28:13 +0100

Hi,

Simon Tournier <zimon.toutoune <at> gmail.com> skribis:

> Another example using Git.  It is from v1.1.0.
>
> $ guix time-machine -q --commit=v1.1.0 -- build -S redkite
> guile: warning: failed to install locale
> The following derivation will be built:
>    /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv
> building /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to `/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin:/gnu/store/jh17p4sns7dvbizwz58gdh953qpic144-tar-1.32/bin'
> Initialized empty Git repository in /gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout/.git/
> fatal: could not read Username for 'https://gitlab.com': No such device or address
> Failed to do a shallow fetch; retrying a full fetch...
> fatal: could not read Username for 'https://gitlab.com': No such device or address
> git-fetch: '/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch origin' failed with exit code 128
> Trying content-addressed mirror at berlin.guixsd.org...
> Trying content-addressed mirror at berlin.guixsd.org...
> Trying to download from Software Heritage...
> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
> SWH: directory d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
> builder for `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed to produce output path `/gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout'
> build of /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv failed
> View build log at '/var/log/guix/drvs/x4/hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv.gz'.
> guix build: error: build of `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed

I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
$5 = #f
--8<---------------cut here---------------end--------------->8---

The bug is that the Vault now returns 302 (redirect) in this case, which
is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.

So I’ll have to push that patch series and add a patch for ‘vault-fetch’
on top of it.

Ludo’.

Message #18 received at 69058-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: 69058-done <at> debbugs.gnu.org
Subject: Re: bug#69058: Failure when fallback to Disarchive+SWH
Date: Mon, 12 Feb 2024 12:24:44 +0100

Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:

> I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:
>
> scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
> SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
> $5 = #f
>
> The bug is that the Vault now returns 302 (redirect) in this case, which
> is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.

In fact it’s the exact same problem.

Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.

Thanks,
Ludo’.

Message #21 received at 69058-done <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 69058-done <at> debbugs.gnu.org
Subject: Re: bug#69058: Failure when fallback to Disarchive+SWH
Date: Mon, 12 Feb 2024 13:59:27 +0100

Hi Ludo,

On lun., 12 févr. 2024 at 12:24, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
> Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:
>
>> I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:
>>
>> scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
>> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
>> SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
>> $5 = #f
>>
>> The bug is that the Vault now returns 302 (redirect) in this case, which
>> is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.
>
> In fact it’s the exact same problem.
>
> Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.

Cool!  Thanks.

However, it means that it stays broken for all commits before this fix,
right?  Somehow, if only SWH is available, then

    guix time-machine -C channels.scm -- shell -m manifest.scm

will fail for all channels older than today.  Right?

For instance, redoing your ReScience paper from 2020 using only SWH as I
did [1] some months ago is now broken (because a change on SWH side).

Well, I am fine to keep this bug closed but for the posterity we need
something like:

    guix swh get -C channels.scm -m manifest.scm

as we discussed elsewhere (off-list IIRC :-)).

1: https://simon.tournier.info/posts/2023-06-23-hackathon-repro.html

Cheers,
simon

Message #24 received at 69058-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: 69058-done <at> debbugs.gnu.org
Subject: Re: bug#69058: Failure when fallback to Disarchive+SWH
Date: Mon, 12 Feb 2024 14:15:00 +0100

Hi,

Simon Tournier <zimon.toutoune <at> gmail.com> skribis:

>> Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.
>
> Cool!  Thanks.
>
> However, it means that it stays broken for all commits before this fix,
> right?  Somehow, if only SWH is available, then
>
>     guix time-machine -C channels.scm -- shell -m manifest.scm
>
> will fail for all channels older than today.  Right?

Mostly, yes (“mostly” because derivations using “builtin:git-download”
will automatically be fixed once we’ve updated the ‘guix’ package, but
that only helps if you’re traveling to after September 2023 (commit
13b0cf85eb31e1b1ea674debbbfb0f35a32d1374).

> For instance, redoing your ReScience paper from 2020 using only SWH as I
> did [1] some months ago is now broken (because a change on SWH side).

Indeed.  :-/

That’s a real bummer, but that’s the nature of software: we cannot
retroactively fix bugs.

> Well, I am fine to keep this bug closed but for the posterity we need
> something like:
>
>     guix swh get -C channels.scm -m manifest.scm

Exactly!  We have the demonstration of just how important that is.

Ludo’.

Message #27 received at 69058-done <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 69058-done <at> debbugs.gnu.org
Subject: Re: bug#69058: Failure when fallback to Disarchive+SWH
Date: Mon, 12 Feb 2024 15:04:14 +0100

> That’s a real bummer, but that’s the nature of software: we cannot
> retroactively fix bugs.

The nature of service and not software appears to me more correct. :-)

(Here it is not the retroactive fix of a bug but the introduction of a
non-backward compatible change.)

Cheers,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.