Subject: Bioconductor tarballs are not archived

From: Ludovic Courtès <[email-masked]>
Date: Fri, 22 Dec 2023 14:40:01 +0100

Hello!

zimoun <[email-masked]> skribis:

> Since 2020, I provided several examples of breakage with bug#39885 [1].
> Here another one:
>
> $ guix time-machine --commit=77e2de365497bf4c8b81cbd78624f78293490485 \
>        -- build r-biocneighbors -S

[...]

> Starting download of /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz
>>>From https://web.archive.org/web/20220718175152/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz...
> download failed "https://web.archive.org/web/20220718175152/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz" 404 "NOT FOUND"
> Trying to use Disarchive to assemble /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz...
> could not find its Disarchive specification
> failed to download "/gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz" from ("https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz" "https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/BiocNeighbors_1.4.1.tar.gz")
> builder for `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' failed to produce output path `/gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz'
> build of /gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv failed
> View build log at '/var/log/guix/drvs/q9/ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv.gz'.
> guix build: error: build of `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' failed
>
> Well, several comments:
>
>  1. Berlin or Bordeaux do not have it as substitutes,
>  2. Diasarchive does not have it,
>  3. Many others neither.

I was wondering whether we’re now doing better for Bioconductor
tarballs.  The answer, based on small sample, seems to be “not quite”:

--8<---------------cut here---------------start------------->8---
$ guix lint -c archival $(guix package -A ^r-bioc | cut -f1)
gnu/packages/bioconductor.scm:19708:12: [email-masked]: Disarchive entry refers to non-existent SWH directory '726af85395d163b5a21e52e4df1bf18aa0072f6b'
gnu/packages/bioconductor.scm:19752:12: [email-masked]: Disarchive entry refers to non-existent SWH directory '12cfedcbc27005a3fb7e01c5c4b727e0116f596f'
gnu/packages/bioconductor.scm:16892:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '6a2d6d909a7cedd56e96f5a98770deeaaaa8d220'
gnu/packages/bioconductor.scm:4540:12: [email-masked]: Disarchive entry refers to non-existent SWH directory '6f19ea14f46dbc75909b77bc08e9023daae6fb9e'
gnu/packages/bioconductor.scm:19785:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '977ff052b4e6c948af7af0fc14ae61f71427cb1a'
gnu/packages/bioconductor.scm:21524:6: [email-masked]: Disarchive entry refers to non-existent SWH directory '29d8fef9a5b386384f20513c612f1e34f6118532'
gnu/packages/bioconductor.scm:13090:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '6d3728b2dee78cceecdeba0318f3e57b6013d96f'
gnu/packages/bioconductor.scm:19957:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '251081d4bc3f061ef8e16338eb042ad4c71ed02d'
gnu/packages/bioconductor.scm:20003:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '0cc9d3dcde06fb353cdd77f3b538845d16a77720'
gnu/packages/bioconductor.scm:6613:12: [email-masked]: Disarchive entry refers to non-existent SWH directory '41e09414898f61655bcc99fdd44d69b0531c0b2d'
gnu/packages/bioconductor.scm:20030:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '55de8618648ed16797a8effd5b508c652a5d7cbe'
gnu/packages/bioconductor.scm:20144:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '1cfa6cac0cb453f2882a35c8f5ae6ddfa713ad2d'
gnu/packages/bioconductor.scm:13276:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '992d3f9d48633fa5d46b9a7640a825054e9538aa'
gnu/packages/bioconductor.scm:19806:12: [email-masked]: Disarchive entry refers to non-existent SWH directory 'bb17c3bd9ac7c373b24782fcfecdde5fa2f0a965'
gnu/packages/bioconductor.scm:22965:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '3d08f77aae1e81ce9ca9bb9ae2adf4d4c7421d11'
gnu/packages/bioconductor.scm:4521:5: [email-masked]: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/bioconductor.scm:19830:12: [email-masked]: Disarchive entry refers to non-existent SWH directory '47e0877ab988469fc09a37505dd769f9626cac2e'
gnu/packages/bioconductor.scm:20182:5: [email-masked]: Disarchive entry refers to non-existent SWH directory '393f3472cc27f632caea3488aef93a7675b403ef'
$ guix describe
Generation 285  Dec 17 2023 23:31:56    (current)
  guix 6ab2426
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 6ab242609daec00e8bd54f7bff54557c92695724
--8<---------------cut here---------------end--------------->8---

In all cases but one, we’re doing the right thing Disarchive-wise, but
our SWH did not archive them.

<https://guix.gnu.org/sources.json> has entries like:

--8<---------------cut here---------------start------------->8---
    {
      "type": "url",
      "urls": [
        "https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz",
        "https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz",
        "https://bordeaux.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas",
        "https://ci.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas",
        "https://tarballs.nixos.org/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas"
      ],
      "integrity": "sha256-WlUXSI41dP7xSTx81ibFsrIsACW5jmzaX5I/lxJ4vCg=",
      "outputHashAlgo": "sha256",
      "outputHashMode": "flat"
    },
--8<---------------cut here---------------end--------------->8---

Note that we have at least one copy on our infra:

--8<---------------cut here---------------start------------->8---
$ wget -qO- "https://bordeaux.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas"|guix hash  - -f base64
WlUXSI41dP7xSTx81ibFsrIsACW5jmzaX5I/lxJ4vCg=
--8<---------------cut here---------------end--------------->8---

<https://ci.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas>
is 404 (but I can see why: for /file, ‘guix publish’ relies on things
being available in the store and we no longer keep them on ci.guix; we
do have a substitute at
<https://ci.guix.gnu.org/nar/6kfpflffl7b4hx6ibb5k879ar8ffcxb7-BiocNeighbors_1.20.0.tar.gz>
though; we should fix this).

What about hypothesis (2)?  This is what we have:

--8<---------------cut here---------------start------------->8---
$ wget -qO- https://disarchive.guix.gnu.org/sha256//5a5517488e3574fef1493c7cd626c5b2b22c0025b98e6cda5f923f971278bc28 |grep swh
                        (swhid "swh:1:dir:6d3728b2dee78cceecdeba0318f3e57b6013d96f"))
--8<---------------cut here---------------end--------------->8---

I checked with folks on #swh-devel and it turns out that “the legacy
nixguix lister that is still used in production did not detect the
fallback URL as a tarball URL” (the bordeaux.guix.gnu.org URL), but this
is fixed in the new lister, which should be in production “soon”.

As for past tarballs, #swh-devel comrades say we could send them a list
of URLs and they’d create “Save Code Now” requests on our behalf (we
cannot do it ourselves since the site doesn’t accept plain tarballs.)

Any volunteer to write a script that’d generate a list of Bioconductor
content-addressed URLs (the bordeaux.guix.gnu.org/file ones) for say the
past couple of years?

Thanks!

Ludo’.