Subject: bug#42162: Recovering source tarballs

From: Christopher Baines <[email-masked]>
Date: Mon, 13 Jul 2020 20:20:27 +0100

Ludovic Courtès <[email-masked]> writes:

> Hi,
>
> Ludovic Courtès <[email-masked]> skribis:
>
>> There’s this other discussion you mentioned, which I hope will have a
>> positive outcome:
>>
>>   https://forge.softwareheritage.org/T2430
>
> This discussion as well as discussions on #swh-devel have made it clear
> that SWH will not archive raw tarballs, at least not in the foreseeable
> future.  Instead, it will keep archiving the contents of tarballs, as it
> has always done—that’s already a huge service.
>
> Not storing raw tarballs makes sense from an engineering perspective,
> but it does mean that we cannot rely on SWH as a content-addressed
> mirror for tarballs.  (In fact, some raw tarballs are available on SWH,
> but that’s mostly “by chance”, for instance because they appear as-is in
> a Git repo that was ingested.)  In fact this is one of the challenges
> mentioned in
> <https://guix.gnu.org/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive/>.
>
> So we need a solution for now (and quite urgently), and a solution for
> the future.
>
> For the now, since 70% of our packages use ‘url-fetch’, we need to be
> able to fetch or to reconstruct tarballs.  There’s no way around it.
>
> In the short term, we should arrange so that the build farm keeps GC
> roots on source tarballs for an indefinite amount of time.  Cuirass
> jobset?  Mcron job to preserve GC roots?  Ideas?

Going forward, being methodical as a project about storing the tarballs
and source material for the packages is probalby the way to ensure it's
available for the future. I'm not sure the data storage cost is
significant, the cost of doing this is probably in working out what to
store, doing so in a redundant manor, and making the data available.

The Guix Data Service knows about fixed output derivations, so it might
be possible to backfill such a store by just attempting to build those
derivations. It might also be possible to use the Guix Data Service to
work out what's available, and what tarballs are missing.

Chris