Subject: Re: bug#47336: Disarchive as a fallback for downloads

From: Ludovic Courtès <[email-masked]>
Date: Sat, 27 Mar 2021 11:39:18 +0100

Hi!

Timothy Sample <[email-masked]> skribis:

> With “gzip -9”, the average metadata file is 6.8KiB.  It’s pretty
> manageable.  There’s room for improvement on the Disarchive side, too.
> It still stores some redundant information.  Uncompressed, it’s more
> like 112KiB per file.  This is still pretty okay, really.  It means we
> might hit tens of GiB over a couple years.  (It would take just over
> 100GiB to store a million uncompressed metadata files.)  The compression
> ratio is what drove me to skip Git for now.

If needed, the sexp serialization could still be made more compact:
using ‘write’ instead of ‘pretty-print’, shortening field names (but
that’d be incompatible).

We could also use CBOR or canonical sexp serialization, though maybe
gzipped sexps are more compact than what we could achieve?

Anyway, these are surface syntax optimizations that can always be made
at a later point in time when we feel a need for them.

Ludo’.