Subject: [bug#59078] [PATCH] lint: Split the derivation lint checker by system.

From: Christopher Baines <[email-masked]>
Date: Sun, 13 Nov 2022 17:27:48 +0000

Ludovic Courtès <[email-masked]> writes:

> Christopher Baines <[email-masked]> skribis:
>
>> Currently, if you attempt to run the derivation checker on all packages, the
>> Guile process will run out of memory. I think a contributing factor to this is
>> that the checker forces an inefficient order when you want to generate
>> derivations for all the supported systems of each package, constantly
>> switching system then package.
>>
>> This problem also impacts the Guix Data Service, since it tries to run the
>> derivation checker for all packages.
>>
>> The changes in this commit to split the derivation lint checker in to several,
>> one for each system, means that you can now treat each system separately,
>> which should be better for caching purposes.
>>
>> If it's desirable to keep some notion of checking all supported systems for a
>> single package, I think lint checker groups could be added, so that you could
>> ask for the "derivation" checker, and this would run all the derivation
>> checkers.
>
> The ‘derivation’ checker was added for this purpose: making sure that a
> package’s derivation can be computed for all the supported systems.
> Previously it was easy to overlook that kind of breakage.
>
> I think it’s important to keep a ‘derivation’ checker that does this.

What aspect of it do you think is important?

I realise just splitting the checker in to several doesn't quite pick up
on all the same problems, but an additional checker could be added that
flags when packages support systems that don't appear as platforms. I
think that would catch everything that the derivation checker currently
does.

If it's a "we need a singular checker" kind of problem, I was thinking
of having some kind of lint checker groups. So you could have a group
called "derivation", which runs all of the relevant checkers.

> Now, the memory consumption you report is unacceptable and this needs to
> be addressed.
>
> Most (all?) caches are now per-session (associated with
> <store-connection>).  Since ‘guix lint’ uses a single session, those
> caches keep growing because there’s no eviction mechanism in place.
>
> A hack like the one below should work around that.  Could you check how
> well it works for you?

I tried in the Guix Data Service processing packages in chunks of 1000
plus closing the store connection after each batch, and that led to a
heap size of 3090MiB. But this is still higher than 1778MiB heap usage I
got just by splitting the derivation linter.

Fundamentally, I think it's still going to be more memory/time efficient
to move towards processing the derivations by system, rather than by
package.

Thanks,

Chris