Subject: Re: [bug#75518] Request for merging "core-packages-team" branch

From: Christopher Baines <[email-masked]>
Date: Fri, 11 Jul 2025 10:13:15 +0100

Andreas Enge <[email-masked]> writes:

> Am Thu, Jul 10, 2025 at 10:49:54PM +0800 schrieb Z572:
>> It seems that there is a problem with the database on ci. Some drvs no longer exist, and all the current eval has failed.
>
> I have seen this, but have no idea what to do about it. Our tooling
> situation is despairingly dire: Only Chris is able to look after QA, and
> only Ludovic (and one or two more persons?) is able to look after CI.
> QA had not evaluated a single commit in almost the last two weeks, whereas
> the main page of ci.guix.gnu.org was almost permanently not reachable over
> the last days with a 504 error; other pages did appear with a bit of
> patience. We urgently need to reach a situation with working tools, and
> with bus factors larger than 1, but I do not know how to do so.
>
> I see that data.qa and bayfront have been reconfigured today, supposedly by
> Chris. The guix-data-service and guix-data-service-process-jobs services
> were marked in bold, so I have restarted them. I think QA started to work
> again before that: There are evaluations visible at
> https://data.qa.guix.gnu.org/ , on master and core-updates.
>
> I have rebased core-packages-team on one of the processed master commits
> from today, added the akonadi and kwin fixes by Z572, and pushed.
> Hopefully data.qa will pick up this commit and start processing the branch
> again. But I suppose it will take a few days until the build farm has
> finished the branch, so it would clearly be desirable to also repair CI.

With QA, things look like they might be starting to work again today,
but it's not been a fun several weeks.

There's been correctness issues with the derivations in the database for
both data.guix.gnu.org and data.qa.guix.gnu.org, and finding, fixing and
trying to figure out how those might have happened took a bunch of time.

There's also been a couple of issues where the symptom was excessive
time doing garbage collection. One in the data service web server, which
turned out to be due to empty revisions I think (as a result of another
bug), and another affecting processing jobs which turned out to be an
issue in Guile Knots (fixed in [1]).

1; https://forge.cbaines.net/cbaines/guile-knots/commit/d18b5b8d5de5beff3b9f84cfb359b73a4dcf2070