Subject: Re: bug#77390: Shepherd: bug in timer scheduling on daylight savings time change

From: Ludovic Courtès <[email-masked]>
Date: Mon, 31 Mar 2025 15:17:00 +0200

Hi Timo,

"Timo Wilken" <[email-masked]> skribis:

> Last night, this hit the exact moment of the CET/CEST change and seems to have
> confused the Shepherd a lot, to the point where it kept spamming the same
> message to /var/log/messages, and used up 80% of my system's total memory,
> according to htop. (This led to various things getting oom-killed, and the
> system grinding to a halt until I had a look at it in the afternoon.)
>
> First, at 01:00, the Shepherd spawned about 30 processes for this one timer:
>
> --8<---------------cut here
> 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process 1691.
> 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'nextcloud-cron' spawned process 1692.
> 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for nextcloud-cron.
> 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'gpsplot' spawned process 1693.
> 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for gpsplot.
> 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'dynamic-dns-ipv4.api.mythic-beasts.com' spawned process 1694.
> 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for dynamic-dns-ipv4.api.mythic-beasts.com.
> 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process 1698.
> 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-gc.

I saw the exact same problem on berlin (aka. ci.guix.gnu.org):

  https://issues.guix.gnu.org/77401

but not on the other servers I co-administer; I suspect the difference
is that these other servers did not have a timer running at 01:00.

So I guess that the fix for <https://issues.guix.gnu.org/75622> was
bogus somehow?

To be continued…

Ludo’.