From unknown Sat Jun 21 04:59:42 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#77390 <77390@debbugs.gnu.org> To: bug#77390 <77390@debbugs.gnu.org> Subject: Status: [Shepherd] Timer spawns many commands on DST change (CET -> CEST) Reply-To: bug#77390 <77390@debbugs.gnu.org> Date: Sat, 21 Jun 2025 11:59:42 +0000 retitle 77390 [Shepherd] Timer spawns many commands on DST change (CET -> C= EST) reassign 77390 guix submitter 77390 "Timo Wilken" severity 77390 important thanks From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 30 16:04:49 2025 Received: (at submit) by debbugs.gnu.org; 30 Mar 2025 20:04:49 +0000 Received: from localhost ([127.0.0.1]:38175 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tyytj-00078e-Us for submit@debbugs.gnu.org; Sun, 30 Mar 2025 16:04:48 -0400 Received: from lists.gnu.org ([2001:470:142::17]:52410) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tyyth-000783-EZ for submit@debbugs.gnu.org; Sun, 30 Mar 2025 16:04:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tyytR-0006ZC-2j for bug-guix@gnu.org; Sun, 30 Mar 2025 16:04:31 -0400 Received: from mx2.mythic-beasts.com ([2a00:1098:0:82:1000:0:2:1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tyytN-00021E-1I for bug-guix@gnu.org; Sun, 30 Mar 2025 16:04:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=twilken.net ; s=mythic-beasts-k1; h=To:From:Subject:Date; bh=uotxTyu6NZJZ61+GBfSqAGv1Akk1RbYpaC1geEiZozc=; b=UO75YAqP10Nz5Lpyd2LqRM1TAd XT7GqU/trfMbHI7J7j45eRGyLkrxPqTG5N77UG6ZRZesvk1usa4U01pG72iOjErVwGNI4Bb6RKo5Y qJx8REbLjYV7SvhtAY0iMdieckQYMe20b0UD4e/MuVTyytRQCY4yTXHT0OTrsylWHkI9JLXBuCO3x SP3yBn6E5YnIqd0TGTeDfvBYv0F7Ggsb7SyYUrRqv5kIFQFZnXX76oW98cO6IdyM++iNcZE2J/0QR oI7aT59QRIJCpflTYKgoxwxj4hbp+L/MF0BCiuLyUz3TDRdHCrJf474bM9tTkv5H1WotwNv/faydQ nfc3Uvkg==; Received: by mailhub-hex-d.mythic-beasts.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tyytK-008UvH-Sv for bug-guix@gnu.org; Sun, 30 Mar 2025 21:04:23 +0100 Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Sun, 30 Mar 2025 22:04:07 +0200 Message-Id: Subject: Shepherd: bug in timer scheduling on daylight savings time change From: "Timo Wilken" To: X-Mailer: aerc 0.20.1 X-BlackCat-Spam-Score: 7 X-Spam-Status: No, score=0.7 Received-SPF: pass client-ip=2a00:1098:0:82:1000:0:2:1; envelope-from=timo@twilken.net; helo=mx2.mythic-beasts.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi, Last night, my server's time changed from CET to CEST automatically, as it = is set to Europe/Berlin time. I have a Shepherd timer set to run every night at 02:00 local time: https://cgit.twilken.net/dotfiles/tree/tw/system.scm?h=3Dcd39f5d3340c0bcd59= bb80d3ff4f12741db75e07#n136 --8<---------------cut here (simple-service 'disk-maintenance shepherd-root-service-type (list (shepherd-service (provision '(guix-gc)) (requirement '(user-processes file-systems)) (documentation "Delete unused files from Guix' store.") (modules '((shepherd service timer))) (start #~(make-timer-constructor (calendar-event #:hours '(2) #:minutes '(0)) (command (list "guix" "gc" "-d" "2w")))) (stop #~(make-timer-destructor)) (actions (list (shepherd-action (name 'trigger) (procedure #~trigger-timer) (documentation "Collect garbage now."))))))) --8<---------------cut here Last night, this hit the exact moment of the CET/CEST change and seems to h= ave confused the Shepherd a lot, to the point where it kept spamming the same message to /var/log/messages, and used up 80% of my system's total memory, according to htop. (This led to various things getting oom-killed, and the system grinding to a halt until I had a look at it in the afternoon.) First, at 01:00, the Shepherd spawned about 30 processes for this one timer= : --8<---------------cut here 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1691. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'nextcloud-cron' spawned p= rocess 1692. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for nextc= loud-cron. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'gpsplot' spawned process = 1693. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for gpspl= ot. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'dynamic-dns-ipv4.api.myth= ic-beasts.com' spawned process 1694. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for dynam= ic-dns-ipv4.api.mythic-beasts.com. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1698. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1703. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1713. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1715. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1723. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1726. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1732. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1736. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1740. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1750. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1754. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1761. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1766. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1768. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1778. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1785. 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1791. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1806. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1811. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1818. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1822. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1829. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1838. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1848. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1851. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1862. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1872. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1876. 2025-03-30 01:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 01:00:02 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1883. 2025-03-30 01:00:02 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 03:00:00 localhost linux: [46479.948177] perf: interrupt took to= o long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 79750 2025-03-30 03:00:01 localhost linux: [47490.137060] perf: interrupt took to= o long (3138 > 3130), lowering kernel.perf_event_max_sample_rate to 63500 2025-03-30 03:00:01 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1887. 2025-03-30 03:00:01 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 03:00:02 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1896. 2025-03-30 03:00:02 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 03:00:03 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1910. 2025-03-30 03:00:03 localhost shepherd[1]: Registering new logger for guix-= gc. 2025-03-30 03:00:03 localhost shepherd[1]: Timer 'guix-gc' spawned process = 1915. --8<---------------cut here I then get 731(!!) lines like the following over the course of 3 seconds: --8<---------------cut here 2025-03-30 03:00:03 localhost shepherd[1]: Waiting anew for timer 'guix-gc'= (resuming from sleep state?). [...] 2025-03-30 03:00:06 localhost shepherd[1]: Waiting anew for timer 'guix-gc'= (resuming from sleep state?). --8<---------------cut here The Shepherd then gets more confused and starts spamming the same message o= ver and over again, but now without newlines, leading to lines like: --8<---------------cut here 2025-03-30 03:00:06 localhost shepherd[1]: Waiting anew for timer 'guix-gc'= (resuming from sleep state?).Waiting anew for timer 'guix-gc' (resuming fr= om sleep state?).Waiting anew for timer 'guix-gc' (resuming from sleep stat= e?).Waiting anew for timer 'guix-gc' (resuming from sleep state?).Waiting a= new for timer 'guix-gc' (resuming from sleep state?).Waiting anew for timer= 'guix-gc' (resuming from sleep state?).Waiting anew for timer 'guix-gc' (r= esuming from sleep state?).Waiting anew for timer 'guix-gc' (resuming from = sleep state?).Waiting anew for 2025-03-30 03:00:06 localhost shepherd[1]: timer 'guix-gc' (resuming from = sleep state?).Waiting anew for timer 'guix-gc' (resuming from sleep state?)= .Waiting anew for timer 'guix-gc' (resuming from sleep state?).Waiting anew= for timer 'guix-gc' (resuming from sleep state?).Waiting anew for timer 'g= uix-gc' (resuming from sleep state?).Waiting anew for timer 'guix-gc' (resu= ming from sleep state?).Waiting anew for timer 'guix-gc' (resuming from sle= ep state?).Waiting anew for timer 'guix-gc' (resuming from sleep state?).Wa= iting anew for timer 'guix-gc' --8<---------------cut here Then there's some output from the various "guix gc" processes spawned by th= e timer, which of course all step on each other's toes. The Shepherd keeps happily writing the above lines out in large blocks. It goes on like this until the afternoon, when the OOM kills start (probabl= y when I start SSH'ing in to see what's going on): --8<---------------cut here 2025-03-30 15:21:44 localhost linux: [93869.155828] sshd-session invoked oo= m-killer: gfp_mask=3D0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=3D0, = oom_score_adj=3D0 --8<---------------cut here After a bit of that, I decide to reboot the machine. I've saved the system logs from that night - let me know if you need more d= etails. I'm running Shepherd 1.0.3, as packaged in Guix: --8<---------------cut here Generation 235 Mar 29 2025 12:07:15 (current) file name: /var/guix/profiles/system-235-link canonical file name: /gnu/store/p7infvmm678nysvdg3xvxlr83xxkj66j-system label: GNU with Linux-Libre 6.13.8 bootloader: grub-efi root device: UUID: c63af3e6-3c2b-43d2-b1e6-944f09a10e0f kernel: /gnu/store/d7z9027yrs1vfmpvds10iqfl69cyvixn-linux-libre-6.13.8/bz= Image channels: tw: repository URL: https://git.twilken.net/dotfiles branch: master commit: 39abf96d68ce500abbd55f9bdbd68e1c63282d36 guix: repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: 67f4a2feadbb6cc04a61b5a3bc7b8f4bace2a0bb [non-free channels omitted] --8<---------------cut here Cheers, Timo From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 31 09:10:14 2025 Received: (at control) by debbugs.gnu.org; 31 Mar 2025 13:10:14 +0000 Received: from localhost ([127.0.0.1]:40311 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tzEu5-0002G0-Jz for submit@debbugs.gnu.org; Mon, 31 Mar 2025 09:10:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40272) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tzEu3-0002Cr-1S for control@debbugs.gnu.org; Mon, 31 Mar 2025 09:10:11 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzEtw-0003J3-Ad for control@debbugs.gnu.org; Mon, 31 Mar 2025 09:10:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to: references; bh=uIs8M3B4Pe7/y98hFk8Bn+1Q0jLwve+PhB/UFjnKQfY=; b=YlHXb2qncuPsyH 6QRtBCDLCMzdiCV29/3yYUW7UIE4KscO8NG4Eb/mt+nANccM6Ce/aIX+ukKtuHnaHuDRRnk8JtC00 G/ES1REt0RqToRD9q3Z8huVeh2jzTaKIRWs54pMac3YBah6q356X0tGja7jjSrGauh2Dk2nxDBXt0 ynKiezwnsKiiUZcMsW6saGXG869O8D4vaUH3hXQE4fErqRKi8ZEzGoobAqY2NQWRCEQCXHw/MS0Ks scxzEE/ihsFsm26ZlamJXJcgFlNUsRW+8mRRkrJpxTPClSJVyRCDQekhUWpFAoGjLq1Ry9sKarVPk I7kX50HyoJbXQyjzfi7Q==; Date: Mon, 31 Mar 2025 15:09:38 +0200 Message-Id: <8734etmjn1.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #77390 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) merge 77390 77401 quit From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 31 09:17:17 2025 Received: (at 77390) by debbugs.gnu.org; 31 Mar 2025 13:17:17 +0000 Received: from localhost ([127.0.0.1]:40347 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tzF0u-0005sz-7f for submit@debbugs.gnu.org; Mon, 31 Mar 2025 09:17:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44394) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tzF0p-0005l1-0m for 77390@debbugs.gnu.org; Mon, 31 Mar 2025 09:17:13 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzF0i-0004fw-TF; Mon, 31 Mar 2025 09:17:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=k6iuGWCMNCWxWGVItwVxIa1IK+XgNQKjEHkCdqMNvLU=; b=aBV/vPiBomi32g2Ni8ZZ +7LQgti0T2LOB9+ZmWMcG8TLxvtzhS/6gRkyu3nKi6MtL2mMd9/MJanOVBe4laPQravEt83PzsgHN MQkRIRXbPWDVhte4Dm0iYt2A7T2hjPusns7BQhsFsc+iDCfjNOMfOykCRYUphJ1H3rna05CQUNeBD 2K+YJ+m/ezmWb/WbrF856EMwBkAWmoB5CecVU1IEZ7LSpRVbd9Au1Sj+FCMJ/6CTrnuV8Q6tPGlpi hhvk3Tg0P81QOzmVXrh9LLWT29br8vREVWk6XjM8gy4o0zZZZBCqyVwbECh52L4sk1jYtV3D+hR5v DSGav3JAVbPRwQ==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: "Timo Wilken" Subject: Re: bug#77390: Shepherd: bug in timer scheduling on daylight savings time change In-Reply-To: (Timo Wilken's message of "Sun, 30 Mar 2025 22:04:07 +0200") References: Date: Mon, 31 Mar 2025 15:17:00 +0200 Message-ID: <87semtl4qb.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 77390 Cc: 77390@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Timo, "Timo Wilken" skribis: > Last night, this hit the exact moment of the CET/CEST change and seems to= have > confused the Shepherd a lot, to the point where it kept spamming the same > message to /var/log/messages, and used up 80% of my system's total memory, > according to htop. (This led to various things getting oom-killed, and the > system grinding to a halt until I had a look at it in the afternoon.) > > First, at 01:00, the Shepherd spawned about 30 processes for this one tim= er: > > --8<---------------cut here > 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned proces= s 1691. > 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'nextcloud-cron' spawned= process 1692. > 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for nex= tcloud-cron. > 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'gpsplot' spawned proces= s 1693. > 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for gps= plot. > 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'dynamic-dns-ipv4.api.my= thic-beasts.com' spawned process 1694. > 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for dyn= amic-dns-ipv4.api.mythic-beasts.com. > 2025-03-30 01:00:00 localhost shepherd[1]: Timer 'guix-gc' spawned proces= s 1698. > 2025-03-30 01:00:00 localhost shepherd[1]: Registering new logger for gui= x-gc. I saw the exact same problem on berlin (aka. ci.guix.gnu.org): https://issues.guix.gnu.org/77401 but not on the other servers I co-administer; I suspect the difference is that these other servers did not have a timer running at 01:00. So I guess that the fix for was bogus somehow? To be continued=E2=80=A6 Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 31 09:18:05 2025 Received: (at control) by debbugs.gnu.org; 31 Mar 2025 13:18:05 +0000 Received: from localhost ([127.0.0.1]:40355 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tzF1g-0006UR-3r for submit@debbugs.gnu.org; Mon, 31 Mar 2025 09:18:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41594) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tzF1L-00068j-NM for control@debbugs.gnu.org; Mon, 31 Mar 2025 09:17:44 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzF1F-0004le-Qr for control@debbugs.gnu.org; Mon, 31 Mar 2025 09:17:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to: references; bh=1zbc2wG5AFWXO8eq9So/SrnPDnJ4Z7TpF604FQZmPbI=; b=HyCPv7BfNWQ+pV lgDnZszSWBMs6Qi61HiycGp0HoMki9chLsB+iW5dLjFyfEzE5HDvMMqD783Jxre1uBi/JtglqoYNv CImQgkOTc/cvIZFNcjrAnAUfsW1x+f0b/caMWKmNckL8sEa2TRERyVfMx1TxG4yHpvw+cBrRpYGgK XvBOY9H1/O7LCUwhL8w2pODnmonlUMJIwmMPRrJtnXQuPuvIr5sHJkAgQo/JWa1+i3Anc0XvQv/UU 7Q7h0q6+nBX+y+ic9jALjjn+5meRP2znvRtdrIyOitooTRWZs1OitOBoHWS7r+ErHzvnJthPMuD/d x/k6vlDK8g+nzCbKhGUA==; Date: Mon, 31 Mar 2025 15:17:29 +0200 Message-Id: <87r02dl4pi.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #77401 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) severity 77401 important quit From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 31 16:29:03 2025 Received: (at control) by debbugs.gnu.org; 31 Mar 2025 20:29:03 +0000 Received: from localhost ([127.0.0.1]:43091 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tzLkk-0000Sz-R6 for submit@debbugs.gnu.org; Mon, 31 Mar 2025 16:29:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48698) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tzLki-0000Rt-Be for control@debbugs.gnu.org; Mon, 31 Mar 2025 16:29:00 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzLkc-00031J-Pf for control@debbugs.gnu.org; Mon, 31 Mar 2025 16:28:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to: references; bh=exCrlfDXhc1MQBEOvZui8/BrQDunupVBEN6UGZZRkvA=; b=iDt7XfXNSr4qT8 0fxExQc5uZVflfu8q6XtTFwGD3KD1VyxzxcISnKXDGT2Rref7i0RouR5iqXjQzywE426PM8w+j/7w bLTfliPEnv0Uzrz88tD5QZuwx6E4JRZubNShxw+p/5Ue47cDvQewI5IkLJeBbFK/SjSMSvWgr0Qgy dniIQhFddjcjBBLafrsiUV9Kgl77n2Du4dZrPVOKV/bR3CMNFtqHu2MbdK5X7SmgZV+gqh5cmve3a DrvhIWV10PgQHFWTDuK8eWDKwN+pE1mAN3Gp+s5YJSPH9CotFne6N+3QVq62MOokbW6IEX/ONJhKO MzWYkOEPZ+Jf5JIhZaXA==; Date: Mon, 31 Mar 2025 22:28:51 +0200 Message-Id: <87jz85j664.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #77401 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) retitle 77401 [Shepherd] Timer spawns many commands on DST change (CET -> CEST) quit From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 31 17:22:20 2025 Received: (at 77390-done) by debbugs.gnu.org; 31 Mar 2025 21:22:21 +0000 Received: from localhost ([127.0.0.1]:43183 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tzMaK-0003eV-5L for submit@debbugs.gnu.org; Mon, 31 Mar 2025 17:22:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40938) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tzMa6-0003cX-N5; Mon, 31 Mar 2025 17:22:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzMZz-0000dz-S1; Mon, 31 Mar 2025 17:22:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=dpYDEiik8WSvE38EXsvbT3jwgmG6XE0PzwD09iRErws=; b=QioeIDajMyILnkXkAUBY fXNbbbY+HTdLtwFDYGLeMT6YKUXeY0cJ5keggbEiYkj5z+ne5B6PDV5Tnb68PFtw64JwCh57pBfbX vKAm9jOIvv9fW4wSAJXomfu+PenKMkY2rxQkUpGUc9OolEl7C9DoGCw8v0gMdospmPal6EmMxchX0 flAkFtAagQBpJcLoNhnG9yB66e6u5gl0QlisYDR4QxCEwEZ1Vwfdj002TEw+/vDN8EdXZljw52J7S A8lMghfT9txXo1Ax0EWTUoC8AuYX/2QRSMCcb5tmqgBHI7l4ajG6O1n14hfBalov0CSdyWezVf3fZ kiJpM2QeY504aw==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: "Timo Wilken" Subject: Re: bug#77401: [Shepherd] Timer spawns too many commands In-Reply-To: (Timo Wilken's message of "Sun, 30 Mar 2025 22:04:07 +0200") References: Date: Mon, 31 Mar 2025 23:21:56 +0200 Message-ID: <87o6xgj3pn.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 77390-done Cc: 77401-done@debbugs.gnu.org, 77390-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Timo, "Timo Wilken" skribis: > Last night, this hit the exact moment of the CET/CEST change and seems to= have > confused the Shepherd a lot, to the point where it kept spamming the same > message to /var/log/messages, and used up 80% of my system's total memory, > according to htop. (This led to various things getting oom-killed, and the > system grinding to a halt until I had a look at it in the afternoon.) > > First, at 01:00, the Shepherd spawned about 30 processes for this one tim= er: This is now fixed: https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=3Db36e97a730596= dbf3c376f940150d5fd1f08ecf1 Let me know if you find something fishy. The irony is that the work I did for helped, but the one test that would have caught this issue was wrong: it was looking at the wrong date when testing the CET->CEST change. Setting #:wait-for-termination? #true would have mitigated this bug (it wouldn=E2=80=99t have triggered). Thanks, Ludo=E2=80=99. From unknown Sat Jun 21 04:59:42 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 29 Apr 2025 11:24:13 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator