From debbugs-submit-bounces@debbugs.gnu.org Wed Mar 26 09:42:41 2025 Received: (at submit) by debbugs.gnu.org; 26 Mar 2025 13:42:41 +0000 Received: from localhost ([127.0.0.1]:42129 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1txR1k-0004m2-PR for submit@debbugs.gnu.org; Wed, 26 Mar 2025 09:42:41 -0400 Received: from lists.gnu.org ([2001:470:142::17]:59538) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1txR1i-0004lh-Dp for submit@debbugs.gnu.org; Wed, 26 Mar 2025 09:42:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1txR1V-0001AK-AA for bug-guix@gnu.org; Wed, 26 Mar 2025 09:42:25 -0400 Received: from mail.webarch.email ([81.95.52.48]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1txR1S-0001BA-RL for bug-guix@gnu.org; Wed, 26 Mar 2025 09:42:24 -0400 Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id E7B3F1A851BA for ; Wed, 26 Mar 2025 13:42:08 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=msavoritias.me; s=20210930; t=1742996530; h=from:subject:date:message-id:to:mime-version:content-type: content-transfer-encoding:content-language; bh=F6J1A/+revhMwMgDfVdcfsWBSELTgZNVMx/ruuCivuo=; b=LbEcl/6Ikr32+fJQEM4bkqaBcQWg82TAbi5gb27e1/HeoftVeneUeV5bzmkjpyIQvaJBpG WGQisuTmGZHN3LYTB3lkS2MwDiFFXlyUvPWXD1celzwGgojcVKU8Ose4lMBw2e2fwhkQN3 xNIfwOmGflGTQQPyFOlvpYoizl0G9FXyMO+DpfDfYcUhIZr4PgJpM7lIxuoJVHRiTAUGfp EZu37O2b1IsDHLCH/KWNkSZSaSbIOFL/oe/kh4oLT/3dubdRuQ0IPCofB3hIGKgvK/Lm4j vOyBSqubKhWl4XUGEj0NpqK5oF3Y2PkefA2GEZQIQd6lMdUaJODDXfAeY4gfwQ== Message-ID: Date: Wed, 26 Mar 2025 15:42:06 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: bug-guix@gnu.org From: MSavoritias Subject: Shepherd can't parse logs. Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Last-TLS-Session-Version: TLSv1.3 Received-SPF: pass client-ip=81.95.52.48; envelope-from=email@msavoritias.me; helo=mail.webarch.email X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Shepherd has been crashing our system with the message: https://upload.magicbroccoli.de/fb6e7843fef693538a5c7c156c0a98fc0aeb6bac/KmA7ovtC2Fcw0Nm0PSCtphwSyDLI1VUF3QrT9PtD/Kuvakaappaus_-_2025-03-26_13-27-11.png This ends up with prosody and openssh crashing, nginx still working. the picture shows shepherd system-log crashing with: uncaught exception, value out of range in fibers. It references an "unknown file" and a string with prosody. I looked into git commits in the last month (bug didn't exist before end of January/beginning of February, it seems that the bug is either shepherd itself, it was updated recently and that is what i was told in the irc. Or it is the certbot timer. Certbot timer is the only service we have installed from the ones that changed. Being certbot makes less sense since prosody is mentioned specifically tho so it seems a bug in how shepherd parses logs. Our whole config is here -> https://codeberg.org/joinjabber/Infra MSavoritias From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 30 17:02:26 2025 Received: (at 77283) by debbugs.gnu.org; 30 Mar 2025 21:02:26 +0000 Received: from localhost ([127.0.0.1]:38249 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tyznW-0003LO-5V for submit@debbugs.gnu.org; Sun, 30 Mar 2025 17:02:26 -0400 Received: from hera.aquilenet.fr ([185.233.100.1]:60238) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tyznT-0003Kt-Gq for 77283@debbugs.gnu.org; Sun, 30 Mar 2025 17:02:24 -0400 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 27EC12B6; Sun, 30 Mar 2025 23:02:10 +0200 (CEST) Authentication-Results: hera.aquilenet.fr; none X-Virus-Scanned: Debian amavis at hera.aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavis, port 10024) with ESMTP id qNjXOr_OtY31; Sun, 30 Mar 2025 23:02:09 +0200 (CEST) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 7EA301F3; Sun, 30 Mar 2025 23:02:07 +0200 (CEST) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: MSavoritias Subject: Re: bug#77283: Shepherd can't parse logs. In-Reply-To: (MSavoritias's message of "Wed, 26 Mar 2025 15:42:06 +0200") References: Date: Sun, 30 Mar 2025 23:02:06 +0200 Message-ID: <87bjtidygh.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 27EC12B6 X-Spamd-Result: default: False [5.46 / 15.00]; SPAM_FLAG(5.00)[]; BAYES_HAM(-3.00)[99.99%]; NEURAL_SPAM(3.00)[1.000]; R_MIXED_CHARSET(0.56)[subject]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spam-Level: ***** X-Rspamd-Action: greylist X-Spamd-Bar: +++++ X-Rspamd-Server: hera X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 77283 Cc: 77283@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi MSavoritias, MSavoritias skribis: > Shepherd has been crashing our system with the message: > > https://upload.magicbroccoli.de/fb6e7843fef693538a5c7c156c0a98fc0aeb6bac/= KmA7ovtC2Fcw0Nm0PSCtphwSyDLI1VUF3QrT9PtD/Kuvakaappaus_-_2025-03-26_13-27-11= .png > > > This ends up with prosody and openssh crashing, nginx still > working. the picture shows shepherd system-log crashing with: > > uncaught exception, value out of range in fibers. It references an > "unknown file" and a string with prosody. The code that throws that out-of-range exception on =E2=80=98substring=E2= =80=99 is (match:substring m 3) in =E2=80=98parse-system-log-message=E2=80=99. Normally, =E2=80=98regexp-exec=E2=80=99 returns match structures with corre= ct boundaries, but in this case it didn=E2=80=99t. I reported it here: https://issues.guix.gnu.org/77392 Several options come to mind: 1. Run shepherd in a UTF-8 locale; a quick workaround is to run: sudo herd eval root '(setlocale LC_ALL "C.UTF-8")' but we could also do that in Guix System. 2. Avoid (ice-9 regex) altogether in =E2=80=98system-log.scm=E2=80=99 and= use a pure Scheme implementation, SRFI-115-style. Thanks for your bug report! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 31 16:44:28 2025 Received: (at control) by debbugs.gnu.org; 31 Mar 2025 20:44:28 +0000 Received: from localhost ([127.0.0.1]:43125 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tzLzg-00027h-8z for submit@debbugs.gnu.org; Mon, 31 Mar 2025 16:44:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45666) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tzLzd-000272-U9 for control@debbugs.gnu.org; Mon, 31 Mar 2025 16:44:26 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzLzU-0004dS-GM for control@debbugs.gnu.org; Mon, 31 Mar 2025 16:44:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to: references; bh=0AUilOlgCxVbSr4fSlsZAIGNWoQZnx2vvqCsIEXUiFA=; b=PTpthguslv+TnC vQ1COHOcr8MbJQ0FNrHeyBxd7cwg+00/DLKqq4VF21ywp+5X5OKdBA4sQdI8oTBuAM+G5Q6Ua79w6 dMCMtaBWbhQJMLHob8NMP/yD8M9nNxzjeD44v8+Z+uWwqzoF0+Gq0q6YCIRx7VldIszN8WlVyhza7 g3RdMjD3TnE7UwzgZUvr/vHvhLBUfCs3uvlMRcux3eraq9rHQxgib4Y0tmcFOyfH67JOEadW05VwA hFYfHNvAl3jGKCL35D4NS+JRdDpNvBKsQ0lSGg2Zsurm7U2SYl9Uri4yXjrEQRvs0caz+6IcpXaFK xYozGzIoMaE+CW5ibt7Q==; Date: Mon, 31 Mar 2025 22:44:13 +0200 Message-Id: <874iz9j5gi.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #77283 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) retitle 77283 [Shepherd] System log crashes when parsing message quit From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 31 17:17:21 2025 Received: (at 77283-done) by debbugs.gnu.org; 31 Mar 2025 21:17:21 +0000 Received: from localhost ([127.0.0.1]:43166 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tzMVU-00035H-OY for submit@debbugs.gnu.org; Mon, 31 Mar 2025 17:17:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60634) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tzMVR-00034e-RY for 77283-done@debbugs.gnu.org; Mon, 31 Mar 2025 17:17:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzMVJ-0008OR-QG; Mon, 31 Mar 2025 17:17:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=4T8RVmNxH8G1loyCuXhU2dmjc42abXhN8QEaZisfUEk=; b=is2sVnSuqdwqOkYhVk3R LstmY2eFSeESB46GpctPRC5x+dfImejaLgQ9kTnV/5H0KgTj4dYAzqxR/RWN8h9TNyadP1q++JRYE Lxz1XeqJZ2RbZfFpOmZ8RtyWEzg5x2zR2PeJ7oyZGbgWgcrmxP003O5+iCzhaUhXW2Ss6ejtXveKy PW+Vr8rVMudhrnGZROZJ53LLlgojkZho7+QlJ622fmEg3PQVLEoheW79GFCcrIloQpqC4+cxT2k/l kEg3GHsi0SPrntz8aHzZDGNw5bMyusB2/C1G/ifchoCsNpeRCkF5M4pimardbwGwQFQliwGZ4x1G4 nd/RRyZIoWSEBg==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: MSavoritias Subject: Re: bug#77283: Shepherd can't parse logs. In-Reply-To: <87bjtidygh.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Sun, 30 Mar 2025 23:02:06 +0200") References: <87bjtidygh.fsf@gnu.org> Date: Mon, 31 Mar 2025 23:17:04 +0200 Message-ID: <87zfh0j3xr.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 77283-done Cc: 77283-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Ludovic Court=C3=A8s skribis: > The code that throws that out-of-range exception on =E2=80=98substring=E2= =80=99 is > (match:substring m 3) in =E2=80=98parse-system-log-message=E2=80=99. > > Normally, =E2=80=98regexp-exec=E2=80=99 returns match structures with cor= rect > boundaries, but in this case it didn=E2=80=99t. I reported it here: > > https://issues.guix.gnu.org/77392 Fixed in d98d61a8a3f20de46d18ce4a8af05c93fab20b89 by sidestepping the Guile bug. > 1. Run shepherd in a UTF-8 locale; a quick workaround is to run: > > sudo herd eval root '(setlocale LC_ALL "C.UTF-8")' I recommend doing this while waiting for 1.0.4 to be out, normally in less than a month. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 09 20:52:19 2025 Received: (at 77283) by debbugs.gnu.org; 10 Apr 2025 00:52:19 +0000 Received: from localhost ([127.0.0.1]:42823 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1u2g9T-0006jy-8c for submit@debbugs.gnu.org; Wed, 09 Apr 2025 20:52:19 -0400 Received: from mugam.systemreboot.net ([139.59.75.54]:51572) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1u2g9P-0006je-OW for 77283@debbugs.gnu.org; Wed, 09 Apr 2025 20:52:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=systemreboot.net; s=default; t=1744246317; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=x/B70E22lYL4Q3MiBmW+jZ3UzRwfgW4My+eOeoHryD0=; b=nRKRKNG8VzOY1ebENIci6xtrAKRnkkMl5jUcqEqc3CL06yCqpMq5kmIIFVLZN6QwZzh3O9 ZvAOYZ7sXFkEmcfEeqm1frw8trH7x7g4BZFYOyXWh4f5PZuIwGG69a7cn7ZUsUsrkgqsi5 jjlcwIeVOjeUutL+rhNaFHdtaBCxSAP8WbRy9CCqR7ObgYeHaRyra28ggLLe8UCFIaWHMv G/XmQ/NJ26YbBLjecH2OgQwN+WhMsfpTcUZWteC1/+mnsN6DsahtZgb2BKCOAp0tonCXm8 gBGcGTuMEKIVTlaZSvBFWyP2q4JmR9bg7tWKn8HaV9LZkkk524ZxAJKqJl0Y/A== Received: from localhost ( [192.168.2.1]) by mugam.systemreboot.net (OpenSMTPD) with ESMTPSA id c5f71adf (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Apr 2025 00:51:57 +0000 (UTC) From: Arun Isaac To: 77283@debbugs.gnu.org Subject: Re: [Shepherd] System log crashes when parsing message In-Reply-To: <87zfh0j3xr.fsf@gnu.org> Date: Thu, 10 Apr 2025 01:51:51 +0100 Message-ID: <87semgg7o8.fsf@systemreboot.net> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 77283 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , MSavoritias X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) I ran into this issue today after a system upgrade. Thank you both for identifying and figuring out the issue! I have applied the suggested workaround now. Cheers! From unknown Wed Sep 10 17:01:38 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 08 May 2025 11:24:10 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator