GNU bug report logs - #77283
[Shepherd] System log crashes when parsing message

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 77283 in the body.
You can then email your comments to 77283 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: MSavoritias <email <at> msavoritias.me>
To: bug-guix <at> gnu.org
Subject: Shepherd can't parse logs.
Date: Wed, 26 Mar 2025 15:42:06 +0200

Shepherd has been crashing our system with the message:

https://upload.magicbroccoli.de/fb6e7843fef693538a5c7c156c0a98fc0aeb6bac/KmA7ovtC2Fcw0Nm0PSCtphwSyDLI1VUF3QrT9PtD/Kuvakaappaus_-_2025-03-26_13-27-11.png


This ends up with prosody and openssh crashing, nginx still working. the 
picture shows shepherd system-log crashing with:

uncaught exception, value out of range in fibers. It references an 
"unknown file" and a string with prosody.


I looked into git commits in the last month (bug didn't exist before end 
of January/beginning of February, it seems that the bug is either 
shepherd itself, it was updated recently and that is what i was told in 
the irc. Or it is the certbot timer. Certbot timer is the only service 
we have installed from the ones that changed.


Being certbot makes less sense since prosody is mentioned specifically 
tho so it seems a bug in how shepherd parses logs.


Our whole config is here -> https://codeberg.org/joinjabber/Infra


MSavoritias

Message #8 received at 77283 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: MSavoritias <email <at> msavoritias.me>
Cc: 77283 <at> debbugs.gnu.org
Subject: Re: bug#77283: Shepherd can't parse logs.
Date: Sun, 30 Mar 2025 23:02:06 +0200

Hi MSavoritias,

MSavoritias <email <at> msavoritias.me> skribis:

> Shepherd has been crashing our system with the message:
>
> https://upload.magicbroccoli.de/fb6e7843fef693538a5c7c156c0a98fc0aeb6bac/KmA7ovtC2Fcw0Nm0PSCtphwSyDLI1VUF3QrT9PtD/Kuvakaappaus_-_2025-03-26_13-27-11.png
>
>
> This ends up with prosody and openssh crashing, nginx still
> working. the picture shows shepherd system-log crashing with:
>
> uncaught exception, value out of range in fibers. It references an
> "unknown file" and a string with prosody.

The code that throws that out-of-range exception on ‘substring’ is
(match:substring m 3) in ‘parse-system-log-message’.

Normally, ‘regexp-exec’ returns match structures with correct
boundaries, but in this case it didn’t.  I reported it here:

  https://issues.guix.gnu.org/77392

Several options come to mind:

  1. Run shepherd in a UTF-8 locale; a quick workaround is to run:

       sudo herd eval root '(setlocale LC_ALL "C.UTF-8")'

     but we could also do that in Guix System.

  2. Avoid (ice-9 regex) altogether in ‘system-log.scm’ and use a pure
     Scheme implementation, SRFI-115-style.

Thanks for your bug report!

Ludo’.

Message #15 received at 77283-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: MSavoritias <email <at> msavoritias.me>
Cc: 77283-done <at> debbugs.gnu.org
Subject: Re: bug#77283: Shepherd can't parse logs.
Date: Mon, 31 Mar 2025 23:17:04 +0200

Hi,

Ludovic Courtès <ludo <at> gnu.org> skribis:

> The code that throws that out-of-range exception on ‘substring’ is
> (match:substring m 3) in ‘parse-system-log-message’.
>
> Normally, ‘regexp-exec’ returns match structures with correct
> boundaries, but in this case it didn’t.  I reported it here:
>
>   https://issues.guix.gnu.org/77392

Fixed in d98d61a8a3f20de46d18ce4a8af05c93fab20b89 by sidestepping the
Guile bug.

>   1. Run shepherd in a UTF-8 locale; a quick workaround is to run:
>
>        sudo herd eval root '(setlocale LC_ALL "C.UTF-8")'

I recommend doing this while waiting for 1.0.4 to be out, normally in
less than a month.

Ludo’.

Message #18 received at 77283 <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: 77283 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 MSavoritias <email <at> msavoritias.me>
Subject: Re: [Shepherd] System log crashes when parsing message
Date: Thu, 10 Apr 2025 01:51:51 +0100

I ran into this issue today after a system upgrade. Thank you both for
identifying and figuring out the issue! I have applied the suggested
workaround now.

Cheers!

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.