GNU bug report logs - #71144
Interactive prompt opened upon shepherd config file error

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 71144 in the body.
You can then email your comments to 71144 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: bug-guix <at> gnu.org
Subject: Interactive prompt opened upon shepherd config file error
Date: Thu, 23 May 2024 12:59:40 +0200

Hello,

One problem we noticed in the analysis of the boot problem of bayfront
after the recent downtime¹ is that an interactive REPL would be opened
after an unbound variable was found in the shepherd config file:

--8<---------------cut here---------------start------------->8---
[   13.098907] shepherd[1]: Service root started.
[   13.100711] shepherd[1]: Service root running with value #t.
[   13.103824] shepherd[1]: Service root has been started.
[   13.426102] shepherd[1]: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
[   13.428099] shepherd[1]: Unbound variable: make-forkexec-constructor/container
[   13.429912] shepherd[1]: 
[   13.431108] shepherd[1]: Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
[   13.441983] shepherd[1]: GNU Guile 3.0.9
[   13.442728] shepherd[1]: Copyright (C) 1995-2023 Free Software Foundation, Inc.
[   13.443947] shepherd[1]: 
[   13.444427] shepherd[1]: Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
[   13.445679] shepherd[1]: This program is free software, and you are welcome to redistribute it
[   13.446919] shepherd[1]: under certain conditions; type `,show c' for 
[   13.447072] shepherd[1]: details.
[   13.448737] shepherd[1]: 
[   13.449239] shepherd[1]: Enter `,help' for help.
--8<---------------cut here---------------end--------------->8---

This was unhelpful because we couldn’t interact with that REPL remotely
(no IPMI).  Even when you can interact, it’s of limited use; in this
case, if you type “,q”, it tries to continue and fails:

--8<---------------cut here---------------start------------->8---
Uncaught exception in task:
In fibers.scm:
    172:8  7 (_)
In ice-9/exceptions.scm:
   406:15  6 (_)
In ice-9/boot-9.scm:
  1752:10  5 (with-exception-handler _ _ #:unwind? _ # _)
In shepherd/service.scm:
   824:39  4 (_)


this is because we’re effectively adding #f in the middle of the list
passed to ‘register-services’ (see below).

This REPL-on-error “feature” comes from Guix System, not Shepherd, in
the config file generated from (gnu services shepherd):

    ;; Arrange to spawn a REPL if something goes wrong.  This is better
    ;; than a kernel panic.
    (call-with-error-handling
      (lambda ()
        (register-services
         (parameterize ((current-warning-port
                         (%make-void-port "w")))
           (map (lambda (file)
                  (save-module-excursion
                   (lambda ()
                     (set-current-module (make-user-module))
                     (load-compiled file))))
                '#$(map scm->go files))))))

The rationale mentioned in the comment no longer holds: starting from
Shepherd 0.10.2, the config file is loaded in the background; if it’s
evaluation fails, shepherd keeps running (see
‘tests/config-failure.sh’, which tests this behavior).

I think we should change the above to log and gracefully handle failure
to load an individual service file.

Ludo’.

¹ https://lists.gnu.org/archive/html/info-guix/2024-05/msg00000.html

Message #10 received at 71144 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 71144 <at> debbugs.gnu.org
Cc: Andreas Enge <andreas <at> enge.fr>, Christopher Baines <mail <at> cbaines.net>
Subject: Re: bug#71144: Interactive prompt opened upon shepherd config file
 error
Date: Sat, 25 May 2024 11:11:04 +0200

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:

> I think we should change the above to log and gracefully handle failure
> to load an individual service file.

With the change below, every service except the offending one is loaded
and started as expected:

--8<---------------cut here---------------start------------->8---
[   22.450515] shepherd[1]: Service root running with value #t.
[   22.454624] shepherd[1]: Service root has been started.
[   22.711738] shepherd[1]: Exception caught while loading '/gnu/store/fjis6iqpjfcnr90fy8rsg9v4j828jslv-shepherd-gwl-web.go': #<&compound-exception components: (#<&undefined-variable> #<&origin origin: #f> #<&message message: "Unbound variable: ~S"> #<&irritants irri
[   22.711839] tants: (make-forkexec-constructor/container)> #<&exception-with-kind-and-args kind: unbound-variable args: (#f "Unbound variable: ~S" (make-forkexec-constructor/container) #f)>)>
[   22.755146] shepherd[1]: starting services...
[   22.756491] shepherd[1]: Configuration successfully loaded from '/gnu/store/mq7y31xnjcjwjkyf6w7qiaq61g6n9f5x-shepherd.conf'.
Uncaught exception in task:
In fibers.scm:
    172:8  7 (_)
In ice-9/exceptions.scm:
   406:15  6 (_)
In ice-9/boot-9.scm:
  1752:10  5 (with-exception-handler _ _ #:unwind? _ # _)
In shepherd/service.scm:
   824:39  4 (_)
In oop/goops.scm:
  1567:11  3 (cache-miss #f)
   1585:2  2 (_ _ _)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1683:16  0 (raise-exception _ #:continuable? _)
ice-9/boot-9.scm:1683:16: In procedure raise-exception:
No applicable method for #<<generic> one-shot-service? (1)> in call (one-shot-service? #f)
[   22.798737] shepherd[1]: Starting service user-file-systems...
[   22.800361] shepherd[1]: Starting service root-file-system...
[   22.802015] shepherd[1]: Starting service host-name...
[   22.803688] shepherd[1]: Starting service pam...
[   22.805372] shepherd[1]: Starting service sysctl...
[   22.806926] shepherd[1]: Starting service loopback...
[   22.808225] shepherd[1]: Starting service firewall...
--8<---------------cut here---------------end--------------->8---

(There’s still this scary-looking but harmless backtrace in the middle:
that’s because (start-in-the-background '(something-that-does-not-exist))
throws like that as of 0.10.4.)

Once booted, shepherd is fine and you can interact normally with it; the
only thing missing is, in this case, the ‘gwl-web’ service, which we
failed to load.

I think that’s a significant improvement.

Thoughts?

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/gnu/services/shepherd.scm b/gnu/services/shepherd.scm
index 455e972535d..f13c52c37ba 100644
--- a/gnu/services/shepherd.scm
+++ b/gnu/services/shepherd.scm
@@ -380,8 +380,7 @@ (define (shepherd-configuration-file services shepherd)
         (scm->go (cute scm->go <> shepherd)))
     (define config
       #~(begin
-          (use-modules (srfi srfi-34)
-                       (system repl error-handling))
+          (use-modules (srfi srfi-1))
 
           (define (make-user-module)
             ;; Copied from (shepherd support), where it's private.
@@ -417,17 +416,22 @@ (define (shepherd-configuration-file services shepherd)
 
           ;; Arrange to spawn a REPL if something goes wrong.  This is better
           ;; than a kernel panic.
-          (call-with-error-handling
-            (lambda ()
-              (register-services
-               (parameterize ((current-warning-port
-                               (%make-void-port "w")))
-                 (map (lambda (file)
-                        (save-module-excursion
-                         (lambda ()
-                           (set-current-module (make-user-module))
-                           (load-compiled file))))
-                      '#$(map scm->go files))))))
+          (register-services
+           (parameterize ((current-warning-port (%make-void-port "w")))
+             (filter-map (lambda (file)
+                           (with-exception-handler
+                               (lambda (exception)
+                                 (format #t "Exception caught \
+while loading '~a': ~s~%"
+                                         file exception)
+                                 #f)
+                             (lambda ()
+                               (save-module-excursion
+                                (lambda ()
+                                  (set-current-module (make-user-module))
+                                  (load-compiled file))))
+                             #:unwind? #t))
+                         '#$(map scm->go files))))
 
           (format #t "starting services...~%")
           (let ((services-to-start

Message #15 received at 71144 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 71144 <at> debbugs.gnu.org, Andreas Enge <andreas <at> enge.fr>
Subject: Re: bug#71144: Interactive prompt opened upon shepherd config file
 error
Date: Sat, 25 May 2024 13:02:05 +0100

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Ludovic Courtès <ludovic.courtes <at> inria.fr> skribis:
>
>> I think we should change the above to log and gracefully handle failure
>> to load an individual service file.
>
> With the change below, every service except the offending one is loaded
> and started as expected:
>
> --8<---------------cut here---------------start------------->8---
> [   22.450515] shepherd[1]: Service root running with value #t.
> [   22.454624] shepherd[1]: Service root has been started.
> [   22.711738] shepherd[1]: Exception caught while loading '/gnu/store/fjis6iqpjfcnr90fy8rsg9v4j828jslv-shepherd-gwl-web.go': #<&compound-exception components: (#<&undefined-variable> #<&origin origin: #f> #<&message message: "Unbound variable: ~S"> #<&irritants irri
> [   22.711839] tants: (make-forkexec-constructor/container)> #<&exception-with-kind-and-args kind: unbound-variable args: (#f "Unbound variable: ~S" (make-forkexec-constructor/container) #f)>)>
> [   22.755146] shepherd[1]: starting services...
> [   22.756491] shepherd[1]: Configuration successfully loaded from '/gnu/store/mq7y31xnjcjwjkyf6w7qiaq61g6n9f5x-shepherd.conf'.
> Uncaught exception in task:
> In fibers.scm:
>     172:8  7 (_)
> In ice-9/exceptions.scm:
>    406:15  6 (_)
> In ice-9/boot-9.scm:
>   1752:10  5 (with-exception-handler _ _ #:unwind? _ # _)
> In shepherd/service.scm:
>    824:39  4 (_)
> In oop/goops.scm:
>   1567:11  3 (cache-miss #f)
>    1585:2  2 (_ _ _)
> In ice-9/boot-9.scm:
>   1685:16  1 (raise-exception _ #:continuable? _)
>   1683:16  0 (raise-exception _ #:continuable? _)
> ice-9/boot-9.scm:1683:16: In procedure raise-exception:
> No applicable method for #<<generic> one-shot-service? (1)> in call (one-shot-service? #f)
> [   22.798737] shepherd[1]: Starting service user-file-systems...
> [   22.800361] shepherd[1]: Starting service root-file-system...
> [   22.802015] shepherd[1]: Starting service host-name...
> [   22.803688] shepherd[1]: Starting service pam...
> [   22.805372] shepherd[1]: Starting service sysctl...
> [   22.806926] shepherd[1]: Starting service loopback...
> [   22.808225] shepherd[1]: Starting service firewall...
> --8<---------------cut here---------------end--------------->8---
>
> (There’s still this scary-looking but harmless backtrace in the middle:
> that’s because (start-in-the-background '(something-that-does-not-exist))
> throws like that as of 0.10.4.)
>
> Once booted, shepherd is fine and you can interact normally with it; the
> only thing missing is, in this case, the ‘gwl-web’ service, which we
> failed to load.
>
> I think that’s a significant improvement.
>
> Thoughts?

That looks good to me, the "Arrange to spawn a REPL if something goes
wrong" comment needs removing/updating, but that's the only thing I
spotted.

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 71144-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 71144-done <at> debbugs.gnu.org, Andreas Enge <andreas <at> enge.fr>
Subject: Re: bug#71144: Interactive prompt opened upon shepherd config file
 error
Date: Sat, 25 May 2024 16:59:16 +0200

Christopher Baines <mail <at> cbaines.net> skribis:

> That looks good to me, the "Arrange to spawn a REPL if something goes
> wrong" comment needs removing/updating, but that's the only thing I
> spotted.

Cool.  I updated the comment and pushed it as
cca25a67693bb68a1884a081b415a43fad1e8641.

Thanks!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.