Subject: bug#58485: closed (Re: bug#58485: [shepherd] EADDRINUSE while restarting ssh-daemon)

From: [email-masked] (GNU bug Tracking System)
Date: Sun, 11 Jun 2023 14:21:01 +0000

Your bug report

#58485: [shepherd] EADDRINUSE while restarting ssh-daemon

which was filed against the guix package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
58485: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=58485
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#58485: [shepherd] EADDRINUSE while restarting ssh-daemon
From: Ludovic Courtès <[email-masked]>
To: Lars-Dominik Braun <[email-masked]>
Cc: Lars-Dominik Braun <[email-masked]>, [email-masked]
Date: Sun, 11 Jun 2023 16:20:21 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi Lars,

Ludovic Courtès <[email-masked]> skribis:

> Ludovic Courtès <[email-masked]> skribis:
>
>>> 1     14:12:15.117035 read(21, "(shepherd-command (version 0) (action restart) (service ssh-daemon) (arguments ()) (directory \"/root\"))", 1024) = 103
>>> 1     14:12:15.117254 close(27)         = 0
>>> 1     14:12:15.117283 close(30)         = 0
>>> 1     14:12:15.117416 newfstatat(AT_FDCWD, "/etc/localtime", {st_dev=makedev(0x8, 0x2), st_ino=110100491, st_mode=S_IFREG|0444, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, s
>>> t_size=2298, st_atime=1676898665 /* 2023-02-20T14:11:05.338746772+0100 */, st_atime_nsec=338746772, st_mtime=1676898664 /* 2023-02-20T14:11:04.874743456+0100 */, st_mtime_nsec=874743456, st_c
>>> time=1676898664 /* 2023-02-20T14:11:04.874743456+0100 */, st_ctime_nsec=874743456}, 0) = 0
>>> 1     14:12:15.117475 write(17, "shepherd[1]: Service ssh-daemon has been stopped.\n", 50) = 50
>>> 1     14:12:15.117524 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 26
>>> 1     14:12:15.117561 setsockopt(26, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
>>> 1     14:12:15.117598 bind(26, {sa_family=AF_INET, sin_port=htons(2222), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
>>> 1     14:12:15.117724 write(21, "(reply (version 0) (result #f) (error (error (version 0) action-exception start ssh-daemon system-error (\"bind\" \"~A\" (\"Address already in use\") (98)))) (messages (\"Service ssh-daemon has been stopped.\")))", 204) = 204
>>> 1     14:12:15.117754 close(21)         = 0
>
> [...]
>
>> Maybe we should just have shepherd retry upon EADDRINUSE (like nginx
>> does, as you wrote), though I’d like to understand under what conditions
>> we can get EADDRINUSE in the first place.
>
> Done:
>
>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=41789ee8d0e164967f9ca196db4e9601400a462e

I’m assuming that this is fixed in Shepherd 0.10.x.  Please reopen if
you stumble upon this issue again.

Ludo’.

Subject: [shepherd] Restarting guix-publish fails
From: Lars-Dominik Braun <[email-masked]>
To: [email-masked]
Cc: Ludovic Courtès <[email-masked]>
Date: Thu, 13 Oct 2022 09:51:58 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi,

it seems that `herd restart guix-publish` stopped working after the
introduction of socket activation into shepherd. This is a problem,
because I restart guix-publish automatically after unattended-upgrades. It
fails with the following error for me:

---snip---
Backtrace:
           7 (primitive-load "/gnu/store/7xrg2sbb529ki6hv99n27svg0fi?")
In ice-9/boot-9.scm:
    724:2  6 (call-with-prompt ("prompt") #<procedure 7f8173184940 ?> ?)
  1752:10  5 (with-exception-handler _ _ #:unwind? _ # _)
In ice-9/eval.scm:
    619:8  4 (_ #(#(#<directory (guile-user) 7f817318ac80>)))
In ice-9/boot-9.scm:
   260:13  3 (for-each #<procedure restart-service (name)> _)
In gnu/services/herd.scm:
    168:4  2 (invoke-action guix-publish restart () #<procedure 7f81?>)
    176:7  1 (failure)
In ice-9/boot-9.scm:
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &action-exception-error:
      service: guix-publish
      action: start
      key: system-error
      args: ("bind" "~A" ("Address already in use") (98))
---snap---

Note that due to the socket activation you must visit the URL at least
once to start up the guix-publish process. Otherwise a restart will
work fine. It also works fine the second time I invoke `herd restart
guix-publish`, because `guix-publish` is dead by that time.

Looking at an strace shepherd is indeed trying to kill `guix-publish`
and re-bind to the same address:

---snip---
1     read(23, "(shepherd-command (version 0) (action restart) (service guix-publish) (arguments ()) (directory \"/root\"))", 1024) = 105
1     getpgid(18096)                    = 18096
1     getpgid(0)                        = 0
1     kill(-18096, SIGTERM)             = 0
1     newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0444, st_size=2298, ...}, 0) = 0
1     write(17, "shepherd[1]: Service guix-publish has been stopped.\n", 52) = 52
1     socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 36
1     setsockopt(36, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
1     bind(36, {sa_family=AF_INET, sin_port=htons(8082), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRINUSE (Address already in use)
1     write(23, "(reply (version 0) (result #f) (error (error (version 0) action-exception start guix-publish system-error (\"bind\" \"~A\" (\"Address already in use\") (98)))) (messages (\"Service guix-publish has been stopped.\")))", 208) = 208
1     close(23)
---snap---

The obvious explanation would be that stopping does not wait for the
process to actually exit. make-kill-destructor does not waitpid it seems
and 'running is set unconditionally to #f after 'stop has finished.

Cheers,
Lars