Subject: bug#56519: Shepherd non-deterministically fails to load the "guix-daemon" service after the "user-processes" service has been started

From: Markus Nilsson <[email-masked]>
Date: Tue, 26 Jul 2022 02:48:54 +0000

Hi Ludo,

Thankyou for taking a look at this issue!

I think I didn't make it clear in the original bug report what the actual
problem was.

I understand that in the beginning of the boot process the
"device-mapping-streetkid_VG_storage-streetkid_LV_home" shepherd service is
failing and this is causing shepherd services that depend on it to fail to
start as a consequence. Later in the boot process the
"device-mapping-streetkid_VG_storage-streetkid_LV_home" does successfully
start though.

What happens is that after the
"device-mapping-streetkid_VG_storage-streetkid_LV_home" does start (after a
couple of tries), other shepherd services will then start including the
"user-processes" shepherd service. The problem is that the "guix-daemon"
shepherd service doesn't always start after that. Sometimes it will, but
sometimes it won't (see my first message for examples). Shouldn't the
"guix-daemon" shepherd service ALWAYS start after the "user-processes"
shepherd service starts?

I also found that the "nscd" shepherd service is failing to start even
though the "user-processes" shepherd service has started.

The following terminal output was taken after my streetkid server had
finished booting up. This should make things clearer:

1 mark@streetkid ~$ sudo herd status
2 Started:
3  + /proc/fs/nfsd
4  + console-font-tty1
5  + console-font-tty2
6  + console-font-tty3
7  + console-font-tty4
8  + console-font-tty5
9  + console-font-tty6
10  + device-mapping-streetkid_VG_storage-streetkid_LV_home
11  + file-system-/dev/pts
12  + file-system-/dev/shm
13  + file-system-/gnu/store
14  + file-system-/home
15  + file-system-/sys/firmware/efi/efivars
16  + file-system-/sys/kernel/debug
17  + file-systems
18  + idmap-daemon
19  + loopback
20  + mcron
21  + networking
22  + nfs
23  + ntpd
24  + root
25  + root-file-system
26  + rpc-pipefs
27  + rpc.mountd
28  + rpc.nfsd
29  + rpc.statd
30  + rpcbind-daemon
31  + ssh-daemon
32  + sshd-1
33  + syslogd
34  + term-tty1
35  + term-tty2
36  + term-tty3
37  + term-tty4
38  + term-tty5
39  + term-tty6
40  + udev
41  + urandom-seed
42  + user-file-systems
43  + user-processes
44  + virtual-terminal
45 Stopped:
46  - guix-daemon
47  - nscd
48  - term-console
49 One-shot:
50  * host-name
51  * sysctl
52  * user-homes

As you can see on line 10
"device-mapping-streetkid_VG_storage-streetkid_LV_home" has been started
and on line 43 "user-processes" has been started. The problem is that
"guix-daemon" hasn't been started (line 46) and "nscd" also hasn't been
started (line 47). Shouldn't Shepherd start these two services once
"user-processes" has been started?

I checked other services that depend on "user-processes" (see the attached
shepherd dependency graph for my system). The "mcron" service and "ntpd"
successfully start (lines 20 and 23). This still leaves the mystery of why
"guix-daemon" and "nscd" won't start even though "user-processes" HAS been
started.

I hope this makes things clearer.

Cheers
Markus