Subject: Re: bug#37309: ‘ssh-daemon’ service fails to start at boot

From: Marius Bakke <[email-masked]>
Date: Sat, 28 Nov 2020 02:08:34 +0100

Christopher Lemmer Webber <[email-masked]> skriver:

> Giovanni Biscuolo writes:
>
>> Hi,
>>
>> following a recent discussion on guix-sysadmin I have to confirm the
>> ssh-daemon issue since it is still happening on some of the machines I
>> administer
>>
>> Previous possibly related bug reports are
>> https://issues.guix.gnu.org/issue/30993 and
>> https://issues.guix.gnu.org/issue/32197
>>
>> Unfortunately this issue is *not* well reproducible, it depends on some
>> mysterious (to me) timing factor; AFAIU it does *not* depend on the
>> shepherd version, probably it depends on "something" related to IPv6
>> (read below the details)
>
> This issue continues to plauge me, and has ever since I started to use
> GuixSD.  However it is much worse now that I am running Guix on
> servers... I frequently have to log in via Linode's (nonfree!) web
> console on every server that is rebooted and kick herd to restart
> openssh.  Once I do that it's fine.

Can you share an excerpt of /var/log/messages (ideally the whole boot
sequence) from when SSH failed to start?

> I don't think my linode machine is on "spinning rust" so I don't think
> this is the cause.  IPv6, maybe?  Dunno what.
>
> However I think that it's probably really a dependency issue somewhere;
> herd is starting opensshd before some other dependent service is
> spawned.  But what?  Maybe something authentication related like
> networking, or something.  But hm, networking is required...
>
> I'm assuming others must be experiencing this still too... right?

FWIW I have never encountered this.  :-/

> Would really like to see it fixed.  It's one of the few things holding
> me back from recommending Guix on servers to others.
>
> Do others have any idea?
>
> I noticed the lsh daemon requires networking.  Why doesn't openssh?

It's really for legacy reasons, from before we had the Guix System
installer.  Then a common way to install was to run dhclient and
"herd start ssh-daemon" manually on the live image, so people could
do the installation over SSH:

  https://issues.guix.gnu.org/26548#5

Nowadays, the installer gives a nice and quick way to deploy a minimal
system, and I suspect the SSH method has fallen out of favor.

> What about the following "fix"?

[...]

>    (list (shepherd-service
>           (documentation "OpenSSH server.")
> -         (requirement '(syslogd loopback))
> +         (requirement '(syslogd networking loopback))

If it works for you, let's do this.  It would be good to find the
underlying cause though...

Not sure what to do about the installer however: perhaps create
yet-another undocumented field of openssh-service-type that makes the
networking requirement optional?