GNU bug report logs - #36364
Potential SSH session exhaustion

Previous Next

Full log

Message #8 received at 36364 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zerodaysfordays <at> sdf.lonestar.org (Jakob L. Kreuze)
Cc: 36364 <at> debbugs.gnu.org
Subject: Re: bug#36364: Potential SSH session exhaustion
Date: Mon, 24 Jun 2019 22:45:33 +0200

[Message part 1 (text/plain, inline)]

Hello Jakob,

zerodaysfordays <at> sdf.lonestar.org (Jakob L. Kreuze) skribis:

> In developing a test suite for the internals of 'guix deploy', the
> virtual machine I spun up spat out an OpenSSH error that I've never
> encountered before.
>
>   Jun 24 16:59:26 localhost sshd[235]: error: no more sessions
>
> This is, initially, quite curious. Creating the SSH session with
> #:log-verbosity set to 'protocol gives some insight.

[...]

>   ;;; [2019/06/24 12:59:26.546104, 2] channel_open: Creating a channel 76 with 64000 window and 32768 max packet
>   ;;; [2019/06/24 12:59:26.546407, 2] ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 76:9
>   ;;; [2019/06/24 12:59:26.546438, 2] ssh_packet_channel_open_conf: Remote window : 0, maxpacket : 32768
>   ;;; [2019/06/24 12:59:26.546839, 2] channel_rcv_change_window: Adding 2097152 bytes to channel (76:9) (from 0 bytes)
>   ;;; [2019/06/24 12:59:26.546865, 2] channel_request: Channel request exec success
>   ;;; [2019/06/24 12:59:26.630055, 2] grow_window: growing window (channel 76:9) to 1280000 bytes
>   ;;; [2019/06/24 12:59:27.272139, 2] channel_open: Creating a channel 77 with 64000 window and 32768 max packet
>   ;;; [2019/06/24 12:59:27.272846, 1] ssh_packet_channel_open_fail: Channel opening failure: channel 77 error (2) open failed
>
> I will admit that my knowledge of the SSH protocol is limited, but the
> rising channel number gives me the impression that channels are going
> out of scope and aren't being cleaned up.

Procedures like ‘send-files’ call ‘channel-get-exit-status’ and
‘close-port’ on ports that represent SSH channels.

AFAICS, ‘close-port’ triggers a call of ‘ptob_close’ in Guile-SSH, which
in turn calls ‘ssh_channel_close’.

sshd_config(5) says:

--8<---------------cut here---------------start------------->8---
   MaxSessions
           Specifies the maximum number of open shell, login or
           subsystem (e.g. sftp) sessions permitted per network
           connection.  Multiple sessions may be established by clients
           that support connection multiplexing.  Setting MaxSessions to
           1 will effectively disable session multiplexing, whereas
           setting it to 0 will prevent all shell, login and subsystem
           sessions while still permitting forwarding.  The default is
           10.
--8<---------------cut here---------------end--------------->8---

So you must be hitting this limit.

I see that ‘remote-eval’ does not close ‘remote’ though, so this channel
port remains open until it’s GC’d, which happens too late.  Could you
try the attached patch?  It allows me to do more than 10 ‘remote-eval’
calls in a row.

Thanks,
Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/guix/remote.scm b/guix/remote.scm
index cc051dee8a..fa19ece112 100644
--- a/guix/remote.scm
+++ b/guix/remote.scm
@@ -116,6 +116,7 @@ remote store."
           (mbegin %store-monad
             (built-derivations to-build)
             ((store-lift send-files) to-send remote #:recursive? #t)
+            (return (close-connection remote))
             (return (%remote-eval lowered session))))
         (let ((to-send (map (lambda (input)
                               (match (gexp-input-thing input)
@@ -127,4 +128,5 @@ remote store."
           (mbegin %store-monad
             ((store-lift send-files) to-send remote #:recursive? #t)
             (return (build-derivations remote to-build))
+            (return (close-connection remote))
             (return (%remote-eval lowered session)))))))

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.