Subject: bug#61646: Bandwidth-induced offload timeout abort whole operating

From: Ludovic Courtès <[email-masked]>
Date: Thu, 23 Feb 2023 23:26:22 +0100

Hi Maxim,

Maxim Cournoyer <[email-masked]> skribis:

> I can reproduce this rather easily on my system:
>
> $ ./pre-inst-env guix build icedove
> The following derivations will be built:
>   /gnu/store/l6r93asndd0kwv7024iyrl71zd0lbpbq-icedove-102.7.2.drv
>   /gnu/store/8zi808086b3vlfjrhdm87fgljziwdqx2-icedove-l10n-102.7.2.drv
>   /gnu/store/v0sq7rb8fk36kjasb27a71z1a27wxb1s-icedove-minimal-102.7.2.drv
> process 19542 acquired build slot '/var/guix/offload/localhost:6666/0'
> normalized load on machine 'localhost' is 0.08
> building /gnu/store/8zi808086b3vlfjrhdm87fgljziwdqx2-icedove-l10n-102.7.2.drv...
> process 19548 acquired build slot '/var/guix/offload/localhost:6666/1'
> normalized load on machine 'localhost' is 0.08
> building /gnu/store/v0sq7rb8fk36kjasb27a71z1a27wxb1s-icedove-minimal-102.7.2.drv...
> guix offload: sending 1 store item (558 MiB) to 'localhost'...
> exporting path `/gnu/store/bwb5hcdyzgq16kmbsva7ax0zq6lzg78z-icedove-102.7.2.tar.xz'
> guix offload: error: failed to connect to 'localhost': Timeout connecting to localhost
> cannot build derivation `/gnu/store/l6r93asndd0kwv7024iyrl71zd0lbpbq-icedove-102.7.2.drv': 1 dependencies couldn't be built
> guix build: error: build of
>   `/gnu/store/l6r93asndd0kwv7024iyrl71zd0lbpbq-icedove-102.7.2.drv' failed
>
> The third derivation tries to get a build slot and times out, because
> the first two have already saturated the bandwidth of the link and it
> takes more time than expected to get a reply.

Weird.  Since the it’s a timeout while connecting, I suppose the patch
below would improve the situation:

diff --git a/guix/scripts/offload.scm b/guix/scripts/offload.scm
index 578b3b9888..90cf97401c 100644
--- a/guix/scripts/offload.scm
+++ b/guix/scripts/offload.scm
@@ -220,7 +220,7 @@ (define* (open-ssh-session machine #:optional max-silent-time)
         (session (make-session #:user (build-machine-user machine)
                                #:host (build-machine-name machine)
                                #:port (build-machine-port machine)
-                               #:timeout 10       ;initial timeout (seconds)
+                               #:timeout 30       ;initial timeout (seconds)
                                ;; #:log-verbosity 'protocol
                                #:identity (build-machine-private-key machine)

WDYT?

Ludo’.