Subject: Re: bug#46942: ci.guix.gnu.org is slow from my system

From: raid5atemyhomework <[email-masked]>
Date: Mon, 15 Mar 2021 00:13:48 +0000

Hello all,

Unfortunately, it seems that the SJTU server is somewhat unreliable, I sometimes get random failures in various parts talking to the substitute server, complaining of strange responses from the server:

```
Backtrace:
In guix/ui.scm:
  2164:12 19 (run-guix-command _ . _)
In guix/scripts/substitute.scm:
    652:2 18 (guix-substitute . _)
In unknown file:
          17 (with-continuation-barrier #<procedure thunk ()>)
In ice-9/boot-9.scm:
  1736:10 16 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          15 (apply-smob/0 #<thunk 7f5de06c9cc0>)
In ice-9/boot-9.scm:
  1736:10 14 (with-exception-handler _ _ #:unwind? _ # _)
  1736:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1731:15 12 (with-exception-handler #<procedure 7f5dddeb8e70 at ic…> …)
In guix/scripts/substitute.scm:
   701:17 11 (_)
    410:7 10 (process-substitution _ "/gnu/store/yfzsz94qy92c7m9w0j…" …)
In ice-9/boot-9.scm:
  1736:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    419:9  8 (_)
In ice-9/boot-9.scm:
  1731:15  7 (with-exception-handler #<procedure 7f5dddeb8990 at ic…> …)
  1669:16  6 (raise-exception _ #:continuable? _)
  1667:16  5 (raise-exception _ #:continuable? _)
  1669:16  4 (raise-exception _ #:continuable? _)
  1764:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1669:16  2 (raise-exception _ #:continuable? _)
  1667:16  1 (raise-exception _ #:continuable? _)
  1669:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `bad-response' with args `("Bad Response-Line: ~s" (""))'.
Backtrace:
In ice-9/boot-9.scm:
  1736:10  4 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
           3 (apply-smob/0 #<thunk 7f5de074e520>)
In ice-9/boot-9.scm:
    718:2  2 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8  1 (_ #(#(#<directory (guile-user) 7f5de0751c80>)))
In guix/ui.scm:
  2164:12  0 (run-guix-command _ . _)

guix/ui.scm:2164:12: In procedure run-guix-command:
Throw to key `bad-response' with args `("Bad Response-Line: ~s" (""))'.
substitution of /gnu/store/yfzsz94qy92c7m9w0jbll7slc2pcap45-guix-packages-base failed
guix pull: error: corrupt input while restoring archive from #<closed: file 7f0dc2367bd0>
```

I configured `guix-daemon` to have two substitute URLs: SJTUG, and the official Cuirass in Berlin.  My expectation is that if the SJTUG server fails, it should fall back on using the Cuirass server.

So my problems are two-fold:

* Apparently the SJTUG mirror sometimes acts in ways that aren't quite how Guix expects it.  Either SJTUG or Guix has to be fixed so they talk in compatible manners.
* Multiple Substitute URLs are not useful for the mirroring configuration where one server is temporarily unable to properly serve; further servers are not checked.

I'm still not satisfied with this solution overall.  Yes it's fast but it's imperfect.

I recently had to rebuild an OS (because I was dumb; the Guix language for shepherd services can easily lead you deadlocking shepherd itself) and had supreme difficulty reinstalling, because the Berlin server was too slow (would have taken ~10hours downloading everything) while the SJTUG server was unreliable and would consistently break the automated install.  I had to `guix system build --fallback` separately and *then* `guix system init` manually.

Thanks
raid5atemyhomework