Subject: bug#46942: ci.guix.gnu.org is slow from my system

From: raid5atemyhomework <[email-masked]>
Date: Fri, 05 Mar 2021 11:22:17 +0000

Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.

One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.

```
downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
 wine64-6.0  54.4MiB                                                                                                                                 10KiB/s 01:55 [                  ]   2.1%
```

This can be very slow, including as slow as 4KiB/s at times. This is fairly awful since sometimes I get this result:

```
downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
 wine64-6.0  54.4MiB                                                                                                                                 49KiB/s 06:01 [#####             ]  31.9%guix substitute: error: TLS error in procedure 'read_from_session_record_port': Error decoding the received TLS packet.
substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 failed
guix package: error: some substitutes for the outputs of derivation `/gnu/store/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
```

Because of the failed incomplete download, I have to ***restart the download again from 0%***, which is awful, awful UX. It would be really nice if:

1. The guix download process would make an effort to ***retry*** this a few times.
2. Continue the previous download instead of restarting from 0%.

The problem is not on my ISP, or at least not solely on my ISP. Doing a `wget` from `github.com`:

```
$ wget https://github.com/zfsonlinux/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
--2021-03-05 17:39:46--  https://github.com/zfsonlinux/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
Resolving github.com (github.com)... 13.229.188.59
Connecting to github.com (github.com)|13.229.188.59|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/openzfs/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz [following]
--2021-03-05 17:39:47--  https://github.com/openzfs/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/437011/71526a80-6ba3-11eb-893f-4ceb55b479d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210305%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210305T093947Z&X-Amz-Expires=300&X-Amz-Signature=9454446348219aa0731af009a5232bf5772e3d45a3a34052fa61f64f04c3c979&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=437011&response-content-disposition=attachment%3B%20filename%3Dzfs-2.0.3.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-03-05 17:39:47--  https://github-releases.githubusercontent.com/437011/71526a80-6ba3-11eb-893f-4ceb55b479d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210305%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210305T093947Z&X-Amz-Expires=300&X-Amz-Signature=9454446348219aa0731af009a5232bf5772e3d45a3a34052fa61f64f04c3c979&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=437011&response-content-disposition=attachment%3B%20filename%3Dzfs-2.0.3.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.108.154, 185.199.109.154, 185.199.110.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.108.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13114404 (13M) [application/octet-stream]
Saving to: ‘zfs-2.0.3.tar.gz’

zfs-2.0.3.tar.gz                                100%[=====================================================================================================>]  12.51M  9.34MB/s    in 1.3s

2021-03-05 17:39:49 (9.34 MB/s) - ‘zfs-2.0.3.tar.gz’ saved [13114404/13114404]

```

As can be seen above, I get 9.34MB/s elsewhere, which is better than the >60Mbit/s promised by my ISP.

It's possible that the problem is between my ISP and ci.guix.gnu.org. If I `wget` directly:

```
$ wget https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
--2021-03-05 18:56:21--  https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0       0%[                                                                                                      ] 167.79K  13.4KB/s    eta 69m 20s^C
```

HOWEVER, if I `torify wget`:

```
$ torify wget https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
--2021-03-05 18:57:00--  https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1    13%[============>                                                                                         ]   7.13M   683KB/s    eta 72s    ^C
```

Which is a big WHAT THE FUCK? Tor should not be 45x faster (683KB/s) than directly from my ISP (13KB/s). I am not using any special workarounds for Tor, like meek or obfs4, which makes me doubt that my ISP is attempting to censor --- if my ISP is the one doing the censoring, it would probably target Tor first before ci.guix.gnu.org, but I can access Tor just fine without special workarounds needed for very oppressive countries.

I've tried this a few times.  Consistently, using `torify wget` is much faster than `wget` alone, I don't have to bang my head in boredom with `torify wget`.  This is not my expectation given what I understand of the Tor network and the public network.

Is `ci.guix.gnu.org` doing some kind of per-IP or per-ISP or other throttling?  This looks very suspicious given the massive speed difference when using Tor and not using Tor.

Doing `torify guix package -m manifest.scm` does not seem to perform the download inside Tor, which makes sense since it's probably `guix-daemon` that is doing the download.

So my questions are:

* WHY IS DIRECT DOWNLOAD FROM MY ISP SLOWER THAN TOR? What can I do to check if it's my ISP that is attempting to censor ci.guix.gnu.org?
* Is there a way to make `guix-daemon` use a Tor proxy?  I have two systems using Guix, one is a Guix System, the other is using a foreign distro, and I'd like to adjust both to use Tor instead since it's faster.
  * If not, is there a way to tell `guix-daemon` that I have these `nar` files and it can put them in its store somehow instead of me waiting for it to ***SLOOOOOOOOWLY*** download it?
* Are there any mirrors I can use for substitutes instead? How do I change my systems (both the Guix System and the foreign distro) to use the mirrors? Is there some configuration option to do this?
* How hard would it be to make the substitute download process at least make an effort to attempt to continue a failed download instead of failing immediately and forcing a restart from 0%? `texlive` is something like 400Mb, it takes hours at 10KiB/s, I once had it fail at >90% which was very painful because it had to restart from 0%, I had to do `guix package -m manifest.scm` in a loop over and over again until it finished download. It would probably cut down on people hammering on the server as well.

Thanks
raid5atemyhomework