bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework]

Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using 
computers.

One is a pure Guix System install without any channels, the other one is a 
foreign Guix install with the-channel-that-cannot-be-named.


```
downloading from 
https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
 wine64-6.0  54.4MiB                                                            
                                                                     10KiB/s 
01:55 [                  ]   2.1%
```

This can be very slow, including as slow as 4KiB/s at times. This is fairly 
awful since sometimes I get this result:

```
downloading from 
https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
 wine64-6.0  54.4MiB                                                            
                                                                     49KiB/s 
06:01 [#####             ]  31.9%guix substitute: error: TLS error in procedure 
'read_from_session_record_port': Error decoding the received TLS packet.
substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 failed
guix package: error: some substitutes for the outputs of derivation 
`/gnu/store/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually 
happens due to networking issues); try `--fallback' to build derivation from 
source
```

Because of the failed incomplete download, I have to ***restart the download 
again from 0%***, which is awful, awful UX. It would be really nice if:

1. The guix download process would make an effort to ***retry*** this a few 
times.
2. Continue the previous download instead of restarting from 0%.

The problem is not on my ISP, or at least not solely on my ISP. Doing a `wget` 
from `github.com`:

```
$ wget 
https://github.com/zfsonlinux/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
--2021-03-05 17:39:46--  
https://github.com/zfsonlinux/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
Resolving github.com (github.com)... 13.229.188.59
Connecting to github.com (github.com)|13.229.188.59|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: 
https://github.com/openzfs/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz 
[following]
--2021-03-05 17:39:47--  
https://github.com/openzfs/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: 
https://github-releases.githubusercontent.com/437011/71526a80-6ba3-11eb-893f-4ceb55b479d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210305%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210305T093947Z&X-Amz-Expires=300&X-Amz-Signature=9454446348219aa0731af009a5232bf5772e3d45a3a34052fa61f64f04c3c979&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=437011&response-content-disposition=attachment%3B%20filename%3Dzfs-2.0.3.tar.gz&response-content-type=application%2Foctet-stream
 [following]
--2021-03-05 17:39:47--  
https://github-releases.githubusercontent.com/437011/71526a80-6ba3-11eb-893f-4ceb55b479d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210305%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210305T093947Z&X-Amz-Expires=300&X-Amz-Signature=9454446348219aa0731af009a5232bf5772e3d45a3a34052fa61f64f04c3c979&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=437011&response-content-disposition=attachment%3B%20filename%3Dzfs-2.0.3.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com 
(github-releases.githubusercontent.com)... 185.199.108.154, 185.199.109.154, 
185.199.110.154, ...
Connecting to github-releases.githubusercontent.com 
(github-releases.githubusercontent.com)|185.199.108.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13114404 (13M) [application/octet-stream]
Saving to: ‘zfs-2.0.3.tar.gz’

zfs-2.0.3.tar.gz                                
100%[=====================================================================================================>]
  12.51M  9.34MB/s    in 1.3s

2021-03-05 17:39:49 (9.34 MB/s) - ‘zfs-2.0.3.tar.gz’ saved [13114404/13114404]

```

As can be seen above, I get 9.34MB/s elsewhere, which is better than the 
>60Mbit/s promised by my ISP.

It's possible that the problem is between my ISP and ci.guix.gnu.org. If I 
`wget` directly:

```
$ wget 
https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
--2021-03-05 18:56:21--  
https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0       0%[                           
                                                                           ] 
167.79K  13.4KB/s    eta 69m 20s^C
```

HOWEVER, if I `torify wget`:

```
$ torify wget 
https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
--2021-03-05 18:57:00--  
https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1    13%[============>              
                                                                           ]   
7.13M   683KB/s    eta 72s    ^C
```

Which is a big WHAT THE FUCK? Tor should not be 45x faster (683KB/s) than 
directly from my ISP (13KB/s). I am not using any special workarounds for Tor, 
like meek or obfs4, which makes me doubt that my ISP is attempting to censor 
--- if my ISP is the one doing the censoring, it would probably target Tor 
first before ci.guix.gnu.org, but I can access Tor just fine without special 
workarounds needed for very oppressive countries.

I've tried this a few times.  Consistently, using `torify wget` is much faster 
than `wget` alone, I don't have to bang my head in boredom with `torify wget`.  
This is not my expectation given what I understand of the Tor network and the 
public network.

Is `ci.guix.gnu.org` doing some kind of per-IP or per-ISP or other throttling?  
This looks very suspicious given the massive speed difference when using Tor 
and not using Tor.

Doing `torify guix package -m manifest.scm` does not seem to perform the 
download inside Tor, which makes sense since it's probably `guix-daemon` that 
is doing the download.

So my questions are:

* WHY IS DIRECT DOWNLOAD FROM MY ISP SLOWER THAN TOR? What can I do to check if 
it's my ISP that is attempting to censor ci.guix.gnu.org?
* Is there a way to make `guix-daemon` use a Tor proxy?  I have two systems 
using Guix, one is a Guix System, the other is using a foreign distro, and I'd 
like to adjust both to use Tor instead since it's faster.
  * If not, is there a way to tell `guix-daemon` that I have these `nar` files 
and it can put them in its store somehow instead of me waiting for it to 
***SLOOOOOOOOWLY*** download it?
* Are there any mirrors I can use for substitutes instead? How do I change my 
systems (both the Guix System and the foreign distro) to use the mirrors? Is 
there some configuration option to do this?
* How hard would it be to make the substitute download process at least make an 
effort to attempt to continue a failed download instead of failing immediately 
and forcing a restart from 0%? `texlive` is something like 400Mb, it takes 
hours at 10KiB/s, I once had it fail at >90% which was very painful because it 
had to restart from 0%, I had to do `guix package -m manifest.scm` in a loop 
over and over again until it finished download. It would probably cut down on 
people hammering on the server as well.

Thanks
raid5atemyhomework