GNU bug report logs - #26201
Downloading substitutes is too slow upon nginx cache misses

Previous Next

Full log

View this message in rfc822 format

From: ludo <at> gnu.org (Ludovic Courtès)
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 26201 <at> debbugs.gnu.org, guix-sysadmin <at> gnu.org
Subject: bug#26201: Bandwidth when retrieving substitutes
Date: Mon, 27 Mar 2017 13:20:42 +0200

Hi there!

ludo <at> gnu.org (Ludovic Courtès) skribis:

> ‘guix publish’ should be faster and less resource-hungry than Hydra.  It
> uses in-process gzip for nar compression instead of bzip2 (I chose level
> 7, which seems to provide compression ratios close to what bzip2
> provides with its default compression level, while being 3 times
> faster).  Unlike Hydra it never forks so for instance, 404 responses for
> .narinfo URLs should be quicker.  Hopefully, that will improve the
> worst-case (cache miss) throughput.

Another interesting data point on the client side this time:

--8<---------------cut here---------------start------------->8---
$ wget -O- https://mirror.hydra.gnu.org/nar/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 |bunzip2 >/dev/null
--2017-03-27 13:12:50--  https://mirror.hydra.gnu.org/nar/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17
Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 2001:4ca0:2001:10:225:90ff:fedb:c720
Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-nix-archive]
Saving to: ‘STDOUT’

-                                [                            <=>                ]  53.01M  9.29MB/s    in 5.5s    

2017-03-27 13:12:55 (9.57 MB/s) - written to stdout [55582050]

$ wget -O- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 |gunzip >/dev/null
--2017-03-27 13:13:00--  https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17
Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 2001:4ca0:2001:10:225:90ff:fedb:c720
Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-nix-archive]
Saving to: ‘STDOUT’

-                                [        <=>                                    ]  59.19M  40.8MB/s    in 1.4s    

2017-03-27 13:13:02 (40.8 MB/s) - written to stdout [62068901]

$ wget -O- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 >/dev/null
--2017-03-27 13:15:58--  https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17
Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 2001:4ca0:2001:10:225:90ff:fedb:c720
Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-nix-archive]
Saving to: ‘STDOUT’

-                                [       <=>                                     ]  59.19M  42.5MB/s    in 1.4s    

2017-03-27 13:16:00 (42.5 MB/s) - written to stdout [62068901]
--8<---------------cut here---------------end--------------->8---

40 MB/s vs. 10 MB/s!  (Both items were cached on mirror.hydra.gnu.org.)

IOW, bunzip2 was the bottleneck when retrieving substitutes (and that’s
on an i7.)  With ‘perf timechart’ we see that bunzip2 is indeed busy
all the time right from the start.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.