From unknown Sun Jun 22 11:33:44 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#26201 <26201@debbugs.gnu.org> To: bug#26201 <26201@debbugs.gnu.org> Subject: Status: Downloading substitutes is too slow upon nginx cache misses Reply-To: bug#26201 <26201@debbugs.gnu.org> Date: Sun, 22 Jun 2025 18:33:44 +0000 retitle 26201 Downloading substitutes is too slow upon nginx cache misses reassign 26201 guix submitter 26201 severity 26201 important tag 26201 fixed thanks From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 20 21:45:10 2017 Received: (at submit) by debbugs.gnu.org; 21 Mar 2017 01:45:10 +0000 Received: from localhost ([127.0.0.1]:37032 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq8rG-0000QF-KC for submit@debbugs.gnu.org; Mon, 20 Mar 2017 21:45:10 -0400 Received: from eggs.gnu.org ([208.118.235.92]:35280) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq8rE-0000Q2-Tx for submit@debbugs.gnu.org; Mon, 20 Mar 2017 21:45:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cq8r8-0006qJ-QS for submit@debbugs.gnu.org; Mon, 20 Mar 2017 21:45:03 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:34164) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cq8r8-0006pt-Nl for submit@debbugs.gnu.org; Mon, 20 Mar 2017 21:45:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54450) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cq8r7-0004Iw-H1 for bug-guix@gnu.org; Mon, 20 Mar 2017 21:45:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cq8r4-0006l6-GY for bug-guix@gnu.org; Mon, 20 Mar 2017 21:45:01 -0400 Received: from sender-pp-092.zoho.com ([135.84.80.237]:25412) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cq8r4-0006kI-8h for bug-guix@gnu.org; Mon, 20 Mar 2017 21:44:58 -0400 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:to:subject:message-id:mime-version:content-type; b=YtcmAhmjaKFHHsuuDhkTY8sbVZGoLDgWu53kBlTwfM9w3pPjDCklK56Vnm0sYdgNQak3mBRmS/RV UwLfLk2xKdvsMhJBBuD20R8rmxX91sNHuSGUTfZeZMdYEcG21V1u Received: from khaalida (ip68-96-178-131.lv.lv.cox.net [68.96.178.131]) by mx.zohomail.com with SMTPS id 1490060694914290.2472656794157; Mon, 20 Mar 2017 18:44:54 -0700 (PDT) Date: Mon, 20 Mar 2017 18:44:49 -0700 From: To: GuixSD Subject: No notification of cache misses when downloading substitutes Message-ID: <20170320184449.5ac06051@khaalida> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) Just ran guix pull and guix package -u, and found some of the programs download VERY slowly (<100kb/s, usually around 95). I asked on #guix and lfam mentioned it was probably a cache miss. It would be nice if there was some notification that a cache miss happened and the download will likely be slow, otherwise a user might wonder what problem there is with their connection. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 20 22:45:35 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 02:45:36 +0000 Received: from localhost ([127.0.0.1]:37073 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq9nj-0001x1-Ml for submit@debbugs.gnu.org; Mon, 20 Mar 2017 22:45:35 -0400 Received: from relay4-d.mail.gandi.net ([217.70.183.196]:33961) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq9nh-0001ws-LQ for 26201@debbugs.gnu.org; Mon, 20 Mar 2017 22:45:34 -0400 Received: from mfilter11-d.gandi.net (mfilter11-d.gandi.net [217.70.178.131]) by relay4-d.mail.gandi.net (Postfix) with ESMTP id 7224A17209A; Tue, 21 Mar 2017 03:45:32 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at mfilter11-d.gandi.net Received: from relay4-d.mail.gandi.net ([IPv6:::ffff:217.70.183.196]) by mfilter11-d.gandi.net (mfilter11-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id LWI5u5KAfC2H; Tue, 21 Mar 2017 03:45:30 +0100 (CET) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 94CA1172094; Tue, 21 Mar 2017 03:45:30 +0100 (CET) Subject: Re: bug#26201: No notification of cache misses when downloading substitutes To: dian_cecht@zoho.com References: <20170320184449.5ac06051@khaalida> From: Tobias Geerinckx-Rice Message-ID: <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> Date: Tue, 21 Mar 2017 03:46:29 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170320184449.5ac06051@khaalida> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="38N86beiqjIh8TDoORNeIu1L6lc0qIjTb" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --38N86beiqjIh8TDoORNeIu1L6lc0qIjTb Content-Type: multipart/mixed; boundary="7jM5ikhO7OUdTV4lrvgqnhxtRHF8UlrGf"; protected-headers="v1" From: Tobias Geerinckx-Rice To: dian_cecht@zoho.com Cc: 26201@debbugs.gnu.org Message-ID: <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> In-Reply-To: <20170320184449.5ac06051@khaalida> --7jM5ikhO7OUdTV4lrvgqnhxtRHF8UlrGf Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hullo, On 21/03/17 02:44, dian_cecht@zoho.com wrote: > Just ran guix pull and guix package -u, and found some of the programs > download VERY slowly (<100kb/s, usually around 95). I asked on #guix > and lfam mentioned it was probably a cache miss. Do you mean that *substitutes* existed, but were not yet on mirror.hydra.gnu.org and so were silently proxied from the much slower hydra.gnu.org? Or did Guix fall back to downloading *source* tarballs from some slow upstream to build locally? (I've no access to IRC at the mo'.) Kind regards, T G-R --7jM5ikhO7OUdTV4lrvgqnhxtRHF8UlrGf-- --38N86beiqjIh8TDoORNeIu1L6lc0qIjTb Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY0JQGDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5kLsQf+Jrkb Fi/NrrYlQ3zSjdCtQKnRsXUtHP+gh6t+w8d2RWryW1xSwNzJWZ+aTaKaopVFyZhW mjEGLUjdRZttXn0WfeXeBmu0fnFqYOrPduEkCug+y/Jbu5eu6hgIHIb5v6SAXA/i yb1Z9MlFUdziI8XJ7ZJdsYLByIxe1sZl1F29HxvjDTKX/UBCA21m6wwLCUsqK+5M VVw4GY87y6GsJV8RMLj1GSccHFJES/pic3YcFjsa87DAjL5P0vdNGPXpHvrkFK/5 bl52x0msp8incaPE/wCzfHyMnErUp6md/5hzYFVEvNc9uggLZx2HKNHc2JeQCXZZ BioXmgJix6MW5TNihg== =2rAl -----END PGP SIGNATURE----- --38N86beiqjIh8TDoORNeIu1L6lc0qIjTb-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 20 22:52:58 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 02:52:58 +0000 Received: from localhost ([127.0.0.1]:37077 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq9us-00027f-Gq for submit@debbugs.gnu.org; Mon, 20 Mar 2017 22:52:58 -0400 Received: from sender-pp-092.zoho.com ([135.84.80.237]:25415) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq9up-00027T-MD for 26201@debbugs.gnu.org; Mon, 20 Mar 2017 22:52:56 -0400 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:to:cc:subject:message-id:in-reply-to:references:mime-version:content-type; b=glqAQLuY6SEDqIH2246gzpl3abhAsejHl9SgObp0GAm/bPnA9zNtvkH5KeG6pJx86SD2LyBCA/NI AgF6GIjc0723W4U1CPo7t8eC/gVG1wM8oiQTrGmo6w/lVzt+bl7O Received: from khaalida (ip68-96-178-131.lv.lv.cox.net [68.96.178.131]) by mx.zohomail.com with SMTPS id 1490064772226856.9671126217788; Mon, 20 Mar 2017 19:52:52 -0700 (PDT) Date: Mon, 20 Mar 2017 19:52:47 -0700 From: To: Tobias Geerinckx-Rice Subject: Re: bug#26201: No notification of cache misses when downloading substitutes Message-ID: <20170320195247.05f72fc9@khaalida> In-Reply-To: <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Score: -2.8 (--) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.8 (--) On Tue, 21 Mar 2017 03:46:29 +0100 Tobias Geerinckx-Rice wrote: > Hullo, > > On 21/03/17 02:44, dian_cecht@zoho.com wrote: > > Just ran guix pull and guix package -u, and found some of the > > programs download VERY slowly (<100kb/s, usually around 95). I > > asked on #guix and lfam mentioned it was probably a cache miss. > > Do you mean that *substitutes* existed, but were not yet on > mirror.hydra.gnu.org and so were silently proxied from the much slower > hydra.gnu.org? The URL displayed during the download was mirror.hydra.gnu.org. > > Or did Guix fall back to downloading *source* tarballs from some slow > upstream to build locally? It was a binary download, not source. At least, I don't recall anything about compiles at any point (and I'm sure it didn't take long enough to do that; one package was icecat which I'm sure wouldn't have downloaded at 90k/s then compiled in less than 15 minutes (fwiw, according to my build logs firefox takes about 2 hours to build, so unless icecat is magically orders of magnitude faster to build, then I'm sure it was just a download + install, and not download + compile + install) From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 20 23:56:16 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 03:56:16 +0000 Received: from localhost ([127.0.0.1]:37103 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqAu8-0005Hv-0e for submit@debbugs.gnu.org; Mon, 20 Mar 2017 23:56:16 -0400 Received: from relay4-d.mail.gandi.net ([217.70.183.196]:37832) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqAu5-0005Hn-L6 for 26201@debbugs.gnu.org; Mon, 20 Mar 2017 23:56:14 -0400 Received: from mfilter15-d.gandi.net (mfilter15-d.gandi.net [217.70.178.143]) by relay4-d.mail.gandi.net (Postfix) with ESMTP id 6DF6B17209D; Tue, 21 Mar 2017 04:56:12 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at mfilter15-d.gandi.net Received: from relay4-d.mail.gandi.net ([IPv6:::ffff:217.70.183.196]) by mfilter15-d.gandi.net (mfilter15-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id c7D476y2m06v; Tue, 21 Mar 2017 04:56:10 +0100 (CET) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 8207C172094; Tue, 21 Mar 2017 04:56:10 +0100 (CET) Subject: Re: bug#26201: No notification of cache misses when downloading substitutes To: dian_cecht@zoho.com References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> From: Tobias Geerinckx-Rice Message-ID: <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> Date: Tue, 21 Mar 2017 04:57:09 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170320195247.05f72fc9@khaalida> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Ci71Ap23b3k0qo0Vex16PJ0aN6XRu8WjD" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Ci71Ap23b3k0qo0Vex16PJ0aN6XRu8WjD Content-Type: multipart/mixed; boundary="8LfU2BlEuEaS6sf37mggES7AMiMmrLPCs"; protected-headers="v1" From: Tobias Geerinckx-Rice To: dian_cecht@zoho.com Cc: 26201@debbugs.gnu.org Message-ID: <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> In-Reply-To: <20170320195247.05f72fc9@khaalida> --8LfU2BlEuEaS6sf37mggES7AMiMmrLPCs Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ahoy, On 21/03/17 03:52, dian_cecht@zoho.com wrote: > The URL displayed during the download was mirror.hydra.gnu.org. > [...] It was a binary download, not source. Oh, OK. I'm not an expert on how Hydra's set up these days, but will assume it's not too different from my own (a fast nginx proxy_cache, mirror.hydra.gnu.org, in front of a slower build farm, hydra.gnu.org). Whenever you're the first to request a substitute, mirror.hydra.gnu.org transparently forwards the request to hydra.gnu.org. The latter has to compress the response on the fly, leading to much slower transfer speeds. It slowly sends it back to the mirror, which slowly sends it on to you while also saving it on disc so all subsequent downloads will be fast =E2=80=94 by Hydra standards =E2=80=93 and not inv= olve hydra.gnu.org. Maybe you knew all this, but it's also the reason that... > On 21/03/17 02:44, dian_cecht@zoho.com wrote: > It would be nice if there was some notification that a cache miss > happened and the download will likely be slow, otherwise a user might > wonder what problem there is with their connection. =2E..I'm afraid this makes no sense from guix's point of view. The term =E2=80=98cache miss=E2=80=99 here is an implementation detail of= our current Hydra set-up, not something guix can or IMO should care about. There are hundreds of reasons why your connection might be slow at any given time. Guix should just tell you so (it does), not guess why. Or worse: know. (But if others disagree, we'll have to extend the Hydra API to somehow relay this information to the client, in the spirit of the modern Web.) HTTP 200=C2=BD: OK, fine, but it's Going to Suck. T G-R --8LfU2BlEuEaS6sf37mggES7AMiMmrLPCs-- --Ci71Ap23b3k0qo0Vex16PJ0aN6XRu8WjD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY0KSWDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5nmgAf/Z43j mtjWxZGLbwnEvaT8hnygDeUH3ml2BnOD52JaAAjVMbTwt9RsFdS1xq90hthza+bR HdWehFTg/W7gYvEh8qPPmdMV6mWdZTI0nMm8wSPX4dx9ljK9zrqLikaDPs+BeAsL K5AF8/rw9dhnfNBe6JK+eNtj6cmFI0L2J4RZmLyihz13NU58ojl+jrW9MBO2XB4l ZCBl8eDc9pBTTKfhWR9LueIJgdl9lDs0ZZeMksDNfZTnXiFQqHxiyGZzx7u8xjU/ StDjXiGB7p+UAiLbvvZxzEaFh8xBwoCkDqMl9J4aZN6+thLv5UTd8vDKqdkBwgNg AUKraRoPp3/RRL6E9g== =WI91 -----END PGP SIGNATURE----- --Ci71Ap23b3k0qo0Vex16PJ0aN6XRu8WjD-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 00:48:18 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 04:48:19 +0000 Received: from localhost ([127.0.0.1]:37116 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqBiU-0006TQ-Kw for submit@debbugs.gnu.org; Tue, 21 Mar 2017 00:48:18 -0400 Received: from sender-pp-092.zoho.com ([135.84.80.237]:25437) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqBiT-0006TJ-G0 for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 00:48:18 -0400 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:to:cc:subject:message-id:in-reply-to:references:mime-version:content-type; b=DZmkwU+YbEL7WPUEqMn9m2Rj9Ur8VfeqYUClt+PwyahMRSIFI20kTJc6mrRo5VaTbMFM/GTIk+nU RRqaWMxTmT3EFT/+Obc+c6SAxpbEcLZBKrMUCLtw2h/kbqVST0Df Received: from khaalida (ip68-96-178-131.lv.lv.cox.net [68.96.178.131]) by mx.zohomail.com with SMTPS id 149007169377356.391909041920485; Mon, 20 Mar 2017 21:48:13 -0700 (PDT) Date: Mon, 20 Mar 2017 21:48:09 -0700 From: To: Tobias Geerinckx-Rice Subject: Re: bug#26201: No notification of cache misses when downloading substitutes Message-ID: <20170320214809.466dc5fe@khaalida> In-Reply-To: <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.8 (--) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.8 (--) On Tue, 21 Mar 2017 04:57:09 +0100 Tobias Geerinckx-Rice wrote: > Ahoy, >=20 > On 21/03/17 03:52, dian_cecht@zoho.com wrote: > > The URL displayed during the download was mirror.hydra.gnu.org. > > [...] It was a binary download, not source. =20 >=20 > Oh, OK. I'm not an expert on how Hydra's set up these days, but will > assume it's not too different from my own (a fast nginx proxy_cache, > mirror.hydra.gnu.org, in front of a slower build farm, hydra.gnu.org). >=20 > Whenever you're the first to request a substitute, > mirror.hydra.gnu.org transparently forwards the request to > hydra.gnu.org. >=20 > The latter has to compress the response on the fly, leading to much > slower transfer speeds. It slowly sends it back to the mirror, which > slowly sends it on to you while also saving it on disc so all > subsequent downloads will be fast =E2=80=94 by Hydra standards =E2=80=93 = and not > involve hydra.gnu.org. >=20 > Maybe you knew all this, but it's also the reason that... I'm not familiar with the implementation details, nor how hydra is currently setup. > > On 21/03/17 02:44, dian_cecht@zoho.com wrote: > > It would be nice if there was some notification that a cache miss > > happened and the download will likely be slow, otherwise a user > > might wonder what problem there is with their connection. =20 >=20 > ...I'm afraid this makes no sense from guix's point of view. >=20 > The term =E2=80=98cache miss=E2=80=99 here is an implementation detail of= our current > Hydra set-up, not something guix can or IMO should care about. There > are hundreds of reasons why your connection might be slow at any > given time. Guix should just tell you so (it does), not guess why. Or > worse: know. I'm not suggesting having Guix tell me why my network is slow, only if the download might be slow because it's having to pull from hydra.gnu.org. Having Guix automagically troubleshoot networking problems is well beyond the scope of a package manager, even one that goes as far beyond simple package management as Guix does. >=20 > (But if others disagree, we'll have to extend the Hydra API to somehow > relay this information to the client, in the spirit of the modern > Web.) AFAIK, Guix devs are working on a replacement for the current build system, so the sane option wouldn't be extending the current hydra system to handle a new API call, but to try and work this type of feature into the next system. Unless, of course, something like this could be done in hydra reasonably easily, in which case why not. Another option would be to have the mirrors automatically cache the files as soon as they are available to try. I'd hope this would be how things are handled already, but one never knows. From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 02:21:04 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 06:21:04 +0000 Received: from localhost ([127.0.0.1]:37136 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqDAG-0000HS-BD for submit@debbugs.gnu.org; Tue, 21 Mar 2017 02:21:04 -0400 Received: from relay5-d.mail.gandi.net ([217.70.183.197]:49510) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqDAD-0000Gh-Bx for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 02:21:02 -0400 Received: from mfilter16-d.gandi.net (mfilter16-d.gandi.net [217.70.178.144]) by relay5-d.mail.gandi.net (Postfix) with ESMTP id 53B5141C084; Tue, 21 Mar 2017 07:21:00 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at mfilter16-d.gandi.net Received: from relay5-d.mail.gandi.net ([IPv6:::ffff:217.70.183.197]) by mfilter16-d.gandi.net (mfilter16-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id l1ohN7-mzIyU; Tue, 21 Mar 2017 07:20:58 +0100 (CET) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id CD35B41C090; Tue, 21 Mar 2017 07:20:56 +0100 (CET) Subject: Re: bug#26201: No notification of cache misses when downloading substitutes To: dian_cecht@zoho.com References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> From: Tobias Geerinckx-Rice Message-ID: Date: Tue, 21 Mar 2017 07:21:54 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170320214809.466dc5fe@khaalida> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="R6TLt5K1Pmfwjmw2et0R22cMa2xBPwWKc" X-Spam-Score: -3.5 (---) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.5 (---) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --R6TLt5K1Pmfwjmw2et0R22cMa2xBPwWKc Content-Type: multipart/mixed; boundary="xlSTDx9slGiGVIPnNAG6IpFnWTk6xjlKh"; protected-headers="v1" From: Tobias Geerinckx-Rice To: dian_cecht@zoho.com Cc: 26201@debbugs.gnu.org Message-ID: Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> In-Reply-To: <20170320214809.466dc5fe@khaalida> --xlSTDx9slGiGVIPnNAG6IpFnWTk6xjlKh Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mornin', On 21/03/17 05:48, dian_cecht@zoho.com wrote: > I'm not suggesting having Guix tell me why my network is slow, I never mentioned your network. Your proxied connection to a substitute server, yes. And, well, this very bug report is for Guix to tell you why that's slow... > only if the download might be slow because it's having to pull from=20 > hydra.gnu.org. (Side note: =E2=80=98it=E2=80=99 here is mirror.hydra.gnu.org, never a we= ll-configured Guix client.) So to implement this, the client would need to display a =E2=80=98warning= =E2=80=98 message or flag sent by the substitute server, to notify the user that their download might be slower... sometimes... by an unknown amount... possibly? But see, that wouldn't be true at all on my system (and surely others), despite being set up nearly identically to Hydra. On the other hand, my home download speed fluctuates wildly, even between simultaneous connections to the same server. Whether or not a file is cached makes no difference. To be told would be noise at best, misleading at worst. I'd be against this only for those reasons, but I promise I'm not. It's just all a bit vague, 's all, and my personal opinion is that once the vagueness is resolved, not much will remain. But who knows. > AFAIK, Guix devs are working on a replacement for the current build=20 > system, so the sane option wouldn't be extending the current hydra=20 > system to handle a new API call, but to try and work this type of=20 > feature into the next system. My point is that it wouldn't be sane, and would be an ugly hack in either system. Cuirass isn't really different from Hydra is this regard. Me shut up now :-) I'm more interested in what others have to say. Kind regards, T G-R --xlSTDx9slGiGVIPnNAG6IpFnWTk6xjlKh-- --R6TLt5K1Pmfwjmw2et0R22cMa2xBPwWKc Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY0MaCDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5my6Qf9ECb0 bwg1kgeoIJYPuyVLIte3wezvwFbT+wSXPbjaeckg8zA8nmJpwydHdQeAw3Aj7lAZ mvciAsA0WqNVuNi/9X6u5O9dTgUHEnl0mChDTKI8/jUbH6edPlPpnLt8JWg7ZVkk aOCtIKF2lB5gTJrC3b4I5fdcEjtN63KHlA6YibXmYblNAUxwb42/FQquWCvwXytY C4AmAcITHcWyghdTMAYflQWeEZFydX5JXVvU7toMa0V1eH2kUhUxiaMR2J+VWHij z4lw7hPsAUOvSwOamIB+KwDUdhdC12GPRes9YGyIuEaUdgcf7/wvjnj4O4rjgnZe VWMYMirrwDn6rk3nmA== =OIdm -----END PGP SIGNATURE----- --R6TLt5K1Pmfwjmw2et0R22cMa2xBPwWKc-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 02:49:25 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 06:49:25 +0000 Received: from localhost ([127.0.0.1]:37141 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqDbg-0000uq-PH for submit@debbugs.gnu.org; Tue, 21 Mar 2017 02:49:25 -0400 Received: from sender-pp-092.zoho.com ([135.84.80.237]:25444) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqDbe-0000uh-Po for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 02:49:23 -0400 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:to:cc:subject:message-id:in-reply-to:references:mime-version:content-type; b=ZJbmP4JnQhcuTDjtRsiKiOGf6dfHPoqdm0YIeEAzM/9H65Ss6eW6r4ksMqERBvgtd/KaBmi8Sxw/ f04PuhlKBbUsdH0IbNftwe19AmVC07neys9eFlRRXbtcS0yoHP3S Received: from khaalida (ip68-96-178-131.lv.lv.cox.net [68.96.178.131]) by mx.zohomail.com with SMTPS id 1490078961406635.3972463931003; Mon, 20 Mar 2017 23:49:21 -0700 (PDT) Date: Mon, 20 Mar 2017 23:49:12 -0700 From: To: Tobias Geerinckx-Rice Subject: Re: bug#26201: No notification of cache misses when downloading substitutes Message-ID: <20170320234912.46680062@khaalida> In-Reply-To: References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.8 (--) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.8 (--) On Tue, 21 Mar 2017 07:21:54 +0100 Tobias Geerinckx-Rice wrote: > > only if the download might be slow because [mirror.hydra] is having > > to pull from hydra.gnu.org. =20 >=20 > So to implement this, the client would need to display a =E2=80=98warning= =E2=80=98 > message or flag sent by the substitute server, to notify the user that > their download might be slower... sometimes... by an unknown amount... > possibly? Simply a notification that mirror.hydra doesn't currently have a cached version of the file and the download might be slower than normal would be fine. As-is, looking up and seeing download speeds that amount to less than 10% of one's normal bandwidth is a bit concerning since it would seem like there is a problem. In this case, Guix would be giving the user some notification that something /is/ out of the ordinary, and possibly save the user some effort trying to determine the cause of the slowdown. > But see, that wouldn't be true at all on my system (and surely > others), despite being set up nearly identically to Hydra. On the > other hand, my home download speed fluctuates wildly, even between > simultaneous connections to the same server. I'm not sure how any of this matters. If you are running a local Hydra instance or whatever, then I'd assume you'd be aware of what, if any, problems that could arise. In this case, I'd hope hydra would allow you to disable this feature. > Whether or not a file is cached makes no difference. To be told would > be noise at best, is leading at worst. Had I been notified that mirror.hydra was currently pulling from hydra, it would have saved me the time of jumping on IRC and asking what was up, which only worked because someone was in #guix and had an idea of what was going on; had that not been the case, I would have started looking for the cause for the slowdown and wasted several minutes (at least) trying to figure out what was wrong, and since it was on mirror.hydra's end, I'd have no way to know the slowdown was on their end and not mine, nor my ISP's problem. > > AFAIK, Guix devs are working on a replacement for the current build=20 > > system, so the sane option wouldn't be extending the current hydra=20 > > system to handle a new API call, but to try and work this type of=20 > > feature into the next system. =20 >=20 > My point is that it wouldn't be sane, and would be an ugly hack in > either system. I don't see how this would have to be "an ugly hack". It's simply a query and response. The simplest way I can see for this to work would be for mirror.hydra to either just send the requested file, or a response that the file isn't cached then start to trickle the file on to the client.=20 From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 08:59:32 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 12:59:32 +0000 Received: from localhost ([127.0.0.1]:37314 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqJNr-0003IS-Nq for submit@debbugs.gnu.org; Tue, 21 Mar 2017 08:59:31 -0400 Received: from pelzflorian.de ([5.45.111.108]:41102 helo=mail.pelzflorian.de) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqJNp-0003IK-Bv for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 08:59:30 -0400 Received: from [131.246.68.214] (eduroam-ipv4-5-0214.triple-a.uni-kl.de [131.246.68.214]) by mail.pelzflorian.de (Postfix) with ESMTPSA id 07187360010; Tue, 21 Mar 2017 13:59:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=pelzflorian.de; s=mail; t=1490101168; bh=xwcSczW8YMyENXIDvb4YF9+Grvwe6iBvt1a/Yh0LFPk=; h=From:Subject:To:References:Cc:Date:In-Reply-To; b=DceIKGa5ERy+6a7exnkjSYT2hc+n4xuZzCVmWTZ0B3WijZgupWrTDaMzayUWmzqiZ mWhVphY/YdP+exj4sY6re1R3F1aUncaXFw6vUTDnPfCThBGuWMpeUZLvo6bLNQvTZ/ 0ueJAjyHaRqMQq3ChMTN70Rftnt5LnBKJ6fOagnw= From: Florian Pelz Subject: Re: bug#26201: No notification of cache misses when downloading substitutes To: dian_cecht@zoho.com, Tobias Geerinckx-Rice References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> Message-ID: Date: Tue, 21 Mar 2017 13:59:27 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170320214809.466dc5fe@khaalida> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On Mon, 2017-03-20 at 21:48 -0700, dian_cecht@zoho.com wrote: > Another option would be to have the mirrors automatically cache the > files as soon as they are available to try. I'd hope this would be how > things are handled already, but one never knows. > If it cached everything, it wouldn’t be a cache? From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 10:54:20 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 14:54:21 +0000 Received: from localhost ([127.0.0.1]:38085 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqLAy-00069Z-Fw for submit@debbugs.gnu.org; Tue, 21 Mar 2017 10:54:20 -0400 Received: from relay6-d.mail.gandi.net ([217.70.183.198]:57302) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqLAv-00069Q-G8 for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 10:54:18 -0400 Received: from mfilter29-d.gandi.net (mfilter29-d.gandi.net [217.70.178.160]) by relay6-d.mail.gandi.net (Postfix) with ESMTP id 58451FBA49; Tue, 21 Mar 2017 15:54:16 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at mfilter29-d.gandi.net Received: from relay6-d.mail.gandi.net ([IPv6:::ffff:217.70.183.198]) by mfilter29-d.gandi.net (mfilter29-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id MnrSY2K0TkNH; Tue, 21 Mar 2017 15:54:14 +0100 (CET) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 98F58FBA11; Tue, 21 Mar 2017 15:54:06 +0100 (CET) Subject: Re: bug#26201: No notification of cache misses when downloading substitutes To: dian_cecht@zoho.com References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> <20170320234912.46680062@khaalida> From: Tobias Geerinckx-Rice Message-ID: <1bbd8ee3-1745-3642-27ed-f095c732dc11@tobias.gr> Date: Tue, 21 Mar 2017 15:55:05 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170320234912.46680062@khaalida> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="sVtwlF7vBIdjQ4PVvfNB2UD97ITDdwMOc" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --sVtwlF7vBIdjQ4PVvfNB2UD97ITDdwMOc Content-Type: multipart/mixed; boundary="iu575CXagERv6leFegvDstT3Ehqs7xP5x"; protected-headers="v1" From: Tobias Geerinckx-Rice To: dian_cecht@zoho.com Cc: 26201@debbugs.gnu.org Message-ID: <1bbd8ee3-1745-3642-27ed-f095c732dc11@tobias.gr> Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> <20170320234912.46680062@khaalida> In-Reply-To: <20170320234912.46680062@khaalida> --iu575CXagERv6leFegvDstT3Ehqs7xP5x Content-Type: multipart/mixed; boundary="------------F5B05DD24CBE0FDA7ED8453B" This is a multi-part message in MIME format. --------------F5B05DD24CBE0FDA7ED8453B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hullo! On 21/03/17 07:49, dian_cecht@zoho.com wrote: > I'm not sure how any of this matters. If you are running a local=20 > Hydra instance or whatever, then I'd assume you'd be aware of what,=20 > if any, problems that could arise. It matters for the reasons mentioned. It's not a =E2=80=98local Hydra=E2=80= =99 & I have no idea what problems you're talking about. My problem is that every invocation of Guix already fills several screens with Guile cache misses. Adding another warning (=E2=80=98warning= ! the system is working exactly as designed!=E2=80=99) will only serve to make = those other warnings look less silly, and I think that would be a shame. To clarify: - Warnings should be scary because warnings should be actionable. There's nothing the user can or needs to do about a cache miss. - It would be randomly shown to everyone, since this happens constantly. - The behaviour warned about is not incorrect or abnormal. - As already noted, it's how caching works. > I don't see how this would have to be "an ugly hack". It's simply a=20 > query and response. The simplest way I can see for this to work would > be for mirror.hydra to either just send the requested file, or a > response that the file isn't cached then start to trickle the file on > to the client. Well, yeah... That's the ugly hack. :-) It's not that your suggestion's hard to implement. In fact, it's just one line for nginx (which it turns out I already had): add_header X-Cache-Status $upstream_cache_status; and 6 lines of lightly-tested Guile (attached)=C2=B9. And presto. This th= ing. Doesn't mean we should. Kind regards, T G-R =C2=B9: Why? Practice. Irony. Light masochism. --------------F5B05DD24CBE0FDA7ED8453B Content-Type: text/x-patch; name="0001-http-client-Warn-on-proxy-cache-misses.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="0001-http-client-Warn-on-proxy-cache-misses.patch" =46rom 6d459a442d73628a0628385283c7cf04dff1b797 Mon Sep 17 00:00:00 2001 From: Tobias Geerinckx-Rice Date: Tue, 21 Mar 2017 15:31:56 +0100 Subject: [PATCH] http-client: Warn on proxy cache misses. Still not a good idea. * guix/http-client.scm (http-fetch): Add #:peek-behind-proxy parameter to expose caching proxy implementation details as a scary warning. * guix/scripts/substitute.scm (fetch): Use it. --- guix/http-client.scm | 10 +++++++++- guix/scripts/substitute.scm | 3 ++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 6874c51..2366f5e 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Court=C3= =A8s ;;; Copyright =C2=A9 2015 Mark H Weaver ;;; Copyright =C2=A9 2012, 2015 Free Software Foundation, Inc. +;;; Copyright =C2=A9 2017 Tobias Geerinckx-Rice ;;; ;;; This file is part of GNU Guix. ;;; @@ -222,7 +223,8 @@ or if EOF is reached." =20 (define* (http-fetch uri #:key port (text? #f) (buffered? #t) keep-alive? (verify-certificate? #t) - (headers '((user-agent . "GNU Guile")))) + (headers '((user-agent . "GNU Guile"))) + (peek-behind-cache? #f)) "Return an input port containing the data at URI, and the expected num= ber of bytes available or #f. If TEXT? is true, the data at URI is considered = to be textual. Follow any HTTP redirection. When BUFFERED? is #f, return an @@ -253,8 +255,14 @@ Raise an '&http-get-error' condition if downloading = fails." (http-get uri #:streaming? #t #:port port #:keep-alive? #t #:headers headers)) + ((headers) + (response-headers resp)) ((code) (response-code resp))) + (when (and peek-behind-cache? + (equal? (assoc-ref headers 'x-cache-status) "MISS")) + (warning (_ "the caching proxy is working properly!~%")) + (warning (_ "and there's nothing you can do about it.~%"))= ) (case code ((200) (values data (response-content-length resp))) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index faeb019..4a4f115 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -216,7 +216,8 @@ provide." (unless (or buffered? (not (file-port? port))) (setvbuf port _IONBF))) (http-fetch uri #:text? #f #:port port - #:verify-certificate? #f)))))) + #:verify-certificate? #f + #:peek-behind-cache? #t)))))) (else (leave (_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) --=20 2.9.3 --------------F5B05DD24CBE0FDA7ED8453B-- --iu575CXagERv6leFegvDstT3Ehqs7xP5x-- --sVtwlF7vBIdjQ4PVvfNB2UD97ITDdwMOc Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY0T7KDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5mNSgf+NNsf Eneqqsg7f9QfkOW0iT6JxZ90JEcsVDLoUvx8YvBtOXwVvhnUxt1GQxu0o9yCUUAY 9nLOl+5cmE81Yjz2ibX6z/oUqp4kxzMO/uftnQyYpYKsOkIZppkVcuk2ksoPMrs+ c/kc0tULupvLIJMCWMAxOKAGryR0jsM3aeaq8BgpPLmlcUrR024Yn0RNM8j9vFHz 5b35tVv6M7pCV3VNQGg10fRid4h1OHbnV/KUALMSCuErRwIrg09J+4bllFODCf3M W2QHdYE1q7fn+1gakrkE3BaYeAVlMXu/uGEuR1fUpmK5nnOKdj1Rz+6mOvIAUEK3 qsXIXX8xm0GuiWOIIg== =M77U -----END PGP SIGNATURE----- --sVtwlF7vBIdjQ4PVvfNB2UD97ITDdwMOc-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 11:32:49 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 15:32:49 +0000 Received: from localhost ([127.0.0.1]:38121 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqLmD-000761-2j for submit@debbugs.gnu.org; Tue, 21 Mar 2017 11:32:49 -0400 Received: from sender-pp-092.zoho.com ([135.84.80.237]:25429) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqLmA-00075s-Uu for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 11:32:47 -0400 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:to:cc:subject:message-id:in-reply-to:references:mime-version:content-type; b=HZzJ8fW3A7oQL2woV/1kYNK6yhC9/zwKBXBmJ4B8idx2KHqlILAcw8jbJFYrTl6y+2NRICX0eTaL c83wDVLgCrsves+hzoatkfJ1wLJpI7/uGRIPHBfYLLp0oZ+pEvCl Received: from khaalida (ip68-96-178-131.lv.lv.cox.net [68.96.178.131]) by mx.zohomail.com with SMTPS id 149011036386214.82736455559882; Tue, 21 Mar 2017 08:32:43 -0700 (PDT) Date: Tue, 21 Mar 2017 08:32:39 -0700 From: To: Tobias Geerinckx-Rice Subject: Re: bug#26201: No notification of cache misses when downloading substitutes Message-ID: <20170321083239.3cbf1e8d@khaalida> In-Reply-To: <1bbd8ee3-1745-3642-27ed-f095c732dc11@tobias.gr> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> <20170320234912.46680062@khaalida> <1bbd8ee3-1745-3642-27ed-f095c732dc11@tobias.gr> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Score: -2.8 (--) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.8 (--) On Tue, 21 Mar 2017 15:55:05 +0100 Tobias Geerinckx-Rice wrote: > To clarify: > > - Warnings should be scary because warnings should be actionable. There are warnings and there are errors. Warnings don't have to be scary; I get them every time I update emacs because of duplicate icons stored in two different directories in the store. Is that actionable? Not as far as I am concerned, unless I want to hand delete something from the store, which, as far as I understand it, shouldn't be done. > There's nothing the user can or needs to do about a cache miss. Please reread the 2nd part of my response in Message #23 in this bugreport for why this is needed. > - It would be randomly shown to everyone, since this happens > constantly. Unless mirror.hydra randomly loses data in it's cache from hydra, it won't be random in the least. > - The behaviour warned about is not incorrect or abnormal. No, but the behavior would inform the user that the unusual and random slowdown isn't another problem and is because mirror.hydra is having to update it's cache, which, as I explained before, is useful information. > [...] Quite frankly I'd like someone else to take a look at this bug, if for no other reason than I'm not sure if we're communicating clearly with each other here. Most of what you are saying makes no sense whatsoever and seems to miss the point I have attempted to make. While I will thank you for actually writing a patch, saying "the caching proxy is working properly! and there's nothing you can do about it." seems rather cynical and clearly misses the point of what I'm requesting here. From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 11:35:47 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 15:35:47 +0000 Received: from localhost ([127.0.0.1]:38127 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqLp5-0007AG-IY for submit@debbugs.gnu.org; Tue, 21 Mar 2017 11:35:47 -0400 Received: from sender-pp-092.zoho.com ([135.84.80.237]:25499) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqLp3-0007A8-Kq for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 11:35:46 -0400 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:to:cc:subject:message-id:in-reply-to:references:mime-version:content-type; b=VXt2LV3f4PC8ab1UFAMNZLKdFTkI6bf+fDZ9JXYEH+w1OLuor8m4l/lwrGavk/3Ry1PXQ9PGo91s 23YDz5J278gJIq/yrw2hSbW4pbwLzBGRO06fGITavUOdkvcvZ47i Received: from khaalida (ip68-96-178-131.lv.lv.cox.net [68.96.178.131]) by mx.zohomail.com with SMTPS id 1490110540949407.90263872806156; Tue, 21 Mar 2017 08:35:40 -0700 (PDT) Date: Tue, 21 Mar 2017 08:35:36 -0700 From: To: Florian Pelz Subject: Re: bug#26201: No notification of cache misses when downloading substitutes Message-ID: <20170321083536.639716a9@khaalida> In-Reply-To: References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.8 (--) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.8 (--) On Tue, 21 Mar 2017 13:59:27 +0100 Florian Pelz wrote: > On Mon, 2017-03-20 at 21:48 -0700, dian_cecht@zoho.com wrote: > > Another option would be to have the mirrors automatically cache the > > files as soon as they are available to try. I'd hope this would be > > how things are handled already, but one never knows. > > =20 >=20 > If it cached everything, it wouldn=E2=80=99t be a cache? If the point is to reduce the load on hydra, then at some point it could have everything. If it doesn't, then why have a mirror when it's just pulling right the source all the time anyways? From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 12:08:15 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 16:08:15 +0000 Received: from localhost ([127.0.0.1]:38175 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqMKV-0007xt-0H for submit@debbugs.gnu.org; Tue, 21 Mar 2017 12:08:15 -0400 Received: from relay7-d.mail.gandi.net ([217.70.183.200]:47270) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqMKT-0007xk-60 for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 12:08:14 -0400 Received: from mfilter29-d.gandi.net (mfilter29-d.gandi.net [217.70.178.160]) by relay7-d.mail.gandi.net (Postfix) with ESMTP id 383C9D0D; Tue, 21 Mar 2017 17:08:12 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at mfilter29-d.gandi.net Received: from relay7-d.mail.gandi.net ([IPv6:::ffff:217.70.183.200]) by mfilter29-d.gandi.net (mfilter29-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id aHXxdkoRxGgU; Tue, 21 Mar 2017 17:07:40 +0100 (CET) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id EAB9C298; Tue, 21 Mar 2017 17:07:28 +0100 (CET) Subject: Re: bug#26201: No notification of cache misses when downloading substitutes To: dian_cecht@zoho.com References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> <20170320234912.46680062@khaalida> <1bbd8ee3-1745-3642-27ed-f095c732dc11@tobias.gr> <20170321083239.3cbf1e8d@khaalida> From: Tobias Geerinckx-Rice Message-ID: <553699c2-fb50-5cf4-a80d-8ee0a70c039d@tobias.gr> Date: Tue, 21 Mar 2017 17:07:56 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170321083239.3cbf1e8d@khaalida> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="gmBFOcBNsOrXpOClpiptaJ39BhhdulCJq" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --gmBFOcBNsOrXpOClpiptaJ39BhhdulCJq Content-Type: multipart/mixed; boundary="ieaFvuLdQqdrqTmbiknVWj4ri3V5uKfBj"; protected-headers="v1" From: Tobias Geerinckx-Rice To: dian_cecht@zoho.com Cc: 26201@debbugs.gnu.org Message-ID: <553699c2-fb50-5cf4-a80d-8ee0a70c039d@tobias.gr> Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> <20170320234912.46680062@khaalida> <1bbd8ee3-1745-3642-27ed-f095c732dc11@tobias.gr> <20170321083239.3cbf1e8d@khaalida> In-Reply-To: <20170321083239.3cbf1e8d@khaalida> --ieaFvuLdQqdrqTmbiknVWj4ri3V5uKfBj Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 21/03/17 16:32, dian_cecht@zoho.com wrote: > Unless mirror.hydra randomly loses data in it's cache from hydra, it > won't be random in the least. It will. Whether one is first to download from the cache after the substitute is built is essentially random. > Quite frankly I'd like someone else to take a look at this bug, Glad you agree. > if for no other reason than I'm not sure if we're communicating clearly= > with each other here. Most of what you are saying makes no sense > whatsoever and seems to miss the point I have attempted to make. I assure you it does not. Kind regards, T G-R --ieaFvuLdQqdrqTmbiknVWj4ri3V5uKfBj-- --gmBFOcBNsOrXpOClpiptaJ39BhhdulCJq Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY0U/hDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5lTCwf9EjwC ZeirzfClf6zpvoZhcUMFYhVzeu+jikCR4khmYX2R669ncg0YzcexrH/H/+3Ak5bk oo2H+lJO22R/YNlMbaShQs05MGLYS7A8VHhbOwpiE1Dpdp/Ip1E9Zt2pGBTRfSl3 hXU6R9Lkj6v4MgTOpgf1jlrC6+dFvx5DWe4lrTT+AjQxTs0UmFVUVpcl8tZEks7P mYcbU1KTvEr7exfEq4gzPFnh3LJV8j5YsmV804rIkjwqoDKd1KaNZ249pWLTNB8K ynuMMLNwBVYeT3a25aUh+3U23X1X+AupEt63Q+6oGR95bLARx8VRyj41YBR9eG4R ZQx2uq1xlUhHsIZm3w== =Aw3Z -----END PGP SIGNATURE----- --gmBFOcBNsOrXpOClpiptaJ39BhhdulCJq-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 12:43:44 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 16:43:44 +0000 Received: from localhost ([127.0.0.1]:38190 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqMsq-0000Ll-4y for submit@debbugs.gnu.org; Tue, 21 Mar 2017 12:43:44 -0400 Received: from eggs.gnu.org ([208.118.235.92]:38470) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqMsn-0000LW-Qq for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 12:43:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cqMse-0008UI-Lw for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 12:43:36 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:57431) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cqMse-0008UE-Ic; Tue, 21 Mar 2017 12:43:32 -0400 Received: from [193.50.110.74] (port=56588 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cqMsd-0005pp-Rx; Tue, 21 Mar 2017 12:43:32 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Tobias Geerinckx-Rice Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 1 Germinal an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Tue, 21 Mar 2017 17:43:29 +0100 In-Reply-To: <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> (Tobias Geerinckx-Rice's message of "Tue, 21 Mar 2017 04:57:09 +0100") Message-ID: <8760j2wpfy.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org, dian_cecht@zoho.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hello! Tobias Geerinckx-Rice skribis: > Oh, OK. I'm not an expert on how Hydra's set up these days, but will > assume it's not too different from my own (a fast nginx proxy_cache, > mirror.hydra.gnu.org, in front of a slower build farm, hydra.gnu.org). I think there=E2=80=99s room for improvement in our nginx config at . For instance, I just discovered =E2=80=98proxy_cache_lock=E2=80=99 while lo= oking at ; looks useful in reducing load on hydra.gnu.org. Surely there are other ways to tweak caching. Besides, I=E2=80=99d like to use =E2=80=98guix publish=E2=80=99 on hydra.gn= u.org. I suspect it=E2=80=99s going to be faster than Starman (the HTTP server behind Hydra)= , and also it uses an in-process gzip by default, as opposed to bzip2 which is what Hydra uses (better compression ratio, but super CPU-intensive). At any rate, clients should not paper over server-side performance issues IMO. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 21 13:07:45 2017 Received: (at 26201) by debbugs.gnu.org; 21 Mar 2017 17:07:45 +0000 Received: from localhost ([127.0.0.1]:38208 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqNG4-0000vZ-Uk for submit@debbugs.gnu.org; Tue, 21 Mar 2017 13:07:45 -0400 Received: from relay8-d.mail.gandi.net ([217.70.183.201]:50073) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqNG2-0000vP-As for 26201@debbugs.gnu.org; Tue, 21 Mar 2017 13:07:43 -0400 Received: from mfilter17-d.gandi.net (mfilter17-d.gandi.net [217.70.178.145]) by relay8-d.mail.gandi.net (Postfix) with ESMTP id 877D740409; Tue, 21 Mar 2017 18:07:40 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at mfilter17-d.gandi.net Received: from relay8-d.mail.gandi.net ([IPv6:::ffff:217.70.183.201]) by mfilter17-d.gandi.net (mfilter17-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id fbnkenxYrAHj; Tue, 21 Mar 2017 18:07:38 +0100 (CET) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 6C0A540408; Tue, 21 Mar 2017 18:07:36 +0100 (CET) Subject: Re: bug#26201: No notification of cache misses when downloading substitutes To: ludo@gnu.org References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> From: Tobias Geerinckx-Rice Message-ID: <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> Date: Tue, 21 Mar 2017 18:08:02 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <8760j2wpfy.fsf@gnu.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="JV9jdaI7mxWVbH5BGog5IQkb2TJ9IWvti" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --JV9jdaI7mxWVbH5BGog5IQkb2TJ9IWvti Content-Type: multipart/mixed; boundary="WV8LXBLt9afGN3kTL4dr87L3aHotFsu4a"; protected-headers="v1" From: Tobias Geerinckx-Rice To: ludo@gnu.org Cc: 26201@debbugs.gnu.org Message-ID: <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> In-Reply-To: <8760j2wpfy.fsf@gnu.org> --WV8LXBLt9afGN3kTL4dr87L3aHotFsu4a Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludo', On 21/03/17 17:43, Ludovic Court=C3=A8s wrote: > I think there=E2=80=99s room for improvement in our nginx config at > . >=20 > For instance, I just discovered =E2=80=98proxy_cache_lock=E2=80=99 whil= e looking at > ; looks usefu= l > in reducing load on hydra.gnu.org. Surely there are other ways to twea= k > caching. Indeed! For reference, here's my cache configuration. That's right. Now you can all=C2=B9 steal some criminally overpriced Belg= ian bandwidth! server { server_name substitutes.tobias.gr; listen [::]:443 ssl http2; listen 443 ssl http2; # FIXME move to main LE cert ssl_certificate substitutes.pem; ssl_certificate_key substitutes.key; # "" means =E2=80=98inherit from upstream=E2=80=99 here. add_header Cache-Control ""; # So does =E2=80=98off=E2=80=99. This is all a bit hacky. expires off; proxy_hide_header Set-Cookie; proxy_ignore_headers Set-Cookie; # Almost all traffic is already compressed. gzip off; ... location / { limit_except GET { deny all; } proxy_pass SUPER_SEKRIT_BACKEND; # https://www.nginx.com/blog/nginx-caching-guide add_header X-Cache-Status $upstream_cache_status; proxy_cache default; # We allow only GET requests, so don't waste key space: proxy_cache_key "$request_uri"; proxy_cache_lock on; proxy_cache_lock_timeout 3h; #yolo proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504; } ... } I'm sure it's hardly optimal (or, erm, =E2=80=98good=E2=80=99) either but= it works. > Besides, I=E2=80=99d like to use =E2=80=98guix publish=E2=80=99 on hydr= a.gnu.org. I suspect > it=E2=80=99s going to be faster than Starman (the HTTP server behind Hy= dra), and > also it uses an in-process gzip by default, as opposed to bzip2 which i= s > what Hydra uses (better compression ratio, but super CPU-intensive). Back when I used Hydra-the-software I do so briefly and I think it worked. But no hard tests. > At any rate, clients should not paper over server-side performance > issues IMO. Entirely off-topic, but this 'tude is a part of what drew me to Guix in the first place. So, like, thanks, in general :-) Kind regards, T G-R =C2=B9: Just put it *after* mirror.hydra.gnu.org, OK? --WV8LXBLt9afGN3kTL4dr87L3aHotFsu4a-- --JV9jdaI7mxWVbH5BGog5IQkb2TJ9IWvti Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY0V3zDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5m5ngf9EZsz U9cze8vfRYyWCK5g0b4Cyz4FYdx1HerKmgnRugAruTMrPY+mRG1MbQGBoe7uQshS fwOkM3IAAVFAcX3pHf1kYXwltCYf/JH5Q6MJYNHlUnUTb6y5MryP0BVamdLiRklc x1wHb2bIJqmdIp50XR4LRzMu4ZihPQa6JPMKHCcEc7CkolzCEJJ5Y/SaROyWv096 4LEDq4twGdVboFVF6akzSCnsEWhVxvNbDitjPworw6BeH+RaM8Awn7Pw9kjxac2X FgGVZPHQIwaeeE6v+Z9H4GuY3SSG2sdVJK6CeRKDTEjB6veFD7yrka8CYuu9ZW5T KhRlmbPb/JHYQL3N3Q== =gOrO -----END PGP SIGNATURE----- --JV9jdaI7mxWVbH5BGog5IQkb2TJ9IWvti-- From debbugs-submit-bounces@debbugs.gnu.org Wed Mar 22 18:06:26 2017 Received: (at 26201) by debbugs.gnu.org; 22 Mar 2017 22:06:26 +0000 Received: from localhost ([127.0.0.1]:39866 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqoOe-0004ub-R1 for submit@debbugs.gnu.org; Wed, 22 Mar 2017 18:06:26 -0400 Received: from eggs.gnu.org ([208.118.235.92]:48332) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqoOd-0004uP-Ep for 26201@debbugs.gnu.org; Wed, 22 Mar 2017 18:06:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cqoOU-00063X-4E for 26201@debbugs.gnu.org; Wed, 22 Mar 2017 18:06:18 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:52225) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cqoOU-00063T-0l; Wed, 22 Mar 2017 18:06:14 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:35316 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cqoOT-0007or-Ez; Wed, 22 Mar 2017 18:06:13 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Tobias Geerinckx-Rice Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 2 Germinal an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Wed, 22 Mar 2017 23:06:11 +0100 In-Reply-To: <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> (Tobias Geerinckx-Rice's message of "Tue, 21 Mar 2017 18:08:02 +0100") Message-ID: <87fui50xws.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hey Tobias, Tobias Geerinckx-Rice skribis: > On 21/03/17 17:43, Ludovic Court=C3=A8s wrote: >> I think there=E2=80=99s room for improvement in our nginx config at >> . >>=20 >> For instance, I just discovered =E2=80=98proxy_cache_lock=E2=80=99 while= looking at >> ; looks useful >> in reducing load on hydra.gnu.org. Surely there are other ways to tweak >> caching. > > Indeed! For reference, here's my cache configuration. > > That's right. Now you can all=C2=B9 steal some criminally overpriced Belg= ian > bandwidth! Heheh. :-) > limit_except GET { deny all; } > proxy_pass SUPER_SEKRIT_BACKEND; > > # https://www.nginx.com/blog/nginx-caching-guide > add_header X-Cache-Status $upstream_cache_status; > > proxy_cache default; > # We allow only GET requests, so don't waste key space: > proxy_cache_key "$request_uri"; > proxy_cache_lock on; > proxy_cache_lock_timeout 3h; #yolo > proxy_cache_use_stale error timeout > http_500 http_502 http_503 http_504; I didn=E2=80=99t fully understand the docs for the last 3 directives here. = For instance, what happens when 10 clients do GET /nar/xyz-texlive? Do the 9 unlucky clients wait for 3 hours and then get 404? Anyway, thanks for sharing your tips. :-) > Entirely off-topic, but this 'tude is a part of what drew me to Guix in > the first place. So, like, thanks, in general :-) :-) Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Mar 22 18:22:54 2017 Received: (at 26201) by debbugs.gnu.org; 22 Mar 2017 22:22:54 +0000 Received: from localhost ([127.0.0.1]:39881 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqoec-0005KE-FP for submit@debbugs.gnu.org; Wed, 22 Mar 2017 18:22:54 -0400 Received: from eggs.gnu.org ([208.118.235.92]:52692) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqoeZ-0005Jy-GY for 26201@debbugs.gnu.org; Wed, 22 Mar 2017 18:22:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cqoeT-0002pS-FK for 26201@debbugs.gnu.org; Wed, 22 Mar 2017 18:22:46 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:52450) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cqoeO-0002op-MX; Wed, 22 Mar 2017 18:22:40 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:35360 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cqoeN-0001GA-UR; Wed, 22 Mar 2017 18:22:40 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Tobias Geerinckx-Rice Subject: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 2 Germinal an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Wed, 22 Mar 2017 23:22:37 +0100 In-Reply-To: <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> (Tobias Geerinckx-Rice's message of "Tue, 21 Mar 2017 18:08:02 +0100") Message-ID: <87r31pyms2.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hi again! Until now hydra.gnu.org was using Hydra (the software) to serve not only the Web interface but also all the .narinfo and /nar URLs (substitute meta-data and substitutes). Starting from now, hydra.gnu.org directs all .narinfo and corresponding nar requests to =E2=80=98guix publish=E2=80=99 instead of Hydra. =E2=80=98guix publish=E2=80=99 should be faster and less resource-hungry th= an Hydra. It uses in-process gzip for nar compression instead of bzip2 (I chose level 7, which seems to provide compression ratios close to what bzip2 provides with its default compression level, while being 3 times faster). Unlike Hydra it never forks so for instance, 404 responses for .narinfo URLs should be quicker. Hopefully, that will improve the worst-case (cache miss) throughput. I configured nginx in such a way that the former Hydra-provided /nar URLs (which are cached in nginx instances, in our /var/guix/substitute/cache directories, etc.) are still available. =E2=80=98guix publish=E2=80=99 uses the /guix/nar URLs while Hydra uses /na= r, so the nginx config redirects to either Hydra or =E2=80=98guix publish=E2=80=99 de= pending on the URL: https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/h= ydra.gnu.org-locations.conf#n29 Hydra-provided .narinfos are still cached here and there; they=E2=80=99ll be progressively expire and be replaced by =E2=80=98guix publish=E2=80=99-prov= ided .narinfos. Let me know if you notice anything fishy! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 23 06:29:39 2017 Received: (at 26201) by debbugs.gnu.org; 23 Mar 2017 10:29:39 +0000 Received: from localhost ([127.0.0.1]:40175 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqzzu-0006XC-Va for submit@debbugs.gnu.org; Thu, 23 Mar 2017 06:29:39 -0400 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21032) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cqzzs-0006X2-Bd for 26201@debbugs.gnu.org; Thu, 23 Mar 2017 06:29:36 -0400 Received: from localhost (141.80.148.97 [141.80.148.97]) by mx.zohomail.com with SMTPS id 1490264970575237.66711416118244; Thu, 23 Mar 2017 03:29:30 -0700 (PDT) References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> User-agent: mu4e 0.9.18; emacs 25.1.1 From: Ricardo Wurmus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish=E2=80=99?= for nars and narinfos In-reply-to: <87r31pyms2.fsf_-_@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Thu, 23 Mar 2017 11:29:29 +0100 Message-ID: <87mvccs2uu.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.8 (-) X-Debbugs-Envelope-To: 26201 Cc: Tobias Geerinckx-Rice , 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.8 (-) Ludovic Courtès writes: > Until now hydra.gnu.org was using Hydra (the software) to serve not only > the Web interface but also all the .narinfo and /nar URLs (substitute > meta-data and substitutes). > > Starting from now, hydra.gnu.org directs all .narinfo and corresponding > nar requests to ‘guix publish’ instead of Hydra. That’s very cool! I’m happy to see more of Hydra replaced. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 23 14:36:41 2017 Received: (at 26201) by debbugs.gnu.org; 23 Mar 2017 18:36:41 +0000 Received: from localhost ([127.0.0.1]:41287 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cr7bE-0006WN-S8 for submit@debbugs.gnu.org; Thu, 23 Mar 2017 14:36:41 -0400 Received: from world.peace.net ([50.252.239.5]:42089) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cr7bC-0006WA-Gx for 26201@debbugs.gnu.org; Thu, 23 Mar 2017 14:36:38 -0400 Received: from pool-72-93-33-151.bstnma.east.verizon.net ([72.93.33.151] helo=jojen) by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1cr7b6-0004LM-F7; Thu, 23 Mar 2017 14:36:32 -0400 From: Mark H Weaver To: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: bug#26201: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish?= =?utf-8?Q?=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> Date: Thu, 23 Mar 2017 14:36:20 -0400 In-Reply-To: <87r31pyms2.fsf_-_@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\?\= \=\?utf-8\?Q\?\=22's\?\= message of "Wed, 22 Mar 2017 23:22:37 +0100") Message-ID: <87inmzrgbf.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 26201 Cc: Tobias Geerinckx-Rice , 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Hi again! > > Until now hydra.gnu.org was using Hydra (the software) to serve not only > the Web interface but also all the .narinfo and /nar URLs (substitute > meta-data and substitutes). > > Starting from now, hydra.gnu.org directs all .narinfo and corresponding > nar requests to =E2=80=98guix publish=E2=80=99 instead of Hydra. > > =E2=80=98guix publish=E2=80=99 should be faster and less resource-hungry = than Hydra. It > uses in-process gzip for nar compression instead of bzip2 (I chose level > 7, which seems to provide compression ratios close to what bzip2 > provides with its default compression level, while being 3 times > faster). Unlike Hydra it never forks so for instance, 404 responses for > .narinfo URLs should be quicker. Hopefully, that will improve the > worst-case (cache miss) throughput. Excellent! Any improvement in 404 response time will be very helpful. I've noticed that spikes of narinfo requests resulting in 404 has been a major source of overloading on Hydra, because these requests cannot be cached for very long. The reason: if we cache those failures for N minutes, this effectively delays the appearance of new nars by N minutes (if it was requested before that). This forces us to choose a small N for negative cache entries, which means the cache is not much help here. One question: what will happen in the case of multiple concurrent requests for the same nar? Will multiple nar-pack-and-bzip2 processes be run on-demand? Recall that the nginx proxy will pass all of those requests through, and not create the cache entry until it has received a complete response. This has caused us severe problems with huge nars such as texinfo-texmf, to the point that we had to crudely block those nar requests. Unfortunately, it is not obvious how to block the associated narinfo requests due to the lack of job name in the URL, so this results in failures on the client side that must be manually worked around. Thanks, Mark From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 23 14:51:40 2017 Received: (at 26201) by debbugs.gnu.org; 23 Mar 2017 18:51:40 +0000 Received: from localhost ([127.0.0.1]:41299 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cr7pk-0006su-I2 for submit@debbugs.gnu.org; Thu, 23 Mar 2017 14:51:40 -0400 Received: from relay6-d.mail.gandi.net ([217.70.183.198]:43418) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cr7pi-0006sm-Fx for 26201@debbugs.gnu.org; Thu, 23 Mar 2017 14:51:39 -0400 Received: from mfilter17-d.gandi.net (mfilter17-d.gandi.net [217.70.178.145]) by relay6-d.mail.gandi.net (Postfix) with ESMTP id 973DAFB8C2; Thu, 23 Mar 2017 19:51:36 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at mfilter17-d.gandi.net Received: from relay6-d.mail.gandi.net ([IPv6:::ffff:217.70.183.198]) by mfilter17-d.gandi.net (mfilter17-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id SkEv3GECalzW; Thu, 23 Mar 2017 19:51:35 +0100 (CET) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 35D2AFB8B6; Thu, 23 Mar 2017 19:51:33 +0100 (CET) Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= To: mhw@netris.org References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> From: Tobias Geerinckx-Rice Message-ID: <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> Date: Thu, 23 Mar 2017 19:52:30 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <87inmzrgbf.fsf@netris.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="dVXht4gfnfdO2ivijg2i1pNaesWUQ1Vmo" X-Spam-Score: -3.5 (---) X-Debbugs-Envelope-To: 26201 Cc: ludo@gnu.org, 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.5 (---) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --dVXht4gfnfdO2ivijg2i1pNaesWUQ1Vmo Content-Type: multipart/mixed; boundary="2UD43kRVETcEQao7brdvNHJdub701gWxn"; protected-headers="v1" From: Tobias Geerinckx-Rice To: mhw@netris.org Cc: ludo@gnu.org, 26201@debbugs.gnu.org, guix-sysadmin@gnu.org Message-ID: <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> In-Reply-To: <87inmzrgbf.fsf@netris.org> --2UD43kRVETcEQao7brdvNHJdub701gWxn Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mark, On 23/03/17 19:36, Mark H Weaver wrote: > One question: what will happen in the case of multiple concurrent > requests for the same nar? Will multiple nar-pack-and-bzip2 processes > be run on-demand? I think this used to be the case with the previous nginx configuration, but the recent changes pushed by Ludo' were aimed in part at preventing that. > Recall that the nginx proxy will pass all of those requests through, Are you sure? I was under the impression=C2=B9 that this is exactly what =E2=80=98proxy_cache_lock on;=E2=80=99 prevents. I'm no nginx guru, obvio= usly, so please =E2=80=94 anyone! =E2=80=94 correct me if I'm misguided. Kind regards, T G-R =C2=B9: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_loc= k --2UD43kRVETcEQao7brdvNHJdub701gWxn-- --dVXht4gfnfdO2ivijg2i1pNaesWUQ1Vmo Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY1BlvDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5mXvggAngOM rGWofD7nBqWPIULF3CPlJmsRUvzstCSYkxvosaBCe2DuYcGZtZ9+qymmOP1XymRC vEx9wxTRHldmocMypKPQrn5ZEj0R9paMpDBljxQ9oURZKGLdlVcihYuF1/7xSj5T CCM4CnWBozoRPcQ1yyCFeP8T32x/mW6/XHAD7HmO4sxihNw45xazBQAo/JZmmYnC 66KOblM+uSD2DVTe5m8WOruXpQslwZXTE1UCNqCJSQYqM20bu2wi15a2iPQv1YAF uvPGxMfj+W8rI8D/aEL+vkQu9P+lRrqxtTinFFiaEJQetwQjqFeWlTPl0AQP3tGP TngOaMOd39K0Ib6oDg== =5ERe -----END PGP SIGNATURE----- --dVXht4gfnfdO2ivijg2i1pNaesWUQ1Vmo-- From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 23 15:24:55 2017 Received: (at 26201) by debbugs.gnu.org; 23 Mar 2017 19:24:55 +0000 Received: from localhost ([127.0.0.1]:41344 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cr8Lv-0007g1-FH for submit@debbugs.gnu.org; Thu, 23 Mar 2017 15:24:55 -0400 Received: from relay6-d.mail.gandi.net ([217.70.183.198]:40748) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cr8Lt-0007fr-QJ for 26201@debbugs.gnu.org; Thu, 23 Mar 2017 15:24:54 -0400 Received: from mfilter10-d.gandi.net (mfilter10-d.gandi.net [217.70.178.139]) by relay6-d.mail.gandi.net (Postfix) with ESMTP id CD012FB88B; Thu, 23 Mar 2017 20:24:51 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at mfilter10-d.gandi.net Received: from relay6-d.mail.gandi.net ([IPv6:::ffff:217.70.183.198]) by mfilter10-d.gandi.net (mfilter10-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id FyG4qKntx7Ja; Thu, 23 Mar 2017 20:24:50 +0100 (CET) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id E5094FB886; Thu, 23 Mar 2017 20:24:47 +0100 (CET) Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= To: ludo@gnu.org References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87fui50xws.fsf@gnu.org> From: Tobias Geerinckx-Rice Message-ID: Date: Thu, 23 Mar 2017 20:25:44 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <87fui50xws.fsf@gnu.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="wQgDkXpQf8pEMS9qaTEMkQRQl3iR3UKfM" X-Spam-Score: -3.5 (---) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.5 (---) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --wQgDkXpQf8pEMS9qaTEMkQRQl3iR3UKfM Content-Type: multipart/mixed; boundary="m29Ouruoh7dDAJbA88siIvb3GW0xLoL85"; protected-headers="v1" From: Tobias Geerinckx-Rice To: ludo@gnu.org Cc: 26201@debbugs.gnu.org Message-ID: Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87fui50xws.fsf@gnu.org> In-Reply-To: <87fui50xws.fsf@gnu.org> --m29Ouruoh7dDAJbA88siIvb3GW0xLoL85 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludo', On 22/03/17 23:06, Ludovic Court=C3=A8s wrote: > Tobias Geerinckx-Rice skribis: >> proxy_cache_lock on; >> proxy_cache_lock_timeout 3h; #yolo >> proxy_cache_use_stale error timeout >> http_500 http_502 http_503 http_504; > I didn=E2=80=99t fully understand the docs for the last 3 directives he= re. For > instance, what happens when 10 clients do GET /nar/xyz-texlive? Do the= > 9 unlucky clients wait for 3 hours and then get 404? =46rom =E2=80=98proxy_cache_lock=E2=80=99 [1]: =E2=80=9CWhen enabled, only one request at a time will be allowed to po= pulate a new cache element identified according to the proxy_cache_key directive by passing a request to a proxied server. Other requests of the same cache element will either wait for a response to appear in the cache or the cache lock for this element to be released, up to the time set by the proxy_cache_lock_timeout directive.=E2=80=9D Hmm. Good point: =E2=80=98to appear in the cache=E2=80=99, when we don't = cache 404s or even 410s. I don't actually know. Kind regards, T G-R [1]: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_loc= k --m29Ouruoh7dDAJbA88siIvb3GW0xLoL85-- --wQgDkXpQf8pEMS9qaTEMkQRQl3iR3UKfM Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY1CE5DRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5kgMAf/ae4U S4Xqn6T5S+HKFm1fEORVEiDlCmNqrJcL68w1OyRJrwIBgv4DWwQZCY+v7hmKkYN4 LC1H9PPqS9B6L/oI26lkpbc9hX64J+J4wv0a7HdayJG6O7s8DpDqW9u1Hl5x97cu FnmG3x2FmuJm3OupEOkxlfT10DK9ESZkdbxAefY/EeZNK27ieuDf1hCZVIW8A2IE fHlsqQ/VvlSFlpXvTpUFxCYPyYfp5t2K2d1JwNWHuw5MXRFDbvkTQrhW8xZoeloS t8OGT8A4utokP9dpHK5IgNY+z2KbFBTPLt4/u53RQooZf5Va1w5u5H4KJ2MPdYjB MfCYvhjt1b4Wakyj/w== =irlP -----END PGP SIGNATURE----- --wQgDkXpQf8pEMS9qaTEMkQRQl3iR3UKfM-- From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 23 22:16:04 2017 Received: (at 26201) by debbugs.gnu.org; 24 Mar 2017 02:16:05 +0000 Received: from localhost ([127.0.0.1]:41584 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1crElo-0000bJ-Il for submit@debbugs.gnu.org; Thu, 23 Mar 2017 22:16:04 -0400 Received: from mail-pf0-f193.google.com ([209.85.192.193]:35735) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1crElm-0000ab-Ga for 26201@debbugs.gnu.org; Thu, 23 Mar 2017 22:16:03 -0400 Received: by mail-pf0-f193.google.com with SMTP id n11so259828pfg.2 for <26201@debbugs.gnu.org>; Thu, 23 Mar 2017 19:16:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=aN+Hx9FrAyh0Qsr6Kk3YkIqAJcGxq9TmIKejgdDWnXo=; b=TFyWmYzFuG69JrD1GgEAmO/AS2W0uj3SM6F/mbukOWN1kyefqbX2iVY8DsA9NfApG0 MsktO69d0OjQepkO8jzbtdwjcVv3H/05HcOy5hZx5/xoguIy0TW7bdHRgRu5KiiNoEgD hByh4H18Gh5lquKdiONYGNdCN3Jd2S0ITeBoASo/fQBriyMwXTOX/UIFJcQk39NBKAkv g3HF2IC4skp0YdKu5RwEwad7X4axR4bzqaEa9UeapFFDpuyqiNIz5y2Z3RWgFsf14Fb+ l807y8AHmvAjfdEz5syBQwYCeYrG5UAQB7xIZfwBZj8cKtsXQy62ynLX4rRHfkxjIlxJ K7jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=aN+Hx9FrAyh0Qsr6Kk3YkIqAJcGxq9TmIKejgdDWnXo=; b=f6iPmA+Slfxe3pFuSWvhQ0UJn+1jfW9w7pY7w4fXt5nHX7P8e5ErIaNCOPHo5dLGAS 5Qanjy+yKQr2DeY4cuzvhVYH8l3ZJfiZgjugzGNaTZTfQIihaQsl5dNdHjnbiz/WJq3w jZrqJKjIYIW0LrWeXegS+iG3QSdNbEua4lYyUZ1H8QL9LnfdIzPLmDFhH2Ut4n4NZMSD lWHp/i9v7mLNAqrJjTWjJC48lvWWmbkFw/BuAduKHo+nZF4GvBMy+/k8pX6lQ/y6HMd1 odXUCNZwMlyoewe07u9o9T5fXDDB96Z+WgDPJh1BPIKZYMhbRyUlp8kEylpXSYIJjuxI HKmQ== X-Gm-Message-State: AFeK/H3cMSshgzo2xwPlZlXup7OWIMsZ3jswn64H5CVQMmRPEAEp7Y+411RojOzgQPttSw== X-Received: by 10.98.166.132 with SMTP id r4mr6243936pfl.191.1490321756731; Thu, 23 Mar 2017 19:15:56 -0700 (PDT) Received: from apteryx ([2400:2652:be0:5100:c2f8:daff:fe5d:2f2f]) by smtp.gmail.com with ESMTPSA id x15sm740770pfk.68.2017.03.23.19.15.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 23 Mar 2017 19:15:55 -0700 (PDT) From: Maxim Cournoyer To: Tobias Geerinckx-Rice Subject: Re: bug#26201: No notification of cache misses when downloading substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <20170320214809.466dc5fe@khaalida> <20170320234912.46680062@khaalida> <1bbd8ee3-1745-3642-27ed-f095c732dc11@tobias.gr> <20170321083239.3cbf1e8d@khaalida> <553699c2-fb50-5cf4-a80d-8ee0a70c039d@tobias.gr> Date: Thu, 23 Mar 2017 19:15:48 -0700 In-Reply-To: <553699c2-fb50-5cf4-a80d-8ee0a70c039d@tobias.gr> (Tobias Geerinckx-Rice's message of "Tue, 21 Mar 2017 17:07:56 +0100") Message-ID: <87efxnzagb.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org, dian_cecht@zoho.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-=-= Content-Type: text/plain Hi! Tobias Geerinckx-Rice writes: > On 21/03/17 16:32, dian_cecht@zoho.com wrote: >> Unless mirror.hydra randomly loses data in it's cache from hydra, it >> won't be random in the least. > > It will. Whether one is first to download from the cache after the > substitute is built is essentially random. > >> Quite frankly I'd like someone else to take a look at this bug, > > Glad you agree. > >> if for no other reason than I'm not sure if we're communicating clearly >> with each other here. Most of what you are saying makes no sense >> whatsoever and seems to miss the point I have attempted to make. > > I assure you it does not. > > Kind regards, > > T G-R Please allow me to jump in and voice my opinion here. To me it doesn't make sense to concern the Guix client with implementation details of how the caching of substitutes happen and its impacts. This situation is bound to change in the future or become irrelevant (say, if a new build farm would be able to sustain higher transfer speeds to the cache mirror), or if the caching implementation changes. If the current cache building implementation is slow to the point of being a problem it should be fixed (or documented). Cheers, Maxim --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAljUgVUACgkQEmDkZILm NWLXkA//fiY5xgNAAbJ+QANhXWNcYsCHfTVm9Zhl/dqq2rnKgUcDs7/vd7AKfQJT wQmoWJf2Uz+lnGJep5plLxCy1Q0DhmnnfVtjrtcD2Z12IIkfCd0jo2DIFiuVH4LO PnyhEzQZnSlF/wYPxiyYRkagp5eNQNBeCA8Ym14VP15PXytb7GvrKldH0o3oBBm6 Eht4WjKQ9wWeu5vwcRyWAMxQyPbD1ITpfFRUru1mNgjCmeNRDH7g/q17lQlyXuNA /QVNoJsT2+FOSdjFhvTPGyWXWtVnWWHzU0XGw3iKYfvAHxxroNP12LzK8Mr/KuUw Oux6MIrpsdwCoMmtLZmqVkQEYFbXAPoqZftN1OXOqXdIXNmh9fE6ZAlLrqVPkTdn 19bdRONIxZGOS39lIB1SS0jJ4gIehjWU1ZiqgoKIZ/4jArjn+5cd81+yB5rsUDCF NgCILJRK6TXoaqHCjEj3N0ci3jxrpwtobsAERkiK80tOegPCTCNvIym94y0Zce0Q pJrSBNjPVq1DFXQ/biGlcDsoVq/eGGY9Ie6WfqGfgjpfmb/Espud/XQYQj7j9Mjm OTGcu8vd0Q3TING1RjW1FDlI2dfRyIxVda8Zosj1ckS72OIQ2HFWRqQmL/DD44NY W2qeBfQ3yYHmTalm7ir65Oj9J80AuBpb9KHsbPC5ZzBhuCiP9Io= =zBw4 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 24 04:13:11 2017 Received: (at 26201) by debbugs.gnu.org; 24 Mar 2017 08:13:11 +0000 Received: from localhost ([127.0.0.1]:41701 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1crKLP-0001Ph-F0 for submit@debbugs.gnu.org; Fri, 24 Mar 2017 04:13:11 -0400 Received: from world.peace.net ([50.252.239.5]:42754) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1crKLN-0001PR-Oa for 26201@debbugs.gnu.org; Fri, 24 Mar 2017 04:13:10 -0400 Received: from pool-72-93-31-97.bstnma.east.verizon.net ([72.93.31.97] helo=jojen) by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1crKLG-0007d6-U7; Fri, 24 Mar 2017 04:13:03 -0400 From: Mark H Weaver To: Tobias Geerinckx-Rice Subject: Re: bug#26201: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish?= =?utf-8?Q?=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> Date: Fri, 24 Mar 2017 04:12:50 -0400 In-Reply-To: <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> (Tobias Geerinckx-Rice's message of "Thu, 23 Mar 2017 19:52:30 +0100") Message-ID: <87y3vvozy5.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 26201 Cc: ludo@gnu.org, 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) Hi, Tobias Geerinckx-Rice writes: > On 23/03/17 19:36, Mark H Weaver wrote: >> One question: what will happen in the case of multiple concurrent >> requests for the same nar? Will multiple nar-pack-and-bzip2 processes >> be run on-demand? > > I think this used to be the case with the previous nginx configuration, > but the recent changes pushed by Ludo' were aimed in part at preventing > that. > >> Recall that the nginx proxy will pass all of those requests through, > > Are you sure? I was under the impression=C2=B9 that this is exactly what > =E2=80=98proxy_cache_lock on;=E2=80=99 prevents. I'm no nginx guru, obvio= usly, so please > =E2=80=94 anyone! =E2=80=94 correct me if I'm misguided. I agree that "proxy_cache_lock on" should prevent multiple concurrent requests for the same URL, but unfortunately its behavior is quite undesirable, and arguably worse than leaving it off in our case. See: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_lock Specifically: Other requests of the same cache element will either wait for a response to appear in the cache or the cache lock for this element to be released, up to the time set by the proxy_cache_lock_timeout directive. In our problem case, it takes more than an hour for Hydra to finish sending a response for the 'texlive-texmf' nar. During that time, the nar will be slowly sent to the first client while it's being packed and bzipped on-demand. IIUC, with "proxy_cache_lock on", we have two choices of how other client requests will be treated: (1) If we increase "proxy_cache_lock_timeout" to a huge value, then there will *no* data sent to the other clients until the first client has received the entire nar, which means they wait over an hour before receiving the first byte. I guess this will result in timeouts on the client side. (2) If "proxy_cache_lock_timeout" is *not* huge, then all other clients will get failure responses until the first client has received the entire nar. Either way, this would cause users to see the same download failures (requiring user work-arounds like --fallback) that this fix is intended to prevent for 'texlive-texmf', but instead of happening only for that one nar, it will now happen for *all* large nars. Or at least that's what I'd expect based on my reading of the nginx docs linked above. I haven't tried it. IMO, the best solution is to *never* generate nars on Hydra in response to client requests, but rather to have the build slaves pack and compress the nars, copy them to Hydra, and then serve them as static files using nginx. A far inferior solution, but possibly acceptable and closer to the current approach, would be to arrange for all concurrent responses for the same nar to be sent incrementally from a single nar-packing process. More concretely, while packing and sending a nar response to the first client, the data would also be written to a file. Subsequent requests for the same nar would be serviced using the equivalent of: tail --bytes=3D+0 --follow FILENAME This way, no one would have to wait an hour to receive the first byte. What do you think? Mark From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 24 05:25:52 2017 Received: (at 26201) by debbugs.gnu.org; 24 Mar 2017 09:25:52 +0000 Received: from localhost ([127.0.0.1]:41728 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1crLTk-0003C5-FR for submit@debbugs.gnu.org; Fri, 24 Mar 2017 05:25:52 -0400 Received: from eggs.gnu.org ([208.118.235.92]:51219) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1crLTi-0003Br-V9 for 26201@debbugs.gnu.org; Fri, 24 Mar 2017 05:25:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1crLTc-0004qx-Au for 26201@debbugs.gnu.org; Fri, 24 Mar 2017 05:25:45 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51858) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1crLTX-0004lp-IG; Fri, 24 Mar 2017 05:25:39 -0400 Received: from [193.50.110.125] (port=40078 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1crLTW-0005Tm-MG; Fri, 24 Mar 2017 05:25:39 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mark H Weaver Subject: Re: bug#26201: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish?= =?utf-8?Q?=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 4 Germinal an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Fri, 24 Mar 2017 10:25:35 +0100 In-Reply-To: <87y3vvozy5.fsf@netris.org> (Mark H. Weaver's message of "Fri, 24 Mar 2017 04:12:50 -0400") Message-ID: <87d1d710xc.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: Tobias Geerinckx-Rice , 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hi! Mark H Weaver skribis: > Tobias Geerinckx-Rice writes: [...] >> Are you sure? I was under the impression=C2=B9 that this is exactly what >> =E2=80=98proxy_cache_lock on;=E2=80=99 prevents. I'm no nginx guru, obvi= ously, so please >> =E2=80=94 anyone! =E2=80=94 correct me if I'm misguided. > > I agree that "proxy_cache_lock on" should prevent multiple concurrent > requests for the same URL, but unfortunately its behavior is quite > undesirable, and arguably worse than leaving it off in our case. See: > > https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_l= ock > > Specifically: > > Other requests of the same cache element will either wait for a > response to appear in the cache or the cache lock for this element to > be released, up to the time set by the proxy_cache_lock_timeout > directive. > > In our problem case, it takes more than an hour for Hydra to finish > sending a response for the 'texlive-texmf' nar. During that time, the > nar will be slowly sent to the first client while it's being packed and > bzipped on-demand. > > IIUC, with "proxy_cache_lock on", we have two choices of how other > client requests will be treated: > > (1) If we increase "proxy_cache_lock_timeout" to a huge value, then > there will *no* data sent to the other clients until the first > client has received the entire nar, which means they wait over an > hour before receiving the first byte. I guess this will result in > timeouts on the client side. > > (2) If "proxy_cache_lock_timeout" is *not* huge, then all other clients > will get failure responses until the first client has received the > entire nar. > > Either way, this would cause users to see the same download failures > (requiring user work-arounds like --fallback) that this fix is intended > to prevent for 'texlive-texmf', but instead of happening only for that > one nar, it will now happen for *all* large nars. My understanding is that proxy_cache_lock allows us to avoid spawning concurrent compression threads of the same item at the same time, while also avoiding starvation (proxy_cache_lock_timeout should ensure that nobody ends up waiting until the nar-compression process is done.) IOW, it should help reduce load in most cases, while introducing small delays in some cases (if you=E2=80=99re downloading a nar that=E2=80=99s al= ready being downloaded.) > IMO, the best solution is to *never* generate nars on Hydra in response > to client requests, but rather to have the build slaves pack and > compress the nars, copy them to Hydra, and then serve them as static > files using nginx. The problem is that we want nars to be signed by the master node. Or, if we don=E2=80=99t require that, we need a PKI that allows us to express t= he fact that hydra.gnu.org delegates to the build machines. > A far inferior solution, but possibly acceptable and closer to the > current approach, would be to arrange for all concurrent responses for > the same nar to be sent incrementally from a single nar-packing process. > More concretely, while packing and sending a nar response to the first > client, the data would also be written to a file. Subsequent requests > for the same nar would be serviced using the equivalent of: > > tail --bytes=3D+0 --follow FILENAME > > This way, no one would have to wait an hour to receive the first byte. Yes. I would think that NGINX does something like that for its caching, but I don=E2=80=99t know exactly when/how. Other solutions I=E2=80=99ve thought about: 1. Produce narinfos and nars periodically rather than on-demand and serve them as static files. pros: better HTTP latency and bandwidth pros: allows us to add a Content-Length for nars cons: doesn=E2=80=99t reduce load on hydra.gnu.org cons: introduces arbitrary delays in delivering nars cons: difficult/expensive to know what new store items are available 2. Produce a narinfo and corresponding nar the first time they are requested. So, the first time we receive =E2=80=9CGET foo.narinfo=E2= =80=9D, return 404 and spawn a thread to compute foo.narinfo and foo.nar. Return 200 only when both are ready. The precomputed nar{,info}s would be kept in a cache and we could make sure a narinfo and its nar have the same lifetime, which addresses one of the problems we have. pros: better HTTP latency and bandwidth pros: allows us to add a Content-Length for nars pros: helps keep narinfo/nar lifetime in sync cons: doesn=E2=80=99t reduce load on hydra.gnu.org cons: exposes inconsistency between the store contents and the HTTP response (you may get 404 even if the thing is actually in store), but maybe that=E2=80=99s not a problem Thoughts? Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 26 13:35:50 2017 Received: (at 26201) by debbugs.gnu.org; 26 Mar 2017 17:35:50 +0000 Received: from localhost ([127.0.0.1]:46121 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1csC50-0003Lc-Fr for submit@debbugs.gnu.org; Sun, 26 Mar 2017 13:35:50 -0400 Received: from relay3-d.mail.gandi.net ([217.70.183.195]:41607) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1csC4z-0003LU-17 for 26201@debbugs.gnu.org; Sun, 26 Mar 2017 13:35:50 -0400 Received: from mfilter18-d.gandi.net (mfilter18-d.gandi.net [217.70.178.146]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id 11CE9A80CB; Sun, 26 Mar 2017 19:35:47 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter18-d.gandi.net Received: from relay3-d.mail.gandi.net ([IPv6:::ffff:217.70.183.195]) by mfilter18-d.gandi.net (mfilter18-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id aRf6XSlhhYwQ; Sun, 26 Mar 2017 19:35:45 +0200 (CEST) X-Originating-IP: 109.131.205.51 Received: from [192.168.1.4] (unknown [109.131.205.51]) (Authenticated sender: me@tobias.gr) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 40152A80BE; Sun, 26 Mar 2017 19:35:44 +0200 (CEST) Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= To: mhw@netris.org References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> From: Tobias Geerinckx-Rice Message-ID: <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> Date: Sun, 26 Mar 2017 19:35:25 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <87y3vvozy5.fsf@netris.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Tao7UKjCLBAtpINvT4ketSuXk9Iww88Mg" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26201 Cc: ludo@gnu.org, 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Tao7UKjCLBAtpINvT4ketSuXk9Iww88Mg Content-Type: multipart/mixed; boundary="LWlJpkDdb2w3EIlUP7ojS6J4u0Lg6kf1c"; protected-headers="v1" From: Tobias Geerinckx-Rice To: mhw@netris.org Cc: ludo@gnu.org, 26201@debbugs.gnu.org, guix-sysadmin@gnu.org Message-ID: <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> In-Reply-To: <87y3vvozy5.fsf@netris.org> --LWlJpkDdb2w3EIlUP7ojS6J4u0Lg6kf1c Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mark, On 24/03/17 09:12, Mark H Weaver wrote: > IIUC, with "proxy_cache_lock on", we have two choices of how other > client requests will be treated: >=20 > [badly, ed.] Eh. You're probably (and disappointingly) right. When configuring my little cache, I had a clear idea of how such a cache should work (basically, your last scenario below), then looked at the nginx documentation to find what I had in mind. =E2=80=98proxy_cache_lock= =E2=80=99 matched. I should have been more pessimistic and done more testing. Shame on me, &c. Too much other things on my mind. :-/ > Or at least that's what I'd expect based on my reading of the nginx doc= s > linked above. I haven't tried it. I can try to do some simple tests tomorrow. > IMO, the best solution is to *never* generate nars on Hydra in response= > to client requests, but rather to have the build slaves pack and > compress the nars, copy them to Hydra, and then serve them as static > files using nginx. A true mirror at last! Do we have the disc space for that? And could Hydra actually handle compressing *everything*, without an infinitely growing back-log? I don't have access to any statistics, but I'm guessing that a fair number of package+versions are never actually requested, and hence never compressed. This would change that. > A far inferior solution, but possibly acceptable and closer to the > current approach, would be to arrange for all concurrent responses for > the same nar to be sent incrementally from a single nar-packing process= =2E > More concretely, while packing and sending a nar response to the first > client, the data would also be written to a file. Subsequent requests > for the same nar would be serviced using the equivalent of: >=20 > tail --bytes=3D+0 --follow FILENAME >=20 > This way, no one would have to wait an hour to receive the first byte. ^ This is so obviously the right solution, that it would be disappointing if nginx really couldn't be made to do it. It already buffers proxy responses to a temporary file anyway... Kind regards, T G-R --LWlJpkDdb2w3EIlUP7ojS6J4u0Lg6kf1c-- --Tao7UKjCLBAtpINvT4ketSuXk9Iww88Mg Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY1/veDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5ln2AgAmIsp fZMKZUsfwzzPNL+Cykyb0CoyfPODfBpUJiJNxiqnNrZDo6br7lLAO+oW5hHMDpwe B6+evORtaXpszT60NSQvRWLs5Re/tOkSU/cb+aoZ/7Fm7zgWezbbjWmzUobE4cbB gaPyMybvxuEXqbgMt/Cf7hjAjG3zY1RC3RteNbFym83st6+4NHw2QufjyFmXCzVY 93f+6j/cpOBk+na1LeVPJmVposs1qlhqxoWLfGVZrnDxFFXWb6W0VJhGDKk9i3So prEc5fUNegwmbMngExs/OD0IDuHedYJ5Eh/5ZseQdDLmNS2fDh6EXmYrf2KYkb3b a9xA/HOzmRFct45+tQ== =Y5jd -----END PGP SIGNATURE----- --Tao7UKjCLBAtpINvT4ketSuXk9Iww88Mg-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 27 07:21:00 2017 Received: (at 26201) by debbugs.gnu.org; 27 Mar 2017 11:21:00 +0000 Received: from localhost ([127.0.0.1]:46872 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1csShn-0003z5-Op for submit@debbugs.gnu.org; Mon, 27 Mar 2017 07:21:00 -0400 Received: from eggs.gnu.org ([208.118.235.92]:34485) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1csShl-0003yp-PC for 26201@debbugs.gnu.org; Mon, 27 Mar 2017 07:20:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1csShf-0003L6-7r for 26201@debbugs.gnu.org; Mon, 27 Mar 2017 07:20:52 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:54176) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csShY-0003CJ-UF; Mon, 27 Mar 2017 07:20:44 -0400 Received: from [193.50.110.68] (port=35156 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1csShY-0000Gk-6S; Mon, 27 Mar 2017 07:20:44 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Tobias Geerinckx-Rice Subject: Bandwidth when retrieving substitutes References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> Date: Mon, 27 Mar 2017 13:20:42 +0200 In-Reply-To: <87r31pyms2.fsf_-_@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\?\= \=\?utf-8\?Q\?\=22's\?\= message of "Wed, 22 Mar 2017 23:22:37 +0100") Message-ID: <8760ivm0dx.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hi there! ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > =E2=80=98guix publish=E2=80=99 should be faster and less resource-hungry = than Hydra. It > uses in-process gzip for nar compression instead of bzip2 (I chose level > 7, which seems to provide compression ratios close to what bzip2 > provides with its default compression level, while being 3 times > faster). Unlike Hydra it never forks so for instance, 404 responses for > .narinfo URLs should be quicker. Hopefully, that will improve the > worst-case (cache miss) throughput. Another interesting data point on the client side this time: --8<---------------cut here---------------start------------->8--- $ wget -O- https://mirror.hydra.gnu.org/nar/v6rq6j9wdx8ixsks05dxhxr26jgmr6z= 3-mysql-5.7.17 |bunzip2 >/dev/null --2017-03-27 13:12:50-- https://mirror.hydra.gnu.org/nar/v6rq6j9wdx8ixsks0= 5dxhxr26jgmr6z3-mysql-5.7.17 Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 200= 1:4ca0:2001:10:225:90ff:fedb:c720 Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:44= 3... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-nix-archive] Saving to: =E2=80=98STDOUT=E2=80=99 - [ <=3D> = ] 53.01M 9.29MB/s in 5.5s=20=20=20=20 2017-03-27 13:12:55 (9.57 MB/s) - written to stdout [55582050] $ wget -O- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxh= xr26jgmr6z3-mysql-5.7.17 |gunzip >/dev/null --2017-03-27 13:13:00-- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9= wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 200= 1:4ca0:2001:10:225:90ff:fedb:c720 Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:44= 3... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-nix-archive] Saving to: =E2=80=98STDOUT=E2=80=99 - [ <=3D> = ] 59.19M 40.8MB/s in 1.4s=20=20=20=20 2017-03-27 13:13:02 (40.8 MB/s) - written to stdout [62068901] $ wget -O- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9wdx8ixsks05dxh= xr26jgmr6z3-mysql-5.7.17 >/dev/null --2017-03-27 13:15:58-- https://mirror.hydra.gnu.org/guix/nar/gzip/v6rq6j9= wdx8ixsks05dxhxr26jgmr6z3-mysql-5.7.17 Resolving mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 200= 1:4ca0:2001:10:225:90ff:fedb:c720 Connecting to mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:44= 3... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-nix-archive] Saving to: =E2=80=98STDOUT=E2=80=99 - [ <=3D> = ] 59.19M 42.5MB/s in 1.4s=20=20=20=20 2017-03-27 13:16:00 (42.5 MB/s) - written to stdout [62068901] --8<---------------cut here---------------end--------------->8--- 40=C2=A0MB/s vs. 10=C2=A0MB/s! (Both items were cached on mirror.hydra.gnu= .org.) IOW, bunzip2 was the bottleneck when retrieving substitutes (and that=E2=80= =99s on an i7.) With =E2=80=98perf timechart=E2=80=99 we see that bunzip2 is in= deed busy all the time right from the start. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 27 14:46:25 2017 Received: (at 26201) by debbugs.gnu.org; 27 Mar 2017 18:46:25 +0000 Received: from localhost ([127.0.0.1]:47729 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1csZeq-0008Ew-VJ for submit@debbugs.gnu.org; Mon, 27 Mar 2017 14:46:25 -0400 Received: from relay5-d.mail.gandi.net ([217.70.183.197]:33133) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1csZeo-0008Eo-3W for 26201@debbugs.gnu.org; Mon, 27 Mar 2017 14:46:22 -0400 Received: from mfilter5-d.gandi.net (mfilter5-d.gandi.net [217.70.178.132]) by relay5-d.mail.gandi.net (Postfix) with ESMTP id EDA8C41C08A; Mon, 27 Mar 2017 20:46:20 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter5-d.gandi.net Received: from relay5-d.mail.gandi.net ([IPv6:::ffff:217.70.183.197]) by mfilter5-d.gandi.net (mfilter5-d.gandi.net [::ffff:10.0.15.180]) (amavisd-new, port 10024) with ESMTP id fM--jBD2BbD9; Mon, 27 Mar 2017 20:46:19 +0200 (CEST) X-Originating-IP: 81.241.166.83 Received: from [192.168.1.24] (83.166-241-81.adsl-dyn.isp.belgacom.be [81.241.166.83]) (Authenticated sender: me@tobias.gr) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id C2E5541C080; Mon, 27 Mar 2017 20:46:18 +0200 (CEST) Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> To: 26201@debbugs.gnu.org From: Tobias Geerinckx-Rice Message-ID: Date: Mon, 27 Mar 2017 20:47:17 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Pk7w1HSJck54nE4IR4jTd42qTMqc35B3H" X-Spam-Score: -3.0 (---) X-Debbugs-Envelope-To: 26201 Cc: ludo@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.0 (---) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Pk7w1HSJck54nE4IR4jTd42qTMqc35B3H Content-Type: multipart/mixed; boundary="XJW6iRmcWJDJMJEj0UK1wR4ebqKfWRX1P"; protected-headers="v1" From: Tobias Geerinckx-Rice To: 26201@debbugs.gnu.org Cc: ludo@gnu.org Message-ID: Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> In-Reply-To: <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> --XJW6iRmcWJDJMJEj0UK1wR4ebqKfWRX1P Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Guix, On 26/03/17 19:35, Tobias Geerinckx-Rice wrote: > I can try to do some simple tests tomorrow. Two observations: - =E2=80=98proxy_cache_lock_timeout=E2=80=99 alone won't suffice to seria= lise requests; =E2=80=98proxy_cache_lock_age=E2=80=99 must also be set to an equally r= idiculously long span. Otherwise, multiple requests will still be sent to =E2=80=98= guix publish=E2=80=99 if they are more than 5s apart. Bleh. (The problem then becomes that clients will stall while the file is being cached, as explained by Mark. curl patiently waited.) - Say client A requests a nar from =E2=80=98guix publish=E2=80=99 (no ngi= nx involved). If another client requests the same nar while A's still downloading, =E2=80=98guix publish=E2=80=99 will... silently drop A's connection? I was not expecting this. Kind regards, T G-R --XJW6iRmcWJDJMJEj0UK1wR4ebqKfWRX1P-- --Pk7w1HSJck54nE4IR4jTd42qTMqc35B3H Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY2V42DRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5lDMQf9HBpU 2KDmZmiLUKgzO4aJoSUpSLyH94HiIhtrgNCDFptLT4g/rN7rA+hTHPUgkdG56sws /faIMxay6A2O0idmMr6wK6yREkxjz6+fPsbsPg6kPmH36ZFGIwX0BsSZtahzgtzz O3848hIy212/RXd3VF/+PghK32iJm1EMoWzEtRL2OA5ESLeazPrAz6x7i0n4BDNc zWLTdnMio9kre0AfiiVota3yy6fMjonJbeQ1qaveeAr8p4IIQl6NmPheJfe9gTQy 60aFj33RSTA8vM7koMVcQmWYD+BOAsLNY39PeK5B8puS3Zk2ovYuBCHU+9oCSsSR ktoob0qOzjCwsAGmSw== =wxkr -----END PGP SIGNATURE----- --Pk7w1HSJck54nE4IR4jTd42qTMqc35B3H-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 28 10:47:28 2017 Received: (at 26201) by debbugs.gnu.org; 28 Mar 2017 14:47:28 +0000 Received: from localhost ([127.0.0.1]:49440 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cssPA-0000QA-3I for submit@debbugs.gnu.org; Tue, 28 Mar 2017 10:47:28 -0400 Received: from eggs.gnu.org ([208.118.235.92]:35796) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cssP8-0000Pw-4B for 26201@debbugs.gnu.org; Tue, 28 Mar 2017 10:47:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cssOz-0006OY-IB for 26201@debbugs.gnu.org; Tue, 28 Mar 2017 10:47:20 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:50841) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cssOz-0006OJ-FF; Tue, 28 Mar 2017 10:47:17 -0400 Received: from [193.50.110.98] (port=54594 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cssOy-00073m-R0; Tue, 28 Mar 2017 10:47:17 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Tobias Geerinckx-Rice Subject: Re: bug#26201: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish?= =?utf-8?Q?=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 Germinal an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Tue, 28 Mar 2017 16:47:14 +0200 In-Reply-To: (Tobias Geerinckx-Rice's message of "Mon, 27 Mar 2017 20:47:17 +0200") Message-ID: <87wpb931cd.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hey! Tobias Geerinckx-Rice skribis: > On 26/03/17 19:35, Tobias Geerinckx-Rice wrote: >> I can try to do some simple tests tomorrow. > > Two observations: > > - =E2=80=98proxy_cache_lock_timeout=E2=80=99 alone won't suffice to seria= lise requests; > =E2=80=98proxy_cache_lock_age=E2=80=99 must also be set to an equally r= idiculously > long span. Otherwise, multiple requests will still be sent to =E2=80=98= guix > publish=E2=80=99 if they are more than 5s apart. Bleh. > > (The problem then becomes that clients will stall while the file is > being cached, as explained by Mark. curl patiently waited.) Setting =E2=80=98proxy_cache_lock_timeout=E2=80=99 to 5s is reasonable I th= ink: if you=E2=80=99re unlucky, you wait for 5 seconds, and then we get =E2=80=98gu= ix publish=E2=80=99 threads serving the same request in parallel; in the most common case, there=E2=80=99s only ever one instance of a given request being served at a given time. > - Say client A requests a nar from =E2=80=98guix publish=E2=80=99 (no ngi= nx involved). > If another client requests the same nar while A's still downloading, > =E2=80=98guix publish=E2=80=99 will... silently drop A's connection? > I was not expecting this. That would be a bug. Do you have an easy way to reproduce? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 08 17:18:06 2017 Received: (at control) by debbugs.gnu.org; 8 Apr 2017 21:18:06 +0000 Received: from localhost ([127.0.0.1]:38533 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cwxkE-0003tm-FL for submit@debbugs.gnu.org; Sat, 08 Apr 2017 17:18:06 -0400 Received: from eggs.gnu.org ([208.118.235.92]:45549) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cwxkD-0003tG-3Y for control@debbugs.gnu.org; Sat, 08 Apr 2017 17:18:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cwxk2-0004GO-TV for control@debbugs.gnu.org; Sat, 08 Apr 2017 17:17:59 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:46961) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cwxk2-0004GK-Pv for control@debbugs.gnu.org; Sat, 08 Apr 2017 17:17:54 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:38760 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cwxk2-0007Cc-4W for control@debbugs.gnu.org; Sat, 08 Apr 2017 17:17:54 -0400 Date: Sat, 08 Apr 2017 23:17:52 +0200 Message-Id: <87wpauoayn.fsf@gnu.org> To: control@debbugs.gnu.org From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: control message for bug #26201 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) retitle 26201 Downloading substitutes is too slow upon nginx cache misses From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 08 17:18:16 2017 Received: (at control) by debbugs.gnu.org; 8 Apr 2017 21:18:17 +0000 Received: from localhost ([127.0.0.1]:38536 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cwxkO-0003uE-Lu for submit@debbugs.gnu.org; Sat, 08 Apr 2017 17:18:16 -0400 Received: from eggs.gnu.org ([208.118.235.92]:45556) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cwxkN-0003tz-AI for control@debbugs.gnu.org; Sat, 08 Apr 2017 17:18:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cwxkD-0004HZ-Bx for control@debbugs.gnu.org; Sat, 08 Apr 2017 17:18:10 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:46962) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cwxkD-0004HV-8z for control@debbugs.gnu.org; Sat, 08 Apr 2017 17:18:05 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:38762 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1cwxkC-0007Cx-Iw for control@debbugs.gnu.org; Sat, 08 Apr 2017 17:18:05 -0400 Date: Sat, 08 Apr 2017 23:18:03 +0200 Message-Id: <87vaqeoayc.fsf@gnu.org> To: control@debbugs.gnu.org From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: control message for bug #26201 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) severity 26201 important From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 17 17:36:24 2017 Received: (at 26201) by debbugs.gnu.org; 17 Apr 2017 21:36:24 +0000 Received: from localhost ([127.0.0.1]:53067 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d0EJs-0003Vl-6O for submit@debbugs.gnu.org; Mon, 17 Apr 2017 17:36:24 -0400 Received: from eggs.gnu.org ([208.118.235.92]:59016) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d0EJq-0003VZ-Os for 26201@debbugs.gnu.org; Mon, 17 Apr 2017 17:36:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d0EJk-0000tk-JL for 26201@debbugs.gnu.org; Mon, 17 Apr 2017 17:36:17 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51591) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d0EJd-0000sA-Ut; Mon, 17 Apr 2017 17:36:09 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:38552 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1d0EJd-0005LC-8E; Mon, 17 Apr 2017 17:36:09 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mark H Weaver Subject: Re: bug#26201: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish?= =?utf-8?Q?=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <87d1d710xc.fsf@gnu.org> Date: Mon, 17 Apr 2017 23:36:06 +0200 In-Reply-To: <87d1d710xc.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 24 Mar 2017 10:25:35 +0100") Message-ID: <87inm2ogxl.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: Tobias Geerinckx-Rice , 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hello, ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > Other solutions I=E2=80=99ve thought about: > > 1. Produce narinfos and nars periodically rather than on-demand and > serve them as static files. > > pros: better HTTP latency and bandwidth > pros: allows us to add a Content-Length for nars > cons: doesn=E2=80=99t reduce load on hydra.gnu.org > cons: introduces arbitrary delays in delivering nars > cons: difficult/expensive to know what new store items are available > > 2. Produce a narinfo and corresponding nar the first time they are > requested. So, the first time we receive =E2=80=9CGET foo.narinfo= =E2=80=9D, return > 404 and spawn a thread to compute foo.narinfo and foo.nar. Return > 200 only when both are ready. > > The precomputed nar{,info}s would be kept in a cache and we could > make sure a narinfo and its nar have the same lifetime, which > addresses one of the problems we have. > > pros: better HTTP latency and bandwidth > pros: allows us to add a Content-Length for nars > pros: helps keep narinfo/nar lifetime in sync > cons: doesn=E2=80=99t reduce load on hydra.gnu.org > cons: exposes inconsistency between the store contents and the HTTP > response (you may get 404 even if the thing is actually in > store), but maybe that=E2=80=99s not a problem The =E2=80=98wip-publish-baking=E2=80=99 implements #2 as a new option to = =E2=80=98guix publish=E2=80=99. It gives some control on the upper bound on CPU usage si= nce we can specify how many worker threads are used. I=E2=80=99ll finish it soon so we can experiment with it. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 18 17:28:03 2017 Received: (at 26201) by debbugs.gnu.org; 18 Apr 2017 21:28:03 +0000 Received: from localhost ([127.0.0.1]:55275 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d0afL-0003FW-Cz for submit@debbugs.gnu.org; Tue, 18 Apr 2017 17:28:03 -0400 Received: from eggs.gnu.org ([208.118.235.92]:53672) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d0afK-0003F3-FY for 26201@debbugs.gnu.org; Tue, 18 Apr 2017 17:28:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d0afE-0000xb-Do for 26201@debbugs.gnu.org; Tue, 18 Apr 2017 17:27:57 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:35368) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d0af6-0000wh-Hi; Tue, 18 Apr 2017 17:27:48 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:36928 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1d0af5-0006vQ-Q9; Tue, 18 Apr 2017 17:27:48 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mark H Weaver Subject: Re: bug#26201: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish?= =?utf-8?Q?=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <87d1d710xc.fsf@gnu.org> Date: Tue, 18 Apr 2017 23:27:44 +0200 In-Reply-To: <87d1d710xc.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 24 Mar 2017 10:25:35 +0100") Message-ID: <87o9vts8xb.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: Tobias Geerinckx-Rice , 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > 2. Produce a narinfo and corresponding nar the first time they are > requested. So, the first time we receive =E2=80=9CGET foo.narinfo= =E2=80=9D, return > 404 and spawn a thread to compute foo.narinfo and foo.nar. Return > 200 only when both are ready. > > The precomputed nar{,info}s would be kept in a cache and we could > make sure a narinfo and its nar have the same lifetime, which > addresses one of the problems we have. > > pros: better HTTP latency and bandwidth > pros: allows us to add a Content-Length for nars > pros: helps keep narinfo/nar lifetime in sync > cons: doesn=E2=80=99t reduce load on hydra.gnu.org > cons: exposes inconsistency between the store contents and the HTTP > response (you may get 404 even if the thing is actually in > store), but maybe that=E2=80=99s not a problem Implemented in commit 00753f7038234a0f5a79be3ec9ab949840a18743. I=E2=80=99ll set up a test instance shortly. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 19 10:25:11 2017 Received: (at 26201) by debbugs.gnu.org; 19 Apr 2017 14:25:11 +0000 Received: from localhost ([127.0.0.1]:56940 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d0qXe-0001Ta-Uh for submit@debbugs.gnu.org; Wed, 19 Apr 2017 10:25:11 -0400 Received: from eggs.gnu.org ([208.118.235.92]:36364) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d0qXc-0001TJ-HY for 26201@debbugs.gnu.org; Wed, 19 Apr 2017 10:25:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d0qXU-0001rv-8e for 26201@debbugs.gnu.org; Wed, 19 Apr 2017 10:25:03 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:47117) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d0qXL-0001ow-C0; Wed, 19 Apr 2017 10:24:51 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:37924 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1d0qXK-0002bU-Hw; Wed, 19 Apr 2017 10:24:50 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mark H Weaver Subject: Heads-up: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix?= publish =?utf-8?Q?--cache=E2=80=99?= References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <87d1d710xc.fsf@gnu.org> <87o9vts8xb.fsf@gnu.org> Date: Wed, 19 Apr 2017 16:24:46 +0200 In-Reply-To: <87o9vts8xb.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 18 Apr 2017 23:27:44 +0200") Message-ID: <87vaq0o4pd.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: Tobias Geerinckx-Rice , 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > >> 2. Produce a narinfo and corresponding nar the first time they are >> requested. So, the first time we receive =E2=80=9CGET foo.narinfo= =E2=80=9D, return >> 404 and spawn a thread to compute foo.narinfo and foo.nar. Return >> 200 only when both are ready. >> >> The precomputed nar{,info}s would be kept in a cache and we could >> make sure a narinfo and its nar have the same lifetime, which >> addresses one of the problems we have. >> >> pros: better HTTP latency and bandwidth >> pros: allows us to add a Content-Length for nars >> pros: helps keep narinfo/nar lifetime in sync >> cons: doesn=E2=80=99t reduce load on hydra.gnu.org >> cons: exposes inconsistency between the store contents and the HTTP >> response (you may get 404 even if the thing is actually in >> store), but maybe that=E2=80=99s not a problem > > Implemented in commit 00753f7038234a0f5a79be3ec9ab949840a18743. > > I=E2=80=99ll set up a test instance shortly. I ended up deploying it on hydra.gnu.org directly. :-) Progressively the cached nar/narinfo at {,mirror.}hydra.gnu.org will be replaced with the new ones. Now, the /guix/nar URLs have a =E2=80=98Content-Length=E2=80=99 header you should see a progress bar when = downloading one of these: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix build vim The following file will be downloaded: /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566 @ substituter-started /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0= 566 /gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/libexec/= guix/substitute Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/ax5cm9gr1741pcq17w7b= hgss5nvq5470-vim-8.0.0566 (23.4MiB installed)... vim-8.0.0566 7.8MiB 385KiB/s 00:21 [= ####################] 100.0% @ substituter-succeeded /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0= .0566 /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566 --8<---------------cut here---------------end--------------->8--- This new caching scheme should put an end to caching of truncated nars in nginx, which has been too frequent lately. It should also mostly avoid the problem where we have a narinfo for something but not the corresponding nar, which leads to user frustration (=E2=80=98guix=E2=80=99 reports that the thing will be downloaded and event= ually fails with 410 =E2=80=9CGone=E2=80=9D while trying to download it), because =E2= =80=98guix publish=E2=80=99 caches narinfo/nar pairs together. I say =E2=80=9Cmostly=E2=80=9D because = nginx caching in front of =E2=80=98guix publish=E2=80=99 makes things more complicated. The bandwidth issue reported at the beginning of this thread should be mostly fixed: serving a narinfo or nar URL is now just sendfile(2), which is the best we can do; 404s on narinfo should be immediate. Of course, when the machine is overloaded, we=E2=80=99ll still experience increased latency and lower bandwidth, but that should be less acute than with the previous setting. Please report any problems you may have! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 25 06:11:32 2017 Received: (at control) by debbugs.gnu.org; 25 Apr 2017 10:11:32 +0000 Received: from localhost ([127.0.0.1]:38842 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d2xRU-0007qj-AI for submit@debbugs.gnu.org; Tue, 25 Apr 2017 06:11:32 -0400 Received: from eggs.gnu.org ([208.118.235.92]:52831) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d2xRS-0007qX-Pz for control@debbugs.gnu.org; Tue, 25 Apr 2017 06:11:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d2xRJ-0004d6-Fe for control@debbugs.gnu.org; Tue, 25 Apr 2017 06:11:25 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:34114) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d2xRJ-0004d1-DA for control@debbugs.gnu.org; Tue, 25 Apr 2017 06:11:21 -0400 Received: from [89.131.103.136] (port=49222 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1d2xRI-0005VR-Gk for control@debbugs.gnu.org; Tue, 25 Apr 2017 06:11:21 -0400 Date: Tue, 25 Apr 2017 12:11:16 +0200 Message-Id: <87h91cdcfv.fsf@gnu.org> To: control@debbugs.gnu.org From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: control message for bug #26201 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) tags 26201 fixed close 26201 From debbugs-submit-bounces@debbugs.gnu.org Wed May 03 04:11:51 2017 Received: (at 26201) by debbugs.gnu.org; 3 May 2017 08:11:52 +0000 Received: from localhost ([127.0.0.1]:51864 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d5pO3-0004nK-M6 for submit@debbugs.gnu.org; Wed, 03 May 2017 04:11:51 -0400 Received: from world.peace.net ([50.252.239.5]:43989) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d5pO1-0004n7-P5 for 26201@debbugs.gnu.org; Wed, 03 May 2017 04:11:50 -0400 Received: from pool-72-93-31-169.bstnma.east.verizon.net ([72.93.31.169] helo=jojen) by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1d5pNv-0006vb-Jc; Wed, 03 May 2017 04:11:43 -0400 From: Mark H Weaver To: Tobias Geerinckx-Rice Subject: Re: bug#26201: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish?= =?utf-8?Q?=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> Date: Wed, 03 May 2017 04:11:31 -0400 In-Reply-To: <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> (Tobias Geerinckx-Rice's message of "Sun, 26 Mar 2017 19:35:25 +0200") Message-ID: <877f1yjr64.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 26201 Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) Reviving an old thread... Tobias Geerinckx-Rice writes: >> IMO, the best solution is to *never* generate nars on Hydra in response >> to client requests, but rather to have the build slaves pack and >> compress the nars, copy them to Hydra, and then serve them as static >> files using nginx. > > A true mirror at last! Do we have the disc space for that? > > And could Hydra actually handle compressing *everything*, without an > infinitely growing back-log? I don't have access to any statistics, but > I'm guessing that a fair number of package+versions are never actually > requested, and hence never compressed. This would change that. Actually, IIUC, the build slaves are _already_ compressing everything, and they always have. They compress the build outputs for transmission back to the master machine. In the current framework, the master machine immediately decompresses them upon receipt, and this compression and decompression is considered an internal detail of the network transport. Currently, the master machine stores all build outputs uncompressed in /gnu/store, and then later recompresses them for transmission to users and other build slaves. The needless decompression and recompression is a tremendous amount of wasted work on our master machine. That it's all stored uncompressed is also a significant waste of disk space, which leads to significant additional costs during garbage collection. Essentially, my proposal is for the build slaves to be modified to prepare the compressed NARs in a form suitable for delivery to end users (and other build slaves) with minimal processing by our master node. The master node would be significantly modified to receive, store, and forward NARs explicitly, without ever decompressing them. As far as I can tell, this would mean strictly less work to do and less data to store for every machine and in every case. Ludovic has pointed out that we cannot do this because Hydra must add its digital signature, and that this digital signature is stored within the compressed NAR. Therefore, we cannot avoid having the master machine decompress and recompress every NAR that is delivered to users. In my opinion, we should change the way we sign NARs. Signatures should be external to the NARs, not internal. Not only would this allow us to decentralize production of our NARs, but more importantly, it would enable a community of independent builders to add their signatures to a common pool of NARs. Having a common pool of NARs enables us to store these NARs in a shared distribution network without duplication. We cannot even have a common pool of NARs if they contain build-farm-specific data such as signatures. Thoughts? Mark From debbugs-submit-bounces@debbugs.gnu.org Wed May 03 05:25:55 2017 Received: (at 26201) by debbugs.gnu.org; 3 May 2017 09:25:55 +0000 Received: from localhost ([127.0.0.1]:51909 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d5qXi-0006UO-IS for submit@debbugs.gnu.org; Wed, 03 May 2017 05:25:55 -0400 Received: from eggs.gnu.org ([208.118.235.92]:47207) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d5qXh-0006UC-4Y for 26201@debbugs.gnu.org; Wed, 03 May 2017 05:25:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d5qXb-0005cl-00 for 26201@debbugs.gnu.org; Wed, 03 May 2017 05:25:48 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:44119) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d5qXW-0005bI-4Z; Wed, 03 May 2017 05:25:42 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:54578 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1d5qXV-00063M-9M; Wed, 03 May 2017 05:25:41 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mark H Weaver Subject: Re: bug#26201: hydra.gnu.org uses =?utf-8?Q?=E2=80=98guix_publish?= =?utf-8?Q?=E2=80=99?= for nars and narinfos References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> <877f1yjr64.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 14 =?utf-8?Q?Flor=C3=A9al?= an 225 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Wed, 03 May 2017 11:25:38 +0200 In-Reply-To: <877f1yjr64.fsf@netris.org> (Mark H. Weaver's message of "Wed, 03 May 2017 04:11:31 -0400") Message-ID: <87k25ywaul.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 26201 Cc: Tobias Geerinckx-Rice , 26201@debbugs.gnu.org, guix-sysadmin@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hello, Mark H Weaver skribis: > Actually, IIUC, the build slaves are _already_ compressing everything, > and they always have. They compress the build outputs for transmission > back to the master machine. In the current framework, the master > machine immediately decompresses them upon receipt, and this compression > and decompression is considered an internal detail of the network > transport. > > Currently, the master machine stores all build outputs uncompressed in > /gnu/store, and then later recompresses them for transmission to users > and other build slaves. The needless decompression and recompression is > a tremendous amount of wasted work on our master machine. That it's all > stored uncompressed is also a significant waste of disk space, which > leads to significant additional costs during garbage collection. > > Essentially, my proposal is for the build slaves to be modified to > prepare the compressed NARs in a form suitable for delivery to end users > (and other build slaves) with minimal processing by our master node. > The master node would be significantly modified to receive, store, and > forward NARs explicitly, without ever decompressing them. As far as I > can tell, this would mean strictly less work to do and less data to > store for every machine and in every case. I agree that the redundant compression/decompression is terrible. Yet I=E2=80=99m not sure how to architect a solution where compression is perfo= rmed by build machines. The main issue is that offloading and publication are two independent mechanisms, as things are. Maybe each build machine for a build farm use-case we could have a =E2=80=9Csemi-offloading=E2=80=9D mechanism whereby the master spawns a rem= ote build without retrieving its result, something akin to: GUIX_DAEMON_SOCKET=3Dssh://build-machine.example.org \ guix build /gnu/store/=E2=80=A6-foo.drv In addition, the build machine would publish its result via =E2=80=98guix publish=E2=80=99, which the master could then simply mirror and cache with nginx. There=E2=80=99s the issue of signatures, but perhaps we could have a more sophisticated PKI and have the master delegate to build machines=E2=80=A6 Then there are other issues such as that of synchronizing the TTL of a narinfo and its corresponding nar, which --cache addresses. Tricky! > Ludovic has pointed out that we cannot do this because Hydra must add > its digital signature, and that this digital signature is stored within > the compressed NAR. Therefore, we cannot avoid having the master > machine decompress and recompress every NAR that is delivered to users. > > In my opinion, we should change the way we sign NARs. Signatures should > be external to the NARs, not internal. Not only would this allow us to > decentralize production of our NARs, but more importantly, it would > enable a community of independent builders to add their signatures to a > common pool of NARs. Having a common pool of NARs enables us to store > these NARs in a shared distribution network without duplication. We > cannot even have a common pool of NARs if they contain > build-farm-specific data such as signatures. Currently the signature is in the narinfos, not in nars proper=C2=B9. So we can already add signatures on an externally provided nar, for instance. There=E2=80=99s a silly limitation currently, which is that the signature is computed over all the fields of the narinfo. That=E2=80=99s silly because = it means that if you change, say, the compression format or the URL of the nar, then the signature becomes invalid. We should fix that at some point. Ludo=E2=80=99. =C2=B9 For =E2=80=98guix publish=E2=80=99. =E2=80=98guix archive --export= =E2=80=99 appends a signature to the nar set. From unknown Sun Jun 22 11:33:44 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 31 May 2017 11:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator