GNU bug report logs - #26201
Downloading substitutes is too slow upon nginx cache misses

Previous Next

Full log

Message #74 received at 26201 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: mhw <at> netris.org
Cc: ludo <at> gnu.org, 26201 <at> debbugs.gnu.org, guix-sysadmin <at> gnu.org
Subject: Re: bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos
Date: Sun, 26 Mar 2017 19:35:25 +0200

[Message part 1 (text/plain, inline)]

Mark,

On 24/03/17 09:12, Mark H Weaver wrote:
> IIUC, with "proxy_cache_lock on", we have two choices of how other
> client requests will be treated:
> 
>   [badly, ed.]

Eh. You're probably (and disappointingly) right.

When configuring my little cache, I had a clear idea of how such a cache
 should work (basically, your last scenario below), then looked at the
nginx documentation to find what I had in mind. ‘proxy_cache_lock’ matched.

I should have been more pessimistic and done more testing.
Shame on me, &c. Too much other things on my mind. :-/

> Or at least that's what I'd expect based on my reading of the nginx docs
> linked above.  I haven't tried it.

I can try to do some simple tests tomorrow.

> IMO, the best solution is to *never* generate nars on Hydra in response
> to client requests, but rather to have the build slaves pack and
> compress the nars, copy them to Hydra, and then serve them as static
> files using nginx.

A true mirror at last! Do we have the disc space for that?

And could Hydra actually handle compressing *everything*, without an
infinitely growing back-log? I don't have access to any statistics, but
I'm guessing that a fair number of package+versions are never actually
requested, and hence never compressed. This would change that.

> A far inferior solution, but possibly acceptable and closer to the
> current approach, would be to arrange for all concurrent responses for
> the same nar to be sent incrementally from a single nar-packing process.
> More concretely, while packing and sending a nar response to the first
> client, the data would also be written to a file.  Subsequent requests
> for the same nar would be serviced using the equivalent of:
> 
>   tail --bytes=+0 --follow FILENAME
> 
> This way, no one would have to wait an hour to receive the first byte.

^ This is so obviously the right solution, that it would be
disappointing if nginx really couldn't be made to do it. It already
buffers proxy responses to a temporary file anyway...

Kind regards,

T G-R

[signature.asc (application/pgp-signature, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.