GNU bug report logs - #39728
[PATCH] Allow parallel downloads and builds

Previous Next

Full log

View this message in rfc822 format

From: Ludovic Courtès <ludo <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 39728 <at> debbugs.gnu.org, Pierre Neidhardt <mail <at> ambrevar.xyz>, zimoun <zimon.toutoune <at> gmail.com>
Subject: [bug#39728] [PATCH v2] Allow parallel downloads and builds
Date: Fri, 26 Nov 2021 11:16:16 +0100

Hi!

Julien Lepiller <julien <at> lepiller.eu> skribis:

> I would still say yes, because the output from berlin is often much
> less than my throughput. With multiple downloads in parallel it at
> least feels quicker, probably because I can download at full speed.

It would be nice to measure that because like I wrote, I think we’re
pretty much network-bound these days, at least with zstd and
uncompressed downloads.

> In any case, I see often a build start while downloads are in progress,
> so I think it's still a win if you can get a few derivations built
> while waiting for a big download to finish at the same time :)

True!  Overlapping downloads and builds sounds like a good idea.

> At some point we might want to prioritize builds/downloads that help
> unlock as much builds as possible early, so we don't have builds
> waiting for downloads.

Right now the daemon starts with substitutes and builds afterwards.

BTW, we’re assuming downloads = substitutes in this whole discussion,
but we could/should take fixed-output derivations into account too.

I’ll take a closer look later on…

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.