GNU bug report logs -
#45018
[PATCH 0/6] Process and connection reuse for substitutions
Previous Next
Reported by: Ludovic Courtès <ludo <at> gnu.org>
Date: Thu, 3 Dec 2020 10:14:02 UTC
Severity: normal
Tags: patch
Done: Ludovic Courtès <ludo <at> gnu.org>
Bug is archived. No further changes may be made.
Full log
View this message in rfc822 format
[Message part 1 (text/plain, inline)]
Your bug report
#45018: [PATCH 0/6] Process and connection reuse for substitutions
which was filed against the guix-patches package, has been closed.
The explanation is attached below, along with your original report.
If you require more details, please reply to 45018 <at> debbugs.gnu.org.
--
45018: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=45018
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems
[Message part 2 (message/rfc822, inline)]
Ludovic Courtès <ludo <at> gnu.org> skribis:
> daemon: 'Agent' constructor takes a list of environment variables.
> daemon: Use 'Agent' to spawn 'guix substitute --query'.
> daemon: Factorize substituter agent spawning.
> daemon: Run 'guix substitute --substitute' as an agent.
> substitute: Cache and reuse connections while substituting.
> daemon: Raise an error if substituter doesn't send the expected hash.
Pushed as bfe4cdf88ee3e88910d22291a4c745462f2d6417, followed by an
update of the ‘guix’ package.
Ludo’.
[Message part 3 (message/rfc822, inline)]
Hi Guix!
The attached patches optimize substitute downloads by: (1) keeping
a single ‘guix substitute’ process for all the substitutes (instead
of respawning a new one each time a store item is substituted), and
(2) reusing connections to substitute servers in between substitutions.
(Note that ‘guix publish’ does not keep connections alive, but
on ci.guix.gnu.org we run nginx in front of ‘guix publish’, and nginx
keeps them alive.)
Anecdotally, the effect is a 10% improvement on the wall-clock time
of ‘guix build --sources=transitive vim’ on my laptop, from 29s to 26s.
Of course it cannot be much better since the rest of the time is spent
actually retrieving bits over the network.
Overall the impact depends on a number of factors. My laptop has
an SSD and I have fiber-to-the-home (FFTH) with low latency: forking
and initiating a TLS connection to ci.guix.gnu.org are both quite cheap,
so I probably don’t benefit that much. The impact may be more acute
on a low-end device.
In the ‘guix build --sources=transitive vim’ case, there are a few
large tarballs and several small ones. The gain is in the lack of
a pause time in between small tarballs. Also this case is quite
advantageous: there’s no decompression and no unpacking happening.
Anyway, I think it’s a welcome improvement, especially when
downloading lots of stuff such as during the initial system installation.
Plus, there are more deletions than insertions. :-)
Thoughts?
Ludo’.
Ludovic Courtès (6):
daemon: 'Agent' constructor takes a list of environment variables.
daemon: Use 'Agent' to spawn 'guix substitute --query'.
daemon: Factorize substituter agent spawning.
daemon: Run 'guix substitute --substitute' as an agent.
substitute: Cache and reuse connections while substituting.
daemon: Raise an error if substituter doesn't send the expected hash.
guix/http-client.scm | 12 +-
guix/progress.scm | 8 +-
guix/scripts/substitute.scm | 215 +++++++++++++++++++++++-------------
nix/libstore/build.cc | 166 ++++++++++++++--------------
nix/libstore/local-store.cc | 170 +++++++---------------------
nix/libstore/local-store.hh | 25 +----
nix/libutil/util.cc | 6 +-
nix/libutil/util.hh | 7 +-
tests/substitute.scm | 98 +++++++++-------
9 files changed, 350 insertions(+), 357 deletions(-)
--
2.29.2
This bug report was last modified 4 years and 243 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.