GNU bug report logs - #22990
Grafts leads to inefficient substitute info retrieval

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: ludo <at> gnu.org (Ludovic Courtès)
Subject: bug#22990: closed (Re: bug#22990: Grafts leads to inefficient
 substitute info retrieval)
Date: Mon, 30 Mar 2020 02:36:51 +0000

[Message part 1 (text/plain, inline)]

Your bug report

#22990: Grafts leads to inefficient substitute info retrieval

which was filed against the guix package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to 22990 <at> debbugs.gnu.org.

-- 
22990: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22990
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Ludovic Courtès <ludo <at> gnu.org>
To: 22990-done <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Sun, 29 Mar 2020 15:41:24 +0200

Hi younger self!

ludo <at> gnu.org (Ludovic Courtès) skribis:

> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>
>   1. Fetching substitute info about the things being built so that it
>      can determine its references, which in turns allows it to determine
>      whether they need to be grafted.
>
>   2. Building stuff, as a last resort, so that it can determine its
>      references.
>
> Case #1 is hopefully going to be the most common.
>
> The problem with #1 is that when building a profile, we do one
> ‘package-derivation’ call for each package in the profile, which
> translates in one ‘graft-derivation’ call for each relevant package¹,
> which translates into one ‘references/substitutes’ call for each.
>
> Concretely, what this means is this:
>
>   $ guix package -u
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   […]
>   The following files would be downloaded:
>
> Each of the initial “updating list” message corresponds to an HTTP
> request for a single narinfo file, which can take around 1 second.

This is finally fixed with commit
710854304b1ab29332edcb76f3de532e0724c197, as discussed earlier this
week¹.  \o/

The solution is simpler and more orthogonal than what was envisioned
earlier in this thread, which is nice.

I also potentially addresses other uses of “dynamic dependencies”
elsewhere in the code base, such as ‘remote-eval’ calls in ‘guix
deploy’.

Feedback welcome!

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00337.html

[Message part 3 (message/rfc822, inline)]

From: ludo <at> gnu.org (Ludovic Courtès)
To: bug-guix <at> gnu.org
Subject: Grafts leads to inefficient substitute info retrieval
Date: Fri, 11 Mar 2016 17:52:59 +0100

As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:

  1. Fetching substitute info about the things being built so that it
     can determine its references, which in turns allows it to determine
     whether they need to be grafted.

  2. Building stuff, as a last resort, so that it can determine its
     references.

Case #1 is hopefully going to be the most common.

The problem with #1 is that when building a profile, we do one
‘package-derivation’ call for each package in the profile, which
translates in one ‘graft-derivation’ call for each relevant package¹,
which translates into one ‘references/substitutes’ call for each.

Concretely, what this means is this:

  $ guix package -u
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  […]
  The following files would be downloaded:

Each of the initial “updating list” message corresponds to an HTTP
request for a single narinfo file, which can take around 1 second.

Instead, the ideal thing would be to fetch the narinfo files for all the
relevant packages at once; that way, we’d spawn ‘guix substitute’ only
once, and it would benefit from HTTP pipelining (one round-trip instead
of N.)

To achieve this, I’m thinking of extending gexp code such that gexp
compilers can return a list of applicable grafts.  The ‘package’
compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
‘graft-derivation’.

I’ll give it a try and report back.

Ludo’.

¹ A package is “relevant” if ‘package-grafts’ returns a non-empty list.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.