GNU bug report logs - #44175
[optimization] Grafting is too slow

Previous Next

To reply to this bug, email your comments to 44175 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Lars-Dominik Braun <ldb <at> leibniz-psychology.org>
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 bug-guix <bug-guix <at> gnu.org>
Subject: [optimization] Grafting is too slow
Date: Fri, 23 Oct 2020 09:07:35 -0400

Hello Lars,

Lars-Dominik Braun <ldb <at> leibniz-psychology.org> writes:

> Hi Maxim,
>
>> Judging from the above, it seems this issue has been resolved.
> grafting is still a performance issue imo. Compare for example:
>
> $ time guix environment --ad-hoc  --search-paths r-learnr
> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 210% cpu 2,844 total
> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 0,08s system 164% cpu 1,277 total

I'm opening a new issue to track optimizing the grafting code, since
it's independent of environments (grafts are applied anytime a
derivation is built, AFAICT).  Grafting is inherently IO-bound, since it
must scan all the built artifacts of a grafted package (from its
dependency that was 'replaced' all the way up), if I understood
correctly.  "info (guix) Security Updates" gives some information.

Perhaps we could benchmark how fast our code currently is for grafting,
and compare it with the fastest sed-like utility out there, as a
starting point, to have an idea of how much there is to gain from
optimization.

Maxim

Message #8 received at 44175 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb <at> leibniz-psychology.org>
Subject: Re: bug#44175: [optimization] Grafting is too slow
Date: Fri, 23 Oct 2020 23:17:49 +0200

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> Lars-Dominik Braun <ldb <at> leibniz-psychology.org> writes:
>
>> Hi Maxim,
>>
>>> Judging from the above, it seems this issue has been resolved.
>> grafting is still a performance issue imo. Compare for example:
>>
>> $ time guix environment --ad-hoc  --search-paths r-learnr
>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 210% cpu 2,844 total
>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 0,08s system 164% cpu 1,277 total
>
> I'm opening a new issue to track optimizing the grafting code, since
> it's independent of environments (grafts are applied anytime a
> derivation is built, AFAICT).  Grafting is inherently IO-bound,

What is slow above is not grafting itself: it’s determining what to
graft that takes CPU time.

I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
should we close this one?

Thanks,
Ludo’.

Message #11 received at 44175 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb <at> leibniz-psychology.org>
Subject: Re: bug#44175: [optimization] Grafting is too slow
Date: Sat, 24 Oct 2020 22:33:08 -0400

Hello!

Ludovic Courtès <ludo <at> gnu.org> writes:

> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> Lars-Dominik Braun <ldb <at> leibniz-psychology.org> writes:
>>
>>> Hi Maxim,
>>>
>>>> Judging from the above, it seems this issue has been resolved.
>>> grafting is still a performance issue imo. Compare for example:
>>>
>>> $ time guix environment --ad-hoc  --search-paths r-learnr
>>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 210% cpu 2,844 total
>>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 0,08s system 164% cpu 1,277 total
>>
>> I'm opening a new issue to track optimizing the grafting code, since
>> it's independent of environments (grafts are applied anytime a
>> derivation is built, AFAICT).  Grafting is inherently IO-bound,
>
> What is slow above is not grafting itself: it’s determining what to
> graft that takes CPU time.

On my system, grafting seems IO rather than CPU bound, I'm guessing
because of the need to scan all the files for strings to replace in the
graft process.

> I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
> should we close this one?

Many optimizations were made in the above issue that were not related to
the grafting process, so to me a fresh entry such as this one is clearer
to follow.  That said, feel free to proceed as you see fit, being the
issue "owner" :-).

Thanks,

Maxim

Message #14 received at 44175 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 44175 <at> debbugs.gnu.org, Lars-Dominik Braun <ldb <at> leibniz-psychology.org>
Subject: Re: bug#44175: [optimization] Grafting is too slow
Date: Sun, 25 Oct 2020 17:43:12 +0100

Hi!

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

>>>> $ time guix environment --ad-hoc  --search-paths r-learnr
>>>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 210% cpu 2,844 total
>>>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>>>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 0,08s system 164% cpu 1,277 total
>>>
>>> I'm opening a new issue to track optimizing the grafting code, since
>>> it's independent of environments (grafts are applied anytime a
>>> derivation is built, AFAICT).  Grafting is inherently IO-bound,
>>
>> What is slow above is not grafting itself: it’s determining what to
>> graft that takes CPU time.
>
> On my system, grafting seems IO rather than CPU bound, I'm guessing
> because of the need to scan all the files for strings to replace in the
> graft process.

Yes, you’re right of course.  But I think in the example above Lars was
reporting the run time of the ‘guix’ command when the graft itself is
already built.  Just the extra code to compute the graft derivation (not
actually building them) leads to x2 wall-clock time or so.

>> I had reopened the initial bug at <https://issues.guix.gnu.org/41702>;
>> should we close this one?
>
> Many optimizations were made in the above issue that were not related to
> the grafting process, so to me a fresh entry such as this one is clearer
> to follow.  That said, feel free to proceed as you see fit, being the
> issue "owner" :-).

Alright, thanks!

Ludo’.

Message #17 received at 44175 <at> debbugs.gnu.org (full text, mbox):

From: Lars-Dominik Braun <ldb <at> leibniz-psychology.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44175 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#44175: [optimization] Grafting is too slow
Date: Mon, 26 Oct 2020 08:56:34 +0100

[Message part 1 (text/plain, inline)]

Hi,

> Yes, you’re right of course.  But I think in the example above Lars was
> reporting the run time of the ‘guix’ command when the graft itself is
> already built.  Just the extra code to compute the graft derivation (not
> actually building them) leads to x2 wall-clock time or so.
yes, Ludo is right. The numbers I reported are with grafts already built, warm caches and
local socket communication. So we’re only seeing the CPU-bound computation
here. Sorry for the confusion. I should have been more specific in the other
issue.

Cheers,
Lars

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.