GNU bug report logs - #51472
substitute servers should be preferred according to their coverage rate

Previous Next

To reply to this bug, email your comments to 51472 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: bug-guix <bug-guix <at> gnu.org>
Subject: substitute servers should be preferred according to their coverage
 rate
Date: Thu, 28 Oct 2021 22:07:39 -0400

Hello,

When using substitute servers discovery, I've noticed that if one of the
substitute servers doesn't have any substitutes available, it'll keep
getting tried instead of others, leading to a slide-show of substitutes
updates such as:

--8<---------------cut here---------------start------------->8---
normalized load on machine '127.0.0.1' is 0.04
building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
normalized load on machine '127.0.0.1' is 0.04
building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://127.0.0.1:8080'... 100.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://127.0.0.1:8080'... 100.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
--8<---------------cut here---------------end--------------->8---

We should implement some scheme to prefer querying high-substitute
servers first, instead of wasting time querying servers always failed
queries; this would greatly improve performance when using substitute
discovery for example combined with low coverage.

Thanks!

Maxim

Message #8 received at 51472 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 51472 <at> debbugs.gnu.org
Subject: Re: bug#51472: substitute servers should be preferred according to
 their coverage rate
Date: Sun, 07 Nov 2021 16:11:04 +0100

Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> When using substitute servers discovery, I've noticed that if one of the
> substitute servers doesn't have any substitutes available, it'll keep
> getting tried instead of others, leading to a slide-show of substitutes
> updates such as:
>
> normalized load on machine '127.0.0.1' is 0.04
> building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
> process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
> normalized load on machine '127.0.0.1' is 0.04
> building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%

We’d need to check why this particular server is checked repeatedly.
The fact that it displays “0.0%” doesn’t mean that the server lacks
substitutes, but that it does not reply to ‘GET /xyz.narinfo’ requests,
for example because it’s off-line (see
<https://issues.guix.gnu.org/48808>.)

> We should implement some scheme to prefer querying high-substitute
> servers first, instead of wasting time querying servers always failed
> queries; this would greatly improve performance when using substitute
> discovery for example combined with low coverage.

There are several problems with that.  First one is that you can’t tell
what substitute coverage is until you’ve actually made those GET
requests.  Second one is that substitute coverage varies and it’s not an
absolute measure; for example, if a server provides substitutes for only
0.1% of all the packages, but that’s precisely the 0.1% you care about,
it’s more valuable than the one that has 99% of the packages but lacks
those you want.

There are other issues such as the fact that current semantics is to
respect the order of substitute URLs, which is presumably chosen by the
user according to their own criteria: download speed, bandwidth usage,
etc.

I hope this makes sense!

Ludo’.

Message #11 received at 51472 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 51472 <at> debbugs.gnu.org, GNU Debbugs <control <at> debbugs.gnu.org>
Subject: Re: bug#51472: substitute servers should be preferred according to
 their coverage rate
Date: Mon, 10 Jan 2022 22:43:33 -0500

merge 48808 51472
thanks

Hello Ludovic,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> When using substitute servers discovery, I've noticed that if one of the
>> substitute servers doesn't have any substitutes available, it'll keep
>> getting tried instead of others, leading to a slide-show of substitutes
>> updates such as:
>>
>> normalized load on machine '127.0.0.1' is 0.04
>> building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
>> process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
>> normalized load on machine '127.0.0.1' is 0.04
>> building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
>> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
>
> We’d need to check why this particular server is checked repeatedly.
> The fact that it displays “0.0%” doesn’t mean that the server lacks
> substitutes, but that it does not reply to ‘GET /xyz.narinfo’ requests,
> for example because it’s off-line (see
> <https://issues.guix.gnu.org/48808>.)
>
>> We should implement some scheme to prefer querying high-substitute
>> servers first, instead of wasting time querying servers always failed
>> queries; this would greatly improve performance when using substitute
>> discovery for example combined with low coverage.
>
> There are several problems with that.  First one is that you can’t tell
> what substitute coverage is until you’ve actually made those GET
> requests.  Second one is that substitute coverage varies and it’s not an
> absolute measure; for example, if a server provides substitutes for only
> 0.1% of all the packages, but that’s precisely the 0.1% you care about,
> it’s more valuable than the one that has 99% of the packages but lacks
> those you want.
>
> There are other issues such as the fact that current semantics is to
> respect the order of substitute URLs, which is presumably chosen by the
> user according to their own criteria: download speed, bandwidth usage,
> etc.
>
> I hope this makes sense!

It does!  I agree that it'd be tricky to get this right; makes me
realize that my problem is probably due to #48808, and fixing that one
would probably have avoided that bug report :-).

I'm merging this one with 48808.

Thank you!

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.