GNU bug report logs - #39813
Running Guix in a Virtual Machine - says 1 GB RAM is enough, but it isn't for guix pull

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 39813 in the body.
You can then email your comments to 39813 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: <bug-guix <at> gnu.org>
Subject: Running Guix in a Virtual Machine - says 1 GB RAM is enough, but it
 isn't for guix pull
Date: Thu, 27 Feb 2020 21:06:01 +0100

[Message part 1 (text/plain, inline)]

Hi,

I just got a report that with Guix in a virtual Machine (like described in the
manual in 8.16), guix pull does not actually work[1] with 1 GB of RAM.
It does work fine with 4 GB of RAM.

[1] build .....guix-packages-base.drv failed

[Message part 2 (application/pgp-signature, inline)]

Message #10 received at 39813-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 39813-done <at> debbugs.gnu.org
Subject: Re: bug#39813: Running Guix in a Virtual Machine - says 1 GB RAM is
 enough, but it isn't for guix pull
Date: Wed, 13 Jul 2022 10:00:56 -0400

Hi Danny,

Danny Milosavljevic <dannym <at> scratchpost.org> writes:

> Hi,
>
> I just got a report that with Guix in a virtual Machine (like described in the
> manual in 8.16), guix pull does not actually work[1] with 1 GB of RAM.
> It does work fine with 4 GB of RAM.

I don't see any reference of 1 GiB being enough in our current version
of the manual.  If you do, please let me know.

Closing for now.

Thanks,

Maxim

Message #13 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Csepp <raingloom <at> riseup.net>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>, bug-guix <at> gnu.org,
 39813-done <at> debbugs.gnu.org
Subject: Re: bug#39813: Running Guix in a Virtual Machine - says 1 GB RAM is
 enough, but it isn't for guix pull
Date: Wed, 13 Jul 2022 18:58:58 +0200

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Hi Danny,
>
> Danny Milosavljevic <dannym <at> scratchpost.org> writes:
>
>> Hi,
>>
>> I just got a report that with Guix in a virtual Machine (like described in the
>> manual in 8.16), guix pull does not actually work[1] with 1 GB of RAM.
>> It does work fine with 4 GB of RAM.
>
> I don't see any reference of 1 GiB being enough in our current version
> of the manual.  If you do, please let me know.
>
> Closing for now.
>
> Thanks,
>
> Maxim

I think it's enough if you only use a single core.
If any guix operations goes out of memory, add --cores=1.
So: guix pull --cores=1

Message #19 received at 39813 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: bug-guix <at> gnu.org, Csepp <raingloom <at> riseup.net>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: dannym <at> scratchpost.org, 39813-done <at> debbugs.gnu.org, 39813 <at> debbugs.gnu.org
Subject: Re: bug#39813: Running Guix in a Virtual Machine - says 1 GB RAM is enough, but it isn't for guix pull
Date: Wed, 13 Jul 2022 19:21:45 +0200

[Message part 1 (text/plain, inline)]

I've heard that theory before. From observation on my late armhf server (two cores):

- it takes just below 2GB to build one of the derivations
- It doesn't swap a single byte
- whether with two or a single core, it takes roughly the same amount of memory
- substitution is nice, it doesn't require lots of memory (and then --cores is useless)

I think it's because we load all the files in a batch before we build them. The biggest amount of memory required is not for running the compiler on a thread, but for loading files and keeping them in memory for the whole duration of the build. With more threads, we still don't load each file more than once (twice to build it), so there's no reason it should take more memory.

Or maybe the process of loading and building is inherently single-threaded? I don't think so, but maybe?

Or it doesn't honor --cores.

Le 13 juillet 2022 18:58:58 GMT+02:00, Csepp <raingloom <at> riseup.net> a écrit :
>
>Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>> Hi Danny,
>>
>> Danny Milosavljevic <dannym <at> scratchpost.org> writes:
>>
>>> Hi,
>>>
>>> I just got a report that with Guix in a virtual Machine (like described in the
>>> manual in 8.16), guix pull does not actually work[1] with 1 GB of RAM.
>>> It does work fine with 4 GB of RAM.
>>
>> I don't see any reference of 1 GiB being enough in our current version
>> of the manual.  If you do, please let me know.
>>
>> Closing for now.
>>
>> Thanks,
>>
>> Maxim
>
>I think it's enough if you only use a single core.
>If any guix operations goes out of memory, add --cores=1.
>So: guix pull --cores=1
>
>
>

[Message part 2 (text/html, inline)]

Message #28 received at 39813 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: Julien Lepiller <julien <at> lepiller.eu>, Csepp <raingloom <at> riseup.net>, 
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: dannym <at> scratchpost.org, 39813 <at> debbugs.gnu.org, 39813-done <at> debbugs.gnu.org
Subject: Re: bug#39813: Running Guix in a Virtual Machine - says 1 GB RAM is
 enough, but it isn't for guix pull
Date: Wed, 13 Jul 2022 22:00:33 +0200

Am Mittwoch, dem 13.07.2022 um 19:21 +0200 schrieb Julien Lepiller:
> I've heard that theory before. From observation on my late armhf
> server
> (two cores):
> 
> - it takes just below 2GB to build one of the derivations
> - It doesn't swap a single byte
> - whether with two or a single core, it takes roughly the same amount
> of memory
> - substitution is nice, it doesn't require lots of memory (and then -
> -
> cores is useless)
> 
> I think it's because we load all the files in a batch before we build
> them. The biggest amount of memory required is not for running the
> compiler on a thread, but for loading files and keeping them in
> memory for the whole duration of the build. With more threads, we
> still don't load each file more than once (twice to build it), so
> there's no reason it should take more memory.
> 
> Or maybe the process of loading and building is inherently single-
> threaded? I don't think so, but maybe?
Loading and building is implemented in build-aux/compile-all.scm, which
does use multiple parallel workers.  However, since all compilation
appears to be done in the same Guile process, I don't think multi-
threading makes too big of an impact (it'll be the same garbage
collector regardless of ordering).

Cheers

Message #34 received at 39813 <at> debbugs.gnu.org (full text, mbox):

From: Csepp <raingloom <at> riseup.net>
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: Julien Lepiller <julien <at> lepiller.eu>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>, dannym <at> scratchpost.org,
 39813 <at> debbugs.gnu.org, 39813-done <at> debbugs.gnu.org,
 Csepp <raingloom <at> riseup.net>
Subject: Re: bug#39813: Running Guix in a Virtual Machine - says 1 GB RAM is
 enough, but it isn't for guix pull
Date: Thu, 14 Jul 2022 14:50:34 +0200

Liliana Marie Prikler <liliana.prikler <at> gmail.com> writes:

> Am Mittwoch, dem 13.07.2022 um 19:21 +0200 schrieb Julien Lepiller:
>> I've heard that theory before. From observation on my late armhf
>> server
>> (two cores):
>> 
>> - it takes just below 2GB to build one of the derivations
>> - It doesn't swap a single byte
>> - whether with two or a single core, it takes roughly the same amount
>> of memory
>> - substitution is nice, it doesn't require lots of memory (and then -
>> -
>> cores is useless)
>> 
>> I think it's because we load all the files in a batch before we build
>> them. The biggest amount of memory required is not for running the
>> compiler on a thread, but for loading files and keeping them in
>> memory for the whole duration of the build. With more threads, we
>> still don't load each file more than once (twice to build it), so
>> there's no reason it should take more memory.
>> 
>> Or maybe the process of loading and building is inherently single-
>> threaded? I don't think so, but maybe?
> Loading and building is implemented in build-aux/compile-all.scm, which
> does use multiple parallel workers.  However, since all compilation
> appears to be done in the same Guile process, I don't think multi-
> threading makes too big of an impact (it'll be the same garbage
> collector regardless of ordering).
>
> Cheers

Hmm, for some reason it finishes much faster on my i686 netbook with
--cores=1. I'll try to look into it further. I enable swap on it but it
doesn't use a lot of it. Maybe using an HDD matters too, since swapping
is much more expensive on it. Disabling parallelism tends to help because
it can halve the worst case memory usage. If foo.c and bar.c both
require at most 1M during build, building them in parallel is 2M in the
worst case, but only 1M when serialized. Oh and this is with
channel-with-substitutes-available. So either that's broken or something
still needs building.
Anyways, I should take a looksie at build-aux/compile-all.scm, maybe I
can decrease the memory usage.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.