GNU bug report logs - #69314
[PATCH] Speed up stage0 bootstrap build using prebuilts

Previous Next

To reply to this bug, email your comments to 69314 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
To: guile-devel <at> gnu.org, bug-guile <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: [PATCH] Speed up stage0 bootstrap build using prebuilts
Date: Thu, 22 Feb 2024 21:09:00 +0100

[Message part 1 (text/plain, inline)]

Dear Guile maintainers,

please consider below patch to speed up the build from release tarballs
that include prebuilt bytecode.

Jonas

On Thu, 2024-01-04 at 11:57 +0100, Jonas Hahnfeld wrote:
> From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17 00:00:00 2001
> From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
> Date: Thu, 4 Jan 2024 11:44:55 +0100
> Subject: [PATCH] Speed up stage0 bootstrap build using prebuilts
> 
> Use prebuilt bytecode of ice-9/eval.go and others for all of stage0,
> it is optimized and evaluation is much faster. In my environment,
> this speeds up the build of guile-3.0.9 from around 29 minutes to
> only 19 minutes.
> 
> * meta/build-env.in: In stage0, prefer prebuilt bytecode over just
> compiled stage0 files.
> ---
>  meta/build-env.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/build-env.in b/meta/build-env.in
> index bdc88ded4..446a536af 100644
> --- a/meta/build-env.in
> +++ b/meta/build-env.in
> @@ -58,7 +58,7 @@ then
>      fi
>      export GUILE_LOAD_PATH
>      case "$GUILE_BOOTSTRAP_STAGE" in
> -        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
> +        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;
>          stage1) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage1:${top_builddir}/stage0" ;;
>          stage2) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage1" ;;
>          *) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage2:${top_builddir}/guile-readline" ;;

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 69314 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
Cc: 69314 <at> debbugs.gnu.org, guile-devel <at> gnu.org
Subject: Re: bug#69314: [PATCH] Speed up stage0 bootstrap build using prebuilts
Date: Mon, 06 May 2024 11:34:33 +0200

Hi Jonas,

Jonas Hahnfeld <hahnjo <at> hahnjo.de> skribis:

> On Thu, 2024-01-04 at 11:57 +0100, Jonas Hahnfeld wrote:
>> From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17 00:00:00 2001
>> From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
>> Date: Thu, 4 Jan 2024 11:44:55 +0100
>> Subject: [PATCH] Speed up stage0 bootstrap build using prebuilts
>> 
>> Use prebuilt bytecode of ice-9/eval.go and others for all of stage0,
>> it is optimized and evaluation is much faster. In my environment,
>> this speeds up the build of guile-3.0.9 from around 29 minutes to
>> only 19 minutes.
>> 
>> * meta/build-env.in: In stage0, prefer prebuilt bytecode over just
>> compiled stage0 files.
>> ---
>>  meta/build-env.in | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/meta/build-env.in b/meta/build-env.in
>> index bdc88ded4..446a536af 100644
>> --- a/meta/build-env.in
>> +++ b/meta/build-env.in
>> @@ -58,7 +58,7 @@ then
>>      fi
>>      export GUILE_LOAD_PATH
>>      case "$GUILE_BOOTSTRAP_STAGE" in
>> -        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
>> +        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;

I don’t understand why changing the order would make a difference.
Surely if .go files are available under prebuilt/, they’ll be found,
even if that directory comes second?  Or am I missing something?

Thanks for the patch!

Ludo’.

Message #11 received at 69314 <at> debbugs.gnu.org (full text, mbox):

From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 69314 <at> debbugs.gnu.org, guile-devel <at> gnu.org
Subject: Re: bug#69314: [PATCH] Speed up stage0 bootstrap build using prebuilts
Date: Mon, 06 May 2024 21:12:44 +0200

[Message part 1 (text/plain, inline)]

On Mon, 2024-05-06 at 11:34 +0200, Ludovic Courtès wrote:
> Hi Jonas,
> 
> Jonas Hahnfeld <hahnjo <at> hahnjo.de> skribis:
> 
> > On Thu, 2024-01-04 at 11:57 +0100, Jonas Hahnfeld wrote:
> > > From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17 00:00:00 2001
> > > From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
> > > Date: Thu, 4 Jan 2024 11:44:55 +0100
> > > Subject: [PATCH] Speed up stage0 bootstrap build using prebuilts
> > > 
> > > Use prebuilt bytecode of ice-9/eval.go and others for all of stage0,
> > > it is optimized and evaluation is much faster. In my environment,
> > > this speeds up the build of guile-3.0.9 from around 29 minutes to
> > > only 19 minutes.
> > > 
> > > * meta/build-env.in: In stage0, prefer prebuilt bytecode over just
> > > compiled stage0 files.
> > > ---
> > >  meta/build-env.in | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/meta/build-env.in b/meta/build-env.in
> > > index bdc88ded4..446a536af 100644
> > > --- a/meta/build-env.in
> > > +++ b/meta/build-env.in
> > > @@ -58,7 +58,7 @@ then
> > >      fi
> > >      export GUILE_LOAD_PATH
> > >      case "$GUILE_BOOTSTRAP_STAGE" in
> > > -        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
> > > +        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;
> 
> I don’t understand why changing the order would make a difference.
> Surely if .go files are available under prebuilt/, they’ll be found,
> even if that directory comes second?  Or am I missing something?

Yes, they are found but with lower priority. As I try to explain in the
commit message (could be improved though), the same bytecode is also
built during stage0, but without optimizations which makes the rest of
stage0 awfully slow.

Jonas

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 69314 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>, Jonas Hahnfeld
 <hahnjo <at> hahnjo.de>
Cc: 69314 <at> debbugs.gnu.org, guile-devel <at> gnu.org
Subject: Re: bug#69314: [PATCH] Speed up stage0 bootstrap build using prebuilts
Date: Tue, 07 May 2024 18:21:01 +0200

Hi Ludo,

On lun., 06 mai 2024 at 11:34, Ludovic Courtès <ludo <at> gnu.org> wrote:

>>> -        stage0) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@" ;;
>>> +        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;
>
> I don’t understand why changing the order would make a difference.
> Surely if .go files are available under prebuilt/, they’ll be found,
> even if that directory comes second?  Or am I missing something?

Well, depending on what had been built before, it might read:

--8<---------------cut here---------------start------------->8---
$ ls -1 {stage0,prebuilt/x86_64-unknown-linux-gnu}/ice-9/{boot-9,eval,psyntax-pp}.go
prebuilt/x86_64-unknown-linux-gnu/ice-9/boot-9.go
prebuilt/x86_64-unknown-linux-gnu/ice-9/eval.go
prebuilt/x86_64-unknown-linux-gnu/ice-9/psyntax-pp.go
stage0/ice-9/boot-9.go
stage0/ice-9/eval.go
stage0/ice-9/psyntax-pp.go
--8<---------------cut here---------------end--------------->8---

Therefore, the order in GUILE_LOAD_COMPILED_PATH matters, no?

Somehow, similarly as:

        stage1) GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage1:${top_builddir}/stage0" ;;

I would set the expected most optimized first, such that:

        stage0) GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_BINARIES@:${top_builddir}/stage0" ;;

assuming ’prebuilt’ provides something more optimized than ’stage0’.


Cheers,
simon

Message #17 received at 69314 <at> debbugs.gnu.org (full text, mbox):

From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 69314 <at> debbugs.gnu.org, guile-devel <at> gnu.org
Subject: Re: bug#69314: [PATCH] Speed up stage0 bootstrap build using prebuilts
Date: Sun, 14 Jul 2024 19:55:54 +0200

[Message part 1 (text/plain, inline)]

On Mon, 2024-05-06 at 21:12 +0200, Jonas Hahnfeld via Developers list
for Guile, the GNU extensibility library wrote:
> On Mon, 2024-05-06 at 11:34 +0200, Ludovic Courtès wrote:
> > Hi Jonas,
> > 
> > Jonas Hahnfeld <hahnjo <at> hahnjo.de> skribis:
> > 
> > > On Thu, 2024-01-04 at 11:57 +0100, Jonas Hahnfeld wrote:
> > > > From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17
> > > > 00:00:00 2001
> > > > From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
> > > > Date: Thu, 4 Jan 2024 11:44:55 +0100
> > > > Subject: [PATCH] Speed up stage0 bootstrap build using
> > > > prebuilts
> > > > 
> > > > Use prebuilt bytecode of ice-9/eval.go and others for all of
> > > > stage0,
> > > > it is optimized and evaluation is much faster. In my
> > > > environment,
> > > > this speeds up the build of guile-3.0.9 from around 29 minutes
> > > > to
> > > > only 19 minutes.
> > > > 
> > > > * meta/build-env.in: In stage0, prefer prebuilt bytecode over
> > > > just
> > > > compiled stage0 files.
> > > > ---
> > > >  meta/build-env.in | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/meta/build-env.in b/meta/build-env.in
> > > > index bdc88ded4..446a536af 100644
> > > > --- a/meta/build-env.in
> > > > +++ b/meta/build-env.in
> > > > @@ -58,7 +58,7 @@ then
> > > >      fi
> > > >      export GUILE_LOAD_PATH
> > > >      case "$GUILE_BOOTSTRAP_STAGE" in
> > > > -        stage0)
> > > > GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/
> > > > prebuilt/@SCM_PREBUILT_BINARIES@" ;;
> > > > +        stage0)
> > > > GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_
> > > > BINARIES@:${top_builddir}/stage0" ;;
> > 
> > I don’t understand why changing the order would make a difference.
> > Surely if .go files are available under prebuilt/, they’ll be
> > found,
> > even if that directory comes second?  Or am I missing something?
> 
> Yes, they are found but with lower priority. As I try to explain in
> the commit message (could be improved though), the same bytecode is
> also built during stage0, but without optimizations which makes the
> rest of stage0 awfully slow.

ping, does this explanation make sense?

Jonas

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 69314 <at> debbugs.gnu.org (full text, mbox):

From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 69314 <at> debbugs.gnu.org, guile-devel <at> gnu.org
Subject: Re: bug#69314: [PATCH] Speed up stage0 bootstrap build using prebuilts
Date: Tue, 28 Jan 2025 22:05:25 +0100

[Message part 1 (text/plain, inline)]

On Sun, 2024-07-14 at 19:55 +0200, Jonas Hahnfeld wrote:
> On Mon, 2024-05-06 at 21:12 +0200, Jonas Hahnfeld wrote:
> > On Mon, 2024-05-06 at 11:34 +0200, Ludovic Courtès wrote:
> > > Hi Jonas,
> > > 
> > > Jonas Hahnfeld <hahnjo <at> hahnjo.de> skribis:
> > > 
> > > > On Thu, 2024-01-04 at 11:57 +0100, Jonas Hahnfeld wrote:
> > > > > From 95f15821c535537c7ad4fdae1988855314d56ece Mon Sep 17
> > > > > 00:00:00 2001
> > > > > From: Jonas Hahnfeld <hahnjo <at> hahnjo.de>
> > > > > Date: Thu, 4 Jan 2024 11:44:55 +0100
> > > > > Subject: [PATCH] Speed up stage0 bootstrap build using
> > > > > prebuilts
> > > > > 
> > > > > Use prebuilt bytecode of ice-9/eval.go and others for all of
> > > > > stage0,
> > > > > it is optimized and evaluation is much faster. In my
> > > > > environment,
> > > > > this speeds up the build of guile-3.0.9 from around 29 minutes
> > > > > to
> > > > > only 19 minutes.
> > > > > 
> > > > > * meta/build-env.in: In stage0, prefer prebuilt bytecode over
> > > > > just
> > > > > compiled stage0 files.
> > > > > ---
> > > > >  meta/build-env.in | 2 +-
> > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/meta/build-env.in b/meta/build-env.in
> > > > > index bdc88ded4..446a536af 100644
> > > > > --- a/meta/build-env.in
> > > > > +++ b/meta/build-env.in
> > > > > @@ -58,7 +58,7 @@ then
> > > > >      fi
> > > > >      export GUILE_LOAD_PATH
> > > > >      case "$GUILE_BOOTSTRAP_STAGE" in
> > > > > -        stage0)
> > > > > GUILE_LOAD_COMPILED_PATH="${top_builddir}/stage0:${top_srcdir}/
> > > > > prebuilt/@SCM_PREBUILT_BINARIES@" ;;
> > > > > +        stage0)
> > > > > GUILE_LOAD_COMPILED_PATH="${top_srcdir}/prebuilt/@SCM_PREBUILT_
> > > > > BINARIES@:${top_builddir}/stage0" ;;
> > > 
> > > I don’t understand why changing the order would make a difference.
> > > Surely if .go files are available under prebuilt/, they’ll be
> > > found,
> > > even if that directory comes second?  Or am I missing something?
> > 
> > Yes, they are found but with lower priority. As I try to explain in
> > the commit message (could be improved though), the same bytecode is
> > also built during stage0, but without optimizations which makes the
> > rest of stage0 awfully slow.
> 
> ping, does this explanation make sense?

another ping...

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.