GNU bug report logs - #74142
[rust-team] rust-bootstrap fails parallel build on x86-64

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 74142 in the body.
You can then email your comments to 74142 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Greg Hogan <code <at> greghogan.com>
To: bug-guix <at> gnu.org
Cc: Efraim Flashner <efraim <at> flashner.co.il>
Subject: [rust-team] rust-bootstrap fails parallel build on x86-64
Date: Thu, 31 Oct 2024 15:13:40 -0400

[Message part 1 (text/plain, inline)]

derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718

Build log attached, tail portion below. Has consistently failed to
build unless cores=1.

The package definition has "#:parallel-build? ,(target-x86-64?)" but
the noted build issue ("race conditions between various dependent
crates") is also seen on x86-64.

It is only the 1.54 build with mrustc that fails in this manner, the
following rust packages in the chain build fine with parallelism
enabled.

Greg

--8<---------------cut here---------------start------------->8---
(36/175) BUILDING autocfg v1.0.1
> /tmp/guix-build-rust-1.54.0.drv-0/mrustc/bin/mrustc rustc-1.54.0-src/vendor/autocfg/src/lib.rs -o output/rustc-build/libautocfg-1_0_1.rlib --crate-name autocfg --crate-type rlib -C emit-depfile=output/rustc-build/libautocfg-1_0_1.rlib.d --crate-tag 1_0_1 --cfg debug_assertions -O -L output -L output/rustc-build
> /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_bitflags-1_2_1_run
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/bitflags/output/rustc-build/build_memchr-2_4_0_H6_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/bitflags/output/rustc-build/build_memchr-2_4_0_H6_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/bitflags/output/rustc-build/build_memchr-2_4_0_H6_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_memchr-2_4_0_H6.txt_failed.txt
for stdout)
(37/175) BUILDING bitflags v1.2.1
> /tmp/guix-build-rust-1.54.0.drv-0/mrustc/bin/mrustc rustc-1.54.0-src/vendor/bitflags/src/lib.rs -o output/rustc-build/libbitflags-1_2_1.rlib --crate-name bitflags --crate-type rlib -C emit-depfile=output/rustc-build/libbitflags-1_2_1.rlib.d --crate-tag 1_2_1 --cfg debug_assertions -O -L output -L output/rustc-build --cfg bitflags_const_fn
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/mrustc/bin/mrustc' - No such file
or directory
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_ryu-1_0_5_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_ryu-1_0_5_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_ryu-1_0_5_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_ryu-1_0_5.txt_failed.txt
for stdout)
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_snap-1_0_5_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_snap-1_0_5_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_snap-1_0_5_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_snap-1_0_5.txt_failed.txt
for stdout)
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_maybe-uninit-2_0_0_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_maybe-uninit-2_0_0_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_maybe-uninit-2_0_0_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_maybe-uninit-2_0_0.txt_failed.txt
for stdout)
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_typenum-1_12_0_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_typenum-1_12_0_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_typenum-1_12_0_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_typenum-1_12_0.txt_failed.txt
for stdout)
BUILD FAILED
make: *** [minicargo.mk:228: output/rustc] Error 1
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "make" arguments: ("-f"
"minicargo.mk" "output/rustc" "RUSTC_TARGET=x86_64-unknown-linux-gnu"
"RUSTC_VERSION=1.54.0" "MRUSTC_TARGET_VER=1.54" "CFLAGS=-std=gnu11"
"OUTDIR_SUF=") exit-status: 2 term-signal: #f stop-signal: #f>
phase `build' failed after 617.7 seconds
command "make" "-f" "minicargo.mk" "output/rustc"
"RUSTC_TARGET=x86_64-unknown-linux-gnu" "RUSTC_VERSION=1.54.0"
"MRUSTC_TARGET_VER=1.54" "CFLAGS=-std=gnu11" "OUTDIR_SUF=" failed with
status 2
build process 18 exited with status 256
--8<---------------cut here---------------end--------------->8---

[qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv.gz (application/gzip, attachment)]

Message #8 received at 74142 <at> debbugs.gnu.org (full text, mbox):

From: Greg Hogan <code <at> greghogan.com>
To: 74142 <at> debbugs.gnu.org
Cc: Greg Hogan <code <at> greghogan.com>
Subject: [PATCH rust-team] gnu: rust-bootstrap: Disable parallel builds.
Date: Thu, 31 Oct 2024 20:26:23 +0000

* gnu/packages/rust.scm (rust-bootstrap): Disable parallel builds for
all architectures.

Change-Id: Icafbbacd30200fad457b6a70730acd466524485b
---
 gnu/packages/rust.scm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index cb58a4b4b3..4f74fc7571 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -208,9 +208,9 @@ (define-public rust-bootstrap
        ;; that doesn't seem to cause issues.
        #:validate-runpath? #f
        ;; Most of the build is single-threaded. This also improves the
-       ;; build time on machines with "only" 8GB of RAM.
-       ;; ppc64le regularly sees race conditions between various dependant crates.
-       #:parallel-build? ,(target-x86-64?)
+       ;; build time on machines with "only" 8GB of RAM and prevents
+       ;; regularly seen race conditions between various dependant crates.
+       #:parallel-build? #f
        #:make-flags
        (list ,(string-append "RUSTC_TARGET="
                              (platform-rust-target

base-commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
-- 
2.46.0

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Greg Hogan <code <at> greghogan.com>
Cc: bug-guix <at> gnu.org
Subject: Re: [rust-team] rust-bootstrap fails parallel build on x86-64
Date: Sun, 3 Nov 2024 09:49:46 +0200

[Message part 1 (text/plain, inline)]

On Thu, Oct 31, 2024 at 03:13:40PM -0400, Greg Hogan wrote:
> derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
> commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
> 
> Build log attached, tail portion below. Has consistently failed to
> build unless cores=1.
> 
> The package definition has "#:parallel-build? ,(target-x86-64?)" but
> the noted build issue ("race conditions between various dependent
> crates") is also seen on x86-64.
> 
> It is only the 1.54 build with mrustc that fails in this manner, the
> following rust packages in the chain build fine with parallelism
> enabled.
> 
> Greg

I see the note there says something about ppc64le having a race
condition too. My machine has "only" 24 cores, so I didn't run into
anything, but I see you're building with 32 cores. IIRC the ppc64le
machine was building with at least 32 cores also.

Can you try building rust-1.54 with only 24 cores and see if that works
for you? If it does then I think we can switch the (job-count) variable
to (min 24 (parallel-job-count))

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Greg Hogan <code <at> greghogan.com>
To: Efraim Flashner <efraim <at> flashner.co.il>, Greg Hogan <code <at> greghogan.com>,
 bug-guix <at> gnu.org
Subject: Re: [rust-team] rust-bootstrap fails parallel build on x86-64
Date: Sun, 3 Nov 2024 18:06:36 -0500

[Message part 1 (text/plain, inline)]

On Sun, Nov 3, 2024 at 2:49 AM Efraim Flashner <efraim <at> flashner.co.il> wrote:
>
> On Thu, Oct 31, 2024 at 03:13:40PM -0400, Greg Hogan wrote:
> > derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
> > commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
> >
> > Build log attached, tail portion below. Has consistently failed to
> > build unless cores=1.
> >
> > The package definition has "#:parallel-build? ,(target-x86-64?)" but
> > the noted build issue ("race conditions between various dependent
> > crates") is also seen on x86-64.
> >
> > It is only the 1.54 build with mrustc that fails in this manner, the
> > following rust packages in the chain build fine with parallelism
> > enabled.
> >
> > Greg
>
> I see the note there says something about ppc64le having a race
> condition too. My machine has "only" 24 cores, so I didn't run into
> anything, but I see you're building with 32 cores. IIRC the ppc64le
> machine was building with at least 32 cores also.
>
> Can you try building rust-1.54 with only 24 cores and see if that works
> for you? If it does then I think we can switch the (job-count) variable
> to (min 24 (parallel-job-count))

Log attached from what looks to be a similar failure on an 8 core instance.

[qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv.gz (application/gzip, attachment)]

Message #21 received at 74142 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Greg Hogan <code <at> greghogan.com>
Cc: 74142 <at> debbugs.gnu.org, efraim <at> flashner.co.il
Subject: Re: bug#74142: [rust-team] rust-bootstrap fails parallel build on
 x86-64
Date: Sat, 07 Dec 2024 22:45:32 +0900

Hi,

Greg Hogan <code <at> greghogan.com> writes:

> On Sun, Nov 3, 2024 at 2:49 AM Efraim Flashner <efraim <at> flashner.co.il> wrote:
>>
>> On Thu, Oct 31, 2024 at 03:13:40PM -0400, Greg Hogan wrote:
>> > derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
>> > commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
>> >
>> > Build log attached, tail portion below. Has consistently failed to
>> > build unless cores=1.
>> >
>> > The package definition has "#:parallel-build? ,(target-x86-64?)" but
>> > the noted build issue ("race conditions between various dependent
>> > crates") is also seen on x86-64.
>> >
>> > It is only the 1.54 build with mrustc that fails in this manner, the
>> > following rust packages in the chain build fine with parallelism
>> > enabled.
>> >
>> > Greg
>>
>> I see the note there says something about ppc64le having a race
>> condition too. My machine has "only" 24 cores, so I didn't run into
>> anything, but I see you're building with 32 cores. IIRC the ppc64le
>> machine was building with at least 32 cores also.
>>
>> Can you try building rust-1.54 with only 24 cores and see if that works
>> for you? If it does then I think we can switch the (job-count) variable
>> to (min 24 (parallel-job-count))
>
> Log attached from what looks to be a similar failure on an 8 core instance.

I encountered the same problem on a 32 "cores" (aka threads) machine.
I've reported it here [0].

[0]  https://github.com/thepowersgang/mrustc/issues/343

-- 
Thanks,
Maxim

Message #24 received at 74142 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 74142 <at> debbugs.gnu.org, Greg Hogan <code <at> greghogan.com>
Subject: Re: bug#74142: [rust-team] rust-bootstrap fails parallel build on
 x86-64
Date: Fri, 13 Dec 2024 13:53:01 +0200

[Message part 1 (text/plain, inline)]

On Sat, Dec 07, 2024 at 10:45:32PM +0900, Maxim Cournoyer wrote:
> Hi,
> 
> Greg Hogan <code <at> greghogan.com> writes:
> 
> > On Sun, Nov 3, 2024 at 2:49 AM Efraim Flashner <efraim <at> flashner.co.il> wrote:
> >>
> >> On Thu, Oct 31, 2024 at 03:13:40PM -0400, Greg Hogan wrote:
> >> > derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
> >> > commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
> >> >
> >> > Build log attached, tail portion below. Has consistently failed to
> >> > build unless cores=1.
> >> >
> >> > The package definition has "#:parallel-build? ,(target-x86-64?)" but
> >> > the noted build issue ("race conditions between various dependent
> >> > crates") is also seen on x86-64.
> >> >
> >> > It is only the 1.54 build with mrustc that fails in this manner, the
> >> > following rust packages in the chain build fine with parallelism
> >> > enabled.
> >> >
> >> > Greg
> >>
> >> I see the note there says something about ppc64le having a race
> >> condition too. My machine has "only" 24 cores, so I didn't run into
> >> anything, but I see you're building with 32 cores. IIRC the ppc64le
> >> machine was building with at least 32 cores also.
> >>
> >> Can you try building rust-1.54 with only 24 cores and see if that works
> >> for you? If it does then I think we can switch the (job-count) variable
> >> to (min 24 (parallel-job-count))
> >
> > Log attached from what looks to be a similar failure on an 8 core instance.
> 
> I encountered the same problem on a 32 "cores" (aka threads) machine.
> I've reported it here [0].
> 
> [0]  https://github.com/thepowersgang/mrustc/issues/343

I followed up with Mutabah (the main author) on IRC and a bit on the bug
tracker, and their suggestion is to use a later commit from mrustc.  I
used the latest commit from the master branch but there I ran into
issues on aarch64 not building correctly. From there they basically said
that they were focused on getting 1.74 bootstrapped, and they weren't
going to be circling back to the master branch until 1.74 was merged.

I went and tried out the rust_1.74 branch which is nearing completion
and that one built without any issues on x86_64 but failed for different
reasons on aarch64, powerpc64le and riscv64.

Someone from Gentoo was also asking around, they're interested in using
mrustc to bootstrap rust-1.74 and continuing on from there, so it's
likely we'll get some help going forward with architecture support.

I think in the meantime while it's not ready I'll change the
parallel-builds? flag to false for everyone, and then I'll continue
working on getting the bootstrap to work on the other architectures. The
good news is it only takes about 15 minutes to fail on aarch64 and
powerpc64le, so those should go pretty quickly through each iteration. I
might need to try some by-hand builds on riscv64; that one was closer to
2 hours.

Mutabah also said that their primary target is x86_64 and they don't
have the hardware to test other architectures, so its more hit-or-miss
there, and they rely on patches from others to get those ports working
as well as they do.  I don't love the answer, but I don't blame them.

I'll probably focus on aarch64 first since I can offload that easily and
I don't need to use a remote GUIX_DAEMON_SOCKET to get it to build, but
unfortunately it looks like we're sticking with 1.54 for a while.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #27 received at 74142 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 74142 <at> debbugs.gnu.org, Greg Hogan <code <at> greghogan.com>
Subject: Re: bug#74142: [rust-team] rust-bootstrap fails parallel build on
 x86-64
Date: Sun, 15 Dec 2024 00:38:52 +0900

Hi Efraim,

Efraim Flashner <efraim <at> flashner.co.il> writes:

[...]

>> [0]  https://github.com/thepowersgang/mrustc/issues/343
>
> I followed up with Mutabah (the main author) on IRC and a bit on the bug
> tracker, and their suggestion is to use a later commit from mrustc.  I
> used the latest commit from the master branch but there I ran into
> issues on aarch64 not building correctly. From there they basically said
> that they were focused on getting 1.74 bootstrapped, and they weren't
> going to be circling back to the master branch until 1.74 was merged.
>
> I went and tried out the rust_1.74 branch which is nearing completion
> and that one built without any issues on x86_64 but failed for different
> reasons on aarch64, powerpc64le and riscv64.
>
> Someone from Gentoo was also asking around, they're interested in using
> mrustc to bootstrap rust-1.74 and continuing on from there, so it's
> likely we'll get some help going forward with architecture support.
>
> I think in the meantime while it's not ready I'll change the
> parallel-builds? flag to false for everyone, and then I'll continue
> working on getting the bootstrap to work on the other architectures.

Sounds good!  Thanks for putting some efforts into shortening the
bootstrap... it currently takes more 3 hours to achieve on a powerful
machine.

-- 
Thanks,
Maxim

Message #32 received at 74142-done <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 74142-done <at> debbugs.gnu.org, Greg Hogan <code <at> greghogan.com>
Subject: Re: bug#74142: [rust-team] rust-bootstrap fails parallel build on
 x86-64
Date: Wed, 25 Dec 2024 16:43:02 +0200

[Message part 1 (text/plain, inline)]

I don't remember if I closed this but I've updated the mrustc snapshot
and now parallel builds are supported by default, on all architectures.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #40 received at 74142-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 74142-done <at> debbugs.gnu.org, Greg Hogan <code <at> greghogan.com>
Subject: Re: bug#74142: [rust-team] rust-bootstrap fails parallel build on
 x86-64
Date: Sat, 28 Dec 2024 14:27:34 +0900

Hi Efraim,

Efraim Flashner <efraim <at> flashner.co.il> writes:

> I don't remember if I closed this but I've updated the mrustc snapshot
> and now parallel builds are supported by default, on all architectures.

Excellent!

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.