GNU bug report logs - #35126
[PATCH] gnu: rust: Enable parallel tests starting with 1.26.

Previous Next

Package: guix-patches;

Reported by: Danny Milosavljevic <dannym <at> scratchpost.org>

Date: Wed, 3 Apr 2019 17:02:02 UTC

Severity: normal

Tags: patch

Done: Danny Milosavljevic <dannym <at> scratchpost.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35126 in the body.
You can then email your comments to 35126 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Wed, 03 Apr 2019 17:02:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Danny Milosavljevic <dannym <at> scratchpost.org>:
New bug report received and forwarded. Copy sent to guix-patches <at> gnu.org. (Wed, 03 Apr 2019 17:02:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: guix-patches <at> gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH] gnu: rust: Enable parallel tests starting with 1.26.
Date: Wed,  3 Apr 2019 19:00:44 +0200
* gnu/packages/rust.scm (rust-1.26)[arguments]<#:phases>[check]: Enable
parallel tests.
---
 gnu/packages/rust.scm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index 63426a12fc..23b28cce23 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -724,6 +724,11 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
           `(modify-phases ,phases
              ;; binaryen was replaced with LLD project from LLVM
              (delete 'dont-build-native)
+             (replace 'check
+               (lambda* _
+                 ;; Enable parallel execution.
+                 (invoke "./x.py" (string-append "-j" (number->string (parallel-job-count))) "test" "-vv")
+                 (invoke "./x.py" (string-append "-j" (number->string (parallel-job-count))) "test" "src/tools/cargo")))
              (replace 'remove-unsupported-tests
                (lambda* _
                  ;; Our ld-wrapper cannot process non-UTF8 bytes in LIBRARY_PATH.




Information forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Thu, 04 Apr 2019 00:53:02 GMT) Full text and rfc822 format available.

Message #8 received at 35126 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 35126 <at> debbugs.gnu.org
Subject: Re: [bug#35126] [PATCH] gnu: rust: Enable parallel tests starting
 with 1.26.
Date: Wed, 3 Apr 2019 17:52:35 -0700
Functionally, it looks good to me (and should speed up all subsequent builds)!

Just curious, any reason why we’re doing this starting 1.26 and not earlier?

—Ivan



Information forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Thu, 04 Apr 2019 18:30:02 GMT) Full text and rfc822 format available.

Message #11 received at 35126 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ivan Petkov <ivanppetkov <at> gmail.com>
Cc: 35126 <at> debbugs.gnu.org
Subject: Re: [bug#35126] [PATCH] gnu: rust: Enable parallel tests starting
 with 1.26.
Date: Thu, 4 Apr 2019 20:29:38 +0200
[Message part 1 (text/plain, inline)]
Hi Ivan,

On Wed, 3 Apr 2019 17:52:35 -0700
Ivan Petkov <ivanppetkov <at> gmail.com> wrote:

> Functionally, it looks good to me (and should speed up all subsequent builds)!
> 
> Just curious, any reason why we’re doing this starting 1.26 and not earlier?

There have been problems with it in very old Rust builds which is why it was
disabled in the first place.

Icecat uses Rust 1.24 - I selfishly don't want to change that one because I
use it ;)

And I'm not sure when (in which version) they fixed the parallel tests (if
they fixed them, that is).

So I'm kinda starting in the middle...

(I'm now up to and excluding rust 1.30.1 with enabled parallel tests with -j4)
(rust 1.30.1 is still building for a few rounds)
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Thu, 04 Apr 2019 22:01:02 GMT) Full text and rfc822 format available.

Message #14 received at 35126 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 35126 <at> debbugs.gnu.org
Subject: Re: [bug#35126] [PATCH] gnu: rust: Enable parallel tests starting
 with 1.26.
Date: Thu, 4 Apr 2019 14:59:53 -0700
Hi Danny,

> On Apr 4, 2019, at 11:29 AM, Danny Milosavljevic <dannym <at> scratchpost.org> wrote:
> 
> There have been problems with it in very old Rust builds which is why it was
> disabled in the first place.
> 
> Icecat uses Rust 1.24 - I selfishly don't want to change that one because I
> use it ;)

Can’t say that I fault you there, I really hate having to rebootstrap the entire rust chain ;)

> And I'm not sure when (in which version) they fixed the parallel tests (if
> they fixed them, that is).
> 
> So I'm kinda starting in the middle...

We can always try moving it up to earlier versions as necessary, so I’m fine with landing
this as long as all the later tests build fine!

—Ivan



Information forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Fri, 05 Apr 2019 14:12:02 GMT) Full text and rfc822 format available.

Message #17 received at 35126 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ivan Petkov <ivanppetkov <at> gmail.com>
Cc: 35126 <at> debbugs.gnu.org
Subject: Re: [bug#35126] [PATCH] gnu: rust: Enable parallel tests starting
 with 1.26.
Date: Fri, 5 Apr 2019 16:11:07 +0200
[Message part 1 (text/plain, inline)]
All tests of all Rust >= 1.26 just successfully completed--with parallel tests.

Should we limit the number of parallel jobs?  I think at least armhf is going
to be very overloaded otherwise.  To which value?
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Fri, 05 Apr 2019 15:38:02 GMT) Full text and rfc822 format available.

Message #20 received at 35126 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 35126 <at> debbugs.gnu.org
Subject: Re: [bug#35126] [PATCH] gnu: rust: Enable parallel tests starting
 with 1.26.
Date: Fri, 5 Apr 2019 08:36:44 -0700
Hi Danny,

> On Apr 5, 2019, at 7:11 AM, Danny Milosavljevic <dannym <at> scratchpost.org> wrote:
> 
> Should we limit the number of parallel jobs?  I think at least armhf is going
> to be very overloaded otherwise.  To which value?

Do the armhf builders define that they have fewer cores available?

If not, we could cap the max jobs to 2, although it would be nice to potentially
use as many cores as are available for non-armhf builds.

—Ivan



Information forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Sat, 06 Apr 2019 10:49:02 GMT) Full text and rfc822 format available.

Message #23 received at 35126 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ivan Petkov <ivanppetkov <at> gmail.com>
Cc: 35126 <at> debbugs.gnu.org
Subject: Re: [bug#35126] [PATCH] gnu: rust: Enable parallel tests starting
 with 1.26.
Date: Sat, 6 Apr 2019 12:48:15 +0200
[Message part 1 (text/plain, inline)]
On Fri, 5 Apr 2019 08:36:44 -0700
Ivan Petkov <ivanppetkov <at> gmail.com> wrote:

> Hi Danny,
> 
> > On Apr 5, 2019, at 7:11 AM, Danny Milosavljevic <dannym <at> scratchpost.org> wrote:
> > 
> > Should we limit the number of parallel jobs?  I think at least armhf is going
> > to be very overloaded otherwise.  To which value?  
> 
> Do the armhf builders define that they have fewer cores available?
> 
> If not, we could cap the max jobs to 2, although it would be nice to potentially
> use as many cores as are available for non-armhf builds.

I think the problem is RAM usage.  Arm machines have little RAM and will choke
if too many memory-expensive things run in parallel.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Sat, 06 Apr 2019 16:10:02 GMT) Full text and rfc822 format available.

Message #26 received at 35126 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 35126 <at> debbugs.gnu.org
Subject: Re: [bug#35126] [PATCH] gnu: rust: Enable parallel tests starting
 with 1.26.
Date: Sat, 6 Apr 2019 09:09:13 -0700

> On Apr 6, 2019, at 3:48 AM, Danny Milosavljevic <dannym <at> scratchpost.org> wrote:
> 
> On Fri, 5 Apr 2019 08:36:44 -0700
> Ivan Petkov <ivanppetkov <at> gmail.com> wrote:
> 
>> Hi Danny,
>> 
>>> On Apr 5, 2019, at 7:11 AM, Danny Milosavljevic <dannym <at> scratchpost.org> wrote:
>>> 
>>> Should we limit the number of parallel jobs?  I think at least armhf is going
>>> to be very overloaded otherwise.  To which value?  
>> 
>> Do the armhf builders define that they have fewer cores available?
>> 
>> If not, we could cap the max jobs to 2, although it would be nice to potentially
>> use as many cores as are available for non-armhf builds.
> 
> I think the problem is RAM usage.  Arm machines have little RAM and will choke
> if too many memory-expensive things run in parallel.

I’d anticipate that building rustc itself would be the most memory intense operation,
I think most of the individual tests aren’t anywhere nearly as big…

—Ivan



Information forwarded to guix-patches <at> gnu.org:
bug#35126; Package guix-patches. (Mon, 08 Apr 2019 01:44:02 GMT) Full text and rfc822 format available.

Message #29 received at 35126 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 35126 <at> debbugs.gnu.org
Subject: Re: [bug#35126] [PATCH] gnu: rust: Enable parallel tests starting
 with 1.26.
Date: Sun, 7 Apr 2019 18:43:32 -0700
Hi Danny,

When it’s time to merge this in, would you mind also changing the
`generate-checksums` procedure in (guix build cargo-utils) to omit
the `src-name` parameter (as mentioned in #35155)?

If we’re going to be having the CI bootstrap almost the whole rustc
chain for this change, we might as well combine them together and
build all at once!

Thanks,
—Ivan



bug closed, send any further explanations to 35126 <at> debbugs.gnu.org and Danny Milosavljevic <dannym <at> scratchpost.org> Request was from Danny Milosavljevic <dannym <at> scratchpost.org> to control <at> debbugs.gnu.org. (Thu, 11 Apr 2019 22:28:02 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Fri, 10 May 2019 11:24:06 GMT) Full text and rfc822 format available.

This bug report was last modified 6 years and 63 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.