GNU bug report logs - #33643
[PATCH] gnu-build-system: Enable xz to decompress in parallel.

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Christopher Baines <mail <at> cbaines.net>
Cc: tracker <at> debbugs.gnu.org
Subject: bug#33643: closed ([PATCH] gnu-build-system: Enable xz to
 decompress in parallel.)
Date: Thu, 14 May 2020 07:39:01 +0000

[Message part 1 (text/plain, inline)]

Your message dated Thu, 14 May 2020 08:37:56 +0100
with message-id <87ftc3nezv.fsf <at> cbaines.net>
and subject line Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel.
has caused the debbugs.gnu.org bug report #33643,
regarding [PATCH] gnu-build-system: Enable xz to decompress in parallel.
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
33643: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=33643
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Christopher Baines <mail <at> cbaines.net>
To: guix-patches <at> gnu.org
Subject: [PATCH] gnu-build-system: Enable xz to decompress in parallel.
Date: Thu,  6 Dec 2018 07:56:15 +0000

It can take a little while to decompress some packages with large xz
compressed source tar files. xz includes support for parallelism, so enable
this using the parallel job count for the overall derivation.

* guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T option
to xz to enable it to work in parallel if appropriate.
---
 guix/build/gnu-build-system.scm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm
index e5f3197b0..9d11e5b1e 100644
--- a/guix/build/gnu-build-system.scm
+++ b/guix/build/gnu-build-system.scm
@@ -147,7 +147,7 @@ chance to be set."
               locale (strerror (system-error-errno args)))
       #t)))
 
-(define* (unpack #:key source #:allow-other-keys)
+(define* (unpack #:key source parallel-build? #:allow-other-keys)
   "Unpack SOURCE in the working directory, and change directory within the
 source.  When SOURCE is a directory, copy it in a sub-directory of the current
 working directory."
@@ -161,6 +161,10 @@ working directory."
         (copy-recursively source "."
                           #:keep-mtime? #t))
       (begin
+        (when parallel-build?
+          (setenv "XZ_OPT"
+                  (format #f "-T~d" (parallel-job-count))))
+
         (if (string-suffix? ".zip" source)
             (invoke "unzip" source)
             (invoke "tar" "xvf" source))
-- 
2.19.2

[Message part 3 (message/rfc822, inline)]

From: Christopher Baines <mail <at> cbaines.net>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 33643-done <at> debbugs.gnu.org
Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in
 parallel.
Date: Thu, 14 May 2020 08:37:56 +0100

[Message part 4 (text/plain, inline)]

Efraim Flashner <efraim <at> flashner.co.il> writes:

> On Wed, May 13, 2020 at 07:20:08PM +0100, Christopher Baines wrote:
>>
>> Christopher Baines <mail <at> cbaines.net> writes:
>>
>> > It can take a little while to decompress some packages with large xz
>> > compressed source tar files. xz includes support for parallelism, so enable
>> > this using the parallel job count for the overall derivation.
>> >
>> > * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T option
>> > to xz to enable it to work in parallel if appropriate.
>> > ---
>> >  guix/build/gnu-build-system.scm | 6 +++++-
>> >  1 file changed, 5 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm
>> > index e5f3197b0..9d11e5b1e 100644
>> > --- a/guix/build/gnu-build-system.scm
>> > +++ b/guix/build/gnu-build-system.scm
>> > @@ -147,7 +147,7 @@ chance to be set."
>> >                locale (strerror (system-error-errno args)))
>> >        #t)))
>> >
>> > -(define* (unpack #:key source #:allow-other-keys)
>> > +(define* (unpack #:key source parallel-build? #:allow-other-keys)
>> >    "Unpack SOURCE in the working directory, and change directory within the
>> >  source.  When SOURCE is a directory, copy it in a sub-directory of the current
>> >  working directory."
>> > @@ -161,6 +161,10 @@ working directory."
>> >          (copy-recursively source "."
>> >                            #:keep-mtime? #t))
>> >        (begin
>> > +        (when parallel-build?
>> > +          (setenv "XZ_OPT"
>> > +                  (format #f "-T~d" (parallel-job-count))))
>> > +
>> >          (if (string-suffix? ".zip" source)
>> >              (invoke "unzip" source)
>> >              (invoke "tar" "xvf" source))
>>
>> It's been a long long while, but now that core-updates has recently been
>> merged, I'd like to try and take a look at this again.
>>
>> I think the consensus was that this will only help for xz compressed
>> files where they have been compressed in parallel. I think it's still
>> worth doing though, as some of the big xz files that need decompressing
>> have been compressed in parallel, and this will speed up the builds when
>> multiple cores are available.
>>
>> Thanks,
>>
>> Chris
>
> I thought the last time we looked into this we figured out that there
> was a mistake in release notes or something and that parallel
> decompression isn't actually supported.

Hmm, I had a look to see if I could find some examples of where this
would apply, but I couldn't find any xz archives that we use in Guix
where it's been compressed in a way that allows multithreaded
decompression...

I'm pretty sure I had some examples before, but maybe somethings changed
in the intervening year.

Anyway, if I discover this again, I'll actually make a note of where
it's applicable.

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.