GNU bug report logs - #32196
[PATCH core-updates] packages, scripts, utils: Disable threaded xz compression.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32196 in the body.
You can then email your comments to 32196 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: guix-patches <at> gnu.org
Subject: [PATCH core-updates] packages, scripts,
 utils: Disable threaded xz compression.
Date: Wed, 18 Jul 2018 14:10:28 +0200

Archives produces by "xz --threads" are not reproducible: they depend on the
number of cores used for compressing.  See <https://bugs.gnu.org/31015>.

* guix/packages.scm (patch-and-repack): Explicitly use 1 thread for compression.
* guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.
* guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
Likewise.
---
 guix/packages.scm     | 2 +-
 guix/scripts/pack.scm | 4 ++--
 guix/utils.scm        | 6 +++---
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index c762fa7c3..ef88e0c59 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -633,7 +633,7 @@ specifies modules in scope when evaluating SNIPPET."
                      ;; threaded compression (introduced in
                      ;; 5.2.0), but it ignores the extra flag.
                      (string-append "--use-compress-program="
-                                    #+xz "/bin/xz --threads=0")
+                                    #+xz "/bin/xz --threads=1")
                      ;; avoid non-determinism in the archive
                      "--mtime=@0"
                      "--owner=root:0"
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 7f087a3a3..e0d6f1dee 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -69,7 +69,7 @@
         (compressor "lzip"  ".lz"
                     #~(#+(file-append lzip "/bin/lzip") "-9"))
         (compressor "xz"    ".xz"
-                    #~(#+(file-append xz "/bin/xz") "-e -T0"))
+                    #~(#+(file-append xz "/bin/xz") "-e -T1"))
         (compressor "bzip2" ".bz2"
                     #~(#+(file-append bzip2 "/bin/bzip2") "-9"))
         (compressor "none" "" #f)))
@@ -77,7 +77,7 @@
 ;; This one is only for use in this module, so don't put it in %compressors.
 (define bootstrap-xz
   (compressor "bootstrap-xz" ".xz"
-              #~(#+(file-append %bootstrap-coreutils&co "/bin/xz") "-e -T0")))
+              #~(#+(file-append %bootstrap-coreutils&co "/bin/xz") "-e -T1")))
 
 (define (lookup-compressor name)
   "Return the compressor object called NAME.  Error out if it could not be
diff --git a/guix/utils.scm b/guix/utils.scm
index a5de9605e..84176a22e 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -175,7 +175,7 @@ a symbol such as 'xz."
   (match compression
     ((or #f 'none) (values input '()))
     ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
-    ('xz           (filtered-port `(,%xz "-dc" "-T0") input))
+    ('xz           (filtered-port `(,%xz "-dc" "-T1") input))
     ('gzip         (filtered-port `(,%gzip "-dc") input))
     (else          (error "unsupported compression scheme" compression))))
 
@@ -185,7 +185,7 @@ a symbol such as 'xz."
   (match compression
     ((or #f 'none) (values input '()))
     ('bzip2        (filtered-port `(,%bzip2 "-c") input))
-    ('xz           (filtered-port `(,%xz "-c" "-T0") input))
+    ('xz           (filtered-port `(,%xz "-c" "-T1") input))
     ('gzip         (filtered-port `(,%gzip "-c") input))
     (else          (error "unsupported compression scheme" compression))))
 
@@ -242,7 +242,7 @@ program--e.g., '(\"--fast\")."
   (match compression
     ((or #f 'none) (values output '()))
     ('bzip2        (filtered-output-port `(,%bzip2 "-c" ,@options) output))
-    ('xz           (filtered-output-port `(,%xz "-c" "-T0" ,@options) output))
+    ('xz           (filtered-output-port `(,%xz "-c" "-T1" ,@options) output))
     ('gzip         (filtered-output-port `(,%gzip "-c" ,@options) output))
     (else          (error "unsupported compression scheme" compression))))
 
-- 
2.18.0

Message #8 received at 32196 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 32196 <at> debbugs.gnu.org
Subject: Re: [bug#32196] [PATCH core-updates] packages, scripts,
 utils: Disable threaded xz compression.
Date: Thu, 19 Jul 2018 11:33:56 +0200

Hi Marius,

Marius Bakke <mbakke <at> fastmail.com> skribis:

> Archives produces by "xz --threads" are not reproducible: they depend on the
> number of cores used for compressing.  See <https://bugs.gnu.org/31015>.
>
> * guix/packages.scm (patch-and-repack): Explicitly use 1 thread for compression.
> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.
> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
> Likewise.

I hadn’t noticed -T0 had made it to ‘master’.

I’d suggest removing 63102406f22412bb922de5549deb89d3594a38c0 on master
(no rebuild needed), with a reference to #31015 in the commit log.  And
then similarly reverting c8a3dea847bb9f87fa1876d0c6c3356d6226f121 on
‘core-updates’.

Sounds good?

If we’re concerned about speed, perhaps we should switch to lzip, which
I think has better behavior.

Thanks,
Ludo’.

Message #13 received at 32196-done <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 32196-done <at> debbugs.gnu.org
Subject: Re: [bug#32196] [PATCH core-updates] packages, scripts,
 utils: Disable threaded xz compression.
Date: Sun, 22 Jul 2018 18:06:04 +0200

[Message part 1 (text/plain, inline)]

ludo <at> gnu.org (Ludovic Courtès) writes:

> Hi Marius,
>
> Marius Bakke <mbakke <at> fastmail.com> skribis:
>
>> Archives produces by "xz --threads" are not reproducible: they depend on the
>> number of cores used for compressing.  See <https://bugs.gnu.org/31015>.
>>
>> * guix/packages.scm (patch-and-repack): Explicitly use 1 thread for compression.
>> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.
>> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
>> Likewise.
>
> I hadn’t noticed -T0 had made it to ‘master’.
>
> I’d suggest removing 63102406f22412bb922de5549deb89d3594a38c0 on master
> (no rebuild needed), with a reference to #31015 in the commit log.  And
> then similarly reverting c8a3dea847bb9f87fa1876d0c6c3356d6226f121 on
> ‘core-updates’.
>
> Sounds good?

Done in commits e9be2c5409f37173d70b202aa06752e3814ccdc2 and
3e95125e9bd0676d4a9add9105217ad3eaef3ff0.

> If we’re concerned about speed, perhaps we should switch to lzip, which
> I think has better behavior.

That sounds great.  lzip has other benefits too, such as the ability to
recover from bit flips.  Recommended reading:
<https://www.nongnu.org/lzip/xz_inadequate.html>.

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.