GNU bug report logs - #62069
Support xz parallel compression in dist-xz

Previous Next

To reply to this bug, email your comments to 62069 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Sam James <sam <at> gentoo.org>
To: bug-automake <at> gnu.org
Subject: Support xz parallel compression in dist-xz
Date: Thu, 09 Mar 2023 08:47:23 +0000

[Message part 1 (text/plain, inline)]

xz supports parallel compression which obviously speeds up
the time taken to run the 'dist-xz' rule, but it also
speeds up *decompression* too, as the parallel compressor
creates different output which can be decompressed in parallel.

There's two options, I think, for automake:
1. Pass -T+1 to xz if xz is >= 5.4.0 (a runtime check
would be needed, but it's cheap to do) as this uses
1 thread, but uses the parallel compressor to do it,
so allows decompression in parallel by users;

2. Pass -T0 (or -Tn, n >= 1) to allow autodetection
of the number of available CPU cores (or n if specified)
which also then naturally uses the threaded compressor.

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 62069 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: Sam James <sam <at> gentoo.org>
Cc: 62069 <at> debbugs.gnu.org
Subject: Re: bug#62069: Support xz parallel compression in dist-xz
Date: Sat, 13 Jan 2024 01:05:49 -0500

[Message part 1 (text/plain, inline)]

On 09 Mar 2023 08:47, Sam James wrote:
> xz supports parallel compression which obviously speeds up
> the time taken to run the 'dist-xz' rule, but it also
> speeds up *decompression* too, as the parallel compressor
> creates different output which can be decompressed in parallel.
> 
> There's two options, I think, for automake:
> 1. Pass -T+1 to xz if xz is >= 5.4.0 (a runtime check
> would be needed, but it's cheap to do) as this uses
> 1 thread, but uses the parallel compressor to do it,
> so allows decompression in parallel by users;
> 
> 2. Pass -T0 (or -Tn, n >= 1) to allow autodetection
> of the number of available CPU cores (or n if specified)
> which also then naturally uses the threaded compressor.

it should be a configure check when dist-xz is enabled in the call to
AM_INIT_AUTOMAKE, and we should default to `-T0`.  this would be the
default XZ_OPT value if it hasn't been set/overridden (where we use
-e now).
-mike

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.