From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 30 18:17:32 2017 Received: (at submit) by debbugs.gnu.org; 30 Mar 2017 22:17:32 +0000 Received: from localhost ([127.0.0.1]:53269 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ctiNn-0000Vu-Px for submit@debbugs.gnu.org; Thu, 30 Mar 2017 18:17:31 -0400 Received: from eggs.gnu.org ([208.118.235.92]:33360) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ctiNl-0000Vd-VO for submit@debbugs.gnu.org; Thu, 30 Mar 2017 18:17:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ctiNf-0002Gc-M7 for submit@debbugs.gnu.org; Thu, 30 Mar 2017 18:17:24 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05,T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:34699) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ctiNf-0002GY-Iu for submit@debbugs.gnu.org; Thu, 30 Mar 2017 18:17:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52547) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ctiNe-0003r5-KY for guix-patches@gnu.org; Thu, 30 Mar 2017 18:17:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ctiNb-0002FE-HA for guix-patches@gnu.org; Thu, 30 Mar 2017 18:17:22 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:49820) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ctiNb-0002EC-6C for guix-patches@gnu.org; Thu, 30 Mar 2017 18:17:19 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id AA2A820AA9; Thu, 30 Mar 2017 18:17:16 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute4.internal (MEProxy); Thu, 30 Mar 2017 18:17:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=content-type:date:from:message-id:mime-version:subject:to :x-me-sender:x-me-sender:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=kRa xl2fY9KAI0/4bk1eKliJZihfd36iPZfBeNWPScqk=; b=qt0wnGiFc2reJgeySBZ /BPRjJgLo9ZaZ1Re/2Y/hmO5tNVfTUYgjBEGmp6DMgBTzGXeH5t6OO61dgkPZw1K sevzTmQuRkjiuY9XjhsnP9QvSDIXcjiJXQSGrQTV4roK/QVA4lr3l52lgtpkk62o kawFDVukVBwNkonkVmuErdOc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-sender:x-me-sender:x-sasl-enc :x-sasl-enc; s=fm1; bh=kRaxl2fY9KAI0/4bk1eKliJZihfd36iPZfBeNWPSc qk=; b=eWyptygapP7WyCvefbCmKqciP9yJZByDZUXQaaOn8LcEY89ZyqpiGk2Fa 2CxccjD5V8yUoTM02jOc/ISFvYN8PWGRnbz00s6WBP7OJOtcyv1EM/NBCUeLQqZH Hk7bZl4LLxtPORSNKSH1NxcR0i0xxgVe0QPdKRaZ8Em23smd/SvKXrbRDvBXtAik wVP/sFYQ8E/RFY8JjWroPk+zsHjnWp/TKfQvJBGVoskwbkECYeeyqaenH4CG6JRP pnYmaDat1wlKj6UKPRpxBLhpYTy8eJVvob8PPpxpzUI16ys77j0tdybEFDuj6D8r YXIRvXvfkUpZZIlNsYyxVIIdjBH6g== X-ME-Sender: X-Sasl-enc: U8tBzYa3EuMa77GppEFOboP5dFIDnjVJ57/COAoAJw/z 1490912236 Received: from localhost (c-73-165-108-70.hsd1.pa.comcast.net [73.165.108.70]) by mail.messagingengine.com (Postfix) with ESMTPA id 708AC7E70C for ; Thu, 30 Mar 2017 18:17:16 -0400 (EDT) Date: Thu, 30 Mar 2017 18:17:15 -0400 From: Leo Famulari To: guix-patches@gnu.org Subject: Enable threaded compression of source tarballs Message-ID: <20170330221715.GA10357@jasmine> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="i0/AhcQY5QxfSsSZ" Content-Disposition: inline User-Agent: Mutt/1.8.0 (2017-02-23) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.1 (----) --i0/AhcQY5QxfSsSZ Content-Type: multipart/mixed; boundary="NzB8fVQJ5HfG6fxh" Content-Disposition: inline --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline This patch enables multi-threaded xz compression in the repacking stage of building source tarballs. With it applied, I get a ~2.2x speedup building the source of linux-libre with 4 threads, compared to 1 thread. Specifically, with the upstream linux-libre tarball in the page cache, `guix build -S linux-libre` took ~366 seconds using a single thread for xz, and ~164 seconds using 4 threads. --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-packages-Enable-threaded-compression-of-source-tarba.patch" Content-Transfer-Encoding: quoted-printable =46rom 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001 =46rom: Leo Famulari Date: Thu, 30 Mar 2017 03:10:48 -0400 Subject: [PATCH] packages: Enable threaded compression of source tarballs. This provides a ~2x speedup when using 4 threads. * guix/packages.scm (patch-and-repack)[build]: Invoke xz with '--threads=3D0' when re-packing tarballs. --- guix/packages.scm | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/guix/packages.scm b/guix/packages.scm index 4bc4b017f..30be5bf38 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -573,7 +573,12 @@ specifies modules in scope when evaluating SNIPPET." #:fail-on-error? #t)))= )) (zero? (apply system* (string-append #+tar "/bin/tar") - "cvfa" #$output + "cvf" #$output + ;; The bootstrap xz does not support + ;; threaded compression (introduced = in + ;; 5.2.0), but it ignores the extra = flag. + (string-append "--use-compress-progr= am=3D" + #+xz "/bin/xz --threa= ds=3D0") ;; avoid non-determinism in the arch= ive "--mtime=3D@0" "--owner=3Droot:0" --=20 2.12.2 --NzB8fVQJ5HfG6fxh-- --i0/AhcQY5QxfSsSZ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAljdg+cACgkQJkb6MLrK fwjeVRAA0icp4AHyeaJKf4U1oovWmJHzY0rF5Qodvm8aA9z2PVWfe/4+L5pFGsCX KxK+sDZtYmW0ZokJZp1PfEUVc+zgzeJbc0LSOzXl0SYFQ/5ExHS1FSLxzlyu0LOg Fo9dxq18eT1HXOScL+FSq8h0+01DFP2ItNnbCRSPlUuvRjUhj/XfNzV/Kmmq9FyM dxFDO1TjDqY5C/1z2++zDUBM2Ajv61770a6LCvc8yWhejjsTmixxAmH6uOPHhoZ0 N5g8i85Cf/+FSCBhSZt9h6Ux57Wb/PO8ETC7rPR3afyyZ/i8p3vL1X8+WTddEras QV/Fi+xPYIMitK/MN3lTfhQEQRpLv9Z3gIlOLu2gU177S0YzIB4fjafncgQ6CS/8 XWXTk6kHGMWDg5U+n4aSLi51GUMOM9ZtNn6LtB3IOTHkm9Z30Bk0XzFwxu80QA2Z Ysgt6s696gYnvwz/M8K4mqQHopwSDdt1UVMQqyoh9Xat1Rirait2Rfz+2hHqpuzh 4V1YJWrT3ewFjXUvdZEtvODZzoVUXceKJpqTBC2FIb4z7mVdB+dgJN4vahp+hhJT gJfIfUNJlG8QaoWIofGoO68imhhEmQW8ho+6aXMTVArTidrF84lXq35xqZoxyKfE SKytwHeRZFjkv9ZmEdqsBrkrDvQkVb+vMoEM8AvVjR9CEn1Hlsk= =QzSR -----END PGP SIGNATURE----- --i0/AhcQY5QxfSsSZ-- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 31 14:15:50 2017 Received: (at 26316) by debbugs.gnu.org; 31 Mar 2017 18:15:50 +0000 Received: from localhost ([127.0.0.1]:54748 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cu15R-0006iM-Tv for submit@debbugs.gnu.org; Fri, 31 Mar 2017 14:15:50 -0400 Received: from lb1.openmailbox.org ([5.79.108.160]:36489 helo=mail.openmailbox.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cu15Q-0006iE-Bg for 26316@debbugs.gnu.org; Fri, 31 Mar 2017 14:15:49 -0400 Received: by mail.openmailbox.org (Postfix, from userid 20002) id 9B752515E17; Fri, 31 Mar 2017 20:15:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=openmailbox.org; s=openmailbox; t=1490984147; bh=XCutXNNMBE9n9C6mG4VWREKmo7sadI8YIx2SpqWIk1g=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=kFjDnDh62TlRqs+eXq9kAQ9tzHfNIovHOJGrhUpMKtyE7PyBmhjGWaiWXgXeSUy/A J8VJKrUlwyd84kgGDBhBfg/OaCx2Seb3A5pPzvChHP9ZHtPw6mfZXIhMV+fYB7BNue DAalNaKDRNpDjMzzsAkpcCt8afwU1fIu8FDL8fag= X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ZDZR002 X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=NO_RECEIVED,NO_RELAYS, T_DKIM_INVALID autolearn=disabled version=3.4.0 From: Kei Kebreau DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=openmailbox.org; s=openmailbox; t=1490984146; bh=XCutXNNMBE9n9C6mG4VWREKmo7sadI8YIx2SpqWIk1g=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=P2qmorwItqIEagiH5ctqXlpxHdDI+oIJSz7sVf6yke4bPEnmKBzw1U+VXXhnOW8VR ohRtAt7ojpbTjS9KhT5tEf0s46gNNq/xTozlZ/HCiEG8L5juPhY6gG/LRhsWbc/xY/ pMm1sPv1uI9l2E7Kd1ooEOlEhpOPh9m9CM5YGiWU= To: Leo Famulari Subject: Re: bug#26316: Enable threaded compression of source tarballs References: <20170330221715.GA10357@jasmine> Date: Fri, 31 Mar 2017 14:15:36 -0400 In-Reply-To: <20170330221715.GA10357@jasmine> (Leo Famulari's message of "Thu, 30 Mar 2017 18:17:15 -0400") Message-ID: <87efxd5n3r.fsf@openmailbox.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 26316 Cc: 26316@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) --=-=-= Content-Type: text/plain Leo Famulari writes: > This patch enables multi-threaded xz compression in the repacking stage > of building source tarballs. > > With it applied, I get a ~2.2x speedup building the source of > linux-libre with 4 threads, compared to 1 thread. > > Specifically, with the upstream linux-libre tarball in the page cache, > `guix build -S linux-libre` took ~366 seconds using a single thread for > xz, and ~164 seconds using 4 threads. > > From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001 > From: Leo Famulari > Date: Thu, 30 Mar 2017 03:10:48 -0400 > Subject: [PATCH] packages: Enable threaded compression of source tarballs. > > This provides a ~2x speedup when using 4 threads. > > * guix/packages.scm (patch-and-repack)[build]: Invoke xz with > '--threads=0' when re-packing tarballs. > --- > guix/packages.scm | 7 ++++++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/guix/packages.scm b/guix/packages.scm > index 4bc4b017f..30be5bf38 100644 > --- a/guix/packages.scm > +++ b/guix/packages.scm > @@ -573,7 +573,12 @@ specifies modules in scope when evaluating SNIPPET." > #:fail-on-error? #t))))) > (zero? (apply system* > (string-append #+tar "/bin/tar") > - "cvfa" #$output > + "cvf" #$output > + ;; The bootstrap xz does not support > + ;; threaded compression (introduced in > + ;; 5.2.0), but it ignores the extra flag. > + (string-append "--use-compress-program=" > + #+xz "/bin/xz --threads=0") > ;; avoid non-determinism in the archive > "--mtime=@0" > "--owner=root:0" LGTM. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEg7ZwOtzKO2lLzi2m5qXuPBlGeg0FAljenMgACgkQ5qXuPBlG eg1Wgg//V2rtEkLIqUcujXU+bEAqkJI0EyLUUXo5N4RTgaDEaKABU939NPO9Z7GK mby7VeCu8xmJFU3zsHYqp+71dM1GgE3HQrKBKwAs4tz+flUKtjCg1c7eWvhQIz3s EhuQnW/mF+9mzjanSWz+DyqXPwgTTz2uWC9l3WJ1J1xf7Ko4AiSNRliEuQ5V1t4x JfWAUHpAeg6lx8LRqfpKD8vGcnMbKCPelcMmd7MTE8BIqY886A1mDQ5+i44LXfzm /bT7SujpHZKzBnkfSXT/Qqps70ZzOerXpozkYNVwXVWvufEWIYUnb0x8S4WXozW8 KQWqAIYH+Gdx8KIv0wtY6zdPYt89padXLcTf6O9mORF1BQvwjCYgGRnK+4THrtao jtlB28Qf0DX29H/5y0Enz6zNbIaCKTtSx3PvQNQ6PA8NvsJp78wwnuuLVP1Jp9xY LwIHpFyHZ4kg/OegPdjtMlEkr3WmwxBqwgdsyZLlJP/BT9aoY1YOmV5ag1K31YV6 m0T1pcGlu0B6bjN5VyUFM/CbLCb8ngECM2KHW/lafqnOEY44l532nBRY1tT7Ogas kEcjf9paNetlX+1avjs15WxeGbzCUvGGayYHt9+WTDgZfjNpqwM2tqsLXoaBVCtM g8fwC9nFj7lSYVkGfnZ9Yw6Spd4npZTpxGQQ0KWeMGvPVb6jTh0= =Gn6v -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 31 19:04:18 2017 Received: (at 26316) by debbugs.gnu.org; 31 Mar 2017 23:04:18 +0000 Received: from localhost ([127.0.0.1]:54885 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cu5ac-00004w-4t for submit@debbugs.gnu.org; Fri, 31 Mar 2017 19:04:18 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:41738) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cu5aZ-0008WS-60 for 26316@debbugs.gnu.org; Fri, 31 Mar 2017 19:04:15 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 23DFB2081D; Fri, 31 Mar 2017 19:04:15 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute4.internal (MEProxy); Fri, 31 Mar 2017 19:04:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=cc:content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc :x-sasl-enc; s=mesmtp; bh=DJPeNa0cZuFPsdAyOf7GYtNOVx6pkQDO1elPGs qXvDc=; b=nMjXqT0PolZn9k6x7ej5xCtdUVBrHICQxfWZLMIBxZSbI74ZttwkhU 3Z67cRlrqWsI8P1bgJuDfT1CUXtapzQurIFKJU7VZBOeCJ5XmpDTOQvLev/ITWnk YbSEnH0FFFqGl2HoIfrXcDf42b5cjLrqjDBo4LMFp3iml+0awGbj8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=DJPeNa0cZuFPsdAyOf 7GYtNOVx6pkQDO1elPGsqXvDc=; b=XLqnh/fTI9FVtvaq/ABDSPzuooGqi36DHn 4yHSxhSAAfr3nrLBFeI0+yq1uEsLXFMCw9jWqacUfCi0FQIOaVKrBmfYXmXyGg+m mP/bErxrgxjRUa/k2Dk8apLfB+WEZadTc3lICF1MhuB+1hqt7FAhoRQXFqFSZX7T txIJ5amXvnrJaj4vDQdrCQPu9eH5Cw0LWuCPmj4si3YLkVTxdDTA6fmx+1L3AIH7 Mj0LhdF8X1AFVJlBhgUz5ZiR8ZxgyQ/lZMRBeQqM/fTeMfokRPh7qn6KfJa4yzIs eTmmZtTgncRHo5x2ZvXdIPnpjIxe5aZ+oE26v8H0xvl3uBVUBFCA== X-ME-Sender: X-Sasl-enc: NrgdUgUkOrrFuk75YDe4A6F4CjanVWOLrJrfSKps990s 1491001454 Received: from localhost (unknown [172.56.28.204]) by mail.messagingengine.com (Postfix) with ESMTPA id D6FDA7E350; Fri, 31 Mar 2017 19:04:14 -0400 (EDT) Date: Fri, 31 Mar 2017 19:04:13 -0400 From: Leo Famulari To: Kei Kebreau Subject: Re: bug#26316: Enable threaded compression of source tarballs Message-ID: <20170331230413.GB9642@jasmine> References: <20170330221715.GA10357@jasmine> <87efxd5n3r.fsf@openmailbox.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Yylu36WmvOXNoKYn" Content-Disposition: inline In-Reply-To: <87efxd5n3r.fsf@openmailbox.org> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26316 Cc: 26316@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --Yylu36WmvOXNoKYn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Mar 31, 2017 at 02:15:36PM -0400, Kei Kebreau wrote: > Leo Famulari writes: > > Subject: [PATCH] packages: Enable threaded compression of source tarbal= ls. > > - "cvfa" #$output > > + "cvf" #$output > > + ;; The bootstrap xz does not sup= port > > + ;; threaded compression (introdu= ced in > > + ;; 5.2.0), but it ignores the ex= tra flag. > > + (string-append "--use-compress-p= rogram=3D" > > + #+xz "/bin/xz --t= hreads=3D0") One question I have is about the removal of 'cvfa'. Here is the relevant documentation: -a, --auto-compress Use archive suffix to determine the compression program. AFAICT, we only build XZ archives, so it's okay to remove 'a' and hard-code the use of xz. Is that correct? > > ;; avoid non-determinism in the = archive > > "--mtime=3D@0" > > "--owner=3Droot:0" >=20 > LGTM. Thanks! This does mean rebuilding the world, so we'd need to save it for the next core-updates cycle. --Yylu36WmvOXNoKYn Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlje4G0ACgkQJkb6MLrK fwicNQ//aBgblz9wl1jiLnjh78oDncpHRm5NjjoVW8HtMXJmiMQj/Ynp02vmLV/S gZFz2SjwRDNiEEWaJQ2lXPYvpVtlz27wEas1+WahkmNUFDr/IqpAiYYj9qf9vaA5 mup3n627krMRJ1CVBd4PTCeCkdB5Au/Zc0cISq6oD7GUkyGMo4cz9BuuO5Mm6WN3 Rcuq6lm03SvX9unFOdHf5eYre4vGVGcNuSp0dWKKidK8JpeaaIEpe8GQrEp/QgUM sX5nJYt02nPVA3d6KmlZ18OzxkafZdw94X1wXCMy4Ral1ilyOt/9XvCazfHKr7vv upPprjMWueBXW6iGPJEP0VoqR22drZfjiYKhjN1NKPdmuX/a08Egz9JfeQOe1ssp a0KvVjqo/HhVNEhIXuZdlzmaU6UyAMbm5GHUw3QIXKOrYpgaZdFCyKNh+s/WnHsa wnl5vQMdolEGHNcV+foJGGxIl2ePy8MWYt+F+K6+bcEJ6RTEx9Q+BARmJcEQYrWa nPcdlC3Z7gdP/fpiRFtEDbvbIotbqjOPVLcui5uv68WJHM2N2XWZnq5BsyqEuNcf 6Q1blRHn/yp+CSKbDhRZO6/mgUlvHiEOn5i0ZCueX0yNW2HHv3+2oNtoiCxmnHhr pGT4cF6wtFdhn+exONnTs/4wOJ1xUlrUIxWy/ocu+GxnDOxxB4g= =UKB9 -----END PGP SIGNATURE----- --Yylu36WmvOXNoKYn-- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 31 19:56:33 2017 Received: (at 26316) by debbugs.gnu.org; 31 Mar 2017 23:56:33 +0000 Received: from localhost ([127.0.0.1]:54891 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cu6PB-0001Ej-BS for submit@debbugs.gnu.org; Fri, 31 Mar 2017 19:56:33 -0400 Received: from lb1.openmailbox.org ([5.79.108.160]:36220 helo=mail.openmailbox.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cu6PA-0001Eb-1X for 26316@debbugs.gnu.org; Fri, 31 Mar 2017 19:56:32 -0400 Received: by mail.openmailbox.org (Postfix, from userid 20002) id D4BD9512C26; Sat, 1 Apr 2017 01:56:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=openmailbox.org; s=openmailbox; t=1491004590; bh=ZnFoQzrFqdSaEkJ3IQbOF8dh+zxm9s6993dv4tLeHgo=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=y8rY9lPwIKAhdQsJ7/UpAlX2x/jut/XMKdJ2vvxzPlOe/LKQSljodfyvXCX81VrO9 Y78uSHa7NE9ao2VjFPSYSGBVwIYvOwGE3O/LkZypUcU+IJpRadqgPKfXhBn+Cs66cK Izve2jxkE+QVbP/NnlUfuVmxH722j2di5VztsCbo= X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ZDZR002 X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=NO_RECEIVED,NO_RELAYS, T_DKIM_INVALID autolearn=disabled version=3.4.0 From: Kei Kebreau DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=openmailbox.org; s=openmailbox; t=1491004590; bh=ZnFoQzrFqdSaEkJ3IQbOF8dh+zxm9s6993dv4tLeHgo=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=y8rY9lPwIKAhdQsJ7/UpAlX2x/jut/XMKdJ2vvxzPlOe/LKQSljodfyvXCX81VrO9 Y78uSHa7NE9ao2VjFPSYSGBVwIYvOwGE3O/LkZypUcU+IJpRadqgPKfXhBn+Cs66cK Izve2jxkE+QVbP/NnlUfuVmxH722j2di5VztsCbo= To: Leo Famulari Subject: Re: bug#26316: Enable threaded compression of source tarballs References: <20170330221715.GA10357@jasmine> <87efxd5n3r.fsf@openmailbox.org> <20170331230413.GB9642@jasmine> Date: Fri, 31 Mar 2017 19:56:18 -0400 In-Reply-To: <20170331230413.GB9642@jasmine> (Leo Famulari's message of "Fri, 31 Mar 2017 19:04:13 -0400") Message-ID: <877f3557bx.fsf@openmailbox.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 26316 Cc: 26316@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Leo Famulari writes: > On Fri, Mar 31, 2017 at 02:15:36PM -0400, Kei Kebreau wrote: >> Leo Famulari writes: >> > Subject: [PATCH] packages: Enable threaded compression of source tarba= lls. > >> > - "cvfa" #$output >> > + "cvf" #$output >> > + ;; The bootstrap xz does not su= pport >> > + ;; threaded compression (introduced in >> > + ;; 5.2.0), but it ignores the extra flag. >> > + (string-append "--use-compress-program=3D" >> > + #+xz "/bin/xz --threads=3D0") > > One question I have is about the removal of 'cvfa'. Here is the relevant > documentation: > > -a, --auto-compress > Use archive suffix to determine the compression > program. > > AFAICT, we only build XZ archives, so it's okay to remove 'a' and > hard-code the use of xz. Is that correct? > The surrounding code definitely seems to suggest that. Reading "up" the code from the gexp->derivation call I can find hardcoded xz usage in the tarxz-name procedure and this bit: (mlet %store-monad ((tar -> (lookup-input "tar")) ->(xz -> (lookup-input "xz")) (only xz here!)/ (patch -> (lookup-input "patch")) (locales -> (lookup-input "locales")) (decomp -> (lookup-input decompression-type)) (patches (sequence %store-monad (map instantiate-patch patches)))) >> > ;; avoid non-determinism in >> > the archive >> > "--mtime=3D@0" >> > "--owner=3Droot:0" >>=20 >> LGTM. > > Thanks! This does mean rebuilding the world, so we'd need to save it for > the next core-updates cycle. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEg7ZwOtzKO2lLzi2m5qXuPBlGeg0FAlje7KMACgkQ5qXuPBlG eg0yJxAAlBLapj0RbRrFF3zNg/peTHD1mowdQp8y538EBhOut121+Ud6tEu0Ygx/ 2T3PxW4xfLXAIGvl55IbINsyjYSbkM7SdM2J/i2DqXKFFmLMoVI67tMhOjA3PIVz eW0z/An/gwqRwHWN77/I1hwMhFmCvb29qV43erMpuL8xfWAw+zUns6n+TxzJtqyu +VASNwRMjVxg5q4AUYJwWpNH4XXteAGD88+mUN7c7P+DRrJiJrFRsACqlzMXS8Rs +PiGyU3uNZkz8hJLGo5ap7wd6ddedKw/hwMzVUWJFkUQK4GXCu5qU0s00uKd1lKO 4buot7MKYkGcHGP+jaKU0IB6CNcmmEN110sSjHl5abIKKTSIZHgoFi3Xpyo2WCrf WmE+YUUd34y2dxA1u+EPxWWrLeKAd+WvmjumHdyf9eBIPoGneebbrXJZB+OFP2BM e+kNMoVIASn+/O91NO1cO2KBLUcRxsc7mIMgeGEysxq9qIYIGSFIUpR2D8RYWqrZ nhku5eXGt0qOZ0OQxfonNm2UHwM0OQP/sOASos80v8MNt2pFtarAZoHGBE3WGaCI XLt794nfkz7wSZfu0w5fV09jewFam57+789P4x6XJ7pfsI4XFtCSC9A0PpEZ4fKs vhLio3Wp1te62oGMo3dxgw5qFPSmhfqY3un2CjWffT7MHf5w2wg= =RsTh -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 31 20:16:03 2017 Received: (at 26316) by debbugs.gnu.org; 1 Apr 2017 00:16:03 +0000 Received: from localhost ([127.0.0.1]:54899 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cu6i2-0001hS-1B for submit@debbugs.gnu.org; Fri, 31 Mar 2017 20:16:02 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:59488) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cu6hy-0001gz-Pd for 26316@debbugs.gnu.org; Fri, 31 Mar 2017 20:16:00 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 1DF8820A5E; Fri, 31 Mar 2017 20:15:58 -0400 (EDT) Received: from frontend2 ([10.202.2.161]) by compute4.internal (MEProxy); Fri, 31 Mar 2017 20:15:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=cc:content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc :x-sasl-enc; s=mesmtp; bh=wPK1jCA9oPEMcpJ7zpSEdOCz0vu8jEfS4jkY9l bgQGo=; b=y2XdETZN8xfnwo+MHH/XEc/BQAx+A3w3F8NY9LWG2w+2w1ourUUkO0 ZSt9xxMc/yLkdAl64Z4Lmsusl26wMQgjHo4Jr1YFZdCngNXjQQxkUVt043UdqNnJ nyaiSM09kJnUmfaIFBf+R3a/duDSAtU09YoR5Xsv5Fwj15+SiSOQA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=wPK1jCA9oPEMcpJ7zp SEdOCz0vu8jEfS4jkY9lbgQGo=; b=klnPDY9tGhXIxAkUPpwtk14qD5fMbwUq1F VJxgYYPG3R20bTVOGYJr8YTZxOGa3kRSst3evNfQ2kez5EWWuoxMQsMRl2qxYI+R sWEz6F3UjxxHYYWCxx2xto7Cy2jRkplc6dJvX2yQ6z/3pwTmmnA/PaQR3KGkddF8 Sdt2F+tIX9bmWOGRhEN6N88ae/I6/tiEBRKEJidCXe42X2XfSQ96pxziFxmPI8OE sZuZlt27MiZ1vyVS43tf2uuY9RviyTQjNNHa50fjjkLM5jdalwpSxHEt6IoGVuud vn5AfAjWcgoIxcHuKqilETqruIzb7glMtVy0Bim/fsbjqKTo9Efg== X-ME-Sender: X-Sasl-enc: gLOgv+uZuKhUPxivVI7tU8w8s7H2EzH/0IhydIlu9IhJ 1491005757 Received: from localhost (unknown [172.56.29.197]) by mail.messagingengine.com (Postfix) with ESMTPA id C17BF2400E; Fri, 31 Mar 2017 20:15:57 -0400 (EDT) Date: Fri, 31 Mar 2017 20:15:55 -0400 From: Leo Famulari To: Kei Kebreau Subject: Re: bug#26316: Enable threaded compression of source tarballs Message-ID: <20170401001555.GC11936@jasmine> References: <20170330221715.GA10357@jasmine> <87efxd5n3r.fsf@openmailbox.org> <20170331230413.GB9642@jasmine> <877f3557bx.fsf@openmailbox.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="vGgW1X5XWziG23Ko" Content-Disposition: inline In-Reply-To: <877f3557bx.fsf@openmailbox.org> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26316 Cc: 26316@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --vGgW1X5XWziG23Ko Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Mar 31, 2017 at 07:56:18PM -0400, Kei Kebreau wrote: > The surrounding code definitely seems to suggest that. Reading "up" the > code from the gexp->derivation call I can find hardcoded xz usage in the > tarxz-name procedure and this bit: >=20 > (mlet %store-monad ((tar -> (lookup-input "tar")) > ->(xz -> (lookup-input "xz")) > (only xz here!)/ (patch -> (lookup-input "patch")) > (locales -> (lookup-input "locales")) > (decomp -> (lookup-input decompression-type)) > (patches (sequence %store-monad > (map instantiate-patch patches)= ))) Right, that's what I was looking at. --vGgW1X5XWziG23Ko Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlje8TsACgkQJkb6MLrK fwgKKg//Qy8ftYEFxjDSyTQJjwzbWeOFQoOnLBZyrSuLr5o4Jw/2ZlcdqnVG2q0q NJ7Y44iYGPScd6/kdztWv3TmbIllUzWygBM+P7l7572h/Qgax0DzzYJRIA+Mt3RR 7/azlrMBoUPY68yBTZndmp1dL4oVw4KqXFuFscgNONwRAc9uATlW2T7V0Dd1YkoD lJ5cDkzrNH77mGxXesCN+I1MGjjxu+roLL0FO2C0iwk8gz3uEzs46efgj2MPF6Q0 pr4r4x20+XCAp6HTKgfhAAWEaJf3uxMJRc/iO9W3+eByZdaTqGG5+m+4VU/E2IhJ qTsbtrq3wo44OCPUrPhfMTgPQpJl+GWI3FAabGqKB96tBgEFVDLzxq4RHhpIPyCW 3c63E1fyxekMv6H6cHw7MvrUQnBPAwwJYRzepXBqjhvFFeUMEYj3rUVPJekfdXwE WKonuZAY626IFZEUXQFI+DstVC+Hz7FihHXAXQ7vbIXbhkp9tezkv1RnEACB0Pfy 3VgLlKUyM4gsnvkXIMoU2IrRobWsNsvTDpw8L3x/kNsF+INMtFbeLbAJYE6q7cLL ngGnMk8eQn1eOnzLsqyHqEvYQbnm2D9qeTfMmWJj7afI5XLFZ0Nbs54KlLGhoOk/ 3VuiT/JfRLfg6XMvB/mcB1kmcpszRHtDIQ7fmzw7zS660AeJQH0= =0v4l -----END PGP SIGNATURE----- --vGgW1X5XWziG23Ko-- From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 15 18:36:03 2017 Received: (at 26316-done) by debbugs.gnu.org; 15 Apr 2017 22:36:03 +0000 Received: from localhost ([127.0.0.1]:49476 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1czWIV-000247-2X for submit@debbugs.gnu.org; Sat, 15 Apr 2017 18:36:03 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:59771) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1czWIS-00023h-UZ for 26316-done@debbugs.gnu.org; Sat, 15 Apr 2017 18:36:01 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 08EE920738; Sat, 15 Apr 2017 18:36:00 -0400 (EDT) Received: from frontend2 ([10.202.2.161]) by compute4.internal (MEProxy); Sat, 15 Apr 2017 18:36:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc :x-sasl-enc; s=mesmtp; bh=bOODkzy/XTV+TYDcvYUJckMcQIUD/GOZtKTz11 IZTVQ=; b=bRaBGD35HpJIGx7FqFAVEung3GsB6nTVbB2rhGgCrqDL+fCzGsCQ+x nV/28mb02GGPgVFwR/aQ3golilTGMHnYwpeEAV50KphabMarJZ3eyDHSkDMBmMNB U6i2MOGQtePc6QEiBGoZUbnGYYLxhqC/69AGPx9x+yMnAPOuOGDTQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=bOODkzy/XTV+TYDcvY UJckMcQIUD/GOZtKTz11IZTVQ=; b=DIbX69LrotHFR8our3IHx5uOW6NGQIbo40 dEDMo3uMKvFMHvoI+4kqisG+7t9dVNrEfTaA1St8xBDwgzItmGgYtlhleph9O+29 vhUpa9uoSeSOhUxwDbYl7HaDTon3PdfDziDBPIJ7Qn/JXN6m3FFoWv3ra24wsO1N hJvkKqwm4+SFEGmQumi7apNU+43yjYV0bmVuNnQ+xSamYObLGaavRrl87jCKyeAJ PEIs2VoE7k3pXCzUMKYgRhjPHBvgZkqRkROFlFLF00uy3gIey9XYIxWaCv9wg0Jm LJyVcDHcXUUu966/8aE4PSkc+zgSkA2YCAcJz6iyBw7JLlT3z0Ng== X-ME-Sender: X-Sasl-enc: 7q/tf8NDGY1mTSdiuRvzsAE2npquSt1TBCeguP15nF5V 1492295759 Received: from localhost (unknown [172.58.225.158]) by mail.messagingengine.com (Postfix) with ESMTPA id B4B42246FC for <26316-done@debbugs.gnu.org>; Sat, 15 Apr 2017 18:35:59 -0400 (EDT) Date: Sat, 15 Apr 2017 18:35:58 -0400 From: Leo Famulari To: 26316-done@debbugs.gnu.org Subject: Re: Enable threaded compression of source tarballs Message-ID: <20170415223558.GA20274@jasmine> References: <20170330221715.GA10357@jasmine> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Nq2Wo0NMKNjxTN9z" Content-Disposition: inline In-Reply-To: <20170330221715.GA10357@jasmine> User-Agent: Mutt/1.8.1 (2017-04-11) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 26316-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --Nq2Wo0NMKNjxTN9z Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 30, 2017 at 06:17:15PM -0400, Leo Famulari wrote: > This patch enables multi-threaded xz compression in the repacking stage > of building source tarballs. >=20 > With it applied, I get a ~2.2x speedup building the source of > linux-libre with 4 threads, compared to 1 thread. >=20 > Specifically, with the upstream linux-libre tarball in the page cache, > `guix build -S linux-libre` took ~366 seconds using a single thread for > xz, and ~164 seconds using 4 threads. > From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001 > From: Leo Famulari > Date: Thu, 30 Mar 2017 03:10:48 -0400 > Subject: [PATCH] packages: Enable threaded compression of source tarballs. >=20 > This provides a ~2x speedup when using 4 threads. >=20 > * guix/packages.scm (patch-and-repack)[build]: Invoke xz with > '--threads=3D0' when re-packing tarballs. I pushed this to core-updates as c8a3dea847bb9f87fa1876d0c6c3356d6226f121. --Nq2Wo0NMKNjxTN9z Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAljyoE4ACgkQJkb6MLrK fwgtORAAwRJijd1s0a8TGZRGPsgIwbWxNG7kU4fftfHXMQGhTJNhuP8coCj9zGZw cIr/uWCQw64nVsyunyVGj2m44F70QIvCJOo/9Ctog/WGLBfY6mnhuC1XsbBslFcE 0E8gPDzh+hvx/ApmWWmxSWGtET0udq6QPYguVQm3LIjMvioIfIGUtHFrsCs6GsOs P4rqQI20nVMBxgcioop+ezAzgzLMap8t6ba1MtHs7wWv74WVMYLM7/qEi+JYP6rG bzVQ/9tmAsUleLqcOUZGMiGxPWxi0h8u1QLETAc/00vdlwRs5AkWuyrUv5YO3QkY bkE6bOP5SBfX+l8EqpD1ZXTO+vbpGD9IaZTjPUzOqa9+ft4XVVYk6+1j4m2GzzEP ta9JW0kY5ushPhEDRL118sY+3Vsrs8s5nIf69D9wHdqmUAq7KH84TowxKSbxhmeU Q/DXDUnauS1+E6vS+T76ppRHjgC3QdCkqMtfEO0WQv7AAKsgNKFn1xeDxehrM25z p53keqPD206SFe4HpY1WF2+/Ly5JYK90TaHDLjs5FzVg0Xi3EAg+2dgObMA8jL3q hdhEjODgomkMwlSPEtLbNZoH2HyIMWcO7qTXV/2WYPEo+DFkNgoT6yYRyS0FscmQ a3+E11ubv71BeepiT/gDamqi0yXwqCjuOicBeMYIHQD3uSVzBtM= =6K3L -----END PGP SIGNATURE----- --Nq2Wo0NMKNjxTN9z-- From unknown Sat Aug 16 18:20:00 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sun, 14 May 2017 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator