From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 06 02:56:46 2018 Received: (at submit) by debbugs.gnu.org; 6 Dec 2018 07:56:46 +0000 Received: from localhost ([127.0.0.1]:34917 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUoWc-0008Qs-1W for submit@debbugs.gnu.org; Thu, 06 Dec 2018 02:56:46 -0500 Received: from eggs.gnu.org ([208.118.235.92]:60856) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUoWa-0008Qf-5H for submit@debbugs.gnu.org; Thu, 06 Dec 2018 02:56:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gUoWN-0001bn-8M for submit@debbugs.gnu.org; Thu, 06 Dec 2018 02:56:36 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,UNPARSEABLE_RELAY autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:58498) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gUoWG-0001Vh-IM for submit@debbugs.gnu.org; Thu, 06 Dec 2018 02:56:27 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42073) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gUoWF-0006Ii-6M for guix-patches@gnu.org; Thu, 06 Dec 2018 02:56:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gUoWB-0001QF-2x for guix-patches@gnu.org; Thu, 06 Dec 2018 02:56:22 -0500 Received: from li622-129.members.linode.com ([212.71.249.129]:43826 helo=mira.cbaines.net) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gUoWA-0001OA-KC for guix-patches@gnu.org; Thu, 06 Dec 2018 02:56:18 -0500 Received: by mira.cbaines.net (Postfix, from userid 113) id D41D216851; Thu, 6 Dec 2018 07:56:16 +0000 (GMT) Received: from localhost (cpc102582-walt20-2-0-cust14.13-2.cable.virginm.net [86.27.34.15]) by mira.cbaines.net (Postfix) with ESMTPSA id A75FC16844 for ; Thu, 6 Dec 2018 07:56:16 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 13c15c60 for ; Thu, 6 Dec 2018 07:56:15 +0000 (UTC) From: Christopher Baines To: guix-patches@gnu.org Subject: [PATCH] gnu-build-system: Enable xz to decompress in parallel. Date: Thu, 6 Dec 2018 07:56:15 +0000 Message-Id: <20181206075615.4637-1-mail@cbaines.net> X-Mailer: git-send-email 2.19.2 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) It can take a little while to decompress some packages with large xz compressed source tar files. xz includes support for parallelism, so enab= le this using the parallel job count for the overall derivation. * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T opt= ion to xz to enable it to work in parallel if appropriate. --- guix/build/gnu-build-system.scm | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-syste= m.scm index e5f3197b0..9d11e5b1e 100644 --- a/guix/build/gnu-build-system.scm +++ b/guix/build/gnu-build-system.scm @@ -147,7 +147,7 @@ chance to be set." locale (strerror (system-error-errno args))) #t))) =20 -(define* (unpack #:key source #:allow-other-keys) +(define* (unpack #:key source parallel-build? #:allow-other-keys) "Unpack SOURCE in the working directory, and change directory within t= he source. When SOURCE is a directory, copy it in a sub-directory of the c= urrent working directory." @@ -161,6 +161,10 @@ working directory." (copy-recursively source "." #:keep-mtime? #t)) (begin + (when parallel-build? + (setenv "XZ_OPT" + (format #f "-T~d" (parallel-job-count)))) + (if (string-suffix? ".zip" source) (invoke "unzip" source) (invoke "tar" "xvf" source)) --=20 2.19.2 From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 06 03:08:43 2018 Received: (at 33643) by debbugs.gnu.org; 6 Dec 2018 08:08:43 +0000 Received: from localhost ([127.0.0.1]:34922 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUoi5-0000HK-6g for submit@debbugs.gnu.org; Thu, 06 Dec 2018 03:08:43 -0500 Received: from li622-129.members.linode.com ([212.71.249.129]:33540 helo=mira.cbaines.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUohz-0000H8-Tl for 33643@debbugs.gnu.org; Thu, 06 Dec 2018 03:08:35 -0500 Received: by mira.cbaines.net (Postfix, from userid 113) id 144E816851; Thu, 6 Dec 2018 08:08:31 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (cpc102582-walt20-2-0-cust14.13-2.cable.virginm.net [86.27.34.15]) by mira.cbaines.net (Postfix) with ESMTPSA id BB8731683E for <33643@debbugs.gnu.org>; Thu, 6 Dec 2018 08:08:30 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 06f451d5 for <33643@debbugs.gnu.org>; Thu, 6 Dec 2018 08:08:30 +0000 (UTC) References: <20181206075615.4637-1-mail@cbaines.net> User-agent: mu4e 1.0; emacs 26.1 From: Christopher Baines To: 33643@debbugs.gnu.org Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. In-reply-to: <20181206075615.4637-1-mail@cbaines.net> Date: Thu, 06 Dec 2018 08:08:29 +0000 Message-ID: <87pnufnkle.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33643 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Christopher Baines writes: > It can take a little while to decompress some packages with large xz > compressed source tar files. xz includes support for parallelism, so enable > this using the parallel job count for the overall derivation. I'm guessing this is only suitable for core-updates, as it'll cause a lot of rebuilds. I'm also not sure if it's worth it, but it does seem to make building some packages at least start faster. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAlwI2P1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9Xe2Xg/9EVDJpU28AcK1fOhFNm8Jo1h0j6loey3ZTc74K7hjr9yMYyJ+TJvYkBb1 RCEp4eRxsjL47JQQb4GAuB3q1x9lyLGwFuvuHkG/e/TBelVAXXLF8snC5aP3HhOc CULFbcK2TKdEbqG8IZS4Tr7nnTK3kOZHrzz1D+2DcrOMZL2tsQMJbveNRl3oSmWh QRuRh4Djtn6+8san16fjUQhDl3TK2zXwgh5YzwMdpZqI43dVTqnXV1ypMOhO79op p/eZVT0vV/XIgzAfo3UJFG1VRZEtgN3Vh1TUjmXYcZAezDnl8AvXmVi/j6jB8P9D 0rGPd0GuXrY/XZxjkQUeISv4VrzZqAm7sGE0hmZCFkSqHL4GSmMeyvObBjI8pi+s ck0f18KlWrH3TmO6uPu70D2tRa7ZclqfkEAFW6EUTs1pF65xkQDwSP8UmxzOVwLg 3QR03K3t1BQpogAPExzFHyzXKnkK1JB+2ERK65gECAhUC3isIs8jzjFGtB2M785z j22IR2Uy7K0+k1t1KI+WVVoKzYaECMVvib7hjTzlK0bnpciR24PAj1VGiQ2oZnH5 0nqgs2o33HPzmmod+KZfjyqq4N15DoJs+dThD6dg1AI0jKaahWDLADy7ldg98QkM YuZKYkjPPm37ZwK3VWlGhmR8cL8MAi3Jn1lADDdUdcGzjBzMljY= =gS3Q -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 06 03:14:13 2018 Received: (at 33643) by debbugs.gnu.org; 6 Dec 2018 08:14:14 +0000 Received: from localhost ([127.0.0.1]:34928 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUonS-0000Q8-61 for submit@debbugs.gnu.org; Thu, 06 Dec 2018 03:14:13 -0500 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:59673) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUonL-0000PR-U2 for 33643@debbugs.gnu.org; Thu, 06 Dec 2018 03:14:08 -0500 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 80FA721DB7; Thu, 6 Dec 2018 03:13:58 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Thu, 06 Dec 2018 03:13:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=mesmtp; bh=YpeZF1DiITtR8dX2rnDzYAjp YbioSNgWF2dZthYPqZ4=; b=pa8Am/jCUCTCaEuu6sa2BMGYiTeujQ4hAsmDG//I zqbcyj1nzQoe93SD2WtK9dEc9TSYgW++W4tc5N3qzYjcdLMEboDCqmtgVhIehNjS 6T5H1jS3qajvfd0sRaSDukwv1KrgmVDO6jCIUyywEKCobOk1D8aFxK1v9g/G20Kt uy8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=YpeZF1 DiITtR8dX2rnDzYAjpYbioSNgWF2dZthYPqZ4=; b=Ymt9QJ/ZNJeWCPxAlXIXcn 27qIPTDUFXjoi9VqADauFjwAM6cmNiFnY/5jPk4kwcEJ3rCy8P67ZNPF7uPYoaGu p2I3QyXyjoaDGrvz5a1NDaXCchiAcAN57OgsLzpefG4tP8EgHdeMsRk5pyPz0HYz 4HnIQ5djm/0GyJL4vEgNc8+Wth82CMjH4Cww/Y+cE1+JrNtlZPT1Mr/JZK7M2MRz luQke1i28O9m31E9yb48C7rc8bWaFFPDv7N4ic6Qs7b911giUie7avQTMHCF7DWF CCqApvJcOSqXMuSzJUbUsMQJzUcytXRv9g9fYgYjXO5YJ1M0SjOSwDtYzVG7r46Q == X-ME-Sender: X-ME-Proxy: Received: from localhost (c-76-124-202-137.hsd1.pa.comcast.net [76.124.202.137]) by mail.messagingengine.com (Postfix) with ESMTPA id 953F5E48FA; Thu, 6 Dec 2018 03:13:57 -0500 (EST) Date: Thu, 6 Dec 2018 03:13:52 -0500 From: Leo Famulari To: Christopher Baines Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. Message-ID: <20181206081352.GA31941@jasmine.lan> References: <20181206075615.4637-1-mail@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="DocE+STaALJfprDB" Content-Disposition: inline In-Reply-To: <20181206075615.4637-1-mail@cbaines.net> User-Agent: Mutt/1.11.0 (2018-11-25) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33643 Cc: 33643@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --DocE+STaALJfprDB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Dec 06, 2018 at 07:56:15AM +0000, Christopher Baines wrote: > It can take a little while to decompress some packages with large xz > compressed source tar files. xz includes support for parallelism, so enable > this using the parallel job count for the overall derivation. The xz man page says that multi-threaded decompression isn't implemented yet, unfortunately. --DocE+STaALJfprDB Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlwI2jsACgkQJkb6MLrK fwi1dw//e1LOv6SRdcQRXyHzDyIdkZDpJ/hIhB0QF3sHyv3Bn55Uw0ZlPO0y8+dF oXuEn10ZFlidOph0IF1J+G9fEqkdEqAqRnZ578b2AM/1UlivryJFLeld5sz0Qmye c2ZOWiLvzN0R9Baw0MiRwFjpfqafTq5Kx4jKkbWkG8uvwxFdUKEGLeF5AksfMIJU p6p7+7Lq9/6Pk1Z9s68ZCIB+ugyj6CqP67fMK9DZ6v/TI2O1IzOrKOs7KXj1ww9y BZVpCXI9W1dovDb4tTCziaxeOLtQhNefN0GQycubxy+dydAn29aAL3wQYbszg0vS USiPWDjz572mI5ROGHMrBYxzschgvWkgfvD95Fup97KUp1nJrAX4X6MeY0SayS6m edWNKY2h2U6DnXWEwqwE9t9+hPd/WKvCpgEoy0Y7mU2fwBpFjHVFz8Onn/w3vpUn fEV9f/yl0zm++vuh6Gma1xpNp8uTFR9bQjqfaI0hFYdcgCLJi9sIvg3aNOEo7zyH mD5yVG3kxc3BOhZqS6AYNTyDnz5xmRZy2Jk/1a1CjB15Cvzlcx78xxhKTf80zPa+ j63rBFauksoEk0lPXqRvBkvQdvTQbJOzvUjZ/TcltOljENc8IwZNcUt2QrNztN4i V6nCYnUE9WIrt+9axhOVO3xHOgElZL5BPuTThfyXRS17F0fHRXA= =HQ1Y -----END PGP SIGNATURE----- --DocE+STaALJfprDB-- From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 06 14:38:34 2018 Received: (at 33643) by debbugs.gnu.org; 6 Dec 2018 19:38:34 +0000 Received: from localhost ([127.0.0.1]:36060 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUzTm-0000pV-9v for submit@debbugs.gnu.org; Thu, 06 Dec 2018 14:38:34 -0500 Received: from li622-129.members.linode.com ([212.71.249.129]:34070 helo=mira.cbaines.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUzTi-0000pJ-1O for 33643@debbugs.gnu.org; Thu, 06 Dec 2018 14:38:30 -0500 Received: by mira.cbaines.net (Postfix, from userid 113) id 4B25616879; Thu, 6 Dec 2018 19:38:29 +0000 (GMT) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mira.cbaines.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from localhost (cpc102582-walt20-2-0-cust14.13-2.cable.virginm.net [86.27.34.15]) by mira.cbaines.net (Postfix) with ESMTPSA id 5423B16822; Thu, 6 Dec 2018 19:38:25 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 13a44de4; Thu, 6 Dec 2018 19:38:24 +0000 (UTC) References: <20181206075615.4637-1-mail@cbaines.net> <20181206081352.GA31941@jasmine.lan> User-agent: mu4e 1.0; emacs 26.1 From: Christopher Baines To: Leo Famulari Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. In-reply-to: <20181206081352.GA31941@jasmine.lan> Date: Thu, 06 Dec 2018 19:38:21 +0000 Message-ID: <87o99yo382.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33643 Cc: 33643@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Leo Famulari writes: > On Thu, Dec 06, 2018 at 07:56:15AM +0000, Christopher Baines wrote: >> It can take a little while to decompress some packages with large xz >> compressed source tar files. xz includes support for parallelism, so enable >> this using the parallel job count for the overall derivation. > > The xz man page says that multi-threaded decompression isn't implemented > yet, unfortunately. Ah, interesting. Having a read myself now, it also says it: "will work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single- threaded mode don't even if --block-size=size is used." So, if -T was used to compress the data, then it sounds like it'll work to decompress it. I guess this adds a little more uncertainty to the benefit of this change, as the impact is dependent on the way the source data is compressed. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAlwJeq1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XfjuA/+JYvjr859j70wQsyAsuHlu+wkuCQuADaxdk5XAXmbMy2Mn4/mMTqpFT1S svDdmFjPwECUG/wtuV0fAgT13g351AMykXKonLnYaAz5gwkJO1avhfdkbenOMZw/ 6Qfd01hgPujP/dLZ630hE6oPaVsaYdSvgo4SYvOXTN2Q51gtASnLz7X92iX8L1cX 5C0qiqWr7OlWTuf81quTnrmOw8iUirhna5k++DY3UlEYhvWql8XGoWlWe4Kx+V41 8oKYm3XKrg+GqqfkCPlUDvRQIqYHMb65bFR/nzExgjRYgKOG+kYv2JQAsPlJTADi 5MKQNGJRBOtca06q0PAvYKpibyO5hKdxc29vF30mNj+91TBrlLBRItiGeeGSVD3m W6yrv/u5uc6RyxYqSgTVGp5C29aY6zjkKCXNeuOqTDX9CqebLYQtut6Fk9EvuaiE hf6SQcGU66mkTgKzmvEbLazctgzJpknOuPTSeebUj8M5r6EDL67FY6vzBH1ZHnK7 PO/91eMGqnHa9RHLgvKNZ42uQJLgBX5qv1WVHqn9xByYspwFjjf4HEzDbUHm+5HW vcpOGLE2AYvM2mKH/jKKOYZf9iW11l+Y96Hm969XZHvXnJLW4FwzRQCO5/Kz3iMk SvxmiDpt9qvAJ0PAo9vfCeP5O09L/dLKyVW286uGI4OH3gOmPXs= =ITch -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 06 16:07:07 2018 Received: (at 33643) by debbugs.gnu.org; 6 Dec 2018 21:07:07 +0000 Received: from localhost ([127.0.0.1]:36078 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gV0rS-0003IB-Sf for submit@debbugs.gnu.org; Thu, 06 Dec 2018 16:07:07 -0500 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:36803) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gV0rP-0003HY-1t for 33643@debbugs.gnu.org; Thu, 06 Dec 2018 16:07:05 -0500 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id B00DB21990; Thu, 6 Dec 2018 16:06:57 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Thu, 06 Dec 2018 16:06:57 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=mesmtp; bh=jIopqqlJiZTaa8Dt7lTLwYpb STLIeuh0J3us36nDC2s=; b=MNn8sP9keRTAtccEoFW8ex6pZejIWXxWC2i/4CL2 gjlBFI/UaZ0/wv+g+ODHkKQxfvwZGqqVENs3hBsJCqJ85uAqeULzYa+z5mE7xsLN sQJVTuY/HzYxDpAcOKJ9fsP9tTtbfQIf4HIGE/dRi9fARyeaFA7x6g9hdWirNjB/ fno= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=jIopqq lJiZTaa8Dt7lTLwYpbSTLIeuh0J3us36nDC2s=; b=FRdcBdPUE5YGrNNBYElMuR C/T2grCZQhq6v/WgN1PENKX0mDkVzBN5WopV1lVtjC7cUWau8KaCnWntDqc5rUKx h4yugdSLJLrOVO+mWDkv0bJBZ1N5puxOsOls2sJRfc4sys+ShYonyvJI8WkUl5HR Op9F/rbnvCKevbu3EEWRi/3Pk2bP50+SBuwQi3OOwHsK47/Gl4IN8NMH/wtHmIT7 zfoJPKqleChpRz/FFqG6srp6X4djr9/Nx0fhWIFXuLVsbdhBbigXd0POzDq1jHeI w/ms/4z3CNUbBuA291BqMGa6gJ109SaA3JpZZ4Ax+dF72hbdF3fscUst/uZjeFwg == X-ME-Sender: X-ME-Proxy: Received: from localhost (c-76-124-202-137.hsd1.pa.comcast.net [76.124.202.137]) by mail.messagingengine.com (Postfix) with ESMTPA id C2B01102F2; Thu, 6 Dec 2018 16:06:55 -0500 (EST) Date: Thu, 6 Dec 2018 16:06:53 -0500 From: Leo Famulari To: Christopher Baines Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. Message-ID: <20181206210653.GA909@jasmine.lan> References: <20181206075615.4637-1-mail@cbaines.net> <20181206081352.GA31941@jasmine.lan> <87o99yo382.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="SLDf9lqlvOQaIe6s" Content-Disposition: inline In-Reply-To: <87o99yo382.fsf@cbaines.net> User-Agent: Mutt/1.11.0 (2018-11-25) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33643 Cc: 33643@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --SLDf9lqlvOQaIe6s Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 06, 2018 at 07:38:21PM +0000, Christopher Baines wrote: > So, if -T was used to compress the data, then it sounds like it'll work > to decompress it. I guess this adds a little more uncertainty to the > benefit of this change, as the impact is dependent on the way the source > data is compressed. Right. When parallel decompression is implemented, I think we should enable it in order to get some benefit from upstream tarballs that may have been created with multi-threaded compression.=20 However, we probably won't be able to use the parallel compression within Guix because it is apparently not deterministic: --SLDf9lqlvOQaIe6s Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlwJj2kACgkQJkb6MLrK fwiCQhAA4rGeNQ1PWm/23zpFcOKanr5Il9U6ppIY94BvLodzT+KfqTsCo2nH4AOg dTWHViH8ArhHlQrre0f0gcKWddMal3Z45lVV5auVgLaU6L5AaFp0h7Xe8cN0Z+vy HcC2E+XLTjOlZZTl2je9+69lxfNE48AtjIr4KfSwOhrOXdD7ahwNV7pLWSD8gc4u 1yT70guVTpNKp7qxUf/Vpv61avg2uy6LXCCIoR4lh8+EENTdwHqEDwh6EnFL8B2V zOzLPrdqSdW8z6o4mLha8LHHFdwcSQZNQWtK0p1I1TGBdZTKk9T73Nr29qLUF8LI iEQ5TH+VIaHDjDXNntJndBtmNdzROJSeul9c3QC0kL31YRg75EKQH9V3+rtScgU5 sHgiYZ9cMJiwYV0Gy9vLy3kZDQDjxdLCdsDMR+iG5q1N54c8ilWRw2vN24HDr3jD bOOG6LwIQsBSD2NmiHPa4byzHHBoMXPQrDbR6OpKGjgKGwEEBNT703tCaiyfbgDY 8rUFsfmD7OcL/BIYc0jO54pwHy04Y01+tNfkZwZegNYc1LkfXspPQTzmxN+GXOQ+ 4GjqcJqvz8uTE6GkfuxlMVDnq0CgcaN53n+zi+tzawXLftbPibDYh/xb8y9T+95I EqckOmrlNcVx52w1NItR2ULEetOMSjm7x3q70LpXonEThhozZY4= =fs3X -----END PGP SIGNATURE----- --SLDf9lqlvOQaIe6s-- From debbugs-submit-bounces@debbugs.gnu.org Sun Dec 09 09:32:13 2018 Received: (at 33643) by debbugs.gnu.org; 9 Dec 2018 14:32:13 +0000 Received: from localhost ([127.0.0.1]:39362 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gW07u-0000N2-8X for submit@debbugs.gnu.org; Sun, 09 Dec 2018 09:32:10 -0500 Received: from flashner.co.il ([178.62.234.194]:37046) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gW07s-0000Mn-EX for 33643@debbugs.gnu.org; Sun, 09 Dec 2018 09:32:08 -0500 Received: from localhost (unknown [5.102.239.133]) by flashner.co.il (Postfix) with ESMTPSA id 52BF94005A; Sun, 9 Dec 2018 14:32:02 +0000 (UTC) Date: Sun, 9 Dec 2018 16:32:01 +0200 From: Efraim Flashner To: Leo Famulari Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. Message-ID: <20181209143201.GA1307@macbook41> References: <20181206075615.4637-1-mail@cbaines.net> <20181206081352.GA31941@jasmine.lan> <87o99yo382.fsf@cbaines.net> <20181206210653.GA909@jasmine.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="7JfCtLOvnd9MIVvH" Content-Disposition: inline In-Reply-To: <20181206210653.GA909@jasmine.lan> User-Agent: Mutt/1.11.0 (2018-11-25) X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33643 Cc: 33643@debbugs.gnu.org, Christopher Baines X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --7JfCtLOvnd9MIVvH Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 06, 2018 at 04:06:53PM -0500, Leo Famulari wrote: > On Thu, Dec 06, 2018 at 07:38:21PM +0000, Christopher Baines wrote: > > So, if -T was used to compress the data, then it sounds like it'll work > > to decompress it. I guess this adds a little more uncertainty to the > > benefit of this change, as the impact is dependent on the way the source > > data is compressed. >=20 > Right. When parallel decompression is implemented, I think we should > enable it in order to get some benefit from upstream tarballs that may > have been created with multi-threaded compression.=20 >=20 > However, we probably won't be able to use the parallel compression > within Guix because it is apparently not deterministic: >=20 > If the tarball is compressed in parallel then it can be decompressed in parallel. As for compressing in parallel, it *might work* to pass it through our non-bootstrap tar for 'tar --sort=3Dname' and then pass it through xz -T(pick-a-num). --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --7JfCtLOvnd9MIVvH Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAlwNJ10ACgkQQarn3Mo9 g1ELVRAAllajPCkWXEEPQXn9pF4gHqjDzkf2N8mDNBKtd+iEILkv5QLlu+KaqPjJ vcg+ZGPD6upDXu/FnGNTuLn4aOcsqJ6ESG6dmHPO0v2yOZXN7StqlggxMkobe2ZE 5MOwW1V49xbmsh9GQBysqqn+Yer81kX3gepw/E5o+fnOlZJXQtkQqtAju8fwl/rF GSibFmRmq7rSdshFFH4fN+0WuIE1tIMTDJeATkX35l+hou6Y6nyg1GnzFwv6ItRx uVrzp+6XrMxK594TrdJ9EPfuWDl3ElKY1dEiDLcZkDArzZQJm7VZG/ONERJ6KtrB 8cLb+hW0ASGKWNMegt4NwmpbGJdnwHbBccYJs38GgzmfQIfivVLiIApIbT9REoM7 UuRUCgQ64zK7FlAsMzbsfZb+r23Eh+tVpz76+LBP2vlI1iRH2evSjeVWLcA0Hz7S loZeFT4sUN3Y2GO8GSawXzD8kSw2esg/mGSXtumfmiwL3VIe2xxahlfI/k6wOQtf 9qI5jJGybjoj0bopJSB2hLgsU7F5mCrbzN2tJqPcXSv/8uJ6tZTfaqeU2Y94jGPt HwbCZz/ikfJuKPeAiUQfnI7cZ6YbguE8c4criyLBiVDKFYL+KLMM0ERSFJgmYtXE lktAbnwMz8SdnN34lJVZORsF1KLbapk5NCWcyvkJvd801Rbsf7k= =GGyt -----END PGP SIGNATURE----- --7JfCtLOvnd9MIVvH-- From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 10 11:24:45 2018 Received: (at 33643) by debbugs.gnu.org; 10 Dec 2018 16:24:45 +0000 Received: from localhost ([127.0.0.1]:42065 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gWOMP-00012S-A8 for submit@debbugs.gnu.org; Mon, 10 Dec 2018 11:24:45 -0500 Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:43557) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gWOMM-00012B-Bp for 33643@debbugs.gnu.org; Mon, 10 Dec 2018 11:24:43 -0500 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 68133B1E; Mon, 10 Dec 2018 11:24:36 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 10 Dec 2018 11:24:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=mesmtp; bh=rSWLHSMAsA34BSH+nyZ3Dr// Uqm7VJWrF+bTVg25+Tc=; b=EG7fAj/oWlnPyG+F3ivc/DZKGxeU0BRRK4G0G+fT A3A6gzsL2NTOgqaFQ8al/3+t6p+JCJXtQIQsgv8QEWtb96FubfNR77IKEH0r1hSi g8JULTRC/n/gRM0NzDkSlKNxqfsdKI/7+vG3PRCSnVlntakM2z0zDSt83bxlHQEY jxA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=rSWLHS MAsA34BSH+nyZ3Dr//Uqm7VJWrF+bTVg25+Tc=; b=zW2HZZZvXr8UxszXYsXmcK w1ycuw0EujvVs7VtRZEcUpEFIMBttkatSm9AEiHyVkM0aRy4HisgRov8E+ysNhRU GcEBKdNLgHFt/OHw6O0QD3eQnPzALuF5JIGOQya0+IEG39CRtVseP8DPwrX3Nw0s mQGW5IlQtCVrWAFYUpGkZ7iUXPuhPgJvbQhS2i8vcQBIuypgN5BrGMKgHqT7Ugg/ t1KyQ+5K2/XoOk7fFGoXeMnP+H/xUKncIlmmrztpA8K/eomIL1a0JopqlbKxMhjg ueHm5M+dWz/L2mtfav0VaE1xA5mUV6bkclPpnhs3Q0TtKahjOXIySAugIQHVUCgg == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedtkedrudeghedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfquhhtnecuuegrihhlohhuthemucef tddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpeffhffvuffkfh ggtggujggfsehgtderredtredvnecuhfhrohhmpefnvghoucfhrghmuhhlrghrihcuoehl vghosehfrghmuhhlrghrihdrnhgrmhgvqeenucfkphepjedurddutdehrddvtddtrdejvd enucfrrghrrghmpehmrghilhhfrhhomheplhgvohesfhgrmhhulhgrrhhirdhnrghmvgen ucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from localhost (pool-71-105-200-72.nycmny.fios.verizon.net [71.105.200.72]) by mail.messagingengine.com (Postfix) with ESMTPA id 0C44910411; Mon, 10 Dec 2018 11:24:34 -0500 (EST) Date: Mon, 10 Dec 2018 11:24:29 -0500 From: Leo Famulari To: Efraim Flashner Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. Message-ID: <20181210162429.GA9407@jasmine.lan> References: <20181206075615.4637-1-mail@cbaines.net> <20181206081352.GA31941@jasmine.lan> <87o99yo382.fsf@cbaines.net> <20181206210653.GA909@jasmine.lan> <20181209143201.GA1307@macbook41> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="AqsLC8rIMeq19msA" Content-Disposition: inline In-Reply-To: <20181209143201.GA1307@macbook41> User-Agent: Mutt/1.11.0 (2018-11-25) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33643 Cc: 33643@debbugs.gnu.org, Christopher Baines X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --AqsLC8rIMeq19msA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Sun, Dec 09, 2018 at 04:32:01PM +0200, Efraim Flashner wrote: > If the tarball is compressed in parallel then it can be decompressed in > parallel. The xz documentation says that parallel decompression is not implemented? Is that no longer the case? > As for compressing in parallel, it *might work* to pass it through our > non-bootstrap tar for 'tar --sort=name' and then pass it through xz > -T(pick-a-num). That could be helpful! --AqsLC8rIMeq19msA Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlwOkzcACgkQJkb6MLrK fwjnkRAAzP70xYA6VIbn7uIJTAgM40T/iuuUCq5WMdfeuh8b15mx04YJXYgSx4zR i3YUoxeDSccXa36dW9yGckQdwU58pveQfId4e4D3Wwj017I3pSO+aRis/CaDL0a2 Kt29sNhb8l1LKlVMXzrFvDm4aKGN5FFJrw7eo2zYSNAFXmn4gjfxomv0kj0g0KeX YOVxhDHiS9XSfrVdEcyOIROjI/eFKBw3uqjKYIVJBymUyFVPanN/A/9fdW8mHawp cWIeTr9REpUeuFej8aU2UdDROcqj+pKh2UdcuKeOV5aWniEheSr3AKtmQOIfv8d8 UMi/O7AnTviASzfVSOBfvzXHoRo7ebbPRcM2lW9bKr2hsAQOPO6VOdPUfuqqPKld mCgQGQYolj0+GKQP+eQTazi4LbErEQ0EzLTc4ZkmHfgqRQv8eXec2hPOP+Gj/ZJc oAMUgUzSsPwsZPwdyjQbuMD21cqhXsP7xiysQvpJMo59llJK0cnNNXx5PwP4e0vH KeC6ZhmHq2jZG+5vt9VxflNJxisutvSB2h6lH1ye8NIfNmF/ercwl0dL6f5uKK1i B0ZjqGnb2+ROk/cLXxemfkWSDBrEE9WYLjU4VRVaR5uZlz4OkYn7Xa7X09Nhcuwu 7V2ZeaS81Kkxh8/00hSRFo6MN6PGEMVgrLGQBa6v7K1wnUXbB7Y= =mQov -----END PGP SIGNATURE----- --AqsLC8rIMeq19msA-- From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 10 13:48:53 2018 Received: (at 33643) by debbugs.gnu.org; 10 Dec 2018 18:48:53 +0000 Received: from localhost ([127.0.0.1]:42148 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gWQbt-0004bu-Ew for submit@debbugs.gnu.org; Mon, 10 Dec 2018 13:48:53 -0500 Received: from flashner.co.il ([178.62.234.194]:39788) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gWQbr-0004bf-HL for 33643@debbugs.gnu.org; Mon, 10 Dec 2018 13:48:52 -0500 Received: from localhost (space.telavivmakers.org [82.80.54.64]) by flashner.co.il (Postfix) with ESMTPSA id 63E464002E; Mon, 10 Dec 2018 18:48:45 +0000 (UTC) Date: Mon, 10 Dec 2018 20:48:43 +0200 From: Efraim Flashner To: Leo Famulari Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. Message-ID: <20181210184843.GA1323@macbook41> References: <20181206075615.4637-1-mail@cbaines.net> <20181206081352.GA31941@jasmine.lan> <87o99yo382.fsf@cbaines.net> <20181206210653.GA909@jasmine.lan> <20181209143201.GA1307@macbook41> <20181210162429.GA9407@jasmine.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="qMm9M+Fa2AknHoGS" Content-Disposition: inline In-Reply-To: <20181210162429.GA9407@jasmine.lan> User-Agent: Mutt/1.11.0 (2018-11-25) X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33643 Cc: 33643@debbugs.gnu.org, Christopher Baines X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --qMm9M+Fa2AknHoGS Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Dec 10, 2018 at 11:24:29AM -0500, Leo Famulari wrote: > On Sun, Dec 09, 2018 at 04:32:01PM +0200, Efraim Flashner wrote: > > If the tarball is compressed in parallel then it can be decompressed in > > parallel. >=20 > The xz documentation says that parallel decompression is not > implemented? Is that no longer the case? Looks like I got caught up with the original release notes. https://git.tukaani.org/?p=3Dxz.git;a=3Dblob;f=3DNEWS;hb=3DHEAD#l94 Looks like it's specifically only compression. >=20 > > As for compressing in parallel, it *might work* to pass it through our > > non-bootstrap tar for 'tar --sort=3Dname' and then pass it through xz > > -T(pick-a-num). >=20 > That could be helpful! --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --qMm9M+Fa2AknHoGS Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAlwOtQUACgkQQarn3Mo9 g1HAzA/6AtSY7TGrc1DjJyrZKjjvjRoPUL/j1875Zvp7gPYDkdI46TQPngS4+sxg KWSEO4LFExteeld29OBtYH/tRJYQxzQv76hKLQEjdRXo3ITeJyVU7hB7T62FLZTe bJyxLRDPTCjWNg+/hqsiUBERgN9tWkFryfHbfeBGJPcDgy81ONgZ6ZXjbPRtLscK gIwlig14BhEScTopspiyy02n18xUs80fv8vAiDzdibAPr7gvsHGdjU3VN8uAbeUj +LwHIUSBAn12hxtEcjzrbmlbRk8gDAXQ89z4h6LQloIfkfFXzsRs0enfZIil0I7o 8z9QJ31Vx/CRdPDJ3QpdQZvcWCUi/q3WcH0WLGDOuoC94mVCpEmrnvjrKOVwpZo9 mcKZTv97e40y0yJEJ7qrXpwKlZw/kvlc7WYhf+FGzd2z98Hha9DWvcH3OPZ7Quqa /UmOmYHgm3Mm4uRgM3zbySDCZd7TpbhmThJFB+k8gwX5V/gQwCQ9UaMdShzAmVwP J9AWZm/QNz7pHDyC+vsae1WsXWc2VsAOL5Nv3hP8zokmw73p/62RGMl4+IUZgELn KAJu3Cu01ioTMeOJGUPO2jfDNLopVBzQ+pgOMVCAyiV0UNcOu52Q8/FzMkpZcdaK v6Z2r3BByuPVGIsmKeZ1zXKx2WnJKPQ6PaBA8uzRAbBDEtP6kCM= =OBRF -----END PGP SIGNATURE----- --qMm9M+Fa2AknHoGS-- From debbugs-submit-bounces@debbugs.gnu.org Wed May 13 14:20:18 2020 Received: (at 33643) by debbugs.gnu.org; 13 May 2020 18:20:18 +0000 Received: from localhost ([127.0.0.1]:59515 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jYvzM-00041n-Te for submit@debbugs.gnu.org; Wed, 13 May 2020 14:20:18 -0400 Received: from mira.cbaines.net ([212.71.252.8]:57690) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jYvzK-00041d-TD for 33643@debbugs.gnu.org; Wed, 13 May 2020 14:20:15 -0400 Received: from localhost (unknown [46.237.173.210]) by mira.cbaines.net (Postfix) with ESMTPSA id B87A827BBE1; Wed, 13 May 2020 19:20:13 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 2156064e; Wed, 13 May 2020 18:20:11 +0000 (UTC) References: <20181206075615.4637-1-mail@cbaines.net> User-agent: mu4e 1.2.0; emacs 26.3 From: Christopher Baines To: 33643@debbugs.gnu.org Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. In-reply-to: <20181206075615.4637-1-mail@cbaines.net> Date: Wed, 13 May 2020 19:20:08 +0100 Message-ID: <87k11fofxj.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 33643 Cc: Efraim Flashner , Leo Famulari X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Christopher Baines writes: > It can take a little while to decompress some packages with large xz > compressed source tar files. xz includes support for parallelism, so enable > this using the parallel job count for the overall derivation. > > * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T option > to xz to enable it to work in parallel if appropriate. > --- > guix/build/gnu-build-system.scm | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm > index e5f3197b0..9d11e5b1e 100644 > --- a/guix/build/gnu-build-system.scm > +++ b/guix/build/gnu-build-system.scm > @@ -147,7 +147,7 @@ chance to be set." > locale (strerror (system-error-errno args))) > #t))) > > -(define* (unpack #:key source #:allow-other-keys) > +(define* (unpack #:key source parallel-build? #:allow-other-keys) > "Unpack SOURCE in the working directory, and change directory within the > source. When SOURCE is a directory, copy it in a sub-directory of the current > working directory." > @@ -161,6 +161,10 @@ working directory." > (copy-recursively source "." > #:keep-mtime? #t)) > (begin > + (when parallel-build? > + (setenv "XZ_OPT" > + (format #f "-T~d" (parallel-job-count)))) > + > (if (string-suffix? ".zip" source) > (invoke "unzip" source) > (invoke "tar" "xvf" source)) It's been a long long while, but now that core-updates has recently been merged, I'd like to try and take a look at this again. I think the consensus was that this will only help for xz compressed files where they have been compressed in parallel. I think it's still worth doing though, as some of the big xz files that need decompressing have been compressed in parallel, and this will speed up the builds when multiple cores are available. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl68OlhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XeIIg//Ttm908GW5504QVqslK2vaUE2acDsgqHjmlsv7ykIKFThXtUp54VKQ5JT wOZl8OV+0HCpQDxF+wTONjGwDjE9eAyFMuyFbF6JKcVu5GNSgd+SbZdwcGBQAKM3 JYY990M6JZiiPwuuXsonLYYv46I8Lb46Jp8Oc7dogOsXXVrF3UZFIL+9pBFQlBco D4gebtkMMTLvTXsm5/pzL+rpRWHudvuBKf9ogBsd+hsKOCFIOLPTF1A0WE3Twrnm VKZ+j7SCqGmKfVkJL+a3Df5ln+fKHX7Yav8XGiYkfSI7LXPg3sJhq1/CDMBAy6O5 7KAFptf/msw7atgc4hazB2Aah7E2o230Gdu86HDy9Dbom0sd6sq6s7F1ICGvcv9C xt/COIetlMXDhnsbvpFL/lmh3kxDPk78sFaHkUfd4fbT4+9nHIXVL03AtWqFOYgF OiChboA8Eh8+vnaMuXwbE9o1O84+E9/o4RuaH3OS6RlDDBp62FC/4bQBhvLagSEH BD1erdFPVzJKV0amSfHExGTrQ/PL9Cov9CgEmWrKE0pPCcZWXbLc0pa7SkXR+jyE xqe1wDQDK4xeOG16sEMurZLgNbSD1ZfRPOoI/jIRbKkrmJtKZqAbAEFi8y6R+Xow QHbq776aESqqV9/x10QHE0mb4sJaoXFTfMc4zrJw9HLwQvUUSMo= =+wNy -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed May 13 15:08:12 2020 Received: (at 33643) by debbugs.gnu.org; 13 May 2020 19:08:12 +0000 Received: from localhost ([127.0.0.1]:59564 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jYwjj-0005Dc-Sf for submit@debbugs.gnu.org; Wed, 13 May 2020 15:08:12 -0400 Received: from flashner.co.il ([178.62.234.194]:51382) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jYwji-0005DN-Ki for 33643@debbugs.gnu.org; Wed, 13 May 2020 15:08:11 -0400 Received: from localhost (unknown [188.120.128.132]) by flashner.co.il (Postfix) with ESMTPSA id AD11940032; Wed, 13 May 2020 19:08:04 +0000 (UTC) Date: Wed, 13 May 2020 22:07:21 +0300 From: Efraim Flashner To: Christopher Baines Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. Message-ID: <20200513190721.GH918@E5400> References: <20181206075615.4637-1-mail@cbaines.net> <87k11fofxj.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="tSiBuZsJmMXpnp7T" Content-Disposition: inline In-Reply-To: <87k11fofxj.fsf@cbaines.net> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33643 Cc: 33643@debbugs.gnu.org, Leo Famulari X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --tSiBuZsJmMXpnp7T Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, May 13, 2020 at 07:20:08PM +0100, Christopher Baines wrote: >=20 > Christopher Baines writes: >=20 > > It can take a little while to decompress some packages with large xz > > compressed source tar files. xz includes support for parallelism, so en= able > > this using the parallel job count for the overall derivation. > > > > * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T o= ption > > to xz to enable it to work in parallel if appropriate. > > --- > > guix/build/gnu-build-system.scm | 6 +++++- > > 1 file changed, 5 insertions(+), 1 deletion(-) > > > > diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-sys= tem.scm > > index e5f3197b0..9d11e5b1e 100644 > > --- a/guix/build/gnu-build-system.scm > > +++ b/guix/build/gnu-build-system.scm > > @@ -147,7 +147,7 @@ chance to be set." > > locale (strerror (system-error-errno args))) > > #t))) > > > > -(define* (unpack #:key source #:allow-other-keys) > > +(define* (unpack #:key source parallel-build? #:allow-other-keys) > > "Unpack SOURCE in the working directory, and change directory within= the > > source. When SOURCE is a directory, copy it in a sub-directory of the= current > > working directory." > > @@ -161,6 +161,10 @@ working directory." > > (copy-recursively source "." > > #:keep-mtime? #t)) > > (begin > > + (when parallel-build? > > + (setenv "XZ_OPT" > > + (format #f "-T~d" (parallel-job-count)))) > > + > > (if (string-suffix? ".zip" source) > > (invoke "unzip" source) > > (invoke "tar" "xvf" source)) >=20 > It's been a long long while, but now that core-updates has recently been > merged, I'd like to try and take a look at this again. >=20 > I think the consensus was that this will only help for xz compressed > files where they have been compressed in parallel. I think it's still > worth doing though, as some of the big xz files that need decompressing > have been compressed in parallel, and this will speed up the builds when > multiple cores are available. >=20 > Thanks, >=20 > Chris I thought the last time we looked into this we figured out that there was a mistake in release notes or something and that parallel decompression isn't actually supported. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --tSiBuZsJmMXpnp7T Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl68RWYACgkQQarn3Mo9 g1Gl6RAAoR1+UxXRaKzafGqbc6dT/zc1zHYdanoa3NWt5EEGiAiKBaEnU1RW1K16 4xYJYfzm4w+FChnp1+Buj6n85cdiGpZLst6mY4KV8qPgYJoZJLi7QRN+06AZ92Lo mgD0479QJhRe0iEwtRXk+pjhEt13xOaSv2TTY5L8cRKHT16GVQhzTmD+kgLBCEiS k3qxjP9hOfGFl5myNReHeFcCVb8jJpDl86MmNqvJTuja7ylEW0g92ujF9JfMs2af m+h7pEtB4GGeDSfAoGIUhLrr8hN6VmiRLcbladkabeZ3ySFdQ669qkxcefGQ11HR C5eu8Ac8b0gJ7CMjFzrXQeW76GvDrq9u+RhyhlJ3EQqa8rAkvOodUXtuh4WhFbfW CZN/qWtO4Vs8BQ4svuJ3oWLjqmu8HhunU7JYCyI09uVKL45pHjz14WBLOmUGvplr oOIvv6z4k/C3HFihrC6zt8DcjuXXpfr51VC3Z7PcleFx6JmduhSRAOfTsat6SP61 +vGtSl75m8zCOxpwgj4zmk0w9efiPaxFgTDHLtz7ZnZbNeReUvfiFcTJRNg8UMQC tH4Uj90eduNDQ50VDJrv6b5Fn+Wmu1tg1MwzcPPuyMVCO4n/0C+cO3qya7FpLZ1m orTniX4rFGc4AMotCPQNWLgjJzgpjMS3UjgWnnDZg11TyNm832A= =XTqp -----END PGP SIGNATURE----- --tSiBuZsJmMXpnp7T-- From debbugs-submit-bounces@debbugs.gnu.org Thu May 14 03:38:02 2020 Received: (at 33643-done) by debbugs.gnu.org; 14 May 2020 07:38:02 +0000 Received: from localhost ([127.0.0.1]:60115 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jZ8RN-0002iY-N5 for submit@debbugs.gnu.org; Thu, 14 May 2020 03:38:02 -0400 Received: from mira.cbaines.net ([212.71.252.8]:52130) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jZ8RM-0002iQ-5A for 33643-done@debbugs.gnu.org; Thu, 14 May 2020 03:38:00 -0400 Received: from localhost (unknown [46.237.173.210]) by mira.cbaines.net (Postfix) with ESMTPSA id 2A65927BBE1; Thu, 14 May 2020 08:37:59 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id ccc3dc77; Thu, 14 May 2020 07:37:57 +0000 (UTC) References: <20181206075615.4637-1-mail@cbaines.net> <87k11fofxj.fsf@cbaines.net> <20200513190721.GH918@E5400> User-agent: mu4e 1.2.0; emacs 26.3 From: Christopher Baines To: Efraim Flashner Subject: Re: [bug#33643] [PATCH] gnu-build-system: Enable xz to decompress in parallel. In-reply-to: <20200513190721.GH918@E5400> Date: Thu, 14 May 2020 08:37:56 +0100 Message-ID: <87ftc3nezv.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 33643-done Cc: 33643-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Efraim Flashner writes: > On Wed, May 13, 2020 at 07:20:08PM +0100, Christopher Baines wrote: >> >> Christopher Baines writes: >> >> > It can take a little while to decompress some packages with large xz >> > compressed source tar files. xz includes support for parallelism, so enable >> > this using the parallel job count for the overall derivation. >> > >> > * guix/build/gnu-build-system.scm (unpack): Set XZ_OPT to pass the -T option >> > to xz to enable it to work in parallel if appropriate. >> > --- >> > guix/build/gnu-build-system.scm | 6 +++++- >> > 1 file changed, 5 insertions(+), 1 deletion(-) >> > >> > diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm >> > index e5f3197b0..9d11e5b1e 100644 >> > --- a/guix/build/gnu-build-system.scm >> > +++ b/guix/build/gnu-build-system.scm >> > @@ -147,7 +147,7 @@ chance to be set." >> > locale (strerror (system-error-errno args))) >> > #t))) >> > >> > -(define* (unpack #:key source #:allow-other-keys) >> > +(define* (unpack #:key source parallel-build? #:allow-other-keys) >> > "Unpack SOURCE in the working directory, and change directory within the >> > source. When SOURCE is a directory, copy it in a sub-directory of the current >> > working directory." >> > @@ -161,6 +161,10 @@ working directory." >> > (copy-recursively source "." >> > #:keep-mtime? #t)) >> > (begin >> > + (when parallel-build? >> > + (setenv "XZ_OPT" >> > + (format #f "-T~d" (parallel-job-count)))) >> > + >> > (if (string-suffix? ".zip" source) >> > (invoke "unzip" source) >> > (invoke "tar" "xvf" source)) >> >> It's been a long long while, but now that core-updates has recently been >> merged, I'd like to try and take a look at this again. >> >> I think the consensus was that this will only help for xz compressed >> files where they have been compressed in parallel. I think it's still >> worth doing though, as some of the big xz files that need decompressing >> have been compressed in parallel, and this will speed up the builds when >> multiple cores are available. >> >> Thanks, >> >> Chris > > I thought the last time we looked into this we figured out that there > was a mistake in release notes or something and that parallel > decompression isn't actually supported. Hmm, I had a look to see if I could find some examples of where this would apply, but I couldn't find any xz archives that we use in Guix where it's been compressed in a way that allows multithreaded decompression... I'm pretty sure I had some examples before, but maybe somethings changed in the intervening year. Anyway, if I discover this again, I'll actually make a note of where it's applicable. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl689VRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XfLAw/+OaHGFflhFa8gpk0UfmZwBA7ZuZBuDEIycWyMsKnisG3suwOPZkDbTIZQ /gx5BCwZKXAEXkVCenRQURzDpTx6jzUsRGQtUo+7dO9lb1RPu+URaQYZqxtmkix5 hT0k7I1J0cI1EVYoKjf8V0pQoMKZ0fwEU6LlJqVdeVJkWJ2lO/YiY9ehjkaRfC8O twqLcxJt7E93EFzOZUKk/fSFO4U6CO1y5HfyFHOsCJpZ6Wv1KAW06Zjxl4dKXAQL oGpZ/O8KH9958wRZneyXCpaHjRFCNABQIRkDceMiAS70uFJhZ8XVb/1b/O13kEBz TMiVRE+nicpg4UKDKRI/ljfwtwvhAVSBXJ0Q4x8oq3jdvOAa1Ax7hKrqUV2BaRBD hn517EaaZschp8QjsoudgHNzf1V+bwD1eKrDuLmrIoo2ObCCmPCsJoMqdnEXZ16S yAu1cdtIoQs45P+Q0cuVywPfoI+pjTD8jlKBo5VK3HhfYWB20YFupSyQbZCRLEwW AjmfFUknkjLxlRt5h9im11GrRup2XEv0sI0A0BWm/Jo1ebSwxywgVh0a0GegcQhK gQb9z41b3hpP3yCW8QwisRlPR4Iy44sm8iRfPGpVfmgaccJfx9dVi9q+lMekt5cG bIEIDhQQo5ONW+Y38r7BhFK78TimI7yKvMrH6RxobnxZklkVblg= =eiqL -----END PGP SIGNATURE----- --=-=-=-- From unknown Fri Jun 20 07:22:31 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 11 Jun 2020 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator