From unknown Tue Jun 17 22:24:54 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#43516] [PATCH core-updates] packages: Enable multi-threaded xz compression when repacking source. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 19 Sep 2020 17:04:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 43516 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43516@debbugs.gnu.org Cc: Maxim Cournoyer X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.160053501220821 (code B ref -1); Sat, 19 Sep 2020 17:04:02 +0000 Received: (at submit) by debbugs.gnu.org; 19 Sep 2020 17:03:32 +0000 Received: from localhost ([127.0.0.1]:47967 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJgGq-0005Pl-5y for submit@debbugs.gnu.org; Sat, 19 Sep 2020 13:03:32 -0400 Received: from lists.gnu.org ([209.51.188.17]:50320) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJgGo-0005Pd-1A for submit@debbugs.gnu.org; Sat, 19 Sep 2020 13:03:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35740) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kJgGk-00029u-F5 for guix-patches@gnu.org; Sat, 19 Sep 2020 13:03:28 -0400 Received: from mail-qv1-xf32.google.com ([2607:f8b0:4864:20::f32]:46723) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kJgGi-0006wU-HY for guix-patches@gnu.org; Sat, 19 Sep 2020 13:03:26 -0400 Received: by mail-qv1-xf32.google.com with SMTP id di5so4902296qvb.13 for ; Sat, 19 Sep 2020 10:03:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=NcsRjvOuO1KQTPvHiHn1cy8NuXQNhh3gNlgwMMwplT4=; b=OCqe5CNd0BIJhIl2MezI1wArs4AmZxuzfBB5laEaKlDRVFcnYpFvssw3TEe+dIle9s ijD7RwFn7itK+/0kFRvtb6lTuOPuWGlRoTWFpY7P4lpV+fJ/Q8ljIi4yTk1TjLMZ0j+B kQzF5o48t4HscGKq+XhJQYGKN2lxZ5OCgynAu8hcVZFqEMPlU+HAJoZ3Pie9MnqJq+8W zGQ0YtDKPZ75ayRFQX8qkv1EXStu2VcgcAYxyVhph3udaSJWMwwOLek8y+xKuALJPeSB WtgQjBJy2EKgVa2UhzxkYRSeJmjuu7J9tqPv1FBd7/uKb6jBDukvoCqIPujH9Ap6uDy2 WKCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=NcsRjvOuO1KQTPvHiHn1cy8NuXQNhh3gNlgwMMwplT4=; b=sgQSzIwAR2fRjh6Q8BkKpZrzTe29Yhb5jEBlh0jfVzo3qI68YmtTj66jnErfkbweSA 2N8Au+2U5Ui7BzBE9Wzhoc+YjyMwl6Xj/uHaDRaJues83R1/Hl0JqVG/voMBLopHI6A4 tBw5nXf5ykBOMW3ZYxhN2GNzRztXn/TM3u+DQA+oyFqVWS/5u+jjasz8ktNWkiBLlDPu wAE6r2ZuSBbhnJN26v6dZsvWsHtxvPYykRZnHi5/L+Cp1O0zSVpEaVxChY1dAGYD1IDx sCG+oSXLz8Nf58F0G2GAg2E/G5nTg07zch4bTHCf7T8WsY/ymWUDOoflocEaezNBXDwM B3eA== X-Gm-Message-State: AOAM533S9a6tpUyCQ3LKhvDFjbyemOcrnvEQHP7ZpYup+f7vA4gwhRmK 4h9Y4dxmceLAENHfN4qmK8oO8GSVFOs= X-Google-Smtp-Source: ABdhPJxW81f6GpGRW1EfN60zV57ZM/2/ecY9Ho4NcDuvvSLI6i5dZzUuBtGd2RN64l3Kn03CTJJdsg== X-Received: by 2002:a0c:e991:: with SMTP id z17mr23025435qvn.39.1600535003437; Sat, 19 Sep 2020 10:03:23 -0700 (PDT) Received: from localhost.localdomain (dsl-159-128.b2b2c.ca. [66.158.159.128]) by smtp.gmail.com with ESMTPSA id g45sm4986008qtb.60.2020.09.19.10.03.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Sep 2020 10:03:23 -0700 (PDT) From: Maxim Cournoyer Date: Sat, 19 Sep 2020 13:03:57 -0400 Message-Id: <20200919170357.13583-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::f32; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qv1-xf32.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) The xz compression is slow; using multiple threads/cores yields a linear performance improvement. * guix/packages.scm (patch-and-repack): Ensure xz is invoked with --threads=N by setting the XZ_DEFAULTS environment variable. --- guix/packages.scm | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/guix/packages.scm b/guix/packages.scm index 6598bd3149..678007a807 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2016 Alex Kost ;;; Copyright © 2017, 2019, 2020 Efraim Flashner ;;; Copyright © 2019 Marius Bakke +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -693,6 +694,11 @@ specifies modules in scope when evaluating SNIPPET." (setenv "PATH" (string-append #+xz "/bin" ":" #+decomp "/bin")) + ;; Enable multi-threaded compression for xz. + (setenv "XZ_DEFAULTS" (string-append "--threads=" + (number->string + (parallel-job-count)))) + ;; SOURCE may be either a directory or a tarball. (if (file-is-directory? #+source) (let* ((store (%store-directory)) -- 2.28.0 From unknown Tue Jun 17 22:24:54 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#43516] [PATCH core-updates] packages: Enable multi-threaded xz compression when repacking source. Resent-From: Leo Famulari Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 19 Sep 2020 19:39:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43516 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 43516@debbugs.gnu.org Received: via spool by 43516-submit@debbugs.gnu.org id=B43516.160054429727493 (code B ref 43516); Sat, 19 Sep 2020 19:39:01 +0000 Received: (at 43516) by debbugs.gnu.org; 19 Sep 2020 19:38:17 +0000 Received: from localhost ([127.0.0.1]:48160 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJigb-00079N-35 for submit@debbugs.gnu.org; Sat, 19 Sep 2020 15:38:17 -0400 Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:37977) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJigY-000798-Pf for 43516@debbugs.gnu.org; Sat, 19 Sep 2020 15:38:16 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id C45AA7DB; Sat, 19 Sep 2020 15:38:08 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Sat, 19 Sep 2020 15:38:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=mesmtp; bh=RFjJjPMmwVPO6i2yXdLF5T5d oOsZ1wX9uv1hjvztc4Y=; b=eL7MZSgk4k6do5haIhv0U9jjFrE1Lboyuo7RDtG0 oIP6AULqzMo5ca4jVupfrOGSLKiQ3tCouEzFbveGgAblmCOdWOL8MomBXJJpKecP s4kTExVcigiu5YGEjb6Fsg7DqbGogXw3QD/ox4SdMbLzdIqY8i9FGm/dSDF9AVmg g7o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=RFjJjP MmwVPO6i2yXdLF5T5doOsZ1wX9uv1hjvztc4Y=; b=bxihV8Rb+Dn505Sb1Zmjt2 R/0XfWh5IXomYveMuPJot9QZoOH4UYZxFHhgIjMjQ6CcuSGVhjqIBb5Ig3bP/oiZ YYTkFpVjtspuDQZcPiJsmRMISwngdiUHzJShC4nD8zvS6xN0CGraraWDjkDsRzGu bjF+RebVj1AnTaIRB/fmjAJ18f7bgIWBX8dBGo3R3mW8L0VeqjQwob3M5JQ5fWSf RTZD/91ndA7mok5LUtdPYKKwPxDLwBPWX5IEXGmwsc2WB982mZXNV+resLx6mti5 Vn9+M+8A81qX2xCkUzzA4fLR2+q8MCUrjVhRnEt1tdjjpUTjliVc2jIkGLlwLrZA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrtdekgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefnvghoucfh rghmuhhlrghrihcuoehlvghosehfrghmuhhlrghrihdrnhgrmhgvqeenucggtffrrghtth gvrhhnpedvvddugefffeeitddthfefvdeuhffgkeeikeegkeevteeghfeftefggeeuudff ieenucffohhmrghinhepghhnuhdrohhrghenucfkphepjeefrddugedurdduvdejrdduge einecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomheplhgv ohesfhgrmhhulhgrrhhirdhnrghmvg X-ME-Proxy: Received: from localhost (c-73-141-127-146.hsd1.pa.comcast.net [73.141.127.146]) by mail.messagingengine.com (Postfix) with ESMTPA id A46CF3064674; Sat, 19 Sep 2020 15:38:07 -0400 (EDT) Date: Sat, 19 Sep 2020 15:38:05 -0400 From: Leo Famulari Message-ID: <20200919193805.GA31344@jasmine.lan> References: <20200919170357.13583-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200919170357.13583-1-maxim.cournoyer@gmail.com> X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) On Sat, Sep 19, 2020 at 01:03:57PM -0400, Maxim Cournoyer wrote: > The xz compression is slow; using multiple threads/cores yields a linear > performance improvement. > > * guix/packages.scm (patch-and-repack): Ensure xz is invoked with --threads=N > by setting the XZ_DEFAULTS environment variable. We tried this previous but reverted it because the archives were not bit-reproducible: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=3e95125e9bd0676d4a9add9105217ad3eaef3ff0 It's really a shame... it would be nice to reduce the time used for XZ compression. But the bandwidth used to move the results is even more expensive in terms of time and money, since most people should get substitutes. From unknown Tue Jun 17 22:24:54 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#43516] Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 22 Sep 2020 02:00:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43516 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43516@debbugs.gnu.org Cc: Leo Famulari Received: via spool by 43516-submit@debbugs.gnu.org id=B43516.16007399723687 (code B ref 43516); Tue, 22 Sep 2020 02:00:01 +0000 Received: (at 43516) by debbugs.gnu.org; 22 Sep 2020 01:59:32 +0000 Received: from localhost ([127.0.0.1]:56785 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKXae-0000xO-D9 for submit@debbugs.gnu.org; Mon, 21 Sep 2020 21:59:32 -0400 Received: from mail-qk1-f195.google.com ([209.85.222.195]:44642) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKXac-0000xC-26 for 43516@debbugs.gnu.org; Mon, 21 Sep 2020 21:59:31 -0400 Received: by mail-qk1-f195.google.com with SMTP id n133so17491894qkn.11 for <43516@debbugs.gnu.org>; Mon, 21 Sep 2020 18:59:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=50ctoBx7OLWNg+r7bPEbPojOoDEq8RmjK0qy1xiW7pM=; b=HLn8rok8kcsbD94dFz1m5oc/WPgErQpJx1qBofIHWoi9hrK7VZwZ586t4QposSO6Q1 CxTstqtxMAHogKJfy3g4gmHdmr8kDU34C4MRMWvzliVZyg3N3ZBJ6qL+/dxTSKDYxCvv DIU813131XORdy3Q5zfaPbnAEA32ZZEG2kigguo+TkLznCo0qZ6kDjpPFXr7fvOgT+EW CPop5uZEAjI0xKw0VPS2E0bRt4PRBLsbcG3jAxdMbVC5eAblDWoxotDde672WD2orR5p QRNIJBRqO/HV3qVdmhJywe+wqHYxR+GOTUu09sr1/1IOUmZgRcpEx6e11DR717YnhYlX so0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=50ctoBx7OLWNg+r7bPEbPojOoDEq8RmjK0qy1xiW7pM=; b=fbTn4cphZPFNZtlf3U0xpM7V+CUZKwJ0GYC7Kgdftxf8sk4myj5JN6c+USE2X6GIIX /rGxq7j0bkDk+Se8/ck4Lmq5Fjbb3l8/lhox4HADN7YIp7wIBpVyn3Rm4jWUYVxaSImE INd2QXzwpiJyYMu7g6jCselt6V0FjijJJsWZZYZRhlb17qVLmIzPslAnNpaS1ea5iAy+ o6NWYwqr0NaON53QkT43hkptoBRcUxO7Yt0yKj88jKehtkQG8nL+VFmmuKC4xYSrEZwa mdE+1rFTEhfm7ku4xflpTuIs64pfPP/1xEnz/kxabEkZsguh/l9GjlAJrAdWkkDnIthz N+mw== X-Gm-Message-State: AOAM533Yfg/8ZJ6NAWWCdDptXSoDhzhJRkNgGOMQHMt/h9W8P7Mt4eHm mQcrbSsIgha8MtosRFsroOfy18oEbyA= X-Google-Smtp-Source: ABdhPJwH2dRbMCX+MQ8DaFzBL7WdOk5duZQHph5wiNCjnoagkRLSWXKtez9MDr3NQLGbJoW9YKwTWQ== X-Received: by 2002:a05:620a:1485:: with SMTP id w5mr2712403qkj.124.1600739964197; Mon, 21 Sep 2020 18:59:24 -0700 (PDT) Received: from localhost.localdomain (dsl-10-131-6.b2b2c.ca. [72.10.131.6]) by smtp.gmail.com with ESMTPSA id x26sm11113945qtr.78.2020.09.21.18.59.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2020 18:59:23 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Sep 2020 22:00:02 -0400 Message-Id: <20200922020003.6954-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200919193805.GA31344@jasmine.lan> References: <20200919193805.GA31344@jasmine.lan> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi Leo! > On Sat, Sep 19, 2020 at 01:03:57PM -0400, Maxim Cournoyer wrote: > > The xz compression is slow; using multiple threads/cores yields a linear > > performance improvement. > > > > * guix/packages.scm [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (maxim.cournoyer[at]gmail.com) -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [209.85.222.195 listed in list.dnswl.org] -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [209.85.222.195 listed in wl.mailspike.net] 2.0 BLANK_SUBJECT Subject is present but empty X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) Hi Leo! > On Sat, Sep 19, 2020 at 01:03:57PM -0400, Maxim Cournoyer wrote: > > The xz compression is slow; using multiple threads/cores yields a linear > > performance improvement. > > > > * guix/packages.scm (patch-and-repack): Ensure xz is invoked with --threads=N > > by setting the XZ_DEFAULTS environment variable. > We tried this previous but reverted it because the archives were not > bit-reproducible: > https://git.savannah.gnu.org/cgit/guix.git/commit/?id=3e95125e9bd0676d4a9add9105217ad3eaef3ff0 Thanks for bringing this to my attention! I've studied what others have done about it, and found a solution that seems to work well on the OpenEmbedded mailing list [0]. Debian uses something similar in their dpkg. The important point is that xz will produce reproducible results as long as it operates in either the single thread mode OR the multi-thread mode (we can't go from one mode to another reproducibly). So the following v2 patch ensures we always use --threads=2 at a minimum, forcing the xz code path into multi-thread operation. The --memlimit=50% argument limits the RAM use of xz to at most half of the available memory, which allows xz to reduce the number of threads used to meet this requirement. I've rebuilt the world or core-updates to test this and got impressive results, such as when building the linux-libre source with 24 cores instead of 1: $ time guix build --source linux-libre --check With this change, on a 24 cores/32 GiB system: 24 cores used, 2.9 GiB max memory used, 36.76 s. On master (same machine): 1 core used, 95 MiB max memory used, 4 m 10 s. [0] https://patchwork.openembedded.org/patch/170475/ [1] https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574 > It's really a shame... it would be nice to reduce the time used for XZ > compression. Seems we can have our cake and eat it, too! Maxim From unknown Tue Jun 17 22:24:54 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#43516] [PATCH core-updates v2] packages: Enable multi-threaded xz compression when repacking source. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 22 Sep 2020 02:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43516 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43516@debbugs.gnu.org Cc: Maxim Cournoyer , Leo Famulari Received: via spool by 43516-submit@debbugs.gnu.org id=B43516.16007399813708 (code B ref 43516); Tue, 22 Sep 2020 02:00:02 +0000 Received: (at 43516) by debbugs.gnu.org; 22 Sep 2020 01:59:41 +0000 Received: from localhost ([127.0.0.1]:56788 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKXam-0000xk-MJ for submit@debbugs.gnu.org; Mon, 21 Sep 2020 21:59:41 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:41514) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKXak-0000xX-PE for 43516@debbugs.gnu.org; Mon, 21 Sep 2020 21:59:39 -0400 Received: by mail-qk1-f193.google.com with SMTP id g72so17513674qke.8 for <43516@debbugs.gnu.org>; Mon, 21 Sep 2020 18:59:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B0qDLOOTwEvsQmrN8iF0azTY+b7L+6nFdwpdi8n1TP8=; b=iZ9hSR9a2y6vYMvAfYrNBE66AyOtO/2A8BsfgeA73ITgs/3sUhPkhcIVd7/SxdpRUX tS/hZARZYPgt06mhGweltepJTHdz3YiPzY4TYzn+zzxyTpwlvbCrRAijqHnql7TjHGGS 3eoM4ryZUX0OZwkFitKQuir10TIYS95EW1PaVybFPXB7bQrW9l/ZUvSo912dG6t1saUA Lsi9WreApF+QuE0cZx0t1Qvskk/AoslpkjKJH3SEgBkxuzEaq34Ru6+4IB7p28HJyTeI MDC460QNN//EVsBdYEU+rukbi8t/Jys3OzG80H0KC2lUQSIOdM1fdS5quoSwgXAgYHqt JrYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B0qDLOOTwEvsQmrN8iF0azTY+b7L+6nFdwpdi8n1TP8=; b=A7pNUFApYRrx7/SsEQl1C1TLdEnyhB0TQJz/+vn/ubZ9V3Drs/Hx8heSJ5SQVqdQBd FjH5e+6/bDVkMjr2qapyrwPJQEIhqCGZBcLW0ffjVeDqDjkFkfPsd0FVSWSdLd/6sMWx auBs1RoTh7F/fdGsAqEbtE6+dV7Oe1ZmETXNOYf5tBaHMQJWdBbscRcRpz69Eq0gwg1E jYgTdg5LeM8isBJaWCpKSUOQZKgSEF12YI0yYnn0dKPxzSMiEzMmme+u1vSfSouk8w9Z GqHWjpt3Im9hTsCgkiEjTrlNI4legYw/800EJdte2HmvvhcXWxi6ivCtK/Jw0VToPzQ9 4InQ== X-Gm-Message-State: AOAM531kxwiXdvi0EftqJgPLYNLf3zcYBFZSPXPaCD8O7boB4CDpSva1 BesauFISNYITFVjhS+GEe0OVYxSbUmI= X-Google-Smtp-Source: ABdhPJz/5R5tqlRiqpIHNtkkgxKwGx2fUxsjkMQoc5GL89ILS0f6P8FIJRAnCG6nOpM/JpsyzhtKjQ== X-Received: by 2002:a05:620a:13f9:: with SMTP id h25mr2560546qkl.283.1600739972901; Mon, 21 Sep 2020 18:59:32 -0700 (PDT) Received: from localhost.localdomain (dsl-10-131-6.b2b2c.ca. [72.10.131.6]) by smtp.gmail.com with ESMTPSA id x26sm11113945qtr.78.2020.09.21.18.59.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2020 18:59:32 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Sep 2020 22:00:03 -0400 Message-Id: <20200922020003.6954-2-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200922020003.6954-1-maxim.cournoyer@gmail.com> References: <20200919193805.GA31344@jasmine.lan> <20200922020003.6954-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) The xz compression is slow; using multiple threads/cores yields a linear performance improvement. * guix/build/utils.scm (%xz-parallel-args): New procedure. * guix/packages.scm (patch-and-repack): Specify the required above xz arguments by setting the XZ_DEFAULTS environment variable. * guix/scripts/pack.scm (%compressors, bootstrap-xz): Modify the commands Gexps so they do not need to be quoted. This allows lazily evaluating the arguments on the builder's side. Specify the required xz arguments. (self-contained-tarball): Do not quote the compressor command value. (docker-image): Likewise. * guix/utils.scm (decompressed-port, compressed-port) (compressed-output-port): Specify the required above xz arguments. --- guix/build/utils.scm | 15 ++++++++++++++- guix/packages.scm | 3 +++ guix/scripts/pack.scm | 27 ++++++++++++++++++--------- guix/utils.scm | 10 ++++++---- 4 files changed, 41 insertions(+), 14 deletions(-) diff --git a/guix/build/utils.scm b/guix/build/utils.scm index e884c26a22..ff4241d088 100644 --- a/guix/build/utils.scm +++ b/guix/build/utils.scm @@ -112,7 +112,9 @@ make-desktop-entry-file - locale-category->string)) + locale-category->string + + %xz-parallel-args)) ;;; @@ -1479,6 +1481,17 @@ returned." LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_TIME))) + +;;; +;;; Others. +;;; + +(define (%xz-parallel-args) + "The xz arguments required to enable bit-reproducible, multi-threaded +compression." + (list "--memlimit=50%" + (format #f "--threads=~a" (max 2 (parallel-job-count))))) + ;;; Local Variables: ;;; eval: (put 'call-with-output-file/atomic 'scheme-indent-function 1) ;;; eval: (put 'call-with-ascii-input-file 'scheme-indent-function 1) diff --git a/guix/packages.scm b/guix/packages.scm index 6598bd3149..865cb81929 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2016 Alex Kost ;;; Copyright © 2017, 2019, 2020 Efraim Flashner ;;; Copyright © 2019 Marius Bakke +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -693,6 +694,8 @@ specifies modules in scope when evaluating SNIPPET." (setenv "PATH" (string-append #+xz "/bin" ":" #+decomp "/bin")) + (setenv "XZ_DEFAULTS" (string-join (%xz-parallel-args))) + ;; SOURCE may be either a directory or a tarball. (if (file-is-directory? #+source) (let* ((store (%store-directory)) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 379e6a3ac6..a0112162e3 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2018 Chris Marusich ;;; Copyright © 2018 Efraim Flashner ;;; Copyright © 2020 Tobias Geerinckx-Rice +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -25,6 +26,7 @@ #:use-module (guix scripts) #:use-module (guix ui) #:use-module (guix gexp) + #:use-module ((guix build utils) #:select (%xz-parallel-args)) #:use-module (guix utils) #:use-module (guix store) #:use-module ((guix status) #:select (with-status-verbosity)) @@ -70,29 +72,34 @@ compressor? (name compressor-name) ;string (e.g., "gzip") (extension compressor-extension) ;string (e.g., ".lz") - (command compressor-command)) ;gexp (e.g., #~("/gnu/store/…/gzip" "-9n")) + (command compressor-command)) ;gexp (e.g., #~(list "/gnu/store/…/gzip" + ; "-9n" )) (define %compressors ;; Available compression tools. (list (compressor "gzip" ".gz" - #~(#+(file-append gzip "/bin/gzip") "-9n")) + #~(list #+(file-append gzip "/bin/gzip") "-9n")) (compressor "lzip" ".lz" - #~(#+(file-append lzip "/bin/lzip") "-9")) + #~(list #+(file-append lzip "/bin/lzip") "-9")) (compressor "xz" ".xz" - #~(#+(file-append xz "/bin/xz") "-e")) + #~(append (list #+(file-append xz "/bin/xz") + "-e") + (%xz-parallel-args))) (compressor "bzip2" ".bz2" - #~(#+(file-append bzip2 "/bin/bzip2") "-9")) + #~(list #+(file-append bzip2 "/bin/bzip2") "-9")) (compressor "zstd" ".zst" ;; The default level 3 compresses better than gzip in a ;; fraction of the time, while the highest level 19 ;; (de)compresses more slowly and worse than xz. - #~(#+(file-append zstd "/bin/zstd") "-3")) + #~(list #+(file-append zstd "/bin/zstd") "-3")) (compressor "none" "" #f))) ;; This one is only for use in this module, so don't put it in %compressors. (define bootstrap-xz (compressor "bootstrap-xz" ".xz" - #~(#+(file-append %bootstrap-coreutils&co "/bin/xz") "-e"))) + #~(append (list #+(file-append %bootstrap-coreutils&co "/bin/xz") + "-e") + (%xz-parallel-args)))) (define (lookup-compressor name) "Return the compressor object called NAME. Error out if it could not be @@ -269,7 +276,7 @@ added to the pack." #+@(if (compressor-command compressor) #~("-I" (string-join - '#+(compressor-command compressor))) + #+(compressor-command compressor))) #~()) "--format=gnu" @@ -541,11 +548,13 @@ the image." ,@(source-module-closure `((guix docker) (guix build store-copy) + (guix build utils) ;for %xz-parallel-args (guix profiles) (guix search-paths)) #:select? not-config?)) #~(begin (use-modules (guix docker) (guix build store-copy) + (guix build utils) (guix profiles) (guix search-paths) (srfi srfi-1) (srfi srfi-19) (ice-9 match)) @@ -602,7 +611,7 @@ the image." #~(list (string-append #$profile "/" #$entry-point))) #:extra-files directives - #:compressor '#+(compressor-command compressor) + #:compressor #+(compressor-command compressor) #:creation-time (make-time time-utc 0 1)))))) (gexp->derivation (string-append name ".tar" diff --git a/guix/utils.scm b/guix/utils.scm index 7cc321205e..ba896623f4 100644 --- a/guix/utils.scm +++ b/guix/utils.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2017 Mathieu Othacehe ;;; Copyright © 2018, 2020 Marius Bakke ;;; Copyright © 2020 Efraim Flashner +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -37,7 +38,7 @@ #:use-module (guix memoization) #:use-module ((guix build utils) #:select (dump-port mkdir-p delete-file-recursively - call-with-temporary-output-file)) + call-with-temporary-output-file %xz-parallel-args)) #:use-module ((guix build syscalls) #:select (mkdtemp! fdatasync)) #:use-module (guix diagnostics) ;, &error-location, etc. #:use-module (ice-9 format) @@ -220,7 +221,7 @@ a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-dc") input)) - ('xz (filtered-port `(,%xz "-dc") input)) + ('xz (filtered-port `(,%xz "-dc" ,@(%xz-parallel-args)) input)) ('gzip (filtered-port `(,%gzip "-dc") input)) ('lzip (values (lzip-port 'make-lzip-input-port input) '())) @@ -232,7 +233,7 @@ a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-c") input)) - ('xz (filtered-port `(,%xz "-c") input)) + ('xz (filtered-port `(,%xz "-c" ,@(%xz-parallel-args)) input)) ('gzip (filtered-port `(,%gzip "-c") input)) ('lzip (values (lzip-port 'make-lzip-input-port/compressed input) '())) @@ -291,7 +292,8 @@ program--e.g., '(\"--fast\")." (match compression ((or #f 'none) (values output '())) ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output)) - ('xz (filtered-output-port `(,%xz "-c" ,@options) output)) + ('xz (filtered-output-port `(,%xz "-c" ,@(%xz-parallel-args) + ,@options) output)) ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output)) ('lzip (values (lzip-port 'make-lzip-output-port output) '())) -- 2.28.0 From unknown Tue Jun 17 22:24:54 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#43516] your mail Resent-From: Leo Famulari Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 22 Sep 2020 15:20:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43516 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 43516@debbugs.gnu.org Received: via spool by 43516-submit@debbugs.gnu.org id=B43516.160078797121097 (code B ref 43516); Tue, 22 Sep 2020 15:20:02 +0000 Received: (at 43516) by debbugs.gnu.org; 22 Sep 2020 15:19:31 +0000 Received: from localhost ([127.0.0.1]:32907 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKk4p-0005UC-9Z for submit@debbugs.gnu.org; Tue, 22 Sep 2020 11:19:31 -0400 Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:49291) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKk4n-0005Tz-Bk for 43516@debbugs.gnu.org; Tue, 22 Sep 2020 11:19:30 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 659B6F11; Tue, 22 Sep 2020 11:19:23 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Tue, 22 Sep 2020 11:19:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=famulari.name; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=mesmtp; bh=PKTus9bAmUEBZdgEsDSEIaQk EByNij5y2T8C7v5TWuE=; b=A2ooKx6u9J8Gbx24kCYoY+w37JpuuXkoyUvXPFA9 eM669mH/VcnQnVVJFCDuW7lFB06sgx7tRefLhgE2Y1QZLX0dGMc7WdwVEwZK/gpz 8GfHPasgQbF7q8b3UfweeOQGgry1rkvRR77jxK7yQWd0dlOGSdqZk+ENotSCk9M8 idw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=PKTus9 bAmUEBZdgEsDSEIaQkEByNij5y2T8C7v5TWuE=; b=FomL+Hq+cfTxlei3Aqc3q+ 6dZk4fwGlx8eZoPmSF4/dkOCvgAo7buB224+fIudMVxQbQvHlghfhK4rfD+aLBhv iEBzjpK+mTDo7Beg59FHy1XHiGt0odNx9MTDz76LwA+rlpNao6AxjgPHjZfbFPXY Yb4Min49mVoQSwJ0zOtMiUeglZLnRP64Z7VkkvdNA5Y54t1VcNW7tWiWYviAmJSm NIbU/+86A+cBpmAqRDtXVBeudY43r4pTaVsFSC4YMjOOvTccIA2YeNIXW6CvVR5j cJG0w/BWrj/LT0TBCIgLkeMyeCeL5Sbc2nJaoQ/z/FCetSda1HA9owpcNrKSF/Qw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrudeggdekjecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkfhggtggujgesthdtredttddtvdenucfhrhhomhepnfgvohcuhfgr mhhulhgrrhhiuceolhgvohesfhgrmhhulhgrrhhirdhnrghmvgeqnecuggftrfgrthhtvg hrnhepueekkedtffdvtddugeejgedtvefhueefiedvjeeitdeigedtveejvdejheffvefg necukfhppeeiledruddvuddrheekrddvfeefnecuvehluhhsthgvrhfuihiivgeptdenuc frrghrrghmpehmrghilhhfrhhomheplhgvohesfhgrmhhulhgrrhhirdhnrghmvg X-ME-Proxy: Received: from localhost (ool-45793ae9.dyn.optonline.net [69.121.58.233]) by mail.messagingengine.com (Postfix) with ESMTPA id A9B3E3280059; Tue, 22 Sep 2020 11:19:22 -0400 (EDT) Date: Tue, 22 Sep 2020 11:19:21 -0400 From: Leo Famulari Message-ID: <20200922151921.GA23722@jasmine.lan> References: <20200919193805.GA31344@jasmine.lan> <20200922020003.6954-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200922020003.6954-1-maxim.cournoyer@gmail.com> X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) On Mon, Sep 21, 2020 at 10:00:02PM -0400, Maxim Cournoyer wrote: > Seems we can have our cake and eat it, too! Amazing! I don't have time to check it myself but please proceed as you see fit. From unknown Tue Jun 17 22:24:54 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Maxim Cournoyer Subject: bug#43516: closed (Re: [bug#43516] your mail) Message-ID: References: <87o8lcrvlr.fsf@gmail.com> <20200919170357.13583-1-maxim.cournoyer@gmail.com> X-Gnu-PR-Message: they-closed 43516 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 43516@debbugs.gnu.org Date: Fri, 09 Oct 2020 02:18:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1602209882-7093-1" This is a multi-part message in MIME format... ------------=_1602209882-7093-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #43516: [PATCH core-updates] packages: Enable multi-threaded xz compression= when repacking source. which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 43516@debbugs.gnu.org. --=20 43516: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D43516 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1602209882-7093-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 43516-done) by debbugs.gnu.org; 9 Oct 2020 02:17:46 +0000 Received: from localhost ([127.0.0.1]:33618 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kQhyc-0001py-E9 for submit@debbugs.gnu.org; Thu, 08 Oct 2020 22:17:46 -0400 Received: from mail-qk1-f169.google.com ([209.85.222.169]:43352) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kQhyZ-0001pk-QD for 43516-done@debbugs.gnu.org; Thu, 08 Oct 2020 22:17:44 -0400 Received: by mail-qk1-f169.google.com with SMTP id c2so9139161qkf.10 for <43516-done@debbugs.gnu.org>; Thu, 08 Oct 2020 19:17:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=wGNnnmk78YKY6dfVqxdm8gVdzv/1nKs2lsA8rOS+fEY=; b=tm+pAnMfwbYnoS2xrcV8HWQSr5o4E+RDPF20d7GmrrKgBpinR3utEv+64PtA+sjyGD v7wjxGQGttw9hAYbrFzfeK7QCzSQARXHdX+hP3mX80HJVSpI0XSmCPXV6J0OieR58Iur RHZZ0A4au8oweeVXvZmKQFfimLmfxy1s1bRtvbw6IMIxFyOaMxtvkw/+8botx1vuOhWO lxhH/EK9tnS69VSxr1rN2tsFLaqyKL4taet/UzfdW1PEu4CKtxLVx4oxtzXjpG7FBHq8 bn9z8N0nHUvEQ0zg3l9GNF8pOtFoO1IpWx9cuzDWif1jAV6tIvYeUutfrwSfK3DO+8CJ 9+tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=wGNnnmk78YKY6dfVqxdm8gVdzv/1nKs2lsA8rOS+fEY=; b=HNnoVQhP5m+qD40RXw9w/XD6UXFnBRiYec0ovJM7jYImRcT7aTIy68nKJlygZAIM/A DV8vWy/+dkqB8YrabYvYFsDxiIe5Ynsznavc2zZ3z8lp8jpivooUPce6Qzifq3lhgk4v 7umvSLOwg+mLML42j0iMOIb1oknJ1FWmjX29M89x5OcdpalscwyXpCm1LUpnOAOvka1/ 05XRZv+c6UgiH92MKaZCpEG3fvvOVMEaSHUU29WBUFTYgR+Zi4717tYBK5nyMg9LP8TY cx73Id8kzRFmUs6EoHzXg+G+UIarASw1xU6Sj+GznYnUsir1Ts90HOfjqg47q5QBs6LO QyvQ== X-Gm-Message-State: AOAM531S+KBvwk3+vPnXz3HZtQyuLSM1Y+ZVt2ZkFv4iu2HcVN0bLcNI Sg8yTtQIcqPHv38mfXJus4PrW0vs5/zZvw== X-Google-Smtp-Source: ABdhPJwW6XBXHv4RbEZiAATOu5XB2rMlHS4nkkxLGU/Zhod6/K0vxV5T/nSx9mrN/lhKythQFRxNsw== X-Received: by 2002:a37:6c04:: with SMTP id h4mr540927qkc.100.1602209858055; Thu, 08 Oct 2020 19:17:38 -0700 (PDT) Received: from hurd (dsl-10-135-56.b2b2c.ca. [72.10.135.56]) by smtp.gmail.com with ESMTPSA id h199sm5207838qke.112.2020.10.08.19.17.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Oct 2020 19:17:37 -0700 (PDT) From: Maxim Cournoyer To: Leo Famulari Subject: Re: [bug#43516] your mail References: <20200919193805.GA31344@jasmine.lan> <20200922020003.6954-1-maxim.cournoyer@gmail.com> <20200922151921.GA23722@jasmine.lan> Date: Thu, 08 Oct 2020 22:17:36 -0400 In-Reply-To: <20200922151921.GA23722@jasmine.lan> (Leo Famulari's message of "Tue, 22 Sep 2020 11:19:21 -0400") Message-ID: <87o8lcrvlr.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 43516-done Cc: 43516-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Leo Famulari writes: > On Mon, Sep 21, 2020 at 10:00:02PM -0400, Maxim Cournoyer wrote: >> Seems we can have our cake and eat it, too! > > Amazing! I don't have time to check it myself but please proceed as you > see fit. Pushed with commit 5a0997ef7f to core-updates. Enjoy! Maxim ------------=_1602209882-7093-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 19 Sep 2020 17:03:32 +0000 Received: from localhost ([127.0.0.1]:47967 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJgGq-0005Pl-5y for submit@debbugs.gnu.org; Sat, 19 Sep 2020 13:03:32 -0400 Received: from lists.gnu.org ([209.51.188.17]:50320) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kJgGo-0005Pd-1A for submit@debbugs.gnu.org; Sat, 19 Sep 2020 13:03:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35740) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kJgGk-00029u-F5 for guix-patches@gnu.org; Sat, 19 Sep 2020 13:03:28 -0400 Received: from mail-qv1-xf32.google.com ([2607:f8b0:4864:20::f32]:46723) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kJgGi-0006wU-HY for guix-patches@gnu.org; Sat, 19 Sep 2020 13:03:26 -0400 Received: by mail-qv1-xf32.google.com with SMTP id di5so4902296qvb.13 for ; Sat, 19 Sep 2020 10:03:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=NcsRjvOuO1KQTPvHiHn1cy8NuXQNhh3gNlgwMMwplT4=; b=OCqe5CNd0BIJhIl2MezI1wArs4AmZxuzfBB5laEaKlDRVFcnYpFvssw3TEe+dIle9s ijD7RwFn7itK+/0kFRvtb6lTuOPuWGlRoTWFpY7P4lpV+fJ/Q8ljIi4yTk1TjLMZ0j+B kQzF5o48t4HscGKq+XhJQYGKN2lxZ5OCgynAu8hcVZFqEMPlU+HAJoZ3Pie9MnqJq+8W zGQ0YtDKPZ75ayRFQX8qkv1EXStu2VcgcAYxyVhph3udaSJWMwwOLek8y+xKuALJPeSB WtgQjBJy2EKgVa2UhzxkYRSeJmjuu7J9tqPv1FBd7/uKb6jBDukvoCqIPujH9Ap6uDy2 WKCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=NcsRjvOuO1KQTPvHiHn1cy8NuXQNhh3gNlgwMMwplT4=; b=sgQSzIwAR2fRjh6Q8BkKpZrzTe29Yhb5jEBlh0jfVzo3qI68YmtTj66jnErfkbweSA 2N8Au+2U5Ui7BzBE9Wzhoc+YjyMwl6Xj/uHaDRaJues83R1/Hl0JqVG/voMBLopHI6A4 tBw5nXf5ykBOMW3ZYxhN2GNzRztXn/TM3u+DQA+oyFqVWS/5u+jjasz8ktNWkiBLlDPu wAE6r2ZuSBbhnJN26v6dZsvWsHtxvPYykRZnHi5/L+Cp1O0zSVpEaVxChY1dAGYD1IDx sCG+oSXLz8Nf58F0G2GAg2E/G5nTg07zch4bTHCf7T8WsY/ymWUDOoflocEaezNBXDwM B3eA== X-Gm-Message-State: AOAM533S9a6tpUyCQ3LKhvDFjbyemOcrnvEQHP7ZpYup+f7vA4gwhRmK 4h9Y4dxmceLAENHfN4qmK8oO8GSVFOs= X-Google-Smtp-Source: ABdhPJxW81f6GpGRW1EfN60zV57ZM/2/ecY9Ho4NcDuvvSLI6i5dZzUuBtGd2RN64l3Kn03CTJJdsg== X-Received: by 2002:a0c:e991:: with SMTP id z17mr23025435qvn.39.1600535003437; Sat, 19 Sep 2020 10:03:23 -0700 (PDT) Received: from localhost.localdomain (dsl-159-128.b2b2c.ca. [66.158.159.128]) by smtp.gmail.com with ESMTPSA id g45sm4986008qtb.60.2020.09.19.10.03.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Sep 2020 10:03:23 -0700 (PDT) From: Maxim Cournoyer To: guix-patches@gnu.org Subject: [PATCH core-updates] packages: Enable multi-threaded xz compression when repacking source. Date: Sat, 19 Sep 2020 13:03:57 -0400 Message-Id: <20200919170357.13583-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::f32; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qv1-xf32.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: submit Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) The xz compression is slow; using multiple threads/cores yields a linear performance improvement. * guix/packages.scm (patch-and-repack): Ensure xz is invoked with --threads=N by setting the XZ_DEFAULTS environment variable. --- guix/packages.scm | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/guix/packages.scm b/guix/packages.scm index 6598bd3149..678007a807 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2016 Alex Kost ;;; Copyright © 2017, 2019, 2020 Efraim Flashner ;;; Copyright © 2019 Marius Bakke +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -693,6 +694,11 @@ specifies modules in scope when evaluating SNIPPET." (setenv "PATH" (string-append #+xz "/bin" ":" #+decomp "/bin")) + ;; Enable multi-threaded compression for xz. + (setenv "XZ_DEFAULTS" (string-append "--threads=" + (number->string + (parallel-job-count)))) + ;; SOURCE may be either a directory or a tarball. (if (file-is-directory? #+source) (let* ((store (%store-directory)) -- 2.28.0 ------------=_1602209882-7093-1--