From unknown Mon Jun 16 23:45:34 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#47137 <47137@debbugs.gnu.org> To: bug#47137 <47137@debbugs.gnu.org> Subject: Status: [PATCH] Adaptive substitute decompression selection Reply-To: bug#47137 <47137@debbugs.gnu.org> Date: Tue, 17 Jun 2025 06:45:34 +0000 retitle 47137 [PATCH] Adaptive substitute decompression selection reassign 47137 guix-patches submitter 47137 Ludovic Court=C3=A8s severity 47137 normal tag 47137 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 14 10:38:54 2021 Received: (at submit) by debbugs.gnu.org; 14 Mar 2021 14:38:54 +0000 Received: from localhost ([127.0.0.1]:34066 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLRtO-0006jy-05 for submit@debbugs.gnu.org; Sun, 14 Mar 2021 10:38:54 -0400 Received: from lists.gnu.org ([209.51.188.17]:50714) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLRtL-0006jp-Vq for submit@debbugs.gnu.org; Sun, 14 Mar 2021 10:38:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49178) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lLRtK-0002OU-To for guix-patches@gnu.org; Sun, 14 Mar 2021 10:38:50 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52429) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLRtK-0002kp-4p for guix-patches@gnu.org; Sun, 14 Mar 2021 10:38:50 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39814 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLRtJ-0001se-GJ for guix-patches@gnu.org; Sun, 14 Mar 2021 10:38:49 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH] Adaptive substitute decompression selection X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 24 =?utf-8?Q?Vent=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 14 Mar 2021 15:38:47 +0100 Message-ID: <87wnu9ls08.fsf@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi! The patch below is a followup to the thread started in December: https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html It provides a na=C3=AFve but apparently good enough way for =E2=80=98guix s= ubstitute=E2=80=99 to choose the compression method that yields the best speed given the CPU and current networking conditions. On a recent x86_64 laptop with fast networking, using ci.guix.gnu.org, the effect so far is to choose gzip substitutes, which indeed provides slightly faster substitute installation. When ci.guix provides zstd substitutes, the speedup will be higher. I have yet to check that it sticks to lzip when bandwidth is low. Thoughts? Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=0001-substitute-Choose-compression-method-based-on-past-C.patch Content-Transfer-Encoding: quoted-printable Content-Description: the patch >From 3f95a1ac04c5e178a7fedfc2d03c07bcb1075ead Mon Sep 17 00:00:00 2001 From: =3D?UTF-8?q?Ludovic=3D20Court=3DC3=3DA8s?=3D Date: Sun, 14 Mar 2021 15:05:30 +0100 Subject: [PATCH] substitute: Choose compression method based on past CPU usage. This stems from the observation that substitute download can be CPU-bound when high-speed networks are in use: https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html * guix/narinfo.scm (decompresses-faster?): New procedure. (narinfo-best-uri): Add #:fast-decompression?. * guix/scripts/substitute.scm (%prefer-fast-decompression?): New variable. (call-with-cpu-usage-monitoring): New procedure. (with-cpu-usage-monitoring): New macro. (display-narinfo-data, process-substitution): Pass #:fast-decompression? to 'narinfo-best-uri'. (process-substitution): Wrap 'restore-file' call in 'with-cpu-usage-monitoring'. Set '%prefer-fast-decompression?'. --- guix/narinfo.scm | 27 ++++++++++++++++--- guix/scripts/substitute.scm | 53 ++++++++++++++++++++++++++++++++----- 2 files changed, 69 insertions(+), 11 deletions(-) diff --git a/guix/narinfo.scm b/guix/narinfo.scm index 2d06124017..72e0f75fda 100644 --- a/guix/narinfo.scm +++ b/guix/narinfo.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright =C2=A9 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovi= c Court=C3=A8s +;;; Copyright =C2=A9 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 = Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2014 Nikita Karetnikov ;;; Copyright =C2=A9 2018 Kyle Meyer ;;; @@ -297,9 +297,21 @@ this is a rough approximation." (_ (or (string=3D? compression2 "none") (string=3D? compression2 "gzip"))))) =20 -(define (narinfo-best-uri narinfo) +(define (decompresses-faster? compression1 compression2) + "Return true if COMPRESSION1 generally has a higher decompression throug= hput +than COMPRESSION2." + (match compression1 + ("none" #t) + ("zstd" #t) + ("gzip" (string=3D? compression2 "lzip")) + (_ #f))) + +(define* (narinfo-best-uri narinfo #:key fast-decompression?) "Select the \"best\" URI to download NARINFO's nar, and return three val= ues: -the URI, its compression method (a string), and the compressed file size." +the URI, its compression method (a string), and the compressed file size. +When FAST-DECOMPRESSION? is true, prefer substitutes with faster +decompression (typically zstd) rather than substitutes with a higher +compression ratio (typically lzip)." (define choices (filter (match-lambda ((uri compression file-size) @@ -321,6 +333,13 @@ the URI, its compression method (a string), and the co= mpressed file size." (compresses-better? compression1 compression2)))) (_ #f))) ;we can't tell =20 - (match (sort choices file-size) (narinfo-references narinfo)) =20 - (let-values (((uri compression file-size) (narinfo-best-uri narinfo))) + (let-values (((uri compression file-size) + (narinfo-best-uri narinfo + #:fast-decompression? + %prefer-fast-decompression?))) (format #t "~a\n~a\n" (or file-size 0) (or (narinfo-size narinfo) 0)))) @@ -438,7 +462,9 @@ the current output port." store-item)) =20 (let-values (((uri compression file-size) - (narinfo-best-uri narinfo))) + (narinfo-best-uri narinfo + #:fast-decompression? + %prefer-fast-decompression?))) (unless print-build-trace? (format (current-error-port) (G_ "Downloading ~a...~%") (uri->string uri))) @@ -476,11 +502,24 @@ the current output port." ((hashed get-hash) (open-hash-input-port algorithm input))) ;; Unpack the Nar at INPUT into DESTINATION. - (restore-file hashed destination - #:dump-file (if (and destination-in-store? - deduplicate?) - dump-file/deduplicate* - dump-file)) + (define cpu-usage + (with-cpu-usage-monitoring + (restore-file hashed destination + #:dump-file (if (and destination-in-store? + deduplicate?) + dump-file/deduplicate* + dump-file)))) + + ;; Create a hysteresis: depending on CPU usage, favor compression + ;; methods with faster decompression (like ztsd) or methods with bet= ter + ;; compression ratios (like lzip). This stems from the observation = that + ;; substitution can be CPU-bound when high-speed networks are used: + ;; . + (when (> cpu-usage .8) + (set! %prefer-fast-decompression? #t)) + (when (< cpu-usage .4) + (set! %prefer-fast-decompression? #f)) + (close-port hashed) (close-port input) =20 --=20 2.30.2 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 21 18:46:31 2021 Received: (at 47137-done) by debbugs.gnu.org; 21 Mar 2021 22:46:31 +0000 Received: from localhost ([127.0.0.1]:55620 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lO6q7-000404-ID for submit@debbugs.gnu.org; Sun, 21 Mar 2021 18:46:31 -0400 Received: from eggs.gnu.org ([209.51.188.92]:34318) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lO6q6-0003zr-7V for 47137-done@debbugs.gnu.org; Sun, 21 Mar 2021 18:46:30 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:45379) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lO6pz-0005NZ-RS for 47137-done@debbugs.gnu.org; Sun, 21 Mar 2021 18:46:24 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39362 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lO6pn-00072O-A1 for 47137-done@debbugs.gnu.org; Sun, 21 Mar 2021 18:46:19 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 47137-done@debbugs.gnu.org Subject: Re: bug#47137: [PATCH] Adaptive substitute decompression selection References: <87wnu9ls08.fsf@inria.fr> Date: Sun, 21 Mar 2021 23:46:08 +0100 In-Reply-To: <87wnu9ls08.fsf@inria.fr> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?= =?utf-8?Q?'s?= message of "Sun, 14 Mar 2021 15:38:47 +0100") Message-ID: <874kh415xr.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47137-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi! Ludovic Court=C3=A8s skribis: > The patch below is a followup to the thread started in December: > > https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html > > It provides a na=C3=AFve but apparently good enough way for =E2=80=98guix= substitute=E2=80=99 > to choose the compression method that yields the best speed given the > CPU and current networking conditions. > > On a recent x86_64 laptop with fast networking, using ci.guix.gnu.org, > the effect so far is to choose gzip substitutes, which indeed provides > slightly faster substitute installation. When ci.guix provides zstd > substitutes, the speedup will be higher. > > I have yet to check that it sticks to lzip when bandwidth is low. I did that, using =E2=80=98tc=E2=80=99, and it works as expected, staying o= n lzip. Pushed as 9da5ec7099b992a8969a17627548cd341c01bd90 with two minor tweaks: lowered the low hysteresis threshold, and added a comment on how to use =E2=80=98tc=E2=80=99 to test the behavior on =E2=80=9Cslow=E2=80=9D = networks. Rather than running =E2=80=98guix build=E2=80=99 followed by =E2=80=98guix = gc=E2=80=99, I found that manually invoking =E2=80=98guix substitute=E2=80=99 was nicer (long line ah= ead!): ( echo substitute /gnu/store/svv4826f8zfj8grl2qa17xnxk3acsppc-elixir-1.11= .4 /tmp/t1; echo substitute /gnu/store/d9dk53m7pwx1dc1p97zm0q323gpk70f9-poe= zio-0.13.1 /tmp/t4; echo substitute /gnu/store/mra8i18y9gjavhmdlkbb10m4miin= irgz-ocaml-4.11.1 /tmp/t2; echo substitute /gnu/store/ay2j5mp20j9vbhibcwp5l= mmcmhqkdnga-vim-full-8.2.2632 /tmp/t3; echo substitute /gnu/store/svv4826f8= zfj8grl2qa17xnxk3acsppc-elixir-1.11.4 /tmp/t5; echo substitute /gnu/store/a= y2j5mp20j9vbhibcwp5lmmcmhqkdnga-vim-full-8.2.2632 /tmp/t6) | GUIX_ALLOW_UNA= UTHENTICATED_SUBSTITUTES=3Dyes ./pre-inst-env guix substitute --substitute = 4>&2 Note that this change won=E2=80=99t take effect until we update the =E2=80= =98guix=E2=80=99 package. Ludo=E2=80=99. From unknown Mon Jun 16 23:45:34 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 19 Apr 2021 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator