From debbugs-submit-bounces@debbugs.gnu.org Mon May 30 04:43:54 2022 Received: (at submit) by debbugs.gnu.org; 30 May 2022 08:43:54 +0000 Received: from localhost ([127.0.0.1]:42698 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvb0E-0000vf-4i for submit@debbugs.gnu.org; Mon, 30 May 2022 04:43:54 -0400 Received: from lists.gnu.org ([209.51.188.17]:52280) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvb09-0000vV-K8 for submit@debbugs.gnu.org; Mon, 30 May 2022 04:43:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48462) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvb09-0000mo-Ci for bug-guix@gnu.org; Mon, 30 May 2022 04:43:49 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:33520) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvb08-0002KY-Ih for bug-guix@gnu.org; Mon, 30 May 2022 04:43:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=jEeq2IgyuhlSL6RWD794ydlSYbLNcQTY2rf8zffEoDo=; b=F2MBNz9mlZbOPY BrkDsWVlubQzAMK8tks8DW5FiKf+Wekjuotm1963n4+AF2TFSx1yCn4z0fVRb1rJqP6+I92JsiYCM blp9U4SqVkgYQloHH1u7WlV7l0uxnShxPuXAgpCCAKkGGKShR0SEh0rgyxsSLyVBAkEbAKUJdFhAW vv6+lf0mNBEcu3SmbAWXe0324opi9fJSn56alh+Pi+cj8zEGhLHdcC0XJxsp8HnN/DVEwbs3o5Wj/ uuqUW8mt22gzLFEpc2GC0AQ0V1a7NtIFJ/afRT4YOwpkKya3BEMFZkyiasiRXfDNjlDScDQutxjgm Q9HT6eQV+eSKknw5heFw==; Received: from [193.50.110.143] (port=37288 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvb08-0003Su-1h for bug-guix@gnu.org; Mon, 30 May 2022 04:43:48 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: bug-guix@gnu.org Subject: Low cache usage in the presence of grafts X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Primidi 11 Prairial an 230 de la =?utf-8?Q?R=C3=A9vo?= =?utf-8?Q?lution=2C?= jour de la Fraise X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 30 May 2022 10:43:46 +0200 Message-ID: <87y1yj5s1p.fsf@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.8 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.8 (--) The use of per-session caches in the presence of grafts is low=E2=80=94specifically when =E2=80=9Cbuild handlers=E2=80=9D are invoked. This is illustrated by the example below: =E2=80=A2 the first run invokes the build handler of =E2=80=98map/accumul= ate-builds=E2=80=99 to download polkit (which has a =E2=80=98replacement=E2=80=99) and then= proceeds (the end result was already in store), taking no less than 6m of CPU time; =E2=80=A2 the second run doesn=E2=80=99t have to hop through the build ha= ndler since everything is already in store and its takes 24s of CPU time. --8<---------------cut here---------------start------------->8--- $ guix gc -D $(guix build polkit --no-grafts) finding garbage collector roots... [0 MiB] deleting '/gnu/store/8vsfk8312m5p461wl4dq5lxjagfbwfhz-polkit-0.121' deleting `/gnu/store/trash' deleting unused links... note: currently hard linking saves 72385.53 MiB $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./pre-= inst-env guix system vm gnu/system/examples/desktop.tmpl=20 0.2 MB will be downloaded polkit-0.121 198KiB = 1.4MiB/s 00:00 [##################] 100.0% /gnu/store/sk1571rnh5jl2ilp0v1k8bqwsdrngwxy-run-vm.sh Reference Cache: fresh caches: 8 lookups: 77581 hits: 52890 (68.2%) cache size: 838 entries Derivation graft cache: fresh caches: 39 lookups: 704635 hits: 627819 (89.1%) cache size: 2946 entries Remote procedure call summary: 33071 RPCs built-in-builders ... 1 query-substitutable-path-infos ... 2 build-things ... 2 add-to-store/tree ... 56 valid-path? ... 59 add-to-store ... 373 add-text-to-store ... 7887 query-references ... 24691 real 5m6.727s user 6m59.438s sys 0m1.633s $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./pre-= inst-env guix system vm gnu/system/examples/desktop.tmpl=20 /gnu/store/sk1571rnh5jl2ilp0v1k8bqwsdrngwxy-run-vm.sh Reference Cache: fresh caches: 1 lookups: 2988 hits: 2150 (72.0%) cache size: 838 entries Derivation graft cache: fresh caches: 5 lookups: 28323 hits: 25376 (89.6%) cache size: 2946 entries Remote procedure call summary: 9159 RPCs query-substitutable-path-infos ... 1 build-things ... 1 built-in-builders ... 1 valid-path? ... 2 add-to-store/tree ... 56 add-to-store ... 373 query-references ... 838 add-text-to-store ... 7887 real 0m17.304s user 0m24.620s sys 0m0.273s $ git log |head commit acb08304c862d65679086a9f160240bfd598289d Author: Wiktor Zelazny Date: Wed May 18 16:20:30 2022 +0000 gnu: Add python-gpy. =20=20=20=20 * gnu/packages/python-science.scm (python-gpy): New variable. =20=20=20=20 Signed-off-by: Ludovic Court=C3=A8s --8<---------------cut here---------------end--------------->8--- >From the first run we can see that the reference cache (used by =E2=80=98references/cached=E2=80=99, itself called form (guix grafts)) is a= lmost unused since we end up making 24K =E2=80=98query-references=E2=80=99 RPCs for only= 838 entries in that cache, meaning the same work is done ~29 times. My hypothesis is that this is a regression introduced by the switch to =E2=80=9Cfunctional caches=E2=80=9D in =E2=80=98core-updates=E2=80=99 back = in 9e5812ac59b01ff011ec0c5b0f437dfe85d6fcc7. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon May 30 05:09:40 2022 Received: (at control) by debbugs.gnu.org; 30 May 2022 09:09:40 +0000 Received: from localhost ([127.0.0.1]:42745 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvbPA-0001c6-Di for submit@debbugs.gnu.org; Mon, 30 May 2022 05:09:40 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36944) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvbP6-0001bp-AB for control@debbugs.gnu.org; Mon, 30 May 2022 05:09:39 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:33736) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvbOz-00078W-MM for control@debbugs.gnu.org; Mon, 30 May 2022 05:09:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to: references; bh=uS8jondEHotb0HUlz97AQGBLqlaYI7918F0T8RSFMAo=; b=dnvXg7WUZ86pkB EGECSE3SrBDUsDSAl/TKlQtV5+kRwKnL1+lXqeIw2EepMSJZi14abxOlyS+VXuVJydkkLq6jd4aIe 7CM3CPM5cBHq2iHEalc/iYR591rUlFYI06kVvnGT/pOUtuB70iKQXz2mcBUX9sJhzU1X4gf8FvEeJ oTMW0a7B03AAyudKPruBzuA4BYgyNmbPzlTJIxiZ5URjD1zhmlIePGJy4oX1TRJBXL1OHtmFgNrhG EPPRUpVnXANcJ6ufzAxfUHecR72rLqYgWPXYGcUAs/0yGGWgvCqknGiBTtzWYLesNn1olgYCCyUlK P8/ib07CmpQqe+P0gJeA==; Received: from [193.50.110.143] (port=35130 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvbOy-0003rp-Md for control@debbugs.gnu.org; Mon, 30 May 2022 05:09:28 -0400 Date: Mon, 30 May 2022 11:09:25 +0200 Message-Id: <87wne35quy.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #55721 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) severity 55721 important quit From debbugs-submit-bounces@debbugs.gnu.org Mon May 30 05:31:59 2022 Received: (at 55721) by debbugs.gnu.org; 30 May 2022 09:31:59 +0000 Received: from localhost ([127.0.0.1]:42776 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvbkl-0004Mq-4Z for submit@debbugs.gnu.org; Mon, 30 May 2022 05:31:59 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47568) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvbki-0004Md-GE for 55721@debbugs.gnu.org; Mon, 30 May 2022 05:31:57 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:33998) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvbkd-0003Xd-8e for 55721@debbugs.gnu.org; Mon, 30 May 2022 05:31:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=k+4hzhz3PbnpiGf0fYDnKHjcwShTug2upnqbtswpSx0=; b=OFtxj11mqFOE4XYkilLY +UIwk682kclJhLGoRh6BgoH5cKRN3sF7ZayAGDZbjA6VIkTXD6+8DbjB8Gaqc4Pa+TILXg+CDxRst R9AlsOHubY3obx0yVu+cV0o2SHXItv/PeCudFT7qf1EuYspXZLAR/ancmPv7tNmTCfr3oKmGNUz8B u3oEK7qaI1AQWUJFWV718UvK5eYVxO4w9KqU2ZqdPaAmyQjc3V3se5WHojaZtIrkFsZ5pEwNq8KSW 1yPDQrNdflTysBzR0XoFmRcFSioo4K3QOB7Y1Ye7HOF9wd2TEnWX9nh/BBzV4tFYWZhA5vkEB84p2 17JcBq+qRD47cQ==; Received: from [193.50.110.143] (port=38260 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvbkc-0004UO-QJ for 55721@debbugs.gnu.org; Mon, 30 May 2022 05:31:51 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 55721@debbugs.gnu.org Subject: Re: bug#55721: Low cache usage in the presence of grafts References: <87y1yj5s1p.fsf@inria.fr> Date: Mon, 30 May 2022 11:31:47 +0200 In-Reply-To: <87y1yj5s1p.fsf@inria.fr> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?= =?utf-8?Q?'s?= message of "Mon, 30 May 2022 10:43:46 +0200") Message-ID: <87pmjv5pto.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -1.8 (-) X-Debbugs-Envelope-To: 55721 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.8 (--) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s skribis: > $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./pr= e-inst-env guix system vm gnu/system/examples/desktop.tmpl=20 > 0.2 MB will be downloaded > polkit-0.121 198KiB = 1.4MiB/s 00:00 [##################] 100.0% > /gnu/store/sk1571rnh5jl2ilp0v1k8bqwsdrngwxy-run-vm.sh > Reference Cache: > fresh caches: 8 > lookups: 77581 > hits: 52890 (68.2%) > cache size: 838 entries > Derivation graft cache: > fresh caches: 39 > lookups: 704635 > hits: 627819 (89.1%) > cache size: 2946 entries > Remote procedure call summary: 33071 RPCs > built-in-builders ... 1 > query-substitutable-path-infos ... 2 > build-things ... 2 > add-to-store/tree ... 56 > valid-path? ... 59 > add-to-store ... 373 > add-text-to-store ... 7887 > query-references ... 24691 > > real 5m6.727s > user 6m59.438s > sys 0m1.633s With this patch: --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/guix/store.scm b/guix/store.scm index efba07bdcd..88cab1bb6a 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -1337,7 +1337,12 @@ (define (build-accumulator expected-store) (if (and (eq? (store-connection-socket store) (store-connection-socket expected-store)) (= mode (build-mode normal))) - (unresolved things continue) + (unresolved things + (lambda (new-store value) + ;; Borrow caches from NEW-STORE. + (set-store-connection-caches! + store (store-connection-caches new-store)) + (continue value))) (continue #t)))) (define default-cutoff @@ -1397,7 +1402,8 @@ (define-values (result rest) (if (unresolved? obj) ;; Pass #f because 'build-things' is now ;; unnecessary. - ((unresolved-continuation obj) #f) + ((unresolved-continuation obj) + store #f) obj)) result #:cutoff cutoff) (map/accumulate-builds store proc rest #:cutoff cutoff))))) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable =E2=80=A6 we=E2=80=99re doing better: --8<---------------cut here---------------start------------->8--- $ guix gc -D $(guix build polkit --no-grafts) finding garbage collector roots... [0 MiB] deleting '/gnu/store/8vsfk8312m5p461wl4dq5lxjagfbwfhz-polkit-0.121' deleting `/gnu/store/trash' deleting unused links... note: currently hard linking saves 72397.13 MiB $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./pre-= inst-env guix system vm gnu/system/examples/desktop.tmpl=20 0.2 MB will be downloaded polkit-0.121 198KiB = 1.6MiB/s 00:00 [##################] 100.0% /gnu/store/wp3lv4xrh6vw79gnkyi5471c1l8j624n-run-vm.sh Reference Cache: fresh caches: 8 lookups: 17232 hits: 14816 (86.0%) cache size: 784 entries Derivation graft cache: fresh caches: 39 lookups: 128402 hits: 111336 (86.7%) cache size: 328 entries Remote procedure call summary: 10796 RPCs built-in-builders ... 1 query-substitutable-path-infos ... 2 build-things ... 2 add-to-store/tree ... 56 valid-path? ... 59 add-to-store ... 373 query-references ... 2416 add-text-to-store ... 7887 real 1m31.550s user 2m7.107s sys 0m0.637s --8<---------------cut here---------------end--------------->8--- There=E2=80=99s still room for improvement though. Ludo=E2=80=99. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon May 30 07:35:31 2022 Received: (at 55721) by debbugs.gnu.org; 30 May 2022 11:35:32 +0000 Received: from localhost ([127.0.0.1]:42954 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvdgJ-0003nJ-Cv for submit@debbugs.gnu.org; Mon, 30 May 2022 07:35:31 -0400 Received: from flashner.co.il ([178.62.234.194]:51146) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvdgE-0003mu-Kj for 55721@debbugs.gnu.org; Mon, 30 May 2022 07:35:30 -0400 Received: from localhost (unknown [141.226.14.232]) by flashner.co.il (Postfix) with ESMTPSA id A3547401C5; Mon, 30 May 2022 11:35:20 +0000 (UTC) Date: Mon, 30 May 2022 14:34:40 +0300 From: Efraim Flashner To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#55721: Low cache usage in the presence of grafts Message-ID: Mail-Followup-To: Efraim Flashner , Ludovic =?utf-8?Q?Court=C3=A8s?= , 55721@debbugs.gnu.org References: <87y1yj5s1p.fsf@inria.fr> <87pmjv5pto.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="el+JekaWj0IAG2lS" Content-Disposition: inline In-Reply-To: <87pmjv5pto.fsf@gnu.org> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 55721 Cc: 55721@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) --el+JekaWj0IAG2lS Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 30, 2022 at 11:31:47AM +0200, Ludovic Court=C3=A8s wrote: > Ludovic Court=C3=A8s skribis: >=20 > > $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./= pre-inst-env guix system vm gnu/system/examples/desktop.tmpl=20 > > 0.2 MB will be downloaded > > polkit-0.121 198KiB = 1.4MiB/s 00:00 [##################] 100.0% > > /gnu/store/sk1571rnh5jl2ilp0v1k8bqwsdrngwxy-run-vm.sh > > Reference Cache: > > fresh caches: 8 > > lookups: 77581 > > hits: 52890 (68.2%) > > cache size: 838 entries > > Derivation graft cache: > > fresh caches: 39 > > lookups: 704635 > > hits: 627819 (89.1%) > > cache size: 2946 entries > > Remote procedure call summary: 33071 RPCs > > built-in-builders ... 1 > > query-substitutable-path-infos ... 2 > > build-things ... 2 > > add-to-store/tree ... 56 > > valid-path? ... 59 > > add-to-store ... 373 > > add-text-to-store ... 7887 > > query-references ... 24691 > > > > real 5m6.727s > > user 6m59.438s > > sys 0m1.633s >=20 > With this patch: >=20 > diff --git a/guix/store.scm b/guix/store.scm > index efba07bdcd..88cab1bb6a 100644 > --- a/guix/store.scm > +++ b/guix/store.scm > @@ -1337,7 +1337,12 @@ (define (build-accumulator expected-store) > (if (and (eq? (store-connection-socket store) > (store-connection-socket expected-store)) > (=3D mode (build-mode normal))) > - (unresolved things continue) > + (unresolved things > + (lambda (new-store value) > + ;; Borrow caches from NEW-STORE. > + (set-store-connection-caches! > + store (store-connection-caches new-store)) > + (continue value))) > (continue #t)))) > =20 > (define default-cutoff > @@ -1397,7 +1402,8 @@ (define-values (result rest) > (if (unresolved? obj) > ;; Pass #f because 'build-thin= gs' is now > ;; unnecessary. > - ((unresolved-continuation obj)= #f) > + ((unresolved-continuation obj) > + store #f) > obj)) > result #:cutoff cutoff) > (map/accumulate-builds store proc rest #:cutoff cutoff))))) >=20 > =E2=80=A6 we=E2=80=99re doing better: >=20 > --8<---------------cut here---------------start------------->8--- > $ guix gc -D $(guix build polkit --no-grafts) > finding garbage collector roots... > [0 MiB] deleting '/gnu/store/8vsfk8312m5p461wl4dq5lxjagfbwfhz-polkit-0.12= 1' > deleting `/gnu/store/trash' > deleting unused links... > note: currently hard linking saves 72397.13 MiB > $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./pr= e-inst-env guix system vm gnu/system/examples/desktop.tmpl=20 > 0.2 MB will be downloaded > polkit-0.121 198KiB = 1.6MiB/s 00:00 [##################] 100.0% > /gnu/store/wp3lv4xrh6vw79gnkyi5471c1l8j624n-run-vm.sh > Reference Cache: > fresh caches: 8 > lookups: 17232 > hits: 14816 (86.0%) > cache size: 784 entries > Derivation graft cache: > fresh caches: 39 > lookups: 128402 > hits: 111336 (86.7%) > cache size: 328 entries > Remote procedure call summary: 10796 RPCs > built-in-builders ... 1 > query-substitutable-path-infos ... 2 > build-things ... 2 > add-to-store/tree ... 56 > valid-path? ... 59 > add-to-store ... 373 > query-references ... 2416 > add-text-to-store ... 7887 >=20 > real 1m31.550s > user 2m7.107s > sys 0m0.637s > --8<---------------cut here---------------end--------------->8--- >=20 > There=E2=80=99s still room for improvement though. >=20 > Ludo=E2=80=99. With that change you've dropped the query-references from 24691 to 2416, more than a 90% drop. If you want to take aim at improving it further I'd say take a look at add-text-to-store and see if you can shrink that number. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --el+JekaWj0IAG2lS Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmKUq80ACgkQQarn3Mo9 g1GdUQ/+N9nCWMi99+De+8gC7kx0N9TfrtbVt0W6vNuzuhc2g5CjDV0/Wm6foo6W 08Mn7bqwMdXJZ4zQshYq5WTKo38Yeie5C7xQ6dFp8cyrWMkicBPhig/XGfo3Whto A3BZ2xOtmpDclUlpwaF4GOxXbcgAEPxLutAjtZYsK4SJdxysAnKEbcWmDQqhFp4G wNZH0ubpnIp2LODdBDi9iL8RmrhUNIhIbFucNYM+QKUGM0A5717vNjS4p/BLPkGu Bw4uqPutweFuvVushzobTctLBN4C9jaizsneukKwBqeZyQSzWqk0j9b1UuUg+3Ev 8bqBL4l0+Jpw/KNl37MNAQ8WZ5gjp92NxfWNeMUDdEweN2eN0sFlEjcDLzVAK5Uc 99+6L4ZsNdlt6cpmmS8zI5pdEvtQCVl9uIzXjXYvexkOjk0cbElXctNv6sPSPhpg HvImJbQTqyGCafQdBaS8V4L5xp7c2TalGooCDgA4nb+vKLy8o5v/36sGA6OjcNp4 uLVV5uzI+U4Ts3UrcHO59/hFy8ZHpE0FKkFzD1PChIIFa5Yt3sMB+d9xjRFsW8pg +iwL85Hgc0DbPhYr9YlB3Onmzj1N01xikLbX9fgJ/2hKNjADxb3c7bqX+nUOBQIa /nvX/GZJlSq0g20lHiPAkRo3w0P1iTBRZxzSs7A5YnfskaqePAk= =f62t -----END PGP SIGNATURE----- --el+JekaWj0IAG2lS-- From debbugs-submit-bounces@debbugs.gnu.org Mon May 30 09:15:38 2022 Received: (at 55721) by debbugs.gnu.org; 30 May 2022 13:15:38 +0000 Received: from localhost ([127.0.0.1]:43292 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvfFB-00074A-VR for submit@debbugs.gnu.org; Mon, 30 May 2022 09:15:38 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52976) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvfF9-00073o-5C for 55721@debbugs.gnu.org; Mon, 30 May 2022 09:15:36 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37740) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvfF3-00018d-08; Mon, 30 May 2022 09:15:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=SNahPy8LZ5D7E5jRZF7czYz1VorlMORnvLke+mDevqg=; b=d/P7Ic20BCzVtDeva49U K7cv5ISGNLq2ozwK8p+ABIHh03phoh+qKuqyI/iwJSmTtPnbHyFJk5OKF2w9I67h7YSXB6ix08CrC vdRkl2+7AlHOU4VSHGiBAyfVWooi0Kg5qUku/95kgAMJU+Mzu2bqn9dW+/oXwgwrBQopKgrmHPXB9 k5DuS9BkCyTf/qL9+KpW+8fB0+LcSY+tN10++r5SdVvrr35GTTJuMfpjITkaLs+la9SC4z4smJSaN UGsgSEUuVgcbaKzx0C0XZqQ8bppcNRvhrTpgOe+/PAA7+k4nVfs8zkJmhH9BqjAM3MfQrNr5F4Y1V z2UHDTn92ZzrHg==; Received: from [193.50.110.143] (port=50948 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvfEk-00080i-03; Mon, 30 May 2022 09:15:27 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Efraim Flashner Subject: Re: bug#55721: Low cache usage in the presence of grafts References: <87y1yj5s1p.fsf@inria.fr> <87pmjv5pto.fsf@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Primidi 11 Prairial an 230 de la =?utf-8?Q?R=C3=A9vo?= =?utf-8?Q?lution=2C?= jour de la Fraise X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 30 May 2022 15:15:07 +0200 In-Reply-To: (Efraim Flashner's message of "Mon, 30 May 2022 14:34:40 +0300") Message-ID: <87bkvf5fhg.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55721 Cc: 55721@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Efraim Flashner skribis: > With that change you've dropped the query-references from 24691 to 2416, > more than a 90% drop. If you want to take aim at improving it further > I'd say take a look at add-text-to-store and see if you can shrink that > number. That=E2=80=99s not related to grafts though: these =E2=80=98add-text-to-sto= re=E2=80=99 RPCs correspond to adding the *-guile-builder files to the store, roughly one per package. This could be improved, for instance by having a single *-guile-builder script for all of =E2=80=98gnu-build-system=E2=80=99; that requires a bit o= f work. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon May 30 10:39:19 2022 Received: (at 55721-done) by debbugs.gnu.org; 30 May 2022 14:39:19 +0000 Received: from localhost ([127.0.0.1]:45708 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvgXw-0002N7-5U for submit@debbugs.gnu.org; Mon, 30 May 2022 10:39:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:55376) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvgXs-0002MR-Ly for 55721-done@debbugs.gnu.org; Mon, 30 May 2022 10:39:03 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39032) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvgXn-0001as-EN for 55721-done@debbugs.gnu.org; Mon, 30 May 2022 10:38:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=1MBqjIrgQwEbyGn2bbZYnp/sXCgKC4EdigN5+XCk3cU=; b=YsV5Xa0/G1gPf2yhCQnS 4zQIg+xWJsAw/AV7iHgOpfjyXtJdmEtiAzypiAgmSM0qRZSGiBof+RuIlOcxwsayMJhSDykT82vcU tgAUs5VTVY5Q0aFCClWZ0B0z42MO5UkWWXTtBiemS7fw5lpOyjuGhSiEFDpUVVL/SX5NGGpzNvVF9 cl/ppGJftMZD6v4eTirym4h8Q+zxgjnfHiCwym2fodYhBNe9ZrtADRSMUjx87gzXcabQ0cflZwY8h V4qmBqWQhOzLQFP728uwQVgeIhLcumS2ehGiQM+1wDZhRgg2S2qpGoFk549qcgsSCQuVYVreZ0aEB mU7btap7borMSw==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=36164 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvgXm-0001kB-HL for 55721-done@debbugs.gnu.org; Mon, 30 May 2022 10:38:55 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 55721-done@debbugs.gnu.org Subject: Re: bug#55721: Low cache usage in the presence of grafts References: <87y1yj5s1p.fsf@inria.fr> <87pmjv5pto.fsf@gnu.org> Date: Mon, 30 May 2022 16:38:52 +0200 In-Reply-To: <87pmjv5pto.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Mon, 30 May 2022 11:31:47 +0200") Message-ID: <877d633x1f.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55721-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) I pushed a variant of the patch I sent as 1ae0e1dc29109cef20dbe9d26eb1c855694716b5. What I get now is: --8<---------------cut here---------------start------------->8--- $ guix gc -D $(guix build polkit --no-grafts) finding garbage collector roots... [0 MiB] deleting '/gnu/store/8vsfk8312m5p461wl4dq5lxjagfbwfhz-polkit-0.121' deleting `/gnu/store/trash' deleting unused links... note: currently hard linking saves 72523.68 MiB $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./pre-= inst-env guix system vm gnu/system/examples/desktop.tmpl=20 0.2 MB will be downloaded polkit-0.121 198KiB = 1.4MiB/s 00:00 [##################] 100.0% /gnu/store/cvfw1jlf9m60s4yjxhnjjv68rg59giag-run-vm.sh guix system: warning: only 4.7 GiB of free space available on /gnu/store hint: Consider deleting old profile generations and collecting garbage, alo= ng these lines: guix gc --delete-generations=3D1m Reference Cache: fresh caches: 1 lookups: 13997 hits: 13091 (93.5%) cache size: 838 entries Derivation graft cache: fresh caches: 5 lookups: 117280 hits: 103398 (88.2%) cache size: 430 entries Remote procedure call summary: 9286 RPCs built-in-builders ... 1 query-substitutable-path-infos ... 2 build-things ... 2 add-to-store/tree ... 56 valid-path? ... 59 add-to-store ... 373 query-references ... 906 add-text-to-store ... 7887 real 0m45.089s user 1m5.976s sys 0m0.433s --8<---------------cut here---------------end--------------->8--- We=E2=80=99re still getting a few extra =E2=80=98query-references=E2=80=99 = RPCs, but it=E2=80=99s much better. Ludo=E2=80=99. From unknown Tue Jun 17 22:29:02 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 28 Jun 2022 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator