From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: computing derivations through inferior takes twice as long Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 24 Apr 2021 21:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 48007@debbugs.gnu.org X-Debbugs-Original-To: bug-guix@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.161929845230718 (code B ref -1); Sat, 24 Apr 2021 21:08:02 +0000 Received: (at submit) by debbugs.gnu.org; 24 Apr 2021 21:07:32 +0000 Received: from localhost ([127.0.0.1]:41599 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1laPUx-0007zN-Vb for submit@debbugs.gnu.org; Sat, 24 Apr 2021 17:07:32 -0400 Received: from lists.gnu.org ([209.51.188.17]:49268) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1laPUt-0007zE-VT for submit@debbugs.gnu.org; Sat, 24 Apr 2021 17:07:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43508) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1laPUt-00051o-R5 for bug-guix@gnu.org; Sat, 24 Apr 2021 17:07:27 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21173) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1laPUr-0005c6-EB for bug-guix@gnu.org; Sat, 24 Apr 2021 17:07:27 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1619298438; cv=none; d=zohomail.com; s=zohoarc; b=YHovs9S5Y7O4jQNv/rOISC/0cnBGJgLaFBv2J/EdYwqWWT2fgeZRaTuuVlDuUcdO7ApMNt2+YQ8vLzJJUsEVYcvFxOjtMe0EqXscvtoemrvDjJ/YUxq6thKQcS5K/+Cdy44b0tb3W1CxkOH+AFgyvbPbRM1qgnpSiK8a8It/Sek= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1619298438; h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To; bh=9qBBAAr+tRTZNB+gK0gY2h6OelR7KIfexvApyGh0bnw=; b=fX8+sdXE93nNzxL6di+97F1EMUlhsVatfwf5JQHS7YeigVSv574m9MwFCAEAz7EF/7bRNZ2TPQczFdNZ5EBUT2f01Fgruh2tlf3EI1TWBt6h3oWWZqfHy/GoxCCk8TBQb4h2a5rcU7X2JEmjnnj8VuaBAX/YzWbaspg4E1pRw5Q= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1619298438; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=9qBBAAr+tRTZNB+gK0gY2h6OelR7KIfexvApyGh0bnw=; b=Hn9hmuCaPhkGVVDWuYeaMgVFoZB/Qk8G5dkcrOzJ3AvjNYDiQKOUbVn+Tojmd0Bq XjxQ1tzecmjtzcUWnsksWcISHovoCWXlK1taokWSEuVnIAzsF0r4Yrp0ClZBdVVIHwj UtJmQdXWZLoTbqXeRC439CX5Qn+sY/BcaYUQSZ/I= Received: from localhost (p4fd5ab0c.dip0.t-ipconnect.de [79.213.171.12]) by mx.zohomail.com with SMTPS id 161929843743942.68504151230297; Sat, 24 Apr 2021 14:07:17 -0700 (PDT) User-agent: mu4e 1.4.15; emacs 27.2 From: Ricardo Wurmus X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Sat, 24 Apr 2021 23:07:14 +0200 Message-ID: <87wnsrpf25.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) This bug report might be related to bug #48005. In the Guix Workflow Language we are always looking up packages=20 through an inferior Guix. That Guix will in most cases be just=20 the current Guix. As I was looking for ways to speed the GWL up,=20 I noticed that the use of inferiors itself contributes to a=20 significant loss in performance. Here is a simple manifest to demonstrate this: --8<---------------cut here---------------start------------->8--- (use-modules (guix inferior) (ice-9 match)) (define inferior (open-inferior (format #false "~a/.config/guix" (getenv=20 "HOME")))) (define packages (list "bash-minimal" "r-corrplot" "r-crosstalk" "r-data-table" "r-deseq2" "r-dt" "r-genomicalignments" "r-genomicranges" "r-ggplot2" "r-ggrepel" "r-gprofiler" "r-knitr" "r-minimal" "r-pheatmap" "r-plotly" "r-reshape2" "r-rmarkdown" "r-rsamtools" "r-rtracklayer" "r-s4vectors" "r-scales" "r-summarizedexperiment" "r-tximport")) (match (getenv "INFERIOR") ("y" (packages->manifest (map (lambda (specification) (match (lookup-inferior-packages inferior=20 specification) ((first . rest) first))) packages))) (_ (specifications->manifest packages))) --8<---------------cut here---------------end--------------->8--- When INFERIOR is set to =E2=80=9Cy=E2=80=9D, each package specification wil= l be=20 looked up in the current Guix via an inferior. For any other=20 values of INFERIOR the specifications are resolved with the=20 current Guix (the very same Guix) directly. Here are the timings: --8<---------------cut here---------------start------------->8--- $ [env] export GUIX_PROFILING=3D"object-cache=20 add-data-to-store-cache rpc" $ [env] time INFERIOR=3Dn guix build -m manifest-test.scm -d /gnu/store/mwg47gbmi98bbrywk07y5l2h9p6d1hz5-bash-minimal-5.0.16.drv /gnu/store/kcjk6z128fa07pzp8irp6lbbyl3g16nr-r-corrplot-0.84.drv /gnu/store/s6hflcww9gaq87g5vaaydd4lphw63xjm-r-crosstalk-1.1.1.drv /gnu/store/qrjgag94sv9lq12028y9iv12j75bva6c-r-data-table-1.14.0.drv /gnu/store/v6xw6pg33xa8pg19nw0cxhz9b7ps26v7-r-deseq2-1.30.1.drv /gnu/store/q1achql92wnij108msymr9mkr8pv2z1h-r-dt-0.17.drv /gnu/store/iym2kzpjiqch22yrhg5lnv9sfazdfphn-r-genomicalignments-1.26.0.drv /gnu/store/k913mn4q11pchgi63xrm8lb3svvqjcix-r-genomicranges-1.42.0.drv /gnu/store/zkpabp1qx6m5yam3f9kninnsxagsgwqh-r-ggplot2-3.3.3.drv /gnu/store/b6w1p6rhbk8shz1ydc2yqb38ypm0ijq9-r-ggrepel-0.9.1.drv /gnu/store/bwmmls5qkf9cfs9m73qzabnr7w5jc8ra-r-gprofiler-0.7.0.drv /gnu/store/j1m8hb4449rkfh3ij1l4379j1lngjr06-r-knitr-1.31.drv /gnu/store/7ig30kf3i65s3rdcw1qik7vsjvspkjxy-r-minimal-4.0.4.drv /gnu/store/mwg8c42sfsvcrbjhbw7mbdcphhz9hq3x-r-pheatmap-1.0.12.drv /gnu/store/xjg40q7a7yl3l9v99kqapjylfjwapwk7-r-plotly-4.9.3.drv /gnu/store/fhs8as885izfb1r6as07sn6jpjgfbl58-r-reshape2-1.4.4.drv /gnu/store/6bcny1hhf83k85js6x3w7h1w3660ii8m-r-rmarkdown-2.7.drv /gnu/store/87pr587bk9rzfkrjmrm4bcfjz95p1n9c-r-rsamtools-2.6.0.drv /gnu/store/l3ibbpd4h7gm565vidbpyamdnhb0czhp-r-rtracklayer-1.50.0.drv /gnu/store/8rf8d204kavcxkw6z71kxd2mzzqzxsk1-r-s4vectors-0.28.1.drv /gnu/store/4nxw4lhcvj3q9j5v6mq9ri4v4vwmxd6h-r-scales-1.1.1.drv /gnu/store/vpf3vkj58vwz92nxcpppil6580c84bb1-r-summarizedexperiment-1.20.0.d= rv /gnu/store/cx3cl0nxwvzpaj484q2xcnz3v7zc1015-r-tximport-1.18.0.drv Store object cache: fresh caches: 2 lookups: 4540 hits: 3568 (78.6%) cache size: 971 entries 'add-data-to-store' cache: lookups: 3450 hits: 492 (14.3%) .drv files: 2087 (60.5%) Scheme files: 1347 (39.0%) Remote procedure call summary: 3412 RPCs built-in-builders ... 1 add-to-store/tree ... 16 add-to-store ... 177 query-references ... 260 add-text-to-store ... 2958 real 0m3.970s user 0m4.055s sys 0m0.173s $ [env] time INFERIOR=3Dy guix build -m manifest-test.scm -d /gnu/store/mwg47gbmi98bbrywk07y5l2h9p6d1hz5-bash-minimal-5.0.16.drv /gnu/store/hmk49rhbfqw2ss55392a7kq34xqg18i7-r-corrplot-0.84.drv /gnu/store/sg8a3pvzxaq2qd4z918mdb2y0vq6w8mg-r-crosstalk-1.1.1.drv /gnu/store/n3vk2kkq7zza7pfrjqqbv6xaxhnzdn2x-r-data-table-1.14.0.drv /gnu/store/44fqdg0s6bcmcgafvgafycf2x82rfl7y-r-deseq2-1.30.1.drv /gnu/store/03snyvyp9fr3nchrln6qhdca00i7lrsz-r-dt-0.17.drv /gnu/store/rl48alwm40sl4b04rnk4cck2h4crr8gc-r-genomicalignments-1.26.0.drv /gnu/store/ryl6hjflgpb72xl91jvp0ab6sl5cblc4-r-genomicranges-1.42.0.drv /gnu/store/1hbg746cvi8s7vn03glzx46m0pdih5pw-r-ggplot2-3.3.3.drv /gnu/store/nwvkjb314hh7z7vag0mk870isynp0hda-r-ggrepel-0.9.1.drv /gnu/store/kvvygkc7vnznrqp4n2rvgsbz9z2jd6ns-r-gprofiler-0.7.0.drv /gnu/store/0jv2zf34b2p1ddpxnzv5smq4717i4hfq-r-knitr-1.31.drv /gnu/store/zgi8sfw54jv7wb33q9cs18ff1vlfy0fm-r-minimal-4.0.4.drv /gnu/store/7w4jp2skqy0vn8i4pr26l94mw8vs8knc-r-pheatmap-1.0.12.drv /gnu/store/xshkhmd8gpjkmi7npz0bw02wgb8mkysg-r-plotly-4.9.3.drv /gnu/store/5jqkb3khygfc2y96nff92hfslc2c53yz-r-reshape2-1.4.4.drv /gnu/store/x0fzqyjg1hq7a4n0wglr9sl71bzxwz0q-r-rmarkdown-2.7.drv /gnu/store/3v78408vx5x28nb3cf42jarr7fy3b16v-r-rsamtools-2.6.0.drv /gnu/store/qp4hjddv5sjxiiss0m55q4cv88k520gd-r-rtracklayer-1.50.0.drv /gnu/store/pgfahjz3wfnppc07z0qbcsdc6mmpri0l-r-s4vectors-0.28.1.drv /gnu/store/aq317mqb3rbc2rnq2y15k781q5qvf9ia-r-scales-1.1.1.drv /gnu/store/w9dirjkx523398mhkjw0v4hxgq7x0b8s-r-summarizedexperiment-1.20.0.d= rv /gnu/store/rfmzii8xsc3fk63s332ix2qgxpvdvrgf-r-tximport-1.18.0.drv Store object cache: fresh caches: 1 lookups: 23 hits: 0 (0.0%) cache size: 23 entries 'add-data-to-store' cache: lookups: 0 hits: 0 (100.0%) .drv files: 0 (100.0%) Scheme files: 0 (100.0%) Remote procedure call summary: 0 RPCs real 0m7.951s user 0m2.191s sys 0m0.240s --8<---------------cut here---------------end--------------->8--- Note that nothing is built. This is merely to compute already=20 existing derivations. Computing the same derivations through an=20 inferior Guix takes about twice as long. I=E2=80=99ll note that in the inferior case there are no=20 =E2=80=9Cadd-data-to-store=E2=80=9D calls compared to 2958 calls in the dir= ect=20 case. I don=E2=80=99t know what to make of this. Is that information=20 lost as we cross over to the inferior Guix=E2=80=A6? Or are there really=20 different / fewer RPCs? Things look similar when we actually instantiate the manifest: --8<---------------cut here---------------start------------->8--- $ [env] time INFERIOR=3Dn guix package -m manifest-test.scm -p=20 /tmp/foo The following packages will be installed: bash-minimal 5.0.16 r-corrplot 0.84 r-crosstalk 1.1.1 r-data-table 1.14.0 r-deseq2 1.30.1 r-dt 0.17 r-genomicalignments 1.26.0 r-genomicranges 1.42.0 r-ggplot2 3.3.3 r-ggrepel 0.9.1 r-gprofiler 0.7.0 r-knitr 1.31 r-minimal 4.0.4 r-pheatmap 1.0.12 r-plotly 4.9.3 r-reshape2 1.4.4 r-rmarkdown 2.7 r-rsamtools 2.6.0 r-rtracklayer 1.50.0 r-s4vectors 0.28.1 r-scales 1.1.1 r-summarizedexperiment 1.20.0 r-tximport 1.18.0 nothing to be done Store object cache: fresh caches: 2 lookups: 41381 hits: 40249 (97.3%) cache size: 1131 entries 'add-data-to-store' cache: lookups: 6083 hits: 2950 (48.5%) .drv files: 3407 (56.0%) Scheme files: 2659 (43.7%) Remote procedure call summary: 3897 RPCs built-in-builders ... 1 add-to-store/tree ... 22 add-to-store ... 178 query-references ... 563 add-text-to-store ... 3133 real 0m12.697s user 0m15.873s sys 0m0.197s $ [env] time INFERIOR=3Dy guix package -m manifest-test.scm -p=20 /tmp/foo The following packages will be installed: bash-minimal 5.0.16 r-corrplot 0.84 r-crosstalk 1.1.1 r-data-table 1.14.0 r-deseq2 1.30.1 r-dt 0.17 r-genomicalignments 1.26.0 r-genomicranges 1.42.0 r-ggplot2 3.3.3 r-ggrepel 0.9.1 r-gprofiler 0.7.0 r-knitr 1.31 r-minimal 4.0.4 r-pheatmap 1.0.12 r-plotly 4.9.3 r-reshape2 1.4.4 r-rmarkdown 2.7 r-rsamtools 2.6.0 r-rtracklayer 1.50.0 r-s4vectors 0.28.1 r-scales 1.1.1 r-summarizedexperiment 1.20.0 r-tximport 1.18.0 nothing to be done Store object cache: fresh caches: 2 lookups: 27162 hits: 26425 (97.3%) cache size: 736 entries 'add-data-to-store' cache: lookups: 887 hits: 52 (5.9%) .drv files: 550 (62.0%) Scheme files: 331 (37.3%) Remote procedure call summary: 1011 RPCs built-in-builders ... 1 add-to-store/tree ... 11 query-references ... 51 add-to-store ... 113 add-text-to-store ... 835 real 0m19.504s user 0m4.014s sys 0m0.411s --8<---------------cut here---------------end--------------->8--- The first case with 12 seconds reproduces bug #48005. The second=20 case (going through the inferior) is much slower with over 19=20 seconds; if we squint this is also close to twice the amount of=20 time compared to the =E2=80=9Cdirect=E2=80=9D computation. --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Fri Apr 30 11:45:54 2021 Received: (at control) by debbugs.gnu.org; 30 Apr 2021 15:45:54 +0000 Received: from localhost ([127.0.0.1]:59612 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lcVL0-0005Tz-0c for submit@debbugs.gnu.org; Fri, 30 Apr 2021 11:45:54 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40638) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lcVKz-0005OH-62 for control@debbugs.gnu.org; Fri, 30 Apr 2021 11:45:53 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49806) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lcVKt-0007Yx-W8 for control@debbugs.gnu.org; Fri, 30 Apr 2021 11:45:48 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=40474 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lcVKt-0000JE-By for control@debbugs.gnu.org; Fri, 30 Apr 2021 11:45:47 -0400 Date: Fri, 30 Apr 2021 17:45:46 +0200 Message-Id: <87mttfbwt1.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #48007 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) severity 48007 important quit From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: computing derivations through inferior takes twice as long Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 26 Jan 2022 20:52:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ricardo Wurmus Cc: 48007@debbugs.gnu.org Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.164323029015335 (code B ref 48007); Wed, 26 Jan 2022 20:52:02 +0000 Received: (at 48007) by debbugs.gnu.org; 26 Jan 2022 20:51:30 +0000 Received: from localhost ([127.0.0.1]:53667 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCpGM-0003zH-6V for submit@debbugs.gnu.org; Wed, 26 Jan 2022 15:51:30 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:50610) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCpGJ-0003z1-LK for 48007@debbugs.gnu.org; Wed, 26 Jan 2022 15:51:28 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id B01D0316; Wed, 26 Jan 2022 21:51:20 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l5c4frjKKyMO; Wed, 26 Jan 2022 21:51:19 +0100 (CET) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id ACA502A2; Wed, 26 Jan 2022 21:51:18 +0100 (CET) From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87wnsrpf25.fsf@elephly.net> Date: Wed, 26 Jan 2022 21:51:18 +0100 In-Reply-To: <87wnsrpf25.fsf@elephly.net> (Ricardo Wurmus's message of "Sat, 24 Apr 2021 23:07:14 +0200") Message-ID: <87r18ufcft.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spamd-Bar: / Authentication-Results: hera.aquilenet.fr; none X-Rspamd-Server: hera X-Rspamd-Queue-Id: B01D0316 X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain,text/x-patch]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spam-Score: 1.0 (+) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi! Ricardo Wurmus skribis: > When INFERIOR is set to =E2=80=9Cy=E2=80=9D, each package specification w= ill be=20 > looked up in the current Guix via an inferior. For any other=20 > values of INFERIOR the specifications are resolved with the=20 > current Guix (the very same Guix) directly. > > Here are the timings: > > $ [env] export GUIX_PROFILING=3D"object-cache=20 > add-data-to-store-cache rpc" > $ [env] time INFERIOR=3Dn guix build -m manifest-test.scm -d > /gnu/store/mwg47gbmi98bbrywk07y5l2h9p6d1hz5-bash-minimal-5.0.16.drv > /gnu/store/kcjk6z128fa07pzp8irp6lbbyl3g16nr-r-corrplot-0.84.drv [...] > $ [env] time INFERIOR=3Dy guix build -m manifest-test.scm -d With the manifest you gave in this message, I get roughly these wall-clock times as of 3993d33d1c0129b1ca6f0fd122fe2bbe48e4f093 for: guix build -m the-manifest.scm -n INFERIOR=3Dn 4.1s INFERIOR=3Dy 36.9s With the patch below, it=E2=80=99s down to: INFERIOR=3Dy 9.3s The trick is to ensure the inferior maintains its object cache across calls. The patch needs to be cleaned up because it peeks into internals, but we should be able to do something along these lines and optimize a couple of other things. If you can give it a spin on a more representative example, that=E2=80=99s great! Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/guix/inferior.scm b/guix/inferior.scm index 572114f626..f6866d2083 100644 --- a/guix/inferior.scm +++ b/guix/inferior.scm @@ -188,6 +188,8 @@ (define* (port->inferior pipe #:optional (close close-port)) (inferior-eval '(use-modules (srfi srfi-34)) result) (inferior-eval '(define %package-table (make-hash-table)) result) + (inferior-eval '(define %previous-object-cache #f) + result) result)) (_ #f))) @@ -559,6 +561,10 @@ (define (inferior-eval-with-store inferior store code) (let ((store (if (defined? 'port->connection) (port->connection socket #:version ,proto) (open-connection)))) + (when %previous-object-cache + (set-store-connection-cache! store (@@ (guix store) %object-cache-id) + %previous-object-cache)) + (dynamic-wind (const #t) (lambda () @@ -570,6 +576,9 @@ (define (inferior-eval-with-store inferior store code) `(store-protocol-error ,(error-message c)))) `(result ,(proc store)))) (lambda () + (set! %previous-object-cache + (store-connection-cache store + (@@ (guix store) %object-cache-id))) (close-connection store) (close-port socket))))) inferior) --=-=-=-- From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: computing derivations through inferior takes twice as long Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 26 Jan 2022 21:37:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 48007@debbugs.gnu.org Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.164323297619633 (code B ref 48007); Wed, 26 Jan 2022 21:37:02 +0000 Received: (at 48007) by debbugs.gnu.org; 26 Jan 2022 21:36:16 +0000 Received: from localhost ([127.0.0.1]:53694 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCpxg-00056b-2b for submit@debbugs.gnu.org; Wed, 26 Jan 2022 16:36:16 -0500 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21121) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCpxe-00056T-DS for 48007@debbugs.gnu.org; Wed, 26 Jan 2022 16:36:14 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1643232909; cv=none; d=zohomail.com; s=zohoarc; b=UNlqA2hVzAKsJ+hoItspTwj5aJlpeyC4kpoSo/YJ6qR63ZEVrg6PxXGMXQcx5Rcc2BEUrtwkMgNWzpYmwMYksqPvE3RMrCb3vCixD11yfOuZJ007Ld6RimcTmfoFf9LcFWZnVEj9K8h3qzkJdgzEnZBErfFkS4jc02pSeP73E24= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1643232909; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=v/6Cx9nN7nP6hCEkiEFv/bhOFTawx7yu3S2uu2e5pl0=; b=i+/hAIy6X+qpVunrDfCBC89/3BvlpAdOUuIMlxFCzsXCHCpTHzzYkwt4zrvezEf6Pg7ItAbQ027VznjsV4KN6EUhSz6jTv35A6JW2sIkcYk7bfIgmkm2obHfyEtBobmqG9gk2gUbkWl/Mp1M172RIaKlZoAB/oxNih4fMEf3ytQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1643232909; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=v/6Cx9nN7nP6hCEkiEFv/bhOFTawx7yu3S2uu2e5pl0=; b=adjvWxzb8yKsqbv5254ZY0ayUrYA7kxSRBIpSrtC4bHNItQEBAv1P7QLCEOdh/BF GI/lwFnKZERT+8mrPszWo9mAJh1LKJpFJu5BR+txqwzvO+8dkcvjfSidW4yzha3c3yY EHatw/atz2fiSVrH5AlD83RZSlGTv4B9QKrREKlY= Received: from localhost (p54ad4fa6.dip0.t-ipconnect.de [84.173.79.166]) by mx.zohomail.com with SMTPS id 1643232898242782.8121641946904; Wed, 26 Jan 2022 13:34:58 -0800 (PST) References: <87wnsrpf25.fsf@elephly.net> <87r18ufcft.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 From: Ricardo Wurmus Date: Wed, 26 Jan 2022 22:32:02 +0100 In-reply-to: <87r18ufcft.fsf@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87czkeji4h.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > The trick is to ensure the inferior maintains its object cache across > calls. The patch needs to be cleaned up because it peeks into > internals, but we should be able to do something along these lines and > optimize a couple of other things. Yeah, this makes sense. Excellent! > If you can give it a spin on a more representative example, that=E2=80=99s > great! I tried it in the GWL with the big RNAseq workflow I adopted from PiGx and the step to generate job scripts (which reference inferior packages) became considerably faster. There is still potential for improvement, but this change is the difference between not too bad (15 seconds) and unusable (> 5 minutes). Thank you! --=20 Ricardo From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: [PATCH 2/4] inferior: Keep the store bridge connected. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 27 Jan 2022 08:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 48007@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.164327330129779 (code B ref 48007); Thu, 27 Jan 2022 08:49:02 +0000 Received: (at 48007) by debbugs.gnu.org; 27 Jan 2022 08:48:21 +0000 Received: from localhost ([127.0.0.1]:54270 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0S4-0007kD-Hu for submit@debbugs.gnu.org; Thu, 27 Jan 2022 03:48:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33142) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0S1-0007js-V1 for 48007@debbugs.gnu.org; Thu, 27 Jan 2022 03:48:18 -0500 Received: from [2001:470:142:3::e] (port=51728 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nD0Rw-0000sF-Dr; Thu, 27 Jan 2022 03:48:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=249/nWybbQ7ipeqdDBp2and84w7AN9yCbk0zSe7Uo/w=; b=NVIecgF0FtFD7O6e9WuD FkAb3xyg4PkZSXysxdHeUsxtu6eY2SULEXcGaDUUiKzEVfHZgXetffTW5r/WAiP4DxWJWzeC58BdX 8ZN1wuJPaP5U1+3/wyzdANor9AFmkSMoRBoJ5jDlW/bpyzUTrfqE8xJeXK6H76B4fye3IKQXnvt3e K4M1eyrqnzsrVsqIgzQrpX8qXFSCQ+acvlzGwqO7rY2/CWhy/xwALytGmOvsPRZQ3wS+zLO5k4cy7 SLutQEO7ZZ472JmXYS1o1+ssTekxWAzi9sTJJe8+q53q29cmUX1JTMugM2DGjN8xMnbgP9AsSU7Z+ XLqMGtQmDsQ5xg==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:60148 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nD0Rv-0002sQ-HY; Thu, 27 Jan 2022 03:48:11 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 27 Jan 2022 09:47:41 +0100 Message-Id: <20220127084743.27130-2-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220127084743.27130-1-ludo@gnu.org> References: <87r18ufcft.fsf@gnu.org> <20220127084743.27130-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Previously, each 'inferior-eval-with-store' would lead the inferior to connect to the named socket the parent is listening to. With this change, the connection is established once for all and reused afterwards. * guix/inferior.scm ()[bridge-file-name]: Remove. (open-bidirectional-pipe): New procedure. (inferior-pipe): Use it instead of 'open-pipe*' and return two values. (port->inferior): Adjust call to 'inferior'. (open-inferior): Adjust to 'inferior-pipe' changes. (close-inferior): Remove 'inferior-bridge-file-name' handling. (open-store-bridge!): Switch back to 'call-with-temporary-directory'. Define '%bridge-socket' in the inferior, connected to the caller. (proxy): Change first argument to be an inferior. Add 'reponse-port' and call to 'drain-input'. Pass 'reponse-port' to 'select' and use it as a loop termination clause. (inferior-eval-with-store): Remove 'socket' and 'connect' calls from the inferior code, and use '%bridge-socket' instead. --- guix/inferior.scm | 167 +++++++++++++++++++++++++++++----------------- 1 file changed, 104 insertions(+), 63 deletions(-) diff --git a/guix/inferior.scm b/guix/inferior.scm index a997c3ead4..1c19527b8f 100644 --- a/guix/inferior.scm +++ b/guix/inferior.scm @@ -25,6 +25,7 @@ (define-module (guix inferior) #:select (source-properties->location)) #:use-module ((guix utils) #:select (%current-system + call-with-temporary-directory version>? version-prefix? cache-directory)) #:use-module ((guix store) @@ -35,8 +36,6 @@ (define-module (guix inferior) &store-protocol-error)) #:use-module ((guix derivations) #:select (read-derivation-from-file)) - #:use-module ((guix build syscalls) - #:select (mkdtemp!)) #:use-module (guix gexp) #:use-module (guix search-paths) #:use-module (guix profiles) @@ -56,7 +55,6 @@ (define-module (guix inferior) #:use-module (srfi srfi-71) #:autoload (ice-9 ftw) (scandir) #:use-module (ice-9 match) - #:use-module (ice-9 popen) #:use-module (ice-9 vlist) #:use-module (ice-9 binary-ports) #:use-module ((rnrs bytevectors) #:select (string->utf8)) @@ -114,7 +112,7 @@ (define-module (guix inferior) ;; Inferior Guix process. (define-record-type (inferior pid socket close version packages table - bridge-file-name bridge-socket) + bridge-socket) inferior? (pid inferior-pid) (socket inferior-socket) @@ -124,8 +122,6 @@ (define-record-type (table inferior-package-table) ;promise of vhash ;; Bridging with a store. - (bridge-file-name inferior-bridge-file-name ;#f | string - set-inferior-bridge-file-name!) (bridge-socket inferior-bridge-socket ;#f | port set-inferior-bridge-socket!)) @@ -138,37 +134,69 @@ (define (write-inferior inferior port) (set-record-type-printer! write-inferior) +(define (open-bidirectional-pipe command . args) + "Open a bidirectional pipe to COMMAND invoked with ARGS and return it, as a +regular file port (socket). + +This is equivalent to (open-pipe* OPEN_BOTH ...) except that the result is a +regular file port that can be passed to 'select' ('open-pipe*' returns a +custom binary port)." + (match (socketpair AF_UNIX SOCK_STREAM 0) + ((parent . child) + (match (primitive-fork) + (0 + (dynamic-wind + (lambda () + #t) + (lambda () + (close-port parent) + (close-fdes 0) + (close-fdes 1) + (dup2 (fileno child) 0) + (dup2 (fileno child) 1) + ;; Mimic 'open-pipe*'. + (unless (file-port? (current-error-port)) + (close-fdes 2) + (dup2 (open-fdes "/dev/null" O_WRONLY) 2)) + (apply execlp command command args)) + (lambda () + (primitive-_exit 127)))) + (pid + (close-port child) + (values parent pid)))))) + (define* (inferior-pipe directory command error-port) - "Return an input/output pipe on the Guix instance in DIRECTORY. This runs -'DIRECTORY/COMMAND repl' if it exists, or falls back to some other method if -it's an old Guix." - (let ((pipe (with-error-to-port error-port - (lambda () - (open-pipe* OPEN_BOTH - (string-append directory "/" command) - "repl" "-t" "machine"))))) + "Return two values: an input/output pipe on the Guix instance in DIRECTORY +and its PID. This runs 'DIRECTORY/COMMAND repl' if it exists, or falls back +to some other method if it's an old Guix." + (let ((pipe pid (with-error-to-port error-port + (lambda () + (open-bidirectional-pipe + (string-append directory "/" command) + "repl" "-t" "machine"))))) (if (eof-object? (peek-char pipe)) (begin - (close-pipe pipe) + (close-port pipe) ;; Older versions of Guix didn't have a 'guix repl' command, so ;; emulate it. (with-error-to-port error-port (lambda () - (open-pipe* OPEN_BOTH "guile" - "-L" (string-append directory "/share/guile/site/" - (effective-version)) - "-C" (string-append directory "/share/guile/site/" - (effective-version)) - "-C" (string-append directory "/lib/guile/" - (effective-version) "/site-ccache") - "-c" - (object->string - `(begin - (primitive-load ,(search-path %load-path - "guix/repl.scm")) - ((@ (guix repl) machine-repl)))))))) - pipe))) + (open-bidirectional-pipe + "guile" + "-L" (string-append directory "/share/guile/site/" + (effective-version)) + "-C" (string-append directory "/share/guile/site/" + (effective-version)) + "-C" (string-append directory "/lib/guile/" + (effective-version) "/site-ccache") + "-c" + (object->string + `(begin + (primitive-load ,(search-path %load-path + "guix/repl.scm")) + ((@ (guix repl) machine-repl)))))))) + (values pipe pid)))) (define* (port->inferior pipe #:optional (close close-port)) "Given PIPE, an input/output port, return an inferior that talks over PIPE. @@ -181,7 +209,7 @@ (define* (port->inferior pipe #:optional (close close-port)) (letrec ((result (inferior 'pipe pipe close (cons 0 rest) (delay (%inferior-packages result)) (delay (%inferior-package-table result)) - #f #f))) + #f))) ;; For protocol (0 1) and later, send the protocol version we support. (match rest @@ -206,10 +234,11 @@ (define* (open-inferior directory (error-port (%make-void-port "w"))) "Open the inferior Guix in DIRECTORY, running 'DIRECTORY/COMMAND repl' or equivalent. Return #f if the inferior could not be launched." - (define pipe - (inferior-pipe directory command error-port)) - - (port->inferior pipe close-pipe)) + (let ((pipe pid (inferior-pipe directory command error-port))) + (port->inferior pipe + (lambda (port) + (close-port port) + (waitpid pid))))) (define (close-inferior inferior) "Close INFERIOR." @@ -218,9 +247,7 @@ (define (close-inferior inferior) ;; Close and delete the store bridge, if any. (when (inferior-bridge-socket inferior) - (close-port (inferior-bridge-socket inferior)) - (delete-file (inferior-bridge-file-name inferior)) - (rmdir (dirname (inferior-bridge-file-name inferior)))))) + (close-port (inferior-bridge-socket inferior))))) ;; Non-self-quoting object of the inferior. (define-record-type @@ -512,22 +539,32 @@ (define (inferior-package-provenance package) 'package-provenance)))) (or provenance (const #f))))) -(define (proxy client backend) ;adapted from (guix ssh) - "Proxy communication between CLIENT and BACKEND until CLIENT closes the -connection, at which point CLIENT is closed (both CLIENT and BACKEND must be -input/output ports.)" +(define (proxy inferior store) ;adapted from (guix ssh) + "Proxy communication between INFERIOR and STORE, until the connection to +STORE is closed or INFERIOR has data available for input (a REPL response)." + (define client + (inferior-bridge-socket inferior)) + (define backend + (store-connection-socket store)) + (define response-port + (inferior-socket inferior)) + ;; Use buffered ports so that 'get-bytevector-some' returns up to the ;; whole buffer like read(2) would--see . (setvbuf client 'block 65536) (setvbuf backend 'block 65536) + ;; RESPONSE-PORT may typically contain a leftover newline that 'read' didn't + ;; consume. Drain it so that 'select' doesn't immediately stop. + (drain-input response-port) + (let loop () - (match (select (list client backend) '() '()) + (match (select (list client backend response-port) '() '()) ((reads () ()) (when (memq client reads) (match (get-bytevector-some client) ((? eof-object?) - (close-port client)) + #t) (bv (put-bytevector backend bv) (force-output backend)))) @@ -536,7 +573,8 @@ (define (proxy client backend) ;adapted from (guix ssh) (bv (put-bytevector client bv) (force-output client)))) - (unless (port-closed? client) + (unless (or (port-closed? client) + (memq response-port reads)) (loop)))))) (define (open-store-bridge! inferior) @@ -547,17 +585,25 @@ (define (open-store-bridge! inferior) ;; its store. This ensures the inferior uses the same store, with the same ;; options, the same per-session GC roots, etc. ;; FIXME: This strategy doesn't work for remote inferiors (SSH). - (define directory - (mkdtemp! (string-append (or (getenv "TMPDIR") "/tmp") - "/guix-inferior.XXXXXX"))) + (call-with-temporary-directory + (lambda (directory) + (chmod directory #o700) + (let ((name (string-append directory "/inferior")) + (socket (socket AF_UNIX SOCK_STREAM 0))) + (bind socket AF_UNIX name) + (listen socket 2) - (chmod directory #o700) - (let ((name (string-append directory "/inferior")) - (socket (socket AF_UNIX SOCK_STREAM 0))) - (bind socket AF_UNIX name) - (listen socket 2) - (set-inferior-bridge-file-name! inferior name) - (set-inferior-bridge-socket! inferior socket))) + (send-inferior-request + `(define %bridge-socket + (let ((socket (socket AF_UNIX SOCK_STREAM 0))) + (connect socket AF_UNIX ,name) + socket)) + inferior) + (match (accept socket) + ((client . address) + (close-port socket) + (set-inferior-bridge-socket! inferior client))) + (read-inferior-response inferior))))) (define (ensure-store-bridge! inferior) "Ensure INFERIOR has a connected bridge." @@ -575,22 +621,19 @@ (define (inferior-eval-with-store inferior store code) (ensure-store-bridge! inferior) (send-inferior-request `(let ((proc ,code) - (socket (socket AF_UNIX SOCK_STREAM 0)) (error? (if (defined? 'store-protocol-error?) store-protocol-error? nix-protocol-error?)) (error-message (if (defined? 'store-protocol-error-message) store-protocol-error-message nix-protocol-error-message))) - (connect socket AF_UNIX - ,(inferior-bridge-file-name inferior)) ;; 'port->connection' appeared in June 2018 and we can hardly ;; emulate it on older versions. Thus fall back to ;; 'open-connection', at the risk of talking to the wrong daemon or ;; having our build result reclaimed (XXX). (let ((store (if (defined? 'port->connection) - (port->connection socket #:version ,proto) + (port->connection %bridge-socket #:version ,proto) (open-connection)))) (dynamic-wind (const #t) @@ -603,12 +646,10 @@ (define (inferior-eval-with-store inferior store code) `(store-protocol-error ,(error-message c)))) `(result ,(proc store)))) (lambda () - (close-connection store) - (close-port socket))))) + (unless (defined? 'port->connection) + (close-port store)))))) inferior) - (match (accept (inferior-bridge-socket inferior)) - ((client . address) - (proxy client (store-connection-socket store)))) + (proxy inferior store) (match (read-inferior-response inferior) (('store-protocol-error message) -- 2.34.0 From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: [PATCH 1/4] inferior: Create the store proxy listening socket only once. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 27 Jan 2022 08:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 48007@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.164327330229787 (code B ref 48007); Thu, 27 Jan 2022 08:49:02 +0000 Received: (at 48007) by debbugs.gnu.org; 27 Jan 2022 08:48:22 +0000 Received: from localhost ([127.0.0.1]:54274 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0S5-0007kG-ED for submit@debbugs.gnu.org; Thu, 27 Jan 2022 03:48:22 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33140) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0S1-0007jr-Sn for 48007@debbugs.gnu.org; Thu, 27 Jan 2022 03:48:19 -0500 Received: from [2001:470:142:3::e] (port=51726 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nD0Rv-0000s9-BA; Thu, 27 Jan 2022 03:48:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=CmsMqTnb6zyXeTmpN2nMLjXGuWMnR5VW0fxREJ53O+w=; b=MQfiR63el2GxCdYTZsfI oGAnVUltvNO3dL4MEbp2sB038CqWZos9/u1pb1KYZd9sPdgkAcXU7PByE3Z/j9VKnt4m/uHmIWV8Y ORN1l63XlVWEK7zcQxa/+SJ+V6L3kVeWXClPH6s4tuegSdTzHVAZTqmLx+qQRvVOuQR00P1CY9552 ii7v2mXRjbPDxWWAA6hNymaq5MGUnwSmV/+cuEKsqfVjwN9NM5oUVd9XXp5EKZ9Mo5BZdZzaxbp2f kEao4uyBj1Wv+aj6nu22IZSpBecT6V1zA+KdzJoxgMpKG634h/r1xXHA6r85XTOk2DV1D7OpIVwLC ISvlGSOBfnZnqw==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:60148 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nD0Ru-0002sQ-RV; Thu, 27 Jan 2022 03:48:11 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 27 Jan 2022 09:47:40 +0100 Message-Id: <20220127084743.27130-1-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <87r18ufcft.fsf@gnu.org> References: <87r18ufcft.fsf@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Previously, each 'inferior-eval-with-store' call would have the calling process create a temporary directory with a listening socket in there. Now that listening socket is created once and reused in subsequent calls. * guix/inferior.scm ()[bridge-file-name, bridge-socket]: New fields. (port->inferior): Adjust accordingly. (close-inferior): Close 'inferior-bridge-socket' and delete 'inferior-bridge-file-name' if set. (open-store-bridge!, ensure-store-bridge!): New procedures. (inferior-eval-with-store): Use them. --- guix/inferior.scm | 154 ++++++++++++++++++++++++++++------------------ 1 file changed, 93 insertions(+), 61 deletions(-) diff --git a/guix/inferior.scm b/guix/inferior.scm index 572114f626..a997c3ead4 100644 --- a/guix/inferior.scm +++ b/guix/inferior.scm @@ -25,7 +25,6 @@ (define-module (guix inferior) #:select (source-properties->location)) #:use-module ((guix utils) #:select (%current-system - call-with-temporary-directory version>? version-prefix? cache-directory)) #:use-module ((guix store) @@ -36,6 +35,8 @@ (define-module (guix inferior) &store-protocol-error)) #:use-module ((guix derivations) #:select (read-derivation-from-file)) + #:use-module ((guix build syscalls) + #:select (mkdtemp!)) #:use-module (guix gexp) #:use-module (guix search-paths) #:use-module (guix profiles) @@ -112,14 +113,21 @@ (define-module (guix inferior) ;; Inferior Guix process. (define-record-type - (inferior pid socket close version packages table) + (inferior pid socket close version packages table + bridge-file-name bridge-socket) inferior? (pid inferior-pid) (socket inferior-socket) (close inferior-close-socket) ;procedure (version inferior-version) ;REPL protocol version (packages inferior-package-promise) ;promise of inferior packages - (table inferior-package-table)) ;promise of vhash + (table inferior-package-table) ;promise of vhash + + ;; Bridging with a store. + (bridge-file-name inferior-bridge-file-name ;#f | string + set-inferior-bridge-file-name!) + (bridge-socket inferior-bridge-socket ;#f | port + set-inferior-bridge-socket!)) (define (write-inferior inferior port) (match inferior @@ -172,7 +180,8 @@ (define* (port->inferior pipe #:optional (close close-port)) (('repl-version 0 rest ...) (letrec ((result (inferior 'pipe pipe close (cons 0 rest) (delay (%inferior-packages result)) - (delay (%inferior-package-table result))))) + (delay (%inferior-package-table result)) + #f #f))) ;; For protocol (0 1) and later, send the protocol version we support. (match rest @@ -205,7 +214,13 @@ (define pipe (define (close-inferior inferior) "Close INFERIOR." (let ((close (inferior-close-socket inferior))) - (close (inferior-socket inferior)))) + (close (inferior-socket inferior)) + + ;; Close and delete the store bridge, if any. + (when (inferior-bridge-socket inferior) + (close-port (inferior-bridge-socket inferior)) + (delete-file (inferior-bridge-file-name inferior)) + (rmdir (dirname (inferior-bridge-file-name inferior)))))) ;; Non-self-quoting object of the inferior. (define-record-type @@ -524,67 +539,84 @@ (define (proxy client backend) ;adapted from (guix ssh) (unless (port-closed? client) (loop)))))) +(define (open-store-bridge! inferior) + "Open a \"store bridge\" for INFERIOR--a named socket in /tmp that will be +used to proxy store RPCs from the inferior to the store of the calling +process." + ;; Create a named socket in /tmp to let INFERIOR connect to it and use it as + ;; its store. This ensures the inferior uses the same store, with the same + ;; options, the same per-session GC roots, etc. + ;; FIXME: This strategy doesn't work for remote inferiors (SSH). + (define directory + (mkdtemp! (string-append (or (getenv "TMPDIR") "/tmp") + "/guix-inferior.XXXXXX"))) + + (chmod directory #o700) + (let ((name (string-append directory "/inferior")) + (socket (socket AF_UNIX SOCK_STREAM 0))) + (bind socket AF_UNIX name) + (listen socket 2) + (set-inferior-bridge-file-name! inferior name) + (set-inferior-bridge-socket! inferior socket))) + +(define (ensure-store-bridge! inferior) + "Ensure INFERIOR has a connected bridge." + (or (inferior-bridge-socket inferior) + (begin + (open-store-bridge! inferior) + (inferior-bridge-socket inferior)))) + (define (inferior-eval-with-store inferior store code) "Evaluate CODE in INFERIOR, passing it STORE as its argument. CODE must thus be the code of a one-argument procedure that accepts a store." - ;; Create a named socket in /tmp and let INFERIOR connect to it and use it - ;; as its store. This ensures the inferior uses the same store, with the - ;; same options, the same per-session GC roots, etc. - ;; FIXME: This strategy doesn't work for remote inferiors (SSH). - (call-with-temporary-directory - (lambda (directory) - (chmod directory #o700) - (let* ((name (string-append directory "/inferior")) - (socket (socket AF_UNIX SOCK_STREAM 0)) - (major (store-connection-major-version store)) - (minor (store-connection-minor-version store)) - (proto (logior major minor))) - (bind socket AF_UNIX name) - (listen socket 1024) - (send-inferior-request - `(let ((proc ,code) - (socket (socket AF_UNIX SOCK_STREAM 0)) - (error? (if (defined? 'store-protocol-error?) - store-protocol-error? - nix-protocol-error?)) - (error-message (if (defined? 'store-protocol-error-message) - store-protocol-error-message - nix-protocol-error-message))) - (connect socket AF_UNIX ,name) + (let* ((major (store-connection-major-version store)) + (minor (store-connection-minor-version store)) + (proto (logior major minor))) + (ensure-store-bridge! inferior) + (send-inferior-request + `(let ((proc ,code) + (socket (socket AF_UNIX SOCK_STREAM 0)) + (error? (if (defined? 'store-protocol-error?) + store-protocol-error? + nix-protocol-error?)) + (error-message (if (defined? 'store-protocol-error-message) + store-protocol-error-message + nix-protocol-error-message))) + (connect socket AF_UNIX + ,(inferior-bridge-file-name inferior)) - ;; 'port->connection' appeared in June 2018 and we can hardly - ;; emulate it on older versions. Thus fall back to - ;; 'open-connection', at the risk of talking to the wrong daemon or - ;; having our build result reclaimed (XXX). - (let ((store (if (defined? 'port->connection) - (port->connection socket #:version ,proto) - (open-connection)))) - (dynamic-wind - (const #t) - (lambda () - ;; Serialize '&store-protocol-error' conditions. The - ;; exception serialization mechanism that - ;; 'read-repl-response' expects is unsuitable for SRFI-35 - ;; error conditions, hence this special case. - (guard (c ((error? c) - `(store-protocol-error ,(error-message c)))) - `(result ,(proc store)))) - (lambda () - (close-connection store) - (close-port socket))))) - inferior) - (match (accept socket) - ((client . address) - (proxy client (store-connection-socket store)))) - (close-port socket) + ;; 'port->connection' appeared in June 2018 and we can hardly + ;; emulate it on older versions. Thus fall back to + ;; 'open-connection', at the risk of talking to the wrong daemon or + ;; having our build result reclaimed (XXX). + (let ((store (if (defined? 'port->connection) + (port->connection socket #:version ,proto) + (open-connection)))) + (dynamic-wind + (const #t) + (lambda () + ;; Serialize '&store-protocol-error' conditions. The + ;; exception serialization mechanism that + ;; 'read-repl-response' expects is unsuitable for SRFI-35 + ;; error conditions, hence this special case. + (guard (c ((error? c) + `(store-protocol-error ,(error-message c)))) + `(result ,(proc store)))) + (lambda () + (close-connection store) + (close-port socket))))) + inferior) + (match (accept (inferior-bridge-socket inferior)) + ((client . address) + (proxy client (store-connection-socket store)))) - (match (read-inferior-response inferior) - (('store-protocol-error message) - (raise (condition - (&store-protocol-error (message message) - (status 1))))) - (('result result) - result)))))) + (match (read-inferior-response inferior) + (('store-protocol-error message) + (raise (condition + (&store-protocol-error (message message) + (status 1))))) + (('result result) + result)))) (define* (inferior-package-derivation store package #:optional base-commit: 3993d33d1c0129b1ca6f0fd122fe2bbe48e4f093 -- 2.34.0 From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: [PATCH 4/4] inferior: Move initialization bits away from 'inferior-eval-with-store'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 27 Jan 2022 08:49:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 48007@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.164327330929812 (code B ref 48007); Thu, 27 Jan 2022 08:49:03 +0000 Received: (at 48007) by debbugs.gnu.org; 27 Jan 2022 08:48:29 +0000 Received: from localhost ([127.0.0.1]:54278 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0SC-0007kk-FB for submit@debbugs.gnu.org; Thu, 27 Jan 2022 03:48:28 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33152) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0S5-0007jz-VG for 48007@debbugs.gnu.org; Thu, 27 Jan 2022 03:48:22 -0500 Received: from [2001:470:142:3::e] (port=51734 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nD0S0-0000sf-Mg; Thu, 27 Jan 2022 03:48:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=H2hIJv5PNG3OcMRCKci/Y9So9WR1BjpdxSjK7R02pQg=; b=S6exTHx28ESeS7INk2KU z6ns1UJV29FqZO49gdvVgFwNISjFsTO9n2uxqYYmbZReqBbBqyYt1ZUu/lc2AC4QangjkrGLqa2bR eexqMxG8SnZ2NCgluzFyeQpzAj4kkxLrBfpVTYfNH+3V3Vt1ouOD683KOKnkbkpImSm+3MegnxPYE GZWkPhpR2pZIOkJ79GXtLqiWXS5sFe9NecgJyEQo+ngE7+oc95VWNsyrw75wXslRIkmCVjG6mglHt FLyvRuq+9eTpvqQT9jHRjIHuIfX20TmONJ7sjBTsP14MAg25ZQo/61g5jNQ3sq1hIbyNnflbDRQX9 jzKu6ppRa0y1gg==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:60148 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nD0S0-0002sQ-Ac; Thu, 27 Jan 2022 03:48:16 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 27 Jan 2022 09:47:43 +0100 Message-Id: <20220127084743.27130-4-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220127084743.27130-1-ludo@gnu.org> References: <87r18ufcft.fsf@gnu.org> <20220127084743.27130-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/inferior.scm (port->inferior): In the inferior, define 'cached-store-connection', 'store-protocol-error?', and 'store-protocol-error-message'. (inferior-eval-with-store): Use them. --- guix/inferior.scm | 76 ++++++++++++++++++++++++++--------------------- 1 file changed, 42 insertions(+), 34 deletions(-) diff --git a/guix/inferior.scm b/guix/inferior.scm index 64dd1ce9b6..fc253dcc4f 100644 --- a/guix/inferior.scm +++ b/guix/inferior.scm @@ -225,7 +225,39 @@ (define* (port->inferior pipe #:optional (close close-port)) (inferior-eval '(use-modules (srfi srfi-34)) result) (inferior-eval '(define %package-table (make-hash-table)) result) - (inferior-eval '(define %store-table (make-hash-table)) + (inferior-eval '(begin + (define %store-table (make-hash-table)) + (define (cached-store-connection store-id version) + ;; Cache connections to store ID. This ensures that + ;; the caches within (in + ;; particular the object cache) are reused across + ;; calls to 'inferior-eval-with-store', which makes a + ;; significant different when it is called + ;; repeatedly. + (or (hashv-ref %store-table store-id) + + ;; 'port->connection' appeared in June 2018 and + ;; we can hardly emulate it on older versions. + ;; Thus fall back to 'open-connection', at the + ;; risk of talking to the wrong daemon or having + ;; our build result reclaimed (XXX). + (let ((store (if (defined? 'port->connection) + (port->connection %bridge-socket + #:version + version) + (open-connection)))) + (hashv-set! %store-table store-id store) + store)))) + result) + (inferior-eval '(begin + (define store-protocol-error? + (if (defined? 'store-protocol-error?) + store-protocol-error? + nix-protocol-error?)) + (define store-protocol-error-message + (if (defined? 'store-protocol-error-message) + store-protocol-error-message + nix-protocol-error-message))) result) result)) (_ @@ -627,39 +659,15 @@ (define (inferior-eval-with-store inferior store code) (store-id (object-address (store-connection-socket store)))) (ensure-store-bridge! inferior) (send-inferior-request - `(let ((proc ,code) - (error? (if (defined? 'store-protocol-error?) - store-protocol-error? - nix-protocol-error?)) - (error-message (if (defined? 'store-protocol-error-message) - store-protocol-error-message - nix-protocol-error-message))) - - ;; Cache connections to STORE-ID. This ensures that the caches within - ;; (in particular the object cache) are reused - ;; across calls to 'inferior-eval-with-store', which makes a - ;; significant different when it is called repeatedly. - (let ((store (or (hashv-ref %store-table ,store-id) - - ;; 'port->connection' appeared in June 2018 and we - ;; can hardly emulate it on older versions. Thus - ;; fall back to 'open-connection', at the risk of - ;; talking to the wrong daemon or having our build - ;; result reclaimed (XXX). - (let ((store (if (defined? 'port->connection) - (port->connection %bridge-socket - #:version ,proto) - (open-connection)))) - (hashv-set! %store-table ,store-id store) - store)))) - - ;; Serialize '&store-protocol-error' conditions. The - ;; exception serialization mechanism that - ;; 'read-repl-response' expects is unsuitable for SRFI-35 - ;; error conditions, hence this special case. - (guard (c ((error? c) - `(store-protocol-error ,(error-message c)))) - `(result ,(proc store))))) + `(let ((proc ,code) + (store (cached-store-connection ,store-id ,proto))) + ;; Serialize '&store-protocol-error' conditions. The exception + ;; serialization mechanism that 'read-repl-response' expects is + ;; unsuitable for SRFI-35 error conditions, hence this special case. + (guard (c ((store-protocol-error? c) + `(store-protocol-error + ,(store-protocol-error-message c)))) + `(result ,(proc store)))) inferior) (proxy inferior store) -- 2.34.0 From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: [PATCH 3/4] inferior: Inferior caches store connections. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 27 Jan 2022 08:49:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 48007@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.164327330929818 (code B ref 48007); Thu, 27 Jan 2022 08:49:03 +0000 Received: (at 48007) by debbugs.gnu.org; 27 Jan 2022 08:48:29 +0000 Received: from localhost ([127.0.0.1]:54280 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0SC-0007km-Rg for submit@debbugs.gnu.org; Thu, 27 Jan 2022 03:48:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33150) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0S5-0007jx-C7 for 48007@debbugs.gnu.org; Thu, 27 Jan 2022 03:48:22 -0500 Received: from [2001:470:142:3::e] (port=51732 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nD0S0-0000sT-3W; Thu, 27 Jan 2022 03:48:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=WPijy0Tmt+Eb0k4WcLUzw/zNkq8zFUXPLJ8AhN9hpBI=; b=m+fkx0QKg41ZplmXmSGM VvS+ibRFx1RuknJiN8Q+m70R0mCv2WrsJY/fEGVrlFtE6Z60VGj+/uLMX/azWpjgYAzYQBo41prLz l4t6tH+lEsUxdk4lXlMi2p1+32AzUYa32DJqYHnEtApTHUg9+gpeJv0bxdIFPzABBMfCpb0DhDEoo RMFa2dMZ4mLPLokklP4HgXvh/pSixsRlfd7+OH3mPHoSqTzrI/kHyuWfL5OUSttCtWTOGP9FJBt2Y icJ+sUob5jc2OySQsOMhv3t/Emyzt+2D8YrVL5r/6BSaGCexz058xPflERUnECce5Er9GNbKl9zL9 Ir2L6etj6pnH0w==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:60148 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nD0Rw-0002sQ-Lr; Thu, 27 Jan 2022 03:48:15 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 27 Jan 2022 09:47:42 +0100 Message-Id: <20220127084743.27130-3-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220127084743.27130-1-ludo@gnu.org> References: <87r18ufcft.fsf@gnu.org> <20220127084743.27130-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Fixes . Reported by Ricardo Wurmus . Previously, at each 'inferior-eval-with-store' call, the inferior would create a new object with empty caches. Consequently, when repeatedly calling 'inferior-package-derivation', we would not benefit from any caching and instead recompute all the derivations for every package. This patch fixes it by caching objects in the inferior. * guix/inferior.scm (port->inferior): Define '%store-table' in the inferior. (inferior-eval-with-store): Cache store connections in %STORE-TABLE. Remove now unneeded 'dynamic-wind' with 'close-port' call. --- guix/inferior.scm | 54 +++++++++++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/guix/inferior.scm b/guix/inferior.scm index 1c19527b8f..64dd1ce9b6 100644 --- a/guix/inferior.scm +++ b/guix/inferior.scm @@ -225,6 +225,8 @@ (define* (port->inferior pipe #:optional (close close-port)) (inferior-eval '(use-modules (srfi srfi-34)) result) (inferior-eval '(define %package-table (make-hash-table)) result) + (inferior-eval '(define %store-table (make-hash-table)) + result) result)) (_ #f))) @@ -617,7 +619,12 @@ (define (inferior-eval-with-store inferior store code) thus be the code of a one-argument procedure that accepts a store." (let* ((major (store-connection-major-version store)) (minor (store-connection-minor-version store)) - (proto (logior major minor))) + (proto (logior major minor)) + + ;; The address of STORE itself is not a good identifier because it + ;; keeps changing through the use of "functional caches". The + ;; address of its socket port makes more sense. + (store-id (object-address (store-connection-socket store)))) (ensure-store-bridge! inferior) (send-inferior-request `(let ((proc ,code) @@ -628,26 +635,31 @@ (define (inferior-eval-with-store inferior store code) store-protocol-error-message nix-protocol-error-message))) - ;; 'port->connection' appeared in June 2018 and we can hardly - ;; emulate it on older versions. Thus fall back to - ;; 'open-connection', at the risk of talking to the wrong daemon or - ;; having our build result reclaimed (XXX). - (let ((store (if (defined? 'port->connection) - (port->connection %bridge-socket #:version ,proto) - (open-connection)))) - (dynamic-wind - (const #t) - (lambda () - ;; Serialize '&store-protocol-error' conditions. The - ;; exception serialization mechanism that - ;; 'read-repl-response' expects is unsuitable for SRFI-35 - ;; error conditions, hence this special case. - (guard (c ((error? c) - `(store-protocol-error ,(error-message c)))) - `(result ,(proc store)))) - (lambda () - (unless (defined? 'port->connection) - (close-port store)))))) + ;; Cache connections to STORE-ID. This ensures that the caches within + ;; (in particular the object cache) are reused + ;; across calls to 'inferior-eval-with-store', which makes a + ;; significant different when it is called repeatedly. + (let ((store (or (hashv-ref %store-table ,store-id) + + ;; 'port->connection' appeared in June 2018 and we + ;; can hardly emulate it on older versions. Thus + ;; fall back to 'open-connection', at the risk of + ;; talking to the wrong daemon or having our build + ;; result reclaimed (XXX). + (let ((store (if (defined? 'port->connection) + (port->connection %bridge-socket + #:version ,proto) + (open-connection)))) + (hashv-set! %store-table ,store-id store) + store)))) + + ;; Serialize '&store-protocol-error' conditions. The + ;; exception serialization mechanism that + ;; 'read-repl-response' expects is unsuitable for SRFI-35 + ;; error conditions, hence this special case. + (guard (c ((error? c) + `(store-protocol-error ,(error-message c)))) + `(result ,(proc store))))) inferior) (proxy inferior store) -- 2.34.0 From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: computing derivations through inferior takes twice as long Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 27 Jan 2022 08:51:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ricardo Wurmus Cc: 48007@debbugs.gnu.org Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.164327341930145 (code B ref 48007); Thu, 27 Jan 2022 08:51:01 +0000 Received: (at 48007) by debbugs.gnu.org; 27 Jan 2022 08:50:19 +0000 Received: from localhost ([127.0.0.1]:54301 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0Ty-0007q9-Px for submit@debbugs.gnu.org; Thu, 27 Jan 2022 03:50:18 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:36008) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD0Tx-0007pt-6M for 48007@debbugs.gnu.org; Thu, 27 Jan 2022 03:50:17 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 70B1C4F8; Thu, 27 Jan 2022 09:50:10 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Sk22AgZtReJr; Thu, 27 Jan 2022 09:50:09 +0100 (CET) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 35D06123; Thu, 27 Jan 2022 09:50:08 +0100 (CET) From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87wnsrpf25.fsf@elephly.net> <87r18ufcft.fsf@gnu.org> <87czkeji4h.fsf@elephly.net> Date: Thu, 27 Jan 2022 09:50:07 +0100 In-Reply-To: <87czkeji4h.fsf@elephly.net> (Ricardo Wurmus's message of "Wed, 26 Jan 2022 22:32:02 +0100") Message-ID: <87mtjhftq8.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / Authentication-Results: hera.aquilenet.fr; none X-Rspamd-Server: hera X-Rspamd-Queue-Id: 70B1C4F8 X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spam-Score: 1.0 (+) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi, Ricardo Wurmus skribis: > I tried it in the GWL with the big RNAseq workflow I adopted from PiGx > and the step to generate job scripts (which reference inferior packages) > became considerably faster. There is still potential for improvement, > but this change is the difference between not too bad (15 seconds) and > unusable (> 5 minutes). Good, that=E2=80=99s a step in the right direction. I=E2=80=99ve sent in this issue a cleaned up patch series that achieves the= same result, plus some micro-optimizations. It=E2=80=99d be great if you could confirm it still works for you. Thanks, Ludo=E2=80=99. From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: computing derivations through inferior takes twice as long Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 27 Jan 2022 09:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 48007@debbugs.gnu.org Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.16432774653960 (code B ref 48007); Thu, 27 Jan 2022 09:58:02 +0000 Received: (at 48007) by debbugs.gnu.org; 27 Jan 2022 09:57:45 +0000 Received: from localhost ([127.0.0.1]:54389 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD1XF-00011o-L6 for submit@debbugs.gnu.org; Thu, 27 Jan 2022 04:57:45 -0500 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21155) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD1XC-00011c-54 for 48007@debbugs.gnu.org; Thu, 27 Jan 2022 04:57:44 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1643277452; cv=none; d=zohomail.com; s=zohoarc; b=n3Bi9EnYMHKpgxvIlZfu4o/Ny2zf9z00YewYnvgpKp5Mq9+kzxtOOEg8LqXWYQMd+NMg4bQNt0q/NUrtR3Ggo2NULuQ3j0AULvd68o6pVBu4L5kU9doFhyCNIhEkT9CZe49quMb5QFJuA2EBlYc/AOvYNFLZQq+5WIzkdjJ4+08= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1643277452; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=CGR2oSHTVBJXUyRQPvQd5as0QOqXmU6uQtOTD+8wF0k=; b=RzjEsct4HKpUBgJs3rotHfsAIQSS81qGKoOw/XdAA3CUMtRJZwsk22Xf8frJf446L24x4POB3fC/pVehEbIZOSg6YdEjadzc7Act7k/rDuTo3JfeBOouNwB7++1F3NOBVfTGePduS/sKybApXkFdjRqtEm2rviMm73gz6++n7oc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1643277452; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=CGR2oSHTVBJXUyRQPvQd5as0QOqXmU6uQtOTD+8wF0k=; b=USeb6PkONopgUYI0ay7LjITcTuAoraZh2T4nR+rQAxYSJRDT8cPYnL3rcAjBRPAH 6Zsph1So/2fJcUlJuiHy5ggR1Ws/uIslX1Ko4jNIwaHY8XTOUO/8nrQYycTWvN7DcAX nizv0ZkHk9h8tt2JKxNabQNUoj0C87fDL/ql8flA= Received: from localhost (p54ad4ff6.dip0.t-ipconnect.de [84.173.79.246]) by mx.zohomail.com with SMTPS id 1643277449654233.21309897494416; Thu, 27 Jan 2022 01:57:29 -0800 (PST) References: <87wnsrpf25.fsf@elephly.net> <87r18ufcft.fsf@gnu.org> <87czkeji4h.fsf@elephly.net> <87mtjhftq8.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 From: Ricardo Wurmus Date: Thu, 27 Jan 2022 10:56:49 +0100 In-reply-to: <87mtjhftq8.fsf@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87a6fhijqy.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > Ricardo Wurmus skribis: > >> I tried it in the GWL with the big RNAseq workflow I adopted from PiGx >> and the step to generate job scripts (which reference inferior packages) >> became considerably faster. There is still potential for improvement, >> but this change is the difference between not too bad (15 seconds) and >> unusable (> 5 minutes). > > Good, that=E2=80=99s a step in the right direction. > > I=E2=80=99ve sent in this issue a cleaned up patch series that achieves t= he same > result, plus some micro-optimizations. It=E2=80=99d be great if you could > confirm it still works for you. These patches look great to me and they work. My real-world test case is now down to about 12 seconds. Thanks! --=20 Ricardo From unknown Sat Jun 14 19:22:56 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Ricardo Wurmus Subject: bug#48007: closed (Re: bug#48007: computing derivations through inferior takes twice as long) Message-ID: References: <877dalfgm0.fsf@gnu.org> <87wnsrpf25.fsf@elephly.net> X-Gnu-PR-Message: they-closed 48007 X-Gnu-PR-Package: guix Reply-To: 48007@debbugs.gnu.org Date: Thu, 27 Jan 2022 13:34:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1643290442-10519-1" This is a multi-part message in MIME format... ------------=_1643290442-10519-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #48007: computing derivations through inferior takes twice as long which was filed against the guix package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 48007@debbugs.gnu.org. --=20 48007: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D48007 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1643290442-10519-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 48007-done) by debbugs.gnu.org; 27 Jan 2022 13:33:41 +0000 Received: from localhost ([127.0.0.1]:54575 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD4uC-0002j4-Qu for submit@debbugs.gnu.org; Thu, 27 Jan 2022 08:33:40 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:42944) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nD4u8-0002ik-7V for 48007-done@debbugs.gnu.org; Thu, 27 Jan 2022 08:33:40 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 96722849; Thu, 27 Jan 2022 14:33:29 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0QTi-Nj3Kiwb; Thu, 27 Jan 2022 14:33:28 +0100 (CET) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 58D2F285; Thu, 27 Jan 2022 14:33:28 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Ricardo Wurmus Subject: Re: bug#48007: computing derivations through inferior takes twice as long References: <87wnsrpf25.fsf@elephly.net> <87r18ufcft.fsf@gnu.org> <87czkeji4h.fsf@elephly.net> <87mtjhftq8.fsf@gnu.org> <87a6fhijqy.fsf@elephly.net> Date: Thu, 27 Jan 2022 14:33:27 +0100 In-Reply-To: <87a6fhijqy.fsf@elephly.net> (Ricardo Wurmus's message of "Thu, 27 Jan 2022 10:56:49 +0100") Message-ID: <877dalfgm0.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / Authentication-Results: hera.aquilenet.fr; none X-Rspamd-Server: hera X-Rspamd-Queue-Id: 96722849 X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 48007-done Cc: 48007-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Ricardo Wurmus skribis: > These patches look great to me and they work. > My real-world test case is now down to about 12 seconds. Good! Fixed the typo you mentioned on IRC and pushed as e778910bdfc68c60a5be59aac93049d32feae904. To summarize, the INFERIOR=3Dy case still takes about twice as long as the INFERIOR=3Dn case (before that it was actually 9 times slower). I suppose this is mostly due to the round trips between the inferior and the parent (one per package). We=E2=80=99d have to analyze more closely, f= or example with =E2=80=98perf timechart=E2=80=99, where the wait times are. I= t=E2=80=99ll always be slower than without an inferior though. Last, we should improve the baseline (4s here for the manifest you gave). That=E2=80=99s the tricky part. Thanks, Ludo=E2=80=99. ------------=_1643290442-10519-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 24 Apr 2021 21:07:32 +0000 Received: from localhost ([127.0.0.1]:41599 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1laPUx-0007zN-Vb for submit@debbugs.gnu.org; Sat, 24 Apr 2021 17:07:32 -0400 Received: from lists.gnu.org ([209.51.188.17]:49268) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1laPUt-0007zE-VT for submit@debbugs.gnu.org; Sat, 24 Apr 2021 17:07:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43508) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1laPUt-00051o-R5 for bug-guix@gnu.org; Sat, 24 Apr 2021 17:07:27 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21173) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1laPUr-0005c6-EB for bug-guix@gnu.org; Sat, 24 Apr 2021 17:07:27 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1619298438; cv=none; d=zohomail.com; s=zohoarc; b=YHovs9S5Y7O4jQNv/rOISC/0cnBGJgLaFBv2J/EdYwqWWT2fgeZRaTuuVlDuUcdO7ApMNt2+YQ8vLzJJUsEVYcvFxOjtMe0EqXscvtoemrvDjJ/YUxq6thKQcS5K/+Cdy44b0tb3W1CxkOH+AFgyvbPbRM1qgnpSiK8a8It/Sek= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1619298438; h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To; bh=9qBBAAr+tRTZNB+gK0gY2h6OelR7KIfexvApyGh0bnw=; b=fX8+sdXE93nNzxL6di+97F1EMUlhsVatfwf5JQHS7YeigVSv574m9MwFCAEAz7EF/7bRNZ2TPQczFdNZ5EBUT2f01Fgruh2tlf3EI1TWBt6h3oWWZqfHy/GoxCCk8TBQb4h2a5rcU7X2JEmjnnj8VuaBAX/YzWbaspg4E1pRw5Q= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1619298438; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=9qBBAAr+tRTZNB+gK0gY2h6OelR7KIfexvApyGh0bnw=; b=Hn9hmuCaPhkGVVDWuYeaMgVFoZB/Qk8G5dkcrOzJ3AvjNYDiQKOUbVn+Tojmd0Bq XjxQ1tzecmjtzcUWnsksWcISHovoCWXlK1taokWSEuVnIAzsF0r4Yrp0ClZBdVVIHwj UtJmQdXWZLoTbqXeRC439CX5Qn+sY/BcaYUQSZ/I= Received: from localhost (p4fd5ab0c.dip0.t-ipconnect.de [79.213.171.12]) by mx.zohomail.com with SMTPS id 161929843743942.68504151230297; Sat, 24 Apr 2021 14:07:17 -0700 (PDT) User-agent: mu4e 1.4.15; emacs 27.2 From: Ricardo Wurmus To: bug-guix@gnu.org Subject: computing derivations through inferior takes twice as long X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Sat, 24 Apr 2021 23:07:14 +0200 Message-ID: <87wnsrpf25.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) This bug report might be related to bug #48005. In the Guix Workflow Language we are always looking up packages=20 through an inferior Guix. That Guix will in most cases be just=20 the current Guix. As I was looking for ways to speed the GWL up,=20 I noticed that the use of inferiors itself contributes to a=20 significant loss in performance. Here is a simple manifest to demonstrate this: --8<---------------cut here---------------start------------->8--- (use-modules (guix inferior) (ice-9 match)) (define inferior (open-inferior (format #false "~a/.config/guix" (getenv=20 "HOME")))) (define packages (list "bash-minimal" "r-corrplot" "r-crosstalk" "r-data-table" "r-deseq2" "r-dt" "r-genomicalignments" "r-genomicranges" "r-ggplot2" "r-ggrepel" "r-gprofiler" "r-knitr" "r-minimal" "r-pheatmap" "r-plotly" "r-reshape2" "r-rmarkdown" "r-rsamtools" "r-rtracklayer" "r-s4vectors" "r-scales" "r-summarizedexperiment" "r-tximport")) (match (getenv "INFERIOR") ("y" (packages->manifest (map (lambda (specification) (match (lookup-inferior-packages inferior=20 specification) ((first . rest) first))) packages))) (_ (specifications->manifest packages))) --8<---------------cut here---------------end--------------->8--- When INFERIOR is set to =E2=80=9Cy=E2=80=9D, each package specification wil= l be=20 looked up in the current Guix via an inferior. For any other=20 values of INFERIOR the specifications are resolved with the=20 current Guix (the very same Guix) directly. Here are the timings: --8<---------------cut here---------------start------------->8--- $ [env] export GUIX_PROFILING=3D"object-cache=20 add-data-to-store-cache rpc" $ [env] time INFERIOR=3Dn guix build -m manifest-test.scm -d /gnu/store/mwg47gbmi98bbrywk07y5l2h9p6d1hz5-bash-minimal-5.0.16.drv /gnu/store/kcjk6z128fa07pzp8irp6lbbyl3g16nr-r-corrplot-0.84.drv /gnu/store/s6hflcww9gaq87g5vaaydd4lphw63xjm-r-crosstalk-1.1.1.drv /gnu/store/qrjgag94sv9lq12028y9iv12j75bva6c-r-data-table-1.14.0.drv /gnu/store/v6xw6pg33xa8pg19nw0cxhz9b7ps26v7-r-deseq2-1.30.1.drv /gnu/store/q1achql92wnij108msymr9mkr8pv2z1h-r-dt-0.17.drv /gnu/store/iym2kzpjiqch22yrhg5lnv9sfazdfphn-r-genomicalignments-1.26.0.drv /gnu/store/k913mn4q11pchgi63xrm8lb3svvqjcix-r-genomicranges-1.42.0.drv /gnu/store/zkpabp1qx6m5yam3f9kninnsxagsgwqh-r-ggplot2-3.3.3.drv /gnu/store/b6w1p6rhbk8shz1ydc2yqb38ypm0ijq9-r-ggrepel-0.9.1.drv /gnu/store/bwmmls5qkf9cfs9m73qzabnr7w5jc8ra-r-gprofiler-0.7.0.drv /gnu/store/j1m8hb4449rkfh3ij1l4379j1lngjr06-r-knitr-1.31.drv /gnu/store/7ig30kf3i65s3rdcw1qik7vsjvspkjxy-r-minimal-4.0.4.drv /gnu/store/mwg8c42sfsvcrbjhbw7mbdcphhz9hq3x-r-pheatmap-1.0.12.drv /gnu/store/xjg40q7a7yl3l9v99kqapjylfjwapwk7-r-plotly-4.9.3.drv /gnu/store/fhs8as885izfb1r6as07sn6jpjgfbl58-r-reshape2-1.4.4.drv /gnu/store/6bcny1hhf83k85js6x3w7h1w3660ii8m-r-rmarkdown-2.7.drv /gnu/store/87pr587bk9rzfkrjmrm4bcfjz95p1n9c-r-rsamtools-2.6.0.drv /gnu/store/l3ibbpd4h7gm565vidbpyamdnhb0czhp-r-rtracklayer-1.50.0.drv /gnu/store/8rf8d204kavcxkw6z71kxd2mzzqzxsk1-r-s4vectors-0.28.1.drv /gnu/store/4nxw4lhcvj3q9j5v6mq9ri4v4vwmxd6h-r-scales-1.1.1.drv /gnu/store/vpf3vkj58vwz92nxcpppil6580c84bb1-r-summarizedexperiment-1.20.0.d= rv /gnu/store/cx3cl0nxwvzpaj484q2xcnz3v7zc1015-r-tximport-1.18.0.drv Store object cache: fresh caches: 2 lookups: 4540 hits: 3568 (78.6%) cache size: 971 entries 'add-data-to-store' cache: lookups: 3450 hits: 492 (14.3%) .drv files: 2087 (60.5%) Scheme files: 1347 (39.0%) Remote procedure call summary: 3412 RPCs built-in-builders ... 1 add-to-store/tree ... 16 add-to-store ... 177 query-references ... 260 add-text-to-store ... 2958 real 0m3.970s user 0m4.055s sys 0m0.173s $ [env] time INFERIOR=3Dy guix build -m manifest-test.scm -d /gnu/store/mwg47gbmi98bbrywk07y5l2h9p6d1hz5-bash-minimal-5.0.16.drv /gnu/store/hmk49rhbfqw2ss55392a7kq34xqg18i7-r-corrplot-0.84.drv /gnu/store/sg8a3pvzxaq2qd4z918mdb2y0vq6w8mg-r-crosstalk-1.1.1.drv /gnu/store/n3vk2kkq7zza7pfrjqqbv6xaxhnzdn2x-r-data-table-1.14.0.drv /gnu/store/44fqdg0s6bcmcgafvgafycf2x82rfl7y-r-deseq2-1.30.1.drv /gnu/store/03snyvyp9fr3nchrln6qhdca00i7lrsz-r-dt-0.17.drv /gnu/store/rl48alwm40sl4b04rnk4cck2h4crr8gc-r-genomicalignments-1.26.0.drv /gnu/store/ryl6hjflgpb72xl91jvp0ab6sl5cblc4-r-genomicranges-1.42.0.drv /gnu/store/1hbg746cvi8s7vn03glzx46m0pdih5pw-r-ggplot2-3.3.3.drv /gnu/store/nwvkjb314hh7z7vag0mk870isynp0hda-r-ggrepel-0.9.1.drv /gnu/store/kvvygkc7vnznrqp4n2rvgsbz9z2jd6ns-r-gprofiler-0.7.0.drv /gnu/store/0jv2zf34b2p1ddpxnzv5smq4717i4hfq-r-knitr-1.31.drv /gnu/store/zgi8sfw54jv7wb33q9cs18ff1vlfy0fm-r-minimal-4.0.4.drv /gnu/store/7w4jp2skqy0vn8i4pr26l94mw8vs8knc-r-pheatmap-1.0.12.drv /gnu/store/xshkhmd8gpjkmi7npz0bw02wgb8mkysg-r-plotly-4.9.3.drv /gnu/store/5jqkb3khygfc2y96nff92hfslc2c53yz-r-reshape2-1.4.4.drv /gnu/store/x0fzqyjg1hq7a4n0wglr9sl71bzxwz0q-r-rmarkdown-2.7.drv /gnu/store/3v78408vx5x28nb3cf42jarr7fy3b16v-r-rsamtools-2.6.0.drv /gnu/store/qp4hjddv5sjxiiss0m55q4cv88k520gd-r-rtracklayer-1.50.0.drv /gnu/store/pgfahjz3wfnppc07z0qbcsdc6mmpri0l-r-s4vectors-0.28.1.drv /gnu/store/aq317mqb3rbc2rnq2y15k781q5qvf9ia-r-scales-1.1.1.drv /gnu/store/w9dirjkx523398mhkjw0v4hxgq7x0b8s-r-summarizedexperiment-1.20.0.d= rv /gnu/store/rfmzii8xsc3fk63s332ix2qgxpvdvrgf-r-tximport-1.18.0.drv Store object cache: fresh caches: 1 lookups: 23 hits: 0 (0.0%) cache size: 23 entries 'add-data-to-store' cache: lookups: 0 hits: 0 (100.0%) .drv files: 0 (100.0%) Scheme files: 0 (100.0%) Remote procedure call summary: 0 RPCs real 0m7.951s user 0m2.191s sys 0m0.240s --8<---------------cut here---------------end--------------->8--- Note that nothing is built. This is merely to compute already=20 existing derivations. Computing the same derivations through an=20 inferior Guix takes about twice as long. I=E2=80=99ll note that in the inferior case there are no=20 =E2=80=9Cadd-data-to-store=E2=80=9D calls compared to 2958 calls in the dir= ect=20 case. I don=E2=80=99t know what to make of this. Is that information=20 lost as we cross over to the inferior Guix=E2=80=A6? Or are there really=20 different / fewer RPCs? Things look similar when we actually instantiate the manifest: --8<---------------cut here---------------start------------->8--- $ [env] time INFERIOR=3Dn guix package -m manifest-test.scm -p=20 /tmp/foo The following packages will be installed: bash-minimal 5.0.16 r-corrplot 0.84 r-crosstalk 1.1.1 r-data-table 1.14.0 r-deseq2 1.30.1 r-dt 0.17 r-genomicalignments 1.26.0 r-genomicranges 1.42.0 r-ggplot2 3.3.3 r-ggrepel 0.9.1 r-gprofiler 0.7.0 r-knitr 1.31 r-minimal 4.0.4 r-pheatmap 1.0.12 r-plotly 4.9.3 r-reshape2 1.4.4 r-rmarkdown 2.7 r-rsamtools 2.6.0 r-rtracklayer 1.50.0 r-s4vectors 0.28.1 r-scales 1.1.1 r-summarizedexperiment 1.20.0 r-tximport 1.18.0 nothing to be done Store object cache: fresh caches: 2 lookups: 41381 hits: 40249 (97.3%) cache size: 1131 entries 'add-data-to-store' cache: lookups: 6083 hits: 2950 (48.5%) .drv files: 3407 (56.0%) Scheme files: 2659 (43.7%) Remote procedure call summary: 3897 RPCs built-in-builders ... 1 add-to-store/tree ... 22 add-to-store ... 178 query-references ... 563 add-text-to-store ... 3133 real 0m12.697s user 0m15.873s sys 0m0.197s $ [env] time INFERIOR=3Dy guix package -m manifest-test.scm -p=20 /tmp/foo The following packages will be installed: bash-minimal 5.0.16 r-corrplot 0.84 r-crosstalk 1.1.1 r-data-table 1.14.0 r-deseq2 1.30.1 r-dt 0.17 r-genomicalignments 1.26.0 r-genomicranges 1.42.0 r-ggplot2 3.3.3 r-ggrepel 0.9.1 r-gprofiler 0.7.0 r-knitr 1.31 r-minimal 4.0.4 r-pheatmap 1.0.12 r-plotly 4.9.3 r-reshape2 1.4.4 r-rmarkdown 2.7 r-rsamtools 2.6.0 r-rtracklayer 1.50.0 r-s4vectors 0.28.1 r-scales 1.1.1 r-summarizedexperiment 1.20.0 r-tximport 1.18.0 nothing to be done Store object cache: fresh caches: 2 lookups: 27162 hits: 26425 (97.3%) cache size: 736 entries 'add-data-to-store' cache: lookups: 887 hits: 52 (5.9%) .drv files: 550 (62.0%) Scheme files: 331 (37.3%) Remote procedure call summary: 1011 RPCs built-in-builders ... 1 add-to-store/tree ... 11 query-references ... 51 add-to-store ... 113 add-text-to-store ... 835 real 0m19.504s user 0m4.014s sys 0m0.411s --8<---------------cut here---------------end--------------->8--- The first case with 12 seconds reproduces bug #48005. The second=20 case (going through the inferior) is much slower with over 19=20 seconds; if we squint this is also close to twice the amount of=20 time compared to the =E2=80=9Cdirect=E2=80=9D computation. --=20 Ricardo ------------=_1643290442-10519-1-- From unknown Sat Jun 14 19:22:56 2025 X-Loop: help-debbugs@gnu.org Subject: bug#48007: computing derivations through inferior takes twice as long Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 18 Feb 2022 11:31:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48007 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 48007@debbugs.gnu.org Received: via spool by 48007-submit@debbugs.gnu.org id=B48007.16451838474968 (code B ref 48007); Fri, 18 Feb 2022 11:31:02 +0000 Received: (at 48007) by debbugs.gnu.org; 18 Feb 2022 11:30:47 +0000 Received: from localhost ([127.0.0.1]:53555 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nL1TL-0001I4-2h for submit@debbugs.gnu.org; Fri, 18 Feb 2022 06:30:47 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:35040) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nL1TJ-0001Hp-N6 for 48007@debbugs.gnu.org; Fri, 18 Feb 2022 06:30:46 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id E4F1B3A7 for <48007@debbugs.gnu.org>; Fri, 18 Feb 2022 12:30:38 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FyvkSUgPuUz2 for <48007@debbugs.gnu.org>; Fri, 18 Feb 2022 12:30:37 +0100 (CET) Received: from ribbon (unknown [IPv6:2001:660:6102:320:e120:2c8f:8909:cdfe]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 2F9311A4 for <48007@debbugs.gnu.org>; Fri, 18 Feb 2022 12:30:37 +0100 (CET) From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87r18ufcft.fsf@gnu.org> <20220127084743.27130-1-ludo@gnu.org> <20220127084743.27130-2-ludo@gnu.org> Date: Fri, 18 Feb 2022 12:30:36 +0100 In-Reply-To: <20220127084743.27130-2-ludo@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Thu, 27 Jan 2022 09:47:41 +0100") Message-ID: <87k0dsjtsz.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / Authentication-Results: hera.aquilenet.fr; none X-Rspamd-Server: hera X-Rspamd-Queue-Id: E4F1B3A7 X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[48007@debbugs.gnu.org]; RCPT_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spam-Score: 1.0 (+) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi, Ludovic Court=C3=A8s skribis: > Previously, each 'inferior-eval-with-store' would lead the inferior to > connect to the named socket the parent is listening to. With this > change, the connection is established once for all and reused > afterwards. > > * guix/inferior.scm ()[bridge-file-name]: Remove. > (open-bidirectional-pipe): New procedure. > (inferior-pipe): Use it instead of 'open-pipe*' and return two values. > (port->inferior): Adjust call to 'inferior'. > (open-inferior): Adjust to 'inferior-pipe' changes. > (close-inferior): Remove 'inferior-bridge-file-name' handling. > (open-store-bridge!): Switch back to 'call-with-temporary-directory'. > Define '%bridge-socket' in the inferior, connected to the caller. > (proxy): Change first argument to be an inferior. Add 'reponse-port' > and call to 'drain-input'. Pass 'reponse-port' to 'select' and use it > as a loop termination clause. > (inferior-eval-with-store): Remove 'socket' and 'connect' calls from the > inferior code, and use '%bridge-socket' instead. [...] > +(define (open-bidirectional-pipe command . args) > + "Open a bidirectional pipe to COMMAND invoked with ARGS and return it,= as a > +regular file port (socket). > + > +This is equivalent to (open-pipe* OPEN_BOTH ...) except that the result = is a > +regular file port that can be passed to 'select' ('open-pipe*' returns a > +custom binary port)." > + (match (socketpair AF_UNIX SOCK_STREAM 0) > + ((parent . child) > + (match (primitive-fork) I noticed that there=E2=80=99s at least one case where this is used from a multi-threaded program, and as we know, fork + threads don=E2=80=99t go well together: --8<---------------cut here---------------start------------->8--- $ make as-derivation [=E2=80=A6] @ build-succeeded /gnu/store/n5jfi8pn1aq1ykmnq75xhr8ba2m7161l-profile.drv - warning: call to primitive-fork while multiple threads are running; further behavior unspecified. See "Processes" in the manual, for more information. warning: call to primitive-fork while multiple threads are running; further behavior unspecified. See "Processes" in the manual, for more information. warning: call to primitive-fork while multiple threads are running; further behavior unspecified. See "Processes" in the manual, for more information. --8<---------------cut here---------------end--------------->8--- The threads are created in =E2=80=98build-aux/cuirass/evaluate.scm=E2=80=99. In practice it=E2=80=99s OK because the code above calls =E2=80=98exec=E2= =80=99 right away; still, it=E2=80=99s annoying. Ludo=E2=80=99.