From unknown Sun Jun 22 00:53:57 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#67238] [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Fri, 17 Nov 2023 11:36:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 67238 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 67238@debbugs.gnu.org Cc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by submit@debbugs.gnu.org id=B.170022095913047 (code B ref -1); Fri, 17 Nov 2023 11:36:02 +0000 Received: (at submit) by debbugs.gnu.org; 17 Nov 2023 11:35:59 +0000 Received: from localhost ([127.0.0.1]:45431 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3x8h-0003OL-1t for submit@debbugs.gnu.org; Fri, 17 Nov 2023 06:35:59 -0500 Received: from lists.gnu.org ([2001:470:142::17]:48978) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3x8c-0003Ns-80 for submit@debbugs.gnu.org; Fri, 17 Nov 2023 06:35:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r3x8R-0000uH-2x for guix-patches@gnu.org; Fri, 17 Nov 2023 06:35:47 -0500 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r3x8O-0003Se-Jf for guix-patches@gnu.org; Fri, 17 Nov 2023 06:35:42 -0500 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 8D8A427BBE2 for ; Fri, 17 Nov 2023 11:35:35 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 6d79035a for ; Fri, 17 Nov 2023 11:35:34 +0000 (UTC) From: Christopher Baines Date: Fri, 17 Nov 2023 11:35:34 +0000 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.9 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.1 (/) strace -c reports over 10,000 readlink syscalls when reading the derivation for the hello package. By just setting the %file-port-name-canonicalization fluid, this drops to less than 10. I'm not sure if this actually improves performance, but doing less is surely better. * guix/derivations.scm (read-derivation-from-file): Set %file-port-name-canonicalization to 'none when calling call-with-input-file. Change-Id: I1ff16a059160576a576f2e9ed881379596e66af3 --- guix/derivations.scm | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/guix/derivations.scm b/guix/derivations.scm index 9fec7f4f0b..e6ecb570c4 100644 --- a/guix/derivations.scm +++ b/guix/derivations.scm @@ -556,7 +556,12 @@ (define (read-derivation-from-file file) ;; and because the same argument is read more than 15 times on average ;; during something like (package-derivation s gdb). (or (and file (hash-ref %derivation-cache file)) - (let ((drv (call-with-input-file file read-derivation))) + (let ((drv + ;; Avoid calling scm_i_relativize_path in + ;; fport_canonicalize_filename since this leads to lots of + ;; readlink calls + (with-fluids ((%file-port-name-canonicalization 'none)) + (call-with-input-file file read-derivation)))) (hash-set! %derivation-cache file drv) drv))) base-commit: e35b7c5386c1bfacf47ed31bac9b503373dd26fc -- 2.41.0 From unknown Sun Jun 22 00:53:57 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#67238] [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 Nov 2023 11:29:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 67238 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , 67238@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines Received: via spool by 67238-submit@debbugs.gnu.org id=B67238.170082533427076 (code B ref 67238); Fri, 24 Nov 2023 11:29:02 +0000 Received: (at 67238) by debbugs.gnu.org; 24 Nov 2023 11:28:54 +0000 Received: from localhost ([127.0.0.1]:35740 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r6UMf-00072e-RC for submit@debbugs.gnu.org; Fri, 24 Nov 2023 06:28:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:40574) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r6UMc-00072K-Tk for 67238@debbugs.gnu.org; Fri, 24 Nov 2023 06:28:51 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r6UMQ-0005Za-IX; Fri, 24 Nov 2023 06:28:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=lV+wie1Hz+eT9Os7+Kd4UTIOwcQWhnvAmQfP1rzEENE=; b=kV3At4pVBu8or4xVrbNp Yq6fpnqBnfB5KFyMFXym2oOmlkJvKUYbASzm0wMTcMKHdzTLtbLrJG3u1JoNFDhltWX1ICEUY4fz3 WARQPe6XiFsTqNCxrjXiPg5/NQpx/s12gcDNn5WJq4PmqTKJz+t7ocknclgfPCr/W6mt4re1H9qbN MuI+5Ax0lRfL+0tEyBgxEA1m/i9Vq0ap0x39xXKISMhXjPgjxaS8R0orAhjcd+A90oCI+nXo7WQDn 8Ygz2mu+jYgKNOhRLb9zqi+J2jZjhCWIvL6X5JY7MMNVQtDHA1eO/WgQBx2XOQbjyU0YNIg6OVDWv EFdwm9HiAPC3Zg==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: (Christopher Baines's message of "Fri, 17 Nov 2023 11:35:34 +0000") References: Date: Fri, 24 Nov 2023 12:28:33 +0100 Message-ID: <87wmu775lq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Christopher Baines skribis: > strace -c reports over 10,000 readlink syscalls when reading the derivati= on > for the hello package. By just setting the %file-port-name-canonicalizati= on > fluid, this drops to less than 10. > > I'm not sure if this actually improves performance, but doing less is sur= ely > better. > > * guix/derivations.scm (read-derivation-from-file): Set > %file-port-name-canonicalization to 'none when calling call-with-input-fi= le. > > Change-Id: I1ff16a059160576a576f2e9ed881379596e66af3 [...] > + (let ((drv > + ;; Avoid calling scm_i_relativize_path in > + ;; fport_canonicalize_filename since this leads to lots of > + ;; readlink calls > + (with-fluids ((%file-port-name-canonicalization 'none)) > + (call-with-input-file file read-derivation)))) This is already done in =E2=80=98run-guix=E2=80=99 in (guix ui), for all th= e =E2=80=98guix=E2=80=99 commands (so this patch would be a slight performance regression for Guix itself). I=E2=80=99d suggest setting this fluid globally in applications that use Gu= ix (the Build Coordinator, etc.), as is done in Guix itself. WDYT? Ludo=E2=80=99. From unknown Sun Jun 22 00:53:57 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Christopher Baines Subject: bug#67238: closed (Re: [bug#67238] [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file.) Message-ID: References: <87edgdmtew.fsf@cbaines.net> X-Gnu-PR-Message: they-closed 67238 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 67238@debbugs.gnu.org Date: Sat, 25 Nov 2023 21:10:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1700946602-14967-1" This is a multi-part message in MIME format... ------------=_1700946602-14967-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #67238: [PATCH] derivations: Avoid readlink syscalls in read-derivation-fro= m-file. which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 67238@debbugs.gnu.org. --=20 67238: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D67238 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1700946602-14967-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 67238-close) by debbugs.gnu.org; 25 Nov 2023 21:09:56 +0000 Received: from localhost ([127.0.0.1]:40359 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r6zuV-0003sx-M7 for submit@debbugs.gnu.org; Sat, 25 Nov 2023 16:09:56 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43048) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r6zuS-0003so-QP for 67238-close@debbugs.gnu.org; Sat, 25 Nov 2023 16:09:53 -0500 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 3A23227BBE2; Sat, 25 Nov 2023 21:09:46 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 0ce3ae0a; Sat, 25 Nov 2023 21:09:45 +0000 (UTC) References: <87wmu775lq.fsf@gnu.org> User-agent: mu4e 1.10.7; emacs 29.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#67238] [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file. Date: Sat, 25 Nov 2023 20:58:03 +0000 In-reply-to: <87wmu775lq.fsf@gnu.org> Message-ID: <87edgdmtew.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 67238-close Cc: 67238-close@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi, > > Christopher Baines skribis: > >> strace -c reports over 10,000 readlink syscalls when reading the derivat= ion >> for the hello package. By just setting the %file-port-name-canonicalizat= ion >> fluid, this drops to less than 10. >> >> I'm not sure if this actually improves performance, but doing less is su= rely >> better. >> >> * guix/derivations.scm (read-derivation-from-file): Set >> %file-port-name-canonicalization to 'none when calling call-with-input-f= ile. >> >> Change-Id: I1ff16a059160576a576f2e9ed881379596e66af3 > > [...] > >> + (let ((drv >> + ;; Avoid calling scm_i_relativize_path in >> + ;; fport_canonicalize_filename since this leads to lots of >> + ;; readlink calls >> + (with-fluids ((%file-port-name-canonicalization 'none)) >> + (call-with-input-file file read-derivation)))) > > This is already done in =E2=80=98run-guix=E2=80=99 in (guix ui), for all = the =E2=80=98guix=E2=80=99 > commands (so this patch would be a slight performance regression for > Guix itself). > > I=E2=80=99d suggest setting this fluid globally in applications that use = Guix > (the Build Coordinator, etc.), as is done in Guix itself. > > WDYT? Ah, I didn't realise it was already set for Guix scripts. But yeah, setting it in other places that read derivations makes sense. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmViYpdfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfcDhAAhpy+TsA6QPr+tmBAZrLtMocrvkU7DHyi HTfdD9lTL082HX9S3R7O7cAcMADiUOUNJfeIMA7NWfgZDBcksgEYlKKM6vff2HFP uylS4tiwOiURPl70HUm3wlLbPrjXO3yVfpNuSgXjxy8ZFVqFUyhFxT5En96+yZj2 pD4yUxbff2J13kO3H4dvgnDM3lQpuQkxP//Scn20/cJRT85ADniDTUXTilRpP0JZ dG0/1w2fDgvfnsOBltPAyRgRZG0chvWD6amNVx19t0rIdDZ2JYhlMJTFpnSZsi5H InxpJcuFEaiHkwOGGe+XJghqckiJHuKBbiXa8PJ8tN1UbwioFLuotcHz1cEct9VQ Zm3mZi0+aZBfrAml+bOz7gFC/mU3uscvZEOjHw6pj4lPtdxAmOWc+/jG794SD+rf oO/SE7V0yWib/EIZX+saUlWPMeAlooLhKHyG/7nhdHqDExYRajmCZ7x9gf740CV7 PGwFXaHTC8yJllZYvszS+VM19KfL18M+koNruE8Ta0deFDPW12VHAm+s955UDkIA 1zFPpc777g1xzKt+EYwMUzpIegb3JBDDlMukUMiD2v4Wn28mNY2gKv3BdJz5Wld7 qBLRBWwbcC6Z2PuS2HFTvlGXNFP2OD6I/gmoF2nK38A1fbVfkGQTDDrHBvB3Wobv uR3xDz01X2E= =Ct4r -----END PGP SIGNATURE----- --=-=-=-- ------------=_1700946602-14967-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 17 Nov 2023 11:35:59 +0000 Received: from localhost ([127.0.0.1]:45431 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3x8h-0003OL-1t for submit@debbugs.gnu.org; Fri, 17 Nov 2023 06:35:59 -0500 Received: from lists.gnu.org ([2001:470:142::17]:48978) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3x8c-0003Ns-80 for submit@debbugs.gnu.org; Fri, 17 Nov 2023 06:35:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r3x8R-0000uH-2x for guix-patches@gnu.org; Fri, 17 Nov 2023 06:35:47 -0500 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r3x8O-0003Se-Jf for guix-patches@gnu.org; Fri, 17 Nov 2023 06:35:42 -0500 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 8D8A427BBE2 for ; Fri, 17 Nov 2023 11:35:35 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 6d79035a for ; Fri, 17 Nov 2023 11:35:34 +0000 (UTC) From: Christopher Baines To: guix-patches@gnu.org Subject: [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file. Date: Fri, 17 Nov 2023 11:35:34 +0000 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.9 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.1 (/) strace -c reports over 10,000 readlink syscalls when reading the derivation for the hello package. By just setting the %file-port-name-canonicalization fluid, this drops to less than 10. I'm not sure if this actually improves performance, but doing less is surely better. * guix/derivations.scm (read-derivation-from-file): Set %file-port-name-canonicalization to 'none when calling call-with-input-file. Change-Id: I1ff16a059160576a576f2e9ed881379596e66af3 --- guix/derivations.scm | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/guix/derivations.scm b/guix/derivations.scm index 9fec7f4f0b..e6ecb570c4 100644 --- a/guix/derivations.scm +++ b/guix/derivations.scm @@ -556,7 +556,12 @@ (define (read-derivation-from-file file) ;; and because the same argument is read more than 15 times on average ;; during something like (package-derivation s gdb). (or (and file (hash-ref %derivation-cache file)) - (let ((drv (call-with-input-file file read-derivation))) + (let ((drv + ;; Avoid calling scm_i_relativize_path in + ;; fport_canonicalize_filename since this leads to lots of + ;; readlink calls + (with-fluids ((%file-port-name-canonicalization 'none)) + (call-with-input-file file read-derivation)))) (hash-set! %derivation-cache file drv) drv))) base-commit: e35b7c5386c1bfacf47ed31bac9b503373dd26fc -- 2.41.0 ------------=_1700946602-14967-1--