From unknown Sat Jun 21 03:22:38 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#67238 <67238@debbugs.gnu.org> To: bug#67238 <67238@debbugs.gnu.org> Subject: Status: [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file. Reply-To: bug#67238 <67238@debbugs.gnu.org> Date: Sat, 21 Jun 2025 10:22:38 +0000 retitle 67238 [PATCH] derivations: Avoid readlink syscalls in read-derivati= on-from-file. reassign 67238 guix-patches submitter 67238 Christopher Baines severity 67238 normal tag 67238 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 17 06:35:59 2023 Received: (at submit) by debbugs.gnu.org; 17 Nov 2023 11:35:59 +0000 Received: from localhost ([127.0.0.1]:45431 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3x8h-0003OL-1t for submit@debbugs.gnu.org; Fri, 17 Nov 2023 06:35:59 -0500 Received: from lists.gnu.org ([2001:470:142::17]:48978) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3x8c-0003Ns-80 for submit@debbugs.gnu.org; Fri, 17 Nov 2023 06:35:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r3x8R-0000uH-2x for guix-patches@gnu.org; Fri, 17 Nov 2023 06:35:47 -0500 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r3x8O-0003Se-Jf for guix-patches@gnu.org; Fri, 17 Nov 2023 06:35:42 -0500 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 8D8A427BBE2 for ; Fri, 17 Nov 2023 11:35:35 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 6d79035a for ; Fri, 17 Nov 2023 11:35:34 +0000 (UTC) From: Christopher Baines To: guix-patches@gnu.org Subject: [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file. Date: Fri, 17 Nov 2023 11:35:34 +0000 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.9 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.1 (/) strace -c reports over 10,000 readlink syscalls when reading the derivation for the hello package. By just setting the %file-port-name-canonicalization fluid, this drops to less than 10. I'm not sure if this actually improves performance, but doing less is surely better. * guix/derivations.scm (read-derivation-from-file): Set %file-port-name-canonicalization to 'none when calling call-with-input-file. Change-Id: I1ff16a059160576a576f2e9ed881379596e66af3 --- guix/derivations.scm | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/guix/derivations.scm b/guix/derivations.scm index 9fec7f4f0b..e6ecb570c4 100644 --- a/guix/derivations.scm +++ b/guix/derivations.scm @@ -556,7 +556,12 @@ (define (read-derivation-from-file file) ;; and because the same argument is read more than 15 times on average ;; during something like (package-derivation s gdb). (or (and file (hash-ref %derivation-cache file)) - (let ((drv (call-with-input-file file read-derivation))) + (let ((drv + ;; Avoid calling scm_i_relativize_path in + ;; fport_canonicalize_filename since this leads to lots of + ;; readlink calls + (with-fluids ((%file-port-name-canonicalization 'none)) + (call-with-input-file file read-derivation)))) (hash-set! %derivation-cache file drv) drv))) base-commit: e35b7c5386c1bfacf47ed31bac9b503373dd26fc -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 24 06:28:54 2023 Received: (at 67238) by debbugs.gnu.org; 24 Nov 2023 11:28:54 +0000 Received: from localhost ([127.0.0.1]:35740 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r6UMf-00072e-RC for submit@debbugs.gnu.org; Fri, 24 Nov 2023 06:28:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:40574) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r6UMc-00072K-Tk for 67238@debbugs.gnu.org; Fri, 24 Nov 2023 06:28:51 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r6UMQ-0005Za-IX; Fri, 24 Nov 2023 06:28:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=lV+wie1Hz+eT9Os7+Kd4UTIOwcQWhnvAmQfP1rzEENE=; b=kV3At4pVBu8or4xVrbNp Yq6fpnqBnfB5KFyMFXym2oOmlkJvKUYbASzm0wMTcMKHdzTLtbLrJG3u1JoNFDhltWX1ICEUY4fz3 WARQPe6XiFsTqNCxrjXiPg5/NQpx/s12gcDNn5WJq4PmqTKJz+t7ocknclgfPCr/W6mt4re1H9qbN MuI+5Ax0lRfL+0tEyBgxEA1m/i9Vq0ap0x39xXKISMhXjPgjxaS8R0orAhjcd+A90oCI+nXo7WQDn 8Ygz2mu+jYgKNOhRLb9zqi+J2jZjhCWIvL6X5JY7MMNVQtDHA1eO/WgQBx2XOQbjyU0YNIg6OVDWv EFdwm9HiAPC3Zg==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: [bug#67238] [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file. In-Reply-To: (Christopher Baines's message of "Fri, 17 Nov 2023 11:35:34 +0000") References: Date: Fri, 24 Nov 2023 12:28:33 +0100 Message-ID: <87wmu775lq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 67238 Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , 67238@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Christopher Baines skribis: > strace -c reports over 10,000 readlink syscalls when reading the derivati= on > for the hello package. By just setting the %file-port-name-canonicalizati= on > fluid, this drops to less than 10. > > I'm not sure if this actually improves performance, but doing less is sur= ely > better. > > * guix/derivations.scm (read-derivation-from-file): Set > %file-port-name-canonicalization to 'none when calling call-with-input-fi= le. > > Change-Id: I1ff16a059160576a576f2e9ed881379596e66af3 [...] > + (let ((drv > + ;; Avoid calling scm_i_relativize_path in > + ;; fport_canonicalize_filename since this leads to lots of > + ;; readlink calls > + (with-fluids ((%file-port-name-canonicalization 'none)) > + (call-with-input-file file read-derivation)))) This is already done in =E2=80=98run-guix=E2=80=99 in (guix ui), for all th= e =E2=80=98guix=E2=80=99 commands (so this patch would be a slight performance regression for Guix itself). I=E2=80=99d suggest setting this fluid globally in applications that use Gu= ix (the Build Coordinator, etc.), as is done in Guix itself. WDYT? Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat Nov 25 16:09:56 2023 Received: (at 67238-close) by debbugs.gnu.org; 25 Nov 2023 21:09:56 +0000 Received: from localhost ([127.0.0.1]:40359 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r6zuV-0003sx-M7 for submit@debbugs.gnu.org; Sat, 25 Nov 2023 16:09:56 -0500 Received: from mira.cbaines.net ([212.71.252.8]:43048) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r6zuS-0003so-QP for 67238-close@debbugs.gnu.org; Sat, 25 Nov 2023 16:09:53 -0500 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 3A23227BBE2; Sat, 25 Nov 2023 21:09:46 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 0ce3ae0a; Sat, 25 Nov 2023 21:09:45 +0000 (UTC) References: <87wmu775lq.fsf@gnu.org> User-agent: mu4e 1.10.7; emacs 29.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#67238] [PATCH] derivations: Avoid readlink syscalls in read-derivation-from-file. Date: Sat, 25 Nov 2023 20:58:03 +0000 In-reply-to: <87wmu775lq.fsf@gnu.org> Message-ID: <87edgdmtew.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 67238-close Cc: 67238-close@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi, > > Christopher Baines skribis: > >> strace -c reports over 10,000 readlink syscalls when reading the derivat= ion >> for the hello package. By just setting the %file-port-name-canonicalizat= ion >> fluid, this drops to less than 10. >> >> I'm not sure if this actually improves performance, but doing less is su= rely >> better. >> >> * guix/derivations.scm (read-derivation-from-file): Set >> %file-port-name-canonicalization to 'none when calling call-with-input-f= ile. >> >> Change-Id: I1ff16a059160576a576f2e9ed881379596e66af3 > > [...] > >> + (let ((drv >> + ;; Avoid calling scm_i_relativize_path in >> + ;; fport_canonicalize_filename since this leads to lots of >> + ;; readlink calls >> + (with-fluids ((%file-port-name-canonicalization 'none)) >> + (call-with-input-file file read-derivation)))) > > This is already done in =E2=80=98run-guix=E2=80=99 in (guix ui), for all = the =E2=80=98guix=E2=80=99 > commands (so this patch would be a slight performance regression for > Guix itself). > > I=E2=80=99d suggest setting this fluid globally in applications that use = Guix > (the Build Coordinator, etc.), as is done in Guix itself. > > WDYT? Ah, I didn't realise it was already set for Guix scripts. But yeah, setting it in other places that read derivations makes sense. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmViYpdfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfcDhAAhpy+TsA6QPr+tmBAZrLtMocrvkU7DHyi HTfdD9lTL082HX9S3R7O7cAcMADiUOUNJfeIMA7NWfgZDBcksgEYlKKM6vff2HFP uylS4tiwOiURPl70HUm3wlLbPrjXO3yVfpNuSgXjxy8ZFVqFUyhFxT5En96+yZj2 pD4yUxbff2J13kO3H4dvgnDM3lQpuQkxP//Scn20/cJRT85ADniDTUXTilRpP0JZ dG0/1w2fDgvfnsOBltPAyRgRZG0chvWD6amNVx19t0rIdDZ2JYhlMJTFpnSZsi5H InxpJcuFEaiHkwOGGe+XJghqckiJHuKBbiXa8PJ8tN1UbwioFLuotcHz1cEct9VQ Zm3mZi0+aZBfrAml+bOz7gFC/mU3uscvZEOjHw6pj4lPtdxAmOWc+/jG794SD+rf oO/SE7V0yWib/EIZX+saUlWPMeAlooLhKHyG/7nhdHqDExYRajmCZ7x9gf740CV7 PGwFXaHTC8yJllZYvszS+VM19KfL18M+koNruE8Ta0deFDPW12VHAm+s955UDkIA 1zFPpc777g1xzKt+EYwMUzpIegb3JBDDlMukUMiD2v4Wn28mNY2gKv3BdJz5Wld7 qBLRBWwbcC6Z2PuS2HFTvlGXNFP2OD6I/gmoF2nK38A1fbVfkGQTDDrHBvB3Wobv uR3xDz01X2E= =Ct4r -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 21 03:22:38 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sun, 24 Dec 2023 12:24:07 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator