From unknown Tue Sep 09 13:20:26 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#41720] [PATCH] store: Use buffered I/O for all protocol writes Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 05 Jun 2020 09:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 41720 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 41720@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.159134831914024 (code B ref -1); Fri, 05 Jun 2020 09:12:01 +0000 Received: (at submit) by debbugs.gnu.org; 5 Jun 2020 09:11:59 +0000 Received: from localhost ([127.0.0.1]:48331 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jh8ON-0003e8-EK for submit@debbugs.gnu.org; Fri, 05 Jun 2020 05:11:59 -0400 Received: from lists.gnu.org ([209.51.188.17]:43676) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jh8OL-0003dy-Ee for submit@debbugs.gnu.org; Fri, 05 Jun 2020 05:11:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37378) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jh8OL-0002ps-A7 for guix-patches@gnu.org; Fri, 05 Jun 2020 05:11:57 -0400 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:46412) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jh8OJ-0008G6-MD for guix-patches@gnu.org; Fri, 05 Jun 2020 05:11:57 -0400 Received: by mail-wr1-x441.google.com with SMTP id x6so8898436wrm.13 for ; Fri, 05 Jun 2020 02:11:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leibniz-psychology-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:subject:message-id:mime-version:content-disposition :user-agent; bh=L9IwGbkNF6IbfvD6DLQ1pqztR5umhq/kTVA+mpy/WP8=; b=pFwLY3sNKpryptdZqNT4g+SzudCRzjsGNcz7Ext8ptpmsLMh2q+FpGoRGSK1pe6KWX bp7Ej3+I8qfWg5G13qUGaht3piznVuAmPf3RrS7cNSOiIVWKchDGsPQ4NFqZAVwCsIHP 4xx9wLlJU4rU1x9hfvR8cBHbCZWVLhuibXb3S06jKACEQTstQ8xU26RgiGXfFoPp3c9H vQzzZTpCf9mUjWZmttlz1ZbYxUEkg+4LBP8gjB686J7ObSEg4Q6Kmd4j/pUTCZR+muu+ vN1fM+5KPdo4YupwVfxclgdeZAtiQI3DW7UoeBDKxLF+J2351qpF9yBJd3xdrkLqnyXJ JkrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition:user-agent; bh=L9IwGbkNF6IbfvD6DLQ1pqztR5umhq/kTVA+mpy/WP8=; b=N7WIhjcvtuDsKoHhxnRg7b1y2A8bbo94WnXfxZQLHAmYsTS+SNlJ/KkugZDPQvoWUD L1u2NRtxH0zXow5KOERnEWmG8rnGZT5MKMBVFzteqD8nmS8JpWVqF8JbKQqWFHv1gtqP +BauwRpZ3LICUKKB74S4I6ypwtKxuRmAD3i06E3JSrsDOJpsWQv9nBQDSShPg98y1l+9 sX7EQk19F2orc2LaOPNQwNU5UyoVJNwJI8ghga3tT8CM4On9UHsrDzdlH8YTAXPjLbXi cbDHhCPSDHSyoX0hdoiDm0jwB3/7IUE8oIDsyWniL9+zvSPhW1xWTr9ESwfyL2PclwAt AYBA== X-Gm-Message-State: AOAM533A90xkqoFFUnaihPfi1QcgKwDPA9060JNE5cOoIQqW3KvtqJYU ohdA1wSBxa49rHzkoNYEAPMRCPtxji2L5/OLkspttQiiZqAan+bozwvZNULNp2LkVZTPXeVn5pH yFtnnMQFnwPL/z4z9fxPVtAFmBVNQ1kTbRiBOj3Dav9PKH7nPo5Pj5J3cdzpPtHGbBZF1OZMe/f 6+FgV05rU= X-Google-Smtp-Source: ABdhPJzFkbCOCS1XZIISvywNN9en3h0k34SE5VfrkUa115Wsa9sY702StKvOHNFvfIcQEOWB5U5+Jg== X-Received: by 2002:a5d:4f81:: with SMTP id d1mr9044251wru.95.1591348313095; Fri, 05 Jun 2020 02:11:53 -0700 (PDT) Received: from localhost (dynamic-2a01-0c22-c874-1100-1842-9ea3-2bc6-059b.c22.pool.telefonica.de. [2a01:c22:c874:1100:1842:9ea3:2bc6:59b]) by smtp.gmail.com with ESMTPSA id g187sm10915271wma.17.2020.06.05.02.11.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Jun 2020 02:11:52 -0700 (PDT) Date: Fri, 5 Jun 2020 11:11:51 +0200 From: Lars-Dominik Braun Message-ID: <20200605091151.GB3225@zpidnp36> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="vOmOzSkFvhd7u8Ms" Content-Disposition: inline User-Agent: Mutt/1.10.1 (2018-07-13) Received-SPF: none client-ip=2a00:1450:4864:20::441; envelope-from=ldb@leibniz-psychology.org; helo=mail-wr1-x441.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --vOmOzSkFvhd7u8Ms Content-Type: multipart/mixed; boundary="XOIedfhf+7KOe/yw" Content-Disposition: inline --XOIedfhf+7KOe/yw Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I did some digging to investigate why SSH-based guix operations are current= ly so slow. One of the reasons seems to be that some operations write single w= ords to the socket and due to NOWAIT these result in a lot of small packets. The attached patch modifies store.scm to buffer all writes, so that won=E2= =80=99t happen any more. I=E2=80=99m seeing about ~40% speedup (6.194s vs 10.075s for the = best out of five runs) for `GUIX_DAEMON_SOCKET=3Dssh://localhost guix environment guix = -- true`, but it also seems to have a negative impact on UNIX domain socket communication by ~10% (1.561s vs 1.385s). For the notorious r-learnr package it=E2=80=99s still ~20% better (ssh, 16.620/21.418) and only ~3% worse (uni= x, 11.489/11.199). Keep in mind localhost has a much larger MTU and much lower latency than usual networks though. It might benefit from having an even bi= gger write buffer (64k vs. currently 8k). Cheers, Lars --XOIedfhf+7KOe/yw Content-Type: text/x-diff; charset=iso-8859-1 Content-Disposition: attachment; filename="0001-store-Use-buffered-I-O-for-all-protocol-writes.patch" Content-Transfer-Encoding: quoted-printable =46rom eb38dc246e5dd33f8a48a99aded8b8dc0c378376 Mon Sep 17 00:00:00 2001 =46rom: Lars-Dominik Braun Date: Fri, 5 Jun 2020 10:38:32 +0200 Subject: [PATCH] store: Use buffered I/O for all protocol writes * guix/store.scm (run-gc) Use buffered output port. (export-path) Same. (add-file-tree-to-store) Same. (set-build-options): Same. Add explicit flush. --- guix/store.scm | 70 ++++++++++++++++++++++++++++---------------------- 1 file changed, 39 insertions(+), 31 deletions(-) diff --git a/guix/store.scm b/guix/store.scm index 014d08aaec..9b3879b4a7 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -3,6 +3,7 @@ ;;; Copyright =A9 2018 Jan Nieuwenhuizen ;;; Copyright =A9 2019, 2020 Mathieu Othacehe ;;; Copyright =A9 2020 Florian Pelz +;;; Copyright =A9 2020 Lars-Dominik Braun ;;; ;;; This file is part of GNU Guix. ;;; @@ -821,8 +822,8 @@ encoding conversion errors." (locale (false-if-exception (setlocale LC_ALL)= ))) ;; Must be called after `open-connection'. =20 - (define socket - (store-connection-socket server)) + (define buffered + (store-connection-output-port server)) =20 (unless (unspecified? use-build-hook?) (warn-about-deprecation #:use-build-hook? #f @@ -831,9 +832,9 @@ encoding conversion errors." (let-syntax ((send (syntax-rules () ((_ (type option) ...) (begin - (write-arg type option socket) + (write-arg type option buffered) ...))))) - (write-int (operation-id set-options) socket) + (write-int (operation-id set-options) buffered) (send (boolean keep-failed?) (boolean keep-going?) (boolean fallback?) (integer verbosity)) (when (< (store-connection-minor-version server) #x61) @@ -896,6 +897,7 @@ encoding conversion errors." `(("locale" . ,locale)) '())))) (send (string-pairs pairs)))) + (write-buffered-output server) (let loop ((done? (process-stderr server))) (or done? (process-stderr server))))) =20 @@ -1108,13 +1110,14 @@ path." ;; We don't use the 'operation' macro so we can pass SELECT? to ;; 'write-file'. (record-operation 'add-to-store) - (let ((port (store-connection-socket server))) - (write-int (operation-id add-to-store) port) - (write-string basename port) - (write-int 1 port) ;obsolete, must be #t - (write-int (if recursive? 1 0) port) - (write-string hash-algo port) - (write-file file-name port #:select? select?) + (let ((port (store-connection-socket server)) + (buffered (store-connection-output-port server))) + (write-int (operation-id add-to-store) buffered) + (write-string basename buffered) + (write-int 1 buffered) ;obsolete, must be #t + (write-int (if recursive? 1 0) buffered) + (write-string hash-algo buffered) + (write-file file-name buffered #:select? select?) (write-buffered-output server) (let loop ((done? (process-stderr server))) (or done? (loop (process-stderr server)))) @@ -1220,13 +1223,14 @@ an arbitrary directory layout in the store without = creating a derivation." ;; We don't use the 'operation' macro so we can use 'write-file-tr= ee' ;; instead of 'write-file'. (record-operation 'add-to-store/tree) - (let ((port (store-connection-socket server))) - (write-int (operation-id add-to-store) port) - (write-string basename port) - (write-int 1 port) ;obsolete, must be #t - (write-int (if recursive? 1 0) port) - (write-string hash-algo port) - (write-file-tree basename port + (let ((port (store-connection-socket server)) + (buffered (store-connection-output-port server))) + (write-int (operation-id add-to-store) buffered) + (write-string basename buffered) + (write-int 1 buffered) ;obsolete, must be #t + (write-int (if recursive? 1 0) buffered) + (write-string hash-algo buffered) + (write-file-tree basename buffered #:file-type+size file-type+size #:file-port file-port #:symlink-target symlink-target @@ -1644,17 +1648,19 @@ the list of store paths to delete. IGNORE-LIVENESS= ? should always be #f. MIN-FREED is the minimum amount of disk space to be freed, in bytes, before the GC can stop. Return the list of store paths delete, and the number of bytes freed." - (let ((s (store-connection-socket server))) - (write-int (operation-id collect-garbage) s) - (write-int action s) - (write-store-path-list to-delete s) - (write-arg boolean #f s) ; ignore-liveness? - (write-long-long min-freed s) - (write-int 0 s) ; obsolete + (let ((s (store-connection-socket server)) + (buffered (store-connection-output-port server))) + (write-int (operation-id collect-garbage) buffered) + (write-int action buffered) + (write-store-path-list to-delete buffered) + (write-arg boolean #f buffered) ; ignore-liveness? + (write-long-long min-freed buffered) + (write-int 0 buffered) ; obsolete (when (>=3D (store-connection-minor-version server) 5) ;; Obsolete `use-atime' and `max-atime' parameters. - (write-int 0 s) - (write-int 0 s)) + (write-int 0 buffered) + (write-int 0 buffered)) + (write-buffered-output server) =20 ;; Loop until the server is done sending error output. (let loop ((done? (process-stderr server))) @@ -1711,10 +1717,12 @@ is raised if the set of paths read from PORT is not= signed (as per =20 (define* (export-path server path port #:key (sign? #t)) "Export PATH to PORT. When SIGN? is true, sign it." - (let ((s (store-connection-socket server))) - (write-int (operation-id export-path) s) - (write-store-path path s) - (write-arg boolean sign? s) + (let ((s (store-connection-socket server)) + (buffered (store-connection-output-port server))) + (write-int (operation-id export-path) buffered) + (write-store-path path buffered) + (write-arg boolean sign? buffered) + (write-buffered-output server) (let loop ((done? (process-stderr server port))) (or done? (loop (process-stderr server port)))) (=3D 1 (read-int s)))) --=20 2.20.1 --XOIedfhf+7KOe/yw-- --vOmOzSkFvhd7u8Ms Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEyk+M9DfXR4/aBV/UQhN3ARo3hEYFAl7aDFMACgkQQhN3ARo3 hEaxtgv/dpe1nWch2+Hfx3WBMZha9WyyFjW8TeasbC5cdy5bccYwasr5Q3Zd+Xbr FYBubNTUkMZz4p5M4qIC1QRloyXf9GlkSYKY7VTYGEXmlIa6mPPzwpkcq0xcJTl3 vypg12EhiBJBXA0KuGpkWgn8Cgxb7y0K1fHeWHa2o+oPXXdVqgJt36oK+bxxkkTH pmv0D4PhDWnxMv0wvYdz6ls5TYiIqUGpSufMIh9ynEiXBkGz/0J4bEGVV5aSKBrF LEB01Pr7TSuR9UDg+OqEz7Q7MMVEN63BaQkxJOi2rBdVLDX/C713LHjgzSahXY37 OzOQZUFX8aCPF4dJSutyUhOUKsdOyR7K3qGxlvuRiCmzHotSym1kWIDuo/9f1e1H h1YOutEHUIiOyiPmo941h5GlGV0qZuFCjKEu4Y7SXT0UYAR1vieL4RIzc6E0IY+m xrLixnn1pSOn4ebeuB+QDxha/VmvyPXo2nUBwOKuXhn9dGld5x0nYWsjrC7Pj91i DSdSf1Xm =THIV -----END PGP SIGNATURE----- --vOmOzSkFvhd7u8Ms-- From unknown Tue Sep 09 13:20:26 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#41720] [PATCH] store: Use buffered I/O for all protocol writes Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 05 Jun 2020 21:34:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41720 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Cc: 41720@debbugs.gnu.org Received: via spool by 41720-submit@debbugs.gnu.org id=B41720.159139279228643 (code B ref 41720); Fri, 05 Jun 2020 21:34:01 +0000 Received: (at 41720) by debbugs.gnu.org; 5 Jun 2020 21:33:12 +0000 Received: from localhost ([127.0.0.1]:50109 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jhJxf-0007Rv-VI for submit@debbugs.gnu.org; Fri, 05 Jun 2020 17:33:12 -0400 Received: from eggs.gnu.org ([209.51.188.92]:35166) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jhJxd-0007Rh-Kn for 41720@debbugs.gnu.org; Fri, 05 Jun 2020 17:33:10 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48417) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jhJxY-0007OT-7N; Fri, 05 Jun 2020 17:33:04 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=50876 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jhJxX-0005Ax-RB; Fri, 05 Jun 2020 17:33:04 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200605091151.GB3225@zpidnp36> Date: Fri, 05 Jun 2020 23:33:02 +0200 In-Reply-To: <20200605091151.GB3225@zpidnp36> (Lars-Dominik Braun's message of "Fri, 5 Jun 2020 11:11:51 +0200") Message-ID: <875zc5xkoh.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Lars-Dominik Braun skribis: > I did some digging to investigate why SSH-based guix operations are curre= ntly > so slow. One of the reasons seems to be that some operations write single= words > to the socket and due to NOWAIT these result in a lot of small packets. Good catch! > The attached patch modifies store.scm to buffer all writes, so that won= =E2=80=99t happen > any more. I=E2=80=99m seeing about ~40% speedup (6.194s vs 10.075s for th= e best out of > five runs) for `GUIX_DAEMON_SOCKET=3Dssh://localhost guix environment gui= x -- > true`, but it also seems to have a negative impact on UNIX domain socket > communication by ~10% (1.561s vs 1.385s). For the notorious r-learnr pack= age > it=E2=80=99s still ~20% better (ssh, 16.620/21.418) and only ~3% worse (u= nix, > 11.489/11.199). Keep in mind localhost has a much larger MTU and much low= er > latency than usual networks though. It might benefit from having an even = bigger > write buffer (64k vs. currently 8k). Nice, though the slowdown on Unix-domain sockets is annoying given that it=E2=80=99s the primary transport. Are the 10% significant and stable over multiple runs? It may be that writing through the buffered port, with the extra allocations, indirection through the custom port, etc., outweighs the gains. OTOH all the other RPCs are buffered, so it makes sense to just do the same. (It may be wiser to make the =E2=80=98operation=E2=80=99 macro smarter and = emit code that directly allocates a bytevector of the right size and sends it. Future work!) > From eb38dc246e5dd33f8a48a99aded8b8dc0c378376 Mon Sep 17 00:00:00 2001 > From: Lars-Dominik Braun > Date: Fri, 5 Jun 2020 10:38:32 +0200 > Subject: [PATCH] store: Use buffered I/O for all protocol writes > > * guix/store.scm (run-gc) Use buffered output port. > (export-path) Same. > (add-file-tree-to-store) Same. > (set-build-options): Same. Add explicit flush. Look: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=3Drpc guix environment guix -- true Remote procedure call summary: 1927 RPCs built-in-builders ... 1 build-things ... 1 query-substitutable-path-infos ... 1 valid-path? ... 2 add-to-store/tree ... 19 query-references ... 108 add-to-store ... 133 add-text-to-store ... 1662 --8<---------------cut here---------------end--------------->8--- So probably the gain you=E2=80=99re seeing comes from =E2=80=98add-file-tre= e-to-store=E2=80=99. Otherwise the patch LGTM as long as it doesn=E2=80=99t introduce any test s= uite regressions. Thanks for investigating! Ludo=E2=80=99. From unknown Tue Sep 09 13:20:26 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#41720] [PATCH] store: Use buffered I/O for all protocol writes Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 08 Jun 2020 06:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41720 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 41720@debbugs.gnu.org Received: via spool by 41720-submit@debbugs.gnu.org id=B41720.159159855718413 (code B ref 41720); Mon, 08 Jun 2020 06:43:02 +0000 Received: (at 41720) by debbugs.gnu.org; 8 Jun 2020 06:42:37 +0000 Received: from localhost ([127.0.0.1]:55798 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jiBUT-0004mu-Lw for submit@debbugs.gnu.org; Mon, 08 Jun 2020 02:42:37 -0400 Received: from mail-ej1-f67.google.com ([209.85.218.67]:46926) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jiBUR-0004mb-Ni for 41720@debbugs.gnu.org; Mon, 08 Jun 2020 02:42:36 -0400 Received: by mail-ej1-f67.google.com with SMTP id e2so16924740eje.13 for <41720@debbugs.gnu.org>; Sun, 07 Jun 2020 23:42:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leibniz-psychology-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=raAstWu4KD5rTFxoeKRofHnEv5q45XMnNBuwaBD4yQI=; b=RcRX/UpX+dN+esFTfY3ItIXl9omD0OKId6vuqs32dhnts5/ljiafa4vs1amCWp2J7I awFubb75xkxqfqTolUAREoEjJVB80c0tuF39nH69S3JMreAz2Ame3N9N8v9uZcgFaD6X An1H7kpMt1pNL4We4oochOJ/4nCE/fLR5GXwacHQsGpZ2N4rkUvkwtI521luAIizvX23 ZT5pA9Z34SLM68IJXh4SmTRk9bHnZqP0yhtWc9RKKQmHUeGkn/k7gv66dJqDUc/TcDSB WPgLaViC/fdzDT/1Y+pjfJ/dTKVzOcTDVSxxWg5qMyU0lOx3mi5FCN8qcR69Y8QqkL/R pz5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=raAstWu4KD5rTFxoeKRofHnEv5q45XMnNBuwaBD4yQI=; b=nvnWy6z0meCASwmJzhbzjpcaufhiXqLbvsJ4thJ++xLs9rNqSp463jHyfPMpMhM41K STxDIyEgT1QiKHD3dl0eamEI4m3zeUmghtWpE6X3lV6K2x6cA28/zzWWF8FiMzEjP7v/ dIDR8mwVHnNy8XFfPQCmZITFIurVHCq+8SF01IBNzMxJAJ9/TJHQNoOXM7Mdhv8sLDX0 hsdElqNhW4HfL1w0EFxjMIgSJFHih/wXCHD5eRLdoMrlv1ELJCaft5G7IM/Jw2PrFnb5 D7qj4lLAJ6WqqG0t3KI/LQIQZIoCquw2LlNPcvNqzc+LoDhSmyNFFNPAAksYdqlZ7GvP Wj3w== X-Gm-Message-State: AOAM533496DRad0Jap/+AO7KI8zI9IhBCMUPYM37cIM0B8i4lW3Ec+Fa Eu37LShZcUNDQWM8cdR5/0hONO+nora65y0GKHr9LrVFAKzmhpOr72EuqAdgYhx3Jm1Yb8Hm9iO hop2qIO+LCycX2ZGKr6uXstDiqFBebQdvkdVfK2mkqzGjbbnJeoxxd/4t8cEw+SVTZfX6cgnkNK 8bsM0= X-Google-Smtp-Source: ABdhPJzdnqdbg76RBI2QrjLZ+BHdRAbC6KDIqTLQTXwU/K1X5YzRdLTLM61bAqGRS8SxZ9I+1lj8PA== X-Received: by 2002:a17:906:768e:: with SMTP id o14mr20243746ejm.497.1591598549639; Sun, 07 Jun 2020 23:42:29 -0700 (PDT) Received: from localhost (dynamic-2a01-0c22-d48d-f900-f58e-6c44-3fc3-d82c.c22.pool.telefonica.de. [2a01:c22:d48d:f900:f58e:6c44:3fc3:d82c]) by smtp.gmail.com with ESMTPSA id q12sm10019343ejn.23.2020.06.07.23.42.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 07 Jun 2020 23:42:28 -0700 (PDT) Date: Mon, 8 Jun 2020 08:42:28 +0200 From: Lars-Dominik Braun Message-ID: <20200608064228.GB3166@zpidnp36> References: <20200605091151.GB3225@zpidnp36> <875zc5xkoh.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="XF85m9dhOBO43t/C" Content-Disposition: inline In-Reply-To: <875zc5xkoh.fsf@gnu.org> User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --XF85m9dhOBO43t/C Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Ludo, > Nice, though the slowdown on Unix-domain sockets is annoying given that > it=E2=80=99s the primary transport. Are the 10% significant and stable o= ver > multiple runs? I don=E2=80=99t want to blind your eyes with meaningless statistics here, b= ut let=E2=80=99s do this properly then. So for `guix environment guix -- true` with 10 trials t= hese are the results: ---snip--- $ ministat -A -c 99 before_ssh after_ssh x before_ssh + after_ssh N Min Max Median Avg Stddev x 10 7.63 10.635 10.1585 9.9382 0.85517585 + 10 5.986 6.424 6.1755 6.1917 0.15398416 Difference at 99.0% confidence -3.7465 +/- 0.790815 -37.698% +/- 5.0788% (Student's t, pooled s =3D 0.614425) $ ministat -A -c 99 before_unix after_unix x before_unix + after_unix N Min Max Median Avg Stddev x 10 1.528 1.624 1.5445 1.5636 0.035842867 + 10 1.576 1.701 1.6145 1.6229 0.037869513 Difference at 99.0% confidence 0.0593 +/- 0.0474548 3.79253% +/- 3.08989% (Student's t, pooled s =3D 0.0368701) ---snap--- And the same for `guix environment --ad-hoc r-learnr`: ---snip--- $ ministat -A -c 99 before_ssh after_ssh x before_ssh + after_ssh N Min Max Median Avg Stddev x 10 20.104 21.629 21.082 21.0115 0.45142472 + 10 16.082 16.713 16.292 16.3416 0.25018224 Difference at 99.0% confidence -4.6699 +/- 0.469719 -22.2254% +/- 1.86734% (Student's t, pooled s =3D 0.364949) $ ministat -A -c 99 before_unix after_unix x before_unix + after_unix N Min Max Median Avg Stddev x 10 11.322 11.529 11.401 11.4127 0.062467858 + 10 11.249 11.469 11.3515 11.368 0.078689262 No difference proven at 99.0% confidence ---snap--- > Otherwise the patch LGTM as long as it doesn=E2=80=99t introduce any test= suite > regressions. Yes, it passes the test suite. Lars --XF85m9dhOBO43t/C Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEyk+M9DfXR4/aBV/UQhN3ARo3hEYFAl7d3dAACgkQQhN3ARo3 hEYkJAv/XGCQySywDtYuQLK2yAKLZvGo27Y6RQ0RiqR1C16Xvcf9h1V3BzuL7zg7 1DGSY4dgExUIKNVXE7TnzD6/Ulg1qNcMunnsWRLnZX/pRVmqWtc7/jHrfxsuPHd4 Guevj+QIWT2kC1Dtj+eTKGJVodDr9BLTfJmWh8orLMOeWCB9dtDmrfWX1DLKdsMH P1IzOEmRlrE4CS0syqD1F/5yKueT6sAvjSqeQ5aQfJZr1ecGwJZ+xFmxfS1kwnSU OWAfJ+pOHKyeZIL5DMYGIem+CtUHBlXZf6rhSmNhS2aCOYFzcSDZfR/nAcCMyzJR m4F9mZQp+38C8ljXYR3UwUeWHguwjDwMFFI2p0kAsD2FigFuYWP7z2BpfuTPWxiJ CnIjRHBgxTAmXUoZfI/tDqX9MzrjX2bPN04Eiqm9+z14Tluq45BmnnMCLOEZBfy4 +nShsaKXerTBqr4LpyYzD11uRiPsO/22QC95Y1S848bWisEkFuh6mCA+8RN9N+6s dNqqHMXJ =KYN5 -----END PGP SIGNATURE----- --XF85m9dhOBO43t/C-- From unknown Tue Sep 09 13:20:26 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Lars-Dominik Braun Subject: bug#41720: closed (Re: [bug#41720] [PATCH] store: Use buffered I/O for all protocol writes) Message-ID: References: <87d068oezm.fsf@gnu.org> <20200605091151.GB3225@zpidnp36> X-Gnu-PR-Message: they-closed 41720 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 41720@debbugs.gnu.org Date: Tue, 09 Jun 2020 07:50:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1591689002-17411-1" This is a multi-part message in MIME format... ------------=_1591689002-17411-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #41720: [PATCH] store: Use buffered I/O for all protocol writes which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 41720@debbugs.gnu.org. --=20 41720: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D41720 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1591689002-17411-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 41720-done) by debbugs.gnu.org; 9 Jun 2020 07:50:00 +0000 Received: from localhost ([127.0.0.1]:58761 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jiZ1E-0004WX-8N for submit@debbugs.gnu.org; Tue, 09 Jun 2020 03:50:00 -0400 Received: from eggs.gnu.org ([209.51.188.92]:34076) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jiZ1C-0004WI-Bk for 41720-done@debbugs.gnu.org; Tue, 09 Jun 2020 03:49:58 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36616) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jiZ16-0001re-Ud; Tue, 09 Jun 2020 03:49:52 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=58658 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jiZ15-0007Ze-Ff; Tue, 09 Jun 2020 03:49:51 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Lars-Dominik Braun Subject: Re: [bug#41720] [PATCH] store: Use buffered I/O for all protocol writes References: <20200605091151.GB3225@zpidnp36> <875zc5xkoh.fsf@gnu.org> <20200608064228.GB3166@zpidnp36> Date: Tue, 09 Jun 2020 09:49:49 +0200 In-Reply-To: <20200608064228.GB3166@zpidnp36> (Lars-Dominik Braun's message of "Mon, 8 Jun 2020 08:42:28 +0200") Message-ID: <87d068oezm.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 41720-done Cc: 41720-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Lars-Dominik Braun skribis: >> Nice, though the slowdown on Unix-domain sockets is annoying given that >> it=E2=80=99s the primary transport. Are the 10% significant and stable = over >> multiple runs? > I don=E2=80=99t want to blind your eyes with meaningless statistics here,= but let=E2=80=99s do > this properly then. So for `guix environment guix -- true` with 10 trials= these > are the results: > > ---snip--- > $ ministat -A -c 99 before_ssh after_ssh > x before_ssh > + after_ssh > N Min Max Median Avg Stdd= ev > x 10 7.63 10.635 10.1585 9.9382 0.855175= 85 > + 10 5.986 6.424 6.1755 6.1917 0.153984= 16 > Difference at 99.0% confidence > -3.7465 +/- 0.790815 > -37.698% +/- 5.0788% > (Student's t, pooled s =3D 0.614425) > > $ ministat -A -c 99 before_unix after_unix > x before_unix > + after_unix > N Min Max Median Avg Stdd= ev > x 10 1.528 1.624 1.5445 1.5636 0.0358428= 67 > + 10 1.576 1.701 1.6145 1.6229 0.0378695= 13 > Difference at 99.0% confidence > 0.0593 +/- 0.0474548 > 3.79253% +/- 3.08989% > (Student's t, pooled s =3D 0.0368701) > ---snap--- > > And the same for `guix environment --ad-hoc r-learnr`: > > ---snip--- > $ ministat -A -c 99 before_ssh after_ssh > x before_ssh > + after_ssh > N Min Max Median Avg Stdd= ev > x 10 20.104 21.629 21.082 21.0115 0.451424= 72 > + 10 16.082 16.713 16.292 16.3416 0.250182= 24 > Difference at 99.0% confidence > -4.6699 +/- 0.469719 > -22.2254% +/- 1.86734% > (Student's t, pooled s =3D 0.364949) > > $ ministat -A -c 99 before_unix after_unix > x before_unix > + after_unix > N Min Max Median Avg Stdd= ev > x 10 11.322 11.529 11.401 11.4127 0.0624678= 58 > + 10 11.249 11.469 11.3515 11.368 0.0786892= 62 > No difference proven at 99.0% confidence > ---snap--- Awesome, thanks a lot. Looks like the difference on Unix-domain sockets is not significant. Applied! What package provides =E2=80=98ministat=E2=80=99? Thanks, Ludo=E2=80=99. ------------=_1591689002-17411-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 5 Jun 2020 09:11:59 +0000 Received: from localhost ([127.0.0.1]:48331 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jh8ON-0003e8-EK for submit@debbugs.gnu.org; Fri, 05 Jun 2020 05:11:59 -0400 Received: from lists.gnu.org ([209.51.188.17]:43676) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jh8OL-0003dy-Ee for submit@debbugs.gnu.org; Fri, 05 Jun 2020 05:11:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37378) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jh8OL-0002ps-A7 for guix-patches@gnu.org; Fri, 05 Jun 2020 05:11:57 -0400 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:46412) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jh8OJ-0008G6-MD for guix-patches@gnu.org; Fri, 05 Jun 2020 05:11:57 -0400 Received: by mail-wr1-x441.google.com with SMTP id x6so8898436wrm.13 for ; Fri, 05 Jun 2020 02:11:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leibniz-psychology-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:subject:message-id:mime-version:content-disposition :user-agent; bh=L9IwGbkNF6IbfvD6DLQ1pqztR5umhq/kTVA+mpy/WP8=; b=pFwLY3sNKpryptdZqNT4g+SzudCRzjsGNcz7Ext8ptpmsLMh2q+FpGoRGSK1pe6KWX bp7Ej3+I8qfWg5G13qUGaht3piznVuAmPf3RrS7cNSOiIVWKchDGsPQ4NFqZAVwCsIHP 4xx9wLlJU4rU1x9hfvR8cBHbCZWVLhuibXb3S06jKACEQTstQ8xU26RgiGXfFoPp3c9H vQzzZTpCf9mUjWZmttlz1ZbYxUEkg+4LBP8gjB686J7ObSEg4Q6Kmd4j/pUTCZR+muu+ vN1fM+5KPdo4YupwVfxclgdeZAtiQI3DW7UoeBDKxLF+J2351qpF9yBJd3xdrkLqnyXJ JkrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition:user-agent; bh=L9IwGbkNF6IbfvD6DLQ1pqztR5umhq/kTVA+mpy/WP8=; b=N7WIhjcvtuDsKoHhxnRg7b1y2A8bbo94WnXfxZQLHAmYsTS+SNlJ/KkugZDPQvoWUD L1u2NRtxH0zXow5KOERnEWmG8rnGZT5MKMBVFzteqD8nmS8JpWVqF8JbKQqWFHv1gtqP +BauwRpZ3LICUKKB74S4I6ypwtKxuRmAD3i06E3JSrsDOJpsWQv9nBQDSShPg98y1l+9 sX7EQk19F2orc2LaOPNQwNU5UyoVJNwJI8ghga3tT8CM4On9UHsrDzdlH8YTAXPjLbXi cbDHhCPSDHSyoX0hdoiDm0jwB3/7IUE8oIDsyWniL9+zvSPhW1xWTr9ESwfyL2PclwAt AYBA== X-Gm-Message-State: AOAM533A90xkqoFFUnaihPfi1QcgKwDPA9060JNE5cOoIQqW3KvtqJYU ohdA1wSBxa49rHzkoNYEAPMRCPtxji2L5/OLkspttQiiZqAan+bozwvZNULNp2LkVZTPXeVn5pH yFtnnMQFnwPL/z4z9fxPVtAFmBVNQ1kTbRiBOj3Dav9PKH7nPo5Pj5J3cdzpPtHGbBZF1OZMe/f 6+FgV05rU= X-Google-Smtp-Source: ABdhPJzFkbCOCS1XZIISvywNN9en3h0k34SE5VfrkUa115Wsa9sY702StKvOHNFvfIcQEOWB5U5+Jg== X-Received: by 2002:a5d:4f81:: with SMTP id d1mr9044251wru.95.1591348313095; Fri, 05 Jun 2020 02:11:53 -0700 (PDT) Received: from localhost (dynamic-2a01-0c22-c874-1100-1842-9ea3-2bc6-059b.c22.pool.telefonica.de. [2a01:c22:c874:1100:1842:9ea3:2bc6:59b]) by smtp.gmail.com with ESMTPSA id g187sm10915271wma.17.2020.06.05.02.11.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Jun 2020 02:11:52 -0700 (PDT) Date: Fri, 5 Jun 2020 11:11:51 +0200 From: Lars-Dominik Braun To: guix-patches@gnu.org Subject: [PATCH] store: Use buffered I/O for all protocol writes Message-ID: <20200605091151.GB3225@zpidnp36> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="vOmOzSkFvhd7u8Ms" Content-Disposition: inline User-Agent: Mutt/1.10.1 (2018-07-13) Received-SPF: none client-ip=2a00:1450:4864:20::441; envelope-from=ldb@leibniz-psychology.org; helo=mail-wr1-x441.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --vOmOzSkFvhd7u8Ms Content-Type: multipart/mixed; boundary="XOIedfhf+7KOe/yw" Content-Disposition: inline --XOIedfhf+7KOe/yw Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I did some digging to investigate why SSH-based guix operations are current= ly so slow. One of the reasons seems to be that some operations write single w= ords to the socket and due to NOWAIT these result in a lot of small packets. The attached patch modifies store.scm to buffer all writes, so that won=E2= =80=99t happen any more. I=E2=80=99m seeing about ~40% speedup (6.194s vs 10.075s for the = best out of five runs) for `GUIX_DAEMON_SOCKET=3Dssh://localhost guix environment guix = -- true`, but it also seems to have a negative impact on UNIX domain socket communication by ~10% (1.561s vs 1.385s). For the notorious r-learnr package it=E2=80=99s still ~20% better (ssh, 16.620/21.418) and only ~3% worse (uni= x, 11.489/11.199). Keep in mind localhost has a much larger MTU and much lower latency than usual networks though. It might benefit from having an even bi= gger write buffer (64k vs. currently 8k). Cheers, Lars --XOIedfhf+7KOe/yw Content-Type: text/x-diff; charset=iso-8859-1 Content-Disposition: attachment; filename="0001-store-Use-buffered-I-O-for-all-protocol-writes.patch" Content-Transfer-Encoding: quoted-printable =46rom eb38dc246e5dd33f8a48a99aded8b8dc0c378376 Mon Sep 17 00:00:00 2001 =46rom: Lars-Dominik Braun Date: Fri, 5 Jun 2020 10:38:32 +0200 Subject: [PATCH] store: Use buffered I/O for all protocol writes * guix/store.scm (run-gc) Use buffered output port. (export-path) Same. (add-file-tree-to-store) Same. (set-build-options): Same. Add explicit flush. --- guix/store.scm | 70 ++++++++++++++++++++++++++++---------------------- 1 file changed, 39 insertions(+), 31 deletions(-) diff --git a/guix/store.scm b/guix/store.scm index 014d08aaec..9b3879b4a7 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -3,6 +3,7 @@ ;;; Copyright =A9 2018 Jan Nieuwenhuizen ;;; Copyright =A9 2019, 2020 Mathieu Othacehe ;;; Copyright =A9 2020 Florian Pelz +;;; Copyright =A9 2020 Lars-Dominik Braun ;;; ;;; This file is part of GNU Guix. ;;; @@ -821,8 +822,8 @@ encoding conversion errors." (locale (false-if-exception (setlocale LC_ALL)= ))) ;; Must be called after `open-connection'. =20 - (define socket - (store-connection-socket server)) + (define buffered + (store-connection-output-port server)) =20 (unless (unspecified? use-build-hook?) (warn-about-deprecation #:use-build-hook? #f @@ -831,9 +832,9 @@ encoding conversion errors." (let-syntax ((send (syntax-rules () ((_ (type option) ...) (begin - (write-arg type option socket) + (write-arg type option buffered) ...))))) - (write-int (operation-id set-options) socket) + (write-int (operation-id set-options) buffered) (send (boolean keep-failed?) (boolean keep-going?) (boolean fallback?) (integer verbosity)) (when (< (store-connection-minor-version server) #x61) @@ -896,6 +897,7 @@ encoding conversion errors." `(("locale" . ,locale)) '())))) (send (string-pairs pairs)))) + (write-buffered-output server) (let loop ((done? (process-stderr server))) (or done? (process-stderr server))))) =20 @@ -1108,13 +1110,14 @@ path." ;; We don't use the 'operation' macro so we can pass SELECT? to ;; 'write-file'. (record-operation 'add-to-store) - (let ((port (store-connection-socket server))) - (write-int (operation-id add-to-store) port) - (write-string basename port) - (write-int 1 port) ;obsolete, must be #t - (write-int (if recursive? 1 0) port) - (write-string hash-algo port) - (write-file file-name port #:select? select?) + (let ((port (store-connection-socket server)) + (buffered (store-connection-output-port server))) + (write-int (operation-id add-to-store) buffered) + (write-string basename buffered) + (write-int 1 buffered) ;obsolete, must be #t + (write-int (if recursive? 1 0) buffered) + (write-string hash-algo buffered) + (write-file file-name buffered #:select? select?) (write-buffered-output server) (let loop ((done? (process-stderr server))) (or done? (loop (process-stderr server)))) @@ -1220,13 +1223,14 @@ an arbitrary directory layout in the store without = creating a derivation." ;; We don't use the 'operation' macro so we can use 'write-file-tr= ee' ;; instead of 'write-file'. (record-operation 'add-to-store/tree) - (let ((port (store-connection-socket server))) - (write-int (operation-id add-to-store) port) - (write-string basename port) - (write-int 1 port) ;obsolete, must be #t - (write-int (if recursive? 1 0) port) - (write-string hash-algo port) - (write-file-tree basename port + (let ((port (store-connection-socket server)) + (buffered (store-connection-output-port server))) + (write-int (operation-id add-to-store) buffered) + (write-string basename buffered) + (write-int 1 buffered) ;obsolete, must be #t + (write-int (if recursive? 1 0) buffered) + (write-string hash-algo buffered) + (write-file-tree basename buffered #:file-type+size file-type+size #:file-port file-port #:symlink-target symlink-target @@ -1644,17 +1648,19 @@ the list of store paths to delete. IGNORE-LIVENESS= ? should always be #f. MIN-FREED is the minimum amount of disk space to be freed, in bytes, before the GC can stop. Return the list of store paths delete, and the number of bytes freed." - (let ((s (store-connection-socket server))) - (write-int (operation-id collect-garbage) s) - (write-int action s) - (write-store-path-list to-delete s) - (write-arg boolean #f s) ; ignore-liveness? - (write-long-long min-freed s) - (write-int 0 s) ; obsolete + (let ((s (store-connection-socket server)) + (buffered (store-connection-output-port server))) + (write-int (operation-id collect-garbage) buffered) + (write-int action buffered) + (write-store-path-list to-delete buffered) + (write-arg boolean #f buffered) ; ignore-liveness? + (write-long-long min-freed buffered) + (write-int 0 buffered) ; obsolete (when (>=3D (store-connection-minor-version server) 5) ;; Obsolete `use-atime' and `max-atime' parameters. - (write-int 0 s) - (write-int 0 s)) + (write-int 0 buffered) + (write-int 0 buffered)) + (write-buffered-output server) =20 ;; Loop until the server is done sending error output. (let loop ((done? (process-stderr server))) @@ -1711,10 +1717,12 @@ is raised if the set of paths read from PORT is not= signed (as per =20 (define* (export-path server path port #:key (sign? #t)) "Export PATH to PORT. When SIGN? is true, sign it." - (let ((s (store-connection-socket server))) - (write-int (operation-id export-path) s) - (write-store-path path s) - (write-arg boolean sign? s) + (let ((s (store-connection-socket server)) + (buffered (store-connection-output-port server))) + (write-int (operation-id export-path) buffered) + (write-store-path path buffered) + (write-arg boolean sign? buffered) + (write-buffered-output server) (let loop ((done? (process-stderr server port))) (or done? (loop (process-stderr server port)))) (=3D 1 (read-int s)))) --=20 2.20.1 --XOIedfhf+7KOe/yw-- --vOmOzSkFvhd7u8Ms Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEyk+M9DfXR4/aBV/UQhN3ARo3hEYFAl7aDFMACgkQQhN3ARo3 hEaxtgv/dpe1nWch2+Hfx3WBMZha9WyyFjW8TeasbC5cdy5bccYwasr5Q3Zd+Xbr FYBubNTUkMZz4p5M4qIC1QRloyXf9GlkSYKY7VTYGEXmlIa6mPPzwpkcq0xcJTl3 vypg12EhiBJBXA0KuGpkWgn8Cgxb7y0K1fHeWHa2o+oPXXdVqgJt36oK+bxxkkTH pmv0D4PhDWnxMv0wvYdz6ls5TYiIqUGpSufMIh9ynEiXBkGz/0J4bEGVV5aSKBrF LEB01Pr7TSuR9UDg+OqEz7Q7MMVEN63BaQkxJOi2rBdVLDX/C713LHjgzSahXY37 OzOQZUFX8aCPF4dJSutyUhOUKsdOyR7K3qGxlvuRiCmzHotSym1kWIDuo/9f1e1H h1YOutEHUIiOyiPmo941h5GlGV0qZuFCjKEu4Y7SXT0UYAR1vieL4RIzc6E0IY+m xrLixnn1pSOn4ebeuB+QDxha/VmvyPXo2nUBwOKuXhn9dGld5x0nYWsjrC7Pj91i DSdSf1Xm =THIV -----END PGP SIGNATURE----- --vOmOzSkFvhd7u8Ms-- ------------=_1591689002-17411-1-- From unknown Tue Sep 09 13:20:26 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#41720] [PATCH] store: Use buffered I/O for all protocol writes Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 09 Jun 2020 08:22:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41720 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 41720-done@debbugs.gnu.org Received: via spool by 41720-done@debbugs.gnu.org id=D41720.159169092120575 (code D ref 41720); Tue, 09 Jun 2020 08:22:02 +0000 Received: (at 41720-done) by debbugs.gnu.org; 9 Jun 2020 08:22:01 +0000 Received: from localhost ([127.0.0.1]:58846 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jiZWC-0005Ln-Fy for submit@debbugs.gnu.org; Tue, 09 Jun 2020 04:22:00 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:54519) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jiZWA-0005LR-QO for 41720-done@debbugs.gnu.org; Tue, 09 Jun 2020 04:21:59 -0400 Received: by mail-wm1-f67.google.com with SMTP id g10so1933215wmh.4 for <41720-done@debbugs.gnu.org>; Tue, 09 Jun 2020 01:21:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leibniz-psychology-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Wv5I/Wl30U1orS4pEb2XWnS6ThC7hd5ieQF+LuieAR0=; b=B3GMZQIfvT5k/t0tIG7wT1kJ5fCkVxoaDC5Lvi7lljCQdG0Fld+X6b8blKD3+ODPSd 99Edmqky3xwFz5MRajacU4fbiByYgbWP8dLZ0vqSWOVlOvlQ4M/Pu8+Z1paak1NwQrbO iYq9W4cLXoB6/FHYPyrRxjzxUfM2FjKyl4iA4hQ9Z/Fd6ybs3ckXDgmOVNjxhrJPR4GG CDNvgU8FU1h+/J6UFGp4rpxMwGYE+Ux4j9TF8Xm9Y4dYblJgbqVTre4q4TUT6HaSEu25 JSlK4eL3VNi2xqDFvnJ/92iP5E7bJ3LFbc8/QB85Grd/7QE4o/eLPigomFBkKRI+POyy dWTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=Wv5I/Wl30U1orS4pEb2XWnS6ThC7hd5ieQF+LuieAR0=; b=DpAyLDuhNnuDcWMnc1RIBFjx5mMGEIpyrBUKP0H07kPGLKcV984CZK44o8TqBK/Ua4 0SrkbbM8w9MVVy5zmpyoP+mYpsJ16Z7uimIwqzIVWDUbqiTXfxjiEZXY2Bn1yqaF+u24 mOGfYs23P+i5AZODgL8gktEpD/AvGpxw585R/sIHzETNEEmHfUGeoD9Uqh9jKOiSArfe 8LWj1RT5jrDlDCRPom34OKU4YLqiLnLj2sYUS/aFX+SW0vEGsR6RBpdhaVyWfsGbbHmZ vOuD8KkwJhV08DeSMZuJZ+fXyIfaVWiedD9qLPCnKsfAEyDsJdl/GadXR9k8CCVYVHTq RZ+A== X-Gm-Message-State: AOAM530MG819IvC0ox/22EcPCJnfTvw6b3qakXYemUf5NYOtDQsnAwmI YR/p1AZWBCIAMvjCxqCeZ/EusWRk08WsXqr7TUB+DiHs4yyg1kuZECIDopSfvmjuJGetqMS2bYu Y7uFARvpAvwgYFnqAZ6M7HX9O50z3VOlOEPmJy+gorp3eL0Ytl9qlb55qV6eIh4OG9MEBN5Jlfs fKvFKxcNfCrjFJNmo= X-Google-Smtp-Source: ABdhPJz7n0edpQEvS+Qn5/7YIap6x4uxZuxhTGZWSIYijHob2Pqm9hLzZWf7GmLcL/6oD1M8hZETYQ== X-Received: by 2002:a1c:a7c3:: with SMTP id q186mr2639052wme.141.1591690912880; Tue, 09 Jun 2020 01:21:52 -0700 (PDT) Received: from localhost (dynamic-2a01-0c23-7c5e-9e00-3d61-c8ee-9cd9-ef49.c23.pool.telefonica.de. [2a01:c23:7c5e:9e00:3d61:c8ee:9cd9:ef49]) by smtp.gmail.com with ESMTPSA id d2sm2452410wrs.95.2020.06.09.01.21.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jun 2020 01:21:51 -0700 (PDT) Date: Tue, 9 Jun 2020 10:21:51 +0200 From: Lars-Dominik Braun Message-ID: <20200609082151.GA8597@zpidnp36> References: <20200605091151.GB3225@zpidnp36> <875zc5xkoh.fsf@gnu.org> <20200608064228.GB3166@zpidnp36> <87d068oezm.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="CE+1k2dSO48ffgeK" Content-Disposition: inline In-Reply-To: <87d068oezm.fsf@gnu.org> User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --CE+1k2dSO48ffgeK Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, great! > What package provides =E2=80=98ministat=E2=80=99? It=E2=80=99s originally a FreeBSD tool[1], but there are several unofficial= Linux ports on GitHub, like [2]. Cheers, Lars [1] https://www.freebsd.org/cgi/man.cgi?query=3Dministat&sektion=3D1&manpat= h=3Dfreebsd-release-ports [2] https://github.com/csjayp/ministat --CE+1k2dSO48ffgeK Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEyk+M9DfXR4/aBV/UQhN3ARo3hEYFAl7fRpwACgkQQhN3ARo3 hEY4Jgv9EegfGhsn0onjdNqJmMmSRcbm9Zg2MGSlkwEZPk9JVfZB2X/hTa5eiUGb ZvO5bS7UrifBbVaMok08EYf0Chkhl1NC9/FSZwEhn5UJLpPYO+Udkw1lEdATbKcI jKmJkn1jR+yddDwG4KLC6y9RsAUUAB7uhTe4hgE7t8YFNRdwjh/aD8kpj+cPO1gB FK5dDoElnoC53jHpmGDu0v/Dg53Yib1wX9wlTNNvNFknaI7QX6va7F32gt899doI G8FumgyWfl2Y6SBotsTWaoLNw8b6ufzg7xXIdQah/xsku04XW/do6s6wUeMSPTsU XkAEO7YsOOzpbEHRpyAZ4Zvh9nipvN2+tnzUWQBz2xEjSnZZH+kRBOJVeq3PUBW4 oafCjYJ/B5w2vqCy4HR/ko8o9S2FZKKa6aOhnjtZH1lYZlsOBuXZu+xRiYVnSbkp EOvuN1SsJAGcuiCCw9YDatf2aicCy9gsTyHvdOGs4Lxw0Zl0amt+/0fO+pIwisVK CvWlzCen =h5uO -----END PGP SIGNATURE----- --CE+1k2dSO48ffgeK--