From unknown Mon Aug 18 11:17:34 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#44567] [PATCH] publish: Improve HTTP performance when not using --cache. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 11 Nov 2020 04:00:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 44567 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 44567@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Maxim Cournoyer X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16050671586174 (code B ref -1); Wed, 11 Nov 2020 04:00:01 +0000 Received: (at submit) by debbugs.gnu.org; 11 Nov 2020 03:59:18 +0000 Received: from localhost ([127.0.0.1]:39732 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kchHy-0001bW-5Z for submit@debbugs.gnu.org; Tue, 10 Nov 2020 22:59:18 -0500 Received: from lists.gnu.org ([209.51.188.17]:40048) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kchHv-0001bN-RC for submit@debbugs.gnu.org; Tue, 10 Nov 2020 22:59:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53288) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kchHv-00058C-4J for guix-patches@gnu.org; Tue, 10 Nov 2020 22:59:15 -0500 Received: from mail-qt1-x842.google.com ([2607:f8b0:4864:20::842]:32793) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kchHt-0001WH-41; Tue, 10 Nov 2020 22:59:14 -0500 Received: by mail-qt1-x842.google.com with SMTP id i12so501765qtj.0; Tue, 10 Nov 2020 19:59:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XaeK7A8gB+CRFWk7fmbBt3z4ezhaDDguNFUKEmtT7pU=; b=qCf8b5wbptyFNlTuFFsAmBuBvjMV38+gu/djoO+frLmIhkd02J/kkugiqS2HZd44rn SvMaeZ3n77AkLA1QhnxnyKlD5V7uo31yQiZiPCmTA1fjM7LSLbqG1CkKOiP0JncU80Hw CQn8p/ZigYuVOhFuAWkK7xd+CQHrXBAIKOtZZJL+64nryEQTJP5bwP2EBRj3qa6hkWR2 c/nfRXl8+tfnupV8YXuJHjlTveOo1eIip53Ead4h7sbjodn82K710WynimoOFhF84QRD WOXmjMIhLbPNMryzvRzO1QQPHV+tKm3HykkEy3CZ0djREzSUZmf3p2RzTJV9UuorY08y oh3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XaeK7A8gB+CRFWk7fmbBt3z4ezhaDDguNFUKEmtT7pU=; b=N0E4+mH+Mhv5LJhuKizb/y0r7MT59MvETaAiVt23PinBLaLqaszPy0TC/QI3uXCwOB BLVhRkGmEl3F3tcm6+4411PSIFBiisXbN1igFIimlhAM8EcdNh5iHtb8vhqjNcMUC3rt 2JWs1l/LV1ybHs5Cqr521O9yP5Fw2sZu46YEOCPF81gEVbRTCHi/f9iyZST68QDv7Syn Yj9bvOWdhZgxlVaTz2IP9CZ2iXvZ37y37kuTbIPT2NPK+n6SC1qzOz/cGgO+RXUyeu7+ dGgTOKXIafiK3wX34pPpUIzYjMM5P4VicpfEAFejEiEoMsl/fLbs5+zmkEHBPC3xRF/a Tf0A== X-Gm-Message-State: AOAM533JK8ld77ULL1KTmFjJMGIkdd+cnlvipvsK3hdK+hqPIOWpl26T Ehmy8sPOOiBxxzvlR0QKywt8m0TT6rtZXg== X-Google-Smtp-Source: ABdhPJwZBAJfqRl1n2GhG9iwpvamVxkjT8pttDeITee/gQK92rAIRJge7dB8Gb7BFty49E71d4hMCg== X-Received: by 2002:aed:30e2:: with SMTP id 89mr22038831qtf.259.1605067150996; Tue, 10 Nov 2020 19:59:10 -0800 (PST) Received: from localhost.localdomain (dsl-10-133-153.b2b2c.ca. [72.10.133.153]) by smtp.gmail.com with ESMTPSA id d12sm986767qtp.77.2020.11.10.19.59.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Nov 2020 19:59:10 -0800 (PST) From: Maxim Cournoyer Date: Tue, 10 Nov 2020 22:57:27 -0500 Message-Id: <20201111035727.11184-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::842; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x842.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) This change harmonizes the way we configure the buffer sizes and the socket options, so that we don't forget to change it at one place like it happened in commit 5e3d169945935b53325e6b738a307ba286751259. Using a greater socket buffer size reportedly improves throughput tenfold. The solution was found by Ludovic Courtès. * guix/scripts/publish.scm (%default-buffer-size) (%default-socket-options): New variables. * guix/scripts/publish.scm (configure-socket): New procedure. (compress-nar): Use %default-buffer-size for the buffer size, increased from 128 to 208 KiB. (nar-response-port): Likewise, increased from 64 to 208 KiB. (http-write): Use configure-socket to set socket options. (open-server-socket): Likewise. --- guix/scripts/publish.scm | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index e8faf379e2..8a07f2300e 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2015 David Thompson ;;; Copyright © 2020 by Amar M. Singh ;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -250,6 +251,18 @@ usage." ("WantMassQuery" . 0) ("Priority" . 100))) +(define %default-buffer-size + (* 208 1024)) + +(define %default-socket-options + (list (list SO_SNDBUF %default-buffer-size))) + +(define* (configure-socket socket #:key (level SOL_SOCKET) + (options %default-socket-options)) + "Apply multiple option tuples in OPTIONS to SOCKET, using LEVEL." + (for-each (cut apply setsockopt socket level <>) + options)) + (define (signed-string s) "Sign the hash of the string S with the daemon's key. Return a canonical sexp for the signature." @@ -569,7 +582,7 @@ requested using POOL." (lambda (port) (write-file item port)) #:level (compression-level compression) - #:buffer-size (* 128 1024)) + #:buffer-size %default-buffer-size) (rename-file (string-append nar ".tmp") nar)) ('lzip ;; Note: the file port gets closed along with the lzip port. @@ -858,7 +871,7 @@ or if EOF is reached." ;; 'make-gzip-output-port' wants a file port. (make-gzip-output-port (response-port response) #:level level - #:buffer-size (* 64 1024))) + #:buffer-size %default-buffer-size)) (($ 'lzip level) (make-lzip-output-port (response-port response) #:level level)) @@ -883,6 +896,7 @@ blocking." client)) (port (begin (force-output client) + (configure-socket client) (nar-response-port response compression)))) ;; XXX: Given our ugly workaround for in ;; 'render-nar', BODY here is just the file name of the store item. @@ -912,7 +926,7 @@ blocking." size) client)) (output (response-port response))) - (setsockopt client SOL_SOCKET SO_SNDBUF (* 128 1024)) + (configure-socket client) (if (file-port? output) (sendfile output input size) (dump-port input output)) @@ -1057,7 +1071,8 @@ methods, return the applicable compression." (define (open-server-socket address) "Return a TCP socket bound to ADDRESS, a socket address." (let ((sock (socket (sockaddr:fam address) SOCK_STREAM 0))) - (setsockopt sock SOL_SOCKET SO_REUSEADDR 1) + (configure-socket sock #:options (cons (list SO_REUSEADDR 1) + %default-socket-options)) (bind sock address) sock)) -- 2.28.0 From unknown Mon Aug 18 11:17:34 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#44567] [PATCH] publish: Improve HTTP performance when not using --cache. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 13 Nov 2020 17:20:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44567 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 44567@debbugs.gnu.org Received: via spool by 44567-submit@debbugs.gnu.org id=B44567.160528795427061 (code B ref 44567); Fri, 13 Nov 2020 17:20:01 +0000 Received: (at 44567) by debbugs.gnu.org; 13 Nov 2020 17:19:14 +0000 Received: from localhost ([127.0.0.1]:49279 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kdcjC-00072P-7F for submit@debbugs.gnu.org; Fri, 13 Nov 2020 12:19:14 -0500 Received: from eggs.gnu.org ([209.51.188.92]:60284) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kdcjA-00072C-9g for 44567@debbugs.gnu.org; Fri, 13 Nov 2020 12:19:12 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:35834) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kdcj4-0004C8-Bd; Fri, 13 Nov 2020 12:19:07 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=37742 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kdcj3-0000i7-B8; Fri, 13 Nov 2020 12:19:05 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20201111035727.11184-1-maxim.cournoyer@gmail.com> Date: Fri, 13 Nov 2020 18:19:03 +0100 In-Reply-To: <20201111035727.11184-1-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Tue, 10 Nov 2020 22:57:27 -0500") Message-ID: <87361dcgzc.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Maxim, Maxim Cournoyer skribis: > This change harmonizes the way we configure the buffer sizes and the sock= et > options, so that we don't forget to change it at one place like it happen= ed in > commit 5e3d169945935b53325e6b738a307ba286751259. Using a greater socket > buffer size reportedly improves throughput tenfold. > > The solution was found by Ludovic Court=C3=A8s. > > * guix/scripts/publish.scm (%default-buffer-size) > (%default-socket-options): New variables. > * guix/scripts/publish.scm (configure-socket): New procedure. > (compress-nar): Use %default-buffer-size for the buffer size, increased f= rom > 128 to 208 KiB. > (nar-response-port): Likewise, increased from 64 to 208 KiB. > (http-write): Use configure-socket to set socket options. > (open-server-socket): Likewise. Apologies for not noticing this before pushing 1cbda46d4aae5ba9bd89a1837f0d81a29653ed7b. What you propose here is nicer. > +(define %default-buffer-size > + (* 208 1024)) Why 208? Did you notice a difference compared to 128KiB? (I=E2=80=99m fine either way, just wondering.) Perhaps add a comment as to what buffer we=E2=80=99re talking about. > +(define %default-socket-options Maybe add: ;; List of options passed to 'setsockopt' when transmitting file= s. > + (list (list SO_SNDBUF %default-buffer-size))) > (make-gzip-output-port (response-port response) > #:level level > - #:buffer-size (* 64 1024))) > + #:buffer-size %default-buffer-size)) Does the gzip buffer size have to match the TCP buffer size? I=E2=80=99d s= ay not necessarily, so perhaps we should have a separate variable for the gzip buffer size (or keep it as is). That=E2=80=99s it, thank you! Ludo=E2=80=99. From unknown Mon Aug 18 11:17:34 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Maxim Cournoyer Subject: bug#44567: closed (Re: [bug#44567] [PATCH] publish: Improve HTTP performance when not using --cache.) Message-ID: References: <87mtzhdgvr.fsf@gmail.com> <20201111035727.11184-1-maxim.cournoyer@gmail.com> X-Gnu-PR-Message: they-closed 44567 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 44567@debbugs.gnu.org Date: Mon, 16 Nov 2020 05:13:01 +0000 Content-Type: multipart/mixed; boundary="----------=_1605503581-12909-1" This is a multi-part message in MIME format... ------------=_1605503581-12909-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #44567: [PATCH] publish: Improve HTTP performance when not using --cache. which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 44567@debbugs.gnu.org. --=20 44567: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D44567 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1605503581-12909-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 44567-done) by debbugs.gnu.org; 16 Nov 2020 05:12:51 +0000 Received: from localhost ([127.0.0.1]:54799 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1keWot-0003Lr-2g for submit@debbugs.gnu.org; Mon, 16 Nov 2020 00:12:51 -0500 Received: from mail-il1-f194.google.com ([209.85.166.194]:40026) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1keWop-0003Lc-Of for 44567-done@debbugs.gnu.org; Mon, 16 Nov 2020 00:12:49 -0500 Received: by mail-il1-f194.google.com with SMTP id n5so13960966ile.7 for <44567-done@debbugs.gnu.org>; Sun, 15 Nov 2020 21:12:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=18UNXV4/bLkfxc1521p+JC5Q8qUPWY16NGtW39GUvA0=; b=Si+1TBxIUOCvWW74bKom1ykmzV9WhONretochbCaRqaSLI3u07vMGN7SVqV5TXj8/e 7dSKJOCexAWPjdozEPHyaphZbcYWIzrk3xTP2oWVLYMv1I5lPK5AZTJTNtFZXFd0ZRAN yAVrtnB5yFRCdKNWIgdO62Mtsjpz4jw++/2FaNWMSF43GRH3Czqev3miUaIzPZ3y5lqR qpDrheI34EfaenyP/Nc6pJP3idwoqyP2JSE7Q4m12bom6qy8Dm1GHXbhP2EzTY/SlfCG Qf7HaaLuiF50qpZB8f6zJRWdfboUs5tvlfX5DGhTTJRERpil2cZxrUOJaGauDIYnXTX8 eSJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=18UNXV4/bLkfxc1521p+JC5Q8qUPWY16NGtW39GUvA0=; b=tdDDRPmtZBmbDP7k85J9tO/QlWY6qEwYH+ILNXeFTGHrTFbHiB82gttRzzrq5SEaDb WfoHx8FghpoR1EzE0J2izFnMUJk0J4DN4wBHtUEzTO5Jth3dCuEDsJ3LFSwxdZ86XREU LRs3wN0WYn/c195LR3ISmn8wZdp2XNEoAvIR23YyMTlC0CGQI6zByLvwHYHXMkOfiyYN WJs/H4RbT7SC8J/JlZlittkkCJ2VV5exZ917NuPeGugnoUd3QJCqNFRLNtdj4uEqiEg8 NBN0/MtBnIB31AmlZT2FjFdC8ax2Q0m3MVO26Tp4rma262pbIO7nZ6SDHCPFQKiJx09p L8sQ== X-Gm-Message-State: AOAM533zhHU7LunE9dJdKZk6hJT0JkugqY3rOEhJ5YZ6A2hS6M9Q7WWf k0fzOGU0OaFhIanZ+tVGG4ON5rcRmyF+IA== X-Google-Smtp-Source: ABdhPJwMEtwR0wsJBZoXvVjg1Fk67U/5oBA1krxK9KvZXAaHsUb9kPx9kfHV5qnixgNETk2KjSh3sg== X-Received: by 2002:a05:6e02:12ab:: with SMTP id f11mr6996136ilr.89.1605503561632; Sun, 15 Nov 2020 21:12:41 -0800 (PST) Received: from hurd (dsl-10-128-91.b2b2c.ca. [72.10.128.91]) by smtp.gmail.com with ESMTPSA id f8sm8604705ioc.24.2020.11.15.21.12.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Nov 2020 21:12:41 -0800 (PST) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#44567] [PATCH] publish: Improve HTTP performance when not using --cache. References: <20201111035727.11184-1-maxim.cournoyer@gmail.com> <87361dcgzc.fsf@gnu.org> Date: Mon, 16 Nov 2020 00:12:40 -0500 In-Reply-To: <87361dcgzc.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Fri, 13 Nov 2020 18:19:03 +0100") Message-ID: <87mtzhdgvr.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 44567-done Cc: 44567-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludovic, Ludovic Court=C3=A8s writes: > Hi Maxim, > > Maxim Cournoyer skribis: > >> This change harmonizes the way we configure the buffer sizes and the soc= ket >> options, so that we don't forget to change it at one place like it happe= ned in >> commit 5e3d169945935b53325e6b738a307ba286751259. Using a greater socket >> buffer size reportedly improves throughput tenfold. >> >> The solution was found by Ludovic Court=C3=A8s. >> >> * guix/scripts/publish.scm (%default-buffer-size) >> (%default-socket-options): New variables. >> * guix/scripts/publish.scm (configure-socket): New procedure. >> (compress-nar): Use %default-buffer-size for the buffer size, increased = from >> 128 to 208 KiB. >> (nar-response-port): Likewise, increased from 64 to 208 KiB. >> (http-write): Use configure-socket to set socket options. >> (open-server-socket): Likewise. > > Apologies for not noticing this before pushing > 1cbda46d4aae5ba9bd89a1837f0d81a29653ed7b. What you propose here is > nicer. No worries! >> +(define %default-buffer-size >> + (* 208 1024)) > > Why 208? Did you notice a difference compared to 128KiB? (I=E2=80=99m f= ine > either way, just wondering.) I didn't observe any meaningful difference, it perhaps felt more 'steady'. The value comes from what Linux, the kernel, configures as a max value by default (see 'man tcp' and 'cat /proc/sys/net/core/wmem_max'). > Perhaps add a comment as to what buffer we=E2=80=99re talking about. Done. >> +(define %default-socket-options > > Maybe add: ;; List of options passed to 'setsockopt' when transmitting fi= les. > >> + (list (list SO_SNDBUF %default-buffer-size))) > >> (make-gzip-output-port (response-port response) >> #:level level >> - #:buffer-size (* 64 1024))) >> + #:buffer-size %default-buffer-size)) > > Does the gzip buffer size have to match the TCP buffer size? I=E2=80=99d= say > not necessarily, so perhaps we should have a separate variable for the > gzip buffer size (or keep it as is). It doesn't, but if we're going to pick some value, it seems the network one (SO_SNDBUF) should be the lowest common denominator. Since we're augmenting the previous value, I've just all made them agree. Another variable should be created the day we have a rationale to give them different values, I'd say. > That=E2=80=99s it, thank you! Thanks for the review! Pushed to the 1.2.0-version branch. Maxim ------------=_1605503581-12909-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 11 Nov 2020 03:59:18 +0000 Received: from localhost ([127.0.0.1]:39732 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kchHy-0001bW-5Z for submit@debbugs.gnu.org; Tue, 10 Nov 2020 22:59:18 -0500 Received: from lists.gnu.org ([209.51.188.17]:40048) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kchHv-0001bN-RC for submit@debbugs.gnu.org; Tue, 10 Nov 2020 22:59:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53288) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kchHv-00058C-4J for guix-patches@gnu.org; Tue, 10 Nov 2020 22:59:15 -0500 Received: from mail-qt1-x842.google.com ([2607:f8b0:4864:20::842]:32793) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kchHt-0001WH-41; Tue, 10 Nov 2020 22:59:14 -0500 Received: by mail-qt1-x842.google.com with SMTP id i12so501765qtj.0; Tue, 10 Nov 2020 19:59:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XaeK7A8gB+CRFWk7fmbBt3z4ezhaDDguNFUKEmtT7pU=; b=qCf8b5wbptyFNlTuFFsAmBuBvjMV38+gu/djoO+frLmIhkd02J/kkugiqS2HZd44rn SvMaeZ3n77AkLA1QhnxnyKlD5V7uo31yQiZiPCmTA1fjM7LSLbqG1CkKOiP0JncU80Hw CQn8p/ZigYuVOhFuAWkK7xd+CQHrXBAIKOtZZJL+64nryEQTJP5bwP2EBRj3qa6hkWR2 c/nfRXl8+tfnupV8YXuJHjlTveOo1eIip53Ead4h7sbjodn82K710WynimoOFhF84QRD WOXmjMIhLbPNMryzvRzO1QQPHV+tKm3HykkEy3CZ0djREzSUZmf3p2RzTJV9UuorY08y oh3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XaeK7A8gB+CRFWk7fmbBt3z4ezhaDDguNFUKEmtT7pU=; b=N0E4+mH+Mhv5LJhuKizb/y0r7MT59MvETaAiVt23PinBLaLqaszPy0TC/QI3uXCwOB BLVhRkGmEl3F3tcm6+4411PSIFBiisXbN1igFIimlhAM8EcdNh5iHtb8vhqjNcMUC3rt 2JWs1l/LV1ybHs5Cqr521O9yP5Fw2sZu46YEOCPF81gEVbRTCHi/f9iyZST68QDv7Syn Yj9bvOWdhZgxlVaTz2IP9CZ2iXvZ37y37kuTbIPT2NPK+n6SC1qzOz/cGgO+RXUyeu7+ dGgTOKXIafiK3wX34pPpUIzYjMM5P4VicpfEAFejEiEoMsl/fLbs5+zmkEHBPC3xRF/a Tf0A== X-Gm-Message-State: AOAM533JK8ld77ULL1KTmFjJMGIkdd+cnlvipvsK3hdK+hqPIOWpl26T Ehmy8sPOOiBxxzvlR0QKywt8m0TT6rtZXg== X-Google-Smtp-Source: ABdhPJwZBAJfqRl1n2GhG9iwpvamVxkjT8pttDeITee/gQK92rAIRJge7dB8Gb7BFty49E71d4hMCg== X-Received: by 2002:aed:30e2:: with SMTP id 89mr22038831qtf.259.1605067150996; Tue, 10 Nov 2020 19:59:10 -0800 (PST) Received: from localhost.localdomain (dsl-10-133-153.b2b2c.ca. [72.10.133.153]) by smtp.gmail.com with ESMTPSA id d12sm986767qtp.77.2020.11.10.19.59.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Nov 2020 19:59:10 -0800 (PST) From: Maxim Cournoyer To: guix-patches@gnu.org Subject: [PATCH] publish: Improve HTTP performance when not using --cache. Date: Tue, 10 Nov 2020 22:57:27 -0500 Message-Id: <20201111035727.11184-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::842; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x842.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.2 (/) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= , Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) This change harmonizes the way we configure the buffer sizes and the socket options, so that we don't forget to change it at one place like it happened in commit 5e3d169945935b53325e6b738a307ba286751259. Using a greater socket buffer size reportedly improves throughput tenfold. The solution was found by Ludovic Courtès. * guix/scripts/publish.scm (%default-buffer-size) (%default-socket-options): New variables. * guix/scripts/publish.scm (configure-socket): New procedure. (compress-nar): Use %default-buffer-size for the buffer size, increased from 128 to 208 KiB. (nar-response-port): Likewise, increased from 64 to 208 KiB. (http-write): Use configure-socket to set socket options. (open-server-socket): Likewise. --- guix/scripts/publish.scm | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index e8faf379e2..8a07f2300e 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2015 David Thompson ;;; Copyright © 2020 by Amar M. Singh ;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -250,6 +251,18 @@ usage." ("WantMassQuery" . 0) ("Priority" . 100))) +(define %default-buffer-size + (* 208 1024)) + +(define %default-socket-options + (list (list SO_SNDBUF %default-buffer-size))) + +(define* (configure-socket socket #:key (level SOL_SOCKET) + (options %default-socket-options)) + "Apply multiple option tuples in OPTIONS to SOCKET, using LEVEL." + (for-each (cut apply setsockopt socket level <>) + options)) + (define (signed-string s) "Sign the hash of the string S with the daemon's key. Return a canonical sexp for the signature." @@ -569,7 +582,7 @@ requested using POOL." (lambda (port) (write-file item port)) #:level (compression-level compression) - #:buffer-size (* 128 1024)) + #:buffer-size %default-buffer-size) (rename-file (string-append nar ".tmp") nar)) ('lzip ;; Note: the file port gets closed along with the lzip port. @@ -858,7 +871,7 @@ or if EOF is reached." ;; 'make-gzip-output-port' wants a file port. (make-gzip-output-port (response-port response) #:level level - #:buffer-size (* 64 1024))) + #:buffer-size %default-buffer-size)) (($ 'lzip level) (make-lzip-output-port (response-port response) #:level level)) @@ -883,6 +896,7 @@ blocking." client)) (port (begin (force-output client) + (configure-socket client) (nar-response-port response compression)))) ;; XXX: Given our ugly workaround for in ;; 'render-nar', BODY here is just the file name of the store item. @@ -912,7 +926,7 @@ blocking." size) client)) (output (response-port response))) - (setsockopt client SOL_SOCKET SO_SNDBUF (* 128 1024)) + (configure-socket client) (if (file-port? output) (sendfile output input size) (dump-port input output)) @@ -1057,7 +1071,8 @@ methods, return the applicable compression." (define (open-server-socket address) "Return a TCP socket bound to ADDRESS, a socket address." (let ((sock (socket (sockaddr:fam address) SOCK_STREAM 0))) - (setsockopt sock SOL_SOCKET SO_REUSEADDR 1) + (configure-socket sock #:options (cons (list SO_REUSEADDR 1) + %default-socket-options)) (bind sock address) sock)) -- 2.28.0 ------------=_1605503581-12909-1-- From unknown Mon Aug 18 11:17:34 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#44567] [PATCH] publish: Improve HTTP performance when not using --cache. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 16 Nov 2020 08:27:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44567 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 44567-done@debbugs.gnu.org Received: via spool by 44567-done@debbugs.gnu.org id=D44567.1605515170426 (code D ref 44567); Mon, 16 Nov 2020 08:27:02 +0000 Received: (at 44567-done) by debbugs.gnu.org; 16 Nov 2020 08:26:10 +0000 Received: from localhost ([127.0.0.1]:55009 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1keZpy-00006n-Bl for submit@debbugs.gnu.org; Mon, 16 Nov 2020 03:26:10 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55738) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1keZpw-00006W-HS for 44567-done@debbugs.gnu.org; Mon, 16 Nov 2020 03:26:08 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58438) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1keZpr-0000aW-9j; Mon, 16 Nov 2020 03:26:03 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=47030 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1keZpp-0008HB-AL; Mon, 16 Nov 2020 03:26:02 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20201111035727.11184-1-maxim.cournoyer@gmail.com> <87361dcgzc.fsf@gnu.org> <87mtzhdgvr.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 26 Brumaire an 229 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 16 Nov 2020 09:26:00 +0100 In-Reply-To: <87mtzhdgvr.fsf@gmail.com> (Maxim Cournoyer's message of "Mon, 16 Nov 2020 00:12:40 -0500") Message-ID: <87d00d4siv.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Maxim Cournoyer skribis: > I didn't observe any meaningful difference, it perhaps felt more > 'steady'. The value comes from what Linux, the kernel, configures as a > max value by default (see 'man tcp' and 'cat > /proc/sys/net/core/wmem_max'). I see, makes sense! >>> (make-gzip-output-port (response-port response) >>> #:level level >>> - #:buffer-size (* 64 1024))) >>> + #:buffer-size %default-buffer-size)) >> >> Does the gzip buffer size have to match the TCP buffer size? I=E2=80=99= d say >> not necessarily, so perhaps we should have a separate variable for the >> gzip buffer size (or keep it as is). > > It doesn't, but if we're going to pick some value, it seems the network > one (SO_SNDBUF) should be the lowest common denominator. Since we're > augmenting the previous value, I've just all made them agree. OK. >> That=E2=80=99s it, thank you! > > Thanks for the review! Pushed to the 1.2.0-version branch. Great, thanks! Ludo=E2=80=99.