From unknown Wed Jun 18 23:10:09 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#78773 <78773@debbugs.gnu.org> To: bug#78773 <78773@debbugs.gnu.org> Subject: Status: [PATCH] Speedup url-retrieve-synchronously for low-latency connections Reply-To: bug#78773 <78773@debbugs.gnu.org> Date: Thu, 19 Jun 2025 06:10:09 +0000 retitle 78773 [PATCH] Speedup url-retrieve-synchronously for low-latency co= nnections reassign 78773 emacs submitter 78773 Steven Allen severity 78773 normal tag 78773 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 00:09:24 2025 Received: (at submit) by debbugs.gnu.org; 12 Jun 2025 04:09:24 +0000 Received: from localhost ([127.0.0.1]:54799 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPZFj-0007ME-BA for submit@debbugs.gnu.org; Thu, 12 Jun 2025 00:09:23 -0400 Received: from lists.gnu.org ([2001:470:142::17]:41560) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPZFg-0007Km-3J for submit@debbugs.gnu.org; Thu, 12 Jun 2025 00:09:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uPZFF-0000ZI-19 for bug-gnu-emacs@gnu.org; Thu, 12 Jun 2025 00:08:53 -0400 Received: from fout-b1-smtp.messagingengine.com ([202.12.124.144]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uPZFB-0008GQ-GT for bug-gnu-emacs@gnu.org; Thu, 12 Jun 2025 00:08:52 -0400 Received: from phl-compute-05.internal (phl-compute-05.phl.internal [10.202.2.45]) by mailfout.stl.internal (Postfix) with ESMTP id 12B261140175; Thu, 12 Jun 2025 00:08:48 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Thu, 12 Jun 2025 00:08:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:message-id:mime-version:reply-to:subject:subject:to :to; s=fm3; t=1749701327; x=1749787727; bh=xCarY22GUUeg9O7eLpsAJ MZVbSv1EAQk8gx6oifdD9c=; b=TJ2hWSEHRZ6hDbjtk5qC1tFNENvqAms5c+pmu MLJ0HZz8I8aplng0JkkRpUEXaMIMrzzkGz2oNvSdJv5FBoAelNCsv7s7anGp1oji lLNrcgDFhDU9a83eWSJTawcSYmsoCUwUhLJyN1yO6VBQbONI+gWCRQLRaB0th7Tj cY76VHRaQ1CteZ3+4jpzXCqo5osfW8r0ibnx7e2EG7wwPoRyuPcPYn7mki/ioSPm BYgYka8p3ETFDN2/Q9QUaltaMaUNhINAWlOMF2ZGlY+MmlC082FpdVQCU8gbTiMx 2Dpvz4/Q8qS9qYlMIfNy/IqAefc55bRtfjFWpXdosqJ50/gcw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1749701327; x= 1749787727; bh=xCarY22GUUeg9O7eLpsAJMZVbSv1EAQk8gx6oifdD9c=; b=o iu8vBTgFpkJrMbyWdXq3D9rXxmHa/zFNJ/NY2diELIFlErSdwIHEjzv3LgYqUWxp HEsR/9y/9Z8fZdegKdMqmlF5xNTYvx4v7iCgJhWzJ70OiqsjH6IXD0m+0qiVq6G3 z0TX0FXoLsHqa2fCFozf4LqYAUjcqUHEj2zZYqSMBNG1NcmR89ApHsJhjc7KEd2c eOyJHtHUKF1RgL86k0KyyirnuYhbSllqTCG6q6l8IaJkHe9d67faopMP0noA8R3O 46ZQqMMmTl7BdoUM8Eb8tHJm+doEvw6Yv11SHgZhnrST/GZSRO7Wgg9gfk8SIm8c 54uAUBYO5d/IrGHgKiuaw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddugedtjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkfggtgesmhdtreertddttdenucfh rhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivghnrd gtohhmqeenucggtffrrghtthgvrhhnpeeftedvfeeggfekvddtvdegkeefkeeggfegveef gfeihfdugfdvgedttdejtdehffenucffohhmrghinhepghhithhhuhgsrdgtohhmpdduvd ejrddtrddtrddupdhgnhhurdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepshhtvghvvghnsehsthgvsggrlhhivghnrdgtohhmpdhnsg gprhgtphhtthhopeefpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegsuhhgqdhg nhhuqdgvmhgrtghssehgnhhurdhorhhgpdhrtghpthhtohepughitghkrdhrrdgthhhirg hnghesghhmrghilhdrtghomhdprhgtphhtthhopehlrghrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 12 Jun 2025 00:08:47 -0400 (EDT) From: Steven Allen To: bug-gnu-emacs@gnu.org Subject: [PATCH] Speedup url-retrieve-synchronously for low-latency connections X-Debbugs-Cc: Date: Wed, 11 Jun 2025 21:08:45 -0700 Message-ID: <8734c51u0i.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=202.12.124.144; envelope-from=steven@stebalien.com; helo=fout-b1-smtp.messagingengine.com X-Spam_score_int: -15 X-Spam_score: -1.6 X-Spam_bar: - X-Spam_report: (-1.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NUMERIC_HTTP_ADDR=1.242, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, WEIRD_PORT=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: submit Cc: Lars Ingebrigtsen , dick X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) --=-=-= Content-Type: text/plain Tags: patch **Present context:** I'm running into an issue in emacs-syncthing [1] where making a few localhost network requests are taking a full second (blocking Emacs) when they should be instantaneous. While digging into `url-retrieve-synchronously', I found that waiting on the correct network process instead of passing nil to `accept-process-output' made these network requests instantaneous (although I have no idea why). See the attached patch. [1]: https://github.com/KeyWeeUsr/emacs-syncthing To test this patch, you can: 1. Run a simple web server on localhost. I usually use python -m http.server --bind 127.0.0.1 8000 2. Evaluate: (benchmark 100 '(url-retrieve-synchronously "http://127.0.0.1:8000")) With this patch, this form is ~16x faster on my machine. I've also tested this against a remote machine with a ~45ms latency and found a 1.25x speedup. I've confirmed that this isn't busy-waiting by modifying this code to print a message each time it loops: it loops the same number of times with or without my patch. I've confirmed that this speedup is strictly due to passing the target process to `accept-process-output' by applying my patch then changing JUST "proc" to "nil": ;; ms, so split the difference. (accept-process-output proc 0.05)) to ;; ms, so split the difference. (accept-process-output nil 0.05)) With this one change, this code goes back to being as slow as it was before. **Historical context** `url-retrieve-synchronously' was changed to wait on "nil" in: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49897, Motivated by this bug report: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49861 However, the patch in question also changed the rest of `url-retrieve-synchronously' so I'm hoping the issue lies elsewhere? In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.18.4) of 2025-06-11 built on Laptop Repository revision: 87244ef1661f9e9d7c08f65047551e8a34cd92b2 Repository branch: makepkg Windowing system distributor 'The X.Org Foundation', version 11.0.12101016 System Description: Arch Linux Configured using: 'configure 'CPPFLAGS=-I/run/user/1000/build/emacs-git/src/mps-git/build/include ' 'LDFLAGS=-L/run/user/1000/build/emacs-git/src/mps-git/build/lib -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-z,pack-relative-relocs -flto=auto' --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games --with-modules --without-m17n-flt --without-selinux --without-pop --without-gconf --disable-gc-mark-trace --with-mps=yes --enable-link-time-optimization --with-native-compilation=yes --with-xinput2 --with-x-toolkit=no --without-toolkit-scroll-bars --without-xaw3d --without-gsettings --with-cairo-xcb --without-xft --with-sound=no --with-tree-sitter --without-gpm --without-compress-install '--program-transform-name=s/\([ec]tags\)/\1.emacs/' 'CFLAGS=-march=native -mtune=native -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fomit-frame-pointer -fno-math-errno -fno-trapping-math -fno-math-errno -fno-trapping-math -flto=auto'' --=-=-= Content-Type: text/patch Content-Disposition: attachment; filename=0001-Speedup-url-retrieve-synchronously-for-low-latency-c.patch >From 4529f63f3c55b40fd43642592cd4dc7162766c7f Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Wed, 11 Jun 2025 20:19:01 -0700 Subject: [PATCH] Speedup url-retrieve-synchronously for low-latency connections On my machine, this provides a ~16x speedup for localhost request and a 1.28x speedup for a 50ms latency connection. * lisp/url/url.el (url-retrieve-synchronously): Wait for output from the target process instead of waiting for output from any process. --- lisp/url/url.el | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/lisp/url/url.el b/lisp/url/url.el index 090a952cf4c..c8dc002b0ac 100644 --- a/lisp/url/url.el +++ b/lisp/url/url.el @@ -270,17 +270,20 @@ url-retrieve-synchronously (kill-buffer proc-buffer)) ;; Accommodate hack in commit 55d1d8b. (setq proc-buffer redirect-buffer))) - (when-let* ((proc (get-buffer-process proc-buffer))) - (when (memq (process-status proc) + (if-let* ((proc (get-buffer-process proc-buffer))) + (if (memq (process-status proc) '(closed exit signal failed)) - ;; Process sentinel vagaries occasionally cause - ;; url-retrieve to fail calling callback. - (unless data-buffer - (url-debug 'retrieval "Dead process %s" url) - (throw 'done 'exception)))) - ;; Querying over consumer internet in the US takes 100 - ;; ms, so split the difference. - (accept-process-output nil 0.05))) + ;; Process sentinel vagaries occasionally cause + ;; url-retrieve to fail calling callback. + (unless data-buffer + (url-debug 'retrieval "Dead process %s" url) + (throw 'done 'exception)) + ;; Querying over consumer internet in the US takes 100 + ;; ms, so split the difference. + (accept-process-output proc 0.05)) + ;; This case shouldn't happen but this keeps us from + ;; spinning and locking up Emacs if it does. + (accept-process-output nil 0.05)))) ;; Kill the process buffer on redirects. (when (and data-buffer (not (eq data-buffer proc-buffer))) -- 2.49.0 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 02:45:31 2025 Received: (at 78773) by debbugs.gnu.org; 12 Jun 2025 06:45:31 +0000 Received: from localhost ([127.0.0.1]:55503 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPbgo-0005Ht-EO for submit@debbugs.gnu.org; Thu, 12 Jun 2025 02:45:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47332) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPbgl-0004xZ-S3 for 78773@debbugs.gnu.org; Thu, 12 Jun 2025 02:45:29 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uPbgf-0000Ty-Fi; Thu, 12 Jun 2025 02:45:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=6WgUHcbL6EIGgvPRIAJgGD1HhM7axmxywI0n/DySrtg=; b=Ciy08OQHewcl EYz1piHWxPv/y5HEMyceofnHi3j+5oa1lecpmxzNl/MknyZmzolSqeqQUwZkAamfUOEUeS1lNmMK8 fSbFBO8p3/pcFq31yzU29NdGFEzJLkMSFvVp8cw+O4wqBasZps7ZeXANweGkqbVBwZJQxlB3rv8FH gyGjJMcIuM5sgTFzRhsKwctT4wrTuKz3vS9Tvsmp1/68Jg+oS7W1IG0X5iodJS8SsRxcwrxR3QOrl um5Q0q3r3PTyB44Xwn5OihXkZ4xkSLr3MiljMDuLOxC79OV+HyeMZLCNHnIhKQS611M4e1NeYGAOn 7Vy0nnF5CSl7HI3q33TVOw==; Date: Thu, 12 Jun 2025 09:45:17 +0300 Message-Id: <8634c5h30i.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen , Robert Pluim In-Reply-To: <8734c51u0i.fsf@stebalien.com> (bug-gnu-emacs@gnu.org) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org, dick.r.chiang@gmail.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > Cc: Lars Ingebrigtsen , dick > Date: Wed, 11 Jun 2025 21:08:45 -0700 > From: Steven Allen via "Bug reports for GNU Emacs, > the Swiss army knife of text editors" > > **Present context:** > > I'm running into an issue in emacs-syncthing [1] where making a few > localhost network requests are taking a full second (blocking Emacs) > when they should be instantaneous. While digging into > `url-retrieve-synchronously', I found that waiting on the correct > network process instead of passing nil to `accept-process-output' made > these network requests instantaneous (although I have no idea why). See > the attached patch. > > [1]: https://github.com/KeyWeeUsr/emacs-syncthing > > To test this patch, you can: > > 1. Run a simple web server on localhost. I usually use > > python -m http.server --bind 127.0.0.1 8000 > > 2. Evaluate: (benchmark 100 '(url-retrieve-synchronously "http://127.0.0.1:8000")) > > With this patch, this form is ~16x faster on my machine. I've also > tested this against a remote machine with a ~45ms latency and found a > 1.25x speedup. > > I've confirmed that this isn't busy-waiting by modifying this code to > print a message each time it loops: it loops the same number of times > with or without my patch. > > I've confirmed that this speedup is strictly due to passing the target > process to `accept-process-output' by applying my patch then changing > JUST "proc" to "nil": > > ;; ms, so split the difference. > (accept-process-output proc 0.05)) > > to > > ;; ms, so split the difference. > (accept-process-output nil 0.05)) > > With this one change, this code goes back to being as slow as it was before. What happens if you leave the 1st argument of accept-process-output at its current nil value, but change the 2nd argument to be 0.005 instead of 0.05 (i.e., 10 times smaller)? Also, how many other sub-processes (of any kind, not just network processes) do you have in that session when you are testing this issue? > **Historical context** > > `url-retrieve-synchronously' was changed to wait on "nil" in: > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49897, > > Motivated by this bug report: > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49861 > > However, the patch in question also changed the rest of > `url-retrieve-synchronously' so I'm hoping the issue lies elsewhere? Unfortunately, I doubt that we will get any useful answers to this question. We need to understand better why asking for output from a single process has such a dramatic effect in your case with localhost requests. If it happens only with localhost requests, perhaps we could make some changes only for that case. Robert, any ideas or suggestions? From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 04:41:52 2025 Received: (at 78773) by debbugs.gnu.org; 12 Jun 2025 08:41:52 +0000 Received: from localhost ([127.0.0.1]:56182 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPdVP-0006P4-P2 for submit@debbugs.gnu.org; Thu, 12 Jun 2025 04:41:52 -0400 Received: from mail-wr1-x42b.google.com ([2a00:1450:4864:20::42b]:49456) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1uPdVN-0006Oc-Lc for 78773@debbugs.gnu.org; Thu, 12 Jun 2025 04:41:50 -0400 Received: by mail-wr1-x42b.google.com with SMTP id ffacd0b85a97d-3a365a6804eso394619f8f.3 for <78773@debbugs.gnu.org>; Thu, 12 Jun 2025 01:41:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749717703; x=1750322503; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=XrJ2pQecjJn1tKoq9OI5hM0FHaF3cDox5Daf0gaOjsI=; b=nffA8udnwjB5sshlXwdV+7zIazcQzw7Ljxje5gcCQ5UovppDA+IXw3VpEWl4mkh5LE WE0PLnLjspNGYdzCMqTmbKxDK+4ciBhX6bpso7t22vFPZDpxG8sZuoGFFmImWVkNc4hd kQ8RqdoC4BDM0kA/iWHS+7CFeq0apQsTRhfNzYvHAirsLfKh3+fs0VfaliRyGoqUd1/Z Tr9k3u646saF4b9sC3cpZ2DZqtmqTwOj8bEdcgXR4H9BLa1E83YmXzAQKbhM9eJaJJpC WH8GbKziLjImq9nFsh5mkGo37d+qAiG6REEJ6HeA9VLNpfuCMfjktviywe4/PWDWnngy NMoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749717703; x=1750322503; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XrJ2pQecjJn1tKoq9OI5hM0FHaF3cDox5Daf0gaOjsI=; b=DU8IZQZ7fV8yZYxjuEDQVRjpehds/xOr5gpEXAd2ETUyjE7tvrr7msF6EUUo6rL0i3 EJJBoW4DbNuKM9lOJX0ViNPx7m718ClXd2Rq8kcXAbwvfrisXX5vah3VOT3bcSj8cLss C6mGaicZijctfY7YUhWCpU9jwmBnaJKe41z5OvueS3DMrV/kzBxHhrJ7IRqUkRYOc9NV 4ZuilC+ufpPH9Aeq76jX+ynmFf/C2gXGtQRx0zTsw/Fi9MboXCYXdgl9s/r6swIaYB/W hvYmt3Wstg+miqfxppJ2zEh2iPZ+0FVwiFz7STOrTqTAU5ubupvLzZHpcGBRBJsfJVu0 rebg== X-Forwarded-Encrypted: i=1; AJvYcCUjLqP00YPqaXZ3fGxkuK4RlauzOBclQXstyP4KwaGs8Zq9UIlnET5yyyLGRohNeIw+ZciS2w==@debbugs.gnu.org X-Gm-Message-State: AOJu0Yy+vza2AnE2CO30Oiv5lHw1GG5WPKDMlnAwzvlIl68w1ccv1+s0 HGd5E9X1zN4Ptal9Rpr7w41PojR9ll2NQtcxzriJg6IMonbWThhjKUcD7msiyWEF X-Gm-Gg: ASbGncsdIZtw56Pig92KfFkg9i3GLqx49CmKDOSS20AC7+KNTl1jZlsilXmq3BkmeGT PWG47jq9pWGL0R4u6T0P46RsaFEmKq7fRH8JsZkVgesywOAauYXB2ChGp2MiaywCyIfbAk+Nk+h OkUXyMo3fYXcklzRT4MgElKiyDtV+rRhwNkQ2Cg0KcIrA3RraYlxVoy9eG6w+7fEFv/XAkY5J5i N0NJC7RUDai9P1qDQEyeV0TaVVGdGQJjI0iUXc3VmjgMpnnodNwIm+NZfL63nZVEw4mg0z19J9Y KknSL14WJWaC2tSfHSTA9dnJZCJwoY8XQTz7Y5lGGShGBAs1aQ== X-Google-Smtp-Source: AGHT+IGFXV4fL+7jg3uacAsPFNonGQV+sNwyTu23dYDfI0ZZMtoiIsG4ShUBfGSQqsGzrpx6usq4dA== X-Received: by 2002:a05:6000:2283:b0:3a3:7ba5:93a5 with SMTP id ffacd0b85a97d-3a55869cd37mr5292303f8f.26.1749717703021; Thu, 12 Jun 2025 01:41:43 -0700 (PDT) Received: from rltb ([2a01:e0a:3f3:fb51:949a:873a:3ff3:56b6]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a561b65ab2sm1324278f8f.96.2025.06.12.01.41.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jun 2025 01:41:42 -0700 (PDT) From: Robert Pluim To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <8634c5h30i.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> Date: Thu, 12 Jun 2025 10:41:41 +0200 Message-ID: <87frg59wsa.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org, Steven Allen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) (I dropped dick chiang from the CC=CA=BCs) >>>>> On Thu, 12 Jun 2025 09:45:17 +0300, Eli Zaretskii said: Eli> What happens if you leave the 1st argument of accept-process-outpu= t at Eli> its current nil value, but change the 2nd argument to be 0.005 ins= tead Eli> of 0.05 (i.e., 10 times smaller)? That speeds it up, although not as dramatically Eli> Also, how many other sub-processes (of any kind, not just network Eli> processes) do you have in that session when you are testing this Eli> issue? The answer to that is important, since there are a couple of loops in `wait_reading_process_output' that depend on that, and can trigger DNS or TLS operations as a result. But I=CA=BCve just tested this in 'emacs -Q' with zero processes, and I see the speedup. >> **Historical context** >>=20 >> `url-retrieve-synchronously' was changed to wait on "nil" in: >>=20 >> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D49897, >>=20 >> Motivated by this bug report: >>=20 >> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D49861 >>=20 >> However, the patch in question also changed the rest of >> `url-retrieve-synchronously' so I'm hoping the issue lies elsewhere? Eli> Unfortunately, I doubt that we will get any useful answers to this Eli> question. We need to understand better why asking for output from= a Eli> single process has such a dramatic effect in your case with localh= ost Eli> requests. If it happens only with localhost requests, perhaps we Eli> could make some changes only for that case. Eli> Robert, any ideas or suggestions? The fact that it=CA=BCs localhost shouldn=CA=BCt matter. The code in questi= on is convoluted, as you well know, so figuring out what=CA=BCs going on needs us to have a good understanding of which cases trigger this behaviour. But the evidence so far points at the handling of the timeout. Robert --=20 From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 07:10:34 2025 Received: (at 78773) by debbugs.gnu.org; 12 Jun 2025 11:10:35 +0000 Received: from localhost ([127.0.0.1]:56843 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPfpK-0002Wn-Em for submit@debbugs.gnu.org; Thu, 12 Jun 2025 07:10:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42258) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPfpI-0002W6-D2 for 78773@debbugs.gnu.org; Thu, 12 Jun 2025 07:10:33 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uPfpC-0004r3-AI; Thu, 12 Jun 2025 07:10:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=tm0rVmm18pcFN8P+GaMLSvijCMxHGm/Hf9RkFBRIN+4=; b=NhrGIJuidVAjnN1G2cIM ReYzxKakniN6ulsQRbk/gZ094uVMxpN1qwIQUp5bY9G2u2A+5JjiOmXMEaAVyQJvxs/fmsPCwzm8f FKg+UPLj9O/wGxVPPKQOHXYtZB2KmQP1GcLxl1Ue6R+sZacPhDDEitRbPjxZ/Yaq96MaOs/8gVwf9 JxJZgkJ+EHLxjauqpCcwuXJhxFNzxzvrh0DCW7fxUYqVwAElFcstt3chp8CFE+yc3yxeoIK36spDc XmUTOl1KjwncbX6uOU/ellNOek+/qrzEMjs8ftihs1lMbAsJE1juIAiF1F4M84RF+gtx/aj7Qznpo 0cp8W4e29Hd6jQ==; Date: Thu, 12 Jun 2025 14:10:23 +0300 Message-Id: <86ldpxfc68.fsf@gnu.org> From: Eli Zaretskii To: Robert Pluim In-Reply-To: <87frg59wsa.fsf@gmail.com> (message from Robert Pluim on Thu, 12 Jun 2025 10:41:41 +0200) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org, steven@stebalien.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Robert Pluim > Cc: Steven Allen , 78773@debbugs.gnu.org, > larsi@gnus.org > Date: Thu, 12 Jun 2025 10:41:41 +0200 > > >>>>> On Thu, 12 Jun 2025 09:45:17 +0300, Eli Zaretskii said: > > Eli> What happens if you leave the 1st argument of accept-process-output at > Eli> its current nil value, but change the 2nd argument to be 0.005 instead > Eli> of 0.05 (i.e., 10 times smaller)? > > That speeds it up, although not as dramatically > > Eli> Also, how many other sub-processes (of any kind, not just network > Eli> processes) do you have in that session when you are testing this > Eli> issue? > > The answer to that is important, since there are a couple of loops in > `wait_reading_process_output' that depend on that, and can trigger DNS > or TLS operations as a result. But Iʼve just tested this in 'emacs -Q' > with zero processes, and I see the speedup. If this happens with a single sub-process, it's strange. It means that we somehow wait for the full timeout even though some output from the single sub-process arrives, which should have ended the wait. Or what am I missing? > Eli> Unfortunately, I doubt that we will get any useful answers to this > Eli> question. We need to understand better why asking for output from a > Eli> single process has such a dramatic effect in your case with localhost > Eli> requests. If it happens only with localhost requests, perhaps we > Eli> could make some changes only for that case. > > Eli> Robert, any ideas or suggestions? > > The fact that itʼs localhost shouldnʼt matter. The code in question is > convoluted, as you well know, so figuring out whatʼs going on needs us > to have a good understanding of which cases trigger this > behaviour. But the evidence so far points at the handling of the > timeout. In the case of a remote host, the speedup was marginal, so I think the localhost does matter, somehow. Perhaps instrumenting wait_reading_process_output with printf's would help to understand the control flow in the case of nil and non-nil PROCESS argument? From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 13:27:25 2025 Received: (at 78773) by debbugs.gnu.org; 12 Jun 2025 17:27:25 +0000 Received: from localhost ([127.0.0.1]:60044 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPli0-0000lq-Ow for submit@debbugs.gnu.org; Thu, 12 Jun 2025 13:27:25 -0400 Received: from fhigh-a6-smtp.messagingengine.com ([103.168.172.157]:37201) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPlhx-0000lQ-Io for 78773@debbugs.gnu.org; Thu, 12 Jun 2025 13:27:22 -0400 Received: from phl-compute-05.internal (phl-compute-05.phl.internal [10.202.2.45]) by mailfhigh.phl.internal (Postfix) with ESMTP id C50101140215; Thu, 12 Jun 2025 13:27:15 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Thu, 12 Jun 2025 13:27:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749749235; x= 1749835635; bh=bVgSSrGMo+vnzx7nAMxa7Q/0vIbgFeZyxYf2j+SgWtA=; b=a wYYcXgK6HPoERVCp72MWkCbX62gDBY5Na98sc3jxE4paLbG1Jzk+Sk3xBkXfG7l8 svqmj30NqsdyAt486zPZ0B+91+L8GxuYciaRWAcmz6W7LnSt4dhU6VVsSKCeoJsi v7yiyWuFpyt2qL7U0FTQS3TGyXZYaLaa1hYzqkVuQfEv6isOkygapRSxywlj17i8 z54qYElUx4OZoYJmNEHfKUM3PaY3xENXFT/QgC3g8oD3HGTQubn7FAan1t9VzTt8 /7J2+TA9/sWzYxIomBH5RvzAdFX4Pqa9Ov172lE2mhh2aluOJYjaOqVt6riL8a/1 q/EhZtZ1yVDTb5l5lxrIg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749749235; x=1749835635; bh=bVgSSrGMo+vnzx7nAMxa7Q/0vIbgFeZyxYf 2j+SgWtA=; b=B6t3gokCduYAamu/MoT97j1tFFO92MWGZMLA6d1hVeU2xoRJXZu 9mHxnbgEbKomqQIRUaCswwxbiXXMvuReLs+Yja1a2cPed27grvDvr7gxuoKcckoN QXgjsl3qW2kjNnOfaFGBJXix7TjOzFxfMxG7fNWrZOHcyESA3VKRStgyXqfm4Hix 5EfLMoQbUd+FcPLCbGSS4VTqIeHLC46Obo+ozpZn5f/qhtL/6Hfvim/yIJcU4WtM zP7sgNDEa7Bo6dAus0p2y4b5hHw1VZc3s556G0zmNtTapK5JJRP+5L4+5v34G10F zR+NZvuFhQY6dcvmG5MWe7kSE0MWY1pSwtA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugdduheeijecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeehgeffteevudefuedvfefhgeduffethfel uedtvddutdejgfetgeetgfeitdevgfenucffohhmrghinhepuddvjedrtddrtddrudenuc evlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsthgvvhgv nhesshhtvggsrghlihgvnhdrtghomhdpnhgspghrtghpthhtohepgedpmhhouggvpehsmh htphhouhhtpdhrtghpthhtohepvghlihiisehgnhhurdhorhhgpdhrtghpthhtoheprhhp lhhuihhmsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjeekjeejfeesuggvsggsuhhgsh drghhnuhdrohhrghdprhgtphhtthhopehlrghrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 12 Jun 2025 13:27:14 -0400 (EDT) From: Steven Allen To: Eli Zaretskii , Robert Pluim Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <8634c5h30i.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> Date: Thu, 12 Jun 2025 10:27:13 -0700 Message-ID: <871proeuq6.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> With this one change, this code goes back to being as slow as it was before. > > What happens if you leave the 1st argument of accept-process-output at > its current nil value, but change the 2nd argument to be 0.005 instead > of 0.05 (i.e., 10 times smaller)? Now testing with: emacs -Q --load=url.el --batch --eval \ "(benchmark 100 '(url-retrieve-synchronously \"http://127.0.0.1:8000\"))" - Changing the timeout from 0.05 to 0.005 drops the time from 1.25 seconds to 0.63 seconds (~2x faster). - My patch drops it to 0.056 (22x faster in a clean environment). However note: changing the timeout from 0.05 to 0.005 makes the wait loop spin 6 times for higher latency (non-localhost, ~40ms) requests whereas it loops exactly once with the default timeout. > Also, how many other sub-processes (of any kind, not just network > processes) do you have in that session when you are testing this > issue? >From list-processes, 13-14 when testing this originally (including network processes, listening sockets, etc.). >> However, the patch in question also changed the rest of >> `url-retrieve-synchronously' so I'm hoping the issue lies elsewhere? > > Unfortunately, I doubt that we will get any useful answers to this > question. We need to understand better why asking for output from a > single process has such a dramatic effect in your case with localhost > requests. If it happens only with localhost requests, perhaps we > could make some changes only for that case. Testing with a server with a 30ms latency (tested with "emacs -Q"), I get: 1. A 1.4x speedup with my patch. 2. A 1.3x speedup by simply changing the timeout. But there's a lot more noise in the higher-latency tests. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 14:02:52 2025 Received: (at 78773) by debbugs.gnu.org; 12 Jun 2025 18:02:52 +0000 Received: from localhost ([127.0.0.1]:60200 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPmGJ-0003C9-FA for submit@debbugs.gnu.org; Thu, 12 Jun 2025 14:02:51 -0400 Received: from fout-b2-smtp.messagingengine.com ([202.12.124.145]:47423) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPmGH-0003Bo-Gi for 78773@debbugs.gnu.org; Thu, 12 Jun 2025 14:02:50 -0400 Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfout.stl.internal (Postfix) with ESMTP id 8692F11401D1; Thu, 12 Jun 2025 14:02:43 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Thu, 12 Jun 2025 14:02:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-transfer-encoding:content-type:content-type :date:date:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:subject:subject:to:to; s=fm3; t=1749751363; x=1749837763; bh=Nz1pAdw9T8ZoJyoRqn0N8Q2VQeLvAp0W taPsTsALrWQ=; b=qfVq2WwkqxMK6TLif4rywZHTkG/oEV8zBdbvffSn+kNxZWQo AFhxdpeL36FAbQWvQlGhndjL5ISzZSDzrS9Bcs9NYPlTGqhIZdUUzOMITWCVjk0r j4W6IPDsL8Y+jjwZv9OBrjw+NkiMRsc6XMo/UtNwIGgu1QkJkYP1kzE+m7KY47qm XsohzdKiZSiooNuCGacDRpmD1sIw05CJQe0p0BF8OwiU2ass7eNzxxJcF8HSrGRo XoRWufyzpLqWGRvJUMLTx2Wv133jPuf5zYNy1XPd920pl0fH7Vvv/IJ9ytM3ZaO8 wlbT4o32mAyxak572PMJQVBD3aFPihMqDFlgtw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1749751363; x= 1749837763; bh=Nz1pAdw9T8ZoJyoRqn0N8Q2VQeLvAp0WtaPsTsALrWQ=; b=h P1L85GWxo9YwFbxxC7xb0awXDoSbrKeeSjJd3isH/wi+cAGsyySzeaNZfqkHFyr6 itxhbZJVfzcLY4CkU5xiffaLq4oED7My/rPQFDHL3y6H5dt7n597QuGCQb4VnjFN rePa5EPRzGpytt6BuCqWa+HHcy793GvZHNqqi7HAaQONso7kVVrWcMAClxz1OVoc gAkB+R7GvCInZVl8PGbjlN6kWCCz+SBrA8u1U5bWpOJBk7JU6JKHQldYZU1BHAnt 2iUKhAt9tvGeXWqH7SQ+y+gYm8QJle6U9IB3fdUM4px8qJwiVc49HiYF+FLg9VfS YdbVEws0iHgyxQsHRR9uA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugdduheejgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgfgsehtqhertddttdej necuhfhrohhmpefuthgvvhgvnhcutehllhgvnhcuoehsthgvvhgvnhesshhtvggsrghlih gvnhdrtghomheqnecuggftrfgrthhtvghrnhepudevlefgtdekheekgffhtdegheduhfeg tdfgffefteetgfetkeeivddthefhkeetnecuffhomhgrihhnpehgnhhurdhorhhgnecuve hluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhtvghvvghn sehsthgvsggrlhhivghnrdgtohhmpdhnsggprhgtphhtthhopeegpdhmohguvgepshhmth hpohhuthdprhgtphhtthhopegvlhhiiiesghhnuhdrohhrghdprhgtphhtthhopehrphhl uhhimhesghhmrghilhdrtghomhdprhgtphhtthhopeejkeejjeefseguvggssghughhsrd hgnhhurdhorhhgpdhrtghpthhtoheplhgrrhhsihesghhnuhhsrdhorhhg X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 12 Jun 2025 14:02:42 -0400 (EDT) From: Steven Allen To: Eli Zaretskii , Robert Pluim Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86ldpxfc68.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> Date: Thu, 12 Jun 2025 11:02:41 -0700 Message-ID: <87y0twdeim.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> The fact that it=CA=BCs localhost shouldn=CA=BCt matter. The code in que= stion is >> convoluted, as you well know, so figuring out what=CA=BCs going on needs= us >> to have a good understanding of which cases trigger this >> behaviour. But the evidence so far points at the handling of the >> timeout. > > In the case of a remote host, the speedup was marginal, so I think > the localhost does matter, somehow. It looks like a latency thing. It's not that it's localhost specific, it's that the effect is more pronounced on low-latency connections because something, somewhere is adding a tiny delay (likely the timeout). When I test against my router (10ms latency, how awful!), I get a 1.25x speedup (1.2x speedup with just your timeout change). I've also tested with IP addresses and reproduced the same results, so it's not related to DNS lookups. > Perhaps instrumenting wait_reading_process_output with printf's would > help to understand the control flow in the case of nil and non-nil > PROCESS argument? I'll do that today. I have a suspicion that fast requests waiting on a single process exit early here: https://https.git.savannah.gnu.org/cgit/git/emacs.git/tree/src/process.c?h= =3D81a3e4e51167be51c63eae682331210bc62f7280#n5562 >> But the evidence so far points at the handling of the timeout. I can say that we're definitely NOT waiting for the timeout: increasing the timeout (to, e.g., 1-2s) doesn't slow things down. I suspect having a really short timeout is causing us to abort wait_reading_process_output early before it hits some expensive part. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 14:29:18 2025 Received: (at 78773) by debbugs.gnu.org; 12 Jun 2025 18:29:19 +0000 Received: from localhost ([127.0.0.1]:60314 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPmfu-0004vp-Ei for submit@debbugs.gnu.org; Thu, 12 Jun 2025 14:29:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36226) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPmfr-0004vb-Lb for 78773@debbugs.gnu.org; Thu, 12 Jun 2025 14:29:16 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uPmfm-0006E1-43; Thu, 12 Jun 2025 14:29:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=Af6wgzBN4MRUYabkAB/9Y0DUSzakS+tFNWI0+eMzQg8=; b=KB89Tz0APyMJ h5wQdXG96oAawwEi+m22YlhoUzKkWBF6xwhvQhutAKfMLVzuC1DSX+hzVnQklvtxBzQ8UGVd0zzp0 6nYOQQyqj5yw1cKXhek5GzVfndJDaUsz+G8z/PybNZS2ySLzogYmBU7Z5wsZa/V/107gLNZyagWvV NV6YyyY0DuJoguogqMjcaePsZQxKlhN0DnW4HGk64Ut9BMVNYzZFeQlAlLlvdimNQHXs12k3LZhns r013wRr9GMDx+KM4XMTph9dIz53RM834RzoM49kNxZVzy/ltXbE8lp66jvXIabBipwjONsXtK309F ywCKSc4eyvO3u2sSFVd9zA==; Date: Thu, 12 Jun 2025 21:28:57 +0300 Message-Id: <86zfecerva.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87y0twdeim.fsf@stebalien.com> (message from Steven Allen on Thu, 12 Jun 2025 11:02:41 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: 78773@debbugs.gnu.org, larsi@gnus.org > Date: Thu, 12 Jun 2025 11:02:41 -0700 > > > Perhaps instrumenting wait_reading_process_output with printf's would > > help to understand the control flow in the case of nil and non-nil > > PROCESS argument? > > I'll do that today. I have a suspicion that fast requests waiting on a > single process exit early here: > > https://https.git.savannah.gnu.org/cgit/git/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5562 The condition for that block is if (wait_proc && ! EQ (wait_proc->status, Qrun) && ! connecting_status (wait_proc->status)) And the comment says "Don't wait for output from a non-running process." Is the case here that the network connection was already closed when we read from the sub-process? I thought that was not the case. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 15:20:41 2025 Received: (at 78773) by debbugs.gnu.org; 12 Jun 2025 19:20:42 +0000 Received: from localhost ([127.0.0.1]:60625 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPnTd-0008Tv-CN for submit@debbugs.gnu.org; Thu, 12 Jun 2025 15:20:41 -0400 Received: from fhigh-b8-smtp.messagingengine.com ([202.12.124.159]:54475) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPnTZ-0008TQ-V4 for 78773@debbugs.gnu.org; Thu, 12 Jun 2025 15:20:38 -0400 Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfhigh.stl.internal (Postfix) with ESMTP id 55C7C25402F1; Thu, 12 Jun 2025 15:20:32 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Thu, 12 Jun 2025 15:20:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749756032; x= 1749842432; bh=Ue/ifrVf8tJ7LVewsm/tpQZk9qn0Np6uF2zM3Mkfc7w=; b=Q bFNo1L28z2zGVDZnCaaWTeGMKJX+Pz9lEuOdmAXZLJpGLoXUfVNolqrvnGML7liS bqhRU4BH3OH+9e2BLxAQoQ/uQ+T5Fo1SERgIBzDJueHh1LlyKV9HW7XKStSvLdmB OW2c0ytNyzhEJ+tM+bkArUNs77SR+JKdgJOKXaQYWV0loEc/4SJa1HhFKkGPnYL/ CRMAxvflz0RDgiFndjCVWxhavfSZfXkRVDR9ruUTn2URK9XlDAJLXx5aDeeoQ1/C k0D3BSVkG6t5JDf1jFWtORoLYJTw8ec7/x8Yiwp+GEATriVandSfnPEa1rolyKBn Lj7YXcZn1AV6WKpiyzRFQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749756032; x=1749842432; bh=Ue/ifrVf8tJ7LVewsm/tpQZk9qn0Np6uF2z M3Mkfc7w=; b=BXbFO/dJq904y5RWhTiOXL5KoJjgexTfdqs8IQ9pbYr59oaBNKU DZ5l+FqL83n7RZ5dfo4nsmKX+x4W0jM4TxTWF3lXPMOGuy8w8ZNz+L+n5nYHXc7t hCd/SQ2GXkCpsZ7e0ax/IaH+p8+7cSE0KeSuuw2R6sj6/cR8hSWdZBlzUUdt/EhM QW7mWzF14EXGLFcpNs2hyOshewoGkkfP/zHsU/LNZy+87JL0gHPlELoc9PNwVNwG oeCK+ikQb8knVjFxmwbxaIUbknULb/LiFTcA8ZQN7vmLkMbndx3qNLf/fxmwAwSJ wlRuUoyRHLgL6w8HF/f+sCgK36cB/4BC/OQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugdduheeklecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeejffefuddvieefgeevkeeggfelhefhffeg leetffetuedttefggeffheeufeektdenucffohhmrghinhepghhnuhdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsthgvvhgvnhes shhtvggsrghlihgvnhdrtghomhdpnhgspghrtghpthhtohepgedpmhhouggvpehsmhhtph houhhtpdhrtghpthhtohepvghlihiisehgnhhurdhorhhgpdhrtghpthhtoheprhhplhhu ihhmsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjeekjeejfeesuggvsggsuhhgshdrgh hnuhdrohhrghdprhgtphhtthhopehlrghrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 12 Jun 2025 15:20:31 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86zfecerva.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> Date: Thu, 12 Jun 2025 12:20:30 -0700 Message-ID: <87v7p0ahs1.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Thu, 12 Jun 2025 11:02:41 -0700 >> >> > Perhaps instrumenting wait_reading_process_output with printf's would >> > help to understand the control flow in the case of nil and non-nil >> > PROCESS argument? >> >> I'll do that today. I have a suspicion that fast requests waiting on a >> single process exit early here: >> >> https://https.git.savannah.gnu.org/cgit/git/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5562 > > The condition for that block is > > if (wait_proc > && ! EQ (wait_proc->status, Qrun) > && ! connecting_status (wait_proc->status)) > > And the comment says "Don't wait for output from a non-running > process." Is the case here that the network connection was already > closed when we read from the sub-process? I thought that was not the > case. With my patch, it does hit that case, but on the third loop through so it's not exiting early. I also tried setting the timeout in `accept-process-output' to nil and that got me the same speedup as my fix but ended up looping 23 times so there's something really strange going on with the timeout here. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 12 16:54:13 2025 Received: (at 78773) by debbugs.gnu.org; 12 Jun 2025 20:54:13 +0000 Received: from localhost ([127.0.0.1]:32843 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPow8-0006Q8-Bs for submit@debbugs.gnu.org; Thu, 12 Jun 2025 16:54:12 -0400 Received: from fhigh-b1-smtp.messagingengine.com ([202.12.124.152]:57641) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPow5-0006Pl-4W for 78773@debbugs.gnu.org; Thu, 12 Jun 2025 16:54:09 -0400 Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfhigh.stl.internal (Postfix) with ESMTP id 5581D2540092; Thu, 12 Jun 2025 16:54:03 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Thu, 12 Jun 2025 16:54:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749761643; x= 1749848043; bh=FynnKFuVSidWqpxJgQvO20Jb2ANTX4zw0IEILpYaSGc=; b=n aiQEjvdmd+GOas/OXNmEWnOJ4ZFemEcggYnFZ/mCsobwXEnl8Uae4ZDjrQ+7iRY6 6nJKm8A2dQbMJgmzYYTOip7OOdR72jCCs1Q4qV28zhP68UBmoRFkY3N3Jvx1uCus TaHl1PdrTMz+7tEl0Q7lfOxJmqiHO4+2pkfFMbiSNHyD+tNjwjrdgKr8fqb5CUhM XFcpfdhNegGFJtrkvzDyYkJz5N/jKTQEv6eNtHH6bw3/N/strETFmLSFnj+dvsNQ w6z6JQduGocBzfKip1acnrILYuDGAMo98woZZv2MhYJyr5GUoBVhB0GbuUpIaQrw D5P5GxRAgNbcsvSUuO/Dg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749761643; x=1749848043; bh=FynnKFuVSidWqpxJgQvO20Jb2ANTX4zw0IE ILpYaSGc=; b=WUZzcKohsm7KBHjA7pbNLN303S527GEZ1c/cDw4sGxOczfz4Tq1 SgI4s4SpHUNiLwdAUvT9YefxTzhYdE1Vn+SuPBEudnL4wMhcOVrcSthHfQBWHx2r PWmo57G89R8T14+uZKrD+nnVspSN7A0oKObWcnHhnnbAXE+6M+hXNzLlNQ0foT8r 50Tl7XCQ1c7ph8WojgGuqf24a0Y5eoI993S5wGMcVwyaDxpmvnVwEF1Llf/FFcHA 2PEBZ6Gso61XBpTe7stViqqxA32vCpak/I73P5sskJqg4tSi8ERmpqRJBJ/MWdIa uPwWqUd8Mhe5TPyt3g2s1CUoa+1wTQMvkiw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugdduiedtkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesmhdtreertddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeeitedvhfffuefhtdeutdegfeegieeiteev vefgueefieevvdehfeejffffgfevjeenucffohhmrghinhepghhnuhdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsthgvvhgvnhes shhtvggsrghlihgvnhdrtghomhdpnhgspghrtghpthhtohepgedpmhhouggvpehsmhhtph houhhtpdhrtghpthhtohepvghlihiisehgnhhurdhorhhgpdhrtghpthhtoheprhhplhhu ihhmsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjeekjeejfeesuggvsggsuhhgshdrgh hnuhdrohhrghdprhgtphhtthhopehlrghrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 12 Jun 2025 16:54:02 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <87v7p0ahs1.fsf@stebalien.com> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> Date: Thu, 12 Jun 2025 13:54:01 -0700 Message-ID: <87o6usadg6.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-=-= Content-Type: text/plain Steven Allen writes: > Eli Zaretskii writes: > >>> From: Steven Allen >>> Cc: 78773@debbugs.gnu.org, larsi@gnus.org >>> Date: Thu, 12 Jun 2025 11:02:41 -0700 >>> >>> > Perhaps instrumenting wait_reading_process_output with printf's would >>> > help to understand the control flow in the case of nil and non-nil >>> > PROCESS argument? >>> >>> I'll do that today. I have a suspicion that fast requests waiting on a >>> single process exit early here: >>> >>> https://https.git.savannah.gnu.org/cgit/git/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5562 >> >> The condition for that block is >> >> if (wait_proc >> && ! EQ (wait_proc->status, Qrun) >> && ! connecting_status (wait_proc->status)) >> >> And the comment says "Don't wait for output from a non-running >> process." Is the case here that the network connection was already >> closed when we read from the sub-process? I thought that was not the >> case. > > With my patch, it does hit that case, but on the third loop through so > it's not exiting early. > > I also tried setting the timeout in `accept-process-output' to nil and > that got me the same speedup as my fix but ended up looping 23 times so > there's something really strange going on with the timeout here. Ok, I think I've figured it out. We loop twice and wait on a timeout (READ_OUTPUT_DELAY_INCREMENT) the second loop because we're already done reading. The first time through, we hit the following, recording that we've gotten some output: https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5974 We hit the following, setting our timeout to READ_OUTPUT_DELAY_INCREMENT: https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5679 Then we hit the select call and wait the full timeout (nothing to read) and we finally exit here (because our timeout has passed): https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5832 *** I think what we need to do is modify: https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5978 from if (wait_proc == XPROCESS (proc)) wait = MINIMUM; to if (!wait_proc || wait_proc == XPROCESS (proc)) wait = MINIMUM; So that we set the timeout to 0 here: https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5439 And exit early. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Exit-accept-process-output-early-when-we-get-output-.patch >From 0772d96d50734fff14a17b2f207661418f5c814c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Thu, 12 Jun 2025 13:48:08 -0700 Subject: [PATCH] Exit accept-process-output early when we get output from any process Previously, the caller requested to wait on output from any process (not a specific process), we'd always end up waiting 10ms (READ_OUTPUT_DELAY_INCREMENT) one final time before returning. * src/process.c (wait_reading_process_output): abort waiting early if we either receive output from the process we were waiting on OR we weren't waiting on a process. --- src/process.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/process.c b/src/process.c index e61ec425f7e..7b8fd1357e3 100644 --- a/src/process.c +++ b/src/process.c @@ -5975,7 +5975,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd, if (nread > 0) { /* Vacuum up any leftovers without waiting. */ - if (wait_proc == XPROCESS (proc)) + if (!wait_proc || wait_proc == XPROCESS (proc)) wait = MINIMUM; /* Since read_process_output can run a filter, which can call accept-process-output, -- 2.49.0 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 02:34:22 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 06:34:22 +0000 Received: from localhost ([127.0.0.1]:37499 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uPxzY-0006WY-Gh for submit@debbugs.gnu.org; Fri, 13 Jun 2025 02:34:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43908) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uPxzT-0006Ub-EQ for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 02:34:17 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uPxzN-0004Q2-IL; Fri, 13 Jun 2025 02:34:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=qDLmA2kuvSkN2wlh+874YbTq+m7PhNT+XNz7xxPRFQY=; b=eR3Zaf4ZadoV Ca7UYX7OsWdahcx6oyzu6Zbf6rwaITBelVm1jDqv+Ni+6pPTt0NVwtoKw1EnHKJs9FnY/m5dx0H59 PnCEBfGSXyoKCnlZNjj9dBy3DCC8qLCK5k3EHRdKbBnXExgMJBK1xRyy+ZiEP2+kvv5WEYsunXG1I LjAvfu25wJrM63KNbBhNQmwvILjHX7ZDhbqumdZEc9tG2hP286HOnlNZEghctNV4KNzxN5VqlnttR H6/em5k4w+pJEcuaB5VJmg6ygyBub9WjAJgvRu3wZJq542/n5AKwt+P5vZeQ+ORG/tnOaCjy4h2gA 40MzuAz+KaWzJ7dCQa0lsg==; Date: Fri, 13 Jun 2025 09:34:06 +0300 Message-Id: <86plf8duap.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87v7p0ahs1.fsf@stebalien.com> (message from Steven Allen on Thu, 12 Jun 2025 12:20:30 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Thu, 12 Jun 2025 12:20:30 -0700 > > Eli Zaretskii writes: > > >> I'll do that today. I have a suspicion that fast requests waiting on a > >> single process exit early here: > >> > >> https://https.git.savannah.gnu.org/cgit/git/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5562 > > > > The condition for that block is > > > > if (wait_proc > > && ! EQ (wait_proc->status, Qrun) > > && ! connecting_status (wait_proc->status)) > > > > And the comment says "Don't wait for output from a non-running > > process." Is the case here that the network connection was already > > closed when we read from the sub-process? I thought that was not the > > case. > > With my patch, it does hit that case, but on the third loop through so > it's not exiting early. How come? What is wait_proc->status in that case? From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 04:58:23 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 08:58:23 +0000 Received: from localhost ([127.0.0.1]:39587 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ0Es-0008K1-Ht for submit@debbugs.gnu.org; Fri, 13 Jun 2025 04:58:23 -0400 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]:48397) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1uQ0Ep-0008JB-PB for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 04:58:16 -0400 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-450cfb6a794so11046875e9.1 for <78773@debbugs.gnu.org>; Fri, 13 Jun 2025 01:58:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749805089; x=1750409889; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=kAwHz/t9LjItrcQzLdHbqvTWfVUk6LKIoMMHQaI3Y/E=; b=cHLtF/07aMPDy0kzhxo19Y8ae/jD5hBePrOPUdXk2QonumNsLzu3tge9k5HfIE5umP TwY6lobR17bZ83iF7zZgjmX/OMlf7C1tIq/mqFdTquRLLv6X2sbDl/c6tCrXUy2Vx60I J/5+TWd3W5TjRUb2Ql6O82iQE5RIgIokk9m7lbgbTncOZOplSBW6xSMDDqgF33pDfn6i ENofu2RUYaQGcg78ArxJDkgrps4h/1VgW3ouPipxvJie5RM6OFh/4gr4OlxkpnVmPPBH 4tD9PamP8DnWWdnX5OlY4vcM4BiBaLcJYNSX0NSADKtUtfrye0OIHd6NwfOp+QFZlBgj 8gRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749805089; x=1750409889; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kAwHz/t9LjItrcQzLdHbqvTWfVUk6LKIoMMHQaI3Y/E=; b=XFEcie6L3mco5b140NqxddJO/Z62n2uyNtSJ7XA/BKI0s9Yj0taXyagwD2BTHWkML3 sqQpuQnF6s2EV2fTTYKIQWbfsOGBJOiOCxPhIGjKmJpifzTpGaON5S4msh9kOJ1SjX/z R1qX1w8SVSqqSaf0cRJTdwfIfMlehrJlbV1GyTuf4yQ+U/HWZRhPwEpJwHvklfrKDWQ1 o3a+wRxhY/7+3lDiPrLU0iPUCVTXpX82Dx/NYnGz/9G3+Qmt3bbMAv1KhJDlWbG98NXm M2UKTkrdxZHmJFlewVDLUFxB9CGjHfciVPu9shb4wDTUtrE2i/i9K7tTmQBb9Q77krjp zwgQ== X-Forwarded-Encrypted: i=1; AJvYcCV2Atw0j6+F+2nGnfgcqz0wOdBEXlGVSoMX58EUYxIoiAyKhO9Cokm1R4cn2iXOSDdLSA9Z7w==@debbugs.gnu.org X-Gm-Message-State: AOJu0YwU5aM/mjLgHmGuoxwsVw/qimcEFFo+VIFlWP2N73iMruOwb3eL VB4itlvASgHwRIovHKW6C7bo/judo/WQHSgJtAYlV1R0Ug3myQZBd4mg X-Gm-Gg: ASbGnctrRK+go5pN1D2dTQDnjy71zMUCe1qifTOTrj0rOqJm2ocUVgX6FzwL6FzCmgD BYpb8Tnob8SklAU0/VFyAOddDf2v7sNMetohL+SI6aAYQfIarr/3k74axY8EF626osbeai+JLRv noJQgO78L+RPxwGrmKe1WHpsn+f/7DCt6Mz3AsmuzbqqHWUVNCqm2lYH6D0i0jRy0xcSSMnvuR3 nDFyJno8Y2fx3QAP/o0Zt/hEYZ9bASF4nTYtFlkQFAADzH0AgUYGb2CysFUGgwulXTCdiSYD5Bp k8ApcJQ1jbzprkEBDKSJL568yLqZEE4eaeghjD8O4D9NCyT/eA== X-Google-Smtp-Source: AGHT+IGf1w7OFlH7QzLbWylvs7oXR5afKc3OoBVoU0V142784OTcwrnWL6Bsyr9ktVU88QJKSJEMYQ== X-Received: by 2002:a05:600c:138b:b0:442:ccf9:e6f2 with SMTP id 5b1f17b1804b1-45334b630ffmr22233415e9.16.1749805088752; Fri, 13 Jun 2025 01:58:08 -0700 (PDT) Received: from rltb ([2a01:e0a:3f3:fb51:c51f:cac0:9d7b:73dd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-453305a0d9dsm39052945e9.21.2025.06.13.01.58.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 01:58:08 -0700 (PDT) From: Robert Pluim To: Steven Allen Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <87o6usadg6.fsf@stebalien.com> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> Date: Fri, 13 Jun 2025 10:58:07 +0200 Message-ID: <87bjqs9fxc.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, Eli Zaretskii , larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) >>>>> On Thu, 12 Jun 2025 13:54:01 -0700, Steven Allen said: Steven> Ok, I think I've figured it out. We loop twice and wait on a ti= meout Steven> (READ_OUTPUT_DELAY_INCREMENT) the second loop because we're alr= eady done Steven> reading. Steven> The first time through, we hit the following, recording that we= 've Steven> gotten some output: Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/pro= cess.c?h=3D81a3e4e51167be51c63eae682331210bc62f7280#n5974 Steven> We hit the following, setting our timeout to READ_OUTPUT_DELAY_= INCREMENT: Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/pro= cess.c?h=3D81a3e4e51167be51c63eae682331210bc62f7280#n5679 Steven> Then we hit the select call and wait the full timeout (nothing = to read) Steven> and we finally exit here (because our timeout has passed): Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/pro= cess.c?h=3D81a3e4e51167be51c63eae682331210bc62f7280#n5832 Steven> *** Steven> I think what we need to do is modify: Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/pro= cess.c?h=3D81a3e4e51167be51c63eae682331210bc62f7280#n5978 Steven> from Steven> if (wait_proc =3D=3D XPROCESS (proc)) Steven> wait =3D MINIMUM; Steven> to Steven> if (!wait_proc || wait_proc =3D=3D XPROCESS (proc)) Steven> wait =3D MINIMUM; Steven> So that we set the timeout to 0 here: Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/pro= cess.c?h=3D81a3e4e51167be51c63eae682331210bc62f7280#n5439 Steven> And exit early. That analysis looks correct to me. It works for the original test case, at least. I=CA=BCm wondering why we=CA=BCre setting the timeout to zero and calling `pselect' again, which means recomputing the wait set again, if I haven=CA=BCt gotten lost in the twisty maze of if=CA=BCs and global variabl= es =F0=9F=98=80. We=CA=BCve got input, why can=CA=BCt we just `break'? There m= ight be some work we don=CA=BCt do, but that will get done the next time `wait_reading_process_output' is called. Robert --=20 From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 06:48:19 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 10:48:19 +0000 Received: from localhost ([127.0.0.1]:41819 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ1xL-0003FE-59 for submit@debbugs.gnu.org; Fri, 13 Jun 2025 06:48:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35268) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQ1xI-0003Ew-QP for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 06:48:17 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQ1xD-0003M6-8m; Fri, 13 Jun 2025 06:48:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=VopnovTtQsgbGimNFkfHFT9KDAOe76rIkpOpBjxXEl0=; b=FOsJlm/gs0LnuwjRjMsF cfMCYQZ2MMME37BO5EyBmRXJ7BTM0YeXxcvF8ItUYZ1Ybx+ZbwyNRwphcdlJg49cr03oq2whZeFSc lNjlRXkelv9dRQRADRHIammMw+45riqNFCxnoW67+4nIGcaIExoojP75pRaKvUT/iPPj8T3jttZzX 2smNUq/pjxPEymB7dQpa7MhedKe7Cv7zczUBIeYsW+J0N5X46694SfscSHnH7ihK4RKpc1Hu5EgSg pJUbNdG7xtdRj6tp7PimbVbNq5x+IWoUmA57KVS63MgkUZJaNj0asVv3TobNxaqWMQ35dXySHw8Vc RouKhlvfygu3sw==; Date: Fri, 13 Jun 2025 13:48:08 +0300 Message-Id: <86jz5fex3r.fsf@gnu.org> From: Eli Zaretskii To: Robert Pluim In-Reply-To: <87bjqs9fxc.fsf@gmail.com> (message from Robert Pluim on Fri, 13 Jun 2025 10:58:07 +0200) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <87bjqs9fxc.fsf@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org, steven@stebalien.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Robert Pluim > Cc: Eli Zaretskii , 78773@debbugs.gnu.org, larsi@gnus.org > Date: Fri, 13 Jun 2025 10:58:07 +0200 > > >>>>> On Thu, 12 Jun 2025 13:54:01 -0700, Steven Allen said: > > Steven> Ok, I think I've figured it out. We loop twice and wait on a timeout > Steven> (READ_OUTPUT_DELAY_INCREMENT) the second loop because we're already done > Steven> reading. > > Steven> The first time through, we hit the following, recording that we've > Steven> gotten some output: > > Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5974 > > > Steven> We hit the following, setting our timeout to READ_OUTPUT_DELAY_INCREMENT: > > Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5679 > > Steven> Then we hit the select call and wait the full timeout (nothing to read) > Steven> and we finally exit here (because our timeout has passed): > > Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5832 > > Steven> *** > > Steven> I think what we need to do is modify: > > Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5978 > > Steven> from > > Steven> if (wait_proc == XPROCESS (proc)) > Steven> wait = MINIMUM; > > Steven> to > > Steven> if (!wait_proc || wait_proc == XPROCESS (proc)) > Steven> wait = MINIMUM; > > Steven> So that we set the timeout to 0 here: > > Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5439 > > Steven> And exit early. > > That analysis looks correct to me. It works for the original test > case, at least. I didn't yet have time to look into this closely enough to have an opinion. > Iʼm wondering why weʼre setting the timeout to zero and calling > `pselect' again, which means recomputing the wait set again, if I > havenʼt gotten lost in the twisty maze of ifʼs and global variables > 😀. Weʼve got input, why canʼt we just `break'? There might be some > work we donʼt do, but that will get done the next time > `wait_reading_process_output' is called. Try looking at the Git history of this code, maybe it will tell you how this code evolved, and bring up past discussions and bug reports which caused us to make the code as it is today. From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 07:59:52 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 11:59:52 +0000 Received: from localhost ([127.0.0.1]:43307 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ34Z-00034S-VP for submit@debbugs.gnu.org; Fri, 13 Jun 2025 07:59:52 -0400 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]:54565) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1uQ34V-00034A-T5 for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 07:59:49 -0400 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-453398e90e9so3353415e9.1 for <78773@debbugs.gnu.org>; Fri, 13 Jun 2025 04:59:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749815981; x=1750420781; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=+wV7vjA31XJOQLX7GGlqVgGVPlt+Y7Brtfr7zLQqRfU=; b=fZG0ZIsTZ3IoG5KmRqjf2ivXeOrFhCq2xV8r7RDR9xRB//EYkHLl6w3tjeeV2mNA7F PbCrZudbLacqyKZSplJTfPgOBn03mROlZglwqLWGdNCjqL6wzjllz76qSzXVWfIJFjID CVhDcd1U+G/BjWrZJc1kSLqghqF1EffwwG80CCQkuC3weVOLYhLi1R3eA/C0B5kFiPRD LH6Z2tooq3my3CTKrm57ySw9v9mtst4eogewT3jU13490cIWA0dIeJ/2Lh25+xTJmlTW zdlLDtCT1ku/BKHexL7WCOTxMJ9jaabhSk9C3YEopXqzim8cvpFbB0q14ZHRV91/vu13 RG3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749815981; x=1750420781; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+wV7vjA31XJOQLX7GGlqVgGVPlt+Y7Brtfr7zLQqRfU=; b=Y4Q5Qm1jTClytZyTpcsaBYvGGwf+y4h9ziltZs1zck9kBng5E2hEVxCl+sdbyT9Mw9 L2L9QzxR/xA2SK7YtJj6PpWnjSwe2rk+GAWuxAxM8EtKGyW65wqCT3Qxw77CWtC2+zBY o7Db+rSKdTEg/z+l3XdYg4p+vzqrfY64Rgf9fznWs9sdpt73G01s++0ItMHyTtemuGrK I1CoqxjZu2u96XLbRNfEyPzujJn1heUrB1CGxp19K4B0UDgyAyiDgksZp+QfwxqBqQH9 rxNzIdnTfCHVaNQw6+GGDHQ165RpKXfGE7GBtqvMC7x9BIG5hDaDYavIyxMdcA1PVyJH vZZA== X-Forwarded-Encrypted: i=1; AJvYcCXx8f5bL7lsBM5LetUQ7J6W7I8sbn5cJCwZGQhGnoMMvGrZ2Ihmsl2+MxW/PeoqIVUjhTWo9A==@debbugs.gnu.org X-Gm-Message-State: AOJu0Yyevppo1al1IW54I5BLmUu49m4NpZ+7sNHAY1WaYrDuNOdH7YEJ 2rzBodWZfNuKgQqTpVbzh35oTK00TVb55FBa740JoxnD6cAo3oFdDIAs X-Gm-Gg: ASbGncvoRu1Fgn3+i0c8cdmZYsQxU45dicHNJEVklXM1ijb0u2QLd3odN2VymA5EsEp 9gj4HHCss3QPklOFw4wHiJ+1qFBZYttqQlal3f4EDIxr5MqERqLvQuqEDMMNyYvEgtdvrTq+agn KPoknpfCpuCZzAGUp2ru+UgS1anBLIZ+Wp8Zr/LYJ3XTikQCrHslEcJCD3jt5soojjgeVLdAJfI CRyfhBYacihxmFrqbdHP5N6k0wgoOnrreWOgN18z3cf7qs/urGeaXhPVWd8JM78oJbDaGZPPhyD fwhAzOEMjk36LgM/EmBtbAV5rx/WME1/0jJDMcizCrLBWZYilvV8dBd54T5J X-Google-Smtp-Source: AGHT+IGXIPVnQDaEz7/VhoxAt4XsTlRAuaSdMeotkRq3yGWQACAic12UeZq3IkFgbNiPqLBIToAdMg== X-Received: by 2002:a05:600c:4f55:b0:43c:f629:66f4 with SMTP id 5b1f17b1804b1-453349acb61mr32003125e9.0.1749815981282; Fri, 13 Jun 2025 04:59:41 -0700 (PDT) Received: from rltb ([2a01:e0a:3f3:fb51:c51f:cac0:9d7b:73dd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4532e2618f0sm49791665e9.37.2025.06.13.04.59.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 04:59:40 -0700 (PDT) From: Robert Pluim To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86jz5fex3r.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <87bjqs9fxc.fsf@gmail.com> <86jz5fex3r.fsf@gnu.org> Date: Fri, 13 Jun 2025 13:59:40 +0200 Message-ID: <874iwjam37.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org, steven@stebalien.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) >>>>> On Fri, 13 Jun 2025 13:48:08 +0300, Eli Zaretskii said: >> I=CA=BCm wondering why we=CA=BCre setting the timeout to zero and ca= lling >> `pselect' again, which means recomputing the wait set again, if I >> haven=CA=BCt gotten lost in the twisty maze of if=CA=BCs and global = variables >> =F0=9F=98=80. We=CA=BCve got input, why can=CA=BCt we just `break'? = There might be some >> work we don=CA=BCt do, but that will get done the next time >> `wait_reading_process_output' is called. Eli> Try looking at the Git history of this code, maybe it will tell you Eli> how this code evolved, and bring up past discussions and bug repor= ts Eli> which caused us to make the code as it is today. Yes, it=CA=BCs gnarly. I think the following would be ok, and it passes the test case for Bug#20978. But I can=CA=BCt detect any difference in Steven=CA=BCs test case, so maybe= we let sleeping pselect=CA=BCs lie. Robert --=20 diff --git a/src/process.c b/src/process.c index e61ec425f7e..0119330f7fe 100644 --- a/src/process.c +++ b/src/process.c @@ -5934,6 +5934,7 @@ wait_reading_process_output (intmax_t time_limit, int= nsecs, int read_kbd, channel_start =3D process_prioritize_lower_fds ? 0 : last_read_channel + 1; =20 + bool break_now =3D false; for (channel =3D channel_start, wrapped =3D false; !wrapped || (channel < channel_start && channel <=3D max_desc); channel++) @@ -5975,8 +5976,11 @@ wait_reading_process_output (intmax_t time_limit, in= t nsecs, int read_kbd, if (nread > 0) { /* Vacuum up any leftovers without waiting. */ - if (wait_proc =3D=3D XPROCESS (proc)) - wait =3D MINIMUM; + if (!wait_proc || wait_proc =3D=3D XPROCESS (proc)) + { + wait =3D MINIMUM; + break_now =3D true; + } /* Since read_process_output can run a filter, which can call accept-process-output, don't try to read from any other processes @@ -6119,6 +6123,8 @@ wait_reading_process_output (intmax_t time_limit, int= nsecs, int read_kbd, } } } /* End for each file descriptor. */ + if (break_now) /* Don't rerun select. */ + break; } /* End while exit conditions not met. */ =20 unbind_to (count, Qnil); From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 10:45:37 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 14:45:37 +0000 Received: from localhost ([127.0.0.1]:47475 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ5ez-0006YK-1v for submit@debbugs.gnu.org; Fri, 13 Jun 2025 10:45:37 -0400 Received: from fhigh-a3-smtp.messagingengine.com ([103.168.172.154]:47049) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQ5ev-0006Y4-J2 for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 10:45:34 -0400 Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfhigh.phl.internal (Postfix) with ESMTP id 03C7A1140163; Fri, 13 Jun 2025 10:45:28 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-09.internal (MEProxy); Fri, 13 Jun 2025 10:45:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749825927; x= 1749912327; bh=wIBHhEdoi+Av/lYNU+MRQTNnMnqFJ4IqF+/gtozwJ0Q=; b=L ce6LkgNfogdac+rF5cRwR78MJhKKJdI1Lhy2pnGziLSBwTIFigK3vby+ahL44w5+ qYx6Dm60oDuzVmFvYrqjNoLsIOV4tFhlYW9t/vPXeIuYV5cU/wavUv8Bj9hZQaJA k5MwsyaRPGytO1RWgVyAYyQ2tEhjwgB6YTyIGgv2cWGQaIlZ9+jVaZd+H0Degt3+ vcIUXCL/u4N4JzYma7Tuvo2ac40C704N/AdZ6RzgcOOKTh+kXaknrW4krpALRdnZ qccCW5cgxKJaOuJgg5ezj4uF0eO/xMnvf32ofWGWHF6NiNb1ol9XbHl6kBEvcYo5 v8YKJWurOZ4b8MCq2iGhA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749825927; x=1749912327; bh=wIBHhEdoi+Av/lYNU+MRQTNnMnqFJ4IqF+/ gtozwJ0Q=; b=mEwCLQVnkjRhmcLjEUZ7itTgDhqCBkEPNNQ9zK6NsN5KOiszO8a LRDLct9lpijbMQlk6m/LuwdJJnZtYq/IsQPB1Dmlbn71DDlsN0kSAbhvDlRA60re tu8Uo7/+za3CkIWQSiHhjOy5Ir4rZsIR/71NGXVdfz28RJ3I1sAUBPJoUMgL6x9u QGYKA9sBIU2kjREFz4m2U0GBkvkJvOZfkCjYjtFTTXV09z3dUX1UFJhcwYtxFhIH 4tJPg9sn5PWomq/ScIS3RFGZ3OVBT1au9DLlIL+0A7MeWEAhpSicUWvM7zZVE/xq 4G+tC3klvLoxmgOsueLc3uy617oUjeZHY7Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddukedvfecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeejffefuddvieefgeevkeeggfelhefhffeg leetffetuedttefggeffheeufeektdenucffohhmrghinhepghhnuhdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsthgvvhgvnhes shhtvggsrghlihgvnhdrtghomhdpnhgspghrtghpthhtohepgedpmhhouggvpehsmhhtph houhhtpdhrtghpthhtohepvghlihiisehgnhhurdhorhhgpdhrtghpthhtoheprhhplhhu ihhmsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjeekjeejfeesuggvsggsuhhgshdrgh hnuhdrohhrghdprhgtphhtthhopehlrghrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 13 Jun 2025 10:45:27 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86plf8duap.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> Date: Fri, 13 Jun 2025 07:45:25 -0700 Message-ID: <87ecvnhf96.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Thu, 12 Jun 2025 12:20:30 -0700 >> >> Eli Zaretskii writes: >> >> >> I'll do that today. I have a suspicion that fast requests waiting on a >> >> single process exit early here: >> >> >> >> https://https.git.savannah.gnu.org/cgit/git/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5562 >> > >> > The condition for that block is >> > >> > if (wait_proc >> > && ! EQ (wait_proc->status, Qrun) >> > && ! connecting_status (wait_proc->status)) >> > >> > And the comment says "Don't wait for output from a non-running >> > process." Is the case here that the network connection was already >> > closed when we read from the sub-process? I thought that was not the >> > case. >> >> With my patch, it does hit that case, but on the third loop through so >> it's not exiting early. > > How come? What is wait_proc->status in that case? Note: this is WITH my original patch (to url.el), the one that makes it only wait on one process. In that case, the status is "run" the 1-2 loops through, then "exit" on the last loop (triggering this branch). From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 10:51:09 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 14:51:09 +0000 Received: from localhost ([127.0.0.1]:47549 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ5kL-0006tE-3h for submit@debbugs.gnu.org; Fri, 13 Jun 2025 10:51:09 -0400 Received: from fout-a1-smtp.messagingengine.com ([103.168.172.144]:35637) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQ5kI-0006sh-ND for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 10:51:07 -0400 Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfout.phl.internal (Postfix) with ESMTP id 1764B13806AB; Fri, 13 Jun 2025 10:51:01 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Fri, 13 Jun 2025 10:51:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-transfer-encoding:content-type:content-type :date:date:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:subject:subject:to:to; s=fm3; t=1749826261; x=1749912661; bh=88FqpJjfKy6u8R1+Z8ZkPkp4ab4l420q 2n59gxff5Tk=; b=L/37h56QFMWZoHkZ0sZYYVKKJWUsxbnIBuZEZKtJAoGqrcMD 8GfXJBWo1JxoINdlw/0sxGPBy0zFyxW3Bwwps1U4J9PnKau+RHyo7U6phdTghoJ1 UHqSeLrAcROYfv9xYKnChfHBd/cqDyscZKRVbN/RvIfNAA0r1RVR7IboK0k+NVdL 4bCFtA6PkXpO0Bl4yDhPxGRLurGO1h6DEhcr9/41MUUiN8r1BDL1YqsxpbFzTzd0 9F7vt1VdUmlvmX2pJlE9HegcrsSITl10979b+yiEEkZrTSCrSaa+zHn+UcpqBYH7 tUi8B/Q05L+FtD7O8RQ9GcsfaBA/+UA/xjQkMw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1749826261; x= 1749912661; bh=88FqpJjfKy6u8R1+Z8ZkPkp4ab4l420q2n59gxff5Tk=; b=B 7G+9YYeUyAzcvWMLwGET1Q3cbegou8N2Hr+8gsguofys4FDxhadD6m00mzJjU0D7 iodc5QiaI2/qDbAzp/vi8W9RnkCRPIhDEA04NkfWepUX33qljyRuCB0E9VjSVf+t ccWcwgYui8akc8jYeHTQwQyZrtUqUgKhZ0VsjJwm2tkLKHsSQgVr8jQtzdvGhEkc qhTkKNHwQdGDfbLRKGPuXecA9c1DHz6r2zv8Vq789PB4+jknybI+NBnIy7Wz4St8 NspApHm3F67o42zdNDqROXuXZitJfSK+LSJcGmdBL1vVTqtdFjVkIfZsqzt71gKC 20UpFnnCPhGXZodyQHQqw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddukedvfecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgfgsehtqhertddttdej necuhfhrohhmpefuthgvvhgvnhcutehllhgvnhcuoehsthgvvhgvnhesshhtvggsrghlih gvnhdrtghomheqnecuggftrfgrthhtvghrnhepudevlefgtdekheekgffhtdegheduhfeg tdfgffefteetgfetkeeivddthefhkeetnecuffhomhgrihhnpehgnhhurdhorhhgnecuve hluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhtvghvvghn sehsthgvsggrlhhivghnrdgtohhmpdhnsggprhgtphhtthhopeegpdhmohguvgepshhmth hpohhuthdprhgtphhtthhopehrphhluhhimhesghhmrghilhdrtghomhdprhgtphhtthho pegvlhhiiiesghhnuhdrohhrghdprhgtphhtthhopeejkeejjeefseguvggssghughhsrd hgnhhurdhorhhgpdhrtghpthhtoheplhgrrhhsihesghhnuhhsrdhorhhg X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 13 Jun 2025 10:51:00 -0400 (EDT) From: Steven Allen To: Robert Pluim , Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <874iwjam37.fsf@gmail.com> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <87bjqs9fxc.fsf@gmail.com> <86jz5fex3r.fsf@gnu.org> <874iwjam37.fsf@gmail.com> Date: Fri, 13 Jun 2025 07:50:59 -0700 Message-ID: <87bjqrhezw.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Robert Pluim writes: >>>>>> On Fri, 13 Jun 2025 13:48:08 +0300, Eli Zaretskii sai= d: > > >> I=CA=BCm wondering why we=CA=BCre setting the timeout to zero and = calling > >> `pselect' again, which means recomputing the wait set again, if I > >> haven=CA=BCt gotten lost in the twisty maze of if=CA=BCs and globa= l variables > >> =F0=9F=98=80. We=CA=BCve got input, why can=CA=BCt we just `break'= ? There might be some > >> work we don=CA=BCt do, but that will get done the next time > >> `wait_reading_process_output' is called. > > Eli> Try looking at the Git history of this code, maybe it will tell = you > Eli> how this code evolved, and bring up past discussions and bug rep= orts > Eli> which caused us to make the code as it is today. > > Yes, it=CA=BCs gnarly. I think the following would be ok, and it passes t= he > test case for Bug#20978. > > But I can=CA=BCt detect any difference in Steven=CA=BCs test case, so may= be we > let sleeping pselect=CA=BCs lie. I think this patch will cause us to the final drain loop when waiting on a single connection. It's probably fine, but I assume this drain loop existed for a reason? https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=3Db= ec823b107ef7d3b51b8e430ccab82c81bd63d24#n5525 > Robert > --=20 > diff --git a/src/process.c b/src/process.c > index e61ec425f7e..0119330f7fe 100644 > --- a/src/process.c > +++ b/src/process.c > @@ -5934,6 +5934,7 @@ wait_reading_process_output (intmax_t time_limit, i= nt nsecs, int read_kbd, > channel_start > =3D process_prioritize_lower_fds ? 0 : last_read_channel + 1; >=20=20 > + bool break_now =3D false; > for (channel =3D channel_start, wrapped =3D false; > !wrapped || (channel < channel_start && channel <=3D max_desc); > channel++) > @@ -5975,8 +5976,11 @@ wait_reading_process_output (intmax_t time_limit, = int nsecs, int read_kbd, > if (nread > 0) > { > /* Vacuum up any leftovers without waiting. */ > - if (wait_proc =3D=3D XPROCESS (proc)) > - wait =3D MINIMUM; > + if (!wait_proc || wait_proc =3D=3D XPROCESS (proc)) > + { > + wait =3D MINIMUM; > + break_now =3D true; > + } > /* Since read_process_output can run a filter, > which can call accept-process-output, > don't try to read from any other processes > @@ -6119,6 +6123,8 @@ wait_reading_process_output (intmax_t time_limit, i= nt nsecs, int read_kbd, > } > } > } /* End for each file descriptor. */ > + if (break_now) /* Don't rerun select. */ > + break; > } /* End while exit conditions not met. */ >=20=20 > unbind_to (count, Qnil); From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 10:59:39 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 14:59:39 +0000 Received: from localhost ([127.0.0.1]:47779 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ5sZ-0007Vk-36 for submit@debbugs.gnu.org; Fri, 13 Jun 2025 10:59:39 -0400 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]:59857) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1uQ5sW-0007VT-RK for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 10:59:37 -0400 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-450ce671a08so13960025e9.3 for <78773@debbugs.gnu.org>; Fri, 13 Jun 2025 07:59:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749826771; x=1750431571; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=WlR6410Qbghbsx7GOU4ekNOt6WMmI3dB6AaCg9Mj9Bk=; b=J/+n5HOn3QTypKlYM5oGVmdxE6uMQMhKtrnG8WkHIcbeoRy50VprM/AeuQiqAZAV78 Q+YUS9j3K36sVbB1NDZqxG4Ry92YbPeHeCQdoFhpcBVbxizNNZAzAwLA6CDZDWMivUei qdEnbHyxObSh4/dft5hOvV3Yw8Mdjx4q33YVamt1lEvV7+XDg9JLlUh8pYfhtFedsY3N g8VOpME/E6MNbW08di6eicBl4e1In06ZzubCuQKJdM1ZSQzcacFEvsUxFzldMYWzQb2x LnsChBOxBvyI11bym1aVPWAvgdsGdY8Cgj+7tbwK6HyF41DapAsMSUQciikaOiEffz1B yOpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749826771; x=1750431571; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WlR6410Qbghbsx7GOU4ekNOt6WMmI3dB6AaCg9Mj9Bk=; b=MizwbR+hdtC+Gu0cSjTrWcoZdpkuDVy5uZClCrSBnifP6PZ1q6FOMUUDTH0KAZjo9R O8Fs/9enLMdGvs6dAjCgoif2fRu373vQn2Sgca3EB5s1t4CzPloQHPNSGOSgXwdwcc4m /xeVGnWv/cot5ZMOi7JbSaweaSWoH3l9gBDKP6bHAMiL2xE3SYkUXp2+TcnlMATDA/vD iiLlKafX9UM5kRzPTTpfkRD3UqK7O8FsDuFdY4HCRweOKW1AWCIvnz/8nR8M5kqJL8MG H+1tzBxxBnEnwhcZTMoqT/d989aI/NYmnLTcQoh6s2FclkUGIQG3e8B/dUWJvYEKrtFH AqNA== X-Forwarded-Encrypted: i=1; AJvYcCXYRohkzW0Ss03/VM7bgBQ5aEnKn18gXSki0URMPMwhUAP+jj3pl1wRRZKVRTXBrj8T7OAqKA==@debbugs.gnu.org X-Gm-Message-State: AOJu0YwWGZk54olLLBseF3mMusGzObMuTDlsWfT+noysNYkAEgHFMITx C8qcxSsquySGyZm7ixpVmZz0iEQbHBjQKu/GB2SgftR+nQc3OJHfdpgeIN2kSwTs X-Gm-Gg: ASbGncsu06rdObMfZlOSVP+L9EC0js8Mvl7eUfTSC+pbVGZTj6Z8h5CS/aGh0XXg24j k2/hiMugwtr7w9xpujQHRIPzOJynPkSMkXiGHmh03f7p03Ln+pbbq4HOdnOuZt+l9DRVy13NSqa nY7qKkUzNqI2xIiylO/F4oz+vWXZqxXIAec6lYbico5W05OmpYUIdkDFKu9M6qtadX6YTqgTkAb Mqu8JL4B54J5ev2xmMyhEXmUEYZr184ecC42KNbrtZGJ/dp3MzBumVgBYCKCLHBZ22H4Pwr9vsd M+NalfqgN9ho9eni3RX0ApzpGBa/TYAw8zzTQ0y9PE51DqxeVg== X-Google-Smtp-Source: AGHT+IEUPylrGlUjs+p/QAywlAtzx0NMWJeoLDOqPpYaEX+91lINIappnT+Qpcpqsd1u1GjKXOZMNA== X-Received: by 2002:a05:600c:4f09:b0:43d:8ea:8d7a with SMTP id 5b1f17b1804b1-45334af5c89mr30044275e9.28.1749826770398; Fri, 13 Jun 2025 07:59:30 -0700 (PDT) Received: from rltb ([2a01:e0a:3f3:fb51:df39:e830:abf3:ba4f]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a568a7fa9dsm2584708f8f.44.2025.06.13.07.59.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 07:59:29 -0700 (PDT) From: Robert Pluim To: Steven Allen Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <87bjqrhezw.fsf@stebalien.com> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <87bjqs9fxc.fsf@gmail.com> <86jz5fex3r.fsf@gnu.org> <874iwjam37.fsf@gmail.com> <87bjqrhezw.fsf@stebalien.com> Date: Fri, 13 Jun 2025 16:59:29 +0200 Message-ID: <87zfeb8z72.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, Eli Zaretskii , larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) >>>>> On Fri, 13 Jun 2025 07:50:59 -0700, Steven Allen said: Steven> I think this patch will cause us to the final drain loop when w= aiting on Steven> a single connection. It's probably fine, but I assume this drai= n loop Steven> existed for a reason? Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/pro= cess.c?h=3Dbec823b107ef7d3b51b8e430ccab82c81bd63d24#n5525 It will, but we=CA=BCll already have read from the process at line 5972 or thereabouts. Since I can=CA=BCt definitively say it won=CA=BCt break anything, we should= go with the minimal patch, ie yours. Robert --=20 From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 11:14:51 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 15:14:51 +0000 Received: from localhost ([127.0.0.1]:48198 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ67H-0000LZ-GH for submit@debbugs.gnu.org; Fri, 13 Jun 2025 11:14:51 -0400 Received: from fhigh-a4-smtp.messagingengine.com ([103.168.172.155]:48989) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQ67F-0000LI-0n for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 11:14:49 -0400 Received: from phl-compute-10.internal (phl-compute-10.phl.internal [10.202.2.50]) by mailfhigh.phl.internal (Postfix) with ESMTP id 8973911400F3; Fri, 13 Jun 2025 11:14:43 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-10.internal (MEProxy); Fri, 13 Jun 2025 11:14:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-transfer-encoding:content-type:content-type :date:date:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:subject:subject:to:to; s=fm3; t=1749827683; x=1749914083; bh=Gee0h8IYCkvUEA535JMAlirqEcK0p6sY fv/NmP0f5L0=; b=ka2Nn3D+ZnquIaLtde+tn1D0oHb/j1xcKUSLKG0tcO2Ltxxr nFRUEhdVYQEWXLl8uYqBqmzPykqvEnYAZohc3/ra9RdclVtlfowGVxLodWB27gCj /YRkiB3kNJRdPNNmYmapNXuf/diWdH0SQwLQVc0BpTtYmRVtl4KkFzMPXhnR3aax n5BidPRFLQRUe4MAYSj4szEbHOXBJzyByu0CMQm+YrojLLmmIZgXuLbYO6zew9r6 +R5aw9uKsRDlFuyr4LV5j78HFw/yGZq4qVg8z+R5q+QuKZV3SazF3DIHd6jx9jYa ZzbTrGSjBTZaQ5mDcNT3qh/VUo3uFwf7MknGug== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1749827683; x= 1749914083; bh=Gee0h8IYCkvUEA535JMAlirqEcK0p6sYfv/NmP0f5L0=; b=M nA0ddv8yxG8WCwojl79T8xvdhcr8JMOJzBMoIGzBtR7Nw+gRETcyTTeSAaWQF0Fb 96CTTv8sgWr+dopgxul85WO/mUar5uSRs07TmaEo6qTdkWN5SQ2ldkegMtZZS4IL Apms/935lhBaub/oZ2DrOUzrE7dOHGTTOkaK1AsHe3eOZdXdtyU/9fOt7PulmKsu 0qWdPrDMswHkrO2bLznRGrPuHEFHxS1Qp05RXzwbXGE3rH935E4BljzeoUuyL2Xd /RAfifhncbsP2Di+0tWv4o+eWwvF5Mo+BQLPNmBKv+8GrbOTsS8cLu2HyrJRsUAg MAH7SdA560q0I0mSHUl1Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddukedvkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgfgsehtqhertddttdej necuhfhrohhmpefuthgvvhgvnhcutehllhgvnhcuoehsthgvvhgvnhesshhtvggsrghlih gvnhdrtghomheqnecuggftrfgrthhtvghrnhepudevlefgtdekheekgffhtdegheduhfeg tdfgffefteetgfetkeeivddthefhkeetnecuffhomhgrihhnpehgnhhurdhorhhgnecuve hluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhtvghvvghn sehsthgvsggrlhhivghnrdgtohhmpdhnsggprhgtphhtthhopeegpdhmohguvgepshhmth hpohhuthdprhgtphhtthhopehrphhluhhimhesghhmrghilhdrtghomhdprhgtphhtthho pegvlhhiiiesghhnuhdrohhrghdprhgtphhtthhopeejkeejjeefseguvggssghughhsrd hgnhhurdhorhhgpdhrtghpthhtoheplhgrrhhsihesghhnuhhsrdhorhhg X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 13 Jun 2025 11:14:42 -0400 (EDT) From: Steven Allen To: Robert Pluim Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <87zfeb8z72.fsf@gmail.com> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <87bjqs9fxc.fsf@gmail.com> <86jz5fex3r.fsf@gnu.org> <874iwjam37.fsf@gmail.com> <87bjqrhezw.fsf@stebalien.com> <87zfeb8z72.fsf@gmail.com> Date: Fri, 13 Jun 2025 08:14:41 -0700 Message-ID: <875xgzhdwe.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, Eli Zaretskii , larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Robert Pluim writes: >>>>>> On Fri, 13 Jun 2025 07:50:59 -0700, Steven Allen said: > > Steven> I think this patch will cause us to the final drain loop when= waiting on > Steven> a single connection. It's probably fine, but I assume this dr= ain loop > Steven> existed for a reason? > > Steven> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/p= rocess.c?h=3Dbec823b107ef7d3b51b8e430ccab82c81bd63d24#n5525 > > It will, but we=CA=BCll already have read from the process at line 5972 or > thereabouts. > > Since I can=CA=BCt definitively say it won=CA=BCt break anything, we shou= ld go > with the minimal patch, ie yours. Your approach is correct, I think. It just may be less performant as the user will now need to accept output twice when we could have just "finished the job" the first time around. From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 11:31:38 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 15:31:39 +0000 Received: from localhost ([127.0.0.1]:48433 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ6NW-0005k8-5i for submit@debbugs.gnu.org; Fri, 13 Jun 2025 11:31:38 -0400 Received: from mail-wr1-x42f.google.com ([2a00:1450:4864:20::42f]:58556) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1uQ6NT-0005io-I4 for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 11:31:36 -0400 Received: by mail-wr1-x42f.google.com with SMTP id ffacd0b85a97d-3a53359dea5so1471776f8f.0 for <78773@debbugs.gnu.org>; Fri, 13 Jun 2025 08:31:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749828689; x=1750433489; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=PKgEUeEfrG16uV/5FaHTW4h9GLSKNe0wiWl7nUSMkJE=; b=GrfkmnmmL9oRobAe5kRivT5Tdo9AzXt9ZTWGCENRWD9gBd3mynkRnXgj0AgpaNLfa+ HxirhAIM85TuJQ+gEDsrOuq1kN8MkBORaxnDgpl9jnWbsNVztU+YeHwdFqqsZnX+Df9K wPRuEXDPAnJp4lMfw5dKqhTQXjSOCKxneBOLIgEyAqUtElP/WI766Y4g+3k+j96T7Gfw MDZhb9ZaPBYqAz+AZW7WzflWPczcwZFX8QMcTkrNuAyfk9hN/HfBn1nU/95W17tGTM+s 8IEu0rM2doF+UVAT61GW3ZTcSBAWb+DogFC+yGe0z7cM7cZWV+iQ4psmenxN37/wMdIT 9ioA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749828689; x=1750433489; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PKgEUeEfrG16uV/5FaHTW4h9GLSKNe0wiWl7nUSMkJE=; b=pjuJcrqLiA3+r5kqTcDRba3wvh0uHnd5tMb5b3faZfdl3Z3IbKnYY9uV0MHr4Hs+lr t7nYmk6w9Z7FSs/vXfNiFlb6xsLXm7n75IAKvPXHF63HenfAJSst+EXZNT0i9VZLTQ+4 yiJ2AyDAPpBkTeqNBi4LhEV6WsbQZd/OULeifaZf8v7J9xzpGvsYUnxZInzZRqbqgA1S SIh8XejcyBzkIScsxw2F8zK/pcaHhS2zc9UqmADHACq3vkBA/Ri2/MYtFgt7rpfbQNhZ iX9ujsYW/tMi5nR6xPfNbHIudkssuLOvuNY9Xz4639FygMafhKJgq3hQRWXalOskk2FY lciQ== X-Forwarded-Encrypted: i=1; AJvYcCUyXx960/f0l/jHluBCrVybBvvnnD0QUCfln77+rhFrF7oDgeLwOkk3ouqlgjNrhqhvnj+2lg==@debbugs.gnu.org X-Gm-Message-State: AOJu0YzjmJjlfcRIUBeYt688DWtr7mxOjRWeYkHBDdHhzLQAWPoVptBt mMtcC5HmG5HEwJ1mNmkLcmpjq6C3zO3r94mQS2xIZ//dNGIvPmZ0r6z8 X-Gm-Gg: ASbGncvqDmz4svPUPzc5i2AbazgzX1Yl8863ZPN69EaWPa3T9OkXqzTUrQQCMy7Kt44 Be9a1dr6rCp6PrJw4fd/418PLPgTXKII93gvLi+v+eC656e889hbCcm6xseUhw9kdFN8IThgfZR YBXCUdXQwPqc/MB+3jmv3M1EBGNWhVKcWIf8m0/GSISP2FQw4KA9WJfnQ/tCAfBC4LT18gQGGV9 tUKViLVJcSd4wFc/+dotyZkIHhFqWtJy8w6FpnZtlWCNMNfj6NYbzW3OpFk/wIK1O4VKEHm+IRz +Zbp3ObXMTYCptzuHNqZP7lTxoY5rKvC2IWBUhwe0amYl7U0hA== X-Google-Smtp-Source: AGHT+IEthoYFNSJDBRjC2Gy0D3m3ZZlYJb4KUFe6N/WcqReo4fThG74WDQpggOw+z2uND7J77b+U+Q== X-Received: by 2002:a05:6000:41dd:b0:3a4:e609:dc63 with SMTP id ffacd0b85a97d-3a57237190fmr206018f8f.20.1749828689071; Fri, 13 Jun 2025 08:31:29 -0700 (PDT) Received: from rltb ([2a01:e0a:3f3:fb51:df39:e830:abf3:ba4f]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a568a72cd3sm2716064f8f.32.2025.06.13.08.31.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 08:31:28 -0700 (PDT) From: Robert Pluim To: Steven Allen Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <875xgzhdwe.fsf@stebalien.com> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <87bjqs9fxc.fsf@gmail.com> <86jz5fex3r.fsf@gnu.org> <874iwjam37.fsf@gmail.com> <87bjqrhezw.fsf@stebalien.com> <87zfeb8z72.fsf@gmail.com> <875xgzhdwe.fsf@stebalien.com> Date: Fri, 13 Jun 2025 17:31:27 +0200 Message-ID: <87v7oz8xps.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, Eli Zaretskii , larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) >>>>> On Fri, 13 Jun 2025 08:14:41 -0700, Steven Allen said: >>=20 >> Since I can=CA=BCt definitively say it won=CA=BCt break anything, we= should go >> with the minimal patch, ie yours. Steven> Your approach is correct, I think. It just may be less performa= nt as the Steven> user will now need to accept output twice when we could have ju= st Steven> "finished the job" the first time around. We=CA=BCll let Eli decide =F0=9F=98=BA Robert --=20 From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 11:48:34 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 15:48:35 +0000 Received: from localhost ([127.0.0.1]:48519 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ6dt-0008GP-Iy for submit@debbugs.gnu.org; Fri, 13 Jun 2025 11:48:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59350) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQ6dq-0008G8-3M for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 11:48:31 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQ6dk-0007B8-CV; Fri, 13 Jun 2025 11:48:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=Pd0/u3oVFDn/dRF1HtT3OU8xGNvSO2YFxlVn8Sm6eNE=; b=bJDcWKVFqc4B qOFg9qD3jIBoyId+H9DzdGzEWoopoi1oL4Sgm/ytxXdXO7fW3IbNkMPjQTK/FzAmsdYO5M4FuD0bU 8h/7WFOIMrpAck/aBVmfKeqSf+XTeMf0Mus3o50j0+5GjJ35u0/CdHLVTd1DZYeOfJ5Bs9jJxACqn GN4eaJHp6AFpJ9NGHCJdZRGCQd56LDC7tAbmSP93li2ICULtQ5pbm0o0r6TVAtXBsjbqn8tPvkCF0 4gszft5jCH0QWvt08jz5zUhsjlPUKb6fgWkcSZWnDkC0QGLURGWJOU5g8WOygguiJm7ImU3s43O0Q AOAmzCtvaMaMc+nyEnM6RQ==; Date: Fri, 13 Jun 2025 18:47:54 +0300 Message-Id: <8634c3ej85.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87ecvnhf96.fsf@stebalien.com> (message from Steven Allen on Fri, 13 Jun 2025 07:45:25 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Fri, 13 Jun 2025 07:45:25 -0700 > > > Eli Zaretskii writes: > > >> From: Steven Allen > >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > >> Date: Thu, 12 Jun 2025 12:20:30 -0700 > >> > >> Eli Zaretskii writes: > >> > >> >> I'll do that today. I have a suspicion that fast requests waiting on a > >> >> single process exit early here: > >> >> > >> >> https://https.git.savannah.gnu.org/cgit/git/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5562 > >> > > >> > The condition for that block is > >> > > >> > if (wait_proc > >> > && ! EQ (wait_proc->status, Qrun) > >> > && ! connecting_status (wait_proc->status)) > >> > > >> > And the comment says "Don't wait for output from a non-running > >> > process." Is the case here that the network connection was already > >> > closed when we read from the sub-process? I thought that was not the > >> > case. > >> > >> With my patch, it does hit that case, but on the third loop through so > >> it's not exiting early. > > > > How come? What is wait_proc->status in that case? > > Note: this is WITH my original patch (to url.el), the one that makes it > only wait on one process. > > In that case, the status is "run" the 1-2 loops through, then "exit" on > the last loop (triggering this branch). So your patch is only useful when the sub-process exits soon after producing its single chunk of output, is that right? From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 13 13:45:18 2025 Received: (at 78773) by debbugs.gnu.org; 13 Jun 2025 17:45:18 +0000 Received: from localhost ([127.0.0.1]:49291 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQ8Ss-0000nQ-0o for submit@debbugs.gnu.org; Fri, 13 Jun 2025 13:45:18 -0400 Received: from fhigh-a5-smtp.messagingengine.com ([103.168.172.156]:42101) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQ8So-0000kq-5v for 78773@debbugs.gnu.org; Fri, 13 Jun 2025 13:45:16 -0400 Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfhigh.phl.internal (Postfix) with ESMTP id D311F11400FD; Fri, 13 Jun 2025 13:45:08 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Fri, 13 Jun 2025 13:45:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749836708; x= 1749923108; bh=qhmmOxugbg4tq7C680s7fnwgB2CzS2hAM695HAhBVRA=; b=l QxCdJvOFlKHYrUzY25NJgk98ggW4L+Ohr7e7offfeepDVwBhArOz65emNGEYLxfG x/d5jziuyzdNKWCcOPPVcOEOIXV0tlVaJ29UjN1DTKRHk1Wx5x4f7Gsu8WZID01x bn3KGYNjSPwshA/JmoHUvs7ooG1s2rNuphwxwZUeJH9UOxHenEKqvL4qNk0C+nDA yf2KZj935qo8N2DTVbevwINyUO0TWqjGtNjt4o2kzQTBQjOTBV22p01gnHJh/2nu elNMOSRddCVgJRpvFH4TSNqhwgE+/RKtgL1SrvOUgUyesu7b0dOJTA91SHLSS+JJ 0yMt8OOSKRYl8SWsg+sgg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749836708; x=1749923108; bh=qhmmOxugbg4tq7C680s7fnwgB2CzS2hAM69 5HAhBVRA=; b=nqQLxhEjCvxlfJSheJJAnwf0cSDUKLHH8ZKbjITie7KVuw5p9wP EgyNpOVkm2JeIozMFATeNPZzAXfw+KlixbYHOascX5RNemctT2AywNbKwZxkMztr fA5G8VjNsbajcrkN1u9wMSwLYlkGPkq2uXgr7dGaqp/iwR5SzBxHv3eaujFJRoZ8 kMJvz6a+R1U8eB/iPs0E9AjzD+sr+OdFoxaA1FfrHlRUScCg3zaBUlIw0SZZH373 imZ0X+HAIRyiGNn3qlmKw5GV41ezbvHC+egFAF6igPBLQPBh5aUJ8Gx+ZKBCCTtO mtaVg7MG3PrWXQojROB/Nf4a/6d60nGhWJA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddukeehkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeejffefuddvieefgeevkeeggfelhefhffeg leetffetuedttefggeffheeufeektdenucffohhmrghinhepghhnuhdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsthgvvhgvnhes shhtvggsrghlihgvnhdrtghomhdpnhgspghrtghpthhtohepgedpmhhouggvpehsmhhtph houhhtpdhrtghpthhtohepvghlihiisehgnhhurdhorhhgpdhrtghpthhtoheprhhplhhu ihhmsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjeekjeejfeesuggvsggsuhhgshdrgh hnuhdrohhrghdprhgtphhtthhopehlrghrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 13 Jun 2025 13:45:08 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <8634c3ej85.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> Date: Fri, 13 Jun 2025 10:45:06 -0700 Message-ID: <8734c3h6xp.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Fri, 13 Jun 2025 07:45:25 -0700 >> >> >> Eli Zaretskii writes: >> >> >> From: Steven Allen >> >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> >> Date: Thu, 12 Jun 2025 12:20:30 -0700 >> >> >> >> Eli Zaretskii writes: >> >> >> >> >> I'll do that today. I have a suspicion that fast requests waiting on a >> >> >> single process exit early here: >> >> >> >> >> >> https://https.git.savannah.gnu.org/cgit/git/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5562 >> >> > >> >> > The condition for that block is >> >> > >> >> > if (wait_proc >> >> > && ! EQ (wait_proc->status, Qrun) >> >> > && ! connecting_status (wait_proc->status)) >> >> > >> >> > And the comment says "Don't wait for output from a non-running >> >> > process." Is the case here that the network connection was already >> >> > closed when we read from the sub-process? I thought that was not the >> >> > case. >> >> >> >> With my patch, it does hit that case, but on the third loop through so >> >> it's not exiting early. >> > >> > How come? What is wait_proc->status in that case? >> >> Note: this is WITH my original patch (to url.el), the one that makes it >> only wait on one process. >> >> In that case, the status is "run" the 1-2 loops through, then "exit" on >> the last loop (triggering this branch). > > So your patch is only useful when the sub-process exits soon after > producing its single chunk of output, is that right? Not quite. That condition will trigger on the last call to `accept-process-output` when a PROCESS is specified, regardless of how short the total response was. But there's a more to the story; let's start at the top. First, the actual bug: When called with no PROCESS, `accept-process-output' would wait an additional 10ms (READ_OUTPUT_DELAY_INCREMENT) after successfully reading data from some (any) process. `url-retrieve-synchronously' was slow for localhost calls because 10ms is a long time for fast requests. This bug would add 10ms of latency to every request. My SECOND patch (0772d96d, the one to process.c) fixes this underlying bug by setting the timeout to 0 in this case: when we manage to read data from some process and aren't waiting on any specific process. My FIRST patch (4529f63f, the one to `url-retrieve-synchronously') worked around this bug by specifying the process to wait on. With that patch I happened to hit the branch in question because the requests were fast enough to complete in one call to `accept-process-output'. However, the FIRST patch would still have worked around the underlying bug (the issue fixed in the SECOND patch) because the FIRST patch specifies the process to wait on and the underlying bug is only triggered when the process is NOT specified. Next steps: 1. The SECOND patch fixes the actual bug and should be installed to fix the underlying bug (assuming I correctly diagnosed the problem/solution, but I'm pretty sure I did). 2. Personally, I would install the FIRST patch as well as it avoids returning to `url-retrieve-synchronously' until we've actually read relevant data. This is how `url-retrieve-synchronously' behaved before bug#49897 changed it to work around then undiagnosed bugs in `accept-process-output'. However, that's very much a maintainer decision because it might cause the bug in `accept-process-output' that motivated this change (bug#49861) to rear its head again. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 07:38:46 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 11:38:46 +0000 Received: from localhost ([127.0.0.1]:36378 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQPDi-000129-7l for submit@debbugs.gnu.org; Sat, 14 Jun 2025 07:38:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34928) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQPDf-00011W-8D for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 07:38:44 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQPDZ-0004to-Dz; Sat, 14 Jun 2025 07:38:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=CjLDmf74OXMTKBToF+XlnJebrhHw/06h8BUtAZhciZw=; b=Bsbre8+JG6NR 43qylcXhPGnxzQIaQtmIHP8AItJT7ucTnEzkJefTt1Lq1Yhv5/io3YaDfZsX0XDj4J1SH9R8/EtAc sCNNQ4dHa0iY4tON/qUBj8d5XsMWyLhaVGsNeLY8nDUavt6a2buxRmuhQjSbpFJxfDnO/vNNFqKAp GqTv2ZpcDJgf1shFTB2pwIXiJoGBW19Yckp9KyDP3G8eI1kDV4foAOOQWKCzU9EZs45L/RmJM7b9E m1HO4GW/Bwm1n3YasSBI76vF2DdZCY8cEEIO4oV3g+UiVZ1kiDaQvY2sElBjNQQmGachtuMrVgAuB d1fLcUV8xAM5zkLdy77PCA==; Date: Sat, 14 Jun 2025 14:38:35 +0300 Message-Id: <86ikkysgck.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87o6usadg6.fsf@stebalien.com> (message from Steven Allen on Thu, 12 Jun 2025 13:54:01 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Thu, 12 Jun 2025 13:54:01 -0700 > > Ok, I think I've figured it out. We loop twice and wait on a timeout > (READ_OUTPUT_DELAY_INCREMENT) the second loop because we're already done > reading. We wait the second time because there could be more input, so I don't see how this could be wrong in general. I also don't quite see the "gotcha!" in your analysis below: > The first time through, we hit the following, recording that we've > gotten some output: > > https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5974 OK. > We hit the following, setting our timeout to READ_OUTPUT_DELAY_INCREMENT: > > https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5679 This is to _shorten_ the wait timeout: /* If we've got some output and haven't limited our timeout with adaptive read buffering, limit it. */ if (got_some_output > 0 && !process_skipped && (timeout.tv_sec || timeout.tv_nsec > READ_OUTPUT_DELAY_INCREMENT)) timeout = make_timespec (0, READ_OUTPUT_DELAY_INCREMENT); So I don't quite see how this could be a problem here. Am I missing something? > Then we hit the select call and wait the full timeout (nothing to read) > and we finally exit here (because our timeout has passed): > > https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5832 The "full timeout" being the value of READ_OUTPUT_DELAY_INCREMENT, is that right? Then why is it not TRT, since it _shortens_ the wait? > I think what we need to do is modify: > > https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5978 > > from > > if (wait_proc == XPROCESS (proc)) > wait = MINIMUM; > > to > > if (!wait_proc || wait_proc == XPROCESS (proc)) > wait = MINIMUM; > > So that we set the timeout to 0 here: > > https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5439 > > And exit early. Why? That code was there for a reason. What if we are waiting for other processes as well? What did I miss in the above description that explains why the current code is wrong? From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 07:45:13 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 11:45:13 +0000 Received: from localhost ([127.0.0.1]:36498 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQPJw-0001Vo-OU for submit@debbugs.gnu.org; Sat, 14 Jun 2025 07:45:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35070) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQPJu-0001UU-7Q for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 07:45:11 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQPJo-0005cL-PF; Sat, 14 Jun 2025 07:45:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=6e9v6rP8UyG0joFroRc/31bUDwtcWivqZUSmxlMVw8k=; b=RjlNq+PRKNqG/6E60axD 9tHLcRVWrmvL9JL8B3DBTH4tTZoaDCqKC9Dj/DXZN4Yxj4+4eM+/UrKtKmdiWmv8tLt4Xx0rqsAg6 leyH9k6ROPuTwZKSXzEIj/CwVt2U81GQ8b/QrddfZ6hWq45iEYlNaoUKbpdeatjXpUp1QpUJPo5PK yCvJAwMEX6iFnJP1FAEJmfpVjedCLSbJUmdZmtFx7Gu2HdWGrwkoy/hjiFiKnBiVCXgP/rMrHj8EP MW89UpS7cr5jJl0KelhrI7NV2dypHNCJLvWJrhJoCQ7JKv7YPQ1s3cHi+35Cd25LpSsm8yuBoLIAp z9bvHgf3VGOhUw==; Date: Sat, 14 Jun 2025 14:45:02 +0300 Message-Id: <86h60isg1t.fsf@gnu.org> From: Eli Zaretskii To: Robert Pluim In-Reply-To: <874iwjam37.fsf@gmail.com> (message from Robert Pluim on Fri, 13 Jun 2025 13:59:40 +0200) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <87bjqs9fxc.fsf@gmail.com> <86jz5fex3r.fsf@gnu.org> <874iwjam37.fsf@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, larsi@gnus.org, steven@stebalien.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Robert Pluim > Cc: steven@stebalien.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Fri, 13 Jun 2025 13:59:40 +0200 > > >>>>> On Fri, 13 Jun 2025 13:48:08 +0300, Eli Zaretskii said: > > >> Iʼm wondering why weʼre setting the timeout to zero and calling > >> `pselect' again, which means recomputing the wait set again, if I > >> havenʼt gotten lost in the twisty maze of ifʼs and global variables > >> 😀. Weʼve got input, why canʼt we just `break'? There might be some > >> work we donʼt do, but that will get done the next time > >> `wait_reading_process_output' is called. > > Eli> Try looking at the Git history of this code, maybe it will tell you > Eli> how this code evolved, and bring up past discussions and bug reports > Eli> which caused us to make the code as it is today. > > Yes, itʼs gnarly. I think the following would be ok, and it passes the > test case for Bug#20978. Please explain why you think this change is okay, by talking us through what will happen with it in the case in pint. I generally don't like to touch this code with a 3-mile pole, and if this means the current use case will run slowly, so be it. So I need a clear evidence that (a) the current code is wrong (not just sub-optimal, but wrong), and (b) that it can never affect other cases handled by wait_reading_process_output, of which there are a legion. > But I canʼt detect any difference in Stevenʼs test case, so maybe we > let sleeping pselectʼs lie. You are saying that Stevenʼs patch also doesn't break bug#20978? If so, why did you post a different one? And I don't yet understand why short-circuiting like Steven suggests is TRT. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 07:52:55 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 11:52:55 +0000 Received: from localhost ([127.0.0.1]:36650 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQPRO-0002AE-PG for submit@debbugs.gnu.org; Sat, 14 Jun 2025 07:52:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38032) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQPRI-00028o-Tw for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 07:52:51 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQPRD-0006Jf-B9; Sat, 14 Jun 2025 07:52:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=gIGIfaQpjlXWrFPInCBLf6YRXYWV65xs0lFTgb5R1ug=; b=oEW7ERwzezsi S/Cu1rgl8P+zQfA3HYr3vP9ZZ2XjfuxoPge3JwEwXijCh0d8UTSbY4Me6nPyfdSrZeyQGdQpQ3Kww B6HxodThvDz0aYlY1HVeyxsB7VF9Tn7P8NdTaiGWnkkaTpHLjtjQ0RxN5oqA8+iOh5BvZfsw8dVvx 0NPS0WKeIg0MmIQh2HwnrcGdDSI8BEGMdvSk8YcQyFTv6PAoe7TdGgKhAs9cEXLBYvA8JpiFFomhS FJaDbWlZl3Oi4ghEEHC+XR+58eJjHMtrvkz5v1tm3ABSzULrknqJfDPrQ6lkWdPCOl09BW/V9yHZg kCCgqOsdiWp8hnx3LHgBYw==; Date: Sat, 14 Jun 2025 14:52:41 +0300 Message-Id: <86frg2sfp2.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <8734c3h6xp.fsf@stebalien.com> (message from Steven Allen on Fri, 13 Jun 2025 10:45:06 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> <8734c3h6xp.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Fri, 13 Jun 2025 10:45:06 -0700 > > First, the actual bug: When called with no PROCESS, > `accept-process-output' would wait an additional 10ms > (READ_OUTPUT_DELAY_INCREMENT) after successfully reading data from some > (any) process. `url-retrieve-synchronously' was slow for localhost calls > because 10ms is a long time for fast requests. This bug would add 10ms > of latency to every request. But it's an unneeded delay only if there's no more output from the subprocess during those 10 ms, isn't that so? And we don't really know if all the process output was read, do we? > My SECOND patch (0772d96d, the one to process.c) fixes this > underlying bug by setting the timeout to 0 in this case: when we manage > to read data from some process and aren't waiting on any specific > process. But what if we are also waiting for output from other processes? Your patch will slow those down, won't it? In you case, you only care about a single process, but that's not so in general. > Next steps: > > 1. The SECOND patch fixes the actual bug and should be installed to fix > the underlying bug (assuming I correctly diagnosed the problem/solution, > but I'm pretty sure I did). I disagree for now. I need to understand better why you think it's TRT in the general case. > 2. Personally, I would install the FIRST patch as well as it avoids > returning to `url-retrieve-synchronously' until we've actually read > relevant data. What if url-retrieve was called for more than a single connection? Won't your FIRST patch cause a regression in that case? > This is how `url-retrieve-synchronously' behaved before > bug#49897 changed it to work around then undiagnosed bugs in > `accept-process-output'. Exactly. I'm unwilling to risk reintroducing bugs we already fixed. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 12:15:15 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 16:15:15 +0000 Received: from localhost ([127.0.0.1]:41875 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQTXG-0002ab-3U for submit@debbugs.gnu.org; Sat, 14 Jun 2025 12:15:14 -0400 Received: from fout-b6-smtp.messagingengine.com ([202.12.124.149]:45009) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQTXB-0002Vp-66 for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 12:15:11 -0400 Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfout.stl.internal (Postfix) with ESMTP id 414AA11400B9; Sat, 14 Jun 2025 12:15:03 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Sat, 14 Jun 2025 12:15:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749917703; x= 1750004103; bh=sdGgE/6u+RnYK/rvA46KYPBFQu4fHdpBdqs9wapcbyY=; b=E bwLIB+uYbjRSeKGRTamcSh00nAJrNAfpJwaWXzMUSMT/zWmPWnVZHPHqq5k+zYVL wOGyo+8WsmuHjZTw32L0NFkkW1OoyF4SUmInYiuspIH7nfwzoEUY5ta668M8pxZk 47+3N21IN3TPnCvqpvqj9OO3AeagTyzAoPcfnCZ1H64I7UU+37jmJL246MSsxwvY eKq0reVsUiuACpHz9s1zM9IpT81k9kOOULpp6uQ/2O8kb8ao3XHEoQwL8MCQ2GBj 5ZQf31lu+ersVGN3Zs2skCZy8f2DsIBcgD9NozBypF+l0HoMyab54aps4yG9lsDO 4DwasFj9M+5Y2ntcy8iSQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749917703; x=1750004103; bh=sdGgE/6u+RnYK/rvA46KYPBFQu4fHdpBdqs 9wapcbyY=; b=g1EX53yQmM/EhhXnoyw7xcHjgHtrMwcp/L00HBpIoHqYm5eSAyw NhbRfkODQrhVu55Bv5YvFz7f2qAbFtmybpvShtT5YNP+IJia4JB59GB6p9WAc4+D TNSBtgoq1JPEIfmX0+XYHOPd/ihy+/dQr6fj02FE/aizy9E+hBbkwMxuZW0b8BWt cIczArORI0Raz2L59Lch/R7eTD0EW8w2sf5OHGMy1Lag0gMRNLyI+it54XJcizIz Jf/jL6Dgs6wXaGXN3ORJ1dqR472Gd+JaCdlz2wZByDkAkUwp2ldbXLDaEcqZws/i F95OZdxBCxdGk05m2WsTX5PSP36gTci4k5g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvuddvkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeejffefuddvieefgeevkeeggfelhefhffeg leetffetuedttefggeffheeufeektdenucffohhmrghinhepghhnuhdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsthgvvhgvnhes shhtvggsrghlihgvnhdrtghomhdpnhgspghrtghpthhtohepgedpmhhouggvpehsmhhtph houhhtpdhrtghpthhtohepvghlihiisehgnhhurdhorhhgpdhrtghpthhtoheprhhplhhu ihhmsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjeekjeejfeesuggvsggsuhhgshdrgh hnuhdrohhrghdprhgtphhtthhopehlrghrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 14 Jun 2025 12:15:02 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86ikkysgck.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> Date: Sat, 14 Jun 2025 09:15:00 -0700 Message-ID: <87ecvm9u63.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Thu, 12 Jun 2025 13:54:01 -0700 >> >> Ok, I think I've figured it out. We loop twice and wait on a timeout >> (READ_OUTPUT_DELAY_INCREMENT) the second loop because we're already done >> reading. > > We wait the second time because there could be more input, so I don't > see how this could be wrong in general. I also don't quite see the > "gotcha!" in your analysis below: There is no available input. There might be more input in 10ms, but we do know that there's no more available right now because we poll pselect with a zero timeout when `wait' is `MINIMAL'. The following check prevents us from exiting the loop (in this case) when there are file descriptors with waiting data, regardless of what the timeout is: https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=bec823b107ef7d3b51b8e430ccab82c81bd63d24#n5810 There is still a design question of how this SHOULD behave (discussed at the end). >> The first time through, we hit the following, recording that we've >> gotten some output: >> >> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5974 > > OK. > >> We hit the following, setting our timeout to READ_OUTPUT_DELAY_INCREMENT: >> >> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5679 > > This is to _shorten_ the wait timeout: > > /* If we've got some output and haven't limited our timeout > with adaptive read buffering, limit it. */ > if (got_some_output > 0 && !process_skipped > && (timeout.tv_sec > || timeout.tv_nsec > READ_OUTPUT_DELAY_INCREMENT)) > timeout = make_timespec (0, READ_OUTPUT_DELAY_INCREMENT); > > So I don't quite see how this could be a problem here. Am I missing > something? Nothing is wrong here, I'm explaining why we're adding exactly 10ms to every call to `accept-process-output' with a nil PROCESS. The fact that it only decreases the timeout also explains why your patch to reduce the timeout in `url-retrieve-synchronously' makes this faster (as long as the new timeout is less than 10ms). >> Then we hit the select call and wait the full timeout (nothing to read) >> and we finally exit here (because our timeout has passed): >> >> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5832 > > The "full timeout" being the value of READ_OUTPUT_DELAY_INCREMENT, is > that right? Then why is it not TRT, since it _shortens_ the wait? Yes. > >> I think what we need to do is modify: >> >> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5978 >> >> from >> >> if (wait_proc == XPROCESS (proc)) >> wait = MINIMUM; >> >> to >> >> if (!wait_proc || wait_proc == XPROCESS (proc)) >> wait = MINIMUM; >> >> So that we set the timeout to 0 here: >> >> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=81a3e4e51167be51c63eae682331210bc62f7280#n5439 >> >> And exit early. > > Why? That code was there for a reason. What if we are waiting for > other processes as well? > > What did I miss in the above description that explains why the current > code is wrong? My understanding is that `accept-process-output' is supposed to return immediately when output has been read BOTH when PROCESS is nil and when it is non-nil. If this is not the case, then: 1. We now understand why passing a nil process to `accept-process-output' is slower than passing a specific process. 2. The fix to the original bug report is to apply the first patch I sent to have `url-retrieve-synchronously' wait on a specific process. Looking at the history, it looks like the code in question was changed in bug#20978. However, from what I can tell, we should only apply adaptive read buffering when `process-adaptive-read-buffering' is non-nil. I think we may need to change the check in question to: if (!process_output_skip && (!wait_proc || wait_proc == XPROCESS (proc))) wait = MINIMUM; But I'm very much not sure about that. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 12:25:51 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 16:25:51 +0000 Received: from localhost ([127.0.0.1]:42111 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQThW-0003rS-J9 for submit@debbugs.gnu.org; Sat, 14 Jun 2025 12:25:51 -0400 Received: from fhigh-b4-smtp.messagingengine.com ([202.12.124.155]:54835) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQThT-0003qu-Qr for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 12:25:48 -0400 Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfhigh.stl.internal (Postfix) with ESMTP id 2B2C82540108; Sat, 14 Jun 2025 12:25:42 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Sat, 14 Jun 2025 12:25:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749918342; x= 1750004742; bh=MITjktcqlpChF5fZSKAaW/EJvOrnKivSx11+LHvKHxY=; b=T Wpqt0FedT7Bq63GRWwD0AbU4ZGnL2f4opBaVUQMTlaAxWd47MKKQPWFEBe3/9iFx Paar/IH5lRdBEqsCuBRd+Itkq1g6laceN6K3HojkNyi02PGgHR3ePFLMeXWEATK2 ekzbVEGl7Er4Ygt4nBxtNuPwvlmVS8wtCXdJ3VyLY0iBagiK+LxrnvYUQ2krbdFk o/y2oYqw4XsLGfGtuE4csOkI8p1mUoSLCtYMmSGXuy0+GubDmRGJL7OpmDGa6tiO PuTZQHk0PDQUbYnDKfvczEi8W2M3dqz3WjsZp3AQwtn9BILqe0iIAEpQswyXcf2N 80t+NZeqN4UVdztj65J3A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749918342; x=1750004742; bh=MITjktcqlpChF5fZSKAaW/EJvOrnKivSx11 +LHvKHxY=; b=CaynswSs5iuxR/hBwMgiT0wSZAkeJymhxaWjQ8g2Bc00i0Gqs31 ionCS81Uc0QriWljKxiBlgj32S0e8SUUvDKcxHnIcsBP1ujtj2aXplYihlpw6ZJp IM21VNceNCF1Zo3oEMu4fvjf2ogwme4q4v4+KSqb+3eGzJAI3pHjnKl45xkifSc2 ilvQSzHVuz1TUOJ8Mo47GbMlzM6NUjUhZbjWGGkYJGmALIg9j5k7V/zRFE7xpGLk mwEjk7TM4J7fsYnjOn4DmMCJJ9kRcLirq7w62W5DIP34EYvSI8bJPxey3EanIcwB vBN1ejBHK7AorHmJrGTp11X5Lng2Sq5riiQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvudeftdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpedvkeehkeegleehheeggfduleektefhhffg ueffteekgedtvdefuddutddtjeejvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehsthgvvhgvnhesshhtvggsrghlihgvnhdrtghomhdpnhgs pghrtghpthhtohepgedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepvghlihiise hgnhhurdhorhhgpdhrtghpthhtoheprhhplhhuihhmsehgmhgrihhlrdgtohhmpdhrtghp thhtohepjeekjeejfeesuggvsggsuhhgshdrghhnuhdrohhrghdprhgtphhtthhopehlrg hrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 14 Jun 2025 12:25:41 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86frg2sfp2.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> <8734c3h6xp.fsf@stebalien.com> <86frg2sfp2.fsf@gnu.org> Date: Sat, 14 Jun 2025 09:25:40 -0700 Message-ID: <87bjqq9tob.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Fri, 13 Jun 2025 10:45:06 -0700 >> >> First, the actual bug: When called with no PROCESS, >> `accept-process-output' would wait an additional 10ms >> (READ_OUTPUT_DELAY_INCREMENT) after successfully reading data from some >> (any) process. `url-retrieve-synchronously' was slow for localhost calls >> because 10ms is a long time for fast requests. This bug would add 10ms >> of latency to every request. > > But it's an unneeded delay only if there's no more output from the > subprocess during those 10 ms, isn't that so? And we don't really > know if all the process output was read, do we? (discussed in the other email) >> My SECOND patch (0772d96d, the one to process.c) fixes this >> underlying bug by setting the timeout to 0 in this case: when we manage >> to read data from some process and aren't waiting on any specific >> process. > > But what if we are also waiting for output from other processes? Your > patch will slow those down, won't it? In you case, you only care > about a single process, but that's not so in general. (discussed in the other email) >> Next steps: >> >> 1. The SECOND patch fixes the actual bug and should be installed to fix >> the underlying bug (assuming I correctly diagnosed the problem/solution, >> but I'm pretty sure I did). > > I disagree for now. I need to understand better why you think it's > TRT in the general case. (discussed in the other email) >> 2. Personally, I would install the FIRST patch as well as it avoids >> returning to `url-retrieve-synchronously' until we've actually read >> relevant data. > > What if url-retrieve was called for more than a single connection? > Won't your FIRST patch cause a regression in that case? I assume you're talking about redirects here? That's the only multiple-process case I know about. In this case, it'll just return back to the top of the `url-retrieve-synchronously' loop (same as it would before) and wait on the next process. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 12:34:10 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 16:34:10 +0000 Received: from localhost ([127.0.0.1]:42151 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQTpY-00058Z-KH for submit@debbugs.gnu.org; Sat, 14 Jun 2025 12:34:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54340) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQTpU-00056n-UB for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 12:34:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQTpP-0005So-9V; Sat, 14 Jun 2025 12:33:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=jWYsTJeMR9drOuYLQND2Ondt2n1Hv1ubQkIjv1gvNf8=; b=J45bJGMnXzFQ hWsVA1fq+ZLSiDHZh06YgxgrM6R9/EfF+a36kWsYyq3a7LGdMoP3Z2/QpyS9MB43dSEAAT+OKtGxL 5rcaYCtZD0CexsbczmOsjgAOHISaOs2CZ8hAJXBaK5+5hTtOJebr8M3BxppfpCyjzgk++i1af+INM F2anU+cJac2t3RTRP7Nx4awwYjRivGuyyQkOuaAcl1QWiKXhDSR/1kkOh5oNYSOOa4nJrEuMXG24/ k6hnWHcnpc7PbU4vlPq5JhAb4EHoAnOns3WL2Iq9I9/ZXnibCp3bcXPWv+Ovz22Lt1W0272IGMtyT s8qGD11ywku882E0OYECWw==; Date: Sat, 14 Jun 2025 19:33:57 +0300 Message-Id: <86msaaqo3u.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87ecvm9u63.fsf@stebalien.com> (message from Steven Allen on Sat, 14 Jun 2025 09:15:00 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Sat, 14 Jun 2025 09:15:00 -0700 > > Eli Zaretskii writes: > > > We wait the second time because there could be more input, so I don't > > see how this could be wrong in general. I also don't quite see the > > "gotcha!" in your analysis below: > > There is no available input. There might be more input in 10ms That's what I meant by "there could be more input". > My understanding is that `accept-process-output' is supposed to return > immediately when output has been read BOTH when PROCESS is nil and when > it is non-nil. That's true, but wait_reading_process_output is called not just by accept-process-output, it is called by several other callers, and in that case it is not necessarily true that we need to exit immediately upon the first available channel. Keep in mind that reading from a subprocess also invokes a filter function, if there is one, so it might make sense in some cases to read from more than one source, perhaps even from all of the ones that have output ready to be read. > Looking at the history, it looks like the code in question was changed > in bug#20978. However, from what I can tell, we should only apply > adaptive read buffering when `process-adaptive-read-buffering' is > non-nil. I think we may need to change the check in question to: > > if (!process_output_skip && (!wait_proc || wait_proc == XPROCESS (proc))) > wait = MINIMUM; That could indeed fly, but process-adaptive-read-buffering is non-nil by default. Does url set it to nil? In any case, I think the addition of process-adaptive-read-buffering test should only affect the case when wait_proc is zero. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 13:04:24 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 17:04:24 +0000 Received: from localhost ([127.0.0.1]:42188 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQUIp-0001f2-Nk for submit@debbugs.gnu.org; Sat, 14 Jun 2025 13:04:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50682) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQUIn-0001de-7k for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 13:04:22 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQUIh-0000B5-4a; Sat, 14 Jun 2025 13:04:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=l/7CxTBa1NMrn+bZu6w4yW/58toob7u9qQn0uC/BIrk=; b=P05U4Oq5v2kf nUJFDHlqmINbr+fNLTQtV6Bq0EdDQtBTKrG8QnnnTUhEff4aiBj7ZHGWxX5qYVuXwgpqijkDzB8n3 MtQxWwypLLOAWlf8VlMaHh9lXmi6V2tYBThClqExhQWP5lCw5+K8EYAmBTFpzKJzIJdfRtHrVU36T /CvjBrQguPppb64CqhADofovFDODO3MmyR2DW820SWFCJaOt9mVOxfgerea5UzoAxp7lNzmeciRMU 2oOsFAIl73U1rkGFW/xLonO9DIxcFIbJBCd0S8+z7WKD64/G/L1AjMtSv1m9hv01uLxWjUo5u/Rjz ihrDeBLKejXfn7F3v4rhqA==; Date: Sat, 14 Jun 2025 20:04:11 +0300 Message-Id: <86ldpuqmpg.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87bjqq9tob.fsf@stebalien.com> (message from Steven Allen on Sat, 14 Jun 2025 09:25:40 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> <8734c3h6xp.fsf@stebalien.com> <86frg2sfp2.fsf@gnu.org> <87bjqq9tob.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Sat, 14 Jun 2025 09:25:40 -0700 > > Eli Zaretskii writes: > > > What if url-retrieve was called for more than a single connection? > > Won't your FIRST patch cause a regression in that case? > > I assume you're talking about redirects here? That's the only > multiple-process case I know about. In this case, it'll just return back > to the top of the `url-retrieve-synchronously' loop (same as it would > before) and wait on the next process. No, I mean other callers of the same low-level functionality, for example, url-retrieve. Once again, the C function where you propose changes is used in a wide variety of scenarios, and several clients of it could be active at the same time. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 14:24:11 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 18:24:11 +0000 Received: from localhost ([127.0.0.1]:43566 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQVY2-0004KL-Ou for submit@debbugs.gnu.org; Sat, 14 Jun 2025 14:24:11 -0400 Received: from fout-b7-smtp.messagingengine.com ([202.12.124.150]:52327) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQVY0-0004Jp-DA for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 14:24:09 -0400 Received: from phl-compute-05.internal (phl-compute-05.phl.internal [10.202.2.45]) by mailfout.stl.internal (Postfix) with ESMTP id 5055411400C4; Sat, 14 Jun 2025 14:24:02 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Sat, 14 Jun 2025 14:24:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749925442; x= 1750011842; bh=oW/9/UdA3/8cdYA546eWz/LWTNCpLvJm5hXcjdOUtys=; b=R nIcQIwEU5+Y0lvTzUSmkgIbJSSUpHGshoP0GBM8KlZ7+kP2AME1gKZdyflmFi5uc vKMZvZcWC5+tWmfNn/c8gc4GB9f4+O9JaJJbeYy9fFBm9Ahh+TMMxydbBuGFQ4jO CGAK1P6kBAp0ywl4z949jUS/LcDEgIsLYnOt6dmkmhIlWxK15/YL5eQQONEG5TEg LW0pQ0lcGz3m3Jxm3cTsLA11Fx1nVT9HkeYwBdosc4FlDlpGrjgeiarG0jU5W54x IvgTl0u2BlpnVnQew/GNyDpj1EFQ6hI8fnEwCbBdA2+trOZnWGEz+tENzvoio7Ba AmX4j/ncuCvSx1K/VtPeg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749925442; x=1750011842; bh=oW/9/UdA3/8cdYA546eWz/LWTNCpLvJm5hX cjdOUtys=; b=RuBNNiCGGQSIsBlSwmY9k66G0owWs2BSX6UwXtKbN+NoXQMw4kY x8WQf/vAqA7ySZbveUVgntG1W+J5B1dPgejan1OU+l1VEK2Q2F+HlYCKiWpgasJ3 zJYDURxtlqxbogOdse6TBjdk9Ozpdc13SbdCpCu4alfAu6Q4C943oIwUTdD+xMCe APDsj+U9hkw3aBWqVgSnowjErhzxtlOyU6HWSoOYncu/GT61Tt2UdzpayPF3fcr6 v+OuABqLWyVOQMdkTidERGC5bIXWFK4qCUN+L/76OC/x2fto54LB1tULWQOzuoT8 OzFr/gu7Jw2KkY8JoWLFxzFOz2c6onS06/Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvudehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpedvkeehkeegleehheeggfduleektefhhffg ueffteekgedtvdefuddutddtjeejvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehsthgvvhgvnhesshhtvggsrghlihgvnhdrtghomhdpnhgs pghrtghpthhtohepgedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepvghlihiise hgnhhurdhorhhgpdhrtghpthhtoheprhhplhhuihhmsehgmhgrihhlrdgtohhmpdhrtghp thhtohepjeekjeejfeesuggvsggsuhhgshdrghhnuhdrohhrghdprhgtphhtthhopehlrg hrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 14 Jun 2025 14:24:01 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86ldpuqmpg.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> <8734c3h6xp.fsf@stebalien.com> <86frg2sfp2.fsf@gnu.org> <87bjqq9tob.fsf@stebalien.com> <86ldpuqmpg.fsf@gnu.org> Date: Sat, 14 Jun 2025 11:23:58 -0700 Message-ID: <871prm9o75.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Sat, 14 Jun 2025 09:25:40 -0700 >> >> Eli Zaretskii writes: >> >> > What if url-retrieve was called for more than a single connection? >> > Won't your FIRST patch cause a regression in that case? >> >> I assume you're talking about redirects here? That's the only >> multiple-process case I know about. In this case, it'll just return back >> to the top of the `url-retrieve-synchronously' loop (same as it would >> before) and wait on the next process. > > No, I mean other callers of the same low-level functionality, for > example, url-retrieve. > > Once again, the C function where you propose changes is used in a wide > variety of scenarios, and several clients of it could be active at the > same time. In that case, `accept-process-output' will process output for those other processes as usual (calling filters/sentinels, etc.) but won't return call until we have data that's relevant to this call to `url-retrieve-synchronously'. We haven't set JUST-THIS-ONE in this call to `accept-process-output' so we won't simply *ignore* all other process output. To be clear, calling `accept-process-output' with a nil process is the exception, not the norm (8 out of 105 call sites). From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 14:29:08 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 18:29:09 +0000 Received: from localhost ([127.0.0.1]:43675 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQVcq-0004yY-00 for submit@debbugs.gnu.org; Sat, 14 Jun 2025 14:29:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58562) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQVcl-0004wa-N6 for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 14:29:05 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQVcf-0001YJ-QK; Sat, 14 Jun 2025 14:28:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=0WYLpoKJNLKFBs5rLNv/QnraNFjmYNZMvjwMnrpwmoo=; b=niD7Uzo0QC9R 0zt0UhWz990I/36SQLnfo2CHWYGrB9YCSatc9mogvmWKvvISfhmsznck4zM+DBKsWc5ULVOtSq0em 14rsE9qu7z9sdS7Vyjd4HGgMJ49A83gTEne0Ev9+qlDV4RX34XFjNPmKyISSkHuB6RnRBz3r3ChAH CJVDxwMH7RC8E+VWKMIu575ZZ/Zn8e8mFxwD/tRW9Ill2ob680TiqCjMdYoeXeQohBCmfBevMLuRd YpKzThyQfg8SCfcC4NlITDIP509266vpXvowQd+hEJH02sg8HhIMLrADSprZAu4GBDzyEhhHhXFZ5 Uxpgu3DJGBR5jaEVqUza0g==; Date: Sat, 14 Jun 2025 21:28:52 +0300 Message-Id: <86ikkyqisb.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <871prm9o75.fsf@stebalien.com> (message from Steven Allen on Sat, 14 Jun 2025 11:23:58 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> <8734c3h6xp.fsf@stebalien.com> <86frg2sfp2.fsf@gnu.org> <87bjqq9tob.fsf@stebalien.com> <86ldpuqmpg.fsf@gnu.org> <871prm9o75.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Sat, 14 Jun 2025 11:23:58 -0700 > > Eli Zaretskii writes: > > > No, I mean other callers of the same low-level functionality, for > > example, url-retrieve. > > > > Once again, the C function where you propose changes is used in a wide > > variety of scenarios, and several clients of it could be active at the > > same time. > > In that case, `accept-process-output' will process output for those > other processes as usual (calling filters/sentinels, etc.) but won't > return call until we have data that's relevant to this call to > `url-retrieve-synchronously'. We haven't set JUST-THIS-ONE in this call > to `accept-process-output' so we won't simply *ignore* all other process > output. > > To be clear, calling `accept-process-output' with a nil process is the > exception, not the norm (8 out of 105 call sites). That might be, but in this particular case we changed the call to give it the nil argument for a reason, and it will be hard to convince me to go back on that decision. So I suggest to explore other ways of solving this first. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 15:37:53 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 19:37:53 +0000 Received: from localhost ([127.0.0.1]:44695 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQWhL-0006Rj-Gd for submit@debbugs.gnu.org; Sat, 14 Jun 2025 15:37:52 -0400 Received: from fhigh-b6-smtp.messagingengine.com ([202.12.124.157]:51505) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQWhI-0006QV-Ee for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 15:37:49 -0400 Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfhigh.stl.internal (Postfix) with ESMTP id 8880325400CA; Sat, 14 Jun 2025 15:37:42 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-02.internal (MEProxy); Sat, 14 Jun 2025 15:37:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749929862; x= 1750016262; bh=eJl5mfFheJ0QFLwxexgFT1IVzHOWaFXceI8Ph3YW4Ec=; b=b Q75vds8qhZOgbCpQb1SVAJO+bBNHFPqPqzVxRehTEh34k4Z0M3H03VwttIDuo8Om f0U7jvo4Rsblhe/bnLdhmwZB6Q1jU5Vba4zhWW9NSDG70i5AY9FCEYWFh7sd7o3C OuTe/HbPyBUa3e3opU49evSUs8tCBXMBty/MGtPHkqne9pvMONJ0JKIcHTkLVqGu jcdPQwHzPWVqOy44EDY9EJEpCMP/9F2iWBSZQKPLqAe3IAv9Cn9OPSAQd4HOj8t3 zHotzeBW6RhEyQ/ORzg6sQcIjBeCxRCnJZwSv7xQQy71C7S7E63ZY/hepGLBkq0N G8eTreoVUm5jZoThA2IoA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749929862; x=1750016262; bh=eJl5mfFheJ0QFLwxexgFT1IVzHOWaFXceI8 Ph3YW4Ec=; b=YBCovt+RbjioIZiAi1eJSs4aZ4dE2t3ctiQKIjEfpvA9UrpKkyI Lf++2VsjqNfHmGvAUTj4UmRT+F4BMQ/OGPpxBOvstO8Ocv2+bZuu3uXxDmAZY0DQ VQl6xkHoaaVEzmkT7hJoRouBcl5hyq/tDuxtT/XKGJ8CPmSnFQ/gGyiQJ50vsRM0 syJ1c6+Xlu6m6RTw0uTo3U8yOq5fDQanF4A5lv5FhJd/N0dam6BG4daV/T1CiT/N fLxGV4eTobN5SYMmtKMTCmprtoyl4RhZezxDyfXKpdDgT8kDWgT6nA70GYF+Hpa9 tkMdVbJRgboBysFyKXTNYweHbCsds7RdM1Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvudeilecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpedvkeehkeegleehheeggfduleektefhhffg ueffteekgedtvdefuddutddtjeejvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehsthgvvhgvnhesshhtvggsrghlihgvnhdrtghomhdpnhgs pghrtghpthhtohepgedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepvghlihiise hgnhhurdhorhhgpdhrtghpthhtoheprhhplhhuihhmsehgmhgrihhlrdgtohhmpdhrtghp thhtohepjeekjeejfeesuggvsggsuhhgshdrghhnuhdrohhrghdprhgtphhtthhopehlrg hrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 14 Jun 2025 15:37:41 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86ikkyqisb.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> <8734c3h6xp.fsf@stebalien.com> <86frg2sfp2.fsf@gnu.org> <87bjqq9tob.fsf@stebalien.com> <86ldpuqmpg.fsf@gnu.org> <871prm9o75.fsf@stebalien.com> <86ikkyqisb.fsf@gnu.org> Date: Sat, 14 Jun 2025 12:37:38 -0700 Message-ID: <87v7oy867x.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Sat, 14 Jun 2025 11:23:58 -0700 >> >> Eli Zaretskii writes: >> >> > No, I mean other callers of the same low-level functionality, for >> > example, url-retrieve. >> > >> > Once again, the C function where you propose changes is used in a wide >> > variety of scenarios, and several clients of it could be active at the >> > same time. >> >> In that case, `accept-process-output' will process output for those >> other processes as usual (calling filters/sentinels, etc.) but won't >> return call until we have data that's relevant to this call to >> `url-retrieve-synchronously'. We haven't set JUST-THIS-ONE in this call >> to `accept-process-output' so we won't simply *ignore* all other process >> output. >> >> To be clear, calling `accept-process-output' with a nil process is the >> exception, not the norm (8 out of 105 call sites). > > That might be, but in this particular case we changed the call to give > it the nil argument for a reason, and it will be hard to convince me > to go back on that decision. It was changed in 93e1248c2085dfb675d7ed916ec5621e3fe6e2c6 as a blind attempt to fix a hard to reproduce bug. The commit message was: * lisp/url/url.el (url-retrieve-synchronously): Use `accept-process-output' on a null process. That is the safer, more conventional way of achieving non-blocking sleep-for (bug#49897). I understand if you want to leave well enough alone, but I also want to find SOME way to fix this bug. Which leads us to... > So I suggest to explore other ways of solving this first. This is what I've been trying to do. There are only two ways to fix this: 1. Change `accept-process-output' to not unconditionally wait an additional 10ms after receiving output when PROCESS is nil "just in case" there's some additional data to read. 2. Change `url-retrieve-synchronously' pass a non-nil PROCESS to `accept-process-output' thereby avoiding the additional 10ms wait. I've been trying to go down the first path (I think that *is* the correct path) but fixing it will necessarily require `accept-process-output` to return immediately after receiving output when PROCESS is nil, instead of waiting around in case some additional output arrives. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 16:01:18 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 20:01:18 +0000 Received: from localhost ([127.0.0.1]:44715 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQX41-0002HZ-B6 for submit@debbugs.gnu.org; Sat, 14 Jun 2025 16:01:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55472) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQX3x-0002G7-TR for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 16:01:15 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQX3r-00032K-Mh; Sat, 14 Jun 2025 16:01:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=IXj/sgr/4EuMr7pN4M6jp3SPfuq3Xsd5qi3IOQQI3WM=; b=JTOzreNd7f6D 1KE5p/gMNVNeY8uPW/LIXUN9g+5rS6agLi12ujd6jDoa2rd1DcfReVIKZntXVouXYLv7xQAbO3zKu XBrlQSGc4ztjUYUGsbF+GyjwIpwxwdMK8v0ChauyAYa/xpH6PSV0PnCAWetRSRPwZPyYIer51xN2D ajU6EbM1Gf3cO+UtDfYD4TIYlmDHZbyyJkvNoxzKA7a0sj2/Vf3l4YvpOtHMckn7qVr2s0fnokLxA DqB773ZVpzFHhLZSVR5NYzY+fjqmTB3mqTUYAjt461QpwhOAa4YDmafReS9zDxawcJtRNTG1CPq9L +FARzWwMlvfAAC3EYT0rkw==; Date: Sat, 14 Jun 2025 23:01:03 +0300 Message-Id: <86frg2qeio.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87v7oy867x.fsf@stebalien.com> (message from Steven Allen on Sat, 14 Jun 2025 12:37:38 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> <8734c3h6xp.fsf@stebalien.com> <86frg2sfp2.fsf@gnu.org> <87bjqq9tob.fsf@stebalien.com> <86ldpuqmpg.fsf@gnu.org> <871prm9o75.fsf@stebalien.com> <86ikkyqisb.fsf@gnu.org> <87v7oy867x.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > Date: Sat, 14 Jun 2025 12:37:38 -0700 > > Eli Zaretskii writes: > > >> To be clear, calling `accept-process-output' with a nil process is the > >> exception, not the norm (8 out of 105 call sites). > > > > That might be, but in this particular case we changed the call to give > > it the nil argument for a reason, and it will be hard to convince me > > to go back on that decision. > > It was changed in 93e1248c2085dfb675d7ed916ec5621e3fe6e2c6 as a blind > attempt to fix a hard to reproduce bug. The commit message was: > > * lisp/url/url.el (url-retrieve-synchronously): Use > `accept-process-output' on a null process. That is the safer, more > conventional way of achieving non-blocking sleep-for (bug#49897). > > I understand if you want to leave well enough alone, but I also want to > find SOME way to fix this bug. Which leads us to... > > > So I suggest to explore other ways of solving this first. > > This is what I've been trying to do. There are only two ways to fix > this: > > 1. Change `accept-process-output' to not unconditionally wait an > additional 10ms after receiving output when PROCESS is nil "just in > case" there's some additional data to read. > > 2. Change `url-retrieve-synchronously' pass a non-nil PROCESS to > `accept-process-output' thereby avoiding the additional 10ms wait. > > I've been trying to go down the first path (I think that *is* the > correct path) but fixing it will necessarily require > `accept-process-output` to return immediately after receiving output > when PROCESS is nil, instead of waiting around in case some additional > output arrives. Taking a step back, you never explained how come waiting for 10 msec causes some operation to take a full second. Could you please describe what happens in more details, so that such a significant effect of such a small delay could be understood without knowing what emacs-syncthing is or does? One aspect of this which I'd like to understand is how general is this problem and what kind of use patterns will bump into it. Because if it turns out that just emacs-syncthing is affected, a simple solution would be for emacs-syncthing to have its own variant of url-retrieve-synchronously, in which case it can do anything it likes with its variant of that function. By contrast, if we want to modify the code in Emacs, we need at least to understand what use cases need that and consider their importance vs the other kinds, which might suffer degradation in performance due to such a change. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 16:12:14 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 20:12:15 +0000 Received: from localhost ([127.0.0.1]:44728 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQXEZ-0004Cu-P3 for submit@debbugs.gnu.org; Sat, 14 Jun 2025 16:12:14 -0400 Received: from fhigh-b6-smtp.messagingengine.com ([202.12.124.157]:35485) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQXEW-0004BI-I1 for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 16:12:09 -0400 Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfhigh.stl.internal (Postfix) with ESMTP id 85A12254008E; Sat, 14 Jun 2025 16:12:02 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Sat, 14 Jun 2025 16:12:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749931922; x= 1750018322; bh=tGodLXKsXGkE49m9GDrmkwPnYb0ZccFTV+I5FifZ+AI=; b=k BCw299J3xUE/CBWMWMpIl/Z6oNbjA7O6iCvGB+4+GjsoWlfClwhQoefs0fk9YREB FiTT8y9X3ZYIh+3AQ63fA8aIKY5GrzAURqlu8R/8oiBmk0q1gM1aPoHoqcODh1s2 6GuMjdmtujda7685MFU9Ra7LWSwU4nT2Q3nm507pgnuyXr8T89gGQebg+8h/6SoP I8J8HCZCxSarkm6V1KrKyXSSHNpz95mqtbnv8QQvs1eoqzIo3y5WtbnnpZ5PrqWd Aw2DmqTz3Hh9FaupMSDcEKZUvSX/EnMBqPA7+Rbb8X2yQ+/JgN7+Q3tbTq5XofWC ziHzlSpb9yY1mvELBuPxQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749931922; x=1750018322; bh=tGodLXKsXGkE49m9GDrmkwPnYb0ZccFTV+I 5FifZ+AI=; b=o+BmSjqVoje5ulgPAsEnCx/GOVj0KLO8akOdUyYzl6DPW8lvGjx oOX2XXvczLu7Ycih8av5PEQ8ax2PzkEBB5++vvt9JKXxmyzsb7ncqScil7mMZtgW EYZUTgm/JafoNbbBXi+qs6VchuE868dNXfGbdqPmPDifH4rDLsXfY4HivDs0B8a7 qKC4+LhTt2+fje3cgrPLVR+h0vqo3S0Qtc5cCbQ1xlocX/5qmKNJjZhxEjKWwMv1 z5YS205q4USFTGpEe6eROpvKH6OsefNQtUqHBM5Vv6ShJXuuOmbGwWrqak+5im7/ aZ3lBQ/rh1vjEg4SRIxaPvbZcF4BDKROCzQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvudejhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesmhdtreertddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeejudefvdeijeeukedttdegudegffevjeeh heeiueelgfffhfelffehfeevhfdvgeenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehsthgvvhgvnhesshhtvggsrghlihgvnhdrtghomhdpnhgs pghrtghpthhtohephedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepvghlihiise hgnhhurdhorhhgpdhrtghpthhtoheprhhplhhuihhmsehgmhgrihhlrdgtohhmpdhrtghp thhtohepjeekjeejfeesuggvsggsuhhgshdrghhnuhdrohhrghdprhgtphhtthhopehlrg hrshhisehgnhhushdrohhrghdprhgtphhtthhopehirghnsehirghnkhgvlhhlihhnghdr ohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 14 Jun 2025 16:12:01 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86msaaqo3u.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> Date: Sat, 14 Jun 2025 13:12:00 -0700 Message-ID: <87o6uq84mn.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, Ian Kelling X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-=-= Content-Type: text/plain Content-Disposition: inline Ian (CCed): I've run into an issue where `accept-process-output' is always waiting an extra 10ms (READ_OUTPUT_DELAY_INCREMENT) after receiving output as long as the PROCESS passed to it is nil. This appears to be related to the change you made in bug#20978 and I'm wondering if you can shed more light on when `accept-process-output' should wait for more input. Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Sat, 14 Jun 2025 09:15:00 -0700 >> >> Eli Zaretskii writes: >> >> > We wait the second time because there could be more input, so I don't >> > see how this could be wrong in general. I also don't quite see the >> > "gotcha!" in your analysis below: >> >> There is no available input. There might be more input in 10ms > > That's what I meant by "there could be more input". I'd rather make use of what we know: we don't know if we'll get more data in the near future but we *do* know that we've just received some data that the caller (of `accept-process-output`) may want to handle. So we might as well return to the caller and give them a chance to do something about it. The exception to this rule is adaptive read buffering where explicitly _don't_ want to yield too little data at a time if we expect we'll receive more in the near future. I agree we should wait a bit longer in that case (the attached patch should handle that). >> My understanding is that `accept-process-output' is supposed to return >> immediately when output has been read BOTH when PROCESS is nil and when >> it is non-nil. > > That's true, but wait_reading_process_output is called not just by > accept-process-output, it is called by several other callers, and in > that case it is not necessarily true that we need to exit immediately > upon the first available channel. Keep in mind that reading from a > subprocess also invokes a filter function, if there is one, so it > might make sense in some cases to read from more than one source, > perhaps even from all of the ones that have output ready to be read. > >> Looking at the history, it looks like the code in question was changed >> in bug#20978. However, from what I can tell, we should only apply >> adaptive read buffering when `process-adaptive-read-buffering' is >> non-nil. I think we may need to change the check in question to: >> >> if (!process_output_skip && (!wait_proc || wait_proc == XPROCESS (proc))) >> wait = MINIMUM; > > That could indeed fly, but process-adaptive-read-buffering is non-nil > by default. Does url set it to nil? It is nil by default as of bug#75574 (8a669b6be523e043423b81571a8c94cb49ccc8e5). > In any case, I think the addition of process-adaptive-read-buffering > test should only affect the case when wait_proc is zero. Looking at this more, I'm becoming more and more convinced that this was a bug inadvertently introduced by 12a2691fb254db20607b2067a12b795a6d7c6649. That patch was trying to make adaptive read buffering work again but it went too far and applied the delay everywhere. I've attached what I think is the correct patch, but I've also CCed Ian (author of #20978 and the associated patch) in hopes of getting a bit more context here. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Exit-accept-process-output-early-when-we-get-output-.patch >From 8ab170b8ad4718e6d5d12bbc85e9bcda02fffc0d Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 14 Jun 2025 10:05:05 -0700 Subject: [PATCH] Exit accept-process-output early when we get output from any process Previously, the caller requested to wait on output from any process (not a specific process), we'd always end up waiting 10ms (READ_OUTPUT_DELAY_INCREMENT) one final time before returning, even when adaptive read buffering was disabled. * src/process.c (wait_reading_process_output): skip adaptive read buffering delays when we're (a) not reading from any specific process and (b) adaptive read buffering is disabled. (Bug#78773) --- src/process.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/src/process.c b/src/process.c index e61ec425f7e..81a4aa32dcf 100644 --- a/src/process.c +++ b/src/process.c @@ -5671,13 +5671,12 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd, process_output_skip = 0; } - /* If we've got some output and haven't limited our timeout - with adaptive read buffering, limit it. */ - if (got_some_output > 0 && !process_skipped - && (timeout.tv_sec - || timeout.tv_nsec > READ_OUTPUT_DELAY_INCREMENT)) - timeout = make_timespec (0, READ_OUTPUT_DELAY_INCREMENT); - + /* If we've got some output and aren't waiting on a process + via adaptive read buffering, switch our waiting mode to + MINIMUM vacuum up any data remaining to be read then + exit. */ + if (got_some_output > 0 && !process_skipped) + wait = MINIMUM; if (NILP (wait_for_cell) && just_wait_proc >= 0 && timespec_valid_p (timer_delay) @@ -5974,9 +5973,6 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd, got_some_output = nread; if (nread > 0) { - /* Vacuum up any leftovers without waiting. */ - if (wait_proc == XPROCESS (proc)) - wait = MINIMUM; /* Since read_process_output can run a filter, which can call accept-process-output, don't try to read from any other processes -- 2.49.0 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 16:51:47 2025 Received: (at 78773) by debbugs.gnu.org; 14 Jun 2025 20:51:47 +0000 Received: from localhost ([127.0.0.1]:45224 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQXqs-0001Lj-SG for submit@debbugs.gnu.org; Sat, 14 Jun 2025 16:51:47 -0400 Received: from fhigh-b7-smtp.messagingengine.com ([202.12.124.158]:32901) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQXqq-0001L0-SE for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 16:51:45 -0400 Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfhigh.stl.internal (Postfix) with ESMTP id 4015225400DB; Sat, 14 Jun 2025 16:51:39 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-09.internal (MEProxy); Sat, 14 Jun 2025 16:51:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749934299; x= 1750020699; bh=HT8HVDSYPgCajXO7RwQdPZdWqvEntj2g5P7DmbWoGXs=; b=Y zZlukpPZt+n7HZiRxmxOVXhIdLPTDHVprQjoCnXQ6T49IHtMvVKKAzxZShluO2UP PhjPMaAz/uEQfVPFeRUwAccQk6/lCtEZrGvfyNwDrPsytmPfMpSzGD+HffTvmdh8 VHn2/GMHQBikjT6WPdo/WDmm/iL0c+gBL987o8Uc2E8MIxZ4qu4oQxyU/LsmMACN 9cDpZScroRAmF6rH0yUThPYVug6mmp+egiWHDybhI2NYWfjSLVZIzil47YG0BE06 s0oagXrBHrn9QhDGU8tvt1zrKwezpJHUhOlhqnu6OKFGqNLrT5W/yHEel/h0tDVX lcNkrtjhElF+yol+v0Cpw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749934299; x=1750020699; bh=HT8HVDSYPgCajXO7RwQdPZdWqvEntj2g5P7 DmbWoGXs=; b=IRIuixT7tCcnXcmX6MOevCxzv6mrpRIEK6xiPgCyO0y13I/EOii PRKOUl1qwC/5Qx0amauUol3IX8uNsxtbnr3aYjRYuHdwtn+/lHMkXtH3EBisNgLe yAm+4gaBIclN5oJ/AAAE4ifMLF0en4qgQG+ZD+MkAMWHQHhuGyHCnXNK+VTc83Yp FTAXihmLrZCnHMxJN3Z+ST1qmhRQqvD+lPi1sdLrxs6Ekse/BOC5dEzVJR5J5AjF LzEdqw+IFXQt2OekNxO3IlBP4tmwBFIwH71l/p0I5oz9txwrJrmt6O59KRjynRhZ WFdUaiLwGIAoj/50+9EJ3Ort6/LriMvBUaA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvudekgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpedvkeehkeegleehheeggfduleektefhhffg ueffteekgedtvdefuddutddtjeejvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehsthgvvhgvnhesshhtvggsrghlihgvnhdrtghomhdpnhgs pghrtghpthhtohephedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepvghlihiise hgnhhurdhorhhgpdhrtghpthhtoheprhhplhhuihhmsehgmhgrihhlrdgtohhmpdhrtghp thhtohepjeekjeejfeesuggvsggsuhhgshdrghhnuhdrohhrghdprhgtphhtthhopehlrg hrshhisehgnhhushdrohhrghdprhgtphhtthhopehirghnsehirghnkhgvlhhlihhnghdr ohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 14 Jun 2025 16:51:38 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <87o6uq84mn.fsf@stebalien.com> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> <87o6uq84mn.fsf@stebalien.com> Date: Sat, 14 Jun 2025 13:51:36 -0700 Message-ID: <87ikkyqc6f.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, Ian Kelling X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Steven Allen writes: > > Looking at this more, I'm becoming more and more convinced that this was > a bug inadvertently introduced by > 12a2691fb254db20607b2067a12b795a6d7c6649. That patch was trying to make > adaptive read buffering work again but it went too far and applied the > delay everywhere. I've attached what I think is the correct patch, but > I've also CCed Ian (author of #20978 and the associated patch) in hopes > of getting a bit more context here. > > From 8ab170b8ad4718e6d5d12bbc85e9bcda02fffc0d Mon Sep 17 00:00:00 2001 > From: Steven Allen > Date: Sat, 14 Jun 2025 10:05:05 -0700 > Subject: [PATCH] Exit accept-process-output early when we get output from any > process > This last patch is definitely buggy (causes random hangs) and didn't make the performance any faster. I'm going to poke at this code a bit more. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 14 20:38:48 2025 Received: (at 78773) by debbugs.gnu.org; 15 Jun 2025 00:38:48 +0000 Received: from localhost ([127.0.0.1]:48162 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQbOZ-0001mB-SC for submit@debbugs.gnu.org; Sat, 14 Jun 2025 20:38:48 -0400 Received: from fhigh-b1-smtp.messagingengine.com ([202.12.124.152]:38097) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQbOU-0001ka-Vy for 78773@debbugs.gnu.org; Sat, 14 Jun 2025 20:38:45 -0400 Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfhigh.stl.internal (Postfix) with ESMTP id 173ED2540118; Sat, 14 Jun 2025 20:38:37 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Sat, 14 Jun 2025 20:38:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1749947916; x= 1750034316; bh=g7s7p9ZgnHVbr57ZTvOA348/Kdr7f2eEeO2bIKKgvRk=; b=m YO7PMOVa5NsHYhhlAZayc8Ct2VpXtwmrcuaID8vagwWSE5RLi1fnIXNpmWVpCe2L 5K3Tr6URpBZ9uDPPKd5+x2JIP4ZD25cfVsVqJeAFhvDgeSGJWRLL5RCH1BUzM46A 9+41N3dV2yqkHZ7GmTOgREJXyH2W/HYvHskaH3A/FZRz4UQgj5ufs/gBQcxG7/2d Oa2RwfVUP6FT3I71RhqAVAS1jozhYxC8ZyWDt41ifYm3IOQf2Kj5cRZWCzhuaUG/ D0wDEUrzjHUvWMUFQ1eL4TLK9pP5Yanv/+LkRMzQXacJDgib8JdMQTR4wVP+wMUQ UQ3NPuu6o6DbwSnX2guSA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1749947916; x=1750034316; bh=g7s7p9ZgnHVbr57ZTvOA348/Kdr7f2eEeO2 bIKKgvRk=; b=jyOsJ9E8V210vHouCzS5xQcKm4Uz+9xYsZsfC12iCXsU8m3XkHX W3OMl4quA4h6XXKAe8wKIT9traTW41QFqLxiAoOtOdqhYUCoK17iGoYVFRcQtZGE n2OJumQkf72niol/AXsC95x+D77qB8h+pfCGP+H3zS1rewQFsfblw1ajpStscITJ WHiFTrP1RfdWNkvNjrMa7+IAJSVzigrDJVK559kaeVHzjBbDyoPahH//nbWBvH1G ZqWqv/MtJ6DRvJE4vphaL6hNKvOcvna5zvae0ZC9Kw0f8SgBzUBxzakCvfvPc5vj mftM57UAM+fcw/iA+cYdBjVzwkL0HkQ1oKw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvvddvlecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesmhdtreertddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeejudefvdeijeeukedttdegudegffevjeeh heeiueelgfffhfelffehfeevhfdvgeenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehsthgvvhgvnhesshhtvggsrghlihgvnhdrtghomhdpnhgs pghrtghpthhtohepgedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepvghlihiise hgnhhurdhorhhgpdhrtghpthhtoheprhhplhhuihhmsehgmhgrihhlrdgtohhmpdhrtghp thhtohepjeekjeejfeesuggvsggsuhhgshdrghhnuhdrohhrghdprhgtphhtthhopehlrg hrshhisehgnhhushdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 14 Jun 2025 20:38:36 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86frg2qeio.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <86plf8duap.fsf@gnu.org> <87ecvnhf96.fsf@stebalien.com> <8634c3ej85.fsf@gnu.org> <8734c3h6xp.fsf@stebalien.com> <86frg2sfp2.fsf@gnu.org> <87bjqq9tob.fsf@stebalien.com> <86ldpuqmpg.fsf@gnu.org> <871prm9o75.fsf@stebalien.com> <86ikkyqisb.fsf@gnu.org> <87v7oy867x.fsf@stebalien.com> <86frg2qeio.fsf@gnu.org> Date: Sat, 14 Jun 2025 17:38:34 -0700 Message-ID: <87sek1q1o5.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-=-= Content-Type: text/plain Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> Date: Sat, 14 Jun 2025 12:37:38 -0700 >> >> Eli Zaretskii writes: >> >> >> To be clear, calling `accept-process-output' with a nil process is the >> >> exception, not the norm (8 out of 105 call sites). >> > >> > That might be, but in this particular case we changed the call to give >> > it the nil argument for a reason, and it will be hard to convince me >> > to go back on that decision. >> >> It was changed in 93e1248c2085dfb675d7ed916ec5621e3fe6e2c6 as a blind >> attempt to fix a hard to reproduce bug. The commit message was: >> >> * lisp/url/url.el (url-retrieve-synchronously): Use >> `accept-process-output' on a null process. That is the safer, more >> conventional way of achieving non-blocking sleep-for (bug#49897). >> >> I understand if you want to leave well enough alone, but I also want to >> find SOME way to fix this bug. Which leads us to... >> >> > So I suggest to explore other ways of solving this first. >> >> This is what I've been trying to do. There are only two ways to fix >> this: >> >> 1. Change `accept-process-output' to not unconditionally wait an >> additional 10ms after receiving output when PROCESS is nil "just in >> case" there's some additional data to read. >> >> 2. Change `url-retrieve-synchronously' pass a non-nil PROCESS to >> `accept-process-output' thereby avoiding the additional 10ms wait. >> >> I've been trying to go down the first path (I think that *is* the >> correct path) but fixing it will necessarily require >> `accept-process-output` to return immediately after receiving output >> when PROCESS is nil, instead of waiting around in case some additional >> output arrives. > > Taking a step back, you never explained how come waiting for 10 msec > causes some operation to take a full second. Could you please > describe what happens in more details, so that such a significant > effect of such a small delay could be understood without knowing what > emacs-syncthing is or does? emacs-syncthing makes 36+ RPC calls to the local syncthing process. Each request takes ~1-2ms on my machine so the 10ms extra delay dominates the request time. However, you do bring up a good point as this change (either the first or second patch) improves performance more than I'd expect: 1. Before this patch, opening the syncthing buffer took 900ms (much greater than the expected 36*10ms). 2. After this patch, it takes 50ms (expected given 1-2ms for 36 RPC calls plus the time to draw the buffer). So it would appear that there's something going on here beyond the 10ms delay from READ_OUTPUT_DELAY_INCREMENT (more like a 25ms delay). I'll do some more digging. > One aspect of this which I'd like to understand is how general is this > problem and what kind of use patterns will bump into it. Because if > it turns out that just emacs-syncthing is affected, a simple solution > would be for emacs-syncthing to have its own variant of > url-retrieve-synchronously, in which case it can do anything it likes > with its variant of that function. By contrast, if we want to modify > the code in Emacs, we need at least to understand what use cases need > that and consider their importance vs the other kinds, which might > suffer degradation in performance due to such a change. This applies to any case where one makes multiple sequential RPC requests to a local server, waiting on that request via `accept-process-output' with a nil PROCESS. However, there appears to be an escape hatch that lets, e.g., jsonrpc avoid this issue: you can throw an error out of `accept-process-output'. I've attached a patch that does this and it appears to work, but throwing from a sentinel seems sketchy. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Rewrite-url-retrieve-synchronously-to-throw-when-don.patch >From 2a7c99cef6644984bc7c240f925171ba14dc089c Mon Sep 17 00:00:00 2001 From: Steven Allen Date: Sat, 14 Jun 2025 15:27:22 -0700 Subject: [PATCH] Rewrite url-retrieve-synchronously to throw when done When called with no PROCESS, `accept-process-output' waits ~10ms after first receiving process output before returning, adding (at least) 10ms of latency to any `url-retrieve-synchronously' calls. This patch avoids this by throwing from the network process sentinel and catching it in the outer `url-retrieve-synchronously', thereby returning immediately when the response is ready. --- lisp/url/url.el | 93 ++++++++++++++++++++++++------------------------- 1 file changed, 45 insertions(+), 48 deletions(-) diff --git a/lisp/url/url.el b/lisp/url/url.el index 090a952cf4c..995c97a1a2f 100644 --- a/lisp/url/url.el +++ b/lisp/url/url.el @@ -235,58 +235,55 @@ url-retrieve-synchronously TIMEOUT is passed, it should be a number that says (in seconds) how long to wait for a response before giving up." (url-do-setup) - (let* (;; Ensure we can stop during connection setup (bug#71295). + (let* ((tag (gensym "url-retrieve-synchronously-tag")) + ;; Ensure we can stop during connection setup (bug#71295). (url-asynchronous (not (null timeout))) - data-buffer - (callback (lambda (&rest _args) - (setq data-buffer (current-buffer)) - (url-debug 'retrieval - "Synchronous fetching done (%S)" - data-buffer))) + (throw-on-input nil) (start-time (current-time)) - (proc-buffer (url-retrieve url callback nil silent - inhibit-cookies))) - (if (not proc-buffer) + (callback (lambda (&rest _args) + (let ((data-buffer (current-buffer))) + (url-debug 'retrieval + "Synchronous fetching done (%S)" + data-buffer) + (throw tag data-buffer)))) + proc-buffer) + (catch tag + (setq proc-buffer (url-retrieve url callback nil silent + inhibit-cookies)) + (unless proc-buffer (url-debug 'retrieval "Synchronous fetching unnecessary %s" url) + (throw nil)) + (unwind-protect - (catch 'done - (while (not data-buffer) - (when (and timeout (time-less-p timeout - (time-since start-time))) - (url-debug 'retrieval "Timed out %s (after %ss)" url - (float-time (time-since start-time))) - (throw 'done 'timeout)) - (url-debug 'retrieval - "Spinning in url-retrieve-synchronously: nil (%S)" - proc-buffer) - (when-let* ((redirect-buffer - (buffer-local-value 'url-redirect-buffer - proc-buffer))) - (unless (eq redirect-buffer proc-buffer) - (url-debug - 'retrieval "Redirect in url-retrieve-synchronously: %S -> %S" - proc-buffer redirect-buffer) - (let (kill-buffer-query-functions) - (kill-buffer proc-buffer)) - ;; Accommodate hack in commit 55d1d8b. - (setq proc-buffer redirect-buffer))) - (when-let* ((proc (get-buffer-process proc-buffer))) - (when (memq (process-status proc) - '(closed exit signal failed)) - ;; Process sentinel vagaries occasionally cause - ;; url-retrieve to fail calling callback. - (unless data-buffer - (url-debug 'retrieval "Dead process %s" url) - (throw 'done 'exception)))) - ;; Querying over consumer internet in the US takes 100 - ;; ms, so split the difference. - (accept-process-output nil 0.05))) - ;; Kill the process buffer on redirects. - (when (and data-buffer - (not (eq data-buffer proc-buffer))) - (let (kill-buffer-query-functions) - (kill-buffer proc-buffer))))) - data-buffer)) + (while t + (when (and timeout (time-less-p timeout (time-since start-time))) + (url-debug 'retrieval "Timed out %s (after %ss)" url + (float-time (time-since start-time))) + (throw tag 'timeout)) + (url-debug 'retrieval + "Spinning in url-retrieve-synchronously: nil (%S)" + proc-buffer) + ;; Querying over consumer internet in the US takes 100 + ;; ms, so split the difference. + (accept-process-output nil 0.05)) + ;; Cleanup + (let ((buf proc-buffer)) + (while buf + ;; If there are still running processes, we must have + ;; canceled the request (quit) and/or timed out. Kill the + ;; process. + (when-let* ((proc (get-buffer-process buf))) + ;; Catch & ignore throws from the sentinel (can happen if + ;; the process is still live). + (catch tag (delete-process proc))) + ;; Follow the chain of redirect buffers and clean them up + ;; one by one. + (setq buf (when-let* ((redirect-buffer + (buffer-local-value + 'url-redirect-buffer buf))) + (let (kill-buffer-query-functions) + (kill-buffer proc-buffer)) + redirect-buffer)))))))) ;; url-mm-callback called from url-mm, which requires mm-decode. (declare-function mm-dissect-buffer "mm-decode" -- 2.49.0 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 15 01:17:48 2025 Received: (at 78773) by debbugs.gnu.org; 15 Jun 2025 05:17:49 +0000 Received: from localhost ([127.0.0.1]:51978 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQfkX-0002uq-9z for submit@debbugs.gnu.org; Sun, 15 Jun 2025 01:17:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:32952) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQfkQ-0002tY-NS for 78773@debbugs.gnu.org; Sun, 15 Jun 2025 01:17:42 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQfkJ-0001SE-5a; Sun, 15 Jun 2025 01:17:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=m8v2SOILg1YSRfOte/cUZQVxkgtHlQrmLTdOgiinhjU=; b=k0RaZXzNjjyO h1jHUjbVnw3GDrxoZO0kOp4IiTP9AqrILIw9+L+HVcCX3mC9t15f+K4DRUHTQxW5JFGCJtu+i5aj8 DoyCwn/e8qnOs4B3C7+thhj2RX3AxyLbRJOjfHl2Rgyznhgve40/1jJz2oBQC25sea0wUdRdII+gI t9Y6X5z1Yg5mx6Nm6GMVsS5pKlLzTxxOQb61pRIv3/59u+fObrdXZi6UsiqBqzAejQY9PkfYtDf6a W+C2q3GGTCJAwlP3H7U308Ew1JOlYruaXRuG4qFvI17NMnNVtwNTpz2ywZlZzwZWrhKOASvXveXwH 9EeUkKQFbVHniv4gyZB91A==; Date: Sun, 15 Jun 2025 08:17:25 +0300 Message-Id: <86ecvlr3bu.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87o6uq84mn.fsf@stebalien.com> (message from Steven Allen on Sat, 14 Jun 2025 13:12:00 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> <87o6uq84mn.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, ian@iankelling.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org, Ian Kelling > > Date: Sat, 14 Jun 2025 13:12:00 -0700 > > Ian (CCed): I've run into an issue where `accept-process-output' is > always waiting an extra 10ms (READ_OUTPUT_DELAY_INCREMENT) after > receiving output as long as the PROCESS passed to it is nil. This > appears to be related to the change you made in bug#20978 and I'm > wondering if you can shed more light on when `accept-process-output' > should wait for more input. Yes, Ian's input will be appreciated, especially if he can explain the rationale for this part of the patch in enough detail for us to understand what parts of it are essential and which aren't. > Eli Zaretskii writes: > > >> From: Steven Allen > >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org > >> Date: Sat, 14 Jun 2025 09:15:00 -0700 > >> > >> Eli Zaretskii writes: > >> > >> > We wait the second time because there could be more input, so I don't > >> > see how this could be wrong in general. I also don't quite see the > >> > "gotcha!" in your analysis below: > >> > >> There is no available input. There might be more input in 10ms > > > > That's what I meant by "there could be more input". > > I'd rather make use of what we know: we don't know if we'll get more > data in the near future but we *do* know that we've just received some > data that the caller (of `accept-process-output`) may want to handle. So > we might as well return to the caller and give them a chance to do > something about it. This would be wrong. In most cases, especially those where performance matters, sub-process output arrives in many chunks, and there almost always is another chunk coming very soon. That's how I understand the rationale for READ_OUTPUT_DELAY_INCREMENT stuff and its use. > > That could indeed fly, but process-adaptive-read-buffering is non-nil > > by default. Does url set it to nil? > > It is nil by default as of bug#75574 (8a669b6be523e043423b81571a8c94cb49ccc8e5). So the problem happens only with the master branch, and doesn't exist in Emacs 30 and older? > Looking at this more, I'm becoming more and more convinced that this was > a bug inadvertently introduced by > 12a2691fb254db20607b2067a12b795a6d7c6649. That patch was trying to make > adaptive read buffering work again but it went too far and applied the > delay everywhere. I've attached what I think is the correct patch, but > I've also CCed Ian (author of #20978 and the associated patch) in hopes > of getting a bit more context here. This cannot be TRT, as you have already discovered. Can we please agree that as long as we think the problem happens due to some specific feature of the code, such as process-adaptive-read-buffering, the fix should be limited _only_ to that feature, and should not affect any other uses? Because that's the only way to make changes in this code without risking regressions. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 15 12:56:03 2025 Received: (at 78773) by debbugs.gnu.org; 15 Jun 2025 16:56:03 +0000 Received: from localhost ([127.0.0.1]:59595 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQqeI-0006yq-Hy for submit@debbugs.gnu.org; Sun, 15 Jun 2025 12:56:03 -0400 Received: from fhigh-b3-smtp.messagingengine.com ([202.12.124.154]:44853) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQqeG-0006xO-20 for 78773@debbugs.gnu.org; Sun, 15 Jun 2025 12:56:00 -0400 Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfhigh.stl.internal (Postfix) with ESMTP id 7B6CC25401A1; Sun, 15 Jun 2025 12:55:54 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-09.internal (MEProxy); Sun, 15 Jun 2025 12:55:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1750006554; x= 1750092954; bh=7yqXLlRM9oBEbqwjkA1pPkKFycmE4+QDGkQ4vNvZOYs=; b=h sRTKsIGBJdB34RDL+nkK5Kl+tb83Oy5dx2QcI2HBXO/KUvaGOhs/DtQKjEKhUI8d jZWu3y+CNjyJhAEVobD1wTTmN1SHltFGHsfIb2kr7XWgaid3jttXxianGwLwngj8 XaRNpyjEO/GOavLzjZGWYMKRl/Vhrimm++mpqgo8k/Agw/GjInTmNT9yC16i3uq4 zK2cMCEUJw6uRuZPi+wvS7oDYiTOlTrj0eELjfkd7EtIh22UUu6lQk5Pp3WG5uO0 wSqiah5lezwmGtHkEURJPA9tCNIagNYL6knbuoiVfqAlvI41T5iBCvjRZ/JCDr2N 7mCSs2i7lmWPhteKtflWA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1750006554; x=1750092954; bh=7yqXLlRM9oBEbqwjkA1pPkKFycmE4+QDGkQ 4vNvZOYs=; b=SzHbjg+4Ual3auc3jrmWOM0XiDbqP/00glEihdWLNMGF7qn1eMv J2QNFVkrsSc6MvLq4TIhQrXS2hPk2XLjGonMDqC1wl4Vp0mQCi2HE5UUNXFuHPZ5 atdVtCXiI8q6y0G1lNSsQfT0KkaRKqDdlFWJZU1B6OBttuq8Kdn0aYFxCNs0MgQA +t7iGdFBABYSH9JO3NdIivxQhkVBv/B54WiZwmX4x0Hxad01ZnPR2se69eTy6Nuq PaITqPEIGkQSbuoCQL3O1ununju1LMrhi1ienb2OSyOcj0rhCAd6dt6TmdORonwt 3zsh5EhtrX4bnR4Jft0NgiU4YTbgj5h8gWw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvgedvgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpeejffefuddvieefgeevkeeggfelhefhffeg leetffetuedttefggeffheeufeektdenucffohhmrghinhepghhnuhdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsthgvvhgvnhes shhtvggsrghlihgvnhdrtghomhdpnhgspghrtghpthhtohephedpmhhouggvpehsmhhtph houhhtpdhrtghpthhtohepvghlihiisehgnhhurdhorhhgpdhrtghpthhtoheprhhplhhu ihhmsehgmhgrihhlrdgtohhmpdhrtghpthhtohepjeekjeejfeesuggvsggsuhhgshdrgh hnuhdrohhrghdprhgtphhtthhopehlrghrshhisehgnhhushdrohhrghdprhgtphhtthho pehirghnsehirghnkhgvlhhlihhnghdrohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 15 Jun 2025 12:55:53 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86ecvlr3bu.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> <87o6uq84mn.fsf@stebalien.com> <86ecvlr3bu.fsf@gnu.org> Date: Sun, 15 Jun 2025 09:55:52 -0700 Message-ID: <87frg1aqqv.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, ian@iankelling.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org, Ian Kelling >> >> Date: Sat, 14 Jun 2025 13:12:00 -0700 >> >> Ian (CCed): I've run into an issue where `accept-process-output' is >> always waiting an extra 10ms (READ_OUTPUT_DELAY_INCREMENT) after >> receiving output as long as the PROCESS passed to it is nil. This >> appears to be related to the change you made in bug#20978 and I'm >> wondering if you can shed more light on when `accept-process-output' >> should wait for more input. > > Yes, Ian's input will be appreciated, especially if he can explain the > rationale for this part of the patch in enough detail for us to > understand what parts of it are essential and which aren't. > >> Eli Zaretskii writes: >> >> >> From: Steven Allen >> >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org >> >> Date: Sat, 14 Jun 2025 09:15:00 -0700 >> >> >> >> Eli Zaretskii writes: >> >> >> >> > We wait the second time because there could be more input, so I don't >> >> > see how this could be wrong in general. I also don't quite see the >> >> > "gotcha!" in your analysis below: >> >> >> >> There is no available input. There might be more input in 10ms >> > >> > That's what I meant by "there could be more input". >> >> I'd rather make use of what we know: we don't know if we'll get more >> data in the near future but we *do* know that we've just received some >> data that the caller (of `accept-process-output`) may want to handle. So >> we might as well return to the caller and give them a chance to do >> something about it. > > This would be wrong. In most cases, especially those where > performance matters, sub-process output arrives in many chunks, and > there almost always is another chunk coming very soon. That's how I > understand the rationale for READ_OUTPUT_DELAY_INCREMENT stuff and its > use. Hence my second paragraph: >> The exception to this rule is adaptive read buffering where explicitly >> _don't_ want to yield too little data at a time if we expect we'll >> receive more in the near future. I agree we should wait a bit longer in >> that case (the attached patch should handle that). When adaptive read buffering is enabled, if we read less then 256 bytes from one or more processes, we wait a bit for more output to arrive from these processes (the process_output_skip flag is set to 1 when the adaptive read buffering algorithm kicks in): https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=ebdad09c5a0a822acb52ec58b3319d77d156f0ce#n5646 However, the current code unconditionally waits 10ms when we're not applying the adaptive read algorithm: https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=ebdad09c5a0a822acb52ec58b3319d77d156f0ce#n5679 >> > That could indeed fly, but process-adaptive-read-buffering is non-nil >> > by default. Does url set it to nil? >> >> It is nil by default as of bug#75574 (8a669b6be523e043423b81571a8c94cb49ccc8e5). > > So the problem happens only with the master branch, and doesn't exist > in Emacs 30 and older? The problem being discussed here happens in the latest release as well. I'm simply pointing out that your assertion that process-adaptive-read-buffering is non-nil by default is incorrect on the latest master so we're all on the same page. Additionally, I should note that adaptive read buffering only applies to system processes and pipes, not network connections (it's not enabled in `make-network-process') so it's not relevant to `url-retrieve-synchronously'. I only bring it up is that I agree that it's important to preserve its behavior in `wait_reading_process_output'. >> Looking at this more, I'm becoming more and more convinced that this was >> a bug inadvertently introduced by >> 12a2691fb254db20607b2067a12b795a6d7c6649. That patch was trying to make >> adaptive read buffering work again but it went too far and applied the >> delay everywhere. I've attached what I think is the correct patch, but >> I've also CCed Ian (author of #20978 and the associated patch) in hopes >> of getting a bit more context here. > > This cannot be TRT, as you have already discovered. > > Can we please agree that as long as we think the problem happens due > to some specific feature of the code, such as > process-adaptive-read-buffering, the fix should be limited _only_ to > that feature, and should not affect any other uses? Because that's > the only way to make changes in this code without risking regressions. I don't think the problem happens because of `process-adaptive-read-buffering'. I think the problem happens because, in an attempt to make adaptive read buffering work again, an unconditional 10ms delay was added to `accept-process-output'. (see my first response at the top of this message) From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 15 13:08:10 2025 Received: (at 78773) by debbugs.gnu.org; 15 Jun 2025 17:08:11 +0000 Received: from localhost ([127.0.0.1]:59740 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uQqq2-0007y5-8q for submit@debbugs.gnu.org; Sun, 15 Jun 2025 13:08:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54622) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uQqpz-0007xX-FD for 78773@debbugs.gnu.org; Sun, 15 Jun 2025 13:08:08 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uQqpt-00058Q-NC; Sun, 15 Jun 2025 13:08:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=wp9YQ6Qsk7IShDkjHXlxDK4zMxxBVGrTr2DmfDuuJ3Y=; b=Z93JXq/dXH2D 2chOsDl2qmy30GauS4MlmBT7t8IF8z2CdORIAM5QeDxDOBtDbTxAjnDNkcakig5OJqu+ErYewWxwg 37MA2IzQ0Z5yO9xUdok9d0ltPcmGyyLt97R9fCeTiFUjx0MN3YCesN0Eck64BbKj+zXQRYtJ7LXWG Mq0hG3SCOFKAhhFDWyjLj47GUr9EUZIdvp6wBWanB3S0ZMYg0CZxtyroC7lUH8JXJoFMBwArflwNT 3SId0kIro8i4/JQcpglStnkQbmsmR6WbC22pH9FDednI1kAb991HbyMfnq7yEMVc6BpHeq4b0Sd1t Z3J8fBV7rA/P6bjRM8qw8Q==; Date: Sun, 15 Jun 2025 20:07:57 +0300 Message-Id: <86msa9orv6.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87frg1aqqv.fsf@stebalien.com> (message from Steven Allen on Sun, 15 Jun 2025 09:55:52 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> <87o6uq84mn.fsf@stebalien.com> <86ecvlr3bu.fsf@gnu.org> <87frg1aqqv.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, ian@iankelling.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org, ian@iankelling.org > Date: Sun, 15 Jun 2025 09:55:52 -0700 > > Eli Zaretskii writes: > > >> > That could indeed fly, but process-adaptive-read-buffering is non-nil > >> > by default. Does url set it to nil? > >> > >> It is nil by default as of bug#75574 (8a669b6be523e043423b81571a8c94cb49ccc8e5). > > > > So the problem happens only with the master branch, and doesn't exist > > in Emacs 30 and older? > > The problem being discussed here happens in the latest release as well. > I'm simply pointing out that your assertion that > process-adaptive-read-buffering is non-nil by default is incorrect on > the latest master so we're all on the same page. > > Additionally, I should note that adaptive read buffering only applies to > system processes and pipes, not network connections (it's not enabled in > `make-network-process') so it's not relevant to > `url-retrieve-synchronously'. I only bring it up is that I agree that > it's important to preserve its behavior in > `wait_reading_process_output'. > > >> Looking at this more, I'm becoming more and more convinced that this was > >> a bug inadvertently introduced by > >> 12a2691fb254db20607b2067a12b795a6d7c6649. That patch was trying to make > >> adaptive read buffering work again but it went too far and applied the > >> delay everywhere. I've attached what I think is the correct patch, but > >> I've also CCed Ian (author of #20978 and the associated patch) in hopes > >> of getting a bit more context here. > > > > This cannot be TRT, as you have already discovered. > > > > Can we please agree that as long as we think the problem happens due > > to some specific feature of the code, such as > > process-adaptive-read-buffering, the fix should be limited _only_ to > > that feature, and should not affect any other uses? Because that's > > the only way to make changes in this code without risking regressions. > > I don't think the problem happens because of > `process-adaptive-read-buffering'. I think the problem happens because, > in an attempt to make adaptive read buffering work again, an > unconditional 10ms delay was added to `accept-process-output'. I don't disagree, I'm just saying that if the problem happens only when process-adaptive-read-buffering is (or should be) nil, then whatever changes we discuss should not touch the cases where process-adaptive-read-buffering is non-nil. Did you have a chance to investigate how come these small waits add up to a full second? I think understanding that is very important for the decision what fixes should be considered. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 16 03:40:02 2025 Received: (at 78773) by debbugs.gnu.org; 16 Jun 2025 07:40:03 +0000 Received: from localhost ([127.0.0.1]:41012 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uR4Rm-00055v-14 for submit@debbugs.gnu.org; Mon, 16 Jun 2025 03:40:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46640) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uR4Rj-00055A-Hz for 78773@debbugs.gnu.org; Mon, 16 Jun 2025 03:40:00 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uR4Rd-0003oa-JA; Mon, 16 Jun 2025 03:39:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=a8QWyybLpofFQcSq/mD9yhuU7CIMvuffFJzSsuHhURY=; b=NvZsIhGmrf1h zEvY1YAgzjo1yDVeFrB1lO6l6rUsGlRrsjBUEgRYkHT7puaFEt3to8EFn5DoB1B22+3vk5c5Cp4/R zujd3mSREsxxTQcPi+jOy65SXXuBoskkur8KNKMYDR9NbHN2od0GF8NSA1uJxHQvuWSbgmLexWNFd q7Lra1jbVUDzmAYAORBJ7a+WPko+TlWnv5TzpJ0V6e+BmSmNIhnFMSPA/r3XrsP5k6F8U/suKIQnK XtaOvPfz34vpauvejfMmupACveDhL29oLDhWQn4L7ZZablddjWlCLTmhkoBvaJ/KWI2YGVeHAvbHI BSEcA562LBmUDbBPANZUYg==; Date: Mon, 16 Jun 2025 10:39:50 +0300 Message-Id: <86cyb4p22h.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87frg1aqqv.fsf@stebalien.com> (message from Steven Allen on Sun, 15 Jun 2025 09:55:52 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> <87o6uq84mn.fsf@stebalien.com> <86ecvlr3bu.fsf@gnu.org> <87frg1aqqv.fsf@stebalien.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, ian@iankelling.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org, ian@iankelling.org > Date: Sun, 15 Jun 2025 09:55:52 -0700 > > However, the current code unconditionally waits 10ms when we're not > applying the adaptive read algorithm: > > https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=ebdad09c5a0a822acb52ec58b3319d77d156f0ce#n5679 No, that's NOT what that code does. It _reduces_ the _existing_ timeout to 10 ms if it is longer than 10 ms. So it can only _shorten_ the wait, it can never add a wait that wasn't there. Here's the code again: /* If we've got some output and haven't limited our timeout with adaptive read buffering, limit it. */ if (got_some_output > 0 && !process_skipped && (timeout.tv_sec || timeout.tv_nsec > READ_OUTPUT_DELAY_INCREMENT)) timeout = make_timespec (0, READ_OUTPUT_DELAY_INCREMENT); This clearly leaves any existing wait that is shorter than 10 ms unaltered. > > Can we please agree that as long as we think the problem happens due > > to some specific feature of the code, such as > > process-adaptive-read-buffering, the fix should be limited _only_ to > > that feature, and should not affect any other uses? Because that's > > the only way to make changes in this code without risking regressions. > > I don't think the problem happens because of > `process-adaptive-read-buffering'. I think the problem happens because, > in an attempt to make adaptive read buffering work again, an > unconditional 10ms delay was added to `accept-process-output'. I'm not talking about the reason _why_ the problem happens, I'm talking about which parts of the code it is okay to touch to fix the problem to minimize the risk of regressions in unrelated scenarios. If the problem happens only when process-adaptive-read-buffering is nil, let's not change code that's used when it's non-nil, and vice versa. From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 16 11:51:11 2025 Received: (at 78773) by debbugs.gnu.org; 16 Jun 2025 15:51:11 +0000 Received: from localhost ([127.0.0.1]:48345 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uRC73-0006Bp-IC for submit@debbugs.gnu.org; Mon, 16 Jun 2025 11:51:11 -0400 Received: from fout-b3-smtp.messagingengine.com ([202.12.124.146]:58423) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uRC6z-00069m-Em for 78773@debbugs.gnu.org; Mon, 16 Jun 2025 11:51:07 -0400 Received: from phl-compute-07.internal (phl-compute-07.phl.internal [10.202.2.47]) by mailfout.stl.internal (Postfix) with ESMTP id C0A02114011A; Mon, 16 Jun 2025 11:50:59 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-07.internal (MEProxy); Mon, 16 Jun 2025 11:50:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1750089059; x= 1750175459; bh=tX88FgN3WqSvtYr1nZ7h7l6a+ONrzz2UQUeWEgwilmg=; b=l l+2/0EDygf9RiPOV8ZUbJbV/LYDOZEQ79gBJjzEW95VItWZT6Vh90KuCTqU9TYJ/ SB5LVrBkfiRBR+jASjWKRAfTmp7lqEb1aidj2zUCeKzs03ouSYlmnEZl0zxUWcKj dX2o2oYqBGxlIPRzICmaa1kPin4WKYj/IEd2w0nt4l2B8WU1hW3AwfdulsR72HmJ Uw7panI9MMoyob/Y4HVpaWzDDY1Vo4MB5nx35aTluaFtpo/El799E6TCds4ZnAM/ 9jgGO+eIkhEq7JWrDtseExTGyfpW3hz78A4BGA/fEO+U7l8H9mv3V1ZVvCi938+3 vUeOBjfqE7bvg3iK/sqAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1750089059; x=1750175459; bh=tX88FgN3WqSvtYr1nZ7h7l6a+ONrzz2UQUe WEgwilmg=; b=WAXviUQbIlPEZ88jucz7wQV+8kc4hULFObjYFcM/aCY4MOBMx+D zDH7ksJRFJmrP6XlyKpuuAHjmLdghNy/vF9wVcryGCReiKYSCi7grbL0St99vvOr YH8y2r3eK9L/YXzX6O4P3VqWDPSLHRygM9v+gO4/8rN36rLoeR4eOswLqfd1EZ11 4EY+LPvonm53SNkOdwShzqakV0NVEKvOPr1Pgigii92mHwWRRNhRjZzgiMneAH4D 2catrqYZ9fn8pCQCJa9tSOMLVAuyxHyjbMj1wCKrfdV0B5o1DXnI2XXdbLWCzD+k hVLu1x8PEVkkSi/tAIsMp0vwyydCtlQbg3g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvieelkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgesthdtredttddttden ucfhrhhomhepufhtvghvvghnucetlhhlvghnuceoshhtvghvvghnsehsthgvsggrlhhivg hnrdgtohhmqeenucggtffrrghtthgvrhhnpedvkeehkeegleehheeggfduleektefhhffg ueffteekgedtvdefuddutddtjeejvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehsthgvvhgvnhesshhtvggsrghlihgvnhdrtghomhdpnhgs pghrtghpthhtohephedpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepvghlihiise hgnhhurdhorhhgpdhrtghpthhtoheprhhplhhuihhmsehgmhgrihhlrdgtohhmpdhrtghp thhtohepjeekjeejfeesuggvsggsuhhgshdrghhnuhdrohhrghdprhgtphhtthhopehlrg hrshhisehgnhhushdrohhrghdprhgtphhtthhopehirghnsehirghnkhgvlhhlihhnghdr ohhrgh X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 16 Jun 2025 11:50:58 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86msa9orv6.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> <87o6uq84mn.fsf@stebalien.com> <86ecvlr3bu.fsf@gnu.org> <87frg1aqqv.fsf@stebalien.com> <86msa9orv6.fsf@gnu.org> Date: Mon, 16 Jun 2025 08:50:56 -0700 Message-ID: <87zfe7ofbz.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, ian@iankelling.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org, ian@iankelling.org >> Date: Sun, 15 Jun 2025 09:55:52 -0700 >> >> Eli Zaretskii writes: >> >> >> > That could indeed fly, but process-adaptive-read-buffering is non-nil >> >> > by default. Does url set it to nil? >> >> >> >> It is nil by default as of bug#75574 (8a669b6be523e043423b81571a8c94cb49ccc8e5). >> > >> > So the problem happens only with the master branch, and doesn't exist >> > in Emacs 30 and older? >> >> The problem being discussed here happens in the latest release as well. >> I'm simply pointing out that your assertion that >> process-adaptive-read-buffering is non-nil by default is incorrect on >> the latest master so we're all on the same page. >> >> Additionally, I should note that adaptive read buffering only applies to >> system processes and pipes, not network connections (it's not enabled in >> `make-network-process') so it's not relevant to >> `url-retrieve-synchronously'. I only bring it up is that I agree that >> it's important to preserve its behavior in >> `wait_reading_process_output'. >> >> >> Looking at this more, I'm becoming more and more convinced that this was >> >> a bug inadvertently introduced by >> >> 12a2691fb254db20607b2067a12b795a6d7c6649. That patch was trying to make >> >> adaptive read buffering work again but it went too far and applied the >> >> delay everywhere. I've attached what I think is the correct patch, but >> >> I've also CCed Ian (author of #20978 and the associated patch) in hopes >> >> of getting a bit more context here. >> > >> > This cannot be TRT, as you have already discovered. >> > >> > Can we please agree that as long as we think the problem happens due >> > to some specific feature of the code, such as >> > process-adaptive-read-buffering, the fix should be limited _only_ to >> > that feature, and should not affect any other uses? Because that's >> > the only way to make changes in this code without risking regressions. >> >> I don't think the problem happens because of >> `process-adaptive-read-buffering'. I think the problem happens because, >> in an attempt to make adaptive read buffering work again, an >> unconditional 10ms delay was added to `accept-process-output'. > > I don't disagree, I'm just saying that if the problem happens only > when process-adaptive-read-buffering is (or should be) nil, then > whatever changes we discuss should not touch the cases where > process-adaptive-read-buffering is non-nil. Fair enough. I'm saying that the problem happens regardless of the value of `process-adaptive-read-buffering'. > Did you have a chance to investigate how come these small waits add up > to a full second? I think understanding that is very important for > the decision what fixes should be considered. Not yet, but that's next on my agenda. From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 16 12:10:44 2025 Received: (at 78773) by debbugs.gnu.org; 16 Jun 2025 16:10:44 +0000 Received: from localhost ([127.0.0.1]:48414 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uRCPy-00018i-DG for submit@debbugs.gnu.org; Mon, 16 Jun 2025 12:10:44 -0400 Received: from fhigh-b1-smtp.messagingengine.com ([202.12.124.152]:44937) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uRCPs-00016v-Gw for 78773@debbugs.gnu.org; Mon, 16 Jun 2025 12:10:37 -0400 Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfhigh.stl.internal (Postfix) with ESMTP id DA3542540136; Mon, 16 Jun 2025 12:10:30 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Mon, 16 Jun 2025 12:10:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stebalien.com; h=cc:cc:content-transfer-encoding:content-type:content-type :date:date:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:subject:subject:to:to; s=fm3; t=1750090230; x=1750176630; bh=uaigl3/69n1imafqIvAk2yD/UDnz8t/8 s0T7hwqnDpo=; b=DECgm4qpNkVzNaqelGzYSG8iCo423wGvtjspR7Nit+BivS5Q BdOCgqFsihWYy5+U+6vr3gI44pWASsYitjITISVwg4xaMUsduVnCgfT21/EiTQcf M+MGtPc/IrDolhVRwhYmTY0u0zY/PS+I0SRNWVNd/DqtaesAge/ICWX6Mq0x24cP /c2sLEs0HW27cqz+NrD3C0qE3ZbDKdUmwsOMwgvAHb0Q62uSXcj4FHbaUqbUMYOu nK1ROBT26R8lWv1Qbq5G3BlJpXvh9uUgt9ipLxIVDd7csMAFfbtK5dpPBKXNb/aZ /gd2Qmm7mG0H1WmEfQJSyX1RsB/f7GOtl198SQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1750090230; x= 1750176630; bh=uaigl3/69n1imafqIvAk2yD/UDnz8t/8s0T7hwqnDpo=; b=S Z5WTFqmK6TnkMW9pvi+bQgAxWPDQW8ZZqcjMXQfphYFZfOZD0mbiy4Fyp0Tu57bj 6VwVYzIwrb5rROVsCYo7FB+ANpjv6URRu1IMqXt9Ko4juLLZDX1A3UDY2buyGaTN zTlXBFC8eWgpcoGUdmfpLzaLmflbkmBwAqe42pezaUWSwC1HgTP8WvtLAh8Cl3TZ 9coVDigQn5do9Jlp+bUE0JvL1yT+2guhfTdxxEmzgjuidSqrTtdp7K5ME15vgSGQ NlFZArYuOCloP6x0obBKqsdG3znSGdH3FHBOv3GcVyHFeOSYxh8lwd0o3klojzRb 2CsBFQMDW26qMMx7lmEkQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddugddvjedtvdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufgjfhffkfggtgfgsehtqhertddttdej necuhfhrohhmpefuthgvvhgvnhcutehllhgvnhcuoehsthgvvhgvnhesshhtvggsrghlih gvnhdrtghomheqnecuggftrfgrthhtvghrnhepudevlefgtdekheekgffhtdegheduhfeg tdfgffefteetgfetkeeivddthefhkeetnecuffhomhgrihhnpehgnhhurdhorhhgnecuve hluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhtvghvvghn sehsthgvsggrlhhivghnrdgtohhmpdhnsggprhgtphhtthhopeehpdhmohguvgepshhmth hpohhuthdprhgtphhtthhopegvlhhiiiesghhnuhdrohhrghdprhgtphhtthhopehrphhl uhhimhesghhmrghilhdrtghomhdprhgtphhtthhopeejkeejjeefseguvggssghughhsrd hgnhhurdhorhhgpdhrtghpthhtoheplhgrrhhsihesghhnuhhsrdhorhhgpdhrtghpthht ohepihgrnhesihgrnhhkvghllhhinhhgrdhorhhg X-ME-Proxy: Feedback-ID: ie8a146a7:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 16 Jun 2025 12:10:29 -0400 (EDT) From: Steven Allen To: Eli Zaretskii Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections In-Reply-To: <86cyb4p22h.fsf@gnu.org> References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> <87o6uq84mn.fsf@stebalien.com> <86ecvlr3bu.fsf@gnu.org> <87frg1aqqv.fsf@stebalien.com> <86cyb4p22h.fsf@gnu.org> Date: Mon, 16 Jun 2025 09:10:28 -0700 Message-ID: <87wm9boeff.fsf@stebalien.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, ian@iankelling.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Eli Zaretskii writes: >> From: Steven Allen >> Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org, ian@iankell= ing.org >> Date: Sun, 15 Jun 2025 09:55:52 -0700 >>=20 >> However, the current code unconditionally waits 10ms when we're not >> applying the adaptive read algorithm: >>=20 >> https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h= =3Debdad09c5a0a822acb52ec58b3319d77d156f0ce#n5679 > > No, that's NOT what that code does. It _reduces_ the _existing_ > timeout to 10 ms if it is longer than 10 ms. So it can only _shorten_ > the wait, it can never add a wait that wasn't there. Here's the code > again: > > /* If we've got some output and haven't limited our timeout > with adaptive read buffering, limit it. */ > if (got_some_output > 0 && !process_skipped > && (timeout.tv_sec > || timeout.tv_nsec > READ_OUTPUT_DELAY_INCREMENT)) > timeout =3D make_timespec (0, READ_OUTPUT_DELAY_INCREMENT); > > This clearly leaves any existing wait that is shorter than 10 ms > unaltered. Fair enough, we can be more precise here: (a) after reading output from any process, (b) when we didn't skip any of those processes due to adaptive read buffering, (c) when we're not waiting on any specific process=E2=80=A0, it waits an additional (after reading the output) 10ms or the remainder of the timeout, whichever is shorter, before returning to the user. My point is that it should reduce the timeout to 0 in this case because we've just read process output (got_some_output > 0) and we KNOW that we haven't read too little output from any of the processes we've read from because process_skipped is false (i.e., we did NOT skip a process because it had too little output available for us to read). =E2=80=A0 If we are waiting on a specific process, we will have set wait to MINIMAL here (on the previous pass through the loop): https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=3D8= 1a3e4e51167be51c63eae682331210bc62f7280#n5979 And then the timeout will already have been set to 0 here: https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/process.c?h=3D8= 1a3e4e51167be51c63eae682331210bc62f7280#n5439 >> > Can we please agree that as long as we think the problem happens due >> > to some specific feature of the code, such as >> > process-adaptive-read-buffering, the fix should be limited _only_ to >> > that feature, and should not affect any other uses? Because that's >> > the only way to make changes in this code without risking regressions. >>=20 >> I don't think the problem happens because of >> `process-adaptive-read-buffering'. I think the problem happens because, >> in an attempt to make adaptive read buffering work again, an >> unconditional 10ms delay was added to `accept-process-output'. > > I'm not talking about the reason _why_ the problem happens, I'm > talking about which parts of the code it is okay to touch to fix the > problem to minimize the risk of regressions in unrelated scenarios. > If the problem happens only when process-adaptive-read-buffering is > nil, let's not change code that's used when it's non-nil, and vice > versa. The problem happens regardless of the value of process-adaptive-read-buffering. From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 16 12:26:13 2025 Received: (at 78773) by debbugs.gnu.org; 16 Jun 2025 16:26:13 +0000 Received: from localhost ([127.0.0.1]:48497 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uRCex-0003jy-Uh for submit@debbugs.gnu.org; Mon, 16 Jun 2025 12:26:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50192) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uRCeu-0003ie-Ml for 78773@debbugs.gnu.org; Mon, 16 Jun 2025 12:26:09 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uRCeo-000235-9Y; Mon, 16 Jun 2025 12:26:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=icud8/QIgHauA/XlUsIVNTweOkcGsvOT/+4WZxemzH0=; b=MvvOheX0l6olRvz9G5Gw 1zTj6KPTRgvJDoCCcYapC92CFliDRsCAezEsFv1JG6ezQ15nuH2SX3Tz0UAcote8oE3YG9hB8gecp k+ZMOMyhDtOK7JhBTYc96Voyekw9E6WOOIt/fkslVH66faGw6Ec9xhBMp2y0h+fiDVhDkjTT4r4zs UtLOh4LK4YVcLbJd2QQ35nkAzE5sPITEHQMLPdUa0VE3cp0T9YqUp6MLZRJVaTFXReopXNzaeU4ee TRv94x0ILC199NAmIwIpFeP71YhA9lyJM0X7qnY9fR5RLNXVNG1AaAEt6GzuoSOZijBgxf8nge6ET kWEqaAeUxTM66w==; Date: Mon, 16 Jun 2025 19:25:59 +0300 Message-Id: <86o6unodpk.fsf@gnu.org> From: Eli Zaretskii To: Steven Allen In-Reply-To: <87wm9boeff.fsf@stebalien.com> (message from Steven Allen on Mon, 16 Jun 2025 09:10:28 -0700) Subject: Re: bug#78773: [PATCH] Speedup url-retrieve-synchronously for low-latency connections References: <8734c51u0i.fsf@stebalien.com> <8634c5h30i.fsf@gnu.org> <87frg59wsa.fsf@gmail.com> <86ldpxfc68.fsf@gnu.org> <87y0twdeim.fsf@stebalien.com> <86zfecerva.fsf@gnu.org> <87v7p0ahs1.fsf@stebalien.com> <87o6usadg6.fsf@stebalien.com> <86ikkysgck.fsf@gnu.org> <87ecvm9u63.fsf@stebalien.com> <86msaaqo3u.fsf@gnu.org> <87o6uq84mn.fsf@stebalien.com> <86ecvlr3bu.fsf@gnu.org> <87frg1aqqv.fsf@stebalien.com> <86cyb4p22h.fsf@gnu.org> <87wm9boeff.fsf@stebalien.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 78773 Cc: 78773@debbugs.gnu.org, rpluim@gmail.com, larsi@gnus.org, ian@iankelling.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Steven Allen > Cc: rpluim@gmail.com, 78773@debbugs.gnu.org, larsi@gnus.org, ian@iankelling.org > Date: Mon, 16 Jun 2025 09:10:28 -0700 > > Eli Zaretskii writes: > > > /* If we've got some output and haven't limited our timeout > > with adaptive read buffering, limit it. */ > > if (got_some_output > 0 && !process_skipped > > && (timeout.tv_sec > > || timeout.tv_nsec > READ_OUTPUT_DELAY_INCREMENT)) > > timeout = make_timespec (0, READ_OUTPUT_DELAY_INCREMENT); > > > > This clearly leaves any existing wait that is shorter than 10 ms > > unaltered. > > Fair enough, we can be more precise here: > > (a) after reading output from any process, > > (b) when we didn't skip any of those processes due to adaptive read > buffering, > > (c) when we're not waiting on any specific process†, > > it waits an additional (after reading the output) 10ms or the remainder > of the timeout, whichever is shorter, before returning to the user. Agreed. > My point is that it should reduce the timeout to 0 in this case because > we've just read process output (got_some_output > 0) and we KNOW that we > haven't read too little output from any of the processes we've read from > because process_skipped is false (i.e., we did NOT skip a process > because it had too little output available for us to read). I think you are looking at this only from the POV of how fast should the caller of wait_reading_process_output return. But that's not the only use of this function: it is also used in applications where reading large amounts of sub-process output should be as fast as possible. In those other applications, waiting for a short while in case there's another chunk soon might be preferable, for minimizing the time it takes to read the entire output from the sub-process. > >> > Can we please agree that as long as we think the problem happens due > >> > to some specific feature of the code, such as > >> > process-adaptive-read-buffering, the fix should be limited _only_ to > >> > that feature, and should not affect any other uses? Because that's > >> > the only way to make changes in this code without risking regressions. > >> > >> I don't think the problem happens because of > >> `process-adaptive-read-buffering'. I think the problem happens because, > >> in an attempt to make adaptive read buffering work again, an > >> unconditional 10ms delay was added to `accept-process-output'. > > > > I'm not talking about the reason _why_ the problem happens, I'm > > talking about which parts of the code it is okay to touch to fix the > > problem to minimize the risk of regressions in unrelated scenarios. > > If the problem happens only when process-adaptive-read-buffering is > > nil, let's not change code that's used when it's non-nil, and vice > > versa. > > The problem happens regardless of the value of > process-adaptive-read-buffering. Once again, I'm talking about which case the modifications should affect. If url-retrieve-synchronously always uses the nil value, then whatever changes we make should not affect the case when this variable is non-nil, because then we are running the risk of introducing unintended regressions.