From unknown Mon Jun 23 07:49:20 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#71094 <71094@debbugs.gnu.org> To: bug#71094 <71094@debbugs.gnu.org> Subject: Status: [PATCH] Prefer to run find and grep in parallel in rgrep Reply-To: bug#71094 <71094@debbugs.gnu.org> Date: Mon, 23 Jun 2025 14:49:20 +0000 retitle 71094 [PATCH] Prefer to run find and grep in parallel in rgrep reassign 71094 emacs submitter 71094 Spencer Baugh severity 71094 normal tag 71094 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Tue May 21 10:35:23 2024 Received: (at submit) by debbugs.gnu.org; 21 May 2024 14:35:23 +0000 Received: from localhost ([127.0.0.1]:49465 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9QaI-0007Gh-Rv for submit@debbugs.gnu.org; Tue, 21 May 2024 10:35:23 -0400 Received: from lists.gnu.org ([209.51.188.17]:56000) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9QaF-0007Ga-DK for submit@debbugs.gnu.org; Tue, 21 May 2024 10:35:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9Qa8-0008Dh-Qa for bug-gnu-emacs@gnu.org; Tue, 21 May 2024 10:35:13 -0400 Received: from mxout1.mail.janestreet.com ([38.105.200.78]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9Qa5-00023o-Ga for bug-gnu-emacs@gnu.org; Tue, 21 May 2024 10:35:12 -0400 From: Spencer Baugh To: bug-gnu-emacs@gnu.org Subject: [PATCH] Prefer to run find and grep in parallel in rgrep Date: Tue, 21 May 2024 10:35:07 -0400 Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1716302107; bh=jglpTl5xjQaurrWecGMRyL2FNqPUq09Tg9kfSp3U0jI=; h=From:To:Cc:Subject:Date; b=wErw/MrsVsv7MhTMgVBH/26TIVWoX8Lwa1kL5bCWLJ+ejpBZkRt3Ed6Yz4aZa2zOa EvBftLDVf9SdNiNmTb79NngR3L3MVThioC4Oygz/fSwu5Cw9bPtR9f0D201KkbQoNn 0uEPZdVrQ13ZQGU/BhPuiGpzwhAbP5BqtH+3bMAXm9u32Ui/z4K3zizrM9b2AHHn2r hmHcrJvQYu/4d2zCG85XsQfGV+mhv/opy5z6SaLj1LleUBKi7ueXiHb8OdxLzBRMno Moy5zdjsXMuNY0H9d+nrpG0FUDITT/pjlqlbmLezKr3Zfn3hTkUw0AAqSMLkctyEsx cizUcGz7boVgQ== Received-SPF: pass client-ip=38.105.200.78; envelope-from=sbaugh@janestreet.com; helo=mxout1.mail.janestreet.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit Cc: Glenn Morris , dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) --=-=-= Content-Type: text/plain Tags: patch grep.el prefers to run "find" and "xargs grep" in a pipeline, which means that "find" can continue searching the filesystem while "xargs grep" searches files. If find and xargs don't support the flags required for this behavior, grep.el will fall back to using the -exec flags to "find", which meant "find" will wait for each "grep" process to complete before continuing to search the filesystem tree. This behavior is controlled by grep-find-use-xargs; `gnu' produces the pipeline and `exec' is the slower fallback. In f3ca7378c1336b3ff98ecb5a99a98c7b2eceece9, the `exec-plus' option was added for grep-find-use-xargs, which improves on `exec' by running one "grep" process to search multiple files, which `gnu' (by using xargs) already did. However, the change erroneously added the `exec-plus' case before the `gnu' case in the autodetection code in grep-compute-defaults, so `exec-plus' would be used even if `gnu' was supported. This change just swaps the two cases, so the faster `gnu' option is once again used in preference to `exec-plus'. In my benchmarking on a large repository, this provides a ~40% speedup. In GNU Emacs 29.2.50 (build 11, x86_64-pc-linux-gnu, X toolkit, cairo version 1.15.12, Xaw scroll bars) of 2024-05-15 built on igm-qws-u22796a Repository revision: 734740051bd377d24899d08d00ec8e1bb8e00e00 Repository branch: emacs-29 Windowing system distributor 'The X.Org Foundation', version 11.0.12011000 System Description: Rocky Linux 8.9 (Green Obsidian) Configured using: 'configure -C --with-x-toolkit=lucid --with-gif=ifavailable' --=-=-= Content-Type: text/patch Content-Disposition: attachment; filename=0001-Prefer-to-run-find-and-grep-in-parallel-in-rgrep.patch >From 06f0683b51088e4c1c080408624f310d6561a381 Mon Sep 17 00:00:00 2001 From: Spencer Baugh Date: Tue, 21 May 2024 10:32:45 -0400 Subject: [PATCH] Prefer to run find and grep in parallel in rgrep grep.el prefers to run "find" and "xargs grep" in a pipeline, which means that "find" can continue searching the filesystem while "xargs grep" searches files. If find and xargs don't support the flags required for this behavior, grep.el will fall back to using the -exec flags to "find", which meant "find" will wait for each "grep" process to complete before continuing to search the filesystem tree. This behavior is controlled by grep-find-use-xargs; `gnu' produces the pipeline and `exec' is the slower fallback. In f3ca7378c1336b3ff98ecb5a99a98c7b2eceece9, the `exec-plus' option was added for grep-find-use-xargs, which improves on `exec' by running one "grep" process to search multiple files, which `gnu' (by using xargs) already did. However, the change erroneously added the `exec-plus' case before the `gnu' case in the autodetection code in grep-compute-defaults, so `exec-plus' would be used even if `gnu' was supported. This change just swaps the two cases, so the faster `gnu' option is once again used in preference to `exec-plus'. In my benchmarking on a large repository, this provides a ~40% speedup. * lisp/progmodes/grep.el (grep-compute-defaults): Prefer `gnu'. for grep-find-use-xargs over `exec-plus'. --- lisp/progmodes/grep.el | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lisp/progmodes/grep.el b/lisp/progmodes/grep.el index 657349cbdff..04056e13685 100644 --- a/lisp/progmodes/grep.el +++ b/lisp/progmodes/grep.el @@ -812,15 +812,15 @@ grep-compute-defaults (unless grep-find-use-xargs (setq grep-find-use-xargs (cond - ((grep-probe find-program - `(nil nil nil ,(null-device) "-exec" "echo" - "{}" "+")) - 'exec-plus) ((and (grep-probe find-program `(nil nil nil ,(null-device) "-print0")) (grep-probe xargs-program '(nil nil nil "-0" "echo"))) 'gnu) + ((grep-probe find-program + `(nil nil nil ,(null-device) "-exec" "echo" + "{}" "+")) + 'exec-plus) (t 'exec)))) (unless grep-find-command -- 2.39.3 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue May 21 16:00:25 2024 Received: (at 71094) by debbugs.gnu.org; 21 May 2024 20:00:25 +0000 Received: from localhost ([127.0.0.1]:50893 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9Veq-0005mI-TA for submit@debbugs.gnu.org; Tue, 21 May 2024 16:00:25 -0400 Received: from fout6-smtp.messagingengine.com ([103.168.172.149]:54189) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9Vem-0005mC-Vd for 71094@debbugs.gnu.org; Tue, 21 May 2024 16:00:23 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfout.nyi.internal (Postfix) with ESMTP id 45C4F13814E6; Tue, 21 May 2024 16:00:10 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Tue, 21 May 2024 16:00:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716321610; x=1716408010; bh=tSIF28CdWL7AuiO28iE4/RINlF4uSxTOgsy7RJqYHQU=; b= SYBaKHvEYTPkMqx/f1+qtMstQrdkGPHaeBj6VGNBmwJDoqKriCfXJJVE7VOOMNUm Q2MjiHmX+QUrFQSiwxeJcVEKsMQEkqq/tDyXRF3VIPxhO/pS03tJd5yVGjmv9zoG VaqUmIQquw7J69TvhHQ+6cGIkyX14ldWHTAiX0URd7/ob9c/eTrnV1/sVynYRaHC tNMJzQaEbuCsRb86zjaMJvaR5OJEtYdnHWlgn3R413ZVP5b43fJhVXEbHB4kAQxn HbePqLLARMPHPqSbQMXpx8wB+tohHGBgPISMzBq9CdvjAulFnuCBVZJmhTpLCZWe NPpsFFJT6yLRtrQbGNgt/A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716321610; x= 1716408010; bh=tSIF28CdWL7AuiO28iE4/RINlF4uSxTOgsy7RJqYHQU=; b=T kYikAzeKfVXE9yJ78/0dF2CIPCbo6oiJzbwgW0Mp59iGq/FzPlWWHOxL+I8PGxvb 3sL3Bhd9ofuzq5iRyZvF7HVuju16Zg0/X4b5QnjMA88Fv/2/0Pzl5hbcPKCx6PD3 kM2iKV2WozIkkgoxIlGmC3quwRIPp9YYIDkK9yrH4JfGV95G8Ri/ItOQBDvGPe7m DCbPGgw7BccuH2nxz9s5wh1o7nPAhV1cNdA9M3s9dJQmeCO2J+O6/85UYjWMx+In Kog+1PoeXdmlIfi4iDURB8O/jji2faMZIUf/1DuMzebhGEoXC5wpfjfF/r3FRASu /nHCOlZsJhTdx4zgcQFXw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeivddgudegudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefkffggfgfuvfevfhfhjggtgfesthejredttddvjeenucfhrhhomhepffhm ihhtrhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrg htthgvrhhnpeetudeljeegheetgfehgeejkeeuhedvveeikeeufedtvddtveefhfdvveeg udejheenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gumhhithhrhiesghhuthhovhdruggvvh X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 21 May 2024 16:00:08 -0400 (EDT) Message-ID: <40400546-1bea-42c5-87d0-407e0b744804@gutov.dev> Date: Tue, 21 May 2024 23:00:06 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Spencer Baugh , 71094@debbugs.gnu.org References: Content-Language: en-US From: Dmitry Gutov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: Glenn Morris X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Spencer, On 21/05/2024 17:35, Spencer Baugh wrote: > In f3ca7378c1336b3ff98ecb5a99a98c7b2eceece9, the `exec-plus' > option was added for grep-find-use-xargs, which improves on > `exec' by running one "grep" process to search multiple files, > which `gnu' (by using xargs) already did. However, the change > erroneously added the `exec-plus' case before the `gnu' case in > the autodetection code in grep-compute-defaults, so `exec-plus' > would be used even if `gnu' was supported. Perhaps the thinking was that piping data through a +1 program, with associated copying, should be more expensive than delegating that to 'find'. > This change just swaps the two cases, so the faster `gnu' option > is once again used in preference to `exec-plus'. In my > benchmarking on a large repository, this provides a ~40% > speedup. I can confirm, an improvement of ~30% here. Specifically in the "many files, few matches" scenario. Nice find. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 08:00:04 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 12:00:04 +0000 Received: from localhost ([127.0.0.1]:55506 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9kdX-0001rJ-Lk for submit@debbugs.gnu.org; Wed, 22 May 2024 08:00:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:55314) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9kdV-0001qH-Ur for 71094@debbugs.gnu.org; Wed, 22 May 2024 08:00:02 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9kdJ-0007ss-Mm; Wed, 22 May 2024 07:59:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=8g3VbakhKVWlbwBLTjtQE4Jclfp9oEUQL+NUB7e0ha8=; b=aha283UF4DBw SBot0gIkVHGpsuRYzgjux5noc5ZfSKoZeAGTJIbLi46gKeM20S8Z+D+xYfUt1bHrSQ+b/Fa5cFSaL GjznuBoaAuuxfY3P5tVFHMb9dNvl7agWRTDu8zqgE6zdcYmyn+WTBOZYhm6dKCjjq/rMfTC1OJVbI xw3rvsqk/lI0YgDevBY2GScmXcH3geGgfSXtJJmRCorTWsV5kWtd6AXLnOesxvtDBgfQCJXEPAcpd hdZlQfdTpkD/+bsAapDYolVUKSj6KcbRH9ECssyep9bt//Xe+RLgHuuNk5FZ5DunAeLL8Heo2gyKm XWeosvhHKXNRXwXnfZchaw==; Date: Wed, 22 May 2024 14:59:39 +0300 Message-Id: <86ttiq6or8.fsf@gnu.org> From: Eli Zaretskii To: Spencer Baugh In-Reply-To: (message from Spencer Baugh on Tue, 21 May 2024 10:35:07 -0400) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > Cc: Glenn Morris , dmitry@gutov.dev > From: Spencer Baugh > Date: Tue, 21 May 2024 10:35:07 -0400 > > grep.el prefers to run "find" and "xargs grep" in a pipeline, > which means that "find" can continue searching the filesystem > while "xargs grep" searches files. If find and xargs don't > support the flags required for this behavior, grep.el will fall > back to using the -exec flags to "find", which meant "find" will > wait for each "grep" process to complete before continuing to > search the filesystem tree. This behavior is controlled by > grep-find-use-xargs; `gnu' produces the pipeline and `exec' is > the slower fallback. > > In f3ca7378c1336b3ff98ecb5a99a98c7b2eceece9, the `exec-plus' > option was added for grep-find-use-xargs, which improves on > `exec' by running one "grep" process to search multiple files, > which `gnu' (by using xargs) already did. However, the change > erroneously added the `exec-plus' case before the `gnu' case in > the autodetection code in grep-compute-defaults, so `exec-plus' > would be used even if `gnu' was supported. > > This change just swaps the two cases, so the faster `gnu' option > is once again used in preference to `exec-plus'. In my > benchmarking on a large repository, this provides a ~40% > speedup. With how many files did you measure the 40% speedup? Can you show the performance with much fewer and much more files than what you used? I suspect that the effect depends on that. (It also depends on the system limit on the number of files and the length of the command line that xargs can use.) The argument about 'find' waiting is no longer relevant with 'exec-plus', since in most cases there will be just one invocation of 'grep'. In any case, please modify the patch so that 'exec-plus' is still preferred on MS-Windows (because most Windows ports of xargs are IME abysmally buggy, so better avoided as much as possible). A comment there with the justification of the order will also be appreciated. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 08:34:25 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 12:34:25 +0000 Received: from localhost ([127.0.0.1]:55694 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9lAm-0002EF-S5 for submit@debbugs.gnu.org; Wed, 22 May 2024 08:34:25 -0400 Received: from fhigh6-smtp.messagingengine.com ([103.168.172.157]:45049) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9lAk-0002E9-Da for 71094@debbugs.gnu.org; Wed, 22 May 2024 08:34:23 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 65BCF1140186; Wed, 22 May 2024 08:34:10 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 22 May 2024 08:34:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716381250; x=1716467650; bh=Bw2pw+KKsuQqZ/gh4mqwA+QH/qC0nRxk9EDXIicx97I=; b= rwqbZfFeVxxbaQLXDb3y84o+z9CuI/c23sVFNuT/w9hSSLmKMaLaNqLbx3CZxJ9m 3cVY+pvY+9puEPlY5PTJyVDV5lqOPSK+rbrKPFqfoxj7Ubd0ulPYtmGy9MdRhMfC KDtVrZqdCULrqpdKuzGq1ws+j+6SLr0SDo9kXmgTgIDdwXzMuzNsDYP2xfvbVB25 PwvoWfpJ0dN5eJoLpTNBHqpk4YgtK/iYy62qkearMjZhBkliH42HU/XkjMgXXw5R r0RbvW5eNHYP+51Sd1u5uONEGzV0iFRe+o2I9fMcgRgQeyrESMMkhFOu6zb8Rqd0 zdhcf+jHBC+pXL83ydu/Pw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716381250; x= 1716467650; bh=Bw2pw+KKsuQqZ/gh4mqwA+QH/qC0nRxk9EDXIicx97I=; b=m UOLMuTfJcITXaw4k2z+GZ9xKM8SKuimqcrFwdsasBcYtO6j0WLzSQcr1gOfsFbX8 96eLW7VEOJyWtN/B9b5TEpayAUByA3AHOBOvrE8JSqhnvDObS/AJFPRKxXdAcASF he35f02EAXIyX0aR0INkmBdWCNG0hJvc9fBCpLNIRj7Mvx2hM10Pp+Ntj7KsqOKq F/7hEUSKYhOsiG3LFFdZKf4jVjD5ZV9/bk0xx3Jy/zrkTGLoSSos7UA8rbpkbQeo QP3nPghKTvvElLrMtGTSreXzVsWKaoThLa9vKGtcesSHEUItC/ATpNXptlMjGtrf wxdut2IzUmBPXOTbZ6zGA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeigedguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 22 May 2024 08:34:09 -0400 (EDT) Message-ID: <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> Date: Wed, 22 May 2024 15:34:06 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Eli Zaretskii , Spencer Baugh References: <86ttiq6or8.fsf@gnu.org> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <86ttiq6or8.fsf@gnu.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, 71094@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 22/05/2024 14:59, Eli Zaretskii wrote: > With how many files did you measure the 40% speedup? Can you show the > performance with much fewer and much more files than what you used? FWIW my test indicated that for a smaller project (such as Emacs) the difference is fairly small - the new code is slightly better or the same. The directory where I saw significant improvement has 300K files. > I > suspect that the effect depends on that. (It also depends on the > system limit on the number of files and the length of the command line > that xargs can use.) The argument about 'find' waiting is no longer > relevant with 'exec-plus', since in most cases there will be just one > invocation of 'grep'. If there's just one invocation, wouldn't that mean that it will happen at the end of the full directory scan? Rather than in parallel. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 08:54:39 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 12:54:39 +0000 Received: from localhost ([127.0.0.1]:55789 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9lUM-0002Ot-Rw for submit@debbugs.gnu.org; Wed, 22 May 2024 08:54:39 -0400 Received: from mxout6.mail.janestreet.com ([64.215.233.21]:59437) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9lUK-0002On-Jn for 71094@debbugs.gnu.org; Wed, 22 May 2024 08:54:37 -0400 From: Spencer Baugh To: Eli Zaretskii Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: <86ttiq6or8.fsf@gnu.org> (Eli Zaretskii's message of "Wed, 22 May 2024 14:59:39 +0300") References: <86ttiq6or8.fsf@gnu.org> Date: Wed, 22 May 2024 08:54:25 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1716382465; bh=McatyCaRI/rnJuGuONJuYNotmBaiGyUAowPHXu13Ti8=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=rEDEK9u2TdoAnzJ2tZb1nh+M6lMRV36ka9q3SCEPZawJuoUMZ/jn3NJpZgAhj2L5t ebqnG4BQurcOD0Bz0cIYjMUV7bAEC6rot6+eLg4o8zrOY7+KKzc+3Fe1nU6PMjIUH7 zODfdKcfWL9eOPFxK8FFDcHEFw1XUIOsBZ42g8HWPP1CnN9fJx4LW/i3qaYxD3uU29 NlMeHH/c2xpo5aW9AzaaEhsENxZHntO0uWeyijudI2lHTM+qP5y5k/HUYYzVPfwMqp fBqBPqSsmmgfPehiljZmeXIxw0UJMT/MT4Wh+NR/PyAQxWcW1SeGOh4idICHOuXV78 PctE1DDET0z2w== X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) --=-=-= Content-Type: text/plain Eli Zaretskii writes: >> Cc: Glenn Morris , dmitry@gutov.dev >> From: Spencer Baugh >> Date: Tue, 21 May 2024 10:35:07 -0400 >> >> grep.el prefers to run "find" and "xargs grep" in a pipeline, >> which means that "find" can continue searching the filesystem >> while "xargs grep" searches files. If find and xargs don't >> support the flags required for this behavior, grep.el will fall >> back to using the -exec flags to "find", which meant "find" will >> wait for each "grep" process to complete before continuing to >> search the filesystem tree. This behavior is controlled by >> grep-find-use-xargs; `gnu' produces the pipeline and `exec' is >> the slower fallback. >> >> In f3ca7378c1336b3ff98ecb5a99a98c7b2eceece9, the `exec-plus' >> option was added for grep-find-use-xargs, which improves on >> `exec' by running one "grep" process to search multiple files, >> which `gnu' (by using xargs) already did. However, the change >> erroneously added the `exec-plus' case before the `gnu' case in >> the autodetection code in grep-compute-defaults, so `exec-plus' >> would be used even if `gnu' was supported. >> >> This change just swaps the two cases, so the faster `gnu' option >> is once again used in preference to `exec-plus'. In my >> benchmarking on a large repository, this provides a ~40% >> speedup. > > With how many files did you measure the 40% speedup? 700k > Can you show the performance with much fewer and much more files than > what you used? Much more is maybe hard, but much fewer is easy: with 212 files (a subset of the original directory I searched), there's no performance change. > I suspect that the effect depends on that. (It also depends on the >system limit on the number of files and the length of the command line >that xargs can use.) The argument about 'find' waiting is no longer >relevant with 'exec-plus', since in most cases there will be just one >invocation of 'grep'. True, it only matters when the directory tree contains more files than can be passed to a single invocation of grep. > In any case, please modify the patch so that 'exec-plus' is still > preferred on MS-Windows (because most Windows ports of xargs are IME > abysmally buggy, so better avoided as much as possible). > > A comment there with the justification of the order will also be > appreciated. Done, attached. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-Prefer-to-run-find-and-grep-in-parallel-in-rgrep.patch >From e7fbfe431ae1f4f004f1d92db2f3b011b30ff682 Mon Sep 17 00:00:00 2001 From: Spencer Baugh Date: Tue, 21 May 2024 10:32:45 -0400 Subject: [PATCH] Prefer to run find and grep in parallel in rgrep grep.el prefers to run "find" and "xargs grep" in a pipeline, which means that "find" can continue searching the filesystem while "xargs grep" searches files. If find and xargs don't support the flags required for this behavior, grep.el will fall back to using the -exec flags to "find", which meant "find" will wait for each "grep" process to complete before continuing to search the filesystem tree. This behavior is controlled by grep-find-use-xargs; `gnu' produces the pipeline and `exec' is the slower fallback. In f3ca7378c1336b3ff98ecb5a99a98c7b2eceece9, the `exec-plus' option was added for grep-find-use-xargs, which improves on `exec' by running one "grep" process to search multiple files, which `gnu' (by using xargs) already did. However, the change erroneously added the `exec-plus' case before the `gnu' case in the autodetection code in grep-compute-defaults, so `exec-plus' would be used even if `gnu' was supported. This change just swaps the two cases, so the faster `gnu' option is once again used in preference to `exec-plus'. In my benchmarking on a large repository, this provides a ~40% speedup. Also, we completely avoid running xargs on MS-Windows, because Eli Zaretskii writes: > most Windows ports of xargs are IME abysmally buggy, so better avoided > as much as possible * lisp/progmodes/grep.el (grep-compute-defaults): Prefer `gnu' for grep-find-use-xargs over `exec-plus', but not on Windows. (bug#71094) --- lisp/progmodes/grep.el | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/lisp/progmodes/grep.el b/lisp/progmodes/grep.el index 0a9de04fce1..ce54c57aabc 100644 --- a/lisp/progmodes/grep.el +++ b/lisp/progmodes/grep.el @@ -812,15 +812,23 @@ grep-compute-defaults (unless grep-find-use-xargs (setq grep-find-use-xargs (cond - ((grep-probe find-program - `(nil nil nil ,(null-device) "-exec" "echo" - "{}" "+")) - 'exec-plus) + ;; For performance, we want: + ;; A. Run grep on batches of files (instead of one grep per file) + ;; B. If the directory is large and we need multiple batches, + ;; run find in parallel with a running grep. + ;; "find | xargs grep" gives both A and B ((and + (not (eq system-type 'windows-nt)) (grep-probe find-program `(nil nil nil ,(null-device) "-print0")) (grep-probe xargs-program '(nil nil nil "-0" "echo"))) 'gnu) + ;; "find -exec {} +" gives A but not B + ((grep-probe find-program + `(nil nil nil ,(null-device) "-exec" "echo" + "{}" "+")) + 'exec-plus) + ;; "find -exec {} ;" gives neither A nor B. (t 'exec)))) (unless grep-find-command -- 2.39.3 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 09:51:26 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 13:51:26 +0000 Received: from localhost ([127.0.0.1]:56085 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9mNK-0003B6-4i for submit@debbugs.gnu.org; Wed, 22 May 2024 09:51:26 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49194) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9mNF-0003B0-Bz for 71094@debbugs.gnu.org; Wed, 22 May 2024 09:51:25 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9mN3-0004WR-A7; Wed, 22 May 2024 09:51:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=3mgLK622PPzlHzwbBWyjSaNUhrzmJ4yiWyFVL4xzynQ=; b=Urzbnn7oqSSU L1BEHOMsoddyjiNFZMRfW9sMteEaGptVeRJ7AKOViluH98ncpa9RaAnEfqRzNcDNB92LWSxkMLqQw Si/IIXl9rBv7aE0MiJF24GOyvlp3y3dIg7K4XG3Lh5jm7zE2adSn4t89go4rExcCtf3MrnVQdAM7c ul9244VCBd5MXVbIKhHRJk0SjZM0Qa4uVTnWoX23P21efaY9opQ0t40QlBjQftFKcJSHr1yHOHs0c aP1uQtrDGpU8+JvYI+bVC7WyqJ+oAABvL847twxCNEFHywE4wLENS/jlOxESlISz9cMQHWj9BELIv sW+w771lUVa+f4EBMVOCDQ==; Date: Wed, 22 May 2024 16:50:57 +0300 Message-Id: <868r026jlq.fsf@gnu.org> From: Eli Zaretskii To: Dmitry Gutov In-Reply-To: <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> (message from Dmitry Gutov on Wed, 22 May 2024 15:34:06 +0300) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > Date: Wed, 22 May 2024 15:34:06 +0300 > Cc: 71094@debbugs.gnu.org, rgm@gnu.org > From: Dmitry Gutov > > On 22/05/2024 14:59, Eli Zaretskii wrote: > > > With how many files did you measure the 40% speedup? Can you show the > > performance with much fewer and much more files than what you used? > > FWIW my test indicated that for a smaller project (such as Emacs) the > difference is fairly small - the new code is slightly better or the same. > > The directory where I saw significant improvement has 300K files. That's what I thought. So we are changing the decade-old defaults to favor huge directories, which is not necessarily the wisest thing to do. > > I > > suspect that the effect depends on that. (It also depends on the > > system limit on the number of files and the length of the command line > > that xargs can use.) The argument about 'find' waiting is no longer > > relevant with 'exec-plus', since in most cases there will be just one > > invocation of 'grep'. > > If there's just one invocation, wouldn't that mean that it will happen > at the end of the full directory scan? Rather than in parallel. That's true, but what is your mental model of how the pipe with xargs works in practice? How many invocations of grep will xargs do, and when will the first invocation happen? From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 10:23:15 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 14:23:15 +0000 Received: from localhost ([127.0.0.1]:56246 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9ms7-0003WD-Do for submit@debbugs.gnu.org; Wed, 22 May 2024 10:23:15 -0400 Received: from fout1-smtp.messagingengine.com ([103.168.172.144]:32869) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9ms3-0003Vm-Jy for 71094@debbugs.gnu.org; Wed, 22 May 2024 10:23:13 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.nyi.internal (Postfix) with ESMTP id 760EA13800B4; Wed, 22 May 2024 10:23:00 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 22 May 2024 10:23:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716387780; x=1716474180; bh=faNf1GI7kINA0SayIDDBlOuXTOKWfK6IsDRbI/Qp0Oc=; b= HQ3B++hqwJJcb/1WqB22BnMUBgspMsypNooMkY0L6CreeKQu7Zxf1FKb1TDZSKnO hZeh/R/ZcOdrb427k7jz5ZSASUm217+e9DQYcw/6Rg/zAr3bK408l7HVEvaNbMDA 3EDuUHHllZJjpqESYvbssvICk1WMwQMY2SREvimU1nxPYL/h6LUVygcbJ8OKG+71 jpQd9K/2fmkigL4V/jXfsCWtxa6f6r3WTOJI8yXvB8BEnAvpH7IEbdcX0LxUp6Qv OaxirYrbScL3Jc86HcJdJ15G2D6n+v9BeYaH+UZukJcZj/VUWKHhdIzh2BvC2vva 3VhTq8fdhb+Mb4djTw+ByQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716387780; x= 1716474180; bh=faNf1GI7kINA0SayIDDBlOuXTOKWfK6IsDRbI/Qp0Oc=; b=I zS1P4tjBo6c9NRBsmTw6gSgYrXRmFPEY914x1ZyRlLtMgMF4Ey0/srM3gd1gTUUm iGT8T12Omqgf7Yo3FHEN0ZpoACKJl4ktfCBa/bkBHdynmVyMGgYCBQUtuNoyIep+ z5ZKBnOf0mXLsqY7kvPxoM0kNNV0s6vzUrH1SkBm5bx0v35FlNXtj8ahPCBR4sOW beuQ+EmqBdDJ5fg5rmi8RBBDsA4k46yrc4rAnfte1CWjWAJSMzRsI3ToffiH1s7p wsaAIVO44H9LUIkoi1PXpHuNv44Ej8DHyjaCmx1k/PfExlCTqcLws75F5xQVeuVy 5COgNnb2K1nE63EXrJ3BQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeigedgudekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 22 May 2024 10:22:58 -0400 (EDT) Message-ID: Date: Wed, 22 May 2024 17:22:56 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Eli Zaretskii References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <868r026jlq.fsf@gnu.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 22/05/2024 16:50, Eli Zaretskii wrote: >> Date: Wed, 22 May 2024 15:34:06 +0300 >> Cc: 71094@debbugs.gnu.org, rgm@gnu.org >> From: Dmitry Gutov >> >> On 22/05/2024 14:59, Eli Zaretskii wrote: >> >>> With how many files did you measure the 40% speedup? Can you show the >>> performance with much fewer and much more files than what you used? >> >> FWIW my test indicated that for a smaller project (such as Emacs) the >> difference is fairly small - the new code is slightly better or the same. >> >> The directory where I saw significant improvement has 300K files. > > That's what I thought. So we are changing the decade-old defaults to > favor huge directories, which is not necessarily the wisest thing to > do. I don't see any regression on small directories, though. And an improvement on big ones. So the way I see it, we're expanding Emacs's applicability to wider audience without any apparent drawbacks. It might actually give us an improvement in smaller projects as well, if we decrease xargs's batch size (with -s or -n). But those are fairly fast already, so it's not critical. >>> I >>> suspect that the effect depends on that. (It also depends on the >>> system limit on the number of files and the length of the command line >>> that xargs can use.) The argument about 'find' waiting is no longer >>> relevant with 'exec-plus', since in most cases there will be just one >>> invocation of 'grep'. >> >> If there's just one invocation, wouldn't that mean that it will happen >> at the end of the full directory scan? Rather than in parallel. > > That's true, but what is your mental model of how the pipe with xargs > works in practice? How many invocations of grep will xargs do, and > when will the first invocation happen? In my mental model xargs acts like an asynchronous queue with batch processing. The first invocation will happen after the output reaches the maximum line number of maximum number of arguments configured. They are system-dependent by default. For example, on my system 'xargs --show-limits' says Size of command buffer we are actually using: 131072 Whereas in the Emacs repository "find ... -print0 | wc" reports 202928 characters. Meaning, it uses just 1.5 'grep' invocations. To see better parallelism there we'll need to either lower the limit or test it in a project at least twice as big. So here is another example: a Linux kernel checkout (76K files). Also about 30% improvement: 1.40s vs 2.00s. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 10:43:10 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 14:43:10 +0000 Received: from localhost ([127.0.0.1]:56340 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9nBO-0003lr-42 for submit@debbugs.gnu.org; Wed, 22 May 2024 10:43:10 -0400 Received: from eggs.gnu.org ([209.51.188.92]:44050) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9nBJ-0003lP-Qk for 71094@debbugs.gnu.org; Wed, 22 May 2024 10:43:08 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9nB7-0005XR-UK; Wed, 22 May 2024 10:42:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=VQyvt4wMYYJdAwlXbko7jtJQ2pI/ZWIGMdHXPQKLsfc=; b=OXg7GMzAqX5v /VUhZyhGxL69ffPj5YCymVBeKOuKs2DuMBbomobLc8sy95Ue/Vm2YsKJfSqgg9imqEjOU67u74fQA zP7FH6WzfdViwMDz/l0OP+JfadjIYeiTjyUztWL/ZdBClqfc6lVO/+1Exq8e+68rM1E3i5WZDs1k6 uCp2DnlRBGmIsyRDLZU8xO9ZQEc7mW1ofdunXSIb+nZ6aKjZtQPRQnuzsBo2LMNfCqnGcusJAJPA3 oA3pn5s7MOyqBgh6p2Rd8J1P0cpOKXVnfJqf2Ea5gvNxNazF4yQOi6fdIJuL0nwz+ubusMjhmg3J4 l6/WnVHKogbqK4ObSUO0uQ==; Date: Wed, 22 May 2024 17:42:50 +0300 Message-Id: <861q5t7vrp.fsf@gnu.org> From: Eli Zaretskii To: Dmitry Gutov In-Reply-To: (message from Dmitry Gutov on Wed, 22 May 2024 17:22:56 +0300) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > Date: Wed, 22 May 2024 17:22:56 +0300 > Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org > From: Dmitry Gutov > > >> The directory where I saw significant improvement has 300K files. > > > > That's what I thought. So we are changing the decade-old defaults to > > favor huge directories, which is not necessarily the wisest thing to > > do. > > I don't see any regression on small directories, though. And an > improvement on big ones. On your system. > > That's true, but what is your mental model of how the pipe with xargs > > works in practice? How many invocations of grep will xargs do, and > > when will the first invocation happen? > > In my mental model xargs acts like an asynchronous queue with batch > processing. The first invocation will happen after the output reaches > the maximum line number of maximum number of arguments configured. They > are system-dependent by default. And can be rather small. But if it is large, then... > For example, on my system 'xargs --show-limits' says > > Size of command buffer we are actually using: 131072 > > Whereas in the Emacs repository "find ... -print0 | wc" reports 202928 > characters. Meaning, it uses just 1.5 'grep' invocations. To see better > parallelism there we'll need to either lower the limit or test it in a > project at least twice as big. ...until xargs collects all those characters, it will not invoke grep, right? So, for directories whose file names total less than those 200K, xargs will still wait until find ends its job, right? > So here is another example: a Linux kernel checkout (76K files). Also > about 30% improvement: 1.40s vs 2.00s. This is all highly system-dependent. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 10:50:58 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 14:50:59 +0000 Received: from localhost ([127.0.0.1]:56387 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9nIw-0003r7-J2 for submit@debbugs.gnu.org; Wed, 22 May 2024 10:50:58 -0400 Received: from fhigh7-smtp.messagingengine.com ([103.168.172.158]:59819) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9nIu-0003r1-GG for 71094@debbugs.gnu.org; Wed, 22 May 2024 10:50:57 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 781C71140192; Wed, 22 May 2024 10:50:45 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Wed, 22 May 2024 10:50:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716389445; x=1716475845; bh=s1Dij0eugnhR43c2DtGd+4b3xenJ9eFS7s0V/Iyg9vQ=; b= jMqRWJ+Gg/vuy172V73snJBe2J5MB0vlpUrPptqblT7/apeHVGo/ZbgvAhpNn7wm JI44c0WpqLrb2zvtnDIidUGTSBu3iw78ns6tr3dDm2+JGdmSnXmtq6/av/fj+fEl Ztm9Msx7awwSpqxPbmeZU5XBea0+ShE6QCMetprxK1wATNZ3klYDu7xW/20UJBum ffOVtysxhfBBSNTeukWUpfdQhM/9zXRhYP52D8ZYZAbdo9r634mNnptN4gf9NtzE IhhUWa32EO06lyLbS5VNRVVoU0Ke7b3t3yDIf2sFowF/8I7l15VzlL8eBWJcpSXH qIaGXjPCDXQibzkyxsi1oQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716389445; x= 1716475845; bh=s1Dij0eugnhR43c2DtGd+4b3xenJ9eFS7s0V/Iyg9vQ=; b=H EvgsA23ZV0woozK8dvBp3Ve3Umau0nlYHI6pBuwt3YW35Ol3OjrDf49Uuy/fKCGo ea/8uElj+083MJEDvKMqrHPxyHRamhP289yvm9SafcB8cAHrF0AzaKjo4xKg5SLR rz4fSzWTCQy2ehdGvd6vpFNUFX4doAqeL7nir9u+NqOwuPv7jmJOlLWCTHFfQupI TxjkM5tXZZUGxidOfcknUMjCn0eQfzhMrbGmbZsgIiEE1WnVGcgb+OszczTHPRen CR6js47gpEkzxL9j7fZp6+7AqbcFkWEAj0GMNhqIdgIMqODY0Ra6QKhgM5EgywpM MK49AkvAeZb1tYdSweNmQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeigedgvddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 22 May 2024 10:50:44 -0400 (EDT) Message-ID: <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> Date: Wed, 22 May 2024 17:50:42 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Eli Zaretskii References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <861q5t7vrp.fsf@gnu.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 22/05/2024 17:42, Eli Zaretskii wrote: >>> That's true, but what is your mental model of how the pipe with xargs >>> works in practice? How many invocations of grep will xargs do, and >>> when will the first invocation happen? >> >> In my mental model xargs acts like an asynchronous queue with batch >> processing. The first invocation will happen after the output reaches >> the maximum line number of maximum number of arguments configured. They >> are system-dependent by default. > > And can be rather small. But if it is large, then... > >> For example, on my system 'xargs --show-limits' says >> >> Size of command buffer we are actually using: 131072 >> >> Whereas in the Emacs repository "find ... -print0 | wc" reports 202928 >> characters. Meaning, it uses just 1.5 'grep' invocations. To see better >> parallelism there we'll need to either lower the limit or test it in a >> project at least twice as big. > > ...until xargs collects all those characters, it will not invoke grep, > right? So, for directories whose file names total less than those > 200K, xargs will still wait until find ends its job, right? That's right. And it's why we're not seeing much of a difference in projects of Emacs's size or smaller. No apparent regression either, though. >> So here is another example: a Linux kernel checkout (76K files). Also >> about 30% improvement: 1.40s vs 2.00s. > > This is all highly system-dependent. Naturally. So it'd be great to see some additional data points from users on other systems. Especially those where the default limit is lower than it is on mine. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 11:27:08 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 15:27:09 +0000 Received: from localhost ([127.0.0.1]:56658 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9nrw-0004NJ-H1 for submit@debbugs.gnu.org; Wed, 22 May 2024 11:27:08 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38428) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9nrs-0004Mt-8O for 71094@debbugs.gnu.org; Wed, 22 May 2024 11:27:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9nrf-0005hz-Cx; Wed, 22 May 2024 11:26:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=7oSdqFsEHKK47UwzpdrnEjie/EJlJ+UT7R0PuVsBCys=; b=se02QEli8gQS 10b1wzWpmAFb6eWPZzFlwnZIFdk5a802hIl7HNV8/Jc/CTZ3h0BBHLybhrkajStfCvYjuLo7SP9jW qABEpWKH19GT2Kw6VFLvGdQSq9ZJRzKOqp176XDUZ1F/ClTdqlW/5S6dFSllaz78fWqc6NfPx27LV EZQ8qhhdt6f/O7OdCydhL87viqDyHn68n2zDU3QQ06AeeULyXFJg9BSy32jOGpeKKng9UdIUzG5xd bD08O5OQAyF8FlO1ED+cOxkDder1kK8S8x01plblhAL+WY3hysVYlpoyfLPi0XKMQC1aeErAO++Rc e1cuUplU9e9mJ9ZBA1hsfA==; Date: Wed, 22 May 2024 18:26:45 +0300 Message-Id: <86wmnl6f62.fsf@gnu.org> From: Eli Zaretskii To: Dmitry Gutov In-Reply-To: <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> (message from Dmitry Gutov on Wed, 22 May 2024 17:50:42 +0300) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > Date: Wed, 22 May 2024 17:50:42 +0300 > Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org > From: Dmitry Gutov > > >> Whereas in the Emacs repository "find ... -print0 | wc" reports 202928 > >> characters. Meaning, it uses just 1.5 'grep' invocations. To see better > >> parallelism there we'll need to either lower the limit or test it in a > >> project at least twice as big. > > > > ...until xargs collects all those characters, it will not invoke grep, > > right? So, for directories whose file names total less than those > > 200K, xargs will still wait until find ends its job, right? > > That's right. And it's why we're not seeing much of a difference in > projects of Emacs's size or smaller. No apparent regression either, though. But we added xargs to the soup. On GNU/Linux, where GNU Findutils are developed, it probably isn't a problem. On other systems, not necessarily... > >> So here is another example: a Linux kernel checkout (76K files). Also > >> about 30% improvement: 1.40s vs 2.00s. > > > > This is all highly system-dependent. > > Naturally. So it'd be great to see some additional data points from > users on other systems. > > Especially those where the default limit is lower than it is on mine. I'd be happy if someone could time these methods on MS-Windows and on some *BSD system, at least. Bonus points for macOS. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 13:48:17 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 17:48:17 +0000 Received: from localhost ([127.0.0.1]:57284 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9q4X-0005fl-4S for submit@debbugs.gnu.org; Wed, 22 May 2024 13:48:17 -0400 Received: from fhigh1-smtp.messagingengine.com ([103.168.172.152]:34791) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9q4S-0005ff-Dy for 71094@debbugs.gnu.org; Wed, 22 May 2024 13:48:15 -0400 Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 2371311401BC; Wed, 22 May 2024 13:48:01 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Wed, 22 May 2024 13:48:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716400081; x=1716486481; bh=oDEme5OS+jkJFZ7CkqHrd0XCZX3vDEP+eml7ZQsxZRw=; b= LnI4DrnF91Hg/Bet3+kxiODVSc+pucF2AOXQGHvrDR/dmAIcnOeSgNcpZhWVmjGU s2vyiiueXGvzwkBqmuhcs09h6ZgtzmeCstVl7fPtXnFMu+U9lWu4WjqKYl1diPfO IE7hrl9YBuWKzTptl7veUdQTa/CGg5wHILynveQDOAdDj8+jqG9g4zCb2JTCn6bg d9fjMrqkGR07vsf07o6mtOFTWqUYF9y8mrtSN5wjmGAWwgSnjoTa9BpaPFKcaQKg 4TKHRv11cnowqWFqaLhf/aRDSVZ+ci2+ZHbWUJ0I0MGwP+i6AEyYR6nYNAoyKku5 +v+ohoWqyWNwHvEHxPj15Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716400081; x= 1716486481; bh=oDEme5OS+jkJFZ7CkqHrd0XCZX3vDEP+eml7ZQsxZRw=; b=Q P8t7FTttwCTozcFm2DnrWHC/Z6qZWkSdUWm72AUvOLlNh1Utd2OC3HdJRjryzhb6 FnyRvHhwW15XLixieQ21QcgYtD4G1fCtUwJCfJwF7pWQc7cgHB8qYS9TJKFJPqPM jI/vsqvC4v2cFl00CmTfPXEFv6wlydxF5IbeBAIMY2yJbuwxDI9e57mdnRGNSp+O +YQm/HqZzSb8n55AuBJUDHNeNM76NQbarfzN4TI566mVnJyRxXfdr1/5FlfB/ZYp 0Kcer1R3gv+n1BIJNXbQ5sPqLhJLZrYkrczb7rUT1JbTGwYP2NbkH285Y/PZk2zQ KsCXM3hqIGgs3NGKMihdg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeigedgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 22 May 2024 13:47:59 -0400 (EDT) Message-ID: <73b2c595-9200-4381-ae0f-2c3e1a2b1f29@gutov.dev> Date: Wed, 22 May 2024 20:47:57 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Eli Zaretskii References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <86wmnl6f62.fsf@gnu.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 22/05/2024 18:26, Eli Zaretskii wrote: > I'd be happy if someone could time these methods on MS-Windows and on > some *BSD system, at least. Bonus points for macOS. As luck would have it, I have an M3 Pro macOS laptop around. The situation with it is odd, as usual. First of all, the default find/xargs/grep installed are some very slow versions from Apple. The patch doesn't seem to change the performance of the search using them, it's just slow either way. Things get better if I install the GNU versions from Homebrew and (setq grep-program "ggrep") at startup. Performance gets better by 4x or so just from that, but still not to the level of my 5-year-old GNU/Linux laptop. The patch doesn't seem to have a make a difference still. If I also set (setq xargs-program "gxargs") then the patch starts improving performance in a large directory (again: Linux kernel), by around 10%. Still more than 3x slower than on my older laptop with Linux. No idea why - the ggrep, gxargs and gfind executables are all reported to be arm64, so I can't blame the x64->arm64 translation layer. To sum up though, the patch under discussion doesn't make things worse on the macOS laptop I tested. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 14:06:57 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 18:06:57 +0000 Received: from localhost ([127.0.0.1]:57379 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9qMb-0005q2-1H for submit@debbugs.gnu.org; Wed, 22 May 2024 14:06:57 -0400 Received: from ledu-giraud.fr ([51.159.28.247]:25875) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9qMY-0005pw-KS for 71094@debbugs.gnu.org; Wed, 22 May 2024 14:06:55 -0400 DKIM-Signature: v=1; a=ed25519-sha256; c=simple/simple; s=ed25519; bh=dCvbjo0R 1QQRke/vf3socaM5EglOSUwjbG0I7JLV2+U=; h=date:references:in-reply-to: subject:cc:to:from; d=ledu-giraud.fr; b=2wTFtui+RVzJAbxFcuD2mRmZBpiOZe zrmq6a1TJoiiGB0nc+hHxtdbjoj0MXKdvxESXzq/D8JQqC35YICNShBg== DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=rsa; bh=dCvbjo0R1QQRke/v f3socaM5EglOSUwjbG0I7JLV2+U=; h=date:references:in-reply-to:subject: cc:to:from; d=ledu-giraud.fr; b=1I3AUDzkl5AghOQARpyyO7KIveU1Wky5tq6FRw HQbSxoIf3r5FjNOEqr8ZprvAmSUb2FMNexW3CJVIsqNYEjmQ700PirseHkcj3vPoeDcUdh 0SFYqxo4GVuiRnzYjJ/QT8j40HtDYBfjcTFMtrfHZRJGYXipcjmCTSemlNejlFMNeqFJx+ 1R5JuvdoeXrI+7cHWZ/Ywy/2ldyFkyYhllSNUc7ixmqAjwXZYqeceEG507vWO31lE8W/gx tzhWTrJyyMHhNRNmnF5upf4a7O2ARSXnmtIuAWms+3LL5z6qzxuJfWDZcUZn/3mR6Lk4nq Whlr8caJ8nzEI9bv0gt+FsuQ== Received: from computer ( [10.1.1.1]) by ledu-giraud.fr (OpenSMTPD) with ESMTPSA id 0aff69f9 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 22 May 2024 20:06:46 +0200 (CEST) From: Manuel Giraud To: Eli Zaretskii Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: <86wmnl6f62.fsf@gnu.org> (Eli Zaretskii's message of "Wed, 22 May 2024 18:26:45 +0300") References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> Date: Wed, 22 May 2024 20:06:44 +0200 Message-ID: <87pltdbu17.fsf@ledu-giraud.fr> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: Dmitry Gutov , 71094@debbugs.gnu.org, rgm@gnu.org, sbaugh@janestreet.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: [...] >> >> So here is another example: a Linux kernel checkout (76K files). Also >> >> about 30% improvement: 1.40s vs 2.00s. >> > >> > This is all highly system-dependent. >> >> Naturally. So it'd be great to see some additional data points from >> users on other systems. >> >> Especially those where the default limit is lower than it is on mine. > > I'd be happy if someone could time these methods on MS-Windows and on > some *BSD system, at least. Bonus points for macOS. I'm not sure it is what you asked for but here is some numbers on OpenBSD (native 'find' and 'xargs'): $ time find ~/emacs-repo -type f -exec grep foo {} + > /dev/null 0m04.09s real 0m03.29s user 0m00.74s system $ time find ~/emacs-repo -type f -print0 | xargs -0 grep foo > /dev/null 0m04.10s real 0m03.45s user 0m00.66s system $ find /usr/src -type f | wc -l 114315 $ time find /usr/src -type f -exec grep foo {} + > /dev/null 0m14.07s real 0m07.68s user 0m06.29s system $ time find /usr/src -type f -print0 | xargs -0 grep foo > /dev/null 0m13.83s real 0m07.94s user 0m06.25s system -- Manuel Giraud From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 14:23:46 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 18:23:46 +0000 Received: from localhost ([127.0.0.1]:57466 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9qcs-0005z4-3X for submit@debbugs.gnu.org; Wed, 22 May 2024 14:23:46 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50594) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9qcp-0005yy-Jm for 71094@debbugs.gnu.org; Wed, 22 May 2024 14:23:44 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9qaX-00039E-71; Wed, 22 May 2024 14:21:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=o+8exqQTyM2iNs0fgypPLxCR7sxg0DpIH/1bcyfQdQk=; b=cC7RRwzv7JYB I3o2+8nFlGjL/j28+YNDVlHInwBViJ7Cc92LmYktCoYuIWnYrckELA4HqERH5e/Nv5Xmz7chQDhJo IwpXePIxj5anP4BsXakXDEJcVnqM+jNKDAiUH+amI8cJiwM9MjFLuTsYljrpB5BfdohXZNnpkCQpQ FJ3HW8uZaZJt1JA7/9hh3YhSqSOx0oxug2yKa1gqGlwz53fdXhJyyDNiiKDjwKtRq8Ee7DHwnkGAd laJUNrLIyFOIPajJ8maDIgcx0T+i6UiLiywMP1bAGjyhi/mFXxYP1ITFv35D2tXRMgXR9VjNCKjjz 6ulZ2OI1pNyr0A5phRas+A==; Date: Wed, 22 May 2024 21:21:14 +0300 Message-Id: <86ttip6739.fsf@gnu.org> From: Eli Zaretskii To: Dmitry Gutov In-Reply-To: <73b2c595-9200-4381-ae0f-2c3e1a2b1f29@gutov.dev> (message from Dmitry Gutov on Wed, 22 May 2024 20:47:57 +0300) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <73b2c595-9200-4381-ae0f-2c3e1a2b1f29@gutov.dev> X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > Date: Wed, 22 May 2024 20:47:57 +0300 > Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org > From: Dmitry Gutov > > To sum up though, the patch under discussion doesn't make things worse > on the macOS laptop I tested. Thanks, it's good to know. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 14:30:45 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 18:30:45 +0000 Received: from localhost ([127.0.0.1]:57503 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9qjd-00081E-Bb for submit@debbugs.gnu.org; Wed, 22 May 2024 14:30:45 -0400 Received: from eggs.gnu.org ([209.51.188.92]:41410) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9qjb-0007nh-T0 for 71094@debbugs.gnu.org; Wed, 22 May 2024 14:30:44 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9qjP-0004IO-CY; Wed, 22 May 2024 14:30:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=JmKZmda7uGbD7hK8vjUUeFudsSpmwHTtNiMvVc/eFnA=; b=eGjkd6/SvCWe uRa0BmavLnlwXGSB7EYkxphGMeumQ5NLGqcmzq7UQXRjVHByhiv+Y7r2PtG3ys28DvGWqhQYUCtVD EmkQ6aBvgZYMb3W2+snR+4XDv8vpsRMy80fO+72EPCgMNynguglgGQDKcWnKk9u1I6i9hVL68SY0P AkylSgqfCtvnOeJVVSCq6m1dG+685NK3dpu3vdQ7Uax1yXfP6NVikkxDrRBMY+mLyDUopT3JslqTK 68kTI0p3Zq60e98o62WSPUxwQVW1SYxqM7EvrsLJxzhdctv16wQPjdqKgkogqYP5xGnulASfN8EpF i24rHGUrEDZXo5lQCBlyaw==; Date: Wed, 22 May 2024 21:30:27 +0300 Message-Id: <86r0dt66nw.fsf@gnu.org> From: Eli Zaretskii To: Manuel Giraud In-Reply-To: <87pltdbu17.fsf@ledu-giraud.fr> (message from Manuel Giraud on Wed, 22 May 2024 20:06:44 +0200) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: dmitry@gutov.dev, 71094@debbugs.gnu.org, rgm@gnu.org, sbaugh@janestreet.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > From: Manuel Giraud > Cc: Dmitry Gutov , sbaugh@janestreet.com, > 71094@debbugs.gnu.org, rgm@gnu.org > Date: Wed, 22 May 2024 20:06:44 +0200 > > Eli Zaretskii writes: > > [...] > > >> >> So here is another example: a Linux kernel checkout (76K files). Also > >> >> about 30% improvement: 1.40s vs 2.00s. > >> > > >> > This is all highly system-dependent. > >> > >> Naturally. So it'd be great to see some additional data points from > >> users on other systems. > >> > >> Especially those where the default limit is lower than it is on mine. > > > > I'd be happy if someone could time these methods on MS-Windows and on > > some *BSD system, at least. Bonus points for macOS. > > I'm not sure it is what you asked for but here is some numbers on > OpenBSD (native 'find' and 'xargs'): > > $ time find ~/emacs-repo -type f -exec grep foo {} + > /dev/null > 0m04.09s real 0m03.29s user 0m00.74s system > $ time find ~/emacs-repo -type f -print0 | xargs -0 grep foo > /dev/null > 0m04.10s real 0m03.45s user 0m00.66s system > > $ find /usr/src -type f | wc -l > 114315 > $ time find /usr/src -type f -exec grep foo {} + > /dev/null > 0m14.07s real 0m07.68s user 0m06.29s system > $ time find /usr/src -type f -print0 | xargs -0 grep foo > /dev/null > 0m13.83s real 0m07.94s user 0m06.25s system Thanks, but we need the timings of the corresponding Emacs commands, not the commands run from the shell prompt. Btw, are you sure that xargs or grep don't pay attention to the fact that their output is redirected to the null device, and do nothing? Some variants of these commands are known to use such a trick, AFAIR. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 14:51:42 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 18:51:42 +0000 Received: from localhost ([127.0.0.1]:57681 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9r3u-0000di-23 for submit@debbugs.gnu.org; Wed, 22 May 2024 14:51:42 -0400 Received: from wfout7-smtp.messagingengine.com ([64.147.123.150]:52225) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9r3p-0000dc-1S for 71094@debbugs.gnu.org; Wed, 22 May 2024 14:51:40 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.west.internal (Postfix) with ESMTP id 085451C00070; Wed, 22 May 2024 14:51:24 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 22 May 2024 14:51:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716403884; x=1716490284; bh=dCftWn62yJWvP3e5qof/1Aj6/05Wc4qyfBlqaQvwaO8=; b= FMGevgBnK8skKMb/M6/rUjT/lgbPQqUov3McgXmmWLINCE1I9T2t5G3VTKETufl9 2EljclPZJcyHfXly4XLuFZhx/UZkn4Ag25OQq81pil8a96IeSxeAiHUb0NZikCyS 5s0yeZAaWBaaZPSPak0oViMMsnghRYoI33T/cg36L9NVeCG9sldRegvIhaoo6Otw NLaNpIPknLJ+APmifAEpTyVEJhjrbdPAOtyalzjBBNRPDt4eg48PkQ3TnQRO8hMd LGGifENdsHB2+pKRLl5ZUge0MVxK4/PXKONE8SKhMO6wx43RIkiNmBSGzTtbEUJg 9tJMCTfCscpAGmKatQUV+g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716403884; x= 1716490284; bh=dCftWn62yJWvP3e5qof/1Aj6/05Wc4qyfBlqaQvwaO8=; b=Y nQDvZhzjKYLHxvmhHodff023glw/0kvvapkuVxUEPlqZzCFPpV0AF1q9onDrzXgr N3QMLZ7cuMxNBNFMkXndK1SObFtxzw7YZO+U0MOILEi4msWyYeKkuG59XY/EuLD/ TrbGoWrxc8uAnXq4iI/lzjEn0fqEk8v0JoWscj88YqlXagx/Vr5mcnVzrOK2SJ0t uqrtTX/EAadZKFpdct+MJKI8gJZWg6OtSG5/TXN3r7P423KqETQASmRH55SuzAEU hoPl5Z24UNrPargCIpvfb61NLp+jsIxIn55/bJmprwk5h1Sz8YGIpmgob7dnPtCE 0Agh3wf1R3hwcMBh26lTA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeigedggeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 22 May 2024 14:51:22 -0400 (EDT) Message-ID: Date: Wed, 22 May 2024 21:51:20 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Manuel Giraud , Eli Zaretskii References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <87pltdbu17.fsf@ledu-giraud.fr> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 22/05/2024 21:06, Manuel Giraud wrote: >> I'd be happy if someone could time these methods on MS-Windows and on >> some *BSD system, at least. Bonus points for macOS. > I'm not sure it is what you asked for but here is some numbers on > OpenBSD (native 'find' and 'xargs'): > > $ time find ~/emacs-repo -type f -exec grep foo {} + > /dev/null > 0m04.09s real 0m03.29s user 0m00.74s system > $ time find ~/emacs-repo -type f -print0 | xargs -0 grep foo > /dev/null > 0m04.10s real 0m03.45s user 0m00.66s system > > $ find /usr/src -type f | wc -l > 114315 > $ time find /usr/src -type f -exec grep foo {} + > /dev/null > 0m14.07s real 0m07.68s user 0m06.29s system > $ time find /usr/src -type f -print0 | xargs -0 grep foo > /dev/null > 0m13.83s real 0m07.94s user 0m06.25s system I'm not sure how many matches for 'foo' are there inside your /usr/src, but if there are a lot, it slows down the last phase (grep output), making the performance gains a wash. For this particular scenario, it's better to search for a string with no matches. Then you won't need to redirect to /dev/null too. From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 15:15:18 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 19:15:18 +0000 Received: from localhost ([127.0.0.1]:57812 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9rQj-0000sh-9i for submit@debbugs.gnu.org; Wed, 22 May 2024 15:15:18 -0400 Received: from ledu-giraud.fr ([51.159.28.247]:44986) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9rQd-0000sa-PB for 71094@debbugs.gnu.org; Wed, 22 May 2024 15:15:16 -0400 DKIM-Signature: v=1; a=ed25519-sha256; c=simple/simple; s=ed25519; bh=vS3Yksva XVMz2LwdgOectFhxkwI0CFDmDxLBR1p6O8U=; h=date:references:in-reply-to: subject:cc:to:from; d=ledu-giraud.fr; b=oSy5YjAq8kDGWrQcOsAgWUPPv6lXiI o8pGHYFGwYUVllfDRnX1MKIwtSJ3G0cq/sg0bKkfySVlxheGXf77MUCw== DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=rsa; bh=vS3YksvaXVMz2Lwd gOectFhxkwI0CFDmDxLBR1p6O8U=; h=date:references:in-reply-to:subject: cc:to:from; d=ledu-giraud.fr; b=KmcjsjRnyyebTqz/TFwQzLFXbh1Xe4xqb54kPl 79VxBzwJfPZzuOVOojFW6gU06xEanm5Zdyxq1Wd1i8dp5AmZmHHpqwBHi9imbWvjBX682u CWjsg1L5Tk2wsyITb5mFHkpMWqO+ltVdWHFObSadE6ktM4iee9Bj44Qk01nvydtvaNZe1B FNqvJqb0nk0Ld7QMPoLEDuO5mFsvwOo39gkPrQhepPYxuNGGMp5B+8p4obevOFbOe57Zv0 QNeWLoEaqbA4WSsyDl3bfI8+xHA9pB42EBIwsHoceQpsxK/oU0ElhGK+6aYl04nbD8fuFk Gp0pGRQxl8yTQ6AiZ+IxFJMg== Received: from computer ( [10.1.1.1]) by ledu-giraud.fr (OpenSMTPD) with ESMTPSA id c20dce1f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 22 May 2024 21:15:04 +0200 (CEST) From: Manuel Giraud To: Eli Zaretskii Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: <86r0dt66nw.fsf@gnu.org> (Eli Zaretskii's message of "Wed, 22 May 2024 21:30:27 +0300") References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> <86r0dt66nw.fsf@gnu.org> Date: Wed, 22 May 2024 21:15:02 +0200 Message-ID: <87le41bqvd.fsf@ledu-giraud.fr> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: dmitry@gutov.dev, 71094@debbugs.gnu.org, rgm@gnu.org, sbaugh@janestreet.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: [...] > Thanks, but we need the timings of the corresponding Emacs commands, > not the commands run from the shell prompt. Ok. What are those commands and how to timed them? > Btw, are you sure that xargs or grep don't pay attention to the fact > that their output is redirected to the null device, and do nothing? > Some variants of these commands are known to use such a trick, AFAIR. I don't know. -- Manuel Giraud From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 15:36:34 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 19:36:34 +0000 Received: from localhost ([127.0.0.1]:57903 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9rlJ-00014G-Kb for submit@debbugs.gnu.org; Wed, 22 May 2024 15:36:33 -0400 Received: from ledu-giraud.fr ([51.159.28.247]:30269) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9rlF-00014A-Kh for 71094@debbugs.gnu.org; Wed, 22 May 2024 15:36:31 -0400 DKIM-Signature: v=1; a=ed25519-sha256; c=simple/simple; s=ed25519; bh=wGGv0ITa KOdlm/w93F2dQ0jE81lchO0TASh3tgrokdU=; h=date:references:in-reply-to: subject:cc:to:from; d=ledu-giraud.fr; b=12gOFV41oFlvkzB42jaR39TxkFfzH7 lMYUNON6gxFobW+x6x18CnS6BczwhcLkeDJ4hqp64qywPk9s78iaX6Cg== DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=rsa; bh=wGGv0ITaKOdlm/w9 3F2dQ0jE81lchO0TASh3tgrokdU=; h=date:references:in-reply-to:subject: cc:to:from; d=ledu-giraud.fr; b=aaQmKCC419IYbgFvn4hjFmbkXTEemwBzxa0Imj M7JeDWjAj8Tkc3fz7QdP/j+tHW0ElldYBAiKNm4gzgqMrOyoTAqXrLPdjS/shsTZRdW+mR Bdu9cNozBbEYAEvSY56WvvFaPx43mtlhbUVm/RhYs0FdM8d+4fKBlMwQ7DxO3HJLR0G5vc HHVXAeW3urZImS+OFlZlD93a40ZPTElpOXvnf4FvuqnU0eIyHPhYmEx3qBjdYATDy+mpK8 z+3MnPgEvDEchICEODJS/Vim34semaFDR2cswPXPn8aZiebv1p0BF9SP3E5j6BN/fSSUgD gu4QXj/q8rOK6vyndebjTQZA== Received: from computer ( [10.1.1.1]) by ledu-giraud.fr (OpenSMTPD) with ESMTPSA id b6114495 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 22 May 2024 21:36:22 +0200 (CEST) From: Manuel Giraud To: Dmitry Gutov Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: (Dmitry Gutov's message of "Wed, 22 May 2024 21:51:20 +0300") References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> Date: Wed, 22 May 2024 21:36:20 +0200 Message-ID: <87h6epbpvv.fsf@ledu-giraud.fr> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, Eli Zaretskii , 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) Dmitry Gutov writes: > On 22/05/2024 21:06, Manuel Giraud wrote: >>> I'd be happy if someone could time these methods on MS-Windows and on >>> some *BSD system, at least. Bonus points for macOS. >> I'm not sure it is what you asked for but here is some numbers on >> OpenBSD (native 'find' and 'xargs'): >> $ time find ~/emacs-repo -type f -exec grep foo {} + > /dev/null >> 0m04.09s real 0m03.29s user 0m00.74s system >> $ time find ~/emacs-repo -type f -print0 | xargs -0 grep foo > /dev/null >> 0m04.10s real 0m03.45s user 0m00.66s system >> $ find /usr/src -type f | wc -l >> 114315 >> $ time find /usr/src -type f -exec grep foo {} + > /dev/null >> 0m14.07s real 0m07.68s user 0m06.29s system >> $ time find /usr/src -type f -print0 | xargs -0 grep foo > /dev/null >> 0m13.83s real 0m07.94s user 0m06.25s system > > I'm not sure how many matches for 'foo' are there inside your > /usr/src, but if there are a lot, it slows down the last phase (grep > output), making the performance gains a wash. > > For this particular scenario, it's better to search for a string with > no matches. Then you won't need to redirect to /dev/null too. Ok, good to know. Here is some new numbers: $ time find /usr/src -type f -exec grep "DIZ_-{)9064gd" {} + 0m11.74s real 0m05.54s user 0m06.14s system $ time find /usr/src -type f -print0 | xargs -0 grep "DIZ_-{)9064gd" 0m11.70s real 0m05.59s user 0m06.50s system -- Manuel Giraud From debbugs-submit-bounces@debbugs.gnu.org Wed May 22 15:59:49 2024 Received: (at 71094) by debbugs.gnu.org; 22 May 2024 19:59:49 +0000 Received: from localhost ([127.0.0.1]:57939 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9s7p-0001Eu-JJ for submit@debbugs.gnu.org; Wed, 22 May 2024 15:59:49 -0400 Received: from wfhigh3-smtp.messagingengine.com ([64.147.123.154]:36787) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9s7m-0001Eo-4s for 71094@debbugs.gnu.org; Wed, 22 May 2024 15:59:48 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.west.internal (Postfix) with ESMTP id 1A2D5180012F; Wed, 22 May 2024 15:59:33 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 22 May 2024 15:59:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716407972; x=1716494372; bh=B5tieLhpcUGAnIQFAhq8v1rqMjH4NaCo69Mjf0aOLBQ=; b= JOItM4Y7569WXEVdC5WewEHjwq8sITMIeK45QIjOo4IwNh0wzP6ybVAZp6TJQf9B NnyLu98FCrN6rMqV/0sehz//eR5r5aj+UumFAb1mVL+L/7ZIkT0svuujo05r6+YX Pm6mGPtKTZqpy0IlJ2anseEc4dp6JhS2I5tuLaXGjbEvEfaelmJ8LmgxoDrrl0SJ vUOcKbPl+WQRHRBWsAMUL8L1PjTi4QDDXgBGOwaLbb3wCGVp24n0+asw66wCT344 znONqyD4IsbKDLy0vAJxQJ7kKAd2Up04cW3vjQSrFjk4z17gvoEVurjchpf1OEEj 1sdcfIc1eoW/cRL7ZNs/CA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716407972; x= 1716494372; bh=B5tieLhpcUGAnIQFAhq8v1rqMjH4NaCo69Mjf0aOLBQ=; b=f DQBobLmq/zeM2pXGaKGywHVYqzLKI08Eypm5EGwk6BCPyX4hKqERPCV6E3j1/tIv dj0wVCBQtqLry5CUBB6Lpk7Y46zOL8sAm/HErmyUthFIpQa+MCu3W9glZXQiEuXq +69sSycaLb43165RxA2Y01gGnfVF5z2O4TYGYADiGxF2xQujFjnXiYPEY72zOAF2 N2pdX2Lz8lj2p4LnuCwcAjTUI7AxIYYKDNFa8OjzygV9MeDLY8H65xGEA5kHzdup wuF8BkHMzVfIu8hXxRRr61eH46pRSUGwx4iUGKRg8Py4AdGDD4S4G6tgZs/DGSvQ y5scAv0cth7+Fo2YPBCZA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeigedgiedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 22 May 2024 15:59:30 -0400 (EDT) Message-ID: <29cb936b-9e1f-4cac-b125-4699eaabac89@gutov.dev> Date: Wed, 22 May 2024 22:59:28 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Manuel Giraud References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> <87h6epbpvv.fsf@ledu-giraud.fr> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <87h6epbpvv.fsf@ledu-giraud.fr> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, Eli Zaretskii , 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 22/05/2024 22:36, Manuel Giraud wrote: > Ok, good to know. Here is some new numbers: > > $ time find /usr/src -type f -exec grep "DIZ_-{)9064gd" {} + > 0m11.74s real 0m05.54s user 0m06.14s system > $ time find /usr/src -type f -print0 | xargs -0 grep "DIZ_-{)9064gd" > 0m11.70s real 0m05.59s user 0m06.50s system Looks about the same. I think that helps, thank you. From debbugs-submit-bounces@debbugs.gnu.org Thu May 23 00:48:45 2024 Received: (at 71094) by debbugs.gnu.org; 23 May 2024 04:48:45 +0000 Received: from localhost ([127.0.0.1]:58348 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sA0Nh-00061J-3y for submit@debbugs.gnu.org; Thu, 23 May 2024 00:48:45 -0400 Received: from eggs.gnu.org ([209.51.188.92]:54512) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sA0Ne-00061D-H4 for 71094@debbugs.gnu.org; Thu, 23 May 2024 00:48:43 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sA0LN-0004tJ-2a; Thu, 23 May 2024 00:46:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=u3Y6QkgrsSD8D0uBstPdrbn4guLUkK5UrgEgb8Es7sU=; b=i421l+NqMzlk 59c7GjVN97kwqfA9QYwF/C5W++nnbt+A+llqzqXNQAoxZGlBKqyoXxgrRH9oIgVl4Wz1yH4l4Y5KO g1R2ZQkXu/73nr3wkhDOdaBes0T+BZrT/ugxkOZBa///NG7L3purkcGb8rTr1bEOP+WUDvVxUtYKb b7ech8hcwGCNkyeg6nefDBHLHQXYJWjcVCN5u0yeT9NCMfCpSHcPXSsyjxjr4l0TlmHrnV01D6X8g dl+8njHRd1pDbg9hBtWWNwhP5KdsQ0f+7Inc3EtMa+0YCOtztMBoJHYx1URhuY94Lhy9w4EvOyV0h 9hJkO80pzwAieQaE/Bh8Ww==; Date: Thu, 23 May 2024 07:46:12 +0300 Message-Id: <86o78x5e5n.fsf@gnu.org> From: Eli Zaretskii To: Manuel Giraud In-Reply-To: <87le41bqvd.fsf@ledu-giraud.fr> (message from Manuel Giraud on Wed, 22 May 2024 21:15:02 +0200) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> <86r0dt66nw.fsf@gnu.org> <87le41bqvd.fsf@ledu-giraud.fr> X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: dmitry@gutov.dev, 71094@debbugs.gnu.org, rgm@gnu.org, sbaugh@janestreet.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > From: Manuel Giraud > Cc: dmitry@gutov.dev, sbaugh@janestreet.com, 71094@debbugs.gnu.org, > rgm@gnu.org > Date: Wed, 22 May 2024 21:15:02 +0200 > > Eli Zaretskii writes: > > [...] > > > Thanks, but we need the timings of the corresponding Emacs commands, > > not the commands run from the shell prompt. > > Ok. What are those commands and how to timed them? It's rgrep, AFAIU, according to the original report in this bug's discussion. Dmitry, would you please show Manual the commands you were running in your benchmarks? > > Btw, are you sure that xargs or grep don't pay attention to the fact > > that their output is redirected to the null device, and do nothing? > > Some variants of these commands are known to use such a trick, AFAIR. > > I don't know. OK, thanks. From debbugs-submit-bounces@debbugs.gnu.org Thu May 23 09:24:33 2024 Received: (at 71094) by debbugs.gnu.org; 23 May 2024 13:24:33 +0000 Received: from localhost ([127.0.0.1]:59060 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sA8Qq-0002cV-JZ for submit@debbugs.gnu.org; Thu, 23 May 2024 09:24:32 -0400 Received: from fout7-smtp.messagingengine.com ([103.168.172.150]:39531) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sA8Ql-0002cN-A3 for 71094@debbugs.gnu.org; Thu, 23 May 2024 09:24:30 -0400 Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfout.nyi.internal (Postfix) with ESMTP id 725EA13803AA; Thu, 23 May 2024 09:24:15 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Thu, 23 May 2024 09:24:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716470655; x=1716557055; bh=dZPrJ80ij9yBP++9vieifQU0SdXUcQsZpXEzLQPXfHM=; b= FxmY6dfYaEhqYQv5aBMREY46WhMeTBloYqGhE7V9HNhRaK12IdVHMwUi9udGtfXJ dMaSfgQORneHw9OOUiHEmhGXZ1ms5zJG8djrw2u1bO3TA1jcWPOQynUuYA2bKl4y ydmic4yK7c66tBy0NAKmY2bz6Rcb3KyEbXW2k2ZDsGomaLgMdwGOfvy9Z4/Uaseg Xs668XBuWaS+vlPIYHO3F8NPFrUjhL9E3urgzV6WI7oIehZDLF7Iau/hB+JYuxFo EfDm8a/x4h+BAC210KmXuT9Q0KjajtftrE4sDsgu1saN2J7JNfQESt4JQS+pI1/f 7FVniLUixbJb0CETpswBkA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716470655; x= 1716557055; bh=dZPrJ80ij9yBP++9vieifQU0SdXUcQsZpXEzLQPXfHM=; b=j Hm9RqKITRK4lFq1EfHOHmxXQf+vpn3c4RM4SAFo0121h+zhrzRHiBTGNwG7YXAcS uslmGjmZP1BL0yZB/Yo1zz+E5APyuZjFzkgx6z0VbtdbCZ3rwoIMqYhEB3/zfLes DsSxWCc4vBAZhlpNI1WA5vl2O4tesMdGdPO0ktZiiEtCAEzBRhwpXFIVaLzLTVeL PYK/sajkuF+KBflWkX7MQ8Om9ToOvnXE5zL8ZZmLK0qXKpHL3Y0+hBOZ34KYH4uv 0y2cVPUG4izrey8PqF4eMehZkhY0wYQRBefdnv6nke75f1B2PyNT0+4YJzjw9aP/ gASDnYVZXmLlkXV6YFnVg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeiiedgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 23 May 2024 09:24:13 -0400 (EDT) Message-ID: <149bf36e-831a-47ac-a424-7a3008462561@gutov.dev> Date: Thu, 23 May 2024 16:24:11 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Eli Zaretskii , Manuel Giraud References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> <86r0dt66nw.fsf@gnu.org> <87le41bqvd.fsf@ledu-giraud.fr> <86o78x5e5n.fsf@gnu.org> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <86o78x5e5n.fsf@gnu.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 23/05/2024 07:46, Eli Zaretskii wrote: >> From: Manuel Giraud >> Cc:dmitry@gutov.dev,sbaugh@janestreet.com,71094@debbugs.gnu.org, >> rgm@gnu.org >> Date: Wed, 22 May 2024 21:15:02 +0200 >> >> Eli Zaretskii writes: >> >> [...] >> >>> Thanks, but we need the timings of the corresponding Emacs commands, >>> not the commands run from the shell prompt. >> Ok. What are those commands and how to timed them? > It's rgrep, AFAIU, according to the original report in this bug's > discussion. Dmitry, would you please show Manual the commands you > were running in your benchmarks? 1. Visit a directory with a fair number of files. 2. M-x rgrep, enter some odd regexp like "asdfasf@!#!" and "*" for the files wildcard. 3. Perform the search. Look at the end of the *grep* output, it will say something like "duration: 4.52 s". Note the number. And you could repeat the same after applying the patch, recompiling Emacs (or at least grep.el) and restarting. Preferably do the scenario 2-3 times to ensure that the filesystem cache is warm, and cold file access speed doesn't skew the numbers. From debbugs-submit-bounces@debbugs.gnu.org Fri May 24 13:45:07 2024 Received: (at 71094) by debbugs.gnu.org; 24 May 2024 17:45:07 +0000 Received: from localhost ([127.0.0.1]:38278 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sAYyY-00024C-GG for submit@debbugs.gnu.org; Fri, 24 May 2024 13:45:07 -0400 Received: from ledu-giraud.fr ([51.159.28.247]:48058) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sAYyJ-00023U-T6 for 71094@debbugs.gnu.org; Fri, 24 May 2024 13:45:05 -0400 DKIM-Signature: v=1; a=ed25519-sha256; c=simple/simple; s=ed25519; bh=HZ/VZjhK RWThAJ3fnl8s3nlo1F3yYH/8hzXtw1V+QeA=; h=date:references:in-reply-to: subject:cc:to:from; d=ledu-giraud.fr; b=Oj/+HtKhMkhBXc0WIwMgI7z7bPcnMT /KLCsJfNAI5YZ5s8Sy+Nw0cpC5ROimvM4uFqSyoLKNNh4a+QLHK9XsBQ== DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=rsa; bh=HZ/VZjhKRWThAJ3f nl8s3nlo1F3yYH/8hzXtw1V+QeA=; h=date:references:in-reply-to:subject: cc:to:from; d=ledu-giraud.fr; b=rlGjhGwhYWrUShEvH62uKU8uuv4az0i889tRZD /Eza6WTAfW2on6RIK6m9jun/ZGwomE0wXLtDGoJAECAggxOV+kTvgpy1J5+exksIsf1WoR Uz5F+e1mEypkFP5fBHBjFuH/zr4A0cQU3OCGS8xy0Ittb/HLRHmr1C7oOSFKS1RoEJMGaF LAXZIow+kc0kOS07gSrwF8yluwO4D3vCFygCff0QXqYXao5cNoun/4+WP1gN2GLj8fUM8M ljxrDx8ioise5R3GshLXLtVomUw9Xs8fsdZzBtHTzqs8iqSLOcj5KK/J5fE204j6yZSBdf jqeq4DpgF7qD6v7QY5oprsUg== Received: from computer ( [10.1.1.1]) by ledu-giraud.fr (OpenSMTPD) with ESMTPSA id 2e8edcd2 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 24 May 2024 19:44:42 +0200 (CEST) From: Manuel Giraud To: Dmitry Gutov Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: <149bf36e-831a-47ac-a424-7a3008462561@gutov.dev> (Dmitry Gutov's message of "Thu, 23 May 2024 16:24:11 +0300") References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> <86r0dt66nw.fsf@gnu.org> <87le41bqvd.fsf@ledu-giraud.fr> <86o78x5e5n.fsf@gnu.org> <149bf36e-831a-47ac-a424-7a3008462561@gutov.dev> Date: Fri, 24 May 2024 19:44:41 +0200 Message-ID: <87bk4vyuie.fsf@ledu-giraud.fr> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, Eli Zaretskii , 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) Dmitry Gutov writes: > On 23/05/2024 07:46, Eli Zaretskii wrote: >>> From: Manuel Giraud >>> Cc:dmitry@gutov.dev,sbaugh@janestreet.com,71094@debbugs.gnu.org, >>> rgm@gnu.org >>> Date: Wed, 22 May 2024 21:15:02 +0200 >>> >>> Eli Zaretskii writes: >>> >>> [...] >>> >>>> Thanks, but we need the timings of the corresponding Emacs commands, >>>> not the commands run from the shell prompt. >>> Ok. What are those commands and how to timed them? >> It's rgrep, AFAIU, according to the original report in this bug's >> discussion. Dmitry, would you please show Manual the commands you >> were running in your benchmarks? > > 1. Visit a directory with a fair number of files. > 2. M-x rgrep, enter some odd regexp like "asdfasf@!#!" and "*" for the > files wildcard. > 3. Perform the search. Look at the end of the *grep* output, it will > say something like "duration: 4.52 s". Note the number. Thanks. Here is what I get first without and then with the patch after cache warming on both runs: --8<---------------cut here---------------start------------->8--- -*- mode: grep; default-directory: "/usr/src/" -*- Grep started at Fri May 24 19:04:44 find -H . -type d \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.src -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o \! -type d \( -name .\#\* -o -name \*.o -o -name \*\~ -o -name \*.bin -o -name \*.lbin -o -name \*.so -o -name \*.a -o -name \*.ln -o -name \*.blg -o -name \*.bbl -o -name \*.elc -o -name \*.lof -o -name \*.glo -o -name \*.idx -o -name \*.lot -o -name \*.fmt -o -name \*.tfm -o -name \*.class -o -name \*.fas -o -name \*.lib -o -name \*.mem -o -name \*.x86f -o -name \*.sparcf -o -name \*.dfsl -o -name \*.pfsl -o -name \*.d64fsl -o -name \*.p64fsl -o -name \*.lx64fsl -o -name \*.lx32fsl -o -name \*.dx64fsl -o -name \*.dx32fsl -o -name \*.fx64fsl -o -name \*.fx32fsl -o -name \*.sx64fsl -o -name \*.sx32fsl -o -name \*.wx64fsl -o -name \*.wx32fsl -o -name \*.fasl -o -name \*.ufsl -o -name \*.fsl -o -name \*.dxl -o -name \*.lo -o -name \*.la -o -name \*.gmo -o -name \*.mo -o -name \*.toc -o -name \*.aux -o -name \*.cp -o -name \*.fn -o -name \*.ky -o -name \*.pg -o -name \*.tp -o -name \*.vr -o -name \*.cps -o -name \*.fns -o -name \*.kys -o -name \*.pgs -o -name \*.tps -o -name \*.vrs -o -name \*.pyc -o -name \*.pyo \) -prune -o -type f \( -name \* -o -name .\* \) -exec grep -i -nH --null -e asdfasf\@\!\#\! \{\} + Grep finished with no matches found at Fri May 24 19:04:54, duration 10.2 s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- -*- mode: grep; default-directory: "/usr/src/" -*- Grep started at Fri May 24 19:37:28 find -H . -type d \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.src -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o \! -type d \( -name .\#\* -o -name \*.o -o -name \*\~ -o -name \*.bin -o -name \*.lbin -o -name \*.so -o -name \*.a -o -name \*.ln -o -name \*.blg -o -name \*.bbl -o -name \*.elc -o -name \*.lof -o -name \*.glo -o -name \*.idx -o -name \*.lot -o -name \*.fmt -o -name \*.tfm -o -name \*.class -o -name \*.fas -o -name \*.lib -o -name \*.mem -o -name \*.x86f -o -name \*.sparcf -o -name \*.dfsl -o -name \*.pfsl -o -name \*.d64fsl -o -name \*.p64fsl -o -name \*.lx64fsl -o -name \*.lx32fsl -o -name \*.dx64fsl -o -name \*.dx32fsl -o -name \*.fx64fsl -o -name \*.fx32fsl -o -name \*.sx64fsl -o -name \*.sx32fsl -o -name \*.wx64fsl -o -name \*.wx32fsl -o -name \*.fasl -o -name \*.ufsl -o -name \*.fsl -o -name \*.dxl -o -name \*.lo -o -name \*.la -o -name \*.gmo -o -name \*.mo -o -name \*.toc -o -name \*.aux -o -name \*.cp -o -name \*.fn -o -name \*.ky -o -name \*.pg -o -name \*.tp -o -name \*.vr -o -name \*.cps -o -name \*.fns -o -name \*.kys -o -name \*.pgs -o -name \*.tps -o -name \*.vrs -o -name \*.pyc -o -name \*.pyo \) -prune -o -type f \( -name \* -o -name .\* \) -print0 | "xargs" -0 grep -i -nH --null -e asdfasf\@\!\#\! Grep finished with no matches found at Fri May 24 19:37:37, duration 9.01 s --8<---------------cut here---------------end--------------->8--- -- Manuel Giraud From debbugs-submit-bounces@debbugs.gnu.org Sun May 26 05:47:53 2024 Received: (at 71094) by debbugs.gnu.org; 26 May 2024 09:47:53 +0000 Received: from localhost ([127.0.0.1]:38419 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sBATp-0007Gb-8X for submit@debbugs.gnu.org; Sun, 26 May 2024 05:47:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:44654) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sBATn-0007GP-LM for 71094@debbugs.gnu.org; Sun, 26 May 2024 05:47:52 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sBATZ-00020W-M7; Sun, 26 May 2024 05:47:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=JObfN9MI8clqkjESsSPo/i5+Vjv9SZBgBPSwhuYLHpE=; b=MZXIYFn1fUb4 kKyqK3at40Hv4CpQo6qBBpt4j0XaG6th7sZbGIyjwqWEJHS73sjFLSfZ26ioPI1ERfWWjW95/3vPi LELTWBKdCPEtIFje2E+m+vLdQBksGPxN+KUKGlaEtqNCNwKYSJ01gBM4iS++eCwhYkx1egMuQCp2S oP7Ss5hW3PLW/UXzydyY9w3IhItdAs3evblkt+yGhnyJmqu2OTp+3HD+lEr5VGwp8we9IuKbi/kb/ E27y+4hBi0CJl0+n78kBPzBHb41e4gbDLnNRuOcY7BigJilw3mdUrO0B1aUNm+NgV3foFIbpR6itk myOZLc6GZ/BzPqyv+oH4Pg==; Date: Sun, 26 May 2024 12:47:32 +0300 Message-Id: <86msocx5u3.fsf@gnu.org> From: Eli Zaretskii To: Spencer Baugh In-Reply-To: (message from Spencer Baugh on Wed, 22 May 2024 08:54:25 -0400) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: <86ttiq6or8.fsf@gnu.org> X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > From: Spencer Baugh > Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev > Date: Wed, 22 May 2024 08:54:25 -0400 > > > In any case, please modify the patch so that 'exec-plus' is still > > preferred on MS-Windows (because most Windows ports of xargs are IME > > abysmally buggy, so better avoided as much as possible). > > > > A comment there with the justification of the order will also be > > appreciated. > > Done, attached. Thanks, LGTM. From debbugs-submit-bounces@debbugs.gnu.org Sun May 26 11:58:16 2024 Received: (at 71094) by debbugs.gnu.org; 26 May 2024 15:58:16 +0000 Received: from localhost ([127.0.0.1]:42323 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sBGGG-0006nw-1E for submit@debbugs.gnu.org; Sun, 26 May 2024 11:58:16 -0400 Received: from wfout8-smtp.messagingengine.com ([64.147.123.151]:56743) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sBGGE-0006nj-Il for 71094@debbugs.gnu.org; Sun, 26 May 2024 11:58:15 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfout.west.internal (Postfix) with ESMTP id 58A431C0010E; Sun, 26 May 2024 11:58:00 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sun, 26 May 2024 11:58:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1716739079; x=1716825479; bh=7nbJX4n/8c8ahuztQ5lUmgJZS45fQKGW4EYPJBtREOM=; b= BAdNcR3yIQcCUogQpSHkhwdcEZVtJHUTEkhUZgWD9/7e9AKFLvgntRGpHDjOS4S4 YsD4lqbdOqKwCEXPwVoieBlPmzvKxNRr5Ji6C+4p+NW7gUtTa8NoLMA0cJPE8/S2 +Z9K1S4ATWj4NqTFkaNUNhiCj43PGVDOpyenZ0AgKyPsidZR1II9pob4K2NWAmAP RDTeqFCZavam0jGqu69n1ebLaHrgasWEj1K8XD1sClLdOSobnDDLQbcdau65V/ER 4ZxY9a1Q3hu+be7xiECtNboIAUPbGE0VnnXX3pwahjEFEz1TE1e1J90S3F/haaAo 9AepCl5h20Vrc4T9jnMzTg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716739079; x= 1716825479; bh=7nbJX4n/8c8ahuztQ5lUmgJZS45fQKGW4EYPJBtREOM=; b=S d2JTQm9puybiOaXUAyZIEpCkYX9SybJpd1kE8OT7kpX0W23Q1fOZyk7IFYPkVnLM XKBv1xihAxBd+p/SrOK2TIKVxFsXwlf8VpkToajb2K+Swr+Zx/NtU+nzDxQX78ZA 9zInlIod9Fj4Sq5rm6a2FnbWT/2+3yOM3y09iP44murZdDBcb6u/A9A8BFTPib+m NIQNOEGltcGOM1+3kbAJvFuUlgMzKKDEi7LSDsCPVGdT2P/L3IBJICpwdr9QPXfn K0EfByQfD3O+N9vaK3ty6x/zArEtQBDbG1+02qqOGUASzpH4YTdXRA/HDLmDdAJ9 p9vy00vWIigm+qZnJK9JQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdejvddgleegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 26 May 2024 11:57:58 -0400 (EDT) Message-ID: <9c6667e4-2b2a-4961-af22-c2c30b395bee@gutov.dev> Date: Sun, 26 May 2024 18:57:55 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Manuel Giraud References: <86ttiq6or8.fsf@gnu.org> <8aedd0ed-58fe-4ac7-98d6-950be2d4700b@gutov.dev> <868r026jlq.fsf@gnu.org> <861q5t7vrp.fsf@gnu.org> <10f62497-dfb1-4c46-b18a-6d1100de4b6a@gutov.dev> <86wmnl6f62.fsf@gnu.org> <87pltdbu17.fsf@ledu-giraud.fr> <86r0dt66nw.fsf@gnu.org> <87le41bqvd.fsf@ledu-giraud.fr> <86o78x5e5n.fsf@gnu.org> <149bf36e-831a-47ac-a424-7a3008462561@gutov.dev> <87bk4vyuie.fsf@ledu-giraud.fr> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <87bk4vyuie.fsf@ledu-giraud.fr> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: sbaugh@janestreet.com, Eli Zaretskii , 71094@debbugs.gnu.org, rgm@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 24/05/2024 20:44, Manuel Giraud wrote: > Thanks. Here is what I get first without and then with the patch after > cache warming on both runs: > > ... > Grep finished with no matches found at Fri May 24 19:04:54, duration > 10.2 s > > ... > Grep finished with no matches found at Fri May 24 19:37:37, duration > 9.01 s That looks like a slight improvement as well, nice. From debbugs-submit-bounces@debbugs.gnu.org Thu May 30 08:29:24 2024 Received: (at 71094) by debbugs.gnu.org; 30 May 2024 12:29:24 +0000 Received: from localhost ([127.0.0.1]:33095 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sCeuJ-0006pQ-MC for submit@debbugs.gnu.org; Thu, 30 May 2024 08:29:23 -0400 Received: from mxout1.mail.janestreet.com ([38.105.200.78]:40049) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sCeuG-0006p6-MW for 71094@debbugs.gnu.org; Thu, 30 May 2024 08:29:21 -0400 From: Spencer Baugh To: Eli Zaretskii Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: <86msocx5u3.fsf@gnu.org> (Eli Zaretskii's message of "Sun, 26 May 2024 12:47:32 +0300") References: <86ttiq6or8.fsf@gnu.org> <86msocx5u3.fsf@gnu.org> Date: Thu, 30 May 2024 08:29:04 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1717072144; bh=/BhkUQ87dmPHqfLoTVyHVzJGjU9OxkJ03XwZfp1HQrg=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=KVgi3ZXZdVMNi6G640LjLJY3K9q1qizVGEv+4KAmlzrz/oky2obYayZ+/Lk/MXHRL 5r/j5EerlrBJ1+o6HGb4OE5890qAk3kXHLO+xsxAdikDHX5N33/2+x/EbV/oS//F8Z DXJp7N1zIRl1r2HEqo2dfl8njE8UQRRfayOYhpFUDRodU6QB8gnYyE/0lBBY6khfI9 gq56ZQYumPFWWTVlksMrSZoMI8lx95F+zT3VfMc1Icw7XhRbr07mp+Go6SNa6bcCFt +iXR6A2ZhgsVgkj4a7NKop0+8Kt01FbTKQYOYID8D+Ypw/HxVNjz5/2WyzDHMbNs1w iB5WxSJp3OHdw== X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) Eli Zaretskii writes: >> From: Spencer Baugh >> Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev >> Date: Wed, 22 May 2024 08:54:25 -0400 >> >> > In any case, please modify the patch so that 'exec-plus' is still >> > preferred on MS-Windows (because most Windows ports of xargs are IME >> > abysmally buggy, so better avoided as much as possible). >> > >> > A comment there with the justification of the order will also be >> > appreciated. >> >> Done, attached. > > Thanks, LGTM. So is this OK to install now? We have had benchmarking on several platforms and they all report either a performance improvement or no change. From debbugs-submit-bounces@debbugs.gnu.org Thu May 30 10:53:04 2024 Received: (at 71094) by debbugs.gnu.org; 30 May 2024 14:53:04 +0000 Received: from localhost ([127.0.0.1]:40834 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sCh9L-0001bE-7Q for submit@debbugs.gnu.org; Thu, 30 May 2024 10:53:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56766) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sCh9I-0001aH-QB for 71094@debbugs.gnu.org; Thu, 30 May 2024 10:53:01 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sCh92-0001si-B7; Thu, 30 May 2024 10:52:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=ZtrX95TsLoft41Kd26+Gflq3SZ78QKNH9sf2sX1orWk=; b=VagqB4Yiboki WhEt9XX5Kw0T4EHCsYO6lPIR61LKV2dtTUSmgzYvTmiAvyD0jDoYnxPwZsw2DPDPKI2UuAnQ22osC iSWHPevWV1D9nVQFwmUSy8CzHhdv5tCIRhN/Rvgvqy/TI6mlwmTd9x8qMkIl+zydJllDoSAVBG3p5 KROYzoGgb43caDrIrUBTd5a27Vr4VlbnTcgsi2TfQgYfDZgD3uWpRdGb1MrlQUYcI6Doaku2Txu/E iJqGHAkRbaYBZKwC+CGJF2nz5umRe/MAgVNUGB0ZmgXC0jVprq10YcQYDfxQ9pnKhxkxVgt5k+3Ui B/BYnR2722WrL7YluW4QNw==; Date: Thu, 30 May 2024 17:52:25 +0300 Message-Id: <86frtzqrme.fsf@gnu.org> From: Eli Zaretskii To: Spencer Baugh In-Reply-To: (message from Spencer Baugh on Thu, 30 May 2024 08:29:04 -0400) Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep References: <86ttiq6or8.fsf@gnu.org> <86msocx5u3.fsf@gnu.org> X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) > From: Spencer Baugh > Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev > Date: Thu, 30 May 2024 08:29:04 -0400 > > Eli Zaretskii writes: > > >> From: Spencer Baugh > >> Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev > >> Date: Wed, 22 May 2024 08:54:25 -0400 > >> > >> > In any case, please modify the patch so that 'exec-plus' is still > >> > preferred on MS-Windows (because most Windows ports of xargs are IME > >> > abysmally buggy, so better avoided as much as possible). > >> > > >> > A comment there with the justification of the order will also be > >> > appreciated. > >> > >> Done, attached. > > > > Thanks, LGTM. > > So is this OK to install now? As far as I'm concerned, yes. I don't know if the other participants of this discussion are okay with this or they still have some comments. From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 28 10:03:10 2024 Received: (at 71094) by debbugs.gnu.org; 28 Jun 2024 14:03:10 +0000 Received: from localhost ([127.0.0.1]:45873 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sNCBy-0004JW-6u for submit@debbugs.gnu.org; Fri, 28 Jun 2024 10:03:10 -0400 Received: from mxout6.mail.janestreet.com ([64.215.233.21]:53967) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sNCBv-0004J6-9H for 71094@debbugs.gnu.org; Fri, 28 Jun 2024 10:03:08 -0400 From: Spencer Baugh To: Eli Zaretskii Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: <86frtzqrme.fsf@gnu.org> (Eli Zaretskii's message of "Thu, 30 May 2024 17:52:25 +0300") References: <86ttiq6or8.fsf@gnu.org> <86msocx5u3.fsf@gnu.org> <86frtzqrme.fsf@gnu.org> Date: Fri, 28 Jun 2024 10:03:01 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1719583381; bh=eSUNR2PFuxP1fLT3w44EhliUu4qxovymz5rllERXQ4Q=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=ZMRvGribGHQ/6Ddx3ndUOV05UtLTCYqAEjB0BTrXr+J9siK5Zc5kjd3/K7Q8c6OtA WP2bTAFAPG0bH00MGCNUf6klRVXbVOtVHgYvq3iKZWiJgYEFrlAquTD66pqyrMWrLA U5RXTogTm2y4ucScxbH/gd5M3RIiDiRVRm2xvyQ6SEFfzEDDfaAx6q/bspttgYfoaW tjqPIoewaR58+GbSCA9wareC76TtwSA/K8RwRvC4pih0hHBlUMZWbF2tNNl3wq2S// Grmm65yDAoZdx7WwAw37e/GlNujJpDOL7rbeaBuQ2hnkxOH9KK6AoFsqg4kashURcf pCxIMUoB1/ILw== X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: >> From: Spencer Baugh >> Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev >> Date: Thu, 30 May 2024 08:29:04 -0400 >> >> Eli Zaretskii writes: >> >> >> From: Spencer Baugh >> >> Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev >> >> Date: Wed, 22 May 2024 08:54:25 -0400 >> >> >> >> > In any case, please modify the patch so that 'exec-plus' is still >> >> > preferred on MS-Windows (because most Windows ports of xargs are IME >> >> > abysmally buggy, so better avoided as much as possible). >> >> > >> >> > A comment there with the justification of the order will also be >> >> > appreciated. >> >> >> >> Done, attached. >> > >> > Thanks, LGTM. >> >> So is this OK to install now? > > As far as I'm concerned, yes. I don't know if the other participants > of this discussion are okay with this or they still have some > comments. Since no-one else has made any comments, could this be installed now? From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 30 01:08:39 2024 Received: (at 71094) by debbugs.gnu.org; 30 Jun 2024 05:08:40 +0000 Received: from localhost ([127.0.0.1]:54418 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sNmnn-0002uS-J7 for submit@debbugs.gnu.org; Sun, 30 Jun 2024 01:08:39 -0400 Received: from mail-ed1-f50.google.com ([209.85.208.50]:42266) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sNmnk-0002u9-PR for 71094@debbugs.gnu.org; Sun, 30 Jun 2024 01:08:37 -0400 Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-57cbc66a0a6so1903193a12.1 for <71094@debbugs.gnu.org>; Sat, 29 Jun 2024 22:08:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719724051; x=1720328851; darn=debbugs.gnu.org; h=cc:to:subject:message-id:date:mime-version:references:in-reply-to :from:from:to:cc:subject:date:message-id:reply-to; bh=pW+zDBuXiWp8TqgnyjcG8CRJ6DWHJJI5ac/okEWB1YI=; b=FjLiAgrkigvd37crkTEXqyxM2WZrNegC4QWfc2Lffm/s5u+axOYJFPw3mja+nf2jHO wvsWqzOE2CzoX7/vUJu9cEHti+eOKHllspWuF/zFK0BvyaWg9KWDvv6hMNw5ESBxmETS zuRgcyGz+jpy5QZTY1FQqkqBLoVfrxa+00jXqQuZF6z7VvYVaE3BvulsOSWqnZoieTRD UNJAwpKIIKbL9IXaRMcWvEBhm3BN1H5XUW+p8ENWGUou34ch09RGsFq1s9PsC5dPRbeZ ifgYxDB50pXOpLo+KEoeX7kyZ546o937Ar68w4Xb3FCugQ8yukdfKPEB/PS5MqLciG5e Z8KA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719724051; x=1720328851; h=cc:to:subject:message-id:date:mime-version:references:in-reply-to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pW+zDBuXiWp8TqgnyjcG8CRJ6DWHJJI5ac/okEWB1YI=; b=f8C+ATbinFcnvmlGExvan+wXXZuNMLa2JOQemvWyYHp9satuTHYG3FFvOav1KRyn4m TwUvAud5R6ROA4kn9F3Uw+pkcMJa582d0TjUmCpwgrzkjReEdD0wj3KS2R/2nPSNIhSR t/9BKGxNUjCOjQlO/OIJyZZgscHWSbYj7ScqkHgjzTfKG1V37bGDuzrNIk0Cjv6MDixC A296e2MdBwcSI03RgumsCP2butRWpQTYew573lNEJI7JEYyEu0SXTVLJp9XwfK3Ulthl eZVvtevHFKiAJHZWjsjICvMtMdX9hT1xBnA6GlPQNrYX1trtzfXdhsnWuaYKbLS4XX0i w+/A== X-Forwarded-Encrypted: i=1; AJvYcCXOA/SUs7esB0S1jhXAt8rPx9GaVBvs+b4CklfyvmPMIaSoQu9PH02yipdhPoyQrKdcon1lgF1Hk3uKNgxaUJvIUTLBJAU= X-Gm-Message-State: AOJu0YyatstilaBD0RssJciTB6If13zj5EqMbiFYXtc693wNPxSeBmXn EhwEs0WHOpqP2erBGg1pobxos5IsSHu5zXnNUlcUr/4pxYLFhVYLQ3hpXr/Cf52sgMMK5DQUJb/ EfU+Iy2lwXRoGA27SQbfkLdf7H0E= X-Google-Smtp-Source: AGHT+IE6d0sysN48QPnFv/7/K7k8BXOFAxaTfnWs+cmWki9fL8DNupQHcbAl2D30cGYvN7KzGbkfY86420ScLuqif40= X-Received: by 2002:a05:6402:2547:b0:57c:c171:2fb6 with SMTP id 4fb4d7f45d1cf-58780068f5emr1970831a12.1.1719724050771; Sat, 29 Jun 2024 22:07:30 -0700 (PDT) Received: from 753933720722 named unknown by gmailapi.google.com with HTTPREST; Sat, 29 Jun 2024 22:07:30 -0700 From: Stefan Kangas In-Reply-To: References: <86ttiq6or8.fsf@gnu.org> <86msocx5u3.fsf@gnu.org> <86frtzqrme.fsf@gnu.org> MIME-Version: 1.0 Date: Sat, 29 Jun 2024 22:07:30 -0700 Message-ID: Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep To: Spencer Baugh , Eli Zaretskii Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Spencer Baugh writes: > Eli Zaretskii writes: >>> From: Spencer Baugh >>> Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev >>> Date: Thu, 30 May 2024 08:29:04 -0400 >>> >>> Eli Zaretskii writes: >>> >>> > Thanks, LGTM. >>> >>> So is this OK to install now? >> >> As far as I'm concerned, yes. I don't know if the other participants >> of this discussion are okay with this or they still have some >> comments. > > Since no-one else has made any comments, could this be installed now? Yes, please go ahead, and thanks. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 03 08:53:54 2024 Received: (at 71094) by debbugs.gnu.org; 3 Jul 2024 12:53:54 +0000 Received: from localhost ([127.0.0.1]:39131 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sOzUg-0008Hc-D8 for submit@debbugs.gnu.org; Wed, 03 Jul 2024 08:53:54 -0400 Received: from mxout6.mail.janestreet.com ([64.215.233.21]:57325) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sOzUe-0008HK-LQ for 71094@debbugs.gnu.org; Wed, 03 Jul 2024 08:53:53 -0400 From: Spencer Baugh To: Stefan Kangas Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: (Stefan Kangas's message of "Sat, 29 Jun 2024 22:07:30 -0700") References: <86ttiq6or8.fsf@gnu.org> <86msocx5u3.fsf@gnu.org> <86frtzqrme.fsf@gnu.org> Date: Wed, 03 Jul 2024 08:53:45 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1720011225; bh=iOi1y5dzta3CWcTAEEr/XPpcBjXl+adycztc+dxSh04=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=dAbFxhDw6BNn8aviVxA4BA9m00eh9zCi4wE/SPa8BTLNXTJy9s8pGP/oV2zdasLNo 32WAdLNv9LWolMka8C8/85vVBa++lKgaxw5cijAF1/0P7DLkf0zaBMbzD1hG3Th334 7asMoyIw/0DeCf5EPby1nNxFOHGU+D7U1i5AdsKxZI7E153qbhIXIiJlk8sCbxOyjR uomx6E3C+rIVJU8Vd/kyL2wgjKyHjhGCh7R8rRc4Qo58Orgf+rmrdWMsidIox1e5WW tci3X5w+5bAuR94YaTMpQ+0ac+RiqUHtUJ40ClH+iN6f9oELmQ3q28SSmeQ3vv9hjy Mx8N/NKZu3p1w== X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 71094 Cc: rgm@gnu.org, Eli Zaretskii , 71094@debbugs.gnu.org, dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Stefan Kangas writes: > Spencer Baugh writes: > >> Eli Zaretskii writes: >>>> From: Spencer Baugh >>>> Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev >>>> Date: Thu, 30 May 2024 08:29:04 -0400 >>>> >>>> Eli Zaretskii writes: >>>> >>>> > Thanks, LGTM. >>>> >>>> So is this OK to install now? >>> >>> As far as I'm concerned, yes. I don't know if the other participants >>> of this discussion are okay with this or they still have some >>> comments. >> >> Since no-one else has made any comments, could this be installed now? > > Yes, please go ahead, and thanks. I don't actually have commit access (AFAIK?) so someone else will need to push it :) From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 03 09:45:34 2024 Received: (at 71094-done) by debbugs.gnu.org; 3 Jul 2024 13:45:34 +0000 Received: from localhost ([127.0.0.1]:39221 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sP0If-0001Nu-0e for submit@debbugs.gnu.org; Wed, 03 Jul 2024 09:45:33 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38810) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sP0IP-0001NL-Sy for 71094-done@debbugs.gnu.org; Wed, 03 Jul 2024 09:45:31 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sP0GB-000581-F9; Wed, 03 Jul 2024 09:42:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=vXjQZnzj2Q5DGH6DvsxdHKRB7+Qv2yKBTKWEHcDm8oo=; b=IAX72N6y8lK6K4vjtRkg mKqAyt/+uwnIkyQj3Ld0IJpj1TMAN7lg4NswzVcE+qRrfxHLdf6zPL4gQyuYbVPX2w3SfBD/Rg6eR AqhUxFb8szocqJSB0wroFAHpwzQrYMcrf46SAQs7KOW6s9iGLn47L3S7a4XfLckSIBmPkz9tQ/WK+ u1uPNcmHl9y/d24o99iYdfM0doodje/Nl3NVP0En98X50GlqLSb7nTbkNmzhPYI4H8XXVcf8EzLTi 43TmWtYJUx+KKLBOkoZa06K7gqUbUrjJOjCWaSNXsqOXvwK+oSLJxLhHwUO2J2pI2sZSRALrlO2Ap C5eM3jbutERtuQ==; Received: from acorallo by fencepost.gnu.org with local (Exim 4.90_1) (envelope-from ) id 1sP0G4-0007a9-IP; Wed, 03 Jul 2024 09:42:55 -0400 From: Andrea Corallo To: Spencer Baugh Subject: Re: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep In-Reply-To: (Spencer Baugh's message of "Wed, 03 Jul 2024 08:53:45 -0400") References: <86ttiq6or8.fsf@gnu.org> <86msocx5u3.fsf@gnu.org> <86frtzqrme.fsf@gnu.org> Date: Wed, 03 Jul 2024 09:42:52 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 71094-done Cc: rgm@gnu.org, Eli Zaretskii , 71094-done@debbugs.gnu.org, Stefan Kangas , dmitry@gutov.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Spencer Baugh writes: > Stefan Kangas writes: >> Spencer Baugh writes: >> >>> Eli Zaretskii writes: >>>>> From: Spencer Baugh >>>>> Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev >>>>> Date: Thu, 30 May 2024 08:29:04 -0400 >>>>> >>>>> Eli Zaretskii writes: >>>>> >>>>> > Thanks, LGTM. >>>>> >>>>> So is this OK to install now? >>>> >>>> As far as I'm concerned, yes. I don't know if the other participants >>>> of this discussion are okay with this or they still have some >>>> comments. >>> >>> Since no-one else has made any comments, could this be installed now? >> >> Yes, please go ahead, and thanks. > > I don't actually have commit access (AFAIK?) so someone else will need > to push it :) Should be done, please double check. Closing meanwhile. Andrea From unknown Mon Jun 23 07:49:20 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 01 Aug 2024 11:24:09 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator