Subject: bug#66806: closed (Re: bug#66806: 30.0.50; [PATCH] 'project-find-regexp' passes Git submodules to the search program)

From: [email-masked] (GNU bug Tracking System)
Date: Mon, 30 Oct 2023 02:02:02 +0000

Your bug report

#66806: 30.0.50; [PATCH] 'project-find-regexp' passes Git submodules to the search program

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
66806: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=66806
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#66806: 30.0.50; [PATCH] 'project-find-regexp' passes Git submodules to the search program
From: Dmitry Gutov <[email-masked]>
To: Jim Porter <[email-masked]>, [email-masked]
Date: Mon, 30 Oct 2023 04:00:51 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Version: 30.1

On 30/10/2023 02:58, Jim Porter wrote:
> On 10/29/2023 2:41 PM, Dmitry Gutov wrote:
>> And that's not to mention usage over Tramp (which would be affected by 
>> the +1 process call that you mentioned as well, but that seems 
>> unavoidable).
> 
> Yeah, I don't see a way around that, unless we constructed a complex 
> command to run via Tramp that does it all in one go.
> 
> (I looked into using the "--stage" argument for "git ls-files", which 
> gets most of the way to fixing this, but you could break that logic with 
> evil file names not in the tree...)

Filtering out gitlinks based on the mode bits? Clever.

Would probably slow down the parsing of the output, though. Not sure by 
how much.

>> Anyway, after recent experience micro-optimizing list operations, I 
>> came up with this version where the impact seems minimal.
>>
>> WDYT?
> 
> Thanks, that helped form the basis for the attached patch. It moves the 
> 'member' call into the lambda for the original 'mapcar', and then we can 
> just 'delq' all the nil elements. Benchmarking against the Emacs repo, 
> this still seems to have a minimal perf impact.

Ah, nice. delq is pretty handy to abbreviate the relinking.

We could combine both steps to eliminate an extra list altogether, but 
that only yields a further 2-3% improvement. Maybe later.

> There might also be a benefit to using "git ls-files 
> --recurse-submodules" when we can (i.e. when not using "-o"), but maybe 
> we can leave that optimization for another day.

As long as it doesn't result in too much code duplication.

Anyway, I've pushed your latest patch. Thanks! And closing.

Subject: 30.0.50; [PATCH] 'project-find-regexp' passes Git submodules to the search program
From: Jim Porter <[email-masked]>
To: [email-masked]
Date: Sat, 28 Oct 2023 22:36:07 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

X-Debbugs-Cc: [email-masked]

To see this in action, you can do the following, starting from "emacs 
-Q" inside of a Git repo that contains submodules:

   M-x trace-function RET project-files RET
   C-x p g some-string RET

If you look at the trace, you'll see that the files in your submodules 
are returned from 'project-files', but so is the submodule directory. 
That's not really correct, since 'project-files' is supposed to return 
*files*, not directories. (There are already workarounds for this in 
'project-search' and 'project-query-replace-regexp'.)

By default, this is just a theoretical problem, but if you customize 
'xref-search-program-alist' and 'xref-search-program' to include some 
other program, this can cause real issues. For example, I tried to add 
"ag" to this list[1], and unfortunately, it just doesn't work in this 
case. The results for submodules get duplicated, and there's no way I 
can see with ag to search only the specified *files*, ignoring any 
directories. (Looking at the definition for ripgrep, I'm guessing the 
"-g '!*/'" is the trick for that program, but nothing similar works for ag.)

Attached are two possible patches for this: a minimal version that just 
fixes 'project-find-regexp', and a maximal version that fixes this in 
general, and should theoretically speed up 'project-search' and 
'project-query-replace-regexp' since they no longer need to call 
'file-regular-p' on every file.

Do either of these patches look ok?

[1] This is a long story, simplified for this message since the gory 
details aren't especially relevant.

From 9f9fc6a606bdd1610d440d9b130aaa30aa66597a Mon Sep 17 00:00:00 2001
From: Jim Porter <[email-masked]>
Date: Sat, 28 Oct 2023 22:11:30 -0700
Subject: [PATCH] Exclude Git submodules when getting list of files for
 'project-find-regexp'

* lisp/progmodes/project.el (project-find-regexp): Filter out
non-regular files.
---
 lisp/progmodes/project.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index fda1081eb62..e245e794318 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -960,7 +960,8 @@ project-find-regexp
          (default-directory (project-root pr))
          (files
           (if (not current-prefix-arg)
-              (project-files pr)
+              ;; XXX: See the comment in project-query-replace-regexp.
+              (cl-delete-if-not #'file-regular-p (project-files pr))
             (let ((dir (read-directory-name "Base directory: "
                                             caller-dir nil t)))
               (project--files-in-directory dir
-- 
2.25.1

From fdb7a99dcd47e3a11057fb47221a5e6f40b2db6a Mon Sep 17 00:00:00 2001
From: Jim Porter <[email-masked]>
Date: Sat, 28 Oct 2023 22:20:41 -0700
Subject: [PATCH] Exclude Git submodules from 'project-files'

* lisp/progmodes/project.el (project--vc-list-files): Exclude Git
submodules.
(project-search, project-query-replace-regexp): Remove now-unneeded
workaround.
---
 lisp/progmodes/project.el | 20 +++++++++-----------
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index fda1081eb62..92b3ed52600 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -647,6 +647,7 @@ project--vc-list-files
             (include-untracked (project--value-in-dir
                                 'project-vc-include-untracked
                                 dir))
+            (submodules (project--git-submodules))
             files)
        (setq args (append args
                           '("-c" "--exclude-standard")
@@ -680,13 +681,14 @@ project--vc-list-files
        (setq files
              (mapcar
               (lambda (file) (concat default-directory file))
-              (split-string
-               (apply #'vc-git--run-command-string nil "ls-files" args)
-               "\0" t)))
+              (cl-set-difference
+               (split-string
+                (apply #'vc-git--run-command-string nil "ls-files" args)
+                "\0" t)
+               submodules :test #'string=)))
        (when (project--vc-merge-submodules-p default-directory)
          ;; Unfortunately, 'ls-files --recurse-submodules' conflicts with '-o'.
-         (let* ((submodules (project--git-submodules))
-                (sub-files
+         (let ((sub-files
                  (mapcar
                   (lambda (module)
                     (when (file-directory-p module)
@@ -1326,8 +1328,7 @@ project-search
   (interactive "sSearch (regexp): ")
   (fileloop-initialize-search
    regexp
-   ;; XXX: See the comment in project-query-replace-regexp.
-   (cl-delete-if-not #'file-regular-p (project-files (project-current t)))
+   (project-files (project-current t))
    'default)
   (fileloop-continue))
 
@@ -1348,10 +1349,7 @@ project-query-replace-regexp
        (list from to))))
   (fileloop-initialize-replace
    from to
-   ;; XXX: Filter out Git submodules, which are not regular files.
-   ;; `project-files' can return those, which is arguably suboptimal,
-   ;; but removing them eagerly has performance cost.
-   (cl-delete-if-not #'file-regular-p (project-files (project-current t)))
+   (project-files (project-current t))
    'default)
   (fileloop-continue))
 
-- 
2.25.1