Subject: bug#56442: closed (Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits)

From: [email-masked] (GNU bug Tracking System)
Date: Thu, 14 Jul 2022 18:51:02 +0000

Your bug report

#56442: gnus-search-run-search: Hits notmuch command line length limits

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
56442: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56442
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits
From: Sean Whitton <[email-masked]>
To: [email-masked], Eric Abrahamsen <[email-masked]>
Date: Thu, 14 Jul 2022 11:50:13 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

Hello,

On Sat 09 Jul 2022 at 11:01AM -07, Sean Whitton wrote:

> On Sat 09 Jul 2022 at 09:11AM -07, Eric Abrahamsen wrote:
>
>> Anyway I'll give this a test this weekend. It would be *very* nice if
>> we could use this syntax rather than the ugly home-grown one.
>
> Let me know the results of your test, and if it's also continuing to
> work for me by then, I'll install the change.  Thanks for taking a look.

I've been testing it with a variety of complex queries for a week now,
and you indicated you were happy with making a change like this, so I've
gone ahead and pushed to master.

-- 
Sean Whitton

Subject: gnus-search-run-search: Hits notmuch command line length limits
From: Sean Whitton <[email-masked]>
To: [email-masked]
Date: Thu, 07 Jul 2022 09:56:05 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

Hello,

I'm running a pretty innocent notmuch query over a fairly small Maildir:

"((List:debian-devel.lists.debian.org) or ... or
(List:debian-haskell.lists.debian.org) or
(List:debconf-discuss.lists.debian.org)) and (not path:annex/**)"

but gnus-search-run-search fails to return any results.  The reason is
that Gnus first runs the query with --output=threads to obtain a list of
thread ids, and then runs another query with --output=files and a query
constructed from the output of the first query: "thread:000000000000d9d0
or thread:000000000000d9e0 or thread:000000000000d268 or ..."

The resulting command fails completely:

    emacs: /usr/bin/notmuch: Argument list too long

Instead of running two searches like this, we can just surround the
whole query like this: "thread:{QUERY}".  The manual says it's exactly
equivalent:

    ... the user should think of the query thread:{<something>} as
    expanding to all of the thread IDs which match <something>; not‐
    much then performs a second search using the expanded query.

This should be faster, too, with only running a single external command.
Here is the patch I'm thinking I'll apply, if anyone has comments.

-- 
Sean Whitton
>From d62398cc27abd317d31d8f07e151a82081444217 Mon Sep 17 00:00:00 2001
From: Sean Whitton <[email-masked]>
Date: Thu, 7 Jul 2022 09:34:04 -0700
Subject: [PATCH] Use notmuch thread:{} operator instead of performing two
 searches

This also avoids some command line length limit problems.

* gnus-search.el
(gnus-search-run-search (engine gnus-search-notmuch)): Delete method.
(gnus-search-indexed-search-command (engine gnus-search-notmuch)):
When searching for threads, wrap whole query in thread:{} operator.
Always use --output=files, never --output=threads.
---
 lisp/gnus/gnus-search.el | 46 +++++-----------------------------------
 1 file changed, 5 insertions(+), 41 deletions(-)

diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index 369df81d9b..53b6d1b4c6 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1672,43 +1672,6 @@ gnus-search-transform-expression
       (format "date:%s.." (notmuch-date (cdr expr))))
      (t (ignore-errors (cl-call-next-method))))))
 
-(cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch)
-					      server query groups)
-  "Handle notmuch's thread-search routine."
-  ;; Notmuch allows for searching threads, but only using its own
-  ;; thread ids.  That means a thread search is a \"double-bounce\":
-  ;; once to find the relevant thread ids, and again to find the
-  ;; actual messages.  This method performs the first \"bounce\".
-  (if (alist-get 'thread query)
-      (with-slots (program proc-buffer) engine
-	(let* ((qstring
-		(gnus-search-make-query-string engine query))
-	       (cp-list (gnus-search-indexed-search-command
-			 engine qstring query groups))
-	       thread-ids proc)
-	  (with-current-buffer proc-buffer
-	    (erase-buffer)
-	    (setq proc (apply #'start-process (format "search-%s" server)
-			      proc-buffer program cp-list))
-	    (while (process-live-p proc)
-	      (accept-process-output proc))
-            (goto-char (point-min))
-	    (while (re-search-forward
-                    "^thread:\\([^[:space:]\n]+\\)"
-                    (point-max) t)
-	      (cl-pushnew (match-string 1) thread-ids :test #'equal)))
-	  (cl-call-next-method
-	   engine server
-	   ;; If we found threads, completely replace the query with
-	   ;; our new thread-based one.
-           (if thread-ids
-               `((query . ,(mapconcat (lambda (thrd)
-                                        (concat "thread:" thrd))
-                                      thread-ids " or ")))
-             query)
-	   nil)))
-    (cl-call-next-method engine server query groups)))
-
 (cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
 						  (qstring string)
 						  query &optional _groups)
@@ -1721,13 +1684,14 @@ gnus-search-indexed-search-command
       (append
        (list (format "--config=%s" config-file)
              "search"
-             (if thread
-                 "--output=threads"
-             "--output=files"))
+             "--output=files")
        (unless thread '("--duplicate=1"))
        (when limit (list (format "--limit=%d" limit)))
        switches
-       (list qstring)))))
+       (list (if thread
+                 (format "thread:\"{%s}\""
+                         (string-replace "\"" "\"\"" qstring))
+               qstring))))))
 
 ;;; Mairix interface
 
-- 
2.30.2