Subject: bug#38011: closed (Re: bug#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly)

From: [email-masked] (GNU bug Tracking System)
Date: Mon, 18 Jan 2021 21:13:01 +0000

Your bug report

#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
38011: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=38011
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly
From: Eric Abrahamsen <[email-masked]>
To: Robert Pluim <[email-masked]>
Cc: Lars Ingebrigtsen <[email-masked]>, [email-masked]
Date: Mon, 18 Jan 2021 13:12:37 -0800
List-Id: <debbugs-submit.debbugs.gnu.org>

Robert Pluim <[email-masked]> writes:

>>>>>> On Sat, 16 Jan 2021 21:00:36 -0800, Eric Abrahamsen <[email-masked]> said:
>
>     Eric> Eric Abrahamsen <[email-masked]> writes:
>     >> On 01/05/21 09:47 AM, Lars Ingebrigtsen wrote:
>     >>> Eric Abrahamsen <[email-masked]> writes:
>     >>> 
>     >>>> Morally the same: the agent has its own `gnus-agent-uncached-articles'
>     >>>> function, which consults the .agentview file, so it uses that instead of
>     >>>> `gnus-sorted-difference'. But the same idea.
>     >>> 
>     >>> Right.
>
>     Eric> [...]
>
>     >> Anyway, I'll keep looking at this patch for a bit, and see if I can find
>     >> any more problems with it.
>
>     Eric> I found a couple more, in agent file writing, and have pushed a commit
>     Eric> fixing that. So far as I can tell this now works as intended. I've run
>     Eric> side-by-side comparisons of master and this feature branch, messing with
>     Eric> the cache and the agent, and feel fairly confident that behavior is the
>     Eric> same in terms of header fetching, and files written to disk. I think I'd
>     Eric> feel okay merging this, but would also be happy to let it mellow longer.
>
> I think previous experience shows that the only way to find bugs in
> this kind of code is to foist it on us :-)

Bombs away, then!

Subject: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly
From: Eric Abrahamsen <[email-masked]>
To: [email-masked]
Cc: Lars Ingebrigtsen <[email-masked]>
Date: Thu, 31 Oct 2019 14:34:06 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

Right now, Gnus backends return article header data by writing it in a
parseable format into the `nntp-server-buffer', and returning one of the
symbols 'nov or 'headers, indicating how the data should be parsed.

This isn't great because it requires backends to first munge their data
into a format that looks like the NNTP format, which is then parsed
again, which is an extra layer of data transformation. It also makes use
of the `nntp-server-buffer', which is something I'd like to work on
reducing because it causes problems with threading and introduces
potential encoding bugs.

This patch provides the possibility for backends to return their own
headers (ie a list of vectors), though it doesn't actually change any of
the backends to do that -- that will be another patch.

I have one question at this stage: the 'nov or 'headers value gets
stored into the `gnus-headers-retrieved-by' variable. That variable is
later checked in a couple of places like so:

      (when (and gnus-fetch-old-headers
		 (eq gnus-headers-retrieved-by 'nov))
	(if (eq gnus-fetch-old-headers 'invisible)
	    (gnus-build-all-threads)
	  (gnus-build-old-threads)))

If the variable is 'headers, the `gnus-build-*-threads' functions don't
get called at all.

What's the difference between 'nov and 'headers, and why can we build
threads in one case and not the other? If backends were to return their
own headers, what value should they return? I'll also note that the nnir
version of this function returns 'nov regardless of what the "real"
backend function returned -- why is that?

I would love to use some other, more direct, heuristic to decide about
building threads or not, and get rid of the
'nov/'headers/gnus-headers-retrieved-by stuff altogether, but I don't
yet know how to do that.

Eric

>From d871f95325ef230f0fda7aeecf1c2b6461d23183 Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <[email-masked]>
Date: Thu, 31 Oct 2019 14:14:44 -0700
Subject: [PATCH] WIP on allowing Gnus backends to return headers directly

* lisp/gnus/gnus-sum.el (gnus-fetch-headers): Allow the
  gnus-retrieve-headers backend function to directly return a list of
  headers, instead of inserting data to parse in the
  nntp-server-buffer.
* lisp/gnus/nnir.el (nnir-retrieve-headers): Handle the same case when
  nnir calls the "real" backend function.
---
 lisp/gnus/gnus-sum.el | 37 ++++++++++++++++++++-----------------
 lisp/gnus/nnir.el     | 35 +++++++++++++++++++++--------------
 2 files changed, 41 insertions(+), 31 deletions(-)

diff --git a/lisp/gnus/gnus-sum.el b/lisp/gnus/gnus-sum.el
index f21bc7584e..e86781bd1a 100644
--- a/lisp/gnus/gnus-sum.el
+++ b/lisp/gnus/gnus-sum.el
@@ -5631,23 +5631,26 @@ gnus-summary-remove-list-identifiers
 (defun gnus-fetch-headers (articles &optional limit force-new dependencies)
   "Fetch headers of ARTICLES."
   (gnus-message 7 "Fetching headers for %s..." gnus-newsgroup-name)
-  (prog1
-      (if (eq 'nov
-	      (setq gnus-headers-retrieved-by
-		    (gnus-retrieve-headers
-		     articles gnus-newsgroup-name
-		     (or limit
-			 ;; We might want to fetch old headers, but
-			 ;; not if there is only 1 article.
-			 (and (or (and
-				   (not (eq gnus-fetch-old-headers 'some))
-				   (not (numberp gnus-fetch-old-headers)))
-				  (> (length articles) 1))
-			      gnus-fetch-old-headers)))))
-	  (gnus-get-newsgroup-headers-xover
-	   articles force-new dependencies gnus-newsgroup-name t)
-	(gnus-get-newsgroup-headers dependencies force-new))
-    (gnus-message 7 "Fetching headers for %s...done" gnus-newsgroup-name)))
+  (let ((res (setq gnus-headers-retrieved-by
+		   (gnus-retrieve-headers
+		    articles gnus-newsgroup-name
+		    (or limit
+			;; We might want to fetch old headers, but
+			;; not if there is only 1 article.
+			(and (or (and
+				  (not (eq gnus-fetch-old-headers 'some))
+				  (not (numberp gnus-fetch-old-headers)))
+				 (> (length articles) 1))
+			     gnus-fetch-old-headers))))))
+    (prog1
+	(pcase res
+	  ('nov
+	   (gnus-get-newsgroup-headers-xover
+	    articles force-new dependencies gnus-newsgroup-name t))
+	  ((pred listp) res)
+	  ;; 'headers is the other likely value.
+	  (_ (gnus-get-newsgroup-headers dependencies force-new)))
+      (gnus-message 7 "Fetching headers for %s...done" gnus-newsgroup-name))))
 
 (defun gnus-select-newsgroup (group &optional read-all select-articles)
   "Select newsgroup GROUP.
diff --git a/lisp/gnus/nnir.el b/lisp/gnus/nnir.el
index 7cb2d1615a..48af6067f0 100644
--- a/lisp/gnus/nnir.el
+++ b/lisp/gnus/nnir.el
@@ -719,22 +719,29 @@ nnir-retrieve-headers
 	     (setq parsefunc 'nnheader-parse-nov))
 	    ('headers
 	     (setq parsefunc 'nnheader-parse-head))
+	    ((pred listp)
+	     (setq headers gnus-headers-retrieved-by))
 	    (_ (error "Unknown header type %s while requesting articles \
                     of group %s" gnus-headers-retrieved-by artgroup)))
-	  (goto-char (point-min))
-	  (while (not (eobp))
-	    (let* ((novitem (funcall parsefunc))
-		   (artno (and novitem
-			       (mail-header-number novitem)))
-		   (art (car (rassq artno articleids))))
-	      (when art
-		(setf (mail-header-number novitem) art)
-		(push novitem headers))
-	      (forward-line 1)))))
-      (setq headers
-	    (sort headers
-		  (lambda (x y)
-		    (< (mail-header-number x) (mail-header-number y)))))
+	  (unless headers
+	    (goto-char (point-min))
+	    (while (not (eobp))
+	      (push (funcall parsefunc) headers)
+	      (forward-line 1)))
+	  (setq headers
+		(sort
+		 (delq nil
+		       (mapcar
+			(lambda (novitem)
+			  (let* ((artno (and novitem
+					     (mail-header-number novitem)))
+				 (art (car-safe (rassq artno articleids))))
+			    (when art
+			      (setf (mail-header-number novitem) art)
+			      novitem)))
+			headers))
+		 (lambda (x y)
+		   (< (mail-header-number x) (mail-header-number y)))))))
       (erase-buffer)
       (mapc 'nnheader-insert-nov headers)
       'nov)))
-- 
2.23.0