Subject: bug#58985: 29.0.50; Gnus setup broken by commit 2cf9e699ef0fc43a4eadaf00a1ed2f876765c64d

From: "J.P." <[email-masked]>
Date: Fri, 25 Nov 2022 06:23:47 -0800

João Távora <[email-masked]> writes:

> The affected item is, I believe, ~/.password-store/local-gmail:imap.gpg
> and likely also ~/.password-store/smtp.gmail.com:465.gpg.  When I set
> auth-source-debug to t, these lines appeared in *Messages*
>
> auth-source-pass: final result: nil
> auth-source-search: found 0 results (max 1) matching (:max 1 :host
> ("local-gmail" "localhost") :port ("imap" "imap" "143") :user
> "[email-masked]" :require (:user :secret) :create t)
> auth-source-pass: final result: nil
> auth-source-search: CREATED 0 results (max 1) matching (:max 1 :host
> ("local-gmail" "localhost") :port ("imap" "imap" "143") :user
> "[email-masked]" :require (:user :secret) :create t)
> Opening nnimap server on local-gmail...failed: 

This was helpful, thanks. It seems

  :require (:user ...)

is clashing with the absence of a "user" component in the affected file
names.

Among other things, the commit in question tries to provide a way of
honoring the `:require' keyword in a manner befitting the doc string of
`auth-source-search':

  :require (A B C) means that only results that contain those
  tokens will be returned.  Thus for instance requiring :secret
  will ensure that any results will actually have a :secret
  property.

The other back ends more or less do the same. (Take a peek at the
attached examples if you're bored.) So, I guess the takeaway here, at
least as things stand, is basically this: if for some reason you really
wanted to enable the option, you'd need to rename the affected files.
Either

  ~/.password-store/[email-masked]@local-gmail:imap.gpg

or

  ~/.password-store/local-gmail:imap/[email-masked]

should do it. Alternatively, if the gnus function that calls
`auth-source-search' were somehow configurable (guessing no), you could
omit the `:require's altogether, increase the `:max' value, and
prioritize the results, which is what ERC does (or tries to do).

>> However, I think it's probably best to forgo all that and do what I was
>> leaning toward from the outset, and that's keeping the new behavior off
>> by default in Emacs 29. It's looking liable to cause too much churn for
>> too many folks [1]. Thus, unless anyone objects or has anything else to
>> add, I will do this in the next 24 hours or so. Apologies again for the
>> disruption and the time spent bisecting.
>
> No problem, and thanks for understanding.

Thank YOU for understanding. (All I did was break your email.)

> I think it is indeed better if you make this opt-in. I can then opt
> into it and help you debug the root cause. But in the meantime, my
> email won't be broken :-)

I've pushed the change, but you may need to clear your auth-source cache
or restart your session to see any effect. Please let me know if that
doesn't do it. And thanks for all your work on Emacs!

J.P.

>From 5b0f0c108578cbae5f0804fe1daa60599a71d4bf Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <[email-masked]>
Date: Thu, 24 Nov 2022 21:03:03 -0800
Subject: [PATCH] [POC] Compare :require among auth-source backends

---
 test/lisp/auth-source-pass-tests.el | 305 ++++++++++++++++++++++++++++
 1 file changed, 305 insertions(+)

diff --git a/test/lisp/auth-source-pass-tests.el b/test/lisp/auth-source-pass-tests.el
index 1107e09b51..ed88cf5476 100644
--- a/test/lisp/auth-source-pass-tests.el
+++ b/test/lisp/auth-source-pass-tests.el
@@ -751,6 +751,311 @@ auth-source-pass-extra-query-keywords--user-priorities
                          (:host "g" :user "u" :port 2 :secret "@") ; **
                          (:host "g" :user "u" :port 2 :secret "/"))))))))
 
+;;;; :require demo
+
+;; Swapping out smtp.gmail.com and 465 with local-gmail and imap
+;; doesn't change the outcome of any cases below.
+
+;; netrc
+
+(ert-deftest auth-source-pass-extra-query-keywords--netrc-joao ()
+  (ert-with-temp-file netrc-file
+    :text "\
+machine local-gmail port imap password a
+machine smtp.gmail.com port 465 password b
+"
+    (let* ((auth-sources (list netrc-file))
+           (auth-source-do-cache nil)
+           (results (auth-source-search :max 1
+                                        :host '("local-gmail" "localhost")
+                                        :port '("imap" "imap" "143")
+                                        :user "[email-masked]"
+                                        :require '(:user :secret))))
+      (should-not results))))
+
+(ert-deftest auth-source-pass-extra-query-keywords--netrc-joao-user ()
+  (ert-with-temp-file netrc-file
+    :text "\
+machine local-gmail login [email-masked] port imap password a
+machine smtp.gmail.com login [email-masked] port 465 password b
+"
+    (let* ((auth-sources (list netrc-file))
+           (auth-source-do-cache nil)
+           (results (auth-source-search :max 1
+                                        :host '("local-gmail" "localhost")
+                                        :port '("imap" "imap" "143")
+                                        :user "[email-masked]"
+                                        :require '(:user :secret))))
+      (dolist (result results)
+        (setf (plist-get result :secret) (auth-info-password result)))
+      (should (equal results '(( :host "local-gmail"
+                                 :user "[email-masked]"
+                                 :port "imap"
+                                 :secret "a")))))))
+
+;; plstore
+
+(require 'epg)
+
+(ert-deftest auth-source-pass-extra-query-keywords--plstore-joao ()
+  (ert-with-temp-file plstore-file
+    :suffix ".plist"
+    :text "\
+;;; public entries -*- mode: plstore -*-
+((\"7c0d0c60eba2b3da2feebff6a007934b73d6ba8c\"
+  :secret-secret t
+  :host \"local-gmail\"
+  :port \"imap\")
+ (\"e579bb71ac879a2fbe90462be686ec090bcb995f\"
+  :secret-secret t
+  :host \"smtp.gmail.com\"
+  :port \"465\"))
+;;; secret entries
+((\"7c0d0c60eba2b3da2feebff6a007934b73d6ba8c\" :secret \"a\")
+ (\"e579bb71ac879a2fbe90462be686ec090bcb995f\" :secret \"b\"))
+"
+    (cl-letf (((symbol-function 'epg-decrypt-string)
+               (lambda (&rest r) (prin1-to-string (cadr r))))
+              ((symbol-function 'epg-find-configuration)
+               (lambda (&rest _) '((program . "/bin/true")))))
+      (let* ((auth-sources (list plstore-file))
+             (auth-source-do-cache nil)
+             (results (auth-source-search :max 1
+                                          :host '("local-gmail" "localhost")
+                                          :port '("imap" "imap" "143")
+                                          :user "[email-masked]"
+                                          :require '(:user :secret))))
+        (should-not results)))))
+
+(ert-deftest auth-source-pass-extra-query-keywords--plstore-joao-user ()
+  (ert-with-temp-file plstore-file
+    :suffix ".plist"
+    :text "\
+;;; public entries -*- mode: plstore -*-
+((\"b0d8e1b370cff2d4c71cd503905d1bfa80247a82\"
+  :secret-secret t
+  :host \"local-gmail\"
+  :user \"[email-masked]\"
+  :port \"imap\")
+ (\"5d05df976779ae4690254c6572c1652748ac4b58\"
+  :secret-secret t
+  :host \"smtp.gmail.com\"
+  :user \"[email-masked]\"
+  :port \"465\"))
+;;; secret entries
+((\"b0d8e1b370cff2d4c71cd503905d1bfa80247a82\" :secret \"a\")
+ (\"5d05df976779ae4690254c6572c1652748ac4b58\" :secret \"b\"))
+"
+    (cl-letf (((symbol-function 'epg-decrypt-string)
+               (lambda (&rest r) (prin1-to-string (cadr r))))
+              ((symbol-function 'epg-find-configuration)
+               (lambda (&rest _) '((program . "/bin/true")))))
+      (let* ((auth-sources (list plstore-file))
+             (auth-source-do-cache nil)
+             (results (auth-source-search :max 1
+                                          :host '("local-gmail" "localhost")
+                                          :port '("imap" "imap" "143")
+                                          :user "[email-masked]"
+                                          :require '(:user :secret))))
+        (dolist (result results)
+          (setf (plist-get result :secret) (auth-info-password result)))
+        (should (equal results '(( :login nil
+                                   :secret "a"
+                                   :host "local-gmail"
+                                   :user "[email-masked]"
+                                   :port "imap"))))))))
+
+;; json
+
+(ert-deftest auth-source-pass-extra-query-keywords--json-joao ()
+  (ert-with-temp-file json-store
+    :suffix ".json"
+    :text "\
+[{\"host\":\"local-gmail\",
+  \"port\":\"imap\",
+  \"secret\":\"a\"},
+ {\"host\":\"smtp.gmail.com\",
+  \"port\":\"465\",
+  \"secret\":\"b\"}]
+"
+    (let ((auth-sources (list json-store))
+          (auth-source-do-cache nil))
+      (should-not (auth-source-search :max 1
+                                      :host '("local-gmail" "localhost")
+                                      :port '("imap" "imap" "143")
+                                      :user "[email-masked]"
+                                      :require '(:user :secret))))))
+
+(ert-deftest auth-source-pass-extra-query-keywords--json-joao-user ()
+  (ert-with-temp-file json-store
+    :suffix ".json"
+    :text "\
+[{\"host\":\"local-gmail\",
+  \"port\":\"imap\",
+  \"user\":\"[email-masked]\",
+  \"secret\":\"a\"},
+ {\"host\":\"smtp.gmail.com\",
+  \"port\":\"465\",
+  \"user\":\"[email-masked]\",
+  \"secret\":\"b\"}]
+"
+    (let* ((auth-sources (list json-store))
+           (auth-source-do-cache nil)
+           (results (auth-source-search :max 1
+                                        :host '("local-gmail" "localhost")
+                                        :port '("imap" "imap" "143")
+                                        :user "[email-masked]"
+                                        :require '(:user :secret))))
+      (dolist (result results)
+        (setf (plist-get result :secret) (auth-info-password result)))
+      (should (equal results
+                     '(( :host "local-gmail"
+                         :port "imap"
+                         :user "[email-masked]"
+                         :secret "a")))))))
+
+;; secrets
+
+;; With the secrets backend, any "attribute" (keyword) specified in
+;; the query must exist in the entry, so `:require' is implied.  The
+;; mocked search below is so contrived as to be meaningless, but it
+;; nevertheless emphasizes the "must match" nature of the upstream
+;; SearchItems DBus method (of the Secret Service API's collections
+;; interface: org.freedesktop.Secret.Collection.SearchItems).
+;;
+;; https://specifications.freedesktop.org/secret-service/latest/re02.html
+;; #org.freedesktop.Secret.Collection.SearchItems
+
+(require 'secrets)
+
+(ert-deftest auth-source-pass-extra-query-keywords--secrets-joao ()
+  (let ((auth-sources '("secrets:Test"))
+        (auth-source-do-cache nil)
+        (entries '(("nil@local-gmail:imap"
+                    (:host . "local-gmail")
+                    (:port . "imap")
+                    (:xdg:schema . "org.freedesktop.Secret.Generic"))
+                   ("[email-masked]:465"
+                    (:host . "smtp.gmail.com")
+                    (:port . "465")
+                    (:xdg:schema . "org.freedesktop.Secret.Generic"))))
+        (secrets '(("nil@local-gmail:imap" . "a")
+                   ("[email-masked]:465" . "b"))))
+
+    (cl-letf (((symbol-function 'secrets-search-items)
+               (lambda (_ &rest r)
+                 (mapcan (lambda (s)
+                           (and (seq-every-p (pcase-lambda (`(,k . ,v))
+                                               (equal v (alist-get k (cdr s))))
+                                             (map-pairs r))
+                                (list (car s))))
+                         entries)))
+              ((symbol-function 'secrets-get-secret)
+               (lambda (_ label) (assoc-default label secrets)))
+              ((symbol-function 'secrets-get-attributes)
+               (lambda (_ label) (assoc-default label entries))))
+
+      (should-not (auth-source-search :max 1
+                                      :host '("local-gmail" "localhost")
+                                      :port '("imap" "imap" "143")
+                                      :user "[email-masked]"
+                                      :require '(:user :secret))))))
+
+(ert-deftest auth-source-pass-extra-query-keywords--secrets-joao-user ()
+  (let ((auth-sources '("secrets:Test"))
+        (auth-source-do-cache nil)
+        (entries '(("[email-masked]@local-gmail:imap"
+                    (:host . "local-gmail")
+                    (:user . "[email-masked]")
+                    (:port . "imap")
+                    (:xdg:schema . "org.freedesktop.Secret.Generic"))
+                   ("[email-masked]@smtp.gmail.com:465"
+                    (:host . "smtp.gmail.com")
+                    (:user . "[email-masked]")
+                    (:port . "465")
+                    (:xdg:schema . "org.freedesktop.Secret.Generic"))))
+        (secrets '(("[email-masked]@local-gmail:imap" . "a")
+                   ("[email-masked]@smtp.gmail.com:465" . "b"))))
+
+    (cl-letf (((symbol-function 'secrets-search-items)
+               (lambda (_ &rest r)
+                 (mapcan (lambda (s)
+                           (and (seq-every-p (pcase-lambda (`(,k . ,v))
+                                               (equal v (alist-get k (cdr s))))
+                                             (map-pairs r))
+                                (list (car s))))
+                         entries)))
+              ((symbol-function 'secrets-get-secret)
+               (lambda (_ label) (assoc-default label secrets)))
+              ((symbol-function 'secrets-get-attributes)
+               (lambda (_ label) (assoc-default label entries))))
+
+      (let ((results (auth-source-search :max 1
+                                         :host '("local-gmail" "localhost")
+                                         :port '("imap" "imap" "143")
+                                         :user "[email-masked]"
+                                         :require '(:user :secret))))
+        (dolist (result results)
+          (setf (plist-get result :secret) (auth-info-password result)))
+        (should (equal results
+                       '(( :login nil
+                           :secret "a"
+                           :host "local-gmail"
+                           :user "[email-masked]"
+                           :port "imap"
+                           :xdg:schema "org.freedesktop.Secret.Generic"))))))))
+
+;; Pass
+
+(ert-deftest auth-source-pass-extra-query-keywords--pass--joao ()
+  (auth-source-pass--with-store '(("smtp.gmail.com:465" (secret . "a"))
+                                  ("local-gmail:imap" (secret . "b")))
+    (auth-source-pass-enable)
+    (let ((auth-source-pass-extra-query-keywords t))
+      (should-not (auth-source-search :max 1
+                                      :host '("local-gmail" "localhost")
+                                      :port '("imap" "imap" "143")
+                                      :user "[email-masked]"
+                                      :require '(:user :secret))))))
+
+(ert-deftest auth-source-pass-extra-query-keywords--pass--joao-user ()
+  ;; "suffix" syntax
+  (auth-source-pass--with-store '(("smtp.gmail.com:465/[email-masked]"
+                                   (secret . "a"))
+                                  ("local-gmail:imap/[email-masked]"
+                                   (secret . "b")))
+    (auth-source-pass-enable)
+    (let* ((auth-source-pass-extra-query-keywords t)
+           (results (auth-source-search :max 1
+                                        :host '("local-gmail" "localhost")
+                                        :port '("imap" "imap" "143")
+                                        :user "[email-masked]"
+                                        :require '(:user :secret))))
+      (dolist (result results)
+        (setf (plist-get result :secret) (auth-info-password result)))
+      (should (equal results '(( :host "local-gmail"
+                                 :user "[email-masked]"
+                                 :port "imap"
+                                 :secret "b"))))))
+  ;; "prefix" syntax
+  (auth-source-pass--with-store '(("[email-masked]@smtp.gmail.com:465"
+                                   (secret . "a"))
+                                  ("[email-masked]@local-gmail:imap"
+                                   (secret . "b")))
+    (auth-source-pass-enable)
+    (let* ((auth-source-pass-extra-query-keywords t)
+           (results (auth-source-search :max 1
+                                        :host '("local-gmail" "localhost")
+                                        :port '("imap" "imap" "143")
+                                        :user "[email-masked]"
+                                        :require '(:user :secret))))
+      (dolist (result results)
+        (setf (plist-get result :secret) (auth-info-password result)))
+      (should (equal results '(( :host "local-gmail"
+                                 :user "[email-masked]"
+                                 :port "imap"
+                                 :secret "b")))))))
+
 (provide 'auth-source-pass-tests)
 
 ;;; auth-source-pass-tests.el ends here
-- 
2.38.1