Subject: bug#31132: AUCTeX, RefTeX and biblatex's multicite commands

From: gusbrs <[email-masked]>
Date: Mon, 8 Apr 2024 11:31:34 -0300

Hi Arash,

On Sun, 7 Apr 2024 at 06:01, Arash Esbati <[email-masked]> wrote:
> Time flies :-)

Oh, reading the old thread was a little embarrassing. I didn't even
know how to reply to a mailing list (full html contents at the end,
etc., sigh... ;-).

>
> It's solely about getting C-& work with things like:
>
> \parencites(Global Pre)(Global Post)[Pre][Post]{mitt:97}[Pre][Post]{lamp:94}
>

Great, and it is very welcome that you are revisiting this. Thank you.

I did test things and the TL;DR is: looks good to me.

The long version: I tested both with AUCTeX and built-in latex-mode,
with the following setups.

With AUCTeX:

    (add-to-list 'load-path "~/.emacs.d/elpa/auctex-14.0.4")
    (load "auctex.el" nil t t)
    (load "reftex.el" nil t t)
    (setq TeX-parse-self t)
    (setq TeX-auto-save t)
    (setq reftex-plug-into-AUCTeX t)
    (add-hook 'LaTeX-mode-hook 'turn-on-reftex)
    (require 'reftex-parse) ;; just being sure...
    ;; and apply Arash's patch

Without AUCTeX:

    (load "reftex.el" nil t t)
    (add-hook 'latex-mode-hook 'turn-on-reftex)
    (require 'reftex-parse) ;; just being sure...
    ;; and apply Arash's patch

I used the same test file I had used in the original report. It is
possibly not up to date six years later, but should be thorough enough
for our purposes.

I tested the citation lists of interest both for "C-c &" and "C-c [",
and they worked as expected in all cases. (The only thing in
`\volcite' and friends where the volume is given in braces, thus
mistaken for a key which is not found, but nothing to do there, I'd
say).

I did not attempt to test for possible side effects in other areas,
given that `reftex-move-to-previous-arg' and `reftex-what-macro' are
general purpose utility functions. But I think the patch at hand is
pretty careful in changing the syntax of the parentheses only locally.
That's what I'd do too and my assessment is that I see no reason it
shouldn't be safe. (That's not what I currently do in my own config,
see below, but that's because I'm lazy and free to shoot myself in the
foot in my own config ;-).

All in all, the patch looks good to me.

The only other thing of note I observed is that the mcite-like
citation commands are not fontified as reference/citation commands
(they get only the generic `font-latex-sedate-face'). I had agreed
with you that the case is tricky and arguably not worth the trouble on
RefTeX's side of things. But fontification for them should be trivial.
So, why not?

> > Also, I have been patching this and some related RefTeX functions,
> > especially for this reason, plus some others. Since you have your
> > hands on this, would you like me to share what I have here for them,
> > to see if you can spot anything which might be worth handling at this
> > point?
>
> Yes, please, share them, and we can see what we can incorporate
> upstream, highly appreciated 👍

Nice, perhaps something useful comes out of it. I'll share the code
first and comment later. I'm using `el-patch' in my config, but it
should be easy enough to read, even if you are not acquainted with it.
And, of course, this is not a real patch or anything, just some stuff
to grab ideas from.

#+begin_src emacs-lisp
(with-eval-after-load 'el-patch
  ;; For 'reftex-what-macro' and 'reftex-move-to-previous-arg'.
  (el-patch-feature reftex-parse)
  (with-eval-after-load 'reftex-parse
    ;; Provide replacement for 'reftex-what-macro' which recognizes 'biblatex'
    ;; citation lists global optional arguments (between parentheses, instead
    ;; of brackets).  These are *renamed* versions of them, just for use in
    ;; 'reftex-figure-out-cite-format' and 'reftex-view-crossref', I don't
    ;; want these changes to affect the behavior of RefTeX elsewhere (playing
    ;; safe...).  Note that the same procedure for 'reftex-what-macro-safe'
    ;; fails because of the syntax table used, so we just go with
    ;; 'reftex-what-macro' on 'reftex-view-crossref'.
    (el-patch-defun (el-patch-swap reftex-what-macro
                                   gb/reftex-what-macro-parens)
      (which &optional bound)
      "Find out if point is within the arguments of any TeX-macro.
The return value is either (\"\\macro\" . (point)) or a list of them.

If WHICH is nil, immediately return nil.
If WHICH is 1, return innermost enclosing macro.
If WHICH is t, return list of all macros enclosing point.
If WHICH is a list of macros, look only for those macros and return the
  name of the first macro in this list found to enclose point.
If the optional BOUND is an integer, bound backwards directed
  searches to this point.  If it is nil, limit to nearest \\section -
  like statement.

This function is pretty stable, but can be fooled if the text contains
things like \\macro{aa}{bb} where \\macro is defined to take only one
argument.  As RefTeX cannot know this, the string \"bb\" would still be
considered an argument of macro \\macro."
      (unless reftex-section-regexp (reftex-compile-variables))
      (catch 'exit
        (if (null which) (throw 'exit nil))
        (let ((bound (or bound (save-excursion (re-search-backward
                                                reftex-section-regexp nil 1)
                                               (point))))
              pos cmd-list cmd cnt cnt-opt entry)
          (save-restriction
            (save-excursion
              (narrow-to-region (max (point-min) bound) (point-max))
              ;; move back out of the current parenthesis
              (while (condition-case nil
                         (let ((forward-sexp-function nil))
                           (up-list -1) t)
                       (error nil))
                (setq cnt 1 cnt-opt 0)
                ;; move back over any touching sexps
                (while (and (el-patch-swap
                              (reftex-move-to-previous-arg bound)
                              (gb/reftex-move-to-previous-arg-parens bound))
                            (condition-case nil
                                (let ((forward-sexp-function nil))
                                  (backward-sexp) t)
                              (error nil)))
                  (if (eq (following-char) ?\[) (cl-incf cnt-opt))
                  (cl-incf cnt))
                (setq pos (point))
                (when (and (or (= (following-char) ?\[)
                               (= (following-char) ?\{)
                               (el-patch-add (= (following-char) ?\()))
                           (re-search-backward "\\\\[*a-zA-Z]+\\=" nil t))
                  (setq cmd (reftex-match-string 0))
                  (when (looking-at "\\\\begin{[^}]*}")
                    (setq cmd (reftex-match-string 0)
                          cnt (1- cnt)))
                  ;; This does ignore optional arguments.  Very hard to fix.
                  (when (setq entry (assoc cmd reftex-env-or-mac-alist))
                    (if (> cnt (or (nth 4 entry) 100))
                        (setq cmd nil)))
                  (cond
                   ((null cmd))
                   ((eq t which)
                    (push (cons cmd (point)) cmd-list))
                   ((or (eq 1 which) (member cmd which))
                    (throw 'exit (cons cmd (point))))))
                (goto-char pos)))
            (nreverse cmd-list)))))

    (el-patch-defsubst (el-patch-swap reftex-move-to-previous-arg
                                      gb/reftex-move-to-previous-arg-parens)
      (&optional bound)
      "Assuming that we are in front of a macro argument,
move backward to the closing parenthesis of the previous argument.
This function understands the splitting of macros over several lines
in TeX."
      (cond
       ;; Just to be quick:
       ((memq (preceding-char) '(?\] ?\} (el-patch-add ?\)))))
       ;; Do a search
       ((and reftex-allow-detached-macro-args
             (re-search-backward
              (el-patch-swap
                "[]}][ \t]*[\n\r]?\\([ \t]*%[^\n\r]*[\n\r]\\)*[ \t]*\\="
                "[]})][ \t]*[\n\r]?\\([ \t]*%[^\n\r]*[\n\r]\\)*[ \t]*\\=")
              bound t))
        (goto-char (1+ (match-beginning 0)))
        t)
       (t nil)))))

(with-eval-after-load 'el-patch
  (el-patch-feature reftex-cite); for 'reftex-figure-out-cite-format'.
  (with-eval-after-load 'reftex-cite
    ;; Select citation format with completion.
    (el-patch-defun reftex-figure-out-cite-format (arg &optional
no-insert format-key)
      "Check if there is already a cite command at point and change cite format
in order to only add another reference in the same cite command."
      (let ((macro (car (el-patch-swap
                          (reftex-what-macro 1)
                          (gb/reftex-what-macro-parens 1))))
            (cite-format-value (reftex-get-cite-format))
            key format)
        (cond
         (no-insert
          ;; Format does not really matter because nothing will be inserted.
          (setq format "%l"))

         ((and (stringp macro)
               (el-patch-swap
                 ;; Match also commands from biblatex ending with `s'
                 ;; (\parencites) or `*' (\parencite*) and `texts?'
                 ;; (\footcitetext and \footcitetexts).
                 (string-match
"\\`\\\\cite\\|cite\\([s*]\\|texts?\\)?\\'" macro)
                 (and (string-match
                       (concat
                        ;; The original regexp.
                        "\\`\\\\cite\\|cite\\([s*]\\|texts?\\)?\\'"
                        ;; Plus some...
                        "\\|\\`\\\\begin{\\(foreign\\|hyphen\\)?displaycquote}")
                       macro)

                      ;; Make sure we are in a mandatory arg.
                      (save-excursion
                        (let ((forward-sexp-function nil))
                          (ignore-errors (up-list -1)))
                        (= (following-char) ?\{)))))
          ;; We are already inside a cite macro
          (if (or (not arg) (not (listp arg)))
              (setq format
                    (concat
                     (if (member (preceding-char) '(?\{ ?,))
                 ""
               reftex-cite-key-separator)
                     "%l"
                     (if (member (following-char) '(?\} ?,))
                 ""
               reftex-cite-key-separator)))
            (setq format "%l")))
         (t
          ;; Figure out the correct format
          (setq format
                (if (and (symbolp cite-format-value)
                         (assq cite-format-value reftex-cite-format-builtin))
                    (nth 2 (assq cite-format-value reftex-cite-format-builtin))
                  cite-format-value))
          (when (listp format)
            (setq key
                  (or format-key
                      (el-patch-swap
                        ;; RefTeX offers a selection by key...
                        (reftex-select-with-char
                         "" (concat "SELECT A CITATION FORMAT\n\n"
                                    (mapconcat
                                     (lambda (x)
                                       (format "[%c] %s  %s" (car x)
                                               (if (> (car x) 31) " " "")
                                               (cdr x)))
                                     format "\n")))
                        ;; ... I prefer to select with completion.
                        ;; I'm stripping the slash and the empty arguments
                        ;; from the candidates with 'replace-regexp-in-string'
                        ;; to have a cleaner list of candidates.  Technically
                        ;; speaking this is not fully general, since we can
                        ;; have two candidates (and thus two keys) which are
                        ;; identical or whose only difference are the
                        ;; arguments (e.g. "\\cite{%l}" and "\\cite[][]{%l}",
                        ;; as in the jurabib format in
                        ;; 'reftex-cite-format-builtin').  But I can well do
                        ;; without it by using an adequate
                        ;; 'gb/reftex-cite-format-biblatex' with no such
                        ;; duplicate entries, particularly since I also let
                        ;; 'reftex-cite-prompt-optional-args' set to 'maybe',
                        ;; and thus am not queried by default for optional
                        ;; args anyway and can always include them on demand
                        ;; with a prefix arg.
                        (let* ((cands (mapcar (lambda (c)
                                                (cons
                                                 (replace-regexp-in-string
                                                  "\\(\\[]\\)*\\({%l}\\)*\\'" ""
                                                  (replace-regexp-in-string
                                                   "\\`\\\\" ""
                                                   (cdr c)))
                                                 (car c)))
                                              format))
                               (ans (completing-read "Citation format: "
                                                     cands nil t)))
                          (cdr (assoc ans cands))))))
            (if (assq key format)
                (setq format (cdr (assq key format)))
              (error "No citation format associated with key `%c'" key)))))
        format))))

(with-eval-after-load 'el-patch
  (el-patch-feature reftex-dcr); for 'reftex-view-crossref' and
                               ; 'reftex-view-crossref-when-idle'.
  (with-eval-after-load 'reftex-dcr
    ;; 'reftex-view-crossref' and the related 'reftex-view-crossref-when-idle'
    ;; have an annoying bug of reporting non-existing bibentries when point is
    ;; actually on pre- or post-notes.  This has been reported
    ;; (https://lists.gnu.org/archive/html/bug-auctex/2019-11/msg00007.html),
    ;; but the bug report was not really well received (see thread
    ;; https://lists.gnu.org/archive/html/bug-auctex/2020-08/msg00007.html).
    ;; So I unfortunately have to patch it here.  This must be done also for
    ;; "ref" commands, since 'zcref' takes an optional argument, and thus is
    ;; subject to the same issue.  I've also added support for 'zcheck' and
    ;; for 'csquotes' integrated display environments here.  Some of this
    ;; stuff could arguably be done by means of 'reftex-view-crossref-extra'
    ;; but, in the end, this is somewhat restrictive, since it only allows to
    ;; enable search in the current buffer, which particularly defeats
    ;; citation keys.
    (el-patch-defun reftex-view-crossref (&optional arg auto-how fail-quietly)
      "View cross reference of macro at point.

Point must be on the KEY argument.  When at a `\\ref' macro, show
corresponding `\\label' definition, also in external
documents (`xr').  When on a label, show a locations where KEY is
referenced.  Subsequent calls find additional locations.  When on
a `\\cite', show the associated `\\bibitem' macro or the BibTeX
database entry.  When on a `\\bibitem', show a `\\cite' macro
which uses this KEY.  When on an `\\index', show other locations
marked by the same index entry.

To define additional cross referencing items, use the option
`reftex-view-crossref-extra'.  See also `reftex-view-crossref-from-bibtex'.
With one or two \\[universal-argument] prefixes, enforce rescanning of
the document.
With argument 2, select the window showing the cross reference.
AUTO-HOW is only for the automatic crossref display and is handed through
to the functions `reftex-view-cr-cite' and `reftex-view-cr-ref'."
      (interactive "P")
      (el-patch-add (require 'reftex-parse))
      ;; See where we are.
      (let* ((macro (car (el-patch-swap
                           (reftex-what-macro-safe 1)
                           (gb/reftex-what-macro-parens 1))))
             (key (reftex-this-word "^{}%\n\r, \t"))
             dw)

        (if (or (null macro) (reftex-in-comment))
            (or fail-quietly
                (error "Not on a crossref macro argument"))

          (setq reftex-call-back-to-this-buffer (current-buffer))

          (cond
           ((el-patch-swap
              (string-match
"\\`\\\\cite\\|cite\\([s*]\\|texts?\\)?\\'\\|bibentry" macro)
              ;; A citation macro: search for bibitems or BibTeX entries.
              ;; Match also commands from biblatex ending with `s'
              ;; (\parencites) or `*' (\parencite*) and `texts?'
              ;; (\footcitetext and \footcitetexts).
              (and (string-match
                    (concat
                     ;; The original regexp.
                     "\\`\\\\cite\\|cite\\([s*]\\|texts?\\)?\\'\\|bibentry"
                     ;; Plus some...
                     "\\|\\`\\\\begin{\\(foreign\\|hyphen\\)?displaycquote}")
                    macro)

                   ;; Make sure we are in a mandatory arg.
                   (save-excursion
                     (let ((forward-sexp-function nil))
                       (ignore-errors (up-list -1)))
                     (= (following-char) ?\{))))
            (setq dw (reftex-view-cr-cite arg key auto-how)))
           ((el-patch-swap
              (string-match "\\`\\\\ref\\|ref\\(range\\|s\\)?\\*?\\'" macro)
              ;; A reference macro: search for labels.
              ;; Match also commands from cleveref ending with `s' (\namecrefs).
              (and (string-match
                    (concat
                     ;; The original regexp.
                     "\\`\\\\ref\\|ref\\(range\\|s\\)?\\*?\\'"
                     ;; Plus some...
                     "\\|\\`\\\\zcheck\\*?\\'")
                    macro)
                   ;; Make sure we are in a mandatory arg.
                   (save-excursion
                     (let ((forward-sexp-function nil))
                       (ignore-errors (up-list -1)))
                     (and (= (following-char) ?\{)
                          ;; And make sure it is the *first* mandatory arg
                          ;; (for '\zcheck').
                          (progn (backward-sexp 1) t)
                          (not (= (following-char) ?\{))))))
            (setq dw (reftex-view-cr-ref arg key auto-how)))
           (auto-how nil)  ;; No further action for automatic display (speed)
           ((or (equal macro "\\label")
                (el-patch-add (equal macro "\\begin{zcregion}"))
                (member macro reftex-macros-with-labels))
            ;; A label macro: search for reference macros
            (reftex-access-scan-info arg)
            (setq dw (reftex-view-regexp-match
                      (format reftex-find-reference-format (regexp-quote key))
                      4 nil nil)))
           ((equal macro "\\bibitem")
            ;; A bibitem macro: search for citations
            (reftex-access-scan-info arg)
            (setq dw (reftex-view-regexp-match
                      (format reftex-find-citation-regexp-format
(regexp-quote key))
                      4 nil nil)))
           ((member macro reftex-macros-with-index)
            (reftex-access-scan-info arg)
            (setq dw (reftex-view-regexp-match
                      (format reftex-find-index-entry-regexp-format
                              (regexp-quote key))
                      3 nil nil)))
           (t
            (reftex-access-scan-info arg)
            (catch 'exit
              (let ((list reftex-view-crossref-extra)
                    entry mre action group)
                (while (setq entry (pop list))
                  (setq mre (car entry)
                        action (nth 1 entry)
                        group (nth 2 entry))
                  (when (string-match mre macro)
                    (setq dw (reftex-view-regexp-match
                              (format action key) group nil nil))
                    (throw 'exit t))))
              (error "Not on a crossref macro argument"))))
          (if (and (eq arg 2) (windowp dw)) (select-window dw)))))

    ;; Allow for 'reftex-view-crossref' to find reference commands with
    ;; multiple labels.  Also add support for '\zcheck'.
    (setopt reftex-find-reference-format
            (concat
             "\\\\\\(ref[a-zA-Z]*\\|[a-zA-Z]*ref\\(range\\)?\\|zcheck\\)\\*?"
             "\\(\\[[^]]*\\]\\|{[^}]*}\\)*"
             "{\\(?:[^}]*,[ \n]*\\)?\\(%s\\)[ ]*[},]"))

    (el-patch-defun reftex-view-crossref-when-idle ()
      ;; Display info about crossref at point in echo area or a window.
      ;; This function was designed to work with an idle timer.
      ;; We try to get out of here as quickly as possible if the call
is useless.
      (and reftex-mode
           ;; Make sure message area is free if we need it.
           (or (eq reftex-auto-view-crossref 'window) (not (current-message)))
           ;; Make sure we are not already displaying this one
           (not (memq last-command '(reftex-view-crossref
                                     reftex-mouse-view-crossref)))
           ;; Quick precheck if this might be a relevant spot
           ;; `reftex-view-crossref' will do a more thorough check.
           (save-excursion
             (search-backward "\\" nil t)
             (looking-at
              (el-patch-swap
                "\\\\[a-zA-Z]*\\(cite\\|ref\\|bibentry\\)"
                (concat
                 ;; The original regexp.
                 "\\\\[a-zA-Z]*\\(cite\\|ref\\|bibentry\\)"
                 ;; Plus some...
                 "\\|\\\\zcheck"
                 "\\|\\\\begin{\\(foreign\\|hyphen\\)?displaycquote}"))))

           (condition-case nil
               (let ((current-prefix-arg nil))
                 (cond
                  ((eq reftex-auto-view-crossref t)
                   (reftex-view-crossref -1 'echo 'quiet))
                  ((eq reftex-auto-view-crossref 'window)
                   (reftex-view-crossref -1 'tmp-window 'quiet))
                  (t nil)))
             (error nil))))))

(with-eval-after-load 'el-patch
  (el-patch-feature reftex-parse); for 'reftex-parse-from-file'
  (with-eval-after-load 'reftex-parse
    ;; With lazy-macro-expansion to avoid startup errors if 'el-patch' is not
    ;; installed.
    (gb/with-lazy-macro-expansion
     ;; Add RefTeX support for zref-xr.  I have made a feature request at
     ;; https://lists.gnu.org/archive/html/auctex-devel/2021-11/msg00012.html
     ;; which met with a positive answer by Arash at
     ;; https://lists.gnu.org/archive/html/auctex-devel/2021-12/msg00000.html.
     ;; But, even if this really makes it, it will take a long time for me to
     ;; enjoy.  In the meantime, we can use a patch, and a "template" at that,
     ;; since this is a tiny change in a large function.
     (el-patch-define-and-eval-template
      (defun reftex-parse-from-file)
      (el-patch-swap
        "\\\\external\\(?:cite\\)?document"
        "\\\\z?external\\(?:cite\\)?document\\*?")))))
#+end_src

Comments, starting with exclusions. Naturally, my changes to
`reftex-what-macro' and `reftex-move-to-previous-arg' will become
obsolete with your patch (Thanks!!). There are obvious idiosyncrasies
of mine, like preferring completing-read to a key dispatcher to choose
the citation command. Disregard those.

Net of this kind of stuff are some things which accumulated over time,
to add support to specific packages, or fixing things I didn't like.
In hindsight, my general take on those is that RefTeX seems to presume
that only the standard, or very traditional, reference and citation
commands exist and, hence, chooses to hard-code most of the relevant
behavior, making it very hard to extend and tweak. Inaccessible to the
less daring users and, more importantly, to AUCTeX style files.

Now, as an author of some non-standard cross-reference packages I'm
certainly more sensitive to these restrictions than the regular user.
Which makes me partial in this regard. But the simple fact that
`csquotes' integrated display environments are not supported by this
feature set of RefTeX is likely sufficient to convince that this
hard-coding bites more widely than the users of some niche package.

The fact that `reftex-view-crossref' does not exclude optional
arguments with its implications to `reftex-view-crossref-when-idle'
would also remain, but this was reported and discussed elsewhere, and
I have nothing new to add on this matter.

In summary, ideally some of this stuff could be exposed so as to be
accessible from AUCTeX style files. Failing that, I'd say adding
support to `csquotes', even if hard-coded, would be very welcome.

> > I'm surprised you dug this back. It is appreciated. Thank you.
>
> I started looking at our old bug reports, let's see if we can close this
> one as well.  Since it is your first report, it should get the attention
> deserved ;-)

I recall you questioning somewhat the relevance of this peculiar
notation of biblatex's citation lists
(https://lists.gnu.org/archive/html/auctex-devel/2023-05/msg00015.html)
so, as a regular user of them, I'm glad and thankful to see you soften
a little your stance in this regard.

Best,
gusbrs