Subject: Re: bug#38249: 12.2.0; reftex-create-bibtex-file and biblatex

From: Arash Esbati <[email-masked]>
Date: Thu, 11 Apr 2024 17:45:57 +0200

gusbrs <[email-masked]> writes:

> I can see you cursing me when you read this one. :-)
> But, worse, much worse, I'm a sinner: I enable
> `reftex-allow-detached-macro-args' and use it widely ("wildly" one
> might say).

I thought you were a nice guy, but that's about to change ;-)
Persoally, I don't like detached args, but that's only me, and it's a
matter of taste, so it doesn't really count.

> An example from an actual document of mine:
>
> destes sobre os benefícios obtidos pelas diferentes partes envolvidas e os
> resultados gerais da valorização.\footcites(Sobre a primeira valorização,
> veja)() {Holloway1978} [40-88]{DelfimNo1959} [vol.~10, \pnfmt{55-218},
>   vol.~11, \pnfmt{35-436}]{Taunay1941} [213-225]{Fausto1989}
> [52-66]{Pelaez1971} [494-501]{Krasner1973} [410-463]{Calogeras1960}
> [13-22]{Fritsch1988} [44-50, 188-195]{Love1980} [62-73, 84-91]{Topik1987}
> [112-134]{Kugelmas1986}
>
> Or how about:
>
> militar, mas que contrastava com ele ao especificar detalhadamente as medidas
> a serem tomadas.\footnote{\cites[1917, vol.~II, sessão de 20/06/1917,
>     \pnfmt{125-131}]{AS}
>   % Assinado:
>   % Pires Ferreira (PI), presidente
>   % F. Mendes de Almeida (MA), relator
>   % A. Indio do Brasil (PA)
>   % Soares dos Santos (RS)
>   % todos militares
>   [1917, vol.~III, sessão de 04/07/1917, \pnfmt{30-34}]{AS}.  A primeira
>   referência é a publicação original da leitura do parecer em plenário, mas
>
> (I hope the email will preserve line breaks, but I think this should
> be enough to grasp how bad it can get).

Can you please try this on your files:

--8<---------------cut here---------------start------------->8---
(defun reftex-all-used-citation-keys ()
  "Return a list of all citation keys used in document."
  (reftex-access-scan-info)
  ;; FIXME: multicites macros provided by biblatex
  ;; are not covered in this function.
  (let ((files (reftex-all-document-files))
        (re (concat "\\\\"
                    "\\(?:"
                    ;; biblatex volcite macros take these args:
                    ;; \volcite[prenote]{volume}[pages]{key}
                    ;; so cater for the first 3 args:
                    (regexp-opt '("volcite"  "Volcite"
                                  "pvolcite" "Pvolcite"
                                  "fvolcite" "ftvolcite"
                                  "svolcite" "Svolcite"
                                  "tvolcite" "Tvolcite"
                                  "avolcite" "Avolcite"))
                    "\\(?:\\[[^]]*\\]\\)?"
                    "{[^}]*}"
                    "\\(?:\\[[^]]*\\]\\)?"
                    "\\|"
                    ;; Other cite macros usually go like:
                    ;; \cite[prenote][postnote]{key}
                    ;; so cater for the optional args:
                    "\\(?:bibentry\\|[a-zA-Z]*[Cc]ite[a-zA-Z*]*\\)"
                    "\\(?:\\[[^]]*\\]\\)\\{0,2\\}"
                    "\\)"
                    ;; Now match the key:
                    "{\\([^}]+\\)}"))
        (re2 (concat "\\\\"
                     (regexp-opt '("cites"       "Cites"
                                   "parencites"  "Parencites"
                                   "footcites"   "footcitetexts"
                                   "smartcites"  "Smartcites"
                                   "textcites"   "Textcites"
                                   "supercites"
                                   "autocites"   "Autocites"
                                   "volcites"    "Volcites"
                                   "pvolcites"   "Pvolcites"
                                   "fvolcites"   "Fvolcites"
                                   "svolcites"   "Svolcites"
                                   "tvolcites"   "Tvolcites"
                                   "avolcites"   "Avolcites"))
                     "\\(?:([^)]*)\\)\\{0,2\\}"))
        (re3 (concat "\\(?:\\[[^]]*\\]\\)\\{0,2\\}"
                     "{\\([^}]+\\)}"))
        file keys kk k)
    (save-current-buffer
      (while (setq file (pop files))
        (set-buffer (reftex-get-file-buffer-force file 'mark))
        (save-excursion
          (save-restriction
            (widen)
            (goto-char (point-min))
            (while (re-search-forward re nil t)
              ;; Make sure we're not inside a comment:
              (unless (save-match-data
                        (nth 4 (syntax-ppss)))
                (setq kk (match-string-no-properties 1))
                (while (string-match "%.*\n?" kk)
                  (setq kk (replace-match "" t t kk)))
                (setq kk (split-string kk "[, \t\r\n]+"))
                (while (setq k (pop kk))
                  (or (member k keys)
                      (setq keys (cons k keys)))))))))
      ;; Re-set `files':
      (setq files (reftex-all-document-files))
      ;; And now search for citation lists:
      (while (setq file (pop files))
        (set-buffer (reftex-get-file-buffer-force file 'mark))
        (save-excursion
          (save-restriction
            (widen)
            (goto-char (point-min))
            (while (re-search-forward re2 nil t)
              ;; Make sure we're not inside a comment:
              (unless (save-match-data
                        (nth 4 (syntax-ppss)))
                (while (progn
                         ;; (and reftex-allow-detached-macro-args
                         (while (looking-at "[ \t\r\n]+\\|%.*\n")
                           (goto-char (match-end 0)))
                         ;; )
                         (and (looking-at re3)
                              (goto-char (match-end 0))))
                  (setq kk (match-string-no-properties 1))
                  (while (string-match "%.*\n?" kk)
                    (setq kk (replace-match "" t t kk)))
                  (setq kk (split-string kk "[, \t\r\n]+"))
                  (while (setq k (pop kk))
                    (or (member k keys)
                        (setq keys (cons k keys)))))))))))
    (reftex-kill-temporary-buffers)
    keys))
--8<---------------cut here---------------end--------------->8---

I think in this particular case, we should ignore the value of
`reftex-allow-detached-macro-args'.

> From AUCTeX/RefTeX's sides pretty much everything works, except that
> you cannot detach the very first argument after the macro. (If I
> recall correctly, this exception is due to an implementation detail in
> font-latex.el, but I don't recall if I had filled this as a
> mal-function or not, I'm just used to it by now. I also don't recall
> if this exception affects only citation lists or is general, probably
> general).

Can you show a concrete example?

> I did some light testing and it does indeed handle the MWE of the
> original report, that is, the simplest cases. But it is easy to beat
> it, you know it. Simply detaching the arguments with white space is
> enough to fool it (that would be easy to fix, I guess). Any macro in
> the pre- or postnotes which itself has an optional argument would fool
> `re3' and break the loop (that would be hard to fix...).

I will think about this one; we could allow one level of balanced braces
inside the arg.

> And, as you have noted yourself, a comment between two arguments would
> also beat it. I'm sure you know much better than me how hard it is to
> parse TeX arguments solely based on regexps (as opposed to sexp
> navigation).

Not always: Try "\foo[opti{]}onal]{}" in a .tex file, put point on [ and
hit 'C-M-f'.

> That said, I really don't recall what the "barely out of noobness five
> year ago me" was up to when "he" opened this bug report. But gladly
> people learn and, nowadays, I'd not even consider for this task
> something which does not rely on generated files (.aux, .bcf, etc.)
> instead of the main file. biber tool mode, bibtool, bibexport, there
> are plenty of options (https://tex.stackexchange.com/q/41821/105447
> seems to be a good list). So, while the "old me" opened this report,
> the "current me" would advise you not to invest much of your time in
> this. It is a losing battle. You can improve things on the margin but
> ultimately it will always remain a "good approximation". Unless, of
> course, you want to go all out and start parsing the relevant
> generated files. But I'm not sure RefTeX should invest in this, given
> there are specialized tools for that.

Agreed.

> True, `reftex-create-bibtex-file' already exists, so it does pose the
> question as to whether "let it wane", "marginally improve" or "replace
> it with something better" is the best option. Not a question for me to
> answer tough. ;-)

I think AUCTeX/RefTeX are here, if we can improve them, we should do it.
And if someone wants to replace it with something better, we can do it
as well.  It is always a matter of "I need that", and best case is one
implements that as well.

Best, Arash