Subject: 2023-08-31; Fix xref errors when matching in unvisited files

From: David Fussner <[email-masked]>
Date: Wed, 13 Sep 2023 11:00:41 +0100

Hi.

I've been working on getting the main xref commands working in TeX
files, and ran into a couple of issues.  To reproduce the first one, in
an empty directory put something like the following text into a file,
mwe.ltx (the file extension matters -- please don't try .tex, as that
encounters a separate bug which is better fixed in Emacs itself, I
think):

\documentclass[12pt]{article}
\usepackage{fontspec}
\usepackage{xparse}
\ExplSyntaxOn
\bool_new:N \g__hook_debug_bool
\cs_new_eq:NN \__hook_debug:n \use_none:n
\ExplSyntaxOff
\begin{document}
\thispagestyle{empty}
\end{document}

Then copy mwe.ltx to mwe2.ltx, and visit mwe.ltx without visiting
mwe2.ltx.  Place point on "fontspec" and hit M-?, RTN, then "..." to
choose a search directory, then choose the current one (the default).
Here I see this error:

Font-lock trying to use keywords before setting them up

The end of the backtrace looks like this:

Debugger entered--Lisp error: (error "Font-lock trying to use keywords
before setting th...")
  signal(error ("Font-lock trying to use keywords before setting th..."))
  error("Font-lock trying to use keywords before setting th...")
  font-lock-compile-keywords((("\\\\\\(?:\\(?:verb\\*?\\)\\)\\(?:\\[[^][]*\\(?:\\[[^][]*\\][^]..."
(1 "\"") (2 ".") (3 "\"")) ("\\(\\\\\\)end
*{\\(?:\\(?:filecontents\\*?\\|verbatim\\*?\\)..." (1 "|" t)) ("^[
\11]*\\\\begin *{\\(?:\\(?:filecontents\\*?\\|verbatim\\..." (1 "|"
t))) t)
  font-lock-fontify-syntactic-keywords-region(1 22)
  #f(compiled-function (start end) #<bytecode 0x835be9f85adc43b>)(1 22)
  syntax-propertize(22)
  xref--collect-matches-1("\\_<fontspec\\_>" "~/src/testing/mwe2.ltx" 2 1 22 0)

The compiled function here is
font-latex--make-syntax-propertize-function, which is a wrapper for
syntax-propertize-via-font-lock, which is a standard value for
syntax-propertize-function, but the in-tree tex-mode.el uses a
different function, and the error doesn't appear there.  Here
syntax-propertize-via-font-lock calls
font-lock-fontify-syntactic-keywords-region, which calls
font-lock-compile-keywords, which errors out because in the temporary
buffer (" *xref-temp*") we haven't yet called font-lock-set-defaults.

The in-tree bibtex.el uses the same syntax-propertize-function but
doesn't error here, because it calls font-lock-set-defaults, with a
comment explaining why, at the end of the definition of bibtex-mode
itself.  If you visit both mwe2.ltx and mwe.ltx, the error won't appear,
because the font-locking mechanism calls font-lock-set-defaults when
setting up the buffer.

I enclose a patch (0001), which puts the required call, with a comment, at
the end of font-latex-set-syntactic-keywords, which seems to work for
all the AUCTeX modes, but there may be a better place for it.

If you apply patch 0001, and make sure font-latex.el is reloaded, you
can then see the next bug: in specific cases, xref-find-references
reports a hit only in visited files, despite there being identical text
in the unvisited one.

To reproduce, move point in mwe.ltx from "fontspec" to
"\__hook_debug:n", then call M-? just as before. You'll only see the hit
in mwe.ltx in the xref buffer, not the one in mwe2.ltx.  The problem is
that the search string called by xref-find-references has the format
"\\_<%s\\_>", so that any search term which doesn't start and end with
chars having either word or symbol syntax won't be returned.  Now,
assuming that parsing is enabled, AUCTeX does the right thing here and
parses this file as containing expl3 constructs, and therefore gives "_"
symbol syntax.  Unfortunately, the style file expl3.el will not be
applied to temporary, non-file-visiting buffers like " *xref-temp*",
because the find-file-hook set at the end of
VirTeX-common-initialization won't be called in such buffers.  The
attached patch (0002) parses and applies styles in the temp buffer, and
only in such non-file-visiting buffers.  It does so both in
LaTeX-common-initialization and in plain-TeX-common-initialization,
which I think covers all the modes.

My profoundest apologies for the long bug report, and thanks,

David.

Emacs  : GNU Emacs 30.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version
3.24.31, cairo version 1.16.0)
 of 2023-09-11
Package: 2023-08-31

current state:
==============
(setq
 AUCTeX-date "2023-08-31"
 window-system 'x
 LaTeX-version "2e"
 TeX-style-path '("/usr/local/var/auctex" "/home/dfussner/src/auctex/style"
          "/home/dfussner/.emacs.d/auctex/auto"
          "/home/dfussner/.emacs.d/auctex/style" "auto" "style")
 TeX-auto-save t
 TeX-parse-self t
 TeX-master nil
 TeX-command-list '(("TeX"
             "%(PDF)%(tex) %(file-line-error) %`%(extraopts)
%S%(PDFout)%(mode)%' %(output-dir) %t"
             TeX-run-TeX nil
             (plain-tex-mode ams-tex-mode texinfo-mode) :help
             "Run plain TeX")
            ("LaTeX" "%`%l%(mode)%' %T" TeX-run-TeX nil
             (latex-mode doctex-mode) :help "Run LaTeX")
            ("Makeinfo" "makeinfo %(extraopts) %(o-dir) %t"
             TeX-run-compile nil (texinfo-mode) :help
             "Run Makeinfo with Info output")
            ("Makeinfo HTML"
             "makeinfo %(extraopts) %(o-dir) --html %t"
             TeX-run-compile nil (texinfo-mode) :help
             "Run Makeinfo with HTML output")
            ("AmSTeX"
             "amstex %(PDFout) %`%(extraopts) %S%(mode)%' %(output-dir) %t"
             TeX-run-TeX nil (ams-tex-mode) :help "Run AMSTeX")
            ("ConTeXt"
             "%(cntxcom) --once --texutil %(extraopts) %(execopts)%t"
             TeX-run-TeX nil (context-mode) :help "Run ConTeXt once")
            ("ConTeXt Full" "%(cntxcom) %(extraopts) %(execopts)%t"
             TeX-run-TeX nil (context-mode) :help
             "Run ConTeXt until completion")
            ("BibTeX" "bibtex %(O?aux)" TeX-run-BibTeX nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode context-mode)
             :help "Run BibTeX")
            ("Biber" "biber %(output-dir) %s" TeX-run-Biber nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run Biber")
            ("Texindex" "texindex %s.??" TeX-run-command nil
             (texinfo-mode) :help "Run Texindex")
            ("Texi2dvi" "%(PDF)texi2dvi %t" TeX-run-command nil
             (texinfo-mode) :help "Run Texi2dvi or Texi2pdf")
            ("View" "%V" TeX-run-discard-or-function t t :help
             "Run Viewer")
            ("Print" "%p" TeX-run-command t t :help "Print the file")
            ("Queue" "%q" TeX-run-background nil t :help
             "View the printer queue" :visible TeX-queue-command)
            ("File" "%(o?)dvips %d -o %f " TeX-run-dvips t
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Generate PostScript file")
            ("Dvips" "%(o?)dvips %d -o %f " TeX-run-dvips nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Convert DVI file to PostScript")
            ("Dvipdfmx" "dvipdfmx -o %(O?pdf) %d" TeX-run-dvipdfmx nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Convert DVI file to PDF with dvipdfmx")
            ("Ps2pdf" "ps2pdf %f %(O?pdf)" TeX-run-ps2pdf nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Convert PostScript file to PDF")
            ("Glossaries" "makeglossaries %(d-dir) %s" TeX-run-command
             nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run makeglossaries to create glossary file")
            ("Index" "makeindex %(O?idx)" TeX-run-index nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run makeindex to create index file")
            ("upMendex" "upmendex %(O?idx)" TeX-run-index t
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run upmendex to create index file")
            ("Xindy" "texindy %s" TeX-run-command nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run xindy to create index file")
            ("Check" "lacheck %s" TeX-run-compile nil (latex-mode)
             :help "Check LaTeX file for correctness")
            ("ChkTeX" "chktex -v6 %s" TeX-run-compile nil (latex-mode)
             :help "Check LaTeX file for common mistakes")
            ("Spell" "(TeX-ispell-document \"\")" TeX-run-function nil
             t :help "Spell-check the document")
            ("Clean" "TeX-clean" TeX-run-function nil t :help
             "Delete generated intermediate files")
            ("Clean All" "(TeX-clean t)" TeX-run-function nil t :help
             "Delete generated intermediate and output files")
            ("Other" "" TeX-run-command t t :help
             "Run an arbitrary command")
            )
 )
From 6df279065bd713c3a9b4524860631732da1519a1 Mon Sep 17 00:00:00 2001
From: David Fussner <[email-masked]>
Date: Mon, 20 Mar 2023 16:57:27 +0000
Subject: [PATCH 1/2] Fix xref error when matching in unvisited files

* font-latex.el (font-latex-set-syntactic-keywords): Add call to
font-lock-set-defaults to fix an error when xref-find-references tries
to fontify the search results in the " *xref-temp*" buffer.
---
 font-latex.el | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/font-latex.el b/font-latex.el
index 8fc567e2..5adda314 100644
--- a/font-latex.el
+++ b/font-latex.el
@@ -1116,9 +1116,14 @@ have changed."
   ;;       (append font-latex-syntactic-keywords
   ;;               ;; For docTeX comment-in-doc.
   ;;               '(("\\(\\^\\)\\^A" (1 (font-latex-doctex-^^A))))))
-  ;; Finally, compute our `syntax-propertize-function' anew.
+  ;; Compute our `syntax-propertize-function' anew.
   (setq-local syntax-propertize-function
-              (font-latex--make-syntax-propertize-function)))
+              (font-latex--make-syntax-propertize-function))
+  ;; We may be using the mode programmatically to extract data, and we
+  ;; then need this to be set up first so that a command like
+  ;; `xref-find-references' doesn't bug out when matching hits in
+  ;; files that Emacs isn't visiting.
+  (font-lock-set-defaults))

 ;;; Syntactic fontification
-- 
2.35.8

From f90c46777a8c7565a1cd34fe4abde62ac2f14d9f Mon Sep 17 00:00:00 2001
From: David Fussner <[email-masked]>
Date: Wed, 13 Sep 2023 10:14:40 +0100
Subject: [PATCH 2/2] Fix xref error when matching in unvisited files

* latex.el (LaTeX-common-initialization):
* plain-tex.el (plain-TeX-common-initialization): Add calls to
TeX-update-style to fix missed matches when xref-find-references
searches unvisited files.
---
 latex.el     | 8 +++++++-
 plain-tex.el | 8 +++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/latex.el b/latex.el
index 548b18a9..d4a5467a 100644
--- a/latex.el
+++ b/latex.el
@@ -8843,7 +8843,13 @@ function would return non-nil and `(match-string 1)' would return
   ;; Set LaTeX-specific help messages for error so that it's available
   ;; in `TeX-help-error'.
   (setq-local TeX-error-description-list-local
-              LaTeX-error-description-list))
+              LaTeX-error-description-list)
+  ;; Complete style initialization in buffers which don't visit files
+  ;; and which are therefore missed by the setting of `find-file-hook'
+  ;; in `VirTeX-common-initialization'.  This is necessary for
+  ;; `xref-find-references', for example.
+  (unless buffer-file-truename
+    (TeX-update-style)))
 
 (defun LaTeX-imenu-create-index-function ()
   "Imenu support function for LaTeX."
diff --git a/plain-tex.el b/plain-tex.el
index 0800b2f3..1e13acc6 100644
--- a/plain-tex.el
+++ b/plain-tex.el
@@ -244,7 +244,13 @@ of `plain-TeX-mode-hook'."
    "parindent"
    "baselineskip"
    "parskip")
-  (TeX-run-style-hooks "TEX"))
+  (TeX-run-style-hooks "TEX")
+  ;; Complete style initialization in buffers which don't visit files
+  ;; and which are therefore missed by the setting of `find-file-hook'
+  ;; in `VirTeX-common-initialization'.  This is necessary for
+  ;; `xref-find-references', for example.
+  (unless buffer-file-truename
+    (TeX-update-style)))

 ;;; Miscellaneous
-- 
2.35.8