Subject: bug#76781: Fontification of expl3 code

From: Ikumi Keita <[email-masked]>
Date: Sat, 22 Mar 2025 17:56:58 +0900

Hi Arash,

>>>>> Arash Esbati <[email-masked]> writes:
> Thanks, and no worries.  I've installed the change with commit
> db11f39d.  I'm closing this report, we can reopen once you have a closer
> look and something is off.

I found out the origin of this bug. The culprit is that font lock is
turned on too early.

1. Let me begin the discussion with quote of the comment in tex.el for
   `find-file-hook':
,----
|   ;; Let `TeX-master-file' be called after a new file was opened and
|   ;; call `TeX-update-style' on any file opened. [...]
|   ;;
|   ;; `TeX-update-style' has to be called before
|   ;; `global-font-lock-mode', which may also be specified in
|   ;; `find-file-hook', gets called.  Otherwise style-based
|   ;; fontification will break (in XEmacs).  That means, `add-hook'
|   ;; cannot be called with a non-nil value of the APPEND argument.
|   ;;
|   ;; `(TeX-master-file nil nil t)' has to be called *before*
|   ;; `TeX-update-style' as the latter will call `TeX-master-file'
|   ;; without the `ask' bit set.
`----
   As this rather old comment implies, AUCTeX assumes that
   `TeX-update-style' runs _before_ font lock is turned on and style files
   are written under that assumption.

   If this assumtion holds, `font-lock-syntax-table' would be
   initialized after "expl3" style hook sets the syntax table by
   (set-syntax-table LaTeX-expl3-syntax-table)
   and fontification would be done as expected.

   However, modern emacsens actually don't satisfy that assumption.
   Major mode functions run `global-font-lock-mode-enable-in-buffer'
   (registered in `after-change-major-mode-hook'), which eventually
   runs `font-lock-set-defaults' which set `font-lock-syntax-table'
   at that time, copied from `docTeX-mode-syntax-table', not
   `LaTeX-expl3-syntax-table'. Thus the fontification was broken.

2. I think that there are two approaches to this bug.
   A: To move the call to `TeX-master-file' and `TeX-update-style' from
      `find-file-hook' to somewhere run _before_
      `after-change-major-mode-hook' and _after_ major mode hook and
      local variables entries are applied.
   B: To abandon the assumption in question and make a new AUCTeX policy
      to write style files assuming that style hooks are run after font
      lock is enabled.

   In my view, A isn't satisfactory:
   o It wouldn't be robust. The detail of the interaction between major
     mode function and turning on of font lock can be subject to further
     changes in future.
   o There is no suitable place for such "somewhere". Looking at the
     definition of `run-mode-hooks', the only possibility is to register
     the calls as local hook of `after-change-major-mode-hook' but I'm
     afraid that such treatment can break in future as described in the
     previous item.
   (On the other hand, there is a good aspect that we can merge call to
   `TeX-update-sytle' for non-file buffer in `TeX-mode-cleanup'.)

   If we take approach B, I think the proper solution is to use
   `font-latex-add-to-syntax-alist' instead of re-calling
   `font-lock-set-defaults' in expl3.el. See the attached patch.
   Additionally, we would have to revise similarly existing style files
   which install their own syntax tables. I expect this doesn't require
   much work; Here is the output of "grep -E set-syntax-table *.el"
   under style/ subdirectory:
brazilian.el:   (set-syntax-table LaTeX-brazilian-mode-syntax-table)
bulgarian.el:   (set-syntax-table LaTeX-bulgarian-mode-syntax-table)
expl3.el:   (set-syntax-table LaTeX-expl3-syntax-table)
german.el:   (set-syntax-table LaTeX-german-mode-syntax-table)
icelandic.el:   (set-syntax-table LaTeX-icelandic-mode-syntax-table)
ngerman.el:   (set-syntax-table LaTeX-german-mode-syntax-table)
plfonts.el:   (set-syntax-table LaTeX-plfonts-mode-syntax-table)
plhb.el:   (set-syntax-table LaTeX-plhb-mode-syntax-table)
polish.el:   (set-syntax-table LaTeX-polish-mode-syntax-table)
portuguese.el:   (set-syntax-table LaTeX-portuguese-mode-syntax-table)

What do you think about this?

Regards,
Ikumi Keita
#StandWithUkraine #StopWarInUkraine
#Gaza #StopMassiveKilling #CeasefireNOW

>From 80e5d1914e1c2a9be17b58a58c6427ed77e3ac54 Mon Sep 17 00:00:00 2001
From: Ikumi Keita <[email-masked]>
Date: Fri, 21 Mar 2025 03:39:46 +0900
Subject: [PATCH] temporal commit

---
 style/expl3.el | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/style/expl3.el b/style/expl3.el
index 60103dc4..e027343c 100644
--- a/style/expl3.el
+++ b/style/expl3.el
@@ -107,7 +107,11 @@ Pass OPTIONAL, PROMPT and NO-PARAM to `TeX-arg-expl3-macro', which see."
               (eq TeX-install-font-lock 'font-latex-setup))
      ;; Fontify _ and : as part of macros.
      (add-to-list 'font-latex-match-simple-include-list "_" t)
-     (add-to-list 'font-latex-match-simple-include-list ":" t))
+     (add-to-list 'font-latex-match-simple-include-list ":" t)
+     ;; We need the next line albeit we have `LaTeX-expl3-syntax-table',
+     ;; because `font-lock-set-defaults' is called before this style
+     ;; hook runs.
+     (font-latex-add-to-syntax-alist '((?\_ . "w") (?\: . "w"))))
 
    ;; Add expl3 macros to the parser.
    (TeX-auto-add-regexp LaTeX-expl3-newmac-regexp)
@@ -710,10 +714,7 @@ Pass OPTIONAL, PROMPT and NO-PARAM to `TeX-arg-expl3-macro', which see."
 
                                 ("cs_undefine:N" "\\")
                                 ("cs_undefine:c" "\\"))
-                              'function)
-     ;; Also tell font-lock to update its internals
-     (setq font-lock-major-mode nil)
-     (font-lock-set-defaults)))
+                              'function)))
  TeX-dialect)
 
 (defvar LaTeX-expl3-package-options-list
-- 
2.48.1