Subject: bug#69431: 30.0.50; Strange fontificaion behavior

From: Andrea Corallo <[email-masked]>
Date: Sun, 31 Mar 2024 16:40:34 -0400

Andrea Corallo <[email-masked]> writes:

> Stefan Monnier <[email-masked]> writes:
>
>>>>> I'm trying to progress on this but I've difficult time at getting a grip
>>>>> on this bug.  One of the reason why is that on my reproducer Emacs
>>>>> doesn't crash nor complain, the other reason is that I'm not really into
>>>>> our fontification.  I'll keep on looking into it but I'd appretiate any
>>>>> hint like: what should be happening to fontify the buffer that is not
>>>>> actually happening here?  With such info I could try investingating the
>>>>> issue from there comparing with the working case.
>>>>
>>>> Hmm... here, after
>>>>
>>>>     rm -rf ~/.emacs.d/eln-cache
>>>>     src/emacs -Q --eval '(setq debug-on-error t)' a.org
>>>>
>>>> I get a buffer with no fontification, that claims to be in Org mode but
>>>> whose `font-lock-keywords` is just (t nil), which explains the lack
>>>> of fontification.
>>>
>>> Where `font-lock-keywords` are supposed to be set?
>>
>> `font-lock-keywords` is normally set from `font-lock-defaults` (which is
>> set by the major-mode) by `font-lock-set-defaults` when `font-lock-mode`
>> is enabled (which usually happens soon after enabling the major mode,
>> via `after-change-major-mode-hook`).
>
> Sorry didn't had much time to progress on this.
>
> I see `font-lock-mode` and `font-lock-set-defaults` are called when
> opening an org file here on my reproducer.
>
> Also I see `font-lock-set-defaults` is actually set! So I'm wondering
> why afterwards the value changes for my test.org.
>
> Will look into it more hopefully tomorrow.

Driven by curiosity I went a little further today...

I've applied the following:

================

1 file changed, 6 insertions(+)
lisp/font-lock.el | 6 ++++++

modified   lisp/font-lock.el
@@ -1874,6 +1874,10 @@ font-lock-refresh-defaults
 (defvar-local font-lock-major-mode nil
   "Major mode for which the font-lock settings have been setup.")

+(defun k-variable-watcher (_symbol newval op _buffer)
+  (message "XXX %s" op)
+  (backtrace))
+
 (defun font-lock-set-defaults ()
   "Set fontification defaults appropriately for this mode.
 Sets various variables using `font-lock-defaults' and
@@ -1934,6 +1938,8 @@ font-lock-set-defaults
       (unless (eq (car font-lock-keywords) t)
         (setq font-lock-keywords
               (font-lock-compile-keywords font-lock-keywords))))
+    (when (string= (buffer-name) "test.org")
+      (add-variable-watcher 'font-lock-keywords #'k-variable-watcher))
     (font-lock-flush)))
 ^L
 ;;; Color etc. support.

================

And this is what it logs on *Messages* depending on the fact that the
eln-cache is wiped or not:

Warm eln-cache (working fontification)
================
For information about GNU Emacs and the GNU system, type C-h C-a.
XXX set
  backtrace()
  normal-mode(t)
  after-find-file(nil t)
  find-file-noselect-1(#<buffer test.org> "~/test.org" nil nil "~/test.org" (14180533 64513))
  find-file-noselect("/home/andcor03/test.org")
  command-line-1(("-eval" "(setq native-comp-jit-compilation t)" "/home/andcor03/test.org"))
  command-line()
  normal-top-level()

XXX set
  backtrace()
  font-lock-fontify-keywords-region(1 7 nil)
  font-lock-default-fontify-region(1 7 nil)
  font-lock-fontify-region(1 7)
  #f(compiled-function (fun) #<bytecode 0x1baf782d70c71cbf>)(font-lock-fontify-region)
  jit-lock--run-functions(1 7)
  jit-lock-fontify-now(1 7)
  jit-lock-function(1)
  redisplay_internal\ \(C\ function\)()
================

Empty eln-cache (fontification broken):
================
For information about GNU Emacs and the GNU system, type C-h C-a.
XXX makunbound
  backtrace()
  k-variable-watcher(font-lock-keywords nil makunbound #<buffer test.org>)
  kill-local-variable(font-lock-keywords)
  org-set-font-lock-defaults()
  org-mode()
  set-auto-mode-0(org-mode nil)
text-mode) ("\\.[tT]e[xX]\\'" . tex-mode) ("\\.ins\\'" . tex-mode) ("\\.ltx\\'" . latex-mode) ("\\.dtx\\'" . doctex-mode) ...) nil nil)
  set-auto-mode()
  normal-mode(t)
  after-find-file(nil t)
  find-file-noselect-1(#<buffer test.org> "~/test.org" nil nil "~/test.org" (14180533 64513))
  find-file-noselect("/home/andcor03/test.org")
  command-line-1(("-eval" "(setq native-comp-jit-compilation t)" "/home/andcor03/test.org"))
  command-line()
  normal-top-level()

XXX set
  backtrace()
  k-variable-watcher(font-lock-keywords (t nil) set nil)
  font-lock-fontify-keywords-region(1 7 nil)
  font-lock-default-fontify-region(1 7 nil)
  font-lock-fontify-region(1 7)
  #f(compiled-function (fun) #<bytecode 0x1baf3a4708041cbf>)(font-lock-fontify-region)
  jit-lock--run-functions(1 7)
  jit-lock-fontify-now(1 7)
  jit-lock-function(1)
  redisplay_internal\ \(C\ function\)()

================

So from what I see in the non working example 'font-lock-keywords' is
cleared at the end of 'org-set-font-lock-defaults' which does
'(kill-local-variable 'font-lock-keywords)'.

Now why this is not happening in the working example I'm not sure ATM,
('org-set-font-lock-defaults' is not even called there AFAICS).

Maybe org maintainers can comment if this is expected or not, or explain
meanwhile the mechanism so debug will be easier.

Thanks!

  Andrea