GNU bug report logs - #60691
29.0.60; Slow tree-sitter font-lock in ruby-ts-mode

Previous Next

Full log

Message #29 received at 60691 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Yuan Fu <casouri <at> gmail.com>
Cc: 60691 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode
Date: Fri, 13 Jan 2023 01:40:56 +0200

On 12/01/2023 23:58, Yuan Fu wrote:
> 
> Dmitry Gutov <dgutov <at> yandex.ru> writes:
> 
>> Yuan? Just making sure you got this message.
> 
> Sorry for the delay :-)
> 
>> On 10/01/2023 16:10, Dmitry Gutov wrote:
>>> Perhaps Yuan has some further ideas. There are some strong oddities here:
>>> - Some time into debugging and repeating the benchmark again and
>>> again, I get the "Pure Lisp storage overflowed" message. Just once
>>> per Emacs session. It doesn't seem to change much, so it might be
>>> unimportant.
> 
> That sounds like 60653. The next time you encounter it, could you record
> the output of M-x memory-usage and M-x memory-report?

Managed to reproduce this after running the test in a couple of 
different files.

But 'M-x memory-usage' says no such command, and 'M-x memory-report' 
ends up with this error:

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  memory-report--gc-elem(nil strings)
  memory-report--garbage-collect()
  memory-report()
  funcall-interactively(memory-report)
  #<subr call-interactively>(memory-report record nil)
  apply(#<subr call-interactively> memory-report (record nil))
  call-interactively <at> ido-cr+-record-current-command(#<subr 
call-interactively> memory-report record nil)
  apply(call-interactively <at> ido-cr+-record-current-command #<subr 
call-interactively> (memory-report record nil))
  call-interactively(memory-report record nil)
  command-execute(memory-report record)
  execute-extended-command(nil "memory-report" nil)
  funcall-interactively(execute-extended-command nil "memory-report" nil)
  #<subr call-interactively>(execute-extended-command nil nil)
  apply(#<subr call-interactively> execute-extended-command (nil nil))
  call-interactively <at> ido-cr+-record-current-command(#<subr 
call-interactively> execute-extended-command nil nil)
  apply(call-interactively <at> ido-cr+-record-current-command #<subr 
call-interactively> (execute-extended-command nil nil))
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

garbage-collect's docstring says:

  However, if there was overflow in pure space, and Emacs was dumped
  using the "unexec" method, ‘garbage-collect’ returns nil, because
  real GC can’t be done.

I don't know if my Emacs was dumped using "unexec", though. ./configure 
says I'm using pdumper.

In case that matters, I'm testing the emacs-29 branch.

>>> - The profiler output looks like this:
>>>     18050  75%                    -
>>> font-lock-fontify-syntactically-region
>>>     15686  65%                     - treesit-font-lock-fontify-region
>>>      3738  15% treesit--children-covering-range-recurse
>>>       188   0%                        treesit-fontify-with-override
>>> - When running the benchmark for the first time in a buffer (such as
>>> ruby.rb), the variable treesit--font-lock-fast-mode is usually
>>> changed to t. In one Emacs session, after I changed it to nil and
>>> re-ran the benchmark, the variable stayed nil, and the benchmark ran
>>> much faster (like 10s vs 36s).
>>> In the next session, after I restarted Emacs, that didn't happen: it
>>> always stayed at t, even if I reset it to nil between runs. But if I
>>> comment out the block in treesit-font-lock-fontify-region that uses
>>> it
>>>       ;; (when treesit--font-lock-fast-mode
>>>       ;;   (setq nodes (treesit--children-covering-range-recurse
>>>       ;;                (car nodes) start end (* 4 jit-lock-chunk-size))))
>>> and evaluate the defun, the benchmark runs much faster again: 11s.
>>> (But then I brought it all back, and re-ran the tests, and the
>>> variable stayed nil that time around; to sum up: the way it's turned
>>> on is unstable.)
>>> Should treesit--font-lock-fast-mode be locally bound inside that
>>> function, so that it's reset between chunks? Or maybe the condition
>>> for its enabling should be tweaked? E.g. I don't think there are any
>>> particularly large or deep nodes in ruby.rb's parse tree. It's a
>>> very shallow file.
> 
> Yeah that is a not-very-clever hack. I’ve got an idea: I can add a C
> function that checks the maximum depth of a parse tree and the maximum
> node span, and turn on the fast-mode if the depth is too large or a node
> is too wide. And we do that check once before doing any fontification.
> 
> I’ll report back once I add it.

Thanks!

And if the check can be fast enough, we could probably do it in the 
beginning of fontifying every chunk.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.