Subject: bug#60691: closed (Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode)

From: [email-masked] (GNU bug Tracking System)
Date: Mon, 30 Jan 2023 00:16:02 +0000

Your bug report

#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
60691: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=60691
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode
From: Dmitry Gutov <[email-masked]>
To: Yuan Fu <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Mon, 30 Jan 2023 02:15:44 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

On 30/01/2023 01:23, Yuan Fu wrote:
> 
>> On Jan 29, 2023, at 3:07 PM, Dmitry Gutov<[email-masked]>  wrote:
>>
>> Hi Yuan,
>>
>> On 29/01/2023 10:25, Yuan Fu wrote:
>>
>>>>>> So if previously it happened once somehow during a certain scenario, now I have to repeat the same scenario 4 times, and the condition is met.
>>>>> I was hoping that the scenario only happen once, oh well 😄 I’ll
>>>>> change the decision based on analyzing the tree’s dimension: too
>>>>> deep or too wide activates the fast mode. Let’s see how it works.
>>>> Thank you, let me know when it's time to test again.
>>> Sorry for the delay. Now treesit-font-lock-fontify-region uses
>>> treesit-subtree-stat to determine whether to enable the "fast mode". Now
>>> it should be impossible to activate the fast mode on moderately sized
>>> buffers.
>> Thank you, it seems to work just fine in my scenario. And treesit-subtree-stat makes sense.
>>
>> I have a few more questions about the current strategy, though.
>>
>> IIUC, we only do the treesit--font-lock-fast-mode test once in treesit-font-lock-fontify-region, and then use the detected value for the whole later life of the buffer. Is that right?
>>
>> What if the buffer didn't originally have the problematic error nodes we are guarding from, and then later the user wrote enough code to have at least one of them? If they didn't close Emacs, or revert the buffer, our logic still wouldn't use the "fast node", would it?
>>
>> Or vice versa: if the buffer started out with error nodes, and consequently, "fast mode", but then the user has edited it so that those error nodes disappeared, shouldn't the buffer stop using the "fast mode"?
>>
>>  From my measurements, in ruby-mode, at least treesit-subtree-stat is 20-40x faster than refontifying the whole buffer. So one possible strategy would be to repeat the test every time. I'm not sure it's fast enough in the "problem" buffers, though, and I don't have any to test.
>>
>> In those I did test, though, it takes ~1 ms.
>>
>> But we could repeat the test only once every couple of seconds and/or after the buffer has changed again. That would hopefully make it a non-bottleneck in all cases.
> I should mention this in the comments, but the fast mode is only for very rare cases, where the file is mechanically generated and has some peculiarities that causes tree-sitter to work poorly. If the file is hand-written and “normal”, even huge files like xdisp.c is well below the bar. Therefore I don’t think “crossing the line” will realistically happen when editing source files.
> 
> Here is the stats of two “problematic files”, named packet and dec_mask, comparing to xdisp.c:
> 
> ;;           max-depth max-width count
> ;; cut-off   100       4000
> ;; packet   (98159     46581 1895137)
> ;; dec mask (3         64301 283995)
> ;; xdisp.c  (29        985   218971)
> 
> I’d say that any regular source file, even mechanically generated, wouldn’t go beyond ~50 levels in depth, and hand-written files should never has a node that has 4000+ direct children in the parse tree.

Oh, thanks for the explanation. Then the current strategy makes sense.

Is xdisp.c absolutely the largest C file in your experience?

According to the above numbers, a file that's only 4x as large could hit 
our current cutoff.

Though, TBH, maybe some extreme files do, and they have font-lock 
performance reduced somewhat. That's not the end of the world, and it 
shouldn't make a difference for the original scenario (diff-syntax 
fontification).

Either way, I'm closing this report. Thank you for your help.

Subject: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode
From: Juri Linkov <[email-masked]>
To: [email-masked]
Date: Mon, 09 Jan 2023 19:16:12 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

X-Debbugs-Cc: Dmitry Gutov <[email-masked]>

After more rules were added recently to ruby-ts--font-lock-settings,
font-lock became slow even on very small files.  Some measurements:

M-: (benchmark-run 1000 (progn (font-lock-mode -1) (font-lock-mode 1) (font-lock-ensure)))

M-x ruby-mode
(1.3564674989999999 0 0.0)

M-x ruby-ts-mode
(8.349582391999999 2 6.489918534000001)

This is not a problem when files are visited infrequently, but
becomes a problem for diff-syntax fontification that wants to
highlight simultaneously many files from git logs.
So a temporary measure would be not to enable ruby-ts-mode
in internal buffers:

(add-hook 'find-file-hook
          (lambda ()
            (when (and (eq major-mode 'ruby-mode)
                       ;; Only when not internal as from diff-syntax
                       (not (string-prefix-p " " (buffer-name))))
              (ruby-ts-mode))))