Subject: Re: bug#56682: Fix the long lines font locking related slowdowns

From: Dmitry Gutov <[email-masked]>
Date: Fri, 5 Aug 2022 22:01:24 +0300

On 05.08.2022 21:14, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 21:02:35 +0300
>> Cc: [email-masked], [email-masked], [email-masked]
>> From: Dmitry Gutov <[email-masked]>
>>
>>> How is (parse-partial-sexp 1 (point-max)) related to the issue at
>>> hand?
>>
>> Or perhaps I should answer this way:
>>
>> We move to near EOB.
>> fontification-functions are called.
>>
>> jit-lock
>> calls
>> (font-lock-fontify-region point-near-buffer-end (point-max))
>> which calls
>> font-lock-fontify-syntactically-region
>> which calls both
>>     (syntax-propertize (point-max))
>>     and
>>     (syntax-ppss point-near-buffer-end) -> and it calls parse-partial-sexp
>>
>> syntax-propertize will also likely call syntax-ppss itself, probably
>> through the major mode's syntax-propertize-function. But if
>> syntax-propertize-function is nil, parse-partial-sexp gets called
>> anyway, over the whole buffer, which makes it the main workload in
>> fontifying near EOB.
>>
>> Now, if syntax-propertize-function is non-nil, parse-partial-sexp will
>> also call it, and it adds its overhead (sometimes a multiple of p-p-s),
>> which also scales linearly with the length of the buffer.
>>
>> So if one can demonstrate that (parse-partial-sexp (point-min)
>> (point-max)) takes about the same time as it takes to fontify the last
>> screen-ful of a buffer, then that says that everything else that
>> jit-lock does to fontify, is negligible, time-wise.
> 
> So you have demonstrated that, if visiting a file and moving inside it
> calls parse-partial-sexp to scan the entire buffer, then this could be
> some, perhaps a large, part of the slowdown.

Yes.

> First, we need to establish that indeed parse-partial-sexp is called
> in that manner in the relevant major modes (not just one of them), or
> by font-lock itself regardless of the mode.

It is called by font-lock itself, which ends up calling syntax-ppss, 
which does its job with parse-partial-sexp. I have outlined the chain of 
calls in the previous message, you can verify it by looking at the sources.

> Second, we need to establish that indeed this takes a large portion of
> the time in the slow operations.  Not just one particular operation,
> but most or all of them.

To establish that, I have described the experiment in the grandparent 
email (with scenarios 1,2a;1,2b;1,2a,2b), and performed it myself as well.

But I'm talking about the slowdown observed when doing 'M->'. Not about 
any operations one might try to perform. Having said that, after the 
initial 'M->' most of navigation operations look snappy to me. So that's 
the slowdown I decided to investigate.

> And after that, we may have some food for thought.

Here's some more:

All major modes we can currently use for JSON (the built-in js-mode and 
the two json-mode's in ELPA) inherit the value of 
syntax-propertize-function from js-mode. But there's no need for it: 
JSON doesn't have division, or regexps, or preprocessor directives, or 
embedded JSX structures.

Setting syntax-propertize-function to nil speeds up parse-partial-sexp 
significantly. Here's a patch you can try to evaluate the effect on 
dictionary.json of that change combined with the previous tweak I 
suggested. Now it takes about 5x faster to fontify the last screenful, 
on my machine. Meaning, 'M->' feels almost (but not quite) instant. And 
the fontification is still correct.

A "proper" change would involve creating a new major mode, probably, 
rather than regexp-matching against buffer-file-name. But I'm not sure 
what name to pick: 'json-mode' would step on the toes of two existing 
packages now. 'js-json-mode', maybe? Or we bring in json-mode from GNU 
ELPA (with a similar change).

Anyway, try this please:

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index eb2a1e4fcc..ae8e980125 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3418,7 +3418,8 @@ js-mode
                (list js--font-lock-keywords nil nil nil nil
                      '(font-lock-syntactic-face-function
                        . js-font-lock-syntactic-face-function)))
-  (setq-local syntax-propertize-function #'js-syntax-propertize)
+  (unless (and buffer-file-name (string-match-p "\\.json\\'" 
buffer-file-name))
+    (setq-local syntax-propertize-function #'js-syntax-propertize))
    (add-hook 'syntax-propertize-extend-region-functions
              #'syntax-propertize-multiline 'append 'local)
    (add-hook 'syntax-propertize-extend-region-functions
diff --git a/src/xdisp.c b/src/xdisp.c
index 099efed2db..fcb2be8768 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4391,20 +4391,6 @@ handle_fontified_prop (struct it *it)

        eassert (it->end_charpos == ZV);

-      if (current_buffer->long_line_optimizations_p)
-	{
-	  ptrdiff_t begv = it->narrowed_begv;
-	  ptrdiff_t zv = it->narrowed_zv;
-	  ptrdiff_t charpos = IT_CHARPOS (*it);
-	  if (charpos < begv || charpos > zv)
-	    {
-	      begv = get_narrowed_begv (it->w, charpos);
-	      zv = get_narrowed_zv (it->w, charpos);
-	    }
-	  narrow_to_region_internal (make_fixnum (begv), make_fixnum (zv), true);
-	  specbind (Qrestrictions_locked, Qt);
-	}
-
        /* Don't allow Lisp that runs from 'fontification-functions'
  	 clear our face and image caches behind our back.  */
        it->f->inhibit_clear_image_cache = true;