Subject: bug#77588: Catastrophic slowdown in eglot via `flymake-diag-region'

From: JD Smith <[email-masked]>
Date: Sun, 6 Apr 2025 18:50:26 -0400

I was diagnosing a painful eglot pause of about 10 seconds that would reliably occur every few edits when completing candidates and working in a long python file (~8k lines).  I am using the basedpyright LSP server.  I narrowed this down to eglot's `textDocument/publishDiagnostics' notification handler.

The file I was working on has several hundred warnings and errors, so that represents a substantial message from the LSP server.  After a fair bit of timing and sleuthing, I narrowed it down further.  A few interrelated misbehaviors combined to produce the huge intermittent pauses:

When completing text directly after entering/removing a newline, the Diagnostics ranges received by eglot are out of date and "one line off".
Some of these off-by-one-line diagnostic message ranges now land on blank lines in the file, so the effective range there is empty.
Seeing an empty range, eglot declares the server has "botched" it, so it tries to make a by-hand recalculation of the range using `flymake-diag-region'.
`flymake-diag-region' for some reason calls (end-of-thing 'sexp) and (end-of-thing 'symbol).  
In some positions in a long python file these commands are VERY SLOW.  Rather than the typical call time of ~1ms, at these certain positions `flymake-diag-region' takes ~400ms.

It doesn't take too many "botched eglot ranges" interacting with slow `thingatpt' misbehavior to add up to a 10s delay.

I'm not certain of the best solution.  A few ideas, from hardest to easiest:

Teach eglot textDocument/diagnostic
++++++++++++++++++++++++++++

 textDocument/publishDiagnostics messages arrive way too frequently IMO, after every buffer change of any kind.  They are pushed to eglot from the LSP server, and if they contain hundreds of errors, this becomes very inefficient (re-painting with flymake the same hundreds of regions over and over after each keystroke).

The best solution here would probably be to adopt "pull" diagnostics using textDocument/diagnostic, perhaps in an idle-timer whose duration the user can configure.  I don't believe EGLOT can do diagnostic pulls at the moment.

Don't use thingatpt in `flymake-diag-region'
+++++++++++++++++++++++++++++++++

`flymake-diag-region' should perhaps not use thingapt, which is subject to the performance vagaries of the major-mode and underlying file.  I am uncertain why it relies on that.  Perhaps the performance of those will be improved with treesitter variants.

Eglot could detect off-by-one diagnostics
+++++++++++++++++++++++++++++++

Hard to know the best heuristic, but lots of null effective ranges is a good one.

Eglot can simply ignore null range diagnostics
+++++++++++++++++++++++++++++++++++

Eglot doesn't need to use `flymake-diag-region' to try to calculate an update range if it encounters a null diagnostic range.  It could simply drop those, as they are probably wrong anyway, and will shortly be updated.