GNU bug report logs - #66759
30.0.50; Flymake (with Eglot) error cleaning up old overlay

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 66759 in the body.
You can then email your comments to 66759 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Richard Copley <rcopley <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org,
 João Távora <joaotavora <at> gmail.com>
Subject: 30.0.50; Flymake (with Eglot) error cleaning up old overlay
Date: Thu, 26 Oct 2023 13:09:07 +0100

[Message part 1 (text/plain, inline)]

I'm afraid I'm unable to consistently reproduce this error. I hope you
can see the issue and devise a testcase from the following
description.

The function `flymake--publish-diagnostics' runs in two phases.
In phase 1 it deletes all the overlays referenced by the (now stale)
existing diagnostics. In phase 2 it creates new overlays for the newly
published diagnostics.

Phase 1 invokes `overlay-buffer' on the value of the :overlay property
of each diagnostic. This signals `wrong-type-argument' if the value is
nil.

Phase 2 intends to set the :overlay property of each diagnostic by
calling `flymake--highlight-line'.

The function `flymake--highlight-line' may return without setting the
:overlay property. For example:

    (when (= (overlay-start ov) (overlay-end ov))
      ;; Some backends report diagnostics with invalid bounds.  Don't
      ;; bother.
      (delete-overlay ov)
      (debug) ;; -- SEE BELOW (last line)
      (cl-return-from flymake--highlight-line nil))

This is likely to occur when `flymake-start` is called by the idle
timer, if there was a diagnostic near the end of the file and the user
has just made the file shorter. In that case, phase 2 inserts a
diagnostic with a null overlay into `flymake--state-diags'.

When `flymake--stated-diags' contains a diagnostic with a null
overlay, the next call to `flymake--publish-diagnostics' signals
`wrong-type-argument' during phase 1. If this next call also has
`flymake-start' in its call stack, the signal is caught in
`flymake--run-backend', which disables the backend.

If the backend continues to call its reporting function (as does Eglot
whenever the language server publishes diagnostics), this leads to a
flood of "Unexpected report from disabled backend %s" errors and an
unusable Emacs session.

A possible fix is to check if `flymake--highlight-line' created an
overlay before inserting a diagnostic into `flymake--state-diags',
in phase 2.

Another is to check that the :overlay property is not null before
attempting to examine and delete the overlay, in phase 1.

Attached is a backtrace from a real-life occurrence of the issue,
obtained by inserting `(debug)' (SEE ABOVE (quoted code)).

[backtrace.txt (text/plain, attachment)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: João Távora <joaotavora <at> gmail.com>
To: Richard Copley <rcopley <at> gmail.com>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: 30.0.50; Flymake (with Eglot) error cleaning up old overlay
Date: Thu, 26 Oct 2023 14:27:42 +0100

Richard Copley <rcopley <at> gmail.com> writes:

> I'm afraid I'm unable to consistently reproduce this error. I hope you
> can see the issue and devise a testcase from the following
> description.

Thanks very much for this report.  This problem could be the same as
https://github.com/joaotavora/eglot/discussions/1311, at least it its
most recent iteration.

Anyway, I think your analysis of the code is excellent and your
conjecture (for at least one possible cause of this problem) is very
promising.  That "don't bother with invalid bounds" was introduced
recently:

   commit 8b1947ffdd9d9eae26a308f0abaac45e06baac22
   Author: João Távora <joaotavora <at> gmail.com>
   Date:   Thu Sep 21 00:03:32 2023 +0100
    
       Flymake: more fixes to flymake--highlight-line
       
       Make it robust to diagonstics with invalid bounds.
       
       * lisp/progmodes/flymake.el (flymake--highlight-line): Robustify.
    
   diff --git a/lisp/progmodes/flymake.el b/lisp/progmodes/flymake.el
   --- a/lisp/progmodes/flymake.el
   +++ b/lisp/progmodes/flymake.el
   @@ -781,1 +782,5 @@
   -    (setq ov (make-overlay end beg))
   +    (when (= (overlay-start ov) (overlay-end ov))
   +      ;; Some backends report diagnostics with invalid bounds.  Don't
   +      ;; bother.
   +      (delete-overlay ov)
   +      (cl-return-from flymake--highlight-line nil)):


And indeed the flymake--diag-overlay slot is not filled in when we get
this early return.  And indeed the overlays considered for deletion are
the ones stored in the "state" map, meaning everything the backend
reported.

So maybe this patch is all that's needed:

diff --git a/lisp/progmodes/flymake.el b/lisp/progmodes/flymake.el
index b27e6527f81..9be40499d37 100644
--- a/lisp/progmodes/flymake.el
+++ b/lisp/progmodes/flymake.el
@@ -809,6 +809,7 @@ flymake--highlight-line
                         (flymake--diag-orig-end e))
                   (flymake--delete-overlay eov)))
     (setq ov (make-overlay beg end))
+    (setf (flymake--diag-overlay diagnostic) ov)
     (when (= (overlay-start ov) (overlay-end ov))
       ;; Some backends report diagnostics with invalid bounds.  Don't
       ;; bother.
@@ -863,7 +864,6 @@ flymake--highlight-line
     (overlay-put ov 'evaporate t)
     (overlay-put ov 'flymake-overlay t)
     (overlay-put ov 'flymake-diagnostic diagnostic)
-    (setf (flymake--diag-overlay diagnostic) ov)
     ;; Handle `flymake-show-diagnostics-at-end-of-line'
     ;;
     (when flymake-show-diagnostics-at-end-of-line


There's a fair chance this fixes the bug effectively, but even if it
doesn't, it is nevertheless a solid change, so I've pushed it and bumped
the Flymake ELPA package version.

Please keep an eye out of this bug.

What language server are you using with Eglot btw?

> A possible fix is to check if `flymake--highlight-line' created an
> overlay before inserting a diagnostic into `flymake--state-diags',
> in phase 2.

This could also work, but is slightly more complex.  And it would
destroy the invariant that that list contains every "domestic"
diagnostic reported by the backend (even invalid ones).

João

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Richard Copley <rcopley <at> gmail.com>
To: João Távora <joaotavora <at> gmail.com>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: 30.0.50; Flymake (with Eglot) error cleaning up old overlay
Date: Thu, 26 Oct 2023 15:17:34 +0100

On Thu, 26 Oct 2023 at 14:24, João Távora <joaotavora <at> gmail.com> wrote:
> And indeed the flymake--diag-overlay slot is not filled in when we get
> this early return.  And indeed the overlays considered for deletion are
> the ones stored in the "state" map, meaning everything the backend
> reported.
>
> So maybe this patch is all that's needed:
>
> diff --git a/lisp/progmodes/flymake.el b/lisp/progmodes/flymake.el
> index b27e6527f81..9be40499d37 100644
> --- a/lisp/progmodes/flymake.el
> +++ b/lisp/progmodes/flymake.el
> @@ -809,6 +809,7 @@ flymake--highlight-line
>                          (flymake--diag-orig-end e))
>                    (flymake--delete-overlay eov)))
>      (setq ov (make-overlay beg end))
> +    (setf (flymake--diag-overlay diagnostic) ov)
>      (when (= (overlay-start ov) (overlay-end ov))
>        ;; Some backends report diagnostics with invalid bounds.  Don't
>        ;; bother.
> @@ -863,7 +864,6 @@ flymake--highlight-line
>      (overlay-put ov 'evaporate t)
>      (overlay-put ov 'flymake-overlay t)
>      (overlay-put ov 'flymake-diagnostic diagnostic)
> -    (setf (flymake--diag-overlay diagnostic) ov)
>      ;; Handle `flymake-show-diagnostics-at-end-of-line'
>      ;;
>      (when flymake-show-diagnostics-at-end-of-line
>
>
> There's a fair chance this fixes the bug effectively, but even if it
> doesn't, it is nevertheless a solid change, so I've pushed it and bumped
> the Flymake ELPA package version.
>
> Please keep an eye out of this bug.

Thanks, will do.

> What language server are you using with Eglot btw?

Lean 4 [1]. There's a supporting Emacs mode [2] based on lsp-mode. I
have a fork which uses Eglot instead [3]. There's nothing missing from
Eglot, but one needs a lot of help from Lean 4 in order to read and
write programs and proofs in Lean 4. Btw, some in the community are
keen for the LSP semantic tokens feature to be implemented (see
[4][5]). The existing `font-lock keywords' in lean4-mode work to a
degree but leave something to be desired, since the language has
user-defined syntax.

[1] https://leanprover-community.github.io/learn.html
[2] https://github.com/leanprover/lean4-mode
[3] https://github.com/bustercopley/lean4-mode
[4] https://github.com/joaotavora/eglot/issues/615
[5] https://github.com/joaotavora/eglot/pull/839

> > A possible fix is to check if `flymake--highlight-line' created an
> > overlay before inserting a diagnostic into `flymake--state-diags',
> > in phase 2.
>
> This could also work, but is slightly more complex.  And it would
> destroy the invariant that that list contains every "domestic"
> diagnostic reported by the backend (even invalid ones).

Ah yes. And risky without a test case.

> João

Message #16 received at 66759-done <at> debbugs.gnu.org (full text, mbox):

From: João Távora <joaotavora <at> gmail.com>
To: Richard Copley <rcopley <at> gmail.com>
Cc: 66759-done <at> debbugs.gnu.org
Subject: Re: 30.0.50; Flymake (with Eglot) error cleaning up old overlay
Date: Thu, 26 Oct 2023 18:10:46 +0100

Richard Copley <rcopley <at> gmail.com> writes:

> On Thu, 26 Oct 2023 at 14:24, João Távora <joaotavora <at> gmail.com> wrote:
>> Please keep an eye out of this bug.
>
> Thanks, will do.

I've now reproduced the bug consistently.  It happens exactly as you
conjectured.  To test, I temporarily hacked Eglot to take 2 seconds
longer to process each request/notification with this patch:

diff --git a/lisp/progmodes/eglot.el b/lisp/progmodes/eglot.el
index eba66503bf7..48845a889a8 100644
--- a/lisp/progmodes/eglot.el
+++ b/lisp/progmodes/eglot.el
@@ -1488,7 +1488,7 @@ eglot--connect
                      ,@more-initargs)))))
          (spread (lambda (fn) (lambda (server method params)
                                 (let ((eglot--cached-server server))
-                                  (apply fn server method (append params nil))))))
+                                  (run-at-time 2 nil #'apply fn server method (append params nil))))))
          (server
           (apply
            #'make-instance class

And then made sure there was a diagnostic at the end of the file,
changed the file and quickly enough (but only after the changes were
sent to server), deleted the region with the diagnostic at the end of
the file.

Without my fix, I get exactly the (overlayp nil) error you reported.
With the fix, everything works correctly.

So I think with the fix I pushed earlier this bug can be closed, which
I'm doing now.

>> What language server are you using with Eglot btw?

> [1] https://leanprover-community.github.io/learn.html
> [2] https://github.com/leanprover/lean4-mode
> [3] https://github.com/bustercopley/lean4-mode
> [4] https://github.com/joaotavora/eglot/issues/615
> [5] https://github.com/joaotavora/eglot/pull/839

Yeah I know about these PRs.  If you want can start a new report in the
Emacs bug tracker proper (i.e. here) )and direct people to it.  Write a
summary of the situation if you can, and present this Lean4 use case as
a argument.

This gives visibility to more Emacs maintainers, and allows me to ping
specialists like Eli more easily on the matter.

João

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.