GNU bug report logs - #30964
python-mode failing "overlapping strings" assertion lately

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30964 in the body.
You can then email your comments to 30964 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: dancol <at> dancol.org
To: bug-gnu-emacs <at> gnu.org
Subject: python-mode failing "overlapping strings" assertion lately
Date: Tue, 27 Mar 2018 11:09:38 -0700

Has something changed in master that might be breaking python-mode? We
keep hitting a parsing assertion.

The surrounding code in python.el claims that

                       ;; The assertion can only fail if syntax table
                       ;; text properties and the `syntax-ppss' cache
                       ;; are somehow out of whack.  This has been
                       ;; observed when using `syntax-ppss' during
                       ;; narrowing.
                       (cl-assert (> string-start last-string-end)
                                  :show-args
                                  "\
Overlapping strings detected (start=%d, last-end=%d)")

I can _sometimes_ repro with (while (not (eobp))
(python-nav-forward-statement)), but I haven't found a reliable test case.

Message #8 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: dancol <at> dancol.org
Cc: 30964 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Sun, 01 Apr 2018 21:11:26 -0400

dancol <at> dancol.org writes:

> Has something changed in master that might be breaking python-mode? We
> keep hitting a parsing assertion.
>
> The surrounding code in python.el claims that
>
>                        ;; The assertion can only fail if syntax table
>                        ;; text properties and the `syntax-ppss' cache
>                        ;; are somehow out of whack.  This has been
>                        ;; observed when using `syntax-ppss' during
>                        ;; narrowing.
>                        (cl-assert (> string-start last-string-end)
>                                   :show-args
>                                   "\
> Overlapping strings detected (start=%d, last-end=%d)")
>
> I can _sometimes_ repro with (while (not (eobp))
> (python-nav-forward-statement)), but I haven't found a reliable test case.

How recent is "lately"?  Could it be related to [1: 14b9558752]?

[1: 14b9558752]: 2017-12-12 23:03:00 -0500
  * src/syntax.c (find_defun_start): Use syntax-ppss
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=14b95587520959c5b54356547a0a69932a9bb480

Message #11 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: charles <at> aurox.ch (Charles A. Roelli)
To: dancol <at> dancol.org
Cc: 30964 <at> debbugs.gnu.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Tue, 03 Apr 2018 20:12:48 +0200

> Date: Tue, 27 Mar 2018 11:09:38 -0700
> From: dancol <at> dancol.org
> 
> Has something changed in master that might be breaking python-mode? We
> keep hitting a parsing assertion.
> 
> The surrounding code in python.el claims that
> 
>                        ;; The assertion can only fail if syntax table
>                        ;; text properties and the `syntax-ppss' cache
>                        ;; are somehow out of whack.  This has been
>                        ;; observed when using `syntax-ppss' during
>                        ;; narrowing.
>                        (cl-assert (> string-start last-string-end)
>                                   :show-args
>                                   "\
> Overlapping strings detected (start=%d, last-end=%d)")
> 
> I can _sometimes_ repro with (while (not (eobp))
> (python-nav-forward-statement)), but I haven't found a reliable test case.

I saw it with the pretest (26.0.91) today, with no narrowing,
font-lock off, and after typing M-3 " to open a docstring.  I didn't
find a reproducible case either, though.

Message #14 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: "Charles A. Roelli" <charles <at> aurox.ch>, dancol <at> dancol.org
Cc: 30964 <at> debbugs.gnu.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Wed, 4 Apr 2018 01:06:26 +0300

On 4/3/18 9:12 PM, Charles A. Roelli wrote:

>> I can _sometimes_ repro with (while (not (eobp))
>> (python-nav-forward-statement)), but I haven't found a reliable test case.

You could try calling (syntax-ppss-flush-cache (point-min)) before that, 
for instance, and see if that changes something.

Anyway, an example file should help, even if the problem shows up only 
intermittently.

Message #17 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Clément Pit-Claudel <clement.pitclaudel <at> live.com>
To: 30964 <at> debbugs.gnu.org
Subject: bug#30964: python-mode failing "overlapping strings" assertion lately
Date: Fri, 18 May 2018 11:46:26 -0400

[Message part 1 (text/plain, inline)]

(Hoping that emailing the bug address is enough to respond to an existing bug, and that I didn't miss recent follow-ups)

(Resending, since I apparently emailed the wrong bug number at first)

Here's a reliable repro:

def f():
    "

def g():
    """That's a docstring"""
    return ''

Place the point after the unclosed quote in f, and run (python-info-current-defun).  You get this:

Debugger entered--Lisp error: (cl-assertion-failed ((> string-start last-string-end) "Overlapping strings detected (start=%d, last-end=%d)" 67 67))
  cl--assertion-failed((> string-start last-string-end) "Overlapping strings detected (start=%d, last-end=%d)" (67 67) nil)
  python-nav-end-of-statement()
  python-nav-end-of-defun()
  python-info-current-defun()
  eval((python-info-current-defun) nil)
  eval-expression((python-info-current-defun) nil nil 127)
  funcall-interactively(eval-expression (python-info-current-defun) nil nil 127)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun.

Clément.

[signature.asc (application/pgp-signature, attachment)]

Message #20 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Fri, 18 May 2018 21:18:23 +0200

On 18.05.2018 17:46, Clément Pit-Claudel wrote:

> This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun.
> 
> Clément.
> 
> 
> 

Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+ 
Version 3.14.5) of 2018-04-10

which-function-mode displays [f]

Message #23 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
Cc: 30964 <at> debbugs.gnu.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Fri, 18 May 2018 21:49:53 -0400

Andreas Röhler <andreas.roehler <at> easy-emacs.de> writes:

> On 18.05.2018 17:46, Clément Pit-Claudel wrote:
>
>> This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun.
>>
>> Clément.
>>
>>
>>
>
> Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+
> Version 3.14.5) of 2018-04-10

I can reproduce, if and only if I make sure to put a newline on the last
line.

Message #26 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: 30964 <at> debbugs.gnu.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Sat, 19 May 2018 08:49:40 +0200

[Message part 1 (text/plain, inline)]

On 19.05.2018 03:49, Noam Postavsky wrote:
> Andreas Röhler <andreas.roehler <at> easy-emacs.de> writes:
> 
>> On 18.05.2018 17:46, Clément Pit-Claudel wrote:
>>
>>> This error triggers in the wild for me due to which-function-mode, which calls python-info-current-defun.
>>>
>>> Clément.
>>>
>>>
>>>
>>
>> Can't reproduce with GNU Emacs 26.1 (build 1, i686-pc-linux-gnu, GTK+
>> Version 3.14.5) of 2018-04-10
> 
> I can reproduce, if and only if I make sure to put a newline on the last
> line.
> 

Okay, got the error at some point when inserting a newline. However a 
newline must not trigger it - see attachment.

Below some reports which might be of interest:

Which-Function mode enabled
next-line: End of buffer
Which-Function mode disabled
next-line: End of bufferError during redisplay: (jit-lock-function 3905) 
signaled (error "Overlapping strings detected")
Error during redisplay: (jit-lock-function 3905) signaled (error 
"Overlapping strings detected")
Which-Function mode enabled
Error during redisplay: (jit-lock-function 3919) signaled (error 
"Overlapping strings detected") [2 times]
Which-Function mode disabled
Error during redisplay: (jit-lock-function 3905) signaled (error 
"Overlapping strings detected") [2 times]
Which-Function mode enabled
C-c e is undefined
M-+ is undefined
C-x c is undefined
Which-Function mode disabled
Which-Function mode enabled
Mark set [4 times]
scroll-up-command: End of buffer
Mark set [2 times]
Saved text from "Which-Function mode disabled
next-line: "
Auto-saving...
Mark set [2 times]

[which-function-mode.png (image/png, attachment)]

Message #29 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Clément Pit-Claudel <clement.pitclaudel <at> live.com>
Cc: Philipp Stephani <p.stephani2 <at> gmail.com>, 30964 <at> debbugs.gnu.org,
 dancol <at> dancol.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Tue, 29 May 2018 20:29:26 -0400

[Message part 1 (text/plain, inline)]

Clément Pit-Claudel <clement.pitclaudel <at> live.com> writes:

> Here's a reliable repro:
>
> def f():
>     "
>
> def g():
>     """That's a docstring"""
>     return ''

It can be reduced a bit:

[bug-30964-py-assert.py (text/x-python, inline)]

'
docstring''

[Message part 3 (text/plain, inline)]

Debugger entered--Lisp error: (cl-assertion-failed ((> string-start last-string-end) "Overlapping strings detected" 13 13))
  cl--assertion-failed((> string-start last-string-end) "Overlapping strings detected" (13 13) nil)
  python-nav-end-of-statement()
  python-info-end-of-statement-p()
  python-info-end-of-block-p()
  python-nav--forward-sexp(-1 nil nil)
  python-nav-forward-sexp(-1 nil nil)
  python-nav-backward-sexp()
  python-info-docstring-p((0 nil nil t nil nil 0 nil 1 nil nil))
  python-font-lock-syntactic-face-function((0 nil nil t nil nil 0 nil 1 nil nil))
  font-lock-fontify-syntactically-region(1 15 nil)
  font-lock-default-fontify-region(1 15 nil)
  font-lock-fontify-region(1 15)
  font-lock-fontify-block(nil)
  funcall-interactively(font-lock-fontify-block nil)
  call-interactively(font-lock-fontify-block nil nil)
  command-execute(font-lock-fontify-block)

I think the problem is that the assertion is off by 1, it currently
asserts that there is at least one space between strings (because the
end position goes *after* the string, while the start position is part
of the string).

I see the assertion was added in [1: 4fbd330fae].  Phillip, does my
proposed change make sense to you?  ('make -C test python-tests' still
passes.)

--- i/lisp/progmodes/python.el
+++ w/lisp/progmodes/python.el
@@ -1506,7 +1506,7 @@ python-nav-end-of-statement
                        ;; are somehow out of whack.  This has been
                        ;; observed when using `syntax-ppss' during
                        ;; narrowing.
-                       (cl-assert (> string-start last-string-end)
+                       (cl-assert (>= string-start last-string-end)
                                   :show-args
                                   "Overlapping strings detected")
                        (goto-char string-start)

[1: 4fbd330fae]: 2017-03-23 23:05:19 +0100
  Protect against an infloop in python-mode
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4fbd330fae54a9c45d4a717127aa86d75e9938d5

Message #32 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Philipp Stephani <p.stephani2 <at> gmail.com>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: Clément Pit-Claudel <clement.pitclaudel <at> live.com>,
 30964 <at> debbugs.gnu.org, dancol <at> dancol.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Sat, 2 Jun 2018 21:03:10 +0200

[Message part 1 (text/plain, inline)]

Noam Postavsky <npostavs <at> gmail.com> schrieb am Mi., 30. Mai 2018 um
02:29 Uhr:

> Clément Pit-Claudel <clement.pitclaudel <at> live.com> writes:
>
> > Here's a reliable repro:
> >
> > def f():
> >     "
> >
> > def g():
> >     """That's a docstring"""
> >     return ''
>
> It can be reduced a bit:
>
>
> Debugger entered--Lisp error: (cl-assertion-failed ((> string-start
> last-string-end) "Overlapping strings detected" 13 13))
>   cl--assertion-failed((> string-start last-string-end) "Overlapping
> strings detected" (13 13) nil)
>   python-nav-end-of-statement()
>   python-info-end-of-statement-p()
>   python-info-end-of-block-p()
>   python-nav--forward-sexp(-1 nil nil)
>   python-nav-forward-sexp(-1 nil nil)
>   python-nav-backward-sexp()
>   python-info-docstring-p((0 nil nil t nil nil 0 nil 1 nil nil))
>   python-font-lock-syntactic-face-function((0 nil nil t nil nil 0 nil 1
> nil nil))
>   font-lock-fontify-syntactically-region(1 15 nil)
>   font-lock-default-fontify-region(1 15 nil)
>   font-lock-fontify-region(1 15)
>   font-lock-fontify-block(nil)
>   funcall-interactively(font-lock-fontify-block nil)
>   call-interactively(font-lock-fontify-block nil nil)
>   command-execute(font-lock-fontify-block)
>
> I think the problem is that the assertion is off by 1, it currently
> asserts that there is at least one space between strings (because the
> end position goes *after* the string, while the start position is part
> of the string).
>
> I see the assertion was added in [1: 4fbd330fae].  Phillip, does my
> proposed change make sense to you?


Maybe :-)
Off-by-one errors are always a bit subtle, so it might definitely be the
case that there is one. If so, the problem could be further reduced to
' '' '
or similar. Could you add a unit test using such input and make sure that
it breaks without your patch?



>   ('make -C test python-tests' still
> passes.)
>
> --- i/lisp/progmodes/python.el
> +++ w/lisp/progmodes/python.el
> @@ -1506,7 +1506,7 @@ python-nav-end-of-statement
>                         ;; are somehow out of whack.  This has been
>                         ;; observed when using `syntax-ppss' during
>                         ;; narrowing.
> -                       (cl-assert (> string-start last-string-end)
> +                       (cl-assert (>= string-start last-string-end)
>                                    :show-args
>                                    "Overlapping strings detected")
>                         (goto-char string-start)
>
> [1: 4fbd330fae]: 2017-03-23 23:05:19 +0100
>   Protect against an infloop in python-mode
>
> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4fbd330fae54a9c45d4a717127aa86d75e9938d5
>
>

[Message part 2 (text/html, inline)]

Message #35 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Philipp Stephani <p.stephani2 <at> gmail.com>
Cc: Clément Pit-Claudel <clement.pitclaudel <at> live.com>,
 30964 <at> debbugs.gnu.org, dancol <at> dancol.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Sat, 02 Jun 2018 16:35:50 -0400

[Message part 1 (text/plain, inline)]

Philipp Stephani <p.stephani2 <at> gmail.com> writes:

>> I think the problem is that the assertion is off by 1, it currently
>> asserts that there is at least one space between strings (because the
>> end position goes *after* the string, while the start position is part
>> of the string).
>>
>> I see the assertion was added in [1: 4fbd330fae].  Phillip, does my
>> proposed change make sense to you?
>
>
> Maybe :-)
> Off-by-one errors are always a bit subtle, so it might definitely be the
> case that there is one. If so, the problem could be further reduced to
> ' '' '
> or similar. Could you add a unit test using such input and make sure that
> it breaks without your patch?

Yeah, the newlines are the important part it seems.

[0001-Fix-off-by-one-error-in-python-mode-assertion-Bug-30.patch (text/x-diff, inline)]

From 209506955227d3b4de030f817c087bf27e5a7426 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs <at> gmail.com>
Date: Sat, 2 Jun 2018 16:22:17 -0400
Subject: [PATCH] Fix off by one error in python-mode assertion (Bug#30964)

* lisp/progmodes/python.el (python-nav-end-of-statement): Don't assert
that string-start is strictly greater than last-string-end, because
the string end is a position outside of the string and may therefore
be the same as the following string's start.
* test/lisp/progmodes/python-tests.el (python-nav-end-of-statement-2):
New test.
---
 lisp/progmodes/python.el            | 2 +-
 test/lisp/progmodes/python-tests.el | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index c7bb2d97c8..87d2f08fee 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -1506,7 +1506,7 @@ python-nav-end-of-statement
                        ;; are somehow out of whack.  This has been
                        ;; observed when using `syntax-ppss' during
                        ;; narrowing.
-                       (cl-assert (> string-start last-string-end)
+                       (cl-assert (>= string-start last-string-end)
                                   :show-args
                                   "Overlapping strings detected")
                        (goto-char string-start)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 4955da02a2..375bd9588d 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -2004,6 +2004,12 @@ python-tests-visible-string
                 (python-util-forward-comment -1)
                 (point))))))
 
+(ert-deftest python-nav-end-of-statement-2 ()
+  "Test the string overlap assertion (Bug#30964)."
+  (python-tests-with-temp-buffer
+   "'\n''\n"
+   (python-nav-end-of-statement)))
+
 (ert-deftest python-nav-forward-statement-1 ()
   (python-tests-with-temp-buffer
    "
-- 
2.11.0

Message #38 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: "Daniel Colascione" <dancol <at> dancol.org>
To: "Noam Postavsky" <npostavs <at> gmail.com>
Cc: "Clément Pit-Claudel"
 <clement.pitclaudel <at> live.com>, Philipp Stephani <p.stephani2 <at> gmail.com>,
 30964 <at> debbugs.gnu.org, dancol <at> dancol.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Sat, 16 Jun 2018 13:55:49 -0700

So it turns out that it wasn't an off-by-one problem at all.  Instead,
python was using (re-search-forward (rx (syntax string-delimiter))), which
was matching incorrectly because re-search-forward didn't update syntax as
it matched. It didn't do that because regex matching in general wasn't
reentrant, and it wasn't reentrant because regex bytecode programs are
self-modifying.

I checked in some work that makes regex searching update syntax as needed.

Message #43 received at 30964 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: "Daniel Colascione" <dancol <at> dancol.org>
Cc: Clément Pit-Claudel <clement.pitclaudel <at> live.com>,
 Philipp Stephani <p.stephani2 <at> gmail.com>, 30964 <at> debbugs.gnu.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Sat, 16 Jun 2018 18:36:29 -0400

"Daniel Colascione" <dancol <at> dancol.org> writes:

> So it turns out that it wasn't an off-by-one problem at all.  Instead,
> python was using (re-search-forward (rx (syntax string-delimiter))), which
> was matching incorrectly because re-search-forward didn't update syntax as
> it matched. It didn't do that because regex matching in general wasn't
> reentrant, and it wasn't reentrant because regex bytecode programs are
> self-modifying.
>
> I checked in some work that makes regex searching update syntax as needed.

It seems there were two separate problems.  The off-by-one error still
happens after your changes, so I pushed my patch to master.

[1: 05345babc9]: 2018-06-16 18:34:19 -0400
  Fix off by one error in python-mode assertion (Bug#30964)
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=05345babc988060cca540770599282102c34f2a7

Message #46 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#30964: python-mode failing "overlapping strings" assertion
 lately
Date: Sun, 17 Jun 2018 09:33:09 +0200

On 17.06.2018 00:36, Noam Postavsky wrote:
> "Daniel Colascione" <dancol <at> dancol.org> writes:
> 
>> So it turns out that it wasn't an off-by-one problem at all.  Instead,
>> python was using (re-search-forward (rx (syntax string-delimiter))), which
>> was matching incorrectly because re-search-forward didn't update syntax as
>> it matched. It didn't do that because regex matching in general wasn't
>> reentrant, and it wasn't reentrant because regex bytecode programs are
>> self-modifying.
>>
>> I checked in some work that makes regex searching update syntax as needed.
> 
> It seems there were two separate problems.  The off-by-one error still
> happens after your changes, so I pushed my patch to master.
> 
> [1: 05345babc9]: 2018-06-16 18:34:19 -0400
>    Fix off by one error in python-mode assertion (Bug#30964)
>    https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=05345babc988060cca540770599282102c34f2a7
> 
> 
> 

Hi folks,

sorry, but as long as syntax-ppss has fundamentally issues, all fixing 
here will be vain. For the very same reason don't be surprised having 
not "found a reliable test case".

Cheers,
Andreas

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.