GNU bug report logs - #59679
29.0.50; python + tree-sitter: Syntax highlighting is sometimes wrong with multi line docstrings

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 59679 in the body.
You can then email your comments to 59679 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Matthias Paulmier <hello <at> mattplm.fr>
To: <bug-gnu-emacs <at> gnu.org>
Subject: 29.0.50; python + tree-sitter: Syntax highlighting is sometimes wrong
 with multi line docstrings
Date: Tue, 29 Nov 2022 13:05:29 +0100

Steps to reproduce:
- emacs -Q
- find-file "foo.py"
- write the following file

```
def test():
     '''This is a function

     It just does nothing and it's ok.
     '''
     pass
```
- at this point, syntax highlighting should be good
- kill-buffer
- find-file "foo.py"
- syntax highlighting doesn't show properly (comment ends after "it'"
and pass is shown with the `font-lock-doc-face')


In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.20, cairo version 1.16.0) of 2022-11-23 built on 20045CLT490
Repository revision: 43060910874f46579fa190d9b5534508e4ea058e
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Ubuntu 20.04.5 LTS

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
value of $LC_ALL: fr_FR.UTF-8
value of $LC_CTYPE: fr_FR.UTF-8
value of $LANG: fr_FR
locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
line-number-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils vc-hg
vc-git diff-mode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs log-view
easy-mmode pcvs-util vc vc-dispatcher python rx project byte-opt gv
bytecomp byte-compile pcase treesit cl-seq comint ansi-osc ring
cl-loaddefs cl-lib ansi-color rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
emacs)

Memory information:
((conses 16 67878 7612)
(symbols 48 8488 0)
(strings 32 25282 1967)
(string-bytes 1 742048)
(vectors 16 16094)
(vector-slots 8 211336 10774)
(floats 8 30 47)
(intervals 56 389 0)
(buffers 984 13))

Message #8 received at 59679 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: hello <at> mattplm.fr
Cc: 59679 <at> debbugs.gnu.org
Subject: Re: bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting  is
 sometimes wrong with multi line docstrings
Date: Wed, 30 Nov 2022 02:31:54 -0800

Matthias Paulmier <hello <at> mattplm.fr> writes:

> Steps to reproduce:
> - emacs -Q
> - find-file "foo.py"
> - write the following file
>
> ```
> def test():
>      '''This is a function
>
>      It just does nothing and it's ok.
>      '''
>      pass
> ```
> - at this point, syntax highlighting should be good
> - kill-buffer
> - find-file "foo.py"
> - syntax highlighting doesn't show properly (comment ends after "it'"
> and pass is shown with the `font-lock-doc-face')

Hmmm, it seems the tabs in the code throws tree-sitter off and makes it
parse incorrectly. But I’m not sure if the tabs are in your code or they
are added by the email provider. Do you expect the code to contain tabs?

Yuan

Message #11 received at 59679 <at> debbugs.gnu.org (full text, mbox):

From: Matthias Paulmier <hello <at> mattplm.fr>
To: Yuan Fu <casouri <at> gmail.com>
Cc: 59679 <at> debbugs.gnu.org
Subject: Re: bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting is
 sometimes wrong with multi line docstrings
Date: Wed, 30 Nov 2022 12:06:01 +0100

> Hmmm, it seems the tabs in the code throws tree-sitter off and makes it
> parse incorrectly. But I’m not sure if the tabs are in your code or they
> are added by the email provider. Do you expect the code to contain tabs?

Hello Yuan,

Thank you for looking at the issue.

The actual file doesn't contain any tabs (they are 4 spaces indent), 
must be my email client trying to format it.

After reflecting a bit, I'm not sure it is related in any way to 
treesitter. I'm not even sure python + treesitter is a thing in emacs in 
29.0.50 (not according to etc/NEWS at least).

I can just confirm that versions <28.2 don't show this issue, that's why 
I thought about treesitter.

--
Matthias

Message #14 received at 59679 <at> debbugs.gnu.org (full text, mbox):

From: Matthias Paulmier <hello <at> mattplm.fr>
To: Yuan Fu <casouri <at> gmail.com>
Cc: 59679 <at> debbugs.gnu.org
Subject: Re: bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting is
 sometimes wrong with multi line docstrings
Date: Wed, 30 Nov 2022 17:50:41 +0100

On 30/11/2022 12:06, Matthias Paulmier wrote:
>
>> Hmmm, it seems the tabs in the code throws tree-sitter off and makes it
>> parse incorrectly. But I’m not sure if the tabs are in your code or they
>> are added by the email provider. Do you expect the code to contain tabs?
>
> Hello Yuan,
>
> Thank you for looking at the issue.
>
> The actual file doesn't contain any tabs (they are 4 spaces indent), 
> must be my email client trying to format it.
>
> After reflecting a bit, I'm not sure it is related in any way to 
> treesitter. I'm not even sure python + treesitter is a thing in emacs 
> in 29.0.50 (not according to etc/NEWS at least).
>
> I can just confirm that versions <28.2 don't show this issue, that's 
> why I thought about treesitter.
>
> -- 
> Matthias


Hello again.

Turns out I had a slightly out of date version (from last week). I 
recompiled and don't have this problem anymore.

Seems to have been resolved in fc9d7b3d04.

Sorry for the confusion.

--

Matthias

Message #19 received at 59679-done <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: hello <at> mattplm.fr
Cc: 59679-done <at> debbugs.gnu.org
Subject: Re: bug#59679: 29.0.50; python + tree-sitter: Syntax highlighting  is
 sometimes wrong with multi line docstrings
Date: Thu, 1 Dec 2022 21:08:16 -0800

Matthias Paulmier <hello <at> mattplm.fr> writes:

> On 30/11/2022 12:06, Matthias Paulmier wrote:
>>
>>> Hmmm, it seems the tabs in the code throws tree-sitter off and makes it
>>> parse incorrectly. But I’m not sure if the tabs are in your code or they
>>> are added by the email provider. Do you expect the code to contain tabs?
>>
>> Hello Yuan,
>>
>> Thank you for looking at the issue.
>>
>> The actual file doesn't contain any tabs (they are 4 spaces indent),
>> must be my email client trying to format it.
>>
>> After reflecting a bit, I'm not sure it is related in any way to
>> treesitter. I'm not even sure python + treesitter is a thing in
>> emacs in 29.0.50 (not according to etc/NEWS at least).
>>
>> I can just confirm that versions <28.2 don't show this issue, that's
>> why I thought about treesitter.
>>
>> -- Matthias
>
>
> Hello again.
>
> Turns out I had a slightly out of date version (from last week). I
> recompiled and don't have this problem anymore.
>
> Seems to have been resolved in fc9d7b3d04.
>
> Sorry for the confusion.

Don’t be sorry :-) Thanks for reporting the bug! I’m closing this report.

Yuan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.