GNU bug report logs - #67262
python-ts-mode cannot identify triple-quoted-strings

Previous Next

Full log

Message #32 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Yuan Fu <casouri <at> gmail.com>, JD Smith <jdtsmith <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Mon, 27 Nov 2023 02:05:10 +0200

On 27/11/2023 01:43, Yuan Fu wrote:
> 
> On 11/26/23 6:58 AM, Dmitry Gutov wrote:
>> On 26/11/2023 04:04, Dmitry Gutov wrote:
>>> As for what to do about this one -- probably something involving 
>>> syntax-propertize-extend-region-functions, adding an entry which 
>>> would initialize the parser, but not call syntax-ppss-flush-cache 
>>> directly (or at least not just that). It would signal the earlier 
>>> position to extend to through some dynamic variable. This is getting 
>>> tricky enough to move from the individual major modes into treesit.el 
>>> proper, I think.
>>
>> Alternatively, we'd trigger updates eagerly from within 
>> treesit_record_change -- that would make it slower, invalidating the 
>> comment above it. Not sure by how much, though.
> 
> It seems to me that what we need is to force a re-parse at the beginning 
> of syntax-propertize or in syntax-ppss-flush-cache; the re-parse would 
> cause the notifier to run, which runs python--treesit-parser-after-change.

syntax-ppss-flush-cache is called by edits (and by the re-parse). It 
seems like it will be odd to have execution the other way around and/or 
add some hook into it which would call the re-parse and extend the 
region to be invalidated.

syntax-propertize could have another hook added, yes. Or an advice.

But it seems better to reuse some of the existing hooks, such as 
syntax-propertize-extend-region-functions. It treesit.c provided a way 
to fetch the newly-invalidated region, the treesit-major-mode-setup 
could add a new function to syntax-propertize-extend-region-functions 
which would invoke that feature. But even now it can instantiate the 
parse, which would call treesit-force-reparse internally, and then 
collect the info from the callbacks.

And yet another way - is to extend the region to be propertized from 
inside the major mode's syntax-propertize-function, invalidating some 
earlier entries too. The main problem with that, I think, is that every 
ts mode will have to repeat that trick. And that authors would have to 
know to do that. How to make that easier and more obvious, is a question.

Finally, if I'm right that bug#66732 has a similar cause, then a shared 
solution that can be reused by syntax and font-lock (or preferably just 
fix both in the same place) would be ideal.

> I'm not quite sure about how do we cause this re-parse. The 
> straightforward approach would be calling treesit-force-reparse[1] in 
> syntax-propertize/syntax-ppss-flush-cache. But ideally I'd like to keep 
> tree-sitter transparent for syntax.el. Maybe we can add a hook in 
> syntax-propertize/syntax-ppss-flush-cache.
> 
> [1] This function doesn't exist yet, but it's easy to define in lisp.

treesit-parser-root-node calls it anyway and does little else, so we 
could get by with just using it.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.