Subject: bug#59477: python-tests fail without tree-sitter

From: Yuan Fu <[email-masked]>
Date: Sat, 26 Nov 2022 14:18:37 -0800

> On Nov 25, 2022, at 8:50 AM, Mattias Engdegård <[email-masked]> wrote:
> 
>> But if I run this in a buffer manually I get the ending newline. I’m not sure what’s the cause of that. Bisecting give 7c5d4348330b206aff1f8e5bc4fd241d6a6dc0b5, but that commit doesn’t change anything filling-related. 
> 
> No idea really, but it might have something to do with the fact that the changes move the assignments
> 
>  (setq-local font-lock-defaults
>              `(,python-font-lock-keywords
>                nil nil nil nil
>                (font-lock-syntactic-face-function
>                 . python-font-lock-syntactic-face-function)))
>  (setq-local syntax-propertize-function
>              python-syntax-propertize-function)
> 
> so that they are executed after
> 
>  (when python-indent-guess-indent-offset
>    (python-indent-guess-indent-offset))
> 
> instead of before. `python-indent-guess-indent-offset` has the side-effect of setting syntax properties, in particular for the string terminator (triple-quote in the test).
> 
> This is important, because python-fill-string (called as part of fill-paragraph in the test) assumes this having already been done and if not, str-end-pos isn't computed correctly and things take a turn for the worse after that.
> 
> Stefan probably knows better how this is supposed to work, but presumably python-fill-string should take measures to ensure accurate syntax properties before doing things like
> 
>  (re-search-forward (rx (syntax string-delimiter)) nil t)
> 
> and so on. Sorry about not being of much help here.
> 

Thanks, that’s a very useful information. And I can only blame myself for breaking the tests :-)

While still unable to find the culprit. I have the following observations:

1. Setting require-final-newline to t doesn’t work
2. If I change with-temp-buffer to with-current-buffer (get-buffer-create "*test*”), the problem disappears, the newlines is not dropped
3. I edebugged fill-paragraph, the newlines in the temp buffer disappears at line 865 in fill.el, where the recursive call returns. Before (funcall function justify) returns (`function` is fill-paragraph itself), the newline still exists, but after we return to the caller at line 865, the newline disappears.

I checked for newline by hitting e and evaluating (char-before (point-max))

Yuan