GNU bug report logs - #48365
13.0.11; Dollars in distinct comments trigger math mode highlighting in text in-between

unarchive 48365

From: jfbu <jfbu <at> free.fr>
To: bug-auctex <at> gnu.org
Subject: 13.0.11;
 Dollars in distinct comments trigger math mode highlighting in text
 in-between
Date: Tue, 11 May 2021 21:17:23 +0200

[Message part 1 (text/plain, inline)]
Consider the following file:

% -*- coding: utf-8; mode: latex; -*-
% $
\def\foo_with_underscore{}
%
%
\def\foo_bar_foo_bar{}
% $
%

The presence of the $ sign in comments trigger math mode highlighting in between, as demonstrated in the attached screenshot

This happens in 13.0.11 (installed via Elpa) but did not happen at 12.2.0 and in fact I sticked with 12.2.0 for a long period because I had it still by accident on another computer where I had started lots of tex coding using very often $ in comments for some special reasons (it had another catcode) and when on my laptop which was with auctex from git I realized I could not work at all anymore because of this highlighting problem I simply rsync'ed from my other computer to my laptop as I had no time to devote to this issue. I thought perhaps the problem was with doctex-mode but today I see I can trigger it in latex-mode.

git bisect indicates first bad commit is

commit 6654955216a42936b87f76dc346aad829b1d52fb
Date:   Wed Jun 3 01:44:32 2020 +0900

    Use search-based fontification for $...$ (bug#33139)

Reading the commit message, I see there was extensive discussion and the change seems to have been discussed in depth and well-motivated. I wonder if you can confirm that you see the same at your locale as me.

Possibly relevant to this in my config are (but I did not try to change them)

 '(font-latex-fontify-script (quote multi-level))

 '(texmathp-search-n-paragraphs 0)

For the latter I dimly recall it was an old issue I had in doctex with highlighting too but I don't recall the details and it might be something else whatsoever. (I think last week I did try removing this to see if it affected the present issue but it did not).

[Message part 2 (text/html, inline)]
[Capture d’écran 2021-05-11 à 20.40.27.png (image/png, inline)]
From: Tassilo Horn <tsdh <at> gnu.org>
To: jfbu <jfbu <at> free.fr>
Cc: bug-auctex <at> gnu.org, 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Tue, 11 May 2021 21:26:50 +0200

Hi Jean-Francois,

> git bisect indicates first bad commit is
>
> commit 6654955216a42936b87f76dc346aad829b1d52fb
> Date:   Wed Jun 3 01:44:32 2020 +0900
>
>     Use search-based fontification for $...$ (bug#33139)
>
> Reading the commit message, I see there was extensive discussion and
> the change seems to have been discussed in depth and well-motivated. I
> wonder if you can confirm that you see the same at your locale as me.

Yes, I see the problem.  I've pushed a fix which ignores $ in comments
just as it already ignored $ in verbatim contexts.

Could you please test and report back?

Bye,
Tassilo

From: jfbu <jfbu <at> free.fr>
To: bug-auctex <at> gnu.org
Cc: 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Tue, 11 May 2021 22:06:12 +0200

Hi Tassilo,

Le 11/05/2021 Ã  21:26, Tassilo Horn a Ã©critÂ :
> Hi Jean-Francois,
> 
>> git bisect indicates first bad commit is
>>
>> commit 6654955216a42936b87f76dc346aad829b1d52fb
>> Date:   Wed Jun 3 01:44:32 2020 +0900
>>
>>      Use search-based fontification for $...$ (bug#33139)
>>
>> Reading the commit message, I see there was extensive discussion and
>> the change seems to have been discussed in depth and well-motivated. I
>> wonder if you can confirm that you see the same at your locale as me.
> 
> Yes, I see the problem.  I've pushed a fix which ignores $ in comments
> just as it already ignored $ in verbatim contexts.
> 
> Could you please test and report back?
> 

Yes, it works also in original context, not only in the short test file.
I realize now that after all I did work for a while with this problem
and I see one file where I have lots of extra $'s added in comments
to create pairs of $'s and avoid the propagation, I removed them
and no ill effect arose, the math highlighting was not applied.

But... wait...

Sadly, I now have another problem. Consider this:

\def\XINT:NE:f:noeval:from:braced:u:p #1#2%
    {\detokenize{\romannumeral`$XINT_expr_null\expandafter#1}~expanded{{#2}}}%

There is $ in there, and it triggers math mode highlighting which is a priori fine,
I can live with that *if I can constrain it to not extend beyond that line*

In the past I would control it this way:

\def\XINT:NE:f:noeval:from:braced:u:p #1#2%
    {\detokenize{\romannumeral`$XINT_expr_null\expandafter#1}~expanded{{#2}}}%$

(observe the added $ at end of second line after the % comment character)

With your patch the %$ does not stop the math highlighting !

This is very problematic to me because that problem can't be solved (as I
could solve earlier one, especially now that I understand better what causes it,
via locating extra $'s inside comments)
by adding a commented out $.

It basically means I have not way to stop the highlighting from propagating...

Jean-FranÃ§ois

> Bye,
> Tassilo
> 
> 
> 
> _______________________________________________
> bug-auctex mailing list
> bug-auctex <at> gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-auctex
> 

From: JF <jfbu <at> free.fr>
To: 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Tue, 11 May 2021 22:21:25 +0200

Le 11/05/2021 Ã  22:06, jfbu a Ã©critÂ :
> 
> Hi Tassilo,
> 
> Le 11/05/2021 Ã  21:26, Tassilo Horn a Ã©critÂ :
>> Hi Jean-Francois,
>>
>>> git bisect indicates first bad commit is
>>>
>>> commit 6654955216a42936b87f76dc346aad829b1d52fb
>>> Date:Â Â  Wed Jun 3 01:44:32 2020 +0900
>>>
>>> Â Â Â Â  Use search-based fontification for $...$ (bug#33139)
>>>
>>> Reading the commit message, I see there was extensive discussion and
>>> the change seems to have been discussed in depth and well-motivated. I
>>> wonder if you can confirm that you see the same at your locale as me.
>>
>> Yes, I see the problem.Â  I've pushed a fix which ignores $ in comments
>> just as it already ignored $ in verbatim contexts.
>>
>> Could you please test and report back?
>>
> 
> Yes, it works also in original context, not only in the short test file.
> I realize now that after all I did work for a while with this problem
> and I see one file where I have lots of extra $'s added in comments
> to create pairs of $'s and avoid the propagation, I removed them
> and no ill effect arose, the math highlighting was not applied.
> 
> But... wait...
> 
> Sadly, I now have another problem. Consider this:
> 
> \def\XINT:NE:f:noeval:from:braced:u:p #1#2%
>  Â Â Â  {\detokenize{\romannumeral`$XINT_expr_null\expandafter#1}~expanded{{#2}}}%
> 
> There is $ in there, and it triggers math mode highlighting which is a priori fine,
> I can live with that *if I can constrain it to not extend beyond that line*
> 
> In the past I would control it this way:
> 
> \def\XINT:NE:f:noeval:from:braced:u:p #1#2%
>  Â Â Â  {\detokenize{\romannumeral`$XINT_expr_null\expandafter#1}~expanded{{#2}}}%$
> 
> (observe the added $ at end of second line after the % comment character)
> 
> With your patch the %$ does not stop the math highlighting !
> 
> This is very problematic to me because that problem can't be solved (as I
> could solve earlier one, especially now that I understand better what causes it,
> via locating extra $'s inside comments)
> by adding a commented out $.
> 
> It basically means I have not way to stop the highlighting from propagating...
> 

not true. I could use another character than % and assign it catcode comment and
use it at end of line

but not that if I am "aguerri" enough for such manoeuvers this might not
be the case of all auctex users and they might do something like
\newcommand\foo{$} and then perhaps they have the same problem as me ?

% -*- coding: utf-8; mode: latex; -*-
%
\def\foo_with_underscore{}
\newcommand\foo{$}% $ does not help here
%
\def\foo_with_underscore{$}% <-- but math highlight will stop here
%
\def\foo_bar_foo_bar{}
%
%

> Jean-FranÃ§ois
> 
> 
> 
>> Bye,
>> Tassilo
>>
>>
>>
>> _______________________________________________
>> bug-auctex mailing list
>> bug-auctex <at> gnu.org
>> https://lists.gnu.org/mailman/listinfo/bug-auctex
>>
> 
> 
> 
> 
> 
> _______________________________________________
> bug-auctex mailing list
> bug-auctex <at> gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-auctex

From: Tassilo Horn <tsdh <at> gnu.org>
To: jfbu <jfbu <at> free.fr>
Cc: bug-auctex <at> gnu.org, 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Tue, 11 May 2021 22:33:04 +0200

> Sadly, I now have another problem. Consider this:
>
> \def\XINT:NE:f:noeval:from:braced:u:p #1#2%
>     {\detokenize{\romannumeral`$XINT_expr_null\expandafter#1}~expanded{{#2}}}%
>
> There is $ in there, and it triggers math mode highlighting which is a
> priori fine, I can live with that *if I can constrain it to not extend
> beyond that line*
>
> In the past I would control it this way:
>
> \def\XINT:NE:f:noeval:from:braced:u:p #1#2%
>     {\detokenize{\romannumeral`$XINT_expr_null\expandafter#1}~expanded{{#2}}}%$
>
> (observe the added $ at end of second line after the % comment character)
>
> With your patch the %$ does not stop the math highlighting !

Now that you've mentioned it, I remember I wanted to do that change also
in the past and Keita-san stopped me from doing so for exactly this
reason.

I've pushed another commit which ignores $ in comments with the single
exception of the "% $ \n" workaround, i.e., a comment with just a $,
arbitrary spaces/tabs before or after the $, and then the end of the
line.

I'm not sure what is worse.  The restriction to have at least balanced $
also in comments, or the now very restrictive workaround form.  We'll
see if someone complains because that breaks his

  % Workaround fix $.

comment fixes.

Bye,
Tassilo

From: jfbu <jfbu <at> free.fr>
To: Tassilo Horn <tsdh <at> gnu.org>
Cc: bug-auctex <at> gnu.org, 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11;
 Dollars in distinct comments trigger math mode highlighting in text
 in-between
Date: Tue, 11 May 2021 22:50:51 +0200

Le 11 mai 2021 Ã  22:33, Tassilo Horn <tsdh <at> gnu.org> a Ã©crit :

> 
>> Sadly, I now have another problem. Consider this:
>> 
>> \def\XINT:NE:f:noeval:from:braced:u:p #1#2%
>>    {\detokenize{\romannumeral`$XINT_expr_null\expandafter#1}~expanded{{#2}}}%
>> 
>> There is $ in there, and it triggers math mode highlighting which is a
>> priori fine, I can live with that *if I can constrain it to not extend
>> beyond that line*
>> 
>> In the past I would control it this way:
>> 
>> \def\XINT:NE:f:noeval:from:braced:u:p #1#2%
>>    {\detokenize{\romannumeral`$XINT_expr_null\expandafter#1}~expanded{{#2}}}%$
>> 
>> (observe the added $ at end of second line after the % comment character)
>> 
>> With your patch the %$ does not stop the math highlighting !
> 
> Now that you've mentioned it, I remember I wanted to do that change also
> in the past and Keita-san stopped me from doing so for exactly this
> reason.
> 
> I've pushed another commit which ignores $ in comments with the single
> exception of the "% $ \n" workaround, i.e., a comment with just a $,
> arbitrary spaces/tabs before or after the $, and then the end of the
> line.
> 
> I'm not sure what is worse.  The restriction to have at least balanced $
> also in comments, or the now very restrictive workaround form.  We'll
> see if someone complains because that breaks his
> 
>  % Workaround fix $.
> 
> comment fixes.
> 

Hi Tassilo, sorry for causing all the trouble.

I pulled your commit and it works. For me that's working solution.

Would it make sense to apply these rules:

1. a $ in comments can never start math mode
2. but it can and will always stop math mode

Cheers,

Jean-FranÃ§ois

From: Tassilo Horn <tsdh <at> gnu.org>
To: jfbu <jfbu <at> free.fr>
Cc: bug-auctex <at> gnu.org, 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Wed, 12 May 2021 08:13:28 +0200

Hi Jean-FranÃ§ois,

>> I'm not sure what is worse.  The restriction to have at least
>> balanced $ also in comments, or the now very restrictive workaround
>> form.  We'll see if someone complains because that breaks his
>> 
>>  % Workaround fix $.
>> 
>> comment fixes.
>
> Hi Tassilo, sorry for causing all the trouble.

Ah, always these users with their strange requirements and bogus
documents! ;-)

> I pulled your commit and it works. For me that's working solution.
>
> Would it make sense to apply these rules:
>
> 1. a $ in comments can never start math mode
> 2. but it can and will always stop math mode

Hm, at first I've thought that would be annoying but it seems to work
quite well.  So I've pushed another commit implementing that.  Please
report if that fits the bill.

Bye,
Tassilo

From: Ikumi Keita <ikumi <at> ikumi.que.jp>
To: jfbu <jfbu <at> free.fr>
Cc: 48365 <at> debbugs.gnu.org, Tassilo Horn <tsdh <at> gnu.org>
Subject: Re: bug#48365: 13.0.11;
 Dollars in distinct comments trigger math mode highlighting in text
 in-between
Date: Wed, 12 May 2021 15:39:20 +0900

[Message part 1 (text/plain, inline)]
Hi Jean and Tassilo,

>>>>> jfbu <jfbu <at> free.fr> writes:
>> Now that you've mentioned it, I remember I wanted to do that change also
>> in the past and Keita-san stopped me from doing so for exactly this
>> reason.

Yes, I did. ;-)

> Would it make sense to apply these rules:

> 1. a $ in comments can never start math mode
> 2. but it can and will always stop math mode

That is a clean idea and, fortunately, easy to implment. Could you test
the attached patch?

One remark: This patch does not follow rigorously your proposition. A $
in comments does not stop math mode if it is preceded by odd numbers of
$'s like this:
foo bar blah % \$

Regards,
Ikumi Keita

[patch (text/x-diff, attachment)]
From: Ikumi Keita <ikumi <at> ikumi.que.jp>
To: Tassilo Horn <tsdh <at> gnu.org>
Cc: 48365 <at> debbugs.gnu.org, jfbu <jfbu <at> free.fr>
Subject: Re: bug#48365: 13.0.11;
 Dollars in distinct comments trigger math mode highlighting in text
 in-between
Date: Wed, 12 May 2021 15:42:09 +0900

Oups, sorry, I sent my previous email just before I saw this message!

>>>>> Tassilo Horn <tsdh <at> gnu.org> writes:
> Hi Jean-FranÃ§ois,
>>> I'm not sure what is worse.  The restriction to have at least
>>> balanced $ also in comments, or the now very restrictive workaround
>>> form.  We'll see if someone complains because that breaks his
>>> 
>>> % Workaround fix $.
>>> 
>>> comment fixes.
>> 
>> Hi Tassilo, sorry for causing all the trouble.

> Ah, always these users with their strange requirements and bogus
> documents! ;-)

>> I pulled your commit and it works. For me that's working solution.
>> 
>> Would it make sense to apply these rules:
>> 
>> 1. a $ in comments can never start math mode
>> 2. but it can and will always stop math mode

> Hm, at first I've thought that would be annoying but it seems to work
> quite well.  So I've pushed another commit implementing that.  Please
> report if that fits the bill.

> Bye,
> Tassilo

> _______________________________________________
> bug-auctex mailing list
> bug-auctex <at> gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-auctex

From: Tassilo Horn <tsdh <at> gnu.org>
To: Ikumi Keita <ikumi <at> ikumi.que.jp>
Cc: 48365 <at> debbugs.gnu.org, jfbu <jfbu <at> free.fr>
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Wed, 12 May 2021 08:49:58 +0200

Ikumi Keita <ikumi <at> ikumi.que.jp> writes:

Hi Ikumi,

>> Would it make sense to apply these rules:
>
>> 1. a $ in comments can never start math mode
>> 2. but it can and will always stop math mode
>
> That is a clean idea and, fortunately, easy to implment. Could you test
> the attached patch?

I've committed a completely identical patch an hour ago with a diff just
in the comment explaining what we are doing.  It's very encouraging when
we both have the very same idea, so I think, it cannot be too wrong. :-)

Thanks,
Tassilo

From: jfbu <jfbu <at> free.fr>
To: Tassilo Horn <tsdh <at> gnu.org>
Cc: Ikumi Keita <ikumi <at> ikumi.que.jp>, 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11;
 Dollars in distinct comments trigger math mode highlighting in text
 in-between
Date: Wed, 12 May 2021 09:22:50 +0200

[Message part 1 (text/plain, inline)]
Hello Tassilo=Keita-san
Le 12 mai 2021 Ã  08:49, Tassilo Horn <tsdh <at> gnu.org> a Ã©crit :

> Ikumi Keita <ikumi <at> ikumi.que.jp> writes:
> 
> Hi Ikumi,
> 
>>> Would it make sense to apply these rules:
>> 
>>> 1. a $ in comments can never start math mode
>>> 2. but it can and will always stop math mode
>> 
>> That is a clean idea and, fortunately, easy to implment. Could you test
>> the attached patch?
> 
> I've committed a completely identical patch an hour ago with a diff just
> in the comment explaining what we are doing.  It's very encouraging when
> we both have the very same idea, so I think, it cannot be too wrong. :-)
> 
> Thanks,
> Tassilo

It is now revealed you are only one entity (perhaps an A.I. based
on quantum computing) pretending to be two humans only to
justify coverage of all time zones !

I have pulled the commit and it appears to work fine in my real-life
files (using doctex mode)

But I notice something weird when comparing effect on those two files

File A behaves as expected
% -*- mode: latex; -*-
%
\def\foo_with_underscore{}
\newcommand\foo{$}%
% a comment
A foo_bar^bar{$}
% a comment
B foo_bar^bar{$}
% a comment
C foo_bar^bar
$
%
%

File B seems to have a problem
% -*- mode: latex; -*-
%
\def\foo_with_underscore{}
\newcommand\foo{$}%$
% a comment
A foo_bar^bar{$}
% a comment
B foo_bar^bar{$}
% a comment
C foo_bar^bar
$
%
%

In case of file B, which differs only by an added %$ at end of the \newcommand line,
the \newcommand for example is not highlighted
and the B line changes colors without applying script style,
see screenshot

In case of file B I was expecting the B line to be styled in math mode,
it only has its color it seems,
and I was not expecting \newcommand\foo to lose its highlighting,

Perhaps this is a symptom of something else?

Best,
Jean-FranÃ§ois

[Message part 2 (text/html, inline)]
[Capture d’écran 2021-05-12 à 09.13.51.png (image/png, inline)]
From: Tassilo Horn <tsdh <at> gnu.org>
To: jfbu <jfbu <at> free.fr>
Cc: Ikumi Keita <ikumi <at> ikumi.que.jp>, 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Wed, 12 May 2021 09:44:42 +0200

jfbu <jfbu <at> free.fr> writes:

> It is now revealed you are only one entity (perhaps an A.I. based on
> quantum computing) pretending to be two humans only to justify
> coverage of all time zones !

Gosh, we're uncovered!

> But I notice something weird when comparing effect on those two files
>
> File A behaves as expected
> % -*- mode: latex; -*-
> %
> \def\foo_with_underscore{}
> \newcommand\foo{$}%
> % a comment
> A foo_bar^bar{$}
> % a comment
> B foo_bar^bar{$}
> % a comment
> C foo_bar^bar
> $
> %
> %
>
> File B seems to have a problem
> % -*- mode: latex; -*-
> %
> \def\foo_with_underscore{}
> \newcommand\foo{$}%$
> % a comment
> A foo_bar^bar{$}
> % a comment
> B foo_bar^bar{$}
> % a comment
> C foo_bar^bar
> $
> %
> %
>
> In case of file B, which differs only by an added %$ at end of the
> \newcommand line, the \newcommand for example is not highlighted and
> the B line changes colors without applying script style, see
> screenshot

Yeah, it appears that if math mode goes out of hands, it may also have
effects on the fontification before.  That has been the case already
before the latest commits.

But if you just add a space before the \newcommand and delete it again
to trigger re-fontification, it becomes fontified correctly.  Not sure
why that is.  I'll have a look when I find some spare time...

Bye,
Tassilo

From: JF <jfbu <at> free.fr>
To: Tassilo Horn <tsdh <at> gnu.org>
Cc: Ikumi Keita <ikumi <at> ikumi.que.jp>, 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Wed, 12 May 2021 10:04:23 +0200

Hi Tassilo

Le 12/05/2021 Ã  09:44, Tassilo Horn a Ã©critÂ :
>> File B seems to have a problem
>> % -*- mode: latex; -*-
>> %
>> \def\foo_with_underscore{}
>> \newcommand\foo{$}%$
>> % a comment
>> A foo_bar^bar{$}
>> % a comment
>> B foo_bar^bar{$}
>> % a comment
>> C foo_bar^bar
>> $
>> %
>> %
>>
>> In case of file B, which differs only by an added %$ at end of the
>> \newcommand line, the \newcommand for example is not highlighted and
>> the B line changes colors without applying script style, see
>> screenshot
> Yeah, it appears that if math mode goes out of hands, it may also have
> effects on the fontification before.  That has been the case already
> before the latest commits.
>
> But if you just add a space before the \newcommand and delete it again
> to trigger re-fontification, it becomes fontified correctly.

Yes, confirmed regarding \newcommand, however

- C-cC-n loses againt the fontification

- the B line still does not behave as expected, it should apply math styling,
but the subscripts are still on the baseline

>    Not sure
> why that is.  I'll have a look when I find some spare time...

No urgency here... I got the feeling my real-life files are ok so, from my
egotistical point of view the matter reached a satisfactory state for me,
the fate of the rest of the world does not interest me extraordinarily,
especially in those days where one has trashed with the approval
of the majority all basic principles
of defence and preservation of liberties, paving the road for very
grave evolutions awaiting us.

Best,
Jean-FranÃ§ois

From: Tassilo Horn <tsdh <at> gnu.org>
To: JF <jfbu <at> free.fr>
Cc: Ikumi Keita <ikumi <at> ikumi.que.jp>, 48365 <at> debbugs.gnu.org
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Wed, 12 May 2021 10:14:28 +0200

> Yes, confirmed regarding \newcommand, however
>
> - C-cC-n loses againt the fontification

Right.

> - the B line still does not behave as expected, it should apply math
> styling, but the subscripts are still on the baseline

Ah, indeed.  I've just seen that it was fontified as math but missed
that the scripts where not raised/lowered...

Bye,
Tassilo

From: Ikumi Keita <ikumi <at> ikumi.que.jp>
To: JF <jfbu <at> free.fr>
Cc: 48365 <at> debbugs.gnu.org, Tassilo Horn <tsdh <at> gnu.org>
Subject: Re: bug#48365: 13.0.11;
 Dollars in distinct comments trigger math mode highlighting in text
 in-between
Date: Thu, 13 May 2021 02:04:03 +0900

Hi Jean,

>>>>> JF <jfbu <at> free.fr> writes:
>>> File B seems to have a problem
>>> % -*- mode: latex; -*-
>>> %
>>> \def\foo_with_underscore{}
>>> \newcommand\foo{$}%$
>>> % a comment
>>> A foo_bar^bar{$}
>>> % a comment
>>> B foo_bar^bar{$}
>>> % a comment
>>> C foo_bar^bar
>>> $
>>> %
>>> %
>>> 
>>> In case of file B, which differs only by an added %$ at end of the
>>> \newcommand line, the \newcommand for example is not highlighted and
>>> the B line changes colors without applying script style, see
>>> screenshot

Fixed in the git repo. Please try.

Regards,
Ikumi Keita

From: Tassilo Horn <tsdh <at> gnu.org>
To: Ikumi Keita <ikumi <at> ikumi.que.jp>
Cc: 48365 <at> debbugs.gnu.org, JF <jfbu <at> free.fr>
Subject: Re: bug#48365: 13.0.11; Dollars in distinct comments trigger math
 mode highlighting in text in-between
Date: Wed, 12 May 2021 20:49:24 +0200

Ikumi Keita <ikumi <at> ikumi.que.jp> writes:

>>>> In case of file B, which differs only by an added %$ at end of the
>>>> \newcommand line, the \newcommand for example is not highlighted and
>>>> the B line changes colors without applying script style, see
>>>> screenshot
>
> Fixed in the git repo. Please try.

Works like a charm, and good catch!  How did you find the culprit?

With `jit-lock-debug-mode' I get as far as getting a message

  Error running timer âjit-lock--debug-fontifyâ: (end-of-buffer)

which clearly tells we're trying to moe farther than end of buffer but
still there is no indication where we're doing that.  Do you have some
trick I'm not aware of yet?

Bye,
Tassilo

From: jfbu <jfbu <at> free.fr>
To: Ikumi Keita <ikumi <at> ikumi.que.jp>
Cc: 48365 <at> debbugs.gnu.org, Tassilo Horn <tsdh <at> gnu.org>
Subject: Re: bug#48365: 13.0.11;
 Dollars in distinct comments trigger math mode highlighting in text
 in-between
Date: Wed, 12 May 2021 23:08:37 +0200

Hi Keita-san
Le 12 mai 2021 Ã  19:04, Ikumi Keita <ikumi <at> ikumi.que.jp> a Ã©crit :

>>>> % -*- mode: latex; -*-
>>>> %
>>>> \def\foo_with_underscore{}
>>>> \newcommand\foo{$}%$
>>>> % a comment
>>>> A foo_bar^bar{$}
>>>> % a comment
>>>> B foo_bar^bar{$}
>>>> % a comment
>>>> C foo_bar^bar
>>>> $
>>>> %
>>>> %
>>>> 
>>>> In case of file B, which differs only by an added %$ at end of the
>>>> \newcommand line, the \newcommand for example is not highlighted and
>>>> the B line changes colors without applying script style, see
>>>> screenshot
> 
> Fixed in the git repo. Please try.

Seems to work indeed
Somehow I have to hit C-cC-n to get fontification inclusive of subscripts getting lowered
to work after adding  a $ at the end of \newcommand\foo{$}%

% -*- mode: latex; -*-
%
\def\foo_with_underscore{}
\newcommand\foo{$}%
% a comment
A foo_bar^bar{$}
% a comment
B foo_bar^bar{$}
% a comment
C foo_bar^bar
$
%
%

but all looks fine to me. Thanks!
Jean-FranÃ§ois

From: Ikumi Keita <ikumi <at> ikumi.que.jp>
To: Tassilo Horn <tsdh <at> gnu.org>
Cc: 48365 <at> debbugs.gnu.org, JF <jfbu <at> free.fr>
Subject: Re: bug#48365: 13.0.11;
 Dollars in distinct comments trigger math mode highlighting in text
 in-between
Date: Thu, 13 May 2021 15:00:23 +0900

>>>>> Tassilo Horn <tsdh <at> gnu.org> writes:
> Works like a charm, and good catch!  How did you find the culprit?

> With `jit-lock-debug-mode' I get as far as getting a message

>   Error running timer âjit-lock--debug-fontifyâ: (end-of-buffer)

> which clearly tells we're trying to moe farther than end of buffer but
> still there is no indication where we're doing that.  Do you have some
> trick I'm not aware of yet?

No, just ordinary hand debug with edebug as an auxiliary tool. :-)

A possible advantage for me is that I encountered a similar example
before and saved it for future debug:
----------------------------------------------------------------------
\documentclass{article}

\begin{document}
\verb|^=}{\|
\verb$'&\$
     $aaa$
\verb|a$bc$d|
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:
----------------------------------------------------------------------
From Jean's example, I learned that unclosed dollar sign is involved in
this symptom. So I suspected that dollar fontification causes some error
to interrupt font lock, ending up with partial fontification.
(1) After some unsuccessful tries&errors, I put the point just before
    the last $ in the Jean's example and did
    M-: (font-latex-match-dollar-math (point-max)) RET
(2) Emacs signaled end-of-buffer error. So I could be confident that the
    problem hides in `font-latex-match-dollar-math' or the vicinity of
    it.
(3) I enabled edebug for `font-latex-match-dollar-math' and tried
    M-: (font-latex-match-dollar-math (point-max)) RET
    after putting the point before the last $ again.
(4) Then I noticed that the second call to
    `font-latex-find-dollar-math' returns t, which should be nil because
    this last $ is unclosed in the buffer.
(5) I looked at `font-latex-find-dollar-math' carefully and noticed that
    it returns false t if LIMIT argument exceeds end of buffer.

Regards,
Ikumi Keita

GNU bug report logs - #48365 13.0.11; Dollars in distinct comments trigger math mode highlighting in text in-between

GNU bug report logs - #48365
13.0.11; Dollars in distinct comments trigger math mode highlighting in text in-between