Subject: Re: bug#36432: 26.2; SMIE does not request forward tokens when point is at point-max

From: Eli Zaretskii <[email-masked]>
Date: Sat, 29 Jun 2019 16:08:01 +0300

> From: Sam Halliday <[email-masked]>
> Date: Sat, 29 Jun 2019 14:01:52 +0100
> Cc: [email-masked]
> 
> On 29/06/2019, Eli Zaretskii <[email-masked]> wrote:
> >> Date: Sat, 29 Jun 2019 15:42:27 +0300
> >> From: Eli Zaretskii <[email-masked]>
> >> Cc: [email-masked]
> >>
> >> How would Emacs know this is the case?  Why don't you also override
> >> point-max to make it consistent with those illusory characters?
> >
> > Or actually insert those characters at EOB, but make them invisible?
> 
> I think I'd like to avoid doing that unless I do it for all of the
> virtual tokens. I don't know how to do that without it impacting the
> underlying source code. Is there more documentation about doing it
> that way? It'd be a drastic change in how the lexer is written.

Why? does the lexer only pay attention to visible characters?

> BTW I had a look through smie.el and I can't see anywhere obvious
> where (eobp) or (point-max) are called that would lead to the bug I'm
> seeing.

It's likely in lower-level code.  Like I said: this assumption is
everywhere in Emacs.