Subject: Re: bug#22256: 25.0.50; multiline font-lock rules broken in C mode

From: Anders Lindgren <[email-masked]>
Date: Fri, 8 Jan 2016 22:34:09 +0100

Hi Alan (and the list),

I made a "git bisect" and found the culprit:

--------
b31d359d182eb252a11f0468a7dc1ee1eafb28e9 is the first bad commit
commit b31d359d182eb252a11f0468a7dc1ee1eafb28e9
Author: Alan Mackenzie <[email-masked]>
Date:   Sun Feb 1 21:20:35 2015 +0000

    CC Mode: Stop Font Lock forcing fontification from BOL.  Fixes
debbugs#19669.
    cc-mode.el (c-font-lock-init): Setq font-lock-extend-region-functions
to nil.
--------

The reason this breaks multiline keywords is that
`font-lock-extend-region-multiline' is normally part of
`font-lock-extend-region-functions'.

    -- Anders

On Wed, Dec 30, 2015 at 6:50 AM, Anders Lindgren <[email-masked]> wrote:
>
>
> Hi!
>
> I just realized that I missed one vital thing in the example I posted.
`font-lock-multiline' must be set. Also, I add code so that it works when
`font-lock-flush' isn't defined.
>
> (defvar my-multiline-test-keywords
>  '(("^X"
>      ("^.+quot;
>       (progn (beginning-of-line)
>              (point-max))
>       nil
>       (0 'highlight)))))
>
> (defun my-multiline-test-add ()
>   (interactive)
>   (font-lock-add-keywords nil my-multiline-test-keywords)
>   (setq font-lock-multiline t)
>   (if (fboundp 'font-lock-flush)
>       (font-lock-flush)
>     (when font-lock-mode
>       (with-no-warnings
>         (font-lock-fontify-buffer)))))
>
> Recipe:
>
> emacs -Q
> Eval the code above
> C-x b test RET
> Insert the text starting with an X (from the first mail)
> M-x c-mode RET
> M-x my-multiline-test-add RET
>
> Alternatively, use emacs-lisp-mode instead of C mode.
>
> When tested on Emacs 24.5 and Emacs 25, the only combination that doesn't
work is Emacs 25 + C mode.
>
> There are a number of font-lock packages that utilize that this work,
including my https://github.com/Lindydancer/preproc-font-lock package.
>
>     -- Anders
>
> On Tue, Dec 29, 2015 at 10:06 PM, Alan Mackenzie <[email-masked]> wrote:
> >
> > Hello, Anders.
> >
> > In article <[email-masked]> you
wrote:
> > > [-- text/plain, encoding 7bit, charset: UTF-8, 143 lines --]
> >
> > > Hi!
> >
> > > In C and related modes, multiline font-lock rules no longer work as
> > > expected. It looks like a few lines are highlighted, but not all of
them.
> >
> > > For example:
> > > Eval the following:
> > > (defvar my-multiline-test-keywords
> > >  '(("^X"
> > >      ("^.+quot;
>
> > >       (progn (beginning-of-line)
> > >              (point-max))
> > >       nil
> > >       (0 'highlight)))))
> >
> > > (defun my-multiline-test-add ()
> > >   (interactive)
> > >   (font-lock-add-keywords nil my-multiline-test-keywords)
> > >   (font-lock-flush))
> >
> > > Insert the following in a new buffer:
> >
> > > X START OF A HIGHLIGHTED BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> > > LONG LINES IN THE BLOCK LONG LINES IN THE BLOCK LONG LINES IN THE
BLOCK
> >
> > > Do:
> > > M-x c-mode RET
> > > M-x my-multiline-test-add RET
> >
> > > I expect the entire buffer to be highlighted. However, only the first
eight
> > > lines are highlighted. When the block is edited, different parts of
the
> > > block is highlighted and unhighlighted.
> >
> > This is caused by the jit-lock mechanism.  The first eight lines are 500
> > characters (jit-lock-chunk-size) rounded up to a whole number of lines.
> >
> > What happens is this: when the first jit-lock-chunk (8 lines) is
> > fontified, the `fontified' property (the one the display engine uses) is
> > set only on these 8 lines.  The `face' property is then set on all the
> > characters of the file, as requested by the my-multiline-test-keywords
> > form.
> >
> > Next thing, jit-lock fontifies the next chunk of ~7 lines starting where
> > the `fontified' property is nil.  The first thing done is to set
> > `fontified' on these ~7 lines, then the `face' property on them is
> > erased.  There is now no matching font lock pattern to apply any new
> > faces to these ~7 lines, since the "X" is many lines back.  The same
> > thing happens with the next 500 byte chunk, and so on till the end of
the
> > buffer.
> >
> > If you set `font-lock-support-mode' to nil and restart font locking, the
> > problem isn't apparent.  (Then set the variable back to 'jit-lock-mode.)
> >
> > This is a fundamental problem with jit-lock-mode: the assumption that
> > text to be fontified has no non-trivial context.  This is a difficult
> > problem to solve in general.  CC Mode uses some ad-hoc tricks to catch,
> > for example, long struct declarations.
> >
> > > As a contrast, when `emacs-lisp-mode' is used, the entire block is
> > > highlighted, and editing does not change the highlighting.
> >
> > I do not see this in Emacs 24.5.  For me, even in emacs-lisp-mode, I
> > still see just the 8 lines being fontified.  If you could give me a
> > recipe (starting from emacs-24.5 -Q) to reproduce this, I'd be very
> > interested.
> >
> > > This worked as intended in Emacs 24.5.
> >
> > >     -- Anders Lindgren
> >
> > > In GNU Emacs 25.0.50.60 (x86_64-apple-darwin14.5.0, NS appkit-1348.17
> > > Version 10.10.5 (Build 14F27))
> > >  of 2015-12-28
> > > Repository revision: e9916d8880561cc06b6cb73bafe7257b93ffbf4c
> > > Windowing system distributor 'Apple', version 10.3.1348
> > > Configured using:
> > >  'configure --without-dbus'
> >
> > > Configured features:
> > > ACL ZLIB TOOLKIT_SCROLL_BARS NS
> >
> > > Important settings:
> > >   value of $LC_CTYPE: UTF-8
> > >   locale-coding-system: utf-8-unix
> >
> > > Major mode: C/l
> >
> > > Minor modes in effect:
> > >   preproc-font-lock-global-mode: t
> > >   preproc-font-lock-mode: t
> > >   tooltip-mode: t
> > >   global-eldoc-mode: t
> > >   electric-indent-mode: t
> > >   mouse-wheel-mode: t
> > >   tool-bar-mode: t
> > >   menu-bar-mode: t
> > >   file-name-shadow-mode: t
> > >   global-font-lock-mode: t
> > >   font-lock-mode: t
> > >   blink-cursor-mode: t
> > >   auto-composition-mode: t
> > >   auto-encryption-mode: t
> > >   auto-compression-mode: t
> > >   line-number-mode: t
> > >   transient-mark-mode: t
> > >   abbrev-mode: t
> >
> > [ .... ]
> >
> > --
> > Alan Mackenzie (Nuremberg, Germany).
> >