Subject: font-lock corrupts `end' position in jit-lock after-change function.

From: Alan Mackenzie <[email-masked]>
Date: Tue, 5 Jan 2016 21:47:28 +0000

Hello, Emacs.

With the latest emacs-25 branch (committed 2016-01-05, ~21:15 +0000),
do emacs -Q, and load (or type in) the follow AWK Mode file (M-x
awk-mode, if required):

########################################################################
#!/usr/bin/gawk -f
BEGIN {
    x = 4
}
#########################################################################

Notice the fontification of "BEGIN".  Type a space anywhere on the first
line.  The following happens:
(i) The fontification on the "B" of "BEGIN" is erased instantly.
(ii) Half a second (jit-lock-context-time) later, the fontification on
  "EGIN" is erased.

The cause of the problem is in
font-lock-extend-jit-lock-region-after-change.  The sequence is as
follows:
(i) In AWK Mode, font-lock-extend-after-change-region-function is
  non-nil, and this function is called to set `beg' and `end', the
  bounds of the region to fontify.  In the above case, this region is
  exactly line 1.
(ii) f-l-extend-j-l-r-a-change adds 1 to end.  The region (beg end) now
  covers line 1 and the "B" at the beginning of line 2.
(iii) After-change fontification fontifies this spurious region, (beg
  end).  Since "B" is "on its own", it appears to be an ordinary
  variable use, hence is fontified without a face.
(iv) Half a second later, context fontification starts from `end', finds
  "EGIN" and fontifies this as an ordinary variable, without a face.

The solution I propose is that should the region have been set by
font-lock-extend-after-change-region-function, the `end' position
should be accepted as is.

Here is a patch to implement this.  I will commit it soon to emacs-25 if
I hear no objections.

diff --git a/lisp/font-lock.el b/lisp/font-lock.el
index 4a92069..3c1f01d 100644
--- a/lisp/font-lock.el
+++ b/lisp/font-lock.el
@@ -1302,15 +1302,18 @@ font-lock-extend-jit-lock-region-after-change
                       (point-min))))
       (when (< end (point-max))
         (setq end
-              (if (get-text-property end 'font-lock-multiline)
-                  (or (text-property-any end (point-max)
-                                         'font-lock-multiline nil)
-                      (point-max))
+              (cond
+               ((get-text-property end 'font-lock-multiline)
+                (or (text-property-any end (point-max)
+                                       'font-lock-multiline nil)
+                    (point-max)))
+               ;; If `end' has been set by the function above, don't corrupt it.
+               (font-lock-extend-after-change-region-function end)
                 ;; Rounding up to a whole number of lines should include the
                 ;; line right after `end'.  Typical case: the first char of
                 ;; the line was deleted.  Or a \n was inserted in the middle
                 ;; of a line.
-                (1+ end))))
+               (t (1+ end)))))
       ;; Finally, pre-enlarge the region to a whole number of lines, to try
       ;; and anticipate what font-lock-default-fontify-region will do, so as to
       ;; avoid double-redisplay.

-- 
Alan Mackenzie (Nuremberg, Germany).