GNU bug report logs - #36092
27.0.50; incorrect highlighting in nXML mode with ' and raw > characters

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 36092 in the body.
You can then email your comments to 36092 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Vincent Lefevre <vincent <at> vinc17.net>
To: bug-gnu-emacs <at> gnu.org
Subject: 27.0.50;
 incorrect highlighting in nXML mode with ' and raw > characters
Date: Tue, 04 Jun 2019 17:04:44 +0200

On the following file, lines 26 and below are not highlighted
(i.e. they get the normal color). This is reproducible with
and without the -Q option.

<?xml version="1.0" encoding="utf-8"?>
<root>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</a>
<a>
<element>
'>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
</element>
</a>
</root>


In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.5)
 of 2019-06-04 built on cventin
Repository revision: dd7bc5de3f59237f21e1c4b70f0ba97549ea1fb4
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Debian GNU/Linux 10 (buster)

Recent messages:
Loading /home/vlefevre/share/emacs/site-lisp/mutteditor.el (source)...done
Loading time...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Using vacuous schema [2 times]
Error during redisplay: (jit-lock-function 1583) signaled (cl-assertion-failed (>= (cadr sgml--syntax-propertize-ppss) 0))

Configured using:
 'configure --prefix=/home/vlefevre/opt/emacs-master'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY INOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LIBSYSTEMD PDUMPER LCMS2
GMP

Important settings:
  value of $LC_COLLATE: POSIX
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_TIME: en_DK
  value of $LANG: POSIX
  locale-coding-system: utf-8-unix

Major mode: nXML

Minor modes in effect:
  display-time-mode: t
  show-paren-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr warnings emacsbug message rmc puny dired
dired-loaddefs format-spec rfc822 mml mml-sec password-cache epa derived
epg epg-config gnus-util rmail rmail-loaddefs text-property-search
time-date mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils vc-dispatcher vc-svn rng-xsd xsd-regexp rng-cmpct
rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt
rng-util rng-pttrn nxml-ns easymenu nxml-mode nxml-outln nxml-rap
sgml-mode seq byte-opt gv bytecomp byte-compile cconv dom nxml-util
nxml-enc xmltok elec-pair time cus-start cus-load paren cc-styles
cc-align cc-engine cc-vars cc-defs edmacro kmacro cl-loaddefs cl-lib
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 81866 9310)
 (symbols 48 10247 1)
 (strings 32 25966 1934)
 (string-bytes 1 826776)
 (vectors 16 13514)
 (vector-slots 8 158635 8782)
 (floats 8 35 19)
 (intervals 56 465 0)
 (buffers 992 14))

Message #8 received at 36092 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Vincent Lefevre <vincent <at> vinc17.net>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, 36092 <at> debbugs.gnu.org
Subject: Re: bug#36092: 27.0.50;
 incorrect highlighting in nXML mode with ' and raw > characters
Date: Tue, 04 Jun 2019 21:14:33 -0400

[Message part 1 (text/plain, inline)]

Vincent Lefevre <vincent <at> vinc17.net> writes:

> On the following file, lines 26 and below are not highlighted
> (i.e. they get the normal color). This is reproducible with
> and without the -Q option.

It's hitting an assertion failure during syntax-propertizing.  The
problem seems to be that the parse-partial-sexp call in
sgml--syntax-propertize-ppss isn't using the right syntax table.  I
wonder why it doesn't happen for sgml-mode though (as far I can tell).
Stefan, any thoughts about that?

Patch below fixes it, but it might just be papering over another
problem.

[0001-Fix-nxml-syntax-propertizing-assertion-failure-Bug-3.patch (text/x-diff, inline)]

From ed59843b7fa456e0614c2c63e1479458642671ea Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs <at> gmail.com>
Date: Tue, 4 Jun 2019 21:08:26 -0400
Subject: [PATCH] Fix nxml syntax propertizing assertion failure (Bug#36092)

* lisp/textmodes/sgml-mode.el (sgml-syntax-propertize): Use the
syntax-ppss-table syntax-table.
* test/lisp/nxml/nxml-mode-tests.el (nxml-mode->-after-quote): New
test.
---
 lisp/textmodes/sgml-mode.el       | 17 +++++++++--------
 test/lisp/nxml/nxml-mode-tests.el | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/lisp/textmodes/sgml-mode.el b/lisp/textmodes/sgml-mode.el
index 0c5d5e56a6..09211e1661 100644
--- a/lisp/textmodes/sgml-mode.el
+++ b/lisp/textmodes/sgml-mode.el
@@ -399,14 +399,15 @@ (eval-and-compile
 
 (defun sgml-syntax-propertize (start end)
   "Syntactic keywords for `sgml-mode'."
-  (setq sgml--syntax-propertize-ppss (cons start (syntax-ppss start)))
-  (cl-assert (>= (cadr sgml--syntax-propertize-ppss) 0))
-  (sgml-syntax-propertize-inside end)
-  (funcall
-   (syntax-propertize-rules sgml-syntax-propertize-rules)
-   start end)
-  ;; Catch any '>' after the last quote.
-  (sgml--syntax-propertize-ppss end))
+  (with-syntax-table syntax-ppss-table
+    (setq sgml--syntax-propertize-ppss (cons start (syntax-ppss start)))
+    (cl-assert (>= (cadr sgml--syntax-propertize-ppss) 0))
+    (sgml-syntax-propertize-inside end)
+    (funcall
+     (syntax-propertize-rules sgml-syntax-propertize-rules)
+     start end)
+    ;; Catch any '>' after the last quote.
+    (sgml--syntax-propertize-ppss end)))
 
 (defun sgml-syntax-propertize-inside (end)
   (let ((ppss (syntax-ppss)))
diff --git a/test/lisp/nxml/nxml-mode-tests.el b/test/lisp/nxml/nxml-mode-tests.el
index 70816bb9de..53416b4280 100644
--- a/test/lisp/nxml/nxml-mode-tests.el
+++ b/test/lisp/nxml/nxml-mode-tests.el
@@ -99,5 +99,20 @@ (ert-deftest nxml-mode-prolog-comment ()
     (should (nth 4 (syntax-ppss)))
     (search-forward "comment3")))
 
+(ert-deftest nxml-mode->-after-quote ()
+  "Reduction from Bug#36092."
+  (with-temp-buffer
+    (insert "<root>\n"
+            (make-string 1794 ?a) "\n"
+            "'>"
+            (make-string 196 ?a) "\n"
+            "</root>")
+    (nxml-mode)
+    (syntax-propertize 2001)
+    (syntax-propertize (point-max))     ; Triggered an assert failure.
+    ;; Check that last tag is parsed as a tag.
+    (should (= 1 (- (car (syntax-ppss (1- (point-max))))
+                    (car (syntax-ppss (point-max))))))))
+
 (provide 'nxml-mode-tests)
 ;;; nxml-mode-tests.el ends here
-- 
2.11.0

Message #11 received at 36092 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: Vincent Lefevre <vincent <at> vinc17.net>, 36092 <at> debbugs.gnu.org
Subject: Re: bug#36092: 27.0.50;
 incorrect highlighting in nXML mode with ' and raw > characters
Date: Tue, 04 Jun 2019 21:48:19 -0400

> It's hitting an assertion failure during syntax-propertizing.  The
> problem seems to be that the parse-partial-sexp call in
> sgml--syntax-propertize-ppss isn't using the right syntax table.

Hmm...

>  (defun sgml-syntax-propertize (start end)
>    "Syntactic keywords for `sgml-mode'."
> -  (setq sgml--syntax-propertize-ppss (cons start (syntax-ppss start)))
> -  (cl-assert (>= (cadr sgml--syntax-propertize-ppss) 0))
> -  (sgml-syntax-propertize-inside end)
> -  (funcall
> -   (syntax-propertize-rules sgml-syntax-propertize-rules)
> -   start end)
> -  ;; Catch any '>' after the last quote.
> -  (sgml--syntax-propertize-ppss end))
> +  (with-syntax-table syntax-ppss-table
> +    (setq sgml--syntax-propertize-ppss (cons start (syntax-ppss start)))
> +    (cl-assert (>= (cadr sgml--syntax-propertize-ppss) 0))
> +    (sgml-syntax-propertize-inside end)
> +    (funcall
> +     (syntax-propertize-rules sgml-syntax-propertize-rules)
> +     start end)
> +    ;; Catch any '>' after the last quote.
> +    (sgml--syntax-propertize-ppss end)))

OMG!

We use syntax-ppss-table in syntax-ppss but not in syntax-propertize!
How dumb could I be!
Of course, it usually works fine because syntax-propertize is usually
called from syntax-ppss, but ... well ... not always,

I just install the patch below into `master` to address that.


        Stefan


diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
index 9c6d5b5829..b6cba44e77 100644
--- a/lisp/emacs-lisp/syntax.el
+++ b/lisp/emacs-lisp/syntax.el
@@ -283,6 +283,9 @@ syntax-propertize-via-font-lock
         ;; In case it was eval'd/compiled.
         (setq keywords font-lock-syntactic-keywords)))))
 
+(defvar-local syntax-ppss-table nil
+  "Syntax-table to use during `syntax-ppss', if any.")
+
 (defun syntax-propertize (pos)
   "Ensure that syntax-table properties are set until POS (a buffer point)."
   (when (< syntax-propertize--done pos)
@@ -301,47 +304,48 @@ syntax-propertize
 	          #'syntax-ppss-flush-cache 99 t))
       (save-excursion
         (with-silent-modifications
-          (make-local-variable 'syntax-propertize--done) ;Just in case!
-          (let* ((start (max (min syntax-propertize--done (point-max))
-                             (point-min)))
-                 (end (max pos
-                           (min (point-max)
-                                (+ start syntax-propertize-chunk-size))))
-                 (funs syntax-propertize-extend-region-functions))
-            (while funs
-              (let ((new (funcall (pop funs) start end))
-                    ;; Avoid recursion!
-                    (syntax-propertize--done most-positive-fixnum))
-                (if (or (null new)
-                        (and (>= (car new) start) (<= (cdr new) end)))
-                    nil
-                  (setq start (car new))
-                  (setq end (cdr new))
-                  ;; If there's been a change, we should go through the
-                  ;; list again since this new position may
-                  ;; warrant a different answer from one of the funs we've
-                  ;; already seen.
-                  (unless (eq funs
-                              (cdr syntax-propertize-extend-region-functions))
-                    (setq funs syntax-propertize-extend-region-functions)))))
-            ;; Flush ppss cache between the original value of `start' and that
-            ;; set above by syntax-propertize-extend-region-functions.
-            (syntax-ppss-flush-cache start)
-            ;; Move the limit before calling the function, so the function
-            ;; can use syntax-ppss.
-            (setq syntax-propertize--done end)
-            ;; (message "syntax-propertizing from %s to %s" start end)
-            (remove-text-properties start end
-                                    '(syntax-table nil syntax-multiline nil))
-            ;; Avoid recursion!
-            (let ((syntax-propertize--done most-positive-fixnum))
-              (funcall syntax-propertize-function start end))))))))
+          (with-syntax-table syntax-ppss-table
+            (make-local-variable 'syntax-propertize--done) ;Just in case!
+            (let* ((start (max (min syntax-propertize--done (point-max))
+                               (point-min)))
+                   (end (max pos
+                             (min (point-max)
+                                  (+ start syntax-propertize-chunk-size))))
+                   (funs syntax-propertize-extend-region-functions))
+              (while funs
+                (let ((new (funcall (pop funs) start end))
+                      ;; Avoid recursion!
+                      (syntax-propertize--done most-positive-fixnum))
+                  (if (or (null new)
+                          (and (>= (car new) start) (<= (cdr new) end)))
+                      nil
+                    (setq start (car new))
+                    (setq end (cdr new))
+                    ;; If there's been a change, we should go through the
+                    ;; list again since this new position may
+                    ;; warrant a different answer from one of the funs we've
+                    ;; already seen.
+                    (unless (eq funs
+                                (cdr syntax-propertize-extend-region-functions))
+                      (setq funs syntax-propertize-extend-region-functions)))))
+              ;; Flush ppss cache between the original value of `start' and that
+              ;; set above by syntax-propertize-extend-region-functions.
+              (syntax-ppss-flush-cache start)
+              ;; Move the limit before calling the function, so the function
+              ;; can use syntax-ppss.
+              (setq syntax-propertize--done end)
+              ;; (message "syntax-propertizing from %s to %s" start end)
+              (remove-text-properties start end
+                                      '(syntax-table nil syntax-multiline nil))
+              ;; Avoid recursion!
+              (let ((syntax-propertize--done most-positive-fixnum))
+                (funcall syntax-propertize-function start end)))))))))
 
 ;;; Link syntax-propertize with syntax.c.
 
 (defvar syntax-propertize-chunks
   ;; We're not sure how far we'll go.  In my tests, using chunks of 2000
-  ;; brings to overhead to something negligible.  Passing ‘charpos’ directly
+  ;; brings the overhead to something negligible.  Passing ‘charpos’ directly
   ;; also works (basically works line-by-line) but results in an overhead which
   ;; I thought was a bit too high (like around 50%).
   2000)
@@ -450,9 +454,6 @@ syntax-ppss--update-stats
     (cl-incf (car pair))
     (cl-incf (cdr pair) (- new old))))
 
-(defvar-local syntax-ppss-table nil
-  "Syntax-table to use during `syntax-ppss', if any.")
-
 (defun syntax-ppss--data ()
   (if (eq (point-min) 1)
       (progn

Message #14 received at 36092 <at> debbugs.gnu.org (full text, mbox):

From: Vincent Lefevre <vincent <at> vinc17.net>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Noam Postavsky <npostavs <at> gmail.com>, 36092 <at> debbugs.gnu.org
Subject: Re: bug#36092: 27.0.50; incorrect highlighting in nXML mode with '
 and raw > characters
Date: Wed, 5 Jun 2019 13:28:35 +0200

On 2019-06-04 21:48:19 -0400, Stefan Monnier wrote:
> I just install the patch below into `master` to address that.

Thanks. Everything seems fine now.

-- 
Vincent Lefèvre <vincent <at> vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Message #17 received at 36092 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Vincent Lefevre <vincent <at> vinc17.net>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, 36092 <at> debbugs.gnu.org
Subject: Re: bug#36092: 27.0.50;
 incorrect highlighting in nXML mode with ' and raw > characters
Date: Wed, 05 Jun 2019 09:14:26 -0400

tags 36092 fixed
close 36092 
quit

Vincent Lefevre <vincent <at> vinc17.net> writes:

> On 2019-06-04 21:48:19 -0400, Stefan Monnier wrote:
>> I just install the patch below into `master` to address that.
>
> Thanks. Everything seems fine now.

Okay, adding just the test case from patch my above and closing.

7ad5e3915f 2019-06-05T09:10:23-04:00 "; Add test for Bug#36092"
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=7ad5e3915fe0fad7c8053b53cab1aa7aa74193fe

dec148939a 2019-06-05T07:29:23-04:00 "Do not use syntax-ppss-table in syntax-propertize when nil (Bug#36095)"
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=dec148939aad8391875e73784af2bcb0a01100bb

0b3982b1a3 2019-06-04T21:48:06-04:00 "* lisp/emacs-lisp/syntax.el: Use syntax-ppss-table for syntax-propertize."
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=0b3982b1a3892486fd9e4916b9cfafa12ddd9137

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.