GNU bug report logs - #13755
24.3.50; regression: incorrect font-lock highlighting

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 13755 in the body.
You can then email your comments to 13755 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: <bug-gnu-emacs <at> gnu.org>
Subject: 24.3.50; regression: incorrect font-lock highlighting
Date: Mon, 18 Feb 2013 14:38:42 -0800

[Message part 1 (text/plain, inline)]

The build from 2013-02-17 broke not only whether/when font-locking
occurs (see bugs #13751 and #13730).  It broke also what gets
highlighted.  A build from Feb 15 shows no font-lock problems.
 
See the two attached screenshots.  The one from 02-17 does not highlight
the command name (`icicle-customize-apropos-options-of-type') completely,
as it should and as does the one from 02-15.
 
Attached are also files showing the values of `font-lock-keywords' in an
Emacs Lisp buffer from builds of Feb 15 and Feb 17.  Diffing them shows
something interesting to you, perhaps (doesn't do much for me).  But
what is significant here is that the part that should govern the
highlighting that I notice broken is identical.  So something else is
wrong, somewhere.

In GNU Emacs 24.3.50.1 (i386-mingw-nt5.1.2600)
 of 2013-02-17 on VBOX-W7
Bzr revision: 111822 rgm <at> gnu.org-20130217190146-mm9bh3227ev56bus
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --with-gcc (4.7) --no-opt --enable-checking --cflags
 -IC:/emacs/libs/libXpm-3.5.10/include -IC:/emacs/libs/libXpm-3.5.10/src
 -IC:/emacs/libs/libpng-dev_1.4.3-1_win32/include
 -IC:/emacs/libs/zlib-dev_1.2.5-2_win32/include
 -IC:/emacs/libs/giflib-4.1.4-1-lib/include
 -IC:/emacs/libs/jpeg-6b-4-lib/include
 -IC:/emacs/libs/tiff-3.8.2-1-lib/include
 -IC:/emacs/libs/libxml2-2.7.8-w32-bin/include/libxml2
 -IC:/emacs/libs/gnutls-3.1.8-w32/include
 -IC:/emacs/libs/libiconv-1.14-2-mingw32-dev/include'
 

[throw-f-l-kw-new.txt (text/plain, attachment)]

[throw-f-l-kw.txt (text/plain, attachment)]

[throw-f-l-bug-02-17.png (image/png, attachment)]

[throw-f-l-bug-02-15.png (image/png, attachment)]

Message #8 received at 13755 <at> debbugs.gnu.org (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: <13755 <at> debbugs.gnu.org>
Subject: RE: bug#13755: 24.3.50; regression: incorrect font-lock highlighting
Date: Mon, 25 Feb 2013 22:16:21 -0800

> The build from 2013-02-17 broke not only whether/when font-locking
> occurs (see bugs #13751 and #13730).  It broke also what gets
> highlighted.  A build from Feb 15 shows no font-lock problems.
>  
> See the two attached screenshots.  The one from 02-17 does 
> not highlight the command name
> (`icicle-customize-apropos-options-of-type') 
> completely, as it should and as does the one from 02-15.

Let me add, in case it helps, that I use this code:

(font-lock-add-keywords
 'emacs-lisp-mode
 `((,(concat "(" (regexp-opt
                  '("icicle-define-add-to-alist-command"
                    "icicle-define-command"
                    "icicle-define-file-command"
                    "icicle-define-sort-command")
                  t)
             "\\>[ \t'\(]*\\(\\sw+\\)?")
    (1 font-lock-keyword-face)
    ;; Index (2 or 3) depends on whether or not shy groups are supported.
    ,(list (if (string-match "\\(?:\\)" "") 2 3)
           'font-lock-function-name-face nil t))
   ("(\\(icicle-condition-case-no-debug\\)\\>"
    1 font-lock-keyword-face)))

Message #11 received at 13755 <at> debbugs.gnu.org (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: <13755 <at> debbugs.gnu.org>
Subject: RE: bug#13755: 24.3.50; regression: incorrect font-lock highlighting
Date: Sat, 9 Mar 2013 06:38:39 -0800

Font-locking was broken in several different ways recently, at the same time.

At least 5 bugs were reported regarding this breakage: 13751, 13755, 13730,
13767, and 13806.  They were all fixed except bug #13755, which has not even
been looked at, AFAICT.

Could someone please take a look at 13755?  Thx.

Message #16 received at 13755-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "Drew Adams" <drew.adams <at> oracle.com>
Cc: 13755-done <at> debbugs.gnu.org
Subject: Re: bug#13755: 24.3.50; regression: incorrect font-lock highlighting
Date: Sat, 09 Mar 2013 10:52:29 -0500

> (font-lock-add-keywords
>  'emacs-lisp-mode
>  `((,(concat "(" (regexp-opt
>                   '("icicle-define-add-to-alist-command"
>                     "icicle-define-command"
>                     "icicle-define-file-command"
>                     "icicle-define-sort-command")
>                   t)
>              "\\>[ \t'\(]*\\(\\sw+\\)?")
                ^^^
There's your bug: you should use \\_>.

>    ("(\\(icicle-condition-case-no-debug\\)\\>"
                                            ^^^
Same here.


        Stefan

Message #21 received at 13755 <at> debbugs.gnu.org (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Stefan Monnier'" <monnier <at> iro.umontreal.ca>
Cc: 13755 <at> debbugs.gnu.org
Subject: RE: bug#13755: 24.3.50; regression: incorrect font-lock highlighting
Date: Sat, 9 Mar 2013 09:16:25 -0800

> > (font-lock-add-keywords
> >  'emacs-lisp-mode
> >  `((,(concat "(" (regexp-opt
> >                   '("icicle-define-add-to-alist-command"
> >                     "icicle-define-command"
> >                     "icicle-define-file-command"
> >                     "icicle-define-sort-command")
> >                   t)
> >              "\\>[ \t'\(]*\\(\\sw+\\)?")
>                 ^^^
> There's your bug: you should use \\_>.

Interesting.  I'm surprised that Emacs has for so long (decades) matched only a
word-end here and not a symbol-end.

Be that as it may, changing from \\> to \\_> here does NOT fix the bug.

Here is the relevant entry from `font-lock-keywords' in Emacs-Lisp mode:

("(\\(icicle-define-\\(?:\\(?:\\(?:add-to-alist\\|file\\|sort\\)-\\)?command\\)\
\)\\_>[ 	'(]*\\(\\sw+\\)?"
   (1 font-lock-keyword-face)
   (2 font-lock-function-name-face nil t))

The problem is not the highlighting of `icicle-define-command', which is
correctly in `font-lock-keyword-face' (when using either \\> or \\_>, for that
matter).

The problem is the highlighting of the command name that follows keyword
`icicle-define-command'.  The command name should all be in
`font-lock-function-name-face'.  But only its first word is highlighted now, as
shown in the screenshots I sent with the bug report.  That is the regression.

It's as if Emacs is not counting regexp groups the same way as before.

Here is the code again (but with your suggested \\_> this time):

(font-lock-add-keywords
 'emacs-lisp-mode
 `((,(concat
      "("
      (regexp-opt
       '("icicle-define-add-to-alist-command"
         "icicle-define-command"
         "icicle-define-file-command"
         "icicle-define-sort-command")
       t)
      "\\_>[ \t'\(]*\\(\\sw+\\)?")
    (1 font-lock-keyword-face)
    ;; Index (2 or 3) depends on whether or not
    ;; shy groups are supported.
    ,(list (if (string-match "\\(?:\\)" "") 2 3)
           'font-lock-function-name-face nil t))
   ("(\\(icicle-condition-case-no-debug\\)\\_>"
    1 font-lock-keyword-face)
   ("(\\(icicle-user-error\\)\\_>"
    1 font-lock-warning-face)))

> >    ("(\\(icicle-condition-case-no-debug\\)\\>"
>                                             ^^^
> Same here.

No.  That highlights correctly.  Again, there is no problem with
`font-lock-keyword-face' highlighting.

Similarly, all the other uses of \\> in Emacs-Lisp mode `font-lock-keywords'
(and there are several, in emacs -Q) also fontify correctly.

AFAICT, you mistook the problem.  Please take another look.

Message #24 received at 13755 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "Drew Adams" <drew.adams <at> oracle.com>
Cc: 13755 <at> debbugs.gnu.org
Subject: Re: bug#13755: 24.3.50; regression: incorrect font-lock highlighting
Date: Sun, 10 Mar 2013 00:39:59 -0500

>       "\\_>[ \t'\(]*\\(\\sw+\\)?")
                         ^^^^
and this needs to be \\(?:\\sw\\|\\s_\\).


        Stefan

Message #27 received at 13755 <at> debbugs.gnu.org (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Stefan Monnier'" <monnier <at> iro.umontreal.ca>
Cc: 13755 <at> debbugs.gnu.org
Subject: RE: bug#13755: 24.3.50; regression: incorrect font-lock highlighting
Date: Sun, 10 Mar 2013 10:16:29 -0700

> >              "\\>[ \t'\(]*\\(\\sw+\\)?")
>                 ^^^
> There's your bug: you should use \\_>.
> 
> >    ("(\\(icicle-condition-case-no-debug\\)\\>"
>                                             ^^^
> Same here.

Actually, \\_> does NOT work for older Emacs versions (e.g. 20).

And \\>, instead of \\_>, DOES work - for all versions, even for the latest
build.  It's not clear to me why that is so, given that you suggested changing
it, and given that hyphen has `symbol', not `word' syntax in Emacs-Lisp mode.
Can you explain this?

> >       "\\_>[ \t'\(]*\\(\\sw+\\)?")
>                          ^^^^
> and this needs to be \\(?:\\sw\\|\\s_\\).

I now have the following, which seems to work for all Emacs versions (20 through
latest 24 trunk build):

(font-lock-add-keywords
 'emacs-lisp-mode
 `((,(concat "(" (regexp-opt
                  '("icicle-define-add-to-alist-command"
                    "icicle-define-bookmark-command"
                    "icicle-define-bookmark-other-window-command"
                    "icicle-define-command"
                    "icicle-define-file-command"
                    "icicle-define-search-bookmark-command"
                    "icicle-define-sort-command")
                  t)
             "\\>[ \t'\(]*\\(\\(\\sw\\|\\s_\\)+\\)?")
    (1 font-lock-keyword-face)
    ,(list (if (string-match "\\(?:\\)" "") 2 3)
           'font-lock-function-name-face nil t))
   ("(\\(icicle-condition-case-no-debug\\)\\>"
    1 font-lock-keyword-face)
   ("(\\(icicle-user-error\\)\\>"
    1 font-lock-warning-face)))

(FWIW, back in 2006 I used "\\s-+\\(\\sw\\(\\sw\\|\\s_\\)+\\)", but I changed
that to "\\>[ \t'\(]*\\(\\sw+\\)?" - I do not recall just why, wrt the last
part.  IOW, I purposely changed to use just \\sw.)

Please help me understand more.  Just what changed wrt font-locking of Lisp code
between the Emacs builds of Feb 15 and Feb 17?  What is the change that makes
such code changes necessary?

I've searched NEWS and looked at font-lock.el, but so far I do not understand
what you have changed in Emacs that broke the regexp that worked previously
(across all Emacs versions):
"\\>[ \t'\(]*\\(\\sw+\\)?".

I would like to understand why `word' syntax worked here before, even though
hyphen has and had then `symbol', not `word' syntax.  I would like to understand
just what has changed in Emacs.  Was there, e.g., a bug before, which made `-'
sometimes be treated as a word constituent?  What is behind this change?  Thx.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.