GNU bug report logs - #22960
25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22960 in the body.
You can then email your comments to 22960 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: bug-gnu-emacs <at> gnu.org
Subject: 25.0.92;
 smie-indent-keyword triggers the "Bumped into unknown token" error
Date: Wed, 09 Mar 2016 17:34:13 +0200

When trying to indent the beginning of a multiline string literal.

Here's an example Ruby code (almost unmodified, from a real project):

foo_bar(
   "# coding:koi8-r
         \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42
         puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1")

Try to reindent the second line.  During its execution,
smie-indent-keyword calls smie-indent-forward-token from the third line
(in the middle of the string literal).  ruby-smie--forward-token returns
"" there (because why not?), and smie-indent-forward-token freaks out.

IMHO, smie-forward-token-function doesn't have to know how to handle
that context, and the fault lies in smie-indent-keyword.

Stefan, could you please take a look?


In GNU Emacs 25.0.92.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.7)
 of 2016-03-05 built on axl
Repository revision: dbb02bfadc38297c06bbd3462e53192d0ced76dc
Windowing system distributor 'The X.Org Foundation', version 11.0.11702000
System Description:	Ubuntu 15.10

Message #8 received at 22960 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 22960 <at> debbugs.gnu.org
Subject: Re: bug#22960: 25.0.92;
 smie-indent-keyword triggers the "Bumped into unknown token" error
Date: Wed, 16 Mar 2016 22:56:45 -0400

> foo_bar(
>    "# coding:koi8-r
>          \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42
>          puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1")
>
> Try to reindent the second line.  During its execution,

I installed the patch below into emacs-25, which should fix it.


        Stefan


diff --git a/lisp/emacs-lisp/smie.el b/lisp/emacs-lisp/smie.el
index 495ba7c..1d8f0cb 100644
--- a/lisp/emacs-lisp/smie.el
+++ b/lisp/emacs-lisp/smie.el
@@ -1493,7 +1493,10 @@ should not be computed on the basis of the following token."
                            (let ((endpos (point)))
                              (goto-char pos)
                              (forward-line 1)
-                             (and (equal res (smie-indent-forward-token))
+                             ;; As seen in bug#22960, pos may be inside
+                             ;; a string, and forward-token may then stumble.
+                             (and (ignore-errors
+                                    (equal res (smie-indent-forward-token)))
                                   (eq (point) endpos)))))
                     nil
                   (goto-char pos)

Message #11 received at 22960 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: 22960 <at> debbugs.gnu.org
Subject: Re: bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into
 unknown token" error
Date: Thu, 17 Mar 2016 14:48:43 +0200

On 03/17/2016 04:56 AM, Stefan Monnier wrote:
>> foo_bar(
>>    "# coding:koi8-r
>>          \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42
>>          puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1")
>>
>> Try to reindent the second line.  During its execution,
>
> I installed the patch below into emacs-25, which should fix it.

Thanks, that did it.

By the way, while testing it, I've noticed that f54e5ccf added new 
behavior to indent-for-tab-command. Do you think it should be mentioned 
in NEWS?

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.