From unknown Mon Jun 23 04:14:50 2025 X-Loop: help-debbugs@gnu.org Subject: bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 09 Mar 2016 15:35:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 22960 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: 22960@debbugs.gnu.org X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.14575376761139 (code B ref -1); Wed, 09 Mar 2016 15:35:01 +0000 Received: (at submit) by debbugs.gnu.org; 9 Mar 2016 15:34:36 +0000 Received: from localhost ([127.0.0.1]:41680 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1adg8B-0000IH-My for submit@debbugs.gnu.org; Wed, 09 Mar 2016 10:34:35 -0500 Received: from eggs.gnu.org ([208.118.235.92]:36678) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1adg8A-0000I6-Tw for submit@debbugs.gnu.org; Wed, 09 Mar 2016 10:34:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1adg7z-0003wT-QR for submit@debbugs.gnu.org; Wed, 09 Mar 2016 10:34:29 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_40,FREEMAIL_FROM, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:37839) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1adg7z-0003wN-NR for submit@debbugs.gnu.org; Wed, 09 Mar 2016 10:34:23 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55861) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1adg7y-0000Go-QW for bug-gnu-emacs@gnu.org; Wed, 09 Mar 2016 10:34:23 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1adg7t-0003vE-1h for bug-gnu-emacs@gnu.org; Wed, 09 Mar 2016 10:34:22 -0500 Received: from mail-wm0-x236.google.com ([2a00:1450:400c:c09::236]:34594) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1adg7s-0003uz-R6 for bug-gnu-emacs@gnu.org; Wed, 09 Mar 2016 10:34:16 -0500 Received: by mail-wm0-x236.google.com with SMTP id p65so197638957wmp.1 for ; Wed, 09 Mar 2016 07:34:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:subject:date:message-id:mime-version; bh=RKAu8hM333BADnpKGk37ugM8a1VbRNdPdvk7rHREg90=; b=U6Oz6ioCrZQVITmv5f98i4v/t6cjsqMNvQBBPAgFoFrft9jMEXBSTnuUYK3HTpuNkC q/LOOL6C606XKeLdKcnQHwN80AKQDpYVZ0b0QG6oN+Qo+Z6voSOMjGasaBRbz88hU0bc ku1ZiH89Ulu82816OgQpvURmxc2AceT5gXnjdRj64Caqz6i95ZDOa1nuI+AQb6QvaFwH cuPl+cdJdwGyja3ithxAS1gehahRHxZPu1Cuo/tMvy8koZmMhyFHKZjJiBmM9iEqpX7k iQlRVk+6BepfPMix23cEfKSo50fJeyKR+8aqVf+IJOHaJgnO16XRAQfDdq3ROxf0+YM0 9oAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:subject:date:message-id :mime-version; bh=RKAu8hM333BADnpKGk37ugM8a1VbRNdPdvk7rHREg90=; b=UnOxKRaR4Pid29iQfAHgpUounNf5hNrDuYkYetlWCXW0n9+88ZvYBXYbNE4SMQD7gO Gjs39T9nHnajj9LX2TU+zBVlO+8i/42rQr8CaD0tIs95p2HCtHAMG0FQ7rRcEKe/WrDq HyAF7jUg4SOOyufyo7so3p5lqdoWOzrfrm9qF1I0evRESrlrihu1hNVQBIHRZMUDtTlF 24z2Gucxq+/wyGi2n8s0IFDs0ojtTyCO6KpgbAQ1+xPruP0efS9LtHfWkdEM21RZoQM2 0xP3CidIfsBWcs8L7vYz56B6G3wMoKIy0vY4gS+irZMkrYiaKrcfii/KhBGrfREu36jq c67A== X-Gm-Message-State: AD7BkJIqEkSTS5zNgETDGiIhjrWdWlkiPQdu5NW2zwoZTp1FjayfqbRta6V0gdmY4zeP4A== X-Received: by 10.194.189.143 with SMTP id gi15mr34865941wjc.54.1457537655970; Wed, 09 Mar 2016 07:34:15 -0800 (PST) Received: from axl ([185.105.175.24]) by smtp.gmail.com with ESMTPSA id t8sm8393569wjy.41.2016.03.09.07.34.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Mar 2016 07:34:15 -0800 (PST) From: Dmitry Gutov Date: Wed, 09 Mar 2016 17:34:13 +0200 Message-ID: <86io0vd5sa.fsf@yandex.ru> MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -3.8 (---) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.8 (---) When trying to indent the beginning of a multiline string literal. Here's an example Ruby code (almost unmodified, from a real project): foo_bar( "# coding:koi8-r \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42 puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1") Try to reindent the second line. During its execution, smie-indent-keyword calls smie-indent-forward-token from the third line (in the middle of the string literal). ruby-smie--forward-token returns "" there (because why not?), and smie-indent-forward-token freaks out. IMHO, smie-forward-token-function doesn't have to know how to handle that context, and the fault lies in smie-indent-keyword. Stefan, could you please take a look? In GNU Emacs 25.0.92.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.7) of 2016-03-05 built on axl Repository revision: dbb02bfadc38297c06bbd3462e53192d0ced76dc Windowing system distributor 'The X.Org Foundation', version 11.0.11702000 System Description: Ubuntu 15.10 From unknown Mon Jun 23 04:14:50 2025 X-Loop: help-debbugs@gnu.org Subject: bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 17 Mar 2016 02:57:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 22960 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: Dmitry Gutov Cc: 22960@debbugs.gnu.org Received: via spool by 22960-submit@debbugs.gnu.org id=B22960.145818341118797 (code B ref 22960); Thu, 17 Mar 2016 02:57:02 +0000 Received: (at 22960) by debbugs.gnu.org; 17 Mar 2016 02:56:51 +0000 Received: from localhost ([127.0.0.1]:50572 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agO7H-0004t5-7p for submit@debbugs.gnu.org; Wed, 16 Mar 2016 22:56:51 -0400 Received: from pruche.dit.umontreal.ca ([132.204.246.22]:46896) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agO7D-0004sk-UT for 22960@debbugs.gnu.org; Wed, 16 Mar 2016 22:56:49 -0400 Received: from fmsmemgm.homelinux.net (lechon.iro.umontreal.ca [132.204.27.242]) by pruche.dit.umontreal.ca (8.14.7/8.14.1) with ESMTP id u2H2ujTf027831; Wed, 16 Mar 2016 22:56:46 -0400 Received: by fmsmemgm.homelinux.net (Postfix, from userid 20848) id 8E177AE6AD; Wed, 16 Mar 2016 22:56:45 -0400 (EDT) From: Stefan Monnier Message-ID: References: <86io0vd5sa.fsf@yandex.ru> Date: Wed, 16 Mar 2016 22:56:45 -0400 In-Reply-To: <86io0vd5sa.fsf@yandex.ru> (Dmitry Gutov's message of "Wed, 09 Mar 2016 17:34:13 +0200") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-NAI-Spam-Flag: NO X-NAI-Spam-Threshold: 5 X-NAI-Spam-Score: 0 X-NAI-Spam-Rules: 1 Rules triggered RV5612=0 X-NAI-Spam-Version: 2.3.0.9418 : core <5612> : inlines <4516> : streams <1604155> : uri <2167782> X-Spam-Score: -1.3 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.3 (-) > foo_bar( > "# coding:koi8-r > \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42 > puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1") > > Try to reindent the second line. During its execution, I installed the patch below into emacs-25, which should fix it. Stefan diff --git a/lisp/emacs-lisp/smie.el b/lisp/emacs-lisp/smie.el index 495ba7c..1d8f0cb 100644 --- a/lisp/emacs-lisp/smie.el +++ b/lisp/emacs-lisp/smie.el @@ -1493,7 +1493,10 @@ should not be computed on the basis of the following token." (let ((endpos (point))) (goto-char pos) (forward-line 1) - (and (equal res (smie-indent-forward-token)) + ;; As seen in bug#22960, pos may be inside + ;; a string, and forward-token may then stumble. + (and (ignore-errors + (equal res (smie-indent-forward-token))) (eq (point) endpos))))) nil (goto-char pos) From unknown Mon Jun 23 04:14:50 2025 X-Loop: help-debbugs@gnu.org Subject: bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 17 Mar 2016 12:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 22960 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: Stefan Monnier Cc: 22960@debbugs.gnu.org Received: via spool by 22960-submit@debbugs.gnu.org id=B22960.145821893414475 (code B ref 22960); Thu, 17 Mar 2016 12:49:02 +0000 Received: (at 22960) by debbugs.gnu.org; 17 Mar 2016 12:48:54 +0000 Received: from localhost ([127.0.0.1]:50725 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agXME-0003lP-1P for submit@debbugs.gnu.org; Thu, 17 Mar 2016 08:48:54 -0400 Received: from mail-wm0-f47.google.com ([74.125.82.47]:38391) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agXMC-0003lC-13 for 22960@debbugs.gnu.org; Thu, 17 Mar 2016 08:48:52 -0400 Received: by mail-wm0-f47.google.com with SMTP id l68so115575342wml.1 for <22960@debbugs.gnu.org>; Thu, 17 Mar 2016 05:48:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=cRxsQwxiT+CINjidxHQVWpSJQzYXBmUz08ybJ7CepRw=; b=Wt1iIuqKT4HveycXLbvCarqsuskCCRMav103BO5+tMYi5fDPWMseRusM4yy8ScrHRr GImAbAUMNzBgsxrZrADEuih18DLFWV7FboT4llqUrXh/gwRHkm8ixLmfZuLvJa3ilLwQ +GThnz4g/UWt9E9+PHYLIAtc5misiUoCz/1wuVBu9G8MUy0EUSElcivwJ3qCZ2OoY8HD QB/bf8mcH/OiFZm1DNDBNBUrRx0XfMjC3F7aSAA3wcTsgD/9ovSh+3xHO1eJ3eoHI1IN 1D53B37L8DsrdrUCeIMchTCqtpf5Bjq3F5HmnG1aljTSzINHgqN2MYM+SIPyBsh0e+8l RcXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:to:references:cc:from:message-id :date:user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=cRxsQwxiT+CINjidxHQVWpSJQzYXBmUz08ybJ7CepRw=; b=ABSxlsJ4OzWIRRdolqcNLgBzqphOgxN4NN2Jo/G/AqSloBW+Mbjb1afhUXRk/kyxuF ut/S8w+g4H5F1zhI65SvxziMXaAWGCPHi3HAwq2ThntNgpvaXRfY69vVgGg1V827CJLv CiQ93rPxZBlo6UKCku+KxWwNFmAhbXMta7eULswzZJD4X5GY2AUkdTRroMc7/UgH8538 smlRoZQIrBKsAwl8KVPHDAIe7Rh1B0Hpqi7/jWO0TDWcRf/aBdB/jgwhfUFAVfhnQpTL acRJhrMDbuwbC0ZcyHBaHVEx6lHyn8ujoRcrl2ereGDMmUeV8JtCe7IDf3WdjSZBvIwY /SHw== X-Gm-Message-State: AD7BkJIxOnyJg2/wxKCeQkIysWfC1Pto3a0fFMKTmELGwaJiykASvtfEZD2S/yWsSC1cAA== X-Received: by 10.194.205.103 with SMTP id lf7mr9609514wjc.147.1458218926290; Thu, 17 Mar 2016 05:48:46 -0700 (PDT) Received: from [192.168.1.2] ([185.105.175.24]) by smtp.googlemail.com with ESMTPSA id jv7sm7570681wjb.36.2016.03.17.05.48.44 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 17 Mar 2016 05:48:45 -0700 (PDT) References: <86io0vd5sa.fsf@yandex.ru> From: Dmitry Gutov Message-ID: <45d9844c-f755-2783-d25f-92094393de8d@yandex.ru> Date: Thu, 17 Mar 2016 14:48:43 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.5 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) On 03/17/2016 04:56 AM, Stefan Monnier wrote: >> foo_bar( >> "# coding:koi8-r >> \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42 >> puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1") >> >> Try to reindent the second line. During its execution, > > I installed the patch below into emacs-25, which should fix it. Thanks, that did it. By the way, while testing it, I've noticed that f54e5ccf added new behavior to indent-for-tab-command. Do you think it should be mentioned in NEWS? From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 17 08:50:15 2016 Received: (at control) by debbugs.gnu.org; 17 Mar 2016 12:50:15 +0000 Received: from localhost ([127.0.0.1]:50729 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agXNX-0003o4-C8 for submit@debbugs.gnu.org; Thu, 17 Mar 2016 08:50:15 -0400 Received: from mail-wm0-f43.google.com ([74.125.82.43]:35109) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agXNU-0003nn-UY for control@debbugs.gnu.org; Thu, 17 Mar 2016 08:50:13 -0400 Received: by mail-wm0-f43.google.com with SMTP id l68so224938662wml.0 for ; Thu, 17 Mar 2016 05:50:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:message-id:to:from:subject; bh=FranmMHKRM9zugvBRkvFPnHql4RadpphwRkxh+ptlQk=; b=oN/66e/JCGzO3U8yCiVyK0Kf947IMbAW27DUgp/RumVbp4gSI2zTBga2tdVJ/7WcKO FjVw47LE7M6j0+0FyIEg+xQmKjIxswg6Cvh2Wy1FtwQGcKDDeL5FNjHnQggqIMFi+HCH 6CCdGUMe6mvNMqoITnb0pNcoUiJnk7IHtlTBQ/CiBmKefmIzZ9f+fsikiJ1u8gfxCWf6 SUkBeitqqfg4g6Bdlu+WEOeaIYOIcLdvmomf7NsAXSgsXLuyHdH6ONmdYVgGm7w8BAxu sNRmlL2xx8IZFvu8pKYdp4vZuqOFjVs8OehueqpptYGoKwdaRBYi22LQ+NM/TXyf+lKg psPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:message-id:to:from:subject; bh=FranmMHKRM9zugvBRkvFPnHql4RadpphwRkxh+ptlQk=; b=aZUMlrSZIGByc8gtIQpfz5DjgaDrT1FGBqJZV1xXzUvR64QGtN6TAXE58A4uZBw0ex JjegmFXJTilMvVLpZd9Bous4jnQOSAgC1+GPoWRXo04tU9ewOLR0tPjKtsywZxZwGvvq J+watIs8U0+Xe7ePXtD2dE3pdzzu1smk7Gl+gFxM/i+fzb46OT6RaanIn0BtHUv/ozkB mA4ag6cV74golYVTcP2VJWArEqb8RgjVY0ldJwhX5jVmSJ85+u3QfL0YvCPvSSLHH0En VUt9SU9s6t6car+6vZXFS7ys9VIlUojibEKD6oZ36ulAHs4oPj7YREuFbyApDGjAiDwL nIGg== X-Gm-Message-State: AD7BkJI/MGuLnKXG115GpQTEDxuyuH7ISbT2sO2okcGYUDjmIIiHxbLhUZdShSvbLXT9mQ== X-Received: by 10.28.101.133 with SMTP id z127mr10839893wmb.84.1458219007435; Thu, 17 Mar 2016 05:50:07 -0700 (PDT) Received: from axl ([185.105.175.24]) by smtp.gmail.com with ESMTPSA id 74sm8032131wmn.17.2016.03.17.05.50.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Mar 2016 05:50:06 -0700 (PDT) Date: Thu, 17 Mar 2016 14:50:05 +0200 Message-Id: <86poutp8ua.fsf@yandex.ru> To: control@debbugs.gnu.org From: Dmitry Gutov Subject: control message for bug #22960 X-Spam-Score: -0.5 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) tags 22960 fixed close 22960 25.1