GNU bug report logs - #15998
24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 15998 in the body.
You can then email your comments to 15998 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Fri, 29 Nov 2013 23:45:00 +0900

`forward-sexp', or `scan-sexps', doesn't seem to skip bool vectors correctly.

Bool vector's format is like #&N"XXX".
`forward-sexp' doesn't skip the whole bool vector, but stop before
double quotation.

And, maybe, just the same thing occurs for circular objects.
It skip only the label, and stop before the object.

Is the current behavior correct?

Regards,
Shigeru

Message #8 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Cc: 15998 <at> debbugs.gnu.org
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Fri, 29 Nov 2013 12:12:50 -0500

> `forward-sexp', or `scan-sexps', doesn't seem to skip bool vectors correctly.
> Bool vector's format is like #&N"XXX".
> `forward-sexp' doesn't skip the whole bool vector, but stop before
> double quotation.

Indeed.

> And, maybe, just the same thing occurs for circular objects.
> It skip only the label, and stop before the object.

Indeed.

> Is the current behavior correct?

No, it's not.  Patches welcome.


        Stefan

Message #11 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>,
 15998 <at> debbugs.gnu.org
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Mon, 02 Dec 2013 05:08:46 +0900

I tried to fix the issue, and seemingly successful.
But I found more to do newly, possibly.  Those are,

1. #s(...)          hash table
2. ^[...] ^^[...]   char table
3. #@NUMBER         comments
4. $!               beginning of executable file
5. ##               empty symbol

I think doing for 1, 2, 5 are practically sufficient.

Shigeru

Message #14 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Cc: 15998 <at> debbugs.gnu.org
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Sun, 01 Dec 2013 15:44:17 -0500

> I tried to fix the issue, and seemingly successful.
> But I found more to do newly, possibly.  Those are,

> 1. #s(...)          hash table
> 2. ^[...] ^^[...]   char table
> 3. #@NUMBER         comments
> 4. $!               beginning of executable file
> 5. ##               empty symbol

3 should only appear in .elc files and the mode used in .elc files
(emacs-lisp-byte-code-mode) already sets up syntax-propertize-function
to handle them correctly.

I don't know what 4 is.  Or do you mean "#!" as the first two chars of
the file?  We could setup syntax-propertize-function to mark them as
comments, indeed.

> I think doing for 1, 2, 5 are practically sufficient.

Sounds right.


        Stefan

Message #17 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>,
 15998 <at> debbugs.gnu.org
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Tue, 03 Dec 2013 18:47:24 +0900

[Message part 1 (text/plain, inline)]

>> 2. ^[...] ^^[...]   char table

Sorry, but it's #^[...] #^^[...]


>> 4. $!               beginning of executable file

>I don't know what 4 is.  Or do you mean "#!" as the first two chars of
>the file?  We could setup syntax-propertize-function to mark them as
>comments, indeed.

It is just magic number of unix files.

(read-from-string "#!/usr/local/bin/emacs\n(pwd)") --> ((pwd) . 28)

see read1 in lread.c.


And, my changes to the latest emacs are,

1) add `scan-sexps' an optional argument, `elisp-syntax'.

2) pass `scan_lists' the optional argument.

3) `scan_lists' handles elisp specific handling when the argument is set.

4) `forward-sexp' calls `scan-sexps' with the optional argument non-nil
value when the current buffer's mode is elisp related mode.
Anso, do additional movement for labels of cyclic object, as is for
prefixes.

(Is a change in the last sentence above unnecessary?)


As a result, I seems doing expectedly on such an object as below.

(a  b  #&3"d"  #&99"ZZZ"  #1=a  #2= b  #1# #3= #40= (c) #40# #2# #10= #&10"A"
   #s(dummy 1 2 3)  #^^[dummy 1 2 3]  #^[dummy 1 2 3]  ##  xyz)


Regards,
Shigeru


---------------------
ChangeLog

	Add support for elisp syntax of bool vector, label of cyclic object,
	hash table, char-table and empty symbol.
	* lisp.el (forward-sexp)

	* syntax.c (scan_lists, Fscan_lists, Fscan_sexps)



lisp.el

(defun forward-sexp (&optional arg)
  "Move forward across one balanced expression (sexp).
With ARG, do it that many times.  Negative arg -N means
move backward across N balanced expressions.
This command assumes point is not in a string or comment.
Calls `forward-sexp-function' to do the work, if that is non-nil."
  (interactive "^p")
  (let ((elisp (memq major-mode '(emacs-lisp-mode
				  ;;eshell-mode
				  inferior-emacs-lisp-mode
				  lisp-interaction-mode))))
    (or arg (setq arg 1))
    (if forward-sexp-function
	(funcall forward-sexp-function arg)
      (goto-char (or (scan-sexps (point) arg elisp) (buffer-end arg)))
      (when (< arg 0)
	(when elisp
	  (save-match-data
	    (while (re-search-backward "#[0-9]+=\\s-*\\=" nil t))))
	(backward-prefix-chars)))))

[syntax.patch (application/octet-stream, attachment)]

Message #20 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Cc: 15998 <at> debbugs.gnu.org
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Tue, 03 Dec 2013 12:09:48 -0500

> 1) add `scan-sexps' an optional argument, `elisp-syntax'.

I don't think I want to add such elisp-specific hacks to the
scan-sexps code.  Better would be to change emacs-lisp-mode to set
forward-sexp-function to a function that handles those cases.


        Stefan

Message #25 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 15998 <at> debbugs.gnu.org, Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Wed, 26 Jun 2019 17:23:05 +0200

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>> 1) add `scan-sexps' an optional argument, `elisp-syntax'.
>
> I don't think I want to add such elisp-specific hacks to the
> scan-sexps code.  Better would be to change emacs-lisp-mode to set
> forward-sexp-function to a function that handles those cases.

But do we want to implement a brand new forward-sexp just for Emacs
Lisp?  I'm assuming that a Lisp-based implementation would be
uncomfortably slow...

To recap, the following doesn't forward-sexp correctly:

1. #s(...)          hash table
2. ^[...] ^^[...]   char table
5. ##               empty symbol
9. #&N"XXX"         bool vector

And especially 1) is rather annoying...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #28 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Shigeru Fukaya <shigeru.fukaya <at> gmail.com>, 15998 <at> debbugs.gnu.org
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Wed, 26 Jun 2019 13:06:22 -0400

> But do we want to implement a brand new forward-sexp just for Emacs Lisp?

No, but I think syntax-propertize can take care of those things.

> I'm assuming that a Lisp-based implementation would be
> uncomfortably slow...
>
> To recap, the following doesn't forward-sexp correctly:
>
> 1. #s(...)          hash table
> 2. ^[...] ^^[...]   char table
> 5. ##               empty symbol
> 9. #&N"XXX"         bool vector

[ I believe for (2) there's a missing # in there somewhere, right?  ]

> And especially 1) is rather annoying...

I do wonder when/where these things show up, tho.
I'd expect those to appear in .elc files and in the output of M-: and
IELM buffers, but not in emacs-lisp-mode buffers.


        Stefan

Message #31 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Shigeru Fukaya <shigeru.fukaya <at> gmail.com>, 15998 <at> debbugs.gnu.org
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Thu, 27 Jun 2019 12:30:25 +0200

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

> I do wonder when/where these things show up, tho.
> I'd expect those to appear in .elc files and in the output of M-: and
> IELM buffers, but not in emacs-lisp-mode buffers.

That's true.  Those things mostly show up when discussing Emacs Lisp and
not when writing Emacs Lisp -- so they're really more in text buffers
than in Lisp buffers...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #34 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 15998 <at> debbugs.gnu.org,
 Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Sat, 06 Jul 2019 22:08:39 -0400

[Message part 1 (text/plain, inline)]

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>> But do we want to implement a brand new forward-sexp just for Emacs Lisp?
>
> No, but I think syntax-propertize can take care of those things.

The patch below seems to work, no new sexp movement commands needed?

[0001-Handle-elisp-syntax-better-Bug-15998.patch (text/x-diff, inline)]

From 463f98c55ff7d6bde576a9538d3ae3a33a576cf5 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs <at> gmail.com>
Date: Sat, 6 Jul 2019 21:55:03 -0400
Subject: [PATCH] Handle elisp #-syntax better (Bug#15998)

* elisp-mode.el (elisp-mode-syntax-propertize): New function.
(emacs-lisp-mode): Set it as syntax-propertize-function.
---
 lisp/progmodes/elisp-mode.el | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/lisp/progmodes/elisp-mode.el b/lisp/progmodes/elisp-mode.el
index cb1b17b447..c86277a309 100644
--- a/lisp/progmodes/elisp-mode.el
+++ b/lisp/progmodes/elisp-mode.el
@@ -213,6 +213,20 @@ emacs-lisp-macroexpand
       (if (bolp) (delete-char -1))
       (indent-region start (point)))))
 
+(defun elisp-mode-syntax-propertize (start end)
+  (goto-char start)
+  (funcall
+   (syntax-propertize-rules
+    ("##" (0 (unless (nth 8 (syntax-ppss)) ; Empty symbol.
+               (string-to-syntax "_"))))
+    ((rx "#" (or (seq (group-n 1 (+ digit) "="))     ; Object label.
+                 (seq (group-n 1 "&" (+ digit)) ?\") ; Bool-vector.
+                 (seq (group-n 1 "s") "(")           ; Record.
+                 (seq (group-n 1 (+ "^")) "[")))     ; Char-table.
+     (1 (unless (save-excursion (nth 8 (syntax-ppss (match-beginning 0))))
+          (string-to-syntax "'")))))
+   start end))
+
 (defcustom emacs-lisp-mode-hook nil
   "Hook run when entering Emacs Lisp mode."
   :options '(eldoc-mode imenu-add-menubar-index checkdoc-minor-mode)
@@ -242,6 +256,7 @@ emacs-lisp-mode
                 #'elisp-eldoc-documentation-function)
   (add-hook 'xref-backend-functions #'elisp--xref-backend nil t)
   (setq-local project-vc-external-roots-function #'elisp-load-path-roots)
+  (setq-local syntax-propertize-function #'elisp-mode-syntax-propertize)
   (add-hook 'completion-at-point-functions
             #'elisp-completion-at-point nil 'local)
   (add-hook 'flymake-diagnostic-functions #'elisp-flymake-checkdoc nil t)
-- 
2.11.0

Message #39 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 15998 <at> debbugs.gnu.org,
 Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Sun, 07 Jul 2019 09:28:08 -0400

>> No, but I think syntax-propertize can take care of those things.
> The patch below seems to work, no new sexp movement commands needed?

It should work indeed, tho I'm not sure it'll fix the original problem,
because I don't know *where* the original problem happened and this
patch only fixes the case where we do such navigation in
emacs-lisp-mode, but things #s(...) and #&"..." are *very* rare in
those buffers.

So I'm not completely sure the cost imposed by this patch is worth
the benefit.  Maybe we could start by adding those rules to the
elisp-byte-code-mode?


        Stefan

Message #42 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Michael Heerdegen <michael_heerdegen <at> web.de>,
 Lars Ingebrigtsen <larsi <at> gnus.org>, Shigeru Fukaya <shigeru.fukaya <at> gmail.com>,
 15998 <at> debbugs.gnu.org
Subject: Re: bug#15998: 24.3;
 forward-sexp (scan-sexps) doesn't do well with some SEXPs
Date: Sun, 07 Jul 2019 09:47:04 -0400

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>>> No, but I think syntax-propertize can take care of those things.
>> The patch below seems to work, no new sexp movement commands needed?
>
> It should work indeed, tho I'm not sure it'll fix the original problem,
> because I don't know *where* the original problem happened and this
> patch only fixes the case where we do such navigation in
> emacs-lisp-mode, but things #s(...) and #&"..." are *very* rare in
> those buffers.

I tend to end up with those things in my *scratch* buffer, after
evaluating various expressions (more so the #s(...) than #&"...", I
don't have much call for messing with bool vectors).  Michael bumped
into an instance of ## in org-list.el (see merged Bug#30132), although
that seems like a mistake in org.

> So I'm not completely sure the cost imposed by this patch is worth
> the benefit.

On the benefit side, it would allow removing the hack I put in for
indentation of #s(...) (that was for IELM, see Bug#31984).  On the other
hand, this seems to be breaking some edebug tests, so that needs more
investigation before it could be installed.

Message #47 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: 30132 <at> debbugs.gnu.org, Shigeru Fukaya <shigeru.fukaya <at> gmail.com>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>, 15998 <at> debbugs.gnu.org
Subject: Re: bug#30132: 27.0.50; scan-sexps and ##
Date: Fri, 06 May 2022 17:50:43 +0200

Noam Postavsky <npostavs <at> gmail.com> writes:

> * elisp-mode.el (elisp-mode-syntax-propertize): New function.
> (emacs-lisp-mode): Set it as syntax-propertize-function.

I've tried the patch, and it seems to work well.  However, it makes
edebug-tests-dotted-forms hang completely, and breaks other edebug
tests.

I've poked at this briefly, but I've been unable to see what makes
things hang here.  It would be nice to get this to work -- did you get
any further in debugging these test problems, Noam?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #52 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: 30132 <at> debbugs.gnu.org, 15998 <at> debbugs.gnu.org,
 Stefan Monnier <monnier <at> iro.umontreal.ca>,
 Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Subject: Re: bug#30132: 27.0.50; scan-sexps and ##
Date: Fri, 06 May 2022 18:30:28 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> I've poked at this briefly, but I've been unable to see what makes
> things hang here.  It would be nice to get this to work -- did you get
> any further in debugging these test problems, Noam?

The offending bit was this:

(seq (group-n 1 (+ digit) "="))     ; Object label.

Which is something that should never appear in Lisp files anyway.  The
rest seem unproblematic (except possible performance problems), but
after testing it for a few minutes, I don't really see anything
noticeable.

So I'm pushing Noam's patch to Emacs 29 (with that object label thing
removed).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #57 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 30132 <at> debbugs.gnu.org, Shigeru Fukaya <shigeru.fukaya <at> gmail.com>,
 Noam Postavsky <npostavs <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 15998 <at> debbugs.gnu.org
Subject: Re: bug#30132: 27.0.50; scan-sexps and ##
Date: Sat, 07 May 2022 02:08:36 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
> > I've poked at this briefly, but I've been unable to see what makes
> > things hang here.  It would be nice to get this to work -- did you get
> > any further in debugging these test problems, Noam?
>
> The offending bit was this:
>
> (seq (group-n 1 (+ digit) "="))     ; Object label.

I wonder why.

> Which is something that should never appear in Lisp files anyway.
> [...]
> So I'm pushing Noam's patch to Emacs 29 (with that object label thing
> removed).

Thanks.  My original use case seems to be fixed.

Michael.

Message #60 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 30132 <at> debbugs.gnu.org, 15998 <at> debbugs.gnu.org,
 Noam Postavsky <npostavs <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Subject: Re: bug#30132: 27.0.50; scan-sexps and ##
Date: Sat, 07 May 2022 06:05:07 +0200

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> > The offending bit was this:
> >
> > (seq (group-n 1 (+ digit) "="))     ; Object label.
>
> I wonder why.

Dunno how the Edebug tests are supposed to run...:  When I load
edebug-tests.el and run those tests with M-x ert-run-tests-interactively
t RET, I always get lots of fails - also with what is installed now (and
even without the patch).

Michael.

Message #63 received at 15998 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 30132 <at> debbugs.gnu.org, 15998 <at> debbugs.gnu.org,
 Noam Postavsky <npostavs <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 Shigeru Fukaya <shigeru.fukaya <at> gmail.com>
Subject: Re: bug#30132: 27.0.50; scan-sexps and ##
Date: Sat, 07 May 2022 12:17:32 +0200

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> Dunno how the Edebug tests are supposed to run...:  When I load
> edebug-tests.el and run those tests with M-x ert-run-tests-interactively
> t RET, I always get lots of fails - also with what is installed now (and
> even without the patch).

I run them with "make check" or "make edebug-tests".  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.