GNU bug report logs - #8218
23.2; scan-sexps sees inconsistent syntax

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 8218 in the body.
You can then email your comments to 8218 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dima Kogan <dkogan <at> cds.caltech.edu>
To: bug-gnu-emacs <at> gnu.org
Subject: 23.2; scan-sexps sees inconsistent syntax
Date: Thu, 10 Mar 2011 01:23:03 -0800

<#part type="text/x-perl" filename="/tmp/tst3.pm" disposition=inline description="bug demo">
<#/part>

I am seeing (scan-sexps) show inconsistent behavior depending on WHEN it
is evaluated. If the attached sample file is opened, the point placed at
the first '{', and (scan-sexps (point) 1) evaluated, the returned
position is at times of the first '}' (wrong) and at other times of the
second '}' (right). If the evaluation is done right when the file is
opened, I observe the wrong behavior. This is reliably triggered with

$ emacs -Q /tmp/tst3.pm --eval '(goto-char (scan-sexps 5 1))'

Any subsequent evaluation of the same form in that buffer produces the
correct behavior. If I turn off font-lock-mode, I consistently get
incorrect behavior. This has wide ramifications with occasional
incorrect indentation and hide-show operation, for instance.

I looked through the source a bit, and I see an inconsistency in the
syntax code returned by the
   code = SYNTAX_WITH_MULTIBYTE_CHECK (c)

line in scan_lists() in syntax.c . When things work, the '$' is
interpreted as punctuation and the following '{' as a block-open. When
things don't work, the '$' is a quote and the '{' is skipped
altogether:

| character | incorrect syntax | correct syntax |
|-----------+------------------+----------------|
| =         | Spunct           | Spunct         |
|           | Swhitespace      | Swhitespace    |
| $         | Scharquote       | Spunct         |
| {         |                  | Sopen          |
| P         | Sword            | Sword          |

syntax.c is a bit difficult to dive into, but it would be great if
someone who has experience with it would take a look.



In GNU Emacs 23.2.1 (x86_64-pc-linux-gnu, GTK+ Version 2.20.1)
 of 2010-12-11 on brahms, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.10707000
configured using `configure  '--build' 'x86_64-linux-gnu' '--build' 'x86_64-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.2/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/23.2/leim' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2' 'LDFLAGS=-g' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Perl

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<up> <up> M-: <help-echo> <down-mouse-2> <mouse-2> 
<return> <up> <up> <up> <up> <up> <up> <up> <down> 
M-x r e p o r t SPC b u g <return> s c a <backspace> 
a n - s e x p s SPC g e n e r a t e s SPC i n c o n 
s i s t e n t SPC s y n t a x C-SPC <home> M-w C-g 
C-g C-g <up> <up> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> M-: ( p o i n t ) <return> <up> M-x <up> <return> 
C-g M-x <up> <return>

Recent messages:
Loading /etc/emacs/site-start.d/50psvn.el (source)...done
Loading /etc/emacs23/site-start.d/50vm-init.el (source)...done
Loading /etc/emacs/site-start.d/50w3m-el.el (source)...done
Loading /etc/emacs/site-start.d/51semi.el (source)...done
Loading /etc/emacs/site-start.d/51speedbar.el (source)...done
Loading /etc/emacs/site-start.d/65wl-beta.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
212 (#o324, #xd4)
Quit [3 times]
92 (#o134, #x5c)
Quit

Load-path shadows:
/usr/share/emacs/23.2/site-lisp/auctex/tex-bar hides /usr/share/emacs/site-lisp/auctex/tex-bar
/usr/share/emacs/23.2/site-lisp/auctex/context-nl hides /usr/share/emacs/site-lisp/auctex/context-nl
/usr/share/emacs/23.2/site-lisp/auctex/tex-buf hides /usr/share/emacs/site-lisp/auctex/tex-buf
/usr/share/emacs/23.2/site-lisp/auctex/texmathp hides /usr/share/emacs/site-lisp/auctex/texmathp
/usr/share/emacs/23.2/site-lisp/auctex/tex hides /usr/share/emacs/site-lisp/auctex/tex
/usr/share/emacs/23.2/site-lisp/auctex/tex-mik hides /usr/share/emacs/site-lisp/auctex/tex-mik
/usr/share/emacs/23.2/site-lisp/auctex/tex-fptex hides /usr/share/emacs/site-lisp/auctex/tex-fptex
/usr/share/emacs/23.2/site-lisp/auctex/bib-cite hides /usr/share/emacs/site-lisp/auctex/bib-cite
/usr/share/emacs/23.2/site-lisp/auctex/tex-info hides /usr/share/emacs/site-lisp/auctex/tex-info
/usr/share/emacs/23.2/site-lisp/auctex/tex-jp hides /usr/share/emacs/site-lisp/auctex/tex-jp
/usr/share/emacs/23.2/site-lisp/auctex/tex-font hides /usr/share/emacs/site-lisp/auctex/tex-font
/usr/share/emacs/23.2/site-lisp/auctex/toolbar-x hides /usr/share/emacs/site-lisp/auctex/toolbar-x
/usr/share/emacs/23.2/site-lisp/auctex/latex hides /usr/share/emacs/site-lisp/auctex/latex
/usr/share/emacs/23.2/site-lisp/auctex/font-latex hides /usr/share/emacs/site-lisp/auctex/font-latex
/usr/share/emacs/23.2/site-lisp/auctex/tex-fold hides /usr/share/emacs/site-lisp/auctex/tex-fold
/usr/share/emacs/23.2/site-lisp/auctex/context hides /usr/share/emacs/site-lisp/auctex/context
/usr/share/emacs/23.2/site-lisp/auctex/context-en hides /usr/share/emacs/site-lisp/auctex/context-en
/usr/share/emacs/23.2/site-lisp/auctex/multi-prompt hides /usr/share/emacs/site-lisp/auctex/multi-prompt
/usr/share/emacs/23.2/site-lisp/auctex/tex-style hides /usr/share/emacs/site-lisp/auctex/tex-style
/usr/local/share/emacs/site-lisp/imaxima-autoconf-variables hides /usr/share/emacs/23.2/site-lisp/maxima/imaxima-autoconf-variables
/usr/local/share/emacs/site-lisp/imaxima hides /usr/share/emacs/23.2/site-lisp/maxima/imaxima
/usr/local/share/emacs/site-lisp/imath hides /usr/share/emacs/23.2/site-lisp/maxima/imath
/usr/share/emacs/site-lisp/autoconf/autotest-mode hides /usr/share/emacs/site-lisp/autotest-mode
/usr/share/emacs/23.2/site-lisp/magit hides /usr/share/emacs/site-lisp/magit
/usr/share/emacs/23.2/site-lisp/cscope/xcscope hides /usr/share/emacs/site-lisp/xcscope
/usr/share/emacs/23.2/site-lisp/cmake-data/cmake-mode hides /usr/share/emacs/site-lisp/cmake-mode
/usr/share/emacs/23.2/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs23/site-lisp/semi/pgg-pgp5 hides /usr/share/emacs/23.2/lisp/pgg-pgp5
/usr/share/emacs23/site-lisp/speedbar/sb-image hides /usr/share/emacs/23.2/lisp/sb-image
/usr/share/emacs23/site-lisp/semi/pgg hides /usr/share/emacs/23.2/lisp/pgg
/usr/share/emacs23/site-lisp/semi/pgg-parse hides /usr/share/emacs/23.2/lisp/pgg-parse
/usr/share/emacs23/site-lisp/cedet-common/ezimage hides /usr/share/emacs/23.2/lisp/ezimage
/usr/share/emacs23/site-lisp/flim/md4 hides /usr/share/emacs/23.2/lisp/md4
/usr/share/emacs23/site-lisp/semi/pgg-gpg hides /usr/share/emacs/23.2/lisp/pgg-gpg
/usr/share/emacs23/site-lisp/speedbar/speedbar hides /usr/share/emacs/23.2/lisp/speedbar
/usr/share/emacs23/site-lisp/flim/sha1 hides /usr/share/emacs/23.2/lisp/sha1
/usr/share/emacs23/site-lisp/semi/pgg-pgp hides /usr/share/emacs/23.2/lisp/pgg-pgp
/usr/share/emacs23/site-lisp/flim/hex-util hides /usr/share/emacs/23.2/lisp/hex-util
/usr/share/emacs23/site-lisp/speedbar/dframe hides /usr/share/emacs/23.2/lisp/dframe
/usr/share/emacs23/site-lisp/semi/pgg-def hides /usr/share/emacs/23.2/lisp/pgg-def
/usr/share/emacs23/site-lisp/dictionaries-common/flyspell hides /usr/share/emacs/23.2/lisp/textmodes/flyspell
/usr/share/emacs23/site-lisp/dictionaries-common/ispell hides /usr/share/emacs/23.2/lisp/textmodes/ispell
/usr/share/emacs/23.2/site-lisp/octave3.1-emacsen/octave-inf hides /usr/share/emacs/23.2/lisp/progmodes/octave-inf
/usr/share/emacs/23.2/site-lisp/octave3.1-emacsen/octave-mod hides /usr/share/emacs/23.2/lisp/progmodes/octave-mod
/usr/share/emacs23/site-lisp/org-mode/org-archive hides /usr/share/emacs/23.2/lisp/org/org-archive
/usr/share/emacs23/site-lisp/org-mode/org-mobile hides /usr/share/emacs/23.2/lisp/org/org-mobile
/usr/share/emacs23/site-lisp/org-mode/org-ascii hides /usr/share/emacs/23.2/lisp/org/org-ascii
/usr/share/emacs23/site-lisp/org-mode/org-clock hides /usr/share/emacs/23.2/lisp/org/org-clock
/usr/share/emacs23/site-lisp/org-mode/org-gnus hides /usr/share/emacs/23.2/lisp/org/org-gnus
/usr/share/emacs23/site-lisp/org-mode/org-indent hides /usr/share/emacs/23.2/lisp/org/org-indent
/usr/share/emacs23/site-lisp/org-mode/org-wl hides /usr/share/emacs/23.2/lisp/org/org-wl
/usr/share/emacs23/site-lisp/org-mode/org-exp-blocks hides /usr/share/emacs/23.2/lisp/org/org-exp-blocks
/usr/share/emacs23/site-lisp/org-mode/org-list hides /usr/share/emacs/23.2/lisp/org/org-list
/usr/share/emacs23/site-lisp/org-mode/org-w3m hides /usr/share/emacs/23.2/lisp/org/org-w3m
/usr/share/emacs23/site-lisp/org-mode/org-colview hides /usr/share/emacs/23.2/lisp/org/org-colview
/usr/share/emacs23/site-lisp/org-mode/org-freemind hides /usr/share/emacs/23.2/lisp/org/org-freemind
/usr/share/emacs23/site-lisp/org-mode/org-macs hides /usr/share/emacs/23.2/lisp/org/org-macs
/usr/share/emacs23/site-lisp/org-mode/org-crypt hides /usr/share/emacs/23.2/lisp/org/org-crypt
/usr/share/emacs23/site-lisp/org-mode/org-mhe hides /usr/share/emacs/23.2/lisp/org/org-mhe
/usr/share/emacs23/site-lisp/org-mode/org-table hides /usr/share/emacs/23.2/lisp/org/org-table
/usr/share/emacs23/site-lisp/org-mode/org hides /usr/share/emacs/23.2/lisp/org/org
/usr/share/emacs23/site-lisp/org-mode/org-bibtex hides /usr/share/emacs/23.2/lisp/org/org-bibtex
/usr/share/emacs23/site-lisp/org-mode/org-src hides /usr/share/emacs/23.2/lisp/org/org-src
/usr/share/emacs23/site-lisp/org-mode/org-compat hides /usr/share/emacs/23.2/lisp/org/org-compat
/usr/share/emacs23/site-lisp/org-mode/org-publish hides /usr/share/emacs/23.2/lisp/org/org-publish
/usr/share/emacs23/site-lisp/org-mode/org-id hides /usr/share/emacs/23.2/lisp/org/org-id
/usr/share/emacs23/site-lisp/org-mode/org-feed hides /usr/share/emacs/23.2/lisp/org/org-feed
/usr/share/emacs23/site-lisp/org-mode/org-habit hides /usr/share/emacs/23.2/lisp/org/org-habit
/usr/share/emacs23/site-lisp/org-mode/org-rmail hides /usr/share/emacs/23.2/lisp/org/org-rmail
/usr/share/emacs23/site-lisp/org-mode/org-plot hides /usr/share/emacs/23.2/lisp/org/org-plot
/usr/share/emacs23/site-lisp/org-mode/org-protocol hides /usr/share/emacs/23.2/lisp/org/org-protocol
/usr/share/emacs23/site-lisp/org-mode/org-faces hides /usr/share/emacs/23.2/lisp/org/org-faces
/usr/share/emacs23/site-lisp/org-mode/org-latex hides /usr/share/emacs/23.2/lisp/org/org-latex
/usr/share/emacs23/site-lisp/org-mode/org-jsinfo hides /usr/share/emacs/23.2/lisp/org/org-jsinfo
/usr/share/emacs23/site-lisp/org-mode/org-xoxo hides /usr/share/emacs/23.2/lisp/org/org-xoxo
/usr/share/emacs23/site-lisp/org-mode/org-vm hides /usr/share/emacs/23.2/lisp/org/org-vm
/usr/share/emacs23/site-lisp/org-mode/org-mouse hides /usr/share/emacs/23.2/lisp/org/org-mouse
/usr/share/emacs23/site-lisp/org-mode/org-timer hides /usr/share/emacs/23.2/lisp/org/org-timer
/usr/share/emacs23/site-lisp/org-mode/org-footnote hides /usr/share/emacs/23.2/lisp/org/org-footnote
/usr/share/emacs23/site-lisp/org-mode/org-html hides /usr/share/emacs/23.2/lisp/org/org-html
/usr/share/emacs23/site-lisp/org-mode/org-inlinetask hides /usr/share/emacs/23.2/lisp/org/org-inlinetask
/usr/share/emacs23/site-lisp/org-mode/org-attach hides /usr/share/emacs/23.2/lisp/org/org-attach
/usr/share/emacs23/site-lisp/org-mode/org-agenda hides /usr/share/emacs/23.2/lisp/org/org-agenda
/usr/share/emacs23/site-lisp/org-mode/org-datetree hides /usr/share/emacs/23.2/lisp/org/org-datetree
/usr/share/emacs23/site-lisp/org-mode/org-bbdb hides /usr/share/emacs/23.2/lisp/org/org-bbdb
/usr/share/emacs23/site-lisp/org-mode/org-irc hides /usr/share/emacs/23.2/lisp/org/org-irc
/usr/share/emacs23/site-lisp/org-mode/org-mew hides /usr/share/emacs/23.2/lisp/org/org-mew
/usr/share/emacs23/site-lisp/org-mode/org-mac-message hides /usr/share/emacs/23.2/lisp/org/org-mac-message
/usr/share/emacs23/site-lisp/org-mode/org-docbook hides /usr/share/emacs/23.2/lisp/org/org-docbook
/usr/share/emacs23/site-lisp/org-mode/org-info hides /usr/share/emacs/23.2/lisp/org/org-info
/usr/share/emacs23/site-lisp/org-mode/org-exp hides /usr/share/emacs/23.2/lisp/org/org-exp
/usr/share/emacs23/site-lisp/org-mode/org-install hides /usr/share/emacs/23.2/lisp/org/org-install
/usr/share/emacs23/site-lisp/org-mode/org-icalendar hides /usr/share/emacs/23.2/lisp/org/org-icalendar
/usr/share/emacs23/site-lisp/org-mode/org-remember hides /usr/share/emacs/23.2/lisp/org/org-remember
/usr/share/emacs23/site-lisp/flim/hmac-def hides /usr/share/emacs/23.2/lisp/net/hmac-def
/usr/share/emacs23/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/23.2/lisp/net/sasl-ntlm
/usr/share/emacs23/site-lisp/flim/sasl hides /usr/share/emacs/23.2/lisp/net/sasl
/usr/share/emacs23/site-lisp/flim/sasl-digest hides /usr/share/emacs/23.2/lisp/net/sasl-digest
/usr/share/emacs23/site-lisp/flim/hmac-md5 hides /usr/share/emacs/23.2/lisp/net/hmac-md5
/usr/share/emacs23/site-lisp/flim/ntlm hides /usr/share/emacs/23.2/lisp/net/ntlm
/usr/share/emacs23/site-lisp/flim/sasl-cram hides /usr/share/emacs/23.2/lisp/net/sasl-cram
/usr/share/emacs23/site-lisp/wl/rfc2368 hides /usr/share/emacs/23.2/lisp/mail/rfc2368
/usr/share/emacs23/site-lisp/cedet-common/cedet-files hides /usr/share/emacs/23.2/lisp/cedet/cedet-files
/usr/share/emacs23/site-lisp/cedet-common/mode-local hides /usr/share/emacs/23.2/lisp/cedet/mode-local
/usr/share/emacs23/site-lisp/cedet-common/inversion hides /usr/share/emacs/23.2/lisp/cedet/inversion
/usr/share/emacs23/site-lisp/cedet-common/cedet hides /usr/share/emacs/23.2/lisp/cedet/cedet

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
easymenu mml-sec password-cache mm-decode mm-bodies mm-encode mailcap
mail-parse rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader
gnus-util netrc time-date mm-util mail-prsvr gmm-utils wid-edit
mailheader canlock sha1 sha1-el hex-util hashcash mail-utils emacsbug
perl-mode w3m-load org-install muse-autoloads emacs-goodies-el
emacs-goodies-custom emacs-goodies-loaddefs easy-mmode cedet-autogen
advice advice-preload autoload help-fns preview-latex tex-site
auto-loads tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win
x-dnd font-setting tool-bar dnd fontset image fringe lisp-mode register
page menu-bar rfn-eshadow timer select scroll-bar mldrag mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind system-font-setting
font-render-setting gtk x-toolkit x multi-tty emacs)

Message #8 received at 8218 <at> debbugs.gnu.org (full text, mbox):

From: Dima Kogan <dkogan <at> cds.caltech.edu>
To: 8218 <at> debbugs.gnu.org
Subject: corrected attachment
Date: Thu, 10 Mar 2011 01:28:50 -0800

[Message part 1 (text/plain, inline)]

I didn't attach the sample file correctly. Here it is.

[tst3.pm (application/x-perl, attachment)]

Message #13 received at 8218-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Dima Kogan <dkogan <at> cds.caltech.edu>
Cc: 8218-done <at> debbugs.gnu.org
Subject: Re: bug#8218: 23.2; scan-sexps sees inconsistent syntax
Date: Thu, 10 Mar 2011 20:49:30 -0500

> I am seeing (scan-sexps) show inconsistent behavior depending on WHEN it
> is evaluated. If the attached sample file is opened, the point placed at
> the first '{', and (scan-sexps (point) 1) evaluated, the returned
> position is at times of the first '}' (wrong) and at other times of the
> second '}' (right). If the evaluation is done right when the file is
> opened, I observe the wrong behavior. This is reliably triggered with

> $ emacs -Q /tmp/tst3.pm --eval '(goto-char (scan-sexps 5 1))'

> Any subsequent evaluation of the same form in that buffer produces the
> correct behavior. If I turn off font-lock-mode, I consistently get
> incorrect behavior. This has wide ramifications with occasional
> incorrect indentation and hide-show operation, for instance.

This is not a bug in syntax.c: the syntax of Perl is too complex for the
simple syntax-table mechanism, so it needs to use the escape hatch,
i.e. the `syntax-table' property.  This property is added to particular
parts of the text via font-lock-syntactic-keywords (in Emacs-23) and via
syntax-propertize (in Emacs-24).  So in Emacs-23, you will only get the
right answer if font-lock is enabled and has processed the relevant
parts of the buffer.  In Emacs-24, it can work right even if font-lock
is disabled, but you still need to make sure the relevant parts of the
buffer have been processed.  For that, you should call
(syntax-propertize POS) where POS is the upper bound of the chunk of
text you're interested.


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.