Subject: bug#41522: closed (Re: bug#41522: 26.3; Narrowing can prevent syntax propertization)

From: [email-masked] (GNU bug Tracking System)
Date: Mon, 25 May 2020 22:13:02 +0000

Your bug report

#41522: 26.3; Narrowing can prevent syntax propertization

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
41522: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=41522
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#41522: 26.3; Narrowing can prevent syntax propertization
From: Dmitry Gutov <[email-masked]>
To: Ikumi Keita <[email-masked]>
Cc: [email-masked]
Date: Tue, 26 May 2020 01:12:49 +0300
List-Id: <debbugs-submit.debbugs.gnu.org>

On 25.05.2020 20:02, Ikumi Keita wrote:
> I suppose that this means that the code should call syntax-ppss (or
> something like it) "before" narrowing is done, right?

Or call (syntax-ppss-flush-cache (point-min)) afterward.

 > Hmm, then elisp programmers have to check whether each function used 
in the code uses narrowing+syntactic functions inside it if this kind of
problem happens. I feel it a bit troublesome but at the same time I 
understand it is very difficult to solve this problem cleanly.

One has to be careful with narrowing. It's an old problem.

 > OK, I try to be aware of this matter afterwards. Thanks for your
advice!

Welcome!

Subject: 26.3; Narrowing can prevent syntax propertization
From: Ikumi Keita <[email-masked]>
To: [email-masked]
Date: Mon, 25 May 2020 17:42:09 +0900
List-Id: <debbugs-submit.debbugs.gnu.org>

When syntax propertization occurs inside narrowed area of a buffer,
the outside area sometimes doesn't get syntax propertization even after
the narrowing is removed.

[How to reproduce]
1. Save the attached sample file as /tmp/sample.sh
2. emacs -Q -geometry 80x25 /tmp/sample.sh
3. Type C-u C-n C-e to go to the end of the (visible) fourth line.
4. Type C-x C-e as the comment there instructs.
5. Then the face of here-doc bleeds over the "EOF" token and continues
   to the end of the buffer.
   Expected result is that the here-doc face ends at "EOF" and the rest
   of the buffer is fontified with various suitable faces.

[Analysis]
In the above example, `narrow-to-region' restricts the buffer into the
last three lines of Local Variables section where syntax propertization
is not yet done. There, `syntax-ppss' triggers syntax propertization and
set `syntax-propertize--done' to (point-max). But actually that
`syntax-ppss' didn't syntax-properize outside the narrowing so it didn't
give the suitable `syntax-table' property to the newline following the
"EOF". Since `syntax-propertize--done' tells emacs that all portion of
the buffer has been propertized already, emacs does not give the
suitable `syntax-table' property to the newline following the "EOF" even
after `widen'. Thus the fontification of here-doc continues beyond "EOF".

In a similar way, if a lisp function goes to the point where syntax
propertization has not yet took place, makes narrowing and calls some
syntax-aware function which sets `syntax-propertize--done' to a value
inside the narrowing, a large portion of the buffer can be left without
suitable `syntax-table' property. This can lead to incorrect result
for not only font-lock but also syntax-aware functions such as
`forward-comment'.

Best regards,
Ikumi Keita

In GNU Emacs 26.3 (build 1, x86_64-unknown-freebsd12.0, GTK+ Version 3.24.9)
 of 2019-08-29 built on freebsd.vmware
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
Recent messages:
Indentation variables are now local.
Indentation setup for shell type sh
 [2 times]
Mark set
Saving file /home/keita/scr/sample.sh...
Wrote /home/keita/scr/sample.sh
Setting up indent for shell type sh
Indentation variables are now local.
Indentation setup for shell type sh

Configured using:
 'configure --with-canna --with-canna-includes=/usr/local/canna/include
--with-canna-libraries=/usr/local/canna/lib --without-xim
--disable-largefile --with-sound=yes --with-file-notification=yes
CFLAGS=-O3'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY ACL GNUTLS
LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS
LCMS2

Important settings:
  value of $EMACSLOADPATH: /home/keita/elisp:
  value of $LANG: ja_JP.eucJP
  locale-coding-system: japanese-iso-8bit-unix

Major mode: Info

Minor modes in effect:
  TeX-PDF-mode: t
  global-magit-file-mode: t
  diff-auto-refine-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
~/elisp/preview hides /usr/local/share/emacs/site-lisp/auctex/preview
~/elisp/reftex-parse hides /usr/local/share/emacs/26.3/lisp/textmodes/reftex-parse
/home/keita/.emacs.d/elpa/soap-client-3.1.5/soap-client hides /usr/local/share/emacs/26.3/lisp/net/soap-client
/home/keita/.emacs.d/elpa/soap-client-3.1.5/soap-inspect hides /usr/local/share/emacs/26.3/lisp/net/soap-inspect

Features:
(ibuf-ext ibuffer ibuffer-loaddefs shadow mh-mime mh-identity
mh-tool-bar mh-letter mh-show goto-addr gnus-cite gnus-art mm-uu mml2015
gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-win gnus-range gnus
nnheader wid-edit mh-xface mh-utils mh-comp mh-scan mh-gnus mm-view
mml-smime smime dig mh-e mh-compat mh-buffers mh-loaddefs emacsbug
sendmail sh-script smie executable perl-mode magit-extras vc-git pp
preview prv-emacs reftex-dcr reftex-auc reftex reftex-loaddefs
reftex-vars tex-bar tex-buf toolbar-x noutline outline font-latex latex
latex-flymake flymake-proc flymake warnings tex-ispell tex-style tex
dbus tex-mode compile latexenc misearch multi-isearch eieio-opt speedbar
sb-image ezimage dframe find-func cl-print thingatpt help-fns radix-tree
bug-reference edmacro kmacro magit-submodule magit-obsolete magit-blame
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch
magit-clone magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit magit-repos magit-apply
magit-wip magit-log which-func imenu magit-diff smerge-mode diff-mode
magit-core magit-autorevert autorevert filenotify magit-margin
magit-transient magit-process magit-mode git-commit transient magit-git
magit-section magit-utils crm log-edit easy-mmode message dired
dired-loaddefs format-spec rfc822 mml mml-sec gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mailabbrev mail-utils
gmm-utils mailheader pcvs-util add-log with-editor async-bytecomp async
shell pcomplete comint ansi-color ring server subr-x dash skeleton
cl-extra help-mode parse-time ucs-normalize json map epa derived epg
url-http tls gnutls url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr url-gw nsm rmc puny jka-compr twittering-mode
advice url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util mailcap xml elec-pair preview-latex
auto-loads tex-site canna-im finder-inf info package easymenu epg-config
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib time-date mule-util japan-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind kqueue lcms2 dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 813972 71846)
 (symbols 48 48314 42)
 (miscs 40 4992 2861)
 (strings 32 186223 9232)
 (string-bytes 1 5882079)
 (vectors 16 83667)
 (vector-slots 8 2273246 58558)
 (floats 8 330 584)
 (intervals 56 12890 4833)
 (buffers 992 41))