From unknown Mon Jun 23 04:14:45 2025 X-Loop: help-debbugs@gnu.org Subject: bug#41522: 26.3; Narrowing can prevent syntax propertization Resent-From: Ikumi Keita Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 25 May 2020 08:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 41522 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: 41522@debbugs.gnu.org X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.159039614420471 (code B ref -1); Mon, 25 May 2020 08:43:02 +0000 Received: (at submit) by debbugs.gnu.org; 25 May 2020 08:42:24 +0000 Received: from localhost ([127.0.0.1]:39744 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jd8gh-0005K7-AW for submit@debbugs.gnu.org; Mon, 25 May 2020 04:42:24 -0400 Received: from lists.gnu.org ([209.51.188.17]:37086) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jd8gf-0005Jz-BQ for submit@debbugs.gnu.org; Mon, 25 May 2020 04:42:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42026) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jd8gf-0002VZ-4C for bug-gnu-emacs@gnu.org; Mon, 25 May 2020 04:42:21 -0400 Received: from smtp1a.inetd.co.jp ([210.129.88.11]:53398) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jd8gc-000610-7X for bug-gnu-emacs@gnu.org; Mon, 25 May 2020 04:42:20 -0400 Received: from localhost (42-146-134-146.rev.home.ne.jp [42.146.134.146]) by smtp1a.inetd.co.jp (Postfix) with ESMTPA id F3D9F2E for ; Mon, 25 May 2020 17:42:10 +0900 (JST) From: Ikumi Keita X-Mailer: MH-E 8.6+git; nmh 1.7.1; GNU Emacs 26.3 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Date: Mon, 25 May 2020 17:42:09 +0900 Message-ID: <14765.1590396129@localhost> Received-SPF: pass client-ip=210.129.88.11; envelope-from=ikumi@ikumi.que.jp; helo=smtp1a.inetd.co.jp X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/25 04:42:11 X-ACL-Warn: Detected OS = Linux 3.1-3.10 X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -1.3 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-=-= Content-Type: text/plain When syntax propertization occurs inside narrowed area of a buffer, the outside area sometimes doesn't get syntax propertization even after the narrowing is removed. [How to reproduce] 1. Save the attached sample file as /tmp/sample.sh 2. emacs -Q -geometry 80x25 /tmp/sample.sh 3. Type C-u C-n C-e to go to the end of the (visible) fourth line. 4. Type C-x C-e as the comment there instructs. 5. Then the face of here-doc bleeds over the "EOF" token and continues to the end of the buffer. Expected result is that the here-doc face ends at "EOF" and the rest of the buffer is fontified with various suitable faces. [Analysis] In the above example, `narrow-to-region' restricts the buffer into the last three lines of Local Variables section where syntax propertization is not yet done. There, `syntax-ppss' triggers syntax propertization and set `syntax-propertize--done' to (point-max). But actually that `syntax-ppss' didn't syntax-properize outside the narrowing so it didn't give the suitable `syntax-table' property to the newline following the "EOF". Since `syntax-propertize--done' tells emacs that all portion of the buffer has been propertized already, emacs does not give the suitable `syntax-table' property to the newline following the "EOF" even after `widen'. Thus the fontification of here-doc continues beyond "EOF". In a similar way, if a lisp function goes to the point where syntax propertization has not yet took place, makes narrowing and calls some syntax-aware function which sets `syntax-propertize--done' to a value inside the narrowing, a large portion of the buffer can be left without suitable `syntax-table' property. This can lead to incorrect result for not only font-lock but also syntax-aware functions such as `forward-comment'. Best regards, Ikumi Keita In GNU Emacs 26.3 (build 1, x86_64-unknown-freebsd12.0, GTK+ Version 3.24.9) of 2019-08-29 built on freebsd.vmware Windowing system distributor 'The X.Org Foundation', version 11.0.12008000 Recent messages: Indentation variables are now local. Indentation setup for shell type sh [2 times] Mark set Saving file /home/keita/scr/sample.sh... Wrote /home/keita/scr/sample.sh Setting up indent for shell type sh Indentation variables are now local. Indentation setup for shell type sh Configured using: 'configure --with-canna --with-canna-includes=/usr/local/canna/include --with-canna-libraries=/usr/local/canna/lib --without-xim --disable-largefile --with-sound=yes --with-file-notification=yes CFLAGS=-O3' Configured features: XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY ACL GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LCMS2 Important settings: value of $EMACSLOADPATH: /home/keita/elisp: value of $LANG: ja_JP.eucJP locale-coding-system: japanese-iso-8bit-unix Major mode: Info Minor modes in effect: TeX-PDF-mode: t global-magit-file-mode: t diff-auto-refine-mode: t magit-auto-revert-mode: t global-git-commit-mode: t async-bytecomp-package-mode: t shell-dirtrack-mode: t tooltip-mode: t global-eldoc-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t buffer-read-only: t line-number-mode: t transient-mark-mode: t Load-path shadows: ~/elisp/preview hides /usr/local/share/emacs/site-lisp/auctex/preview ~/elisp/reftex-parse hides /usr/local/share/emacs/26.3/lisp/textmodes/reftex-parse /home/keita/.emacs.d/elpa/soap-client-3.1.5/soap-client hides /usr/local/share/emacs/26.3/lisp/net/soap-client /home/keita/.emacs.d/elpa/soap-client-3.1.5/soap-inspect hides /usr/local/share/emacs/26.3/lisp/net/soap-inspect Features: (ibuf-ext ibuffer ibuffer-loaddefs shadow mh-mime mh-identity mh-tool-bar mh-letter mh-show goto-addr gnus-cite gnus-art mm-uu mml2015 gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-win gnus-range gnus nnheader wid-edit mh-xface mh-utils mh-comp mh-scan mh-gnus mm-view mml-smime smime dig mh-e mh-compat mh-buffers mh-loaddefs emacsbug sendmail sh-script smie executable perl-mode magit-extras vc-git pp preview prv-emacs reftex-dcr reftex-auc reftex reftex-loaddefs reftex-vars tex-bar tex-buf toolbar-x noutline outline font-latex latex latex-flymake flymake-proc flymake warnings tex-ispell tex-style tex dbus tex-mode compile latexenc misearch multi-isearch eieio-opt speedbar sb-image ezimage dframe find-func cl-print thingatpt help-fns radix-tree bug-reference edmacro kmacro magit-submodule magit-obsolete magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit magit-sequence magit-notes magit-worktree magit-tag magit-merge magit-branch magit-reset magit-files magit-refs magit-status magit magit-repos magit-apply magit-wip magit-log which-func imenu magit-diff smerge-mode diff-mode magit-core magit-autorevert autorevert filenotify magit-margin magit-transient magit-process magit-mode git-commit transient magit-git magit-section magit-utils crm log-edit easy-mmode message dired dired-loaddefs format-spec rfc822 mml mml-sec gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode mailabbrev mail-utils gmm-utils mailheader pcvs-util add-log with-editor async-bytecomp async shell pcomplete comint ansi-color ring server subr-x dash skeleton cl-extra help-mode parse-time ucs-normalize json map epa derived epg url-http tls gnutls url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr url-gw nsm rmc puny jka-compr twittering-mode advice url url-proxy url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util mailcap xml elec-pair preview-latex auto-loads tex-site canna-im finder-inf info package easymenu epg-config url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib time-date mule-util japan-util tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite charscript charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face macroexp files text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote threads dbusbind kqueue lcms2 dynamic-setting system-font-setting font-render-setting move-toolbar gtk x-toolkit x multi-tty make-network-process emacs) Memory information: ((conses 16 813972 71846) (symbols 48 48314 42) (miscs 40 4992 2861) (strings 32 186223 9232) (string-bytes 1 5882079) (vectors 16 83667) (vector-slots 8 2273246 58558) (floats 8 330 584) (intervals 56 12890 4833) (buffers 992 41)) --=-=-= Content-Type: text/x-shellscript Content-Disposition: attachment; filename=sample.sh Content-Description: sample to reproduce bug #!/bin/sh # Type C-x C-e at the end of the next line: # (progn (goto-char (point-max)) (narrow-to-region (point) (- (point) 50)) (syntax-ppss) (widen)) cat < Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 25 May 2020 15:05:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41522 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: Ikumi Keita , 41522@debbugs.gnu.org Received: via spool by 41522-submit@debbugs.gnu.org id=B41522.159041905225980 (code B ref 41522); Mon, 25 May 2020 15:05:01 +0000 Received: (at 41522) by debbugs.gnu.org; 25 May 2020 15:04:12 +0000 Received: from localhost ([127.0.0.1]:42131 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jdEeB-0006kx-QY for submit@debbugs.gnu.org; Mon, 25 May 2020 11:04:12 -0400 Received: from mail-wr1-f46.google.com ([209.85.221.46]:46775) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jdEe9-0006kj-LK for 41522@debbugs.gnu.org; Mon, 25 May 2020 11:04:10 -0400 Received: by mail-wr1-f46.google.com with SMTP id x6so3731916wrm.13 for <41522@debbugs.gnu.org>; Mon, 25 May 2020 08:04:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=ii+IK1F+5tT934qiJOV/CfQn7EggctbG5pplzFFkYHo=; b=gxZKBIm2o+sQM33bvIgAJHI0TYAYDKvk5yKiItzZE0y0n4ngfA0hbSw4oZeENaDne1 VCuTkW8nS2EJUuGEDt1bGo74jwsL1JRFJfVSaS7FnrGsQ8PO0oAG9W5wuRwO/3ff/TYS NFCwzodQfk2rmxkbyka3hD2SL8UUWb/gAhUVrYVOojl01ET+t1zq+0BTKgwvT/ENFnn6 HFEFVFzwiio/BQSOvqz5o+AVVW01AyS2akOte3oRwOfVGTM5A1pnfxzvH6juDDXeEQx/ iFaxEeTax2wxWnNKcVJAAcCpeivNXAYU6QEqH6vJskDmdjhGFNQH09CN695KIDL46JJd G2NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ii+IK1F+5tT934qiJOV/CfQn7EggctbG5pplzFFkYHo=; b=tpQ01lsk0cc57QymnSpTHVpWi7dc0uFF2QaggH3dXj/hBVOe4vKvUlYNQgRfxWgzDs nafh8JYzl9pQkzD60gOHCJa4f9xd2y38DJjrxmBY7ScaEFINo8g9V3vrGIpbEK9bn1n1 v17wVTeabCgrYbP8wSVPdcpEmYjg/RvTYaip0qKb9PAtumPqVp1f5407sXWqa90xGxY+ 41ZUCT3hv2uFw5vOwv7B7LVrkxA1+9z5SDKlOa0ekFA4vMjeRJM/9x1uh60xU7b8BT+g H8vPlxRBgUvFS/RcV70rLWCF73lOuDiLlp6FOFgI1uIjwt+Q20jbvZ+gwmlq3TLWP1v5 Nzbw== X-Gm-Message-State: AOAM532Z4A8qJy3STM0tdq9YR41akClRZjSmhpOWqKSsy1aUXPrez5Sl rSjSIzJNE+hMlwKduRLrAFEcHTb9 X-Google-Smtp-Source: ABdhPJx/y/fw+qW8yPjr9DiooAIVv5+fQZ1KhEeKMhLJRwC82KYwwmPRKOu4McFVX+zZn6hHTwXXqw== X-Received: by 2002:adf:ed51:: with SMTP id u17mr6282346wro.285.1590419043360; Mon, 25 May 2020 08:04:03 -0700 (PDT) Received: from [192.168.0.3] ([66.205.73.129]) by smtp.googlemail.com with ESMTPSA id m13sm17967387wmi.42.2020.05.25.08.04.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 25 May 2020 08:04:02 -0700 (PDT) References: <14765.1590396129@localhost> From: Dmitry Gutov Message-ID: <8f0d1e94-f918-4315-d5d5-ac1ee4df3477@yandex.ru> Date: Mon, 25 May 2020 18:04:01 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: <14765.1590396129@localhost> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Hi! On 25.05.2020 11:42, Ikumi Keita wrote: > In a similar way, if a lisp function goes to the point where syntax > propertization has not yet took place, makes narrowing and calls some > syntax-aware function which sets `syntax-propertize--done' to a value > inside the narrowing, a large portion of the buffer can be left without > suitable `syntax-table' property. This can lead to incorrect result > for not only font-lock but also syntax-aware functions such as > `forward-comment'. Our current stance on the matter is that, in the simple case, code should call syntax-ppss from "widened" state. And the code which doesn't should be fixed. Or if it really need to do that (for some reason), it should take care to handle any undesired sife-effects itself. See the definition of syntax-ppss-wide, which contains caches for both narrowed and "wide" syntax states. From unknown Mon Jun 23 04:14:45 2025 X-Loop: help-debbugs@gnu.org Subject: bug#41522: 26.3; Narrowing can prevent syntax propertization Resent-From: Ikumi Keita Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 25 May 2020 17:03:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41522 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: Dmitry Gutov Cc: 41522@debbugs.gnu.org Received: via spool by 41522-submit@debbugs.gnu.org id=B41522.159042614213150 (code B ref 41522); Mon, 25 May 2020 17:03:02 +0000 Received: (at 41522) by debbugs.gnu.org; 25 May 2020 17:02:22 +0000 Received: from localhost ([127.0.0.1]:42368 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jdGUY-0003Q2-0i for submit@debbugs.gnu.org; Mon, 25 May 2020 13:02:22 -0400 Received: from smtp1a.inetd.co.jp ([210.129.88.11]:58132) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jdGUV-0003Ps-Jk for 41522@debbugs.gnu.org; Mon, 25 May 2020 13:02:20 -0400 Received: from localhost (42-146-134-146.rev.home.ne.jp [42.146.134.146]) by smtp1a.inetd.co.jp (Postfix) with ESMTPA id 106192E; Tue, 26 May 2020 02:02:17 +0900 (JST) From: Ikumi Keita In-reply-to: <8f0d1e94-f918-4315-d5d5-ac1ee4df3477@yandex.ru> References: <14765.1590396129@localhost> <8f0d1e94-f918-4315-d5d5-ac1ee4df3477@yandex.ru> Comments: In-reply-to Dmitry Gutov message dated "Mon, 25 May 2020 18:04:01 +0300." X-Mailer: MH-E 8.6+git; nmh 1.7.1; GNU Emacs 26.3 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <9797.1590426135.1@localhost> Date: Tue, 26 May 2020 02:02:15 +0900 Message-ID: <9801.1590426135@localhost> X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Dmitry, thanks for your reply. >>>>> Dmitry Gutov writes: > Our current stance on the matter is that, in the simple case, code > should call syntax-ppss from "widened" state. I suppose that this means that the code should call syntax-ppss (or something like it) "before" narrowing is done, right? This code works for me ---------------------------------------------------------------------- (progn (goto-char (point-max)) (syntax-propertize (point)) ; <- (narrow-to-region (point) (- (point) 50)) (syntax-ppss) (widen)) ---------------------------------------------------------------------- while these don't: ---------------------------------------------------------------------- (progn (goto-char (point-max)) (narrow-to-region (point) (- (point) 50)) (syntax-ppss) (widen) (syntax-ppss) ; <- ) ---------------------------------------------------------------------- ---------------------------------------------------------------------- (progn (goto-char (point-max)) (narrow-to-region (point) (- (point) 50)) (syntax-ppss) (widen) (syntax-propertize (point)) ; <- ) ---------------------------------------------------------------------- > And the code which doesn't should be fixed. Or if it really need to do > that (for some reason), it should take care to handle any undesired > sife-effects itself. Hmm, then elisp programmers have to check whether each function used in the code uses narrowing+syntactic functions inside it if this kind of problem happens. I feel it a bit troublesome but at the same time I understand it is very difficult to solve this problem cleanly. OK, I try to be aware of this matter afterwards. Thanks for your advice! Regards, Ikumi Keita From unknown Mon Jun 23 04:14:45 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Ikumi Keita Subject: bug#41522: closed (Re: bug#41522: 26.3; Narrowing can prevent syntax propertization) Message-ID: References: <14765.1590396129@localhost> X-Gnu-PR-Message: they-closed 41522 X-Gnu-PR-Package: emacs Reply-To: 41522@debbugs.gnu.org Date: Mon, 25 May 2020 22:13:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1590444782-2152-1" This is a multi-part message in MIME format... ------------=_1590444782-2152-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #41522: 26.3; Narrowing can prevent syntax propertization which was filed against the emacs package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 41522@debbugs.gnu.org. --=20 41522: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D41522 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1590444782-2152-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 41522-done) by debbugs.gnu.org; 25 May 2020 22:12:59 +0000 Received: from localhost ([127.0.0.1]:42719 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jdLL9-0000YU-Hz for submit@debbugs.gnu.org; Mon, 25 May 2020 18:12:59 -0400 Received: from mail-wm1-f53.google.com ([209.85.128.53]:52824) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jdLL7-0000YH-NA for 41522-done@debbugs.gnu.org; Mon, 25 May 2020 18:12:58 -0400 Received: by mail-wm1-f53.google.com with SMTP id r9so1322133wmh.2 for <41522-done@debbugs.gnu.org>; Mon, 25 May 2020 15:12:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=HCU53NBGOZXPzqiNQk/6OP63JXjaL99lZoXitbNLFLo=; b=aSt8QYP+dlJQ9K+pBD2gUt1J7/2F/JIFPwWMrdWfRYDIdf7HWlLxCwZc3zmWoNgK4p A5Gt5ZJSMItVCDi9lBMYjXQqjAr3cE/Y9gnMkReU5ofZrCSXsijxlfeOWekcHT+stM4Q bUKLUiUCooIPji9KWHC7pCXYUvJeZ1iGComGanPe4N4xBn3jmyX1Dr4grNvksDP2yl0L nUJxAv7dWvYhp8/tseDGbNkZxcckx9gzZcFnt8DAlmXml8wa7HU8o4v6XrEgj0JgsAA9 wh2kBRSkcgMlI68WBMiFotTkxOF/0+gIStI4ZDaHiowPGgeEaVBrFEUfDh28e+oob12I MsUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:cc:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=HCU53NBGOZXPzqiNQk/6OP63JXjaL99lZoXitbNLFLo=; b=XS0pXd7/VbGskPdcMikpQQai5uyxHZRa3BvquyK8SYv81OPTeu9HN5kT+RPaNOtsE/ PZoxe1Hj+nCVGzWIEyIRPlmQThUhTQoMgeIPF5I1thRjUfPTWEO8nRfKguQ7w0bpJ/aj e/UHfxK/vyj+XctcsHVmW5NFExA66ldhdppbXm/cRsJHu5h8c1nLqovvEUdqqxdJvalH oEZ+lZm9mJbpB2PFPr6FZngqPMHQxcpdSWHnNC7JcqGz3RtedpqbwrKUOay5x0l/+knj Bnebzow/J2ZqGk8Qusc8IYjJZAEDFGlI43oVZ1OrimTLZxgcOwScOwNBMGXtuj+iYpdi 55+A== X-Gm-Message-State: AOAM532DxrKEB+fvpFyl3cXkE/sx1EX+s+m6VaYozFYfxlj815wR7uKB k6Yp8SvHjH470NE+iAfZxqUGFc9R X-Google-Smtp-Source: ABdhPJxmK9KDhH/8Vc/PQnKuZOnCmUD4Or91BPvuGpJKbERslc5Q2mzrOSVlB+50i6qL+20Chw0y7w== X-Received: by 2002:a1c:4302:: with SMTP id q2mr7622544wma.54.1590444771590; Mon, 25 May 2020 15:12:51 -0700 (PDT) Received: from [192.168.0.3] ([66.205.73.129]) by smtp.googlemail.com with ESMTPSA id p7sm14476103wro.26.2020.05.25.15.12.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 25 May 2020 15:12:51 -0700 (PDT) Subject: Re: bug#41522: 26.3; Narrowing can prevent syntax propertization To: Ikumi Keita References: <14765.1590396129@localhost> <8f0d1e94-f918-4315-d5d5-ac1ee4df3477@yandex.ru> <9801.1590426135@localhost> From: Dmitry Gutov Message-ID: Date: Tue, 26 May 2020 01:12:49 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: <9801.1590426135@localhost> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41522-done Cc: 41522-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) On 25.05.2020 20:02, Ikumi Keita wrote: > I suppose that this means that the code should call syntax-ppss (or > something like it) "before" narrowing is done, right? Or call (syntax-ppss-flush-cache (point-min)) afterward. > Hmm, then elisp programmers have to check whether each function used in the code uses narrowing+syntactic functions inside it if this kind of problem happens. I feel it a bit troublesome but at the same time I understand it is very difficult to solve this problem cleanly. One has to be careful with narrowing. It's an old problem. > OK, I try to be aware of this matter afterwards. Thanks for your advice! Welcome! ------------=_1590444782-2152-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 25 May 2020 08:42:24 +0000 Received: from localhost ([127.0.0.1]:39744 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jd8gh-0005K7-AW for submit@debbugs.gnu.org; Mon, 25 May 2020 04:42:24 -0400 Received: from lists.gnu.org ([209.51.188.17]:37086) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jd8gf-0005Jz-BQ for submit@debbugs.gnu.org; Mon, 25 May 2020 04:42:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42026) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jd8gf-0002VZ-4C for bug-gnu-emacs@gnu.org; Mon, 25 May 2020 04:42:21 -0400 Received: from smtp1a.inetd.co.jp ([210.129.88.11]:53398) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jd8gc-000610-7X for bug-gnu-emacs@gnu.org; Mon, 25 May 2020 04:42:20 -0400 Received: from localhost (42-146-134-146.rev.home.ne.jp [42.146.134.146]) by smtp1a.inetd.co.jp (Postfix) with ESMTPA id F3D9F2E for ; Mon, 25 May 2020 17:42:10 +0900 (JST) From: Ikumi Keita To: bug-gnu-emacs@gnu.org Subject: 26.3; Narrowing can prevent syntax propertization X-Mailer: MH-E 8.6+git; nmh 1.7.1; GNU Emacs 26.3 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Date: Mon, 25 May 2020 17:42:09 +0900 Message-ID: <14765.1590396129@localhost> Received-SPF: pass client-ip=210.129.88.11; envelope-from=ikumi@ikumi.que.jp; helo=smtp1a.inetd.co.jp X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/25 04:42:11 X-ACL-Warn: Detected OS = Linux 3.1-3.10 X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-=-= Content-Type: text/plain When syntax propertization occurs inside narrowed area of a buffer, the outside area sometimes doesn't get syntax propertization even after the narrowing is removed. [How to reproduce] 1. Save the attached sample file as /tmp/sample.sh 2. emacs -Q -geometry 80x25 /tmp/sample.sh 3. Type C-u C-n C-e to go to the end of the (visible) fourth line. 4. Type C-x C-e as the comment there instructs. 5. Then the face of here-doc bleeds over the "EOF" token and continues to the end of the buffer. Expected result is that the here-doc face ends at "EOF" and the rest of the buffer is fontified with various suitable faces. [Analysis] In the above example, `narrow-to-region' restricts the buffer into the last three lines of Local Variables section where syntax propertization is not yet done. There, `syntax-ppss' triggers syntax propertization and set `syntax-propertize--done' to (point-max). But actually that `syntax-ppss' didn't syntax-properize outside the narrowing so it didn't give the suitable `syntax-table' property to the newline following the "EOF". Since `syntax-propertize--done' tells emacs that all portion of the buffer has been propertized already, emacs does not give the suitable `syntax-table' property to the newline following the "EOF" even after `widen'. Thus the fontification of here-doc continues beyond "EOF". In a similar way, if a lisp function goes to the point where syntax propertization has not yet took place, makes narrowing and calls some syntax-aware function which sets `syntax-propertize--done' to a value inside the narrowing, a large portion of the buffer can be left without suitable `syntax-table' property. This can lead to incorrect result for not only font-lock but also syntax-aware functions such as `forward-comment'. Best regards, Ikumi Keita In GNU Emacs 26.3 (build 1, x86_64-unknown-freebsd12.0, GTK+ Version 3.24.9) of 2019-08-29 built on freebsd.vmware Windowing system distributor 'The X.Org Foundation', version 11.0.12008000 Recent messages: Indentation variables are now local. Indentation setup for shell type sh [2 times] Mark set Saving file /home/keita/scr/sample.sh... Wrote /home/keita/scr/sample.sh Setting up indent for shell type sh Indentation variables are now local. Indentation setup for shell type sh Configured using: 'configure --with-canna --with-canna-includes=/usr/local/canna/include --with-canna-libraries=/usr/local/canna/lib --without-xim --disable-largefile --with-sound=yes --with-file-notification=yes CFLAGS=-O3' Configured features: XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY ACL GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LCMS2 Important settings: value of $EMACSLOADPATH: /home/keita/elisp: value of $LANG: ja_JP.eucJP locale-coding-system: japanese-iso-8bit-unix Major mode: Info Minor modes in effect: TeX-PDF-mode: t global-magit-file-mode: t diff-auto-refine-mode: t magit-auto-revert-mode: t global-git-commit-mode: t async-bytecomp-package-mode: t shell-dirtrack-mode: t tooltip-mode: t global-eldoc-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t buffer-read-only: t line-number-mode: t transient-mark-mode: t Load-path shadows: ~/elisp/preview hides /usr/local/share/emacs/site-lisp/auctex/preview ~/elisp/reftex-parse hides /usr/local/share/emacs/26.3/lisp/textmodes/reftex-parse /home/keita/.emacs.d/elpa/soap-client-3.1.5/soap-client hides /usr/local/share/emacs/26.3/lisp/net/soap-client /home/keita/.emacs.d/elpa/soap-client-3.1.5/soap-inspect hides /usr/local/share/emacs/26.3/lisp/net/soap-inspect Features: (ibuf-ext ibuffer ibuffer-loaddefs shadow mh-mime mh-identity mh-tool-bar mh-letter mh-show goto-addr gnus-cite gnus-art mm-uu mml2015 gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-win gnus-range gnus nnheader wid-edit mh-xface mh-utils mh-comp mh-scan mh-gnus mm-view mml-smime smime dig mh-e mh-compat mh-buffers mh-loaddefs emacsbug sendmail sh-script smie executable perl-mode magit-extras vc-git pp preview prv-emacs reftex-dcr reftex-auc reftex reftex-loaddefs reftex-vars tex-bar tex-buf toolbar-x noutline outline font-latex latex latex-flymake flymake-proc flymake warnings tex-ispell tex-style tex dbus tex-mode compile latexenc misearch multi-isearch eieio-opt speedbar sb-image ezimage dframe find-func cl-print thingatpt help-fns radix-tree bug-reference edmacro kmacro magit-submodule magit-obsolete magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit magit-sequence magit-notes magit-worktree magit-tag magit-merge magit-branch magit-reset magit-files magit-refs magit-status magit magit-repos magit-apply magit-wip magit-log which-func imenu magit-diff smerge-mode diff-mode magit-core magit-autorevert autorevert filenotify magit-margin magit-transient magit-process magit-mode git-commit transient magit-git magit-section magit-utils crm log-edit easy-mmode message dired dired-loaddefs format-spec rfc822 mml mml-sec gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode mailabbrev mail-utils gmm-utils mailheader pcvs-util add-log with-editor async-bytecomp async shell pcomplete comint ansi-color ring server subr-x dash skeleton cl-extra help-mode parse-time ucs-normalize json map epa derived epg url-http tls gnutls url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr url-gw nsm rmc puny jka-compr twittering-mode advice url url-proxy url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util mailcap xml elec-pair preview-latex auto-loads tex-site canna-im finder-inf info package easymenu epg-config url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib time-date mule-util japan-util tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite charscript charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face macroexp files text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote threads dbusbind kqueue lcms2 dynamic-setting system-font-setting font-render-setting move-toolbar gtk x-toolkit x multi-tty make-network-process emacs) Memory information: ((conses 16 813972 71846) (symbols 48 48314 42) (miscs 40 4992 2861) (strings 32 186223 9232) (string-bytes 1 5882079) (vectors 16 83667) (vector-slots 8 2273246 58558) (floats 8 330 584) (intervals 56 12890 4833) (buffers 992 41)) --=-=-= Content-Type: text/x-shellscript Content-Disposition: attachment; filename=sample.sh Content-Description: sample to reproduce bug #!/bin/sh # Type C-x C-e at the end of the next line: # (progn (goto-char (point-max)) (narrow-to-region (point) (- (point) 50)) (syntax-ppss) (widen)) cat <