From unknown Fri Aug 15 12:51:18 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#68445 <68445@debbugs.gnu.org> To: bug#68445 <68445@debbugs.gnu.org> Subject: Status: [PATCH] Problem with python--treesit-syntax-propertize Reply-To: bug#68445 <68445@debbugs.gnu.org> Date: Fri, 15 Aug 2025 19:51:18 +0000 retitle 68445 [PATCH] Problem with python--treesit-syntax-propertize reassign 68445 emacs submitter 68445 kobarity severity 68445 normal tag 68445 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 14 04:15:37 2024 Received: (at submit) by debbugs.gnu.org; 14 Jan 2024 09:15:37 +0000 Received: from localhost ([127.0.0.1]:41676 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOwaf-00066O-1m for submit@debbugs.gnu.org; Sun, 14 Jan 2024 04:15:37 -0500 Received: from lists.gnu.org ([2001:470:142::17]:55436) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOwad-0005vL-Eo for submit@debbugs.gnu.org; Sun, 14 Jan 2024 04:15:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOwaY-0004Ds-Ai for bug-gnu-emacs@gnu.org; Sun, 14 Jan 2024 04:15:30 -0500 Received: from mail-oa1-x2a.google.com ([2001:4860:4864:20::2a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rOwaV-0004Q9-GE for bug-gnu-emacs@gnu.org; Sun, 14 Jan 2024 04:15:29 -0500 Received: by mail-oa1-x2a.google.com with SMTP id 586e51a60fabf-20757468411so504355fac.3 for ; Sun, 14 Jan 2024 01:15:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705223724; x=1705828524; darn=gnu.org; h=mime-version:user-agent:subject:to:from:message-id:date:from:to:cc :subject:date:message-id:reply-to; bh=xuf4PC/fCqM60hEnBJW9N+cNQynQprwxZRR4HNijxkY=; b=G3tAhoYcBApvEcsvA+13dfGWEzIhAIOeFbfq5a/siR9zy7pjMabT1wWBbo90QKEwlx telqZ5YMGvQc4EZ0PVSI9L8L9ug2/9I7R78jhz3AFeEFMMNR7v9l2notokY0O9Mkbq2s T4b89LMdFRwM3Dqfx3ZlyjAw9/ICBindzM+Sa8byGsEpFchwzmQxHGc1xxICOvgrUiPk hlZ9+jWDcpmYoOfJWesSWYsEJqEDNZPmFYhEIQ/PODu7nBTiHjQJaUVrvCyjFGy3PXqQ klh3dE/Xl1FOUJoYnh/e5sDw9UXwwhONZywxtt4N0x9PvnJ7Ab6dPVkNC2aPfTIbNz30 nFuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705223724; x=1705828524; h=mime-version:user-agent:subject:to:from:message-id:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xuf4PC/fCqM60hEnBJW9N+cNQynQprwxZRR4HNijxkY=; b=esJuYubOtNodZng43fhDbV7ygX0HvEefXohGUuZzLCP/OEJZuOyUecIzU05N/WC7a3 WKbsvZ7uL4TCJIgIFkgGGk26x3lkclva1n3hgCc6kkO2pxhmMN15KPKJBBVaLTi46hat BylBlPBOl72djhULh0+/0lJZlx8ccEqT4uy/0p1VAsYJYQr25cjHjzvdu9TWa9zEF+Wv C64Xoe70jVdDiS99EMERwLdwy8+0lqVmG0Z0pm5Z0yPejB7PuPLr58Ov8vzI5ApsuoyM I+GiAh3yuoyZQSAnIBh1VusgjiyxOeqd0fG5qSujeCh/0Wn64K+dGHEHOX84VEPT1/+4 rKBw== X-Gm-Message-State: AOJu0YwM9bnYD47LL4rGmVS1od01uiP8eZ3CVoZ1F+bwTEKnG1HTyhFD 67v3U2e6rzgVLkgPKooAsYGxG3N7sKE= X-Google-Smtp-Source: AGHT+IFivoo2fR+zuf2pAmKG6JTQkWddSi5lmJFhclDx6C9lqqWyFOMMLfFZQ4zfvOlz8mMgfXnw3Q== X-Received: by 2002:a05:6871:d08b:b0:204:32a:7b96 with SMTP id mw11-20020a056871d08b00b00204032a7b96mr5786335oac.95.1705223724406; Sun, 14 Jan 2024 01:15:24 -0800 (PST) Received: from localhost (58x12x133x161.ap58.ftth.ucom.ne.jp. [58.12.133.161]) by smtp.gmail.com with ESMTPSA id w7-20020a636207000000b005cdc081bd2asm6182060pgb.24.2024.01.14.01.15.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Jan 2024 01:15:23 -0800 (PST) Date: Sun, 14 Jan 2024 18:15:07 +0900 Message-ID: From: kobarity To: bug-gnu-emacs@gnu.org Subject: [PATCH] Problem with python--treesit-syntax-propertize X-Debbugs-Cc: Yuan Fu , Dmitry Gutov User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.0.50 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: multipart/mixed; boundary="Multipart_Sun_Jan_14_18:15:07_2024-1" Received-SPF: pass client-ip=2001:4860:4864:20::2a; envelope-from=kobarity@gmail.com; helo=mail-oa1-x2a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, WEIRD_QUOTING=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) --Multipart_Sun_Jan_14_18:15:07_2024-1 Content-Type: text/plain; charset=US-ASCII Hi, I found a problem with python--treesit-syntax-propertize recently introduced by the Bug#67977 patch. 1. emacs -Q 2. Open a file in python-ts-mode with the following contents: #+begin_src python """Docstring. test. """ S = """string.""" #+end_src 3. Locate the point on the third line. 4. M-q 5. An empty line will be inserted. 6. M-q 7. The string literal on the last line will be split as follows: S = "" "string.""" This problem does not occur in python-mode. The direct cause of this problem is that the string-delimiter property set in the docstring is removed. python--treesit-syntax-propertize is called to set the property, but it fails to set it properly. Here is the trace of python--treesit-syntax-propertize from step 4 above. ====================================================================== 1 -> (python--treesit-syntax-propertize 1 45) 1 <- python--treesit-syntax-propertize: nil ====================================================================== 1 -> (python--treesit-syntax-propertize 16 45) 1 <- python--treesit-syntax-propertize: nil python--treesit-syntax-propertize is called with argument START 16. This is the position inside the docstring. It seems to me that python--treesit-syntax-propertize assumes that the START argument is outside the triple-quoted string. So one solution might be to change START to the start of the string if it is within a string, as in the attached patch. However, I'm not sure this is the right approach. Should we use syntax-propertize-extend-region-functions? -- In GNU Emacs 30.0.50 (build 5, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw scroll bars) of 2024-01-13 built on ubuntu Repository revision: 106cd9aafe8248ef91d7e89161adc5f912ea54eb Repository branch: master System Description: Ubuntu 22.04.3 LTS --Multipart_Sun_Jan_14_18:15:07_2024-1 Content-Type: text/plain; type=patch; name="0001-Fix-python-treesit-syntax-propertize.patch"; charset=US-ASCII Content-Disposition: attachment; filename="0001-Fix-python-treesit-syntax-propertize.patch" Content-Transfer-Encoding: 7bit >From 809c506504913b9e6af9b7de13dd5892cf91fec3 Mon Sep 17 00:00:00 2001 From: kobarity Date: Sun, 14 Jan 2024 17:50:16 +0900 Subject: [PATCH] Fix python--treesit-syntax-propertize * lisp/progmodes/python.el (python--treesit-syntax-propertize): If START is in a string, move start to the beginning of the string. --- lisp/progmodes/python.el | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index e2f614f52c2..1430fd84141 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -1358,6 +1358,9 @@ python--treesit-syntax-propertize "Propertize triple-quote strings between START and END." (save-excursion (goto-char start) + (let ((node (treesit-node-at (point)))) + (when (equal (treesit-node-type node) "string_content") + (goto-char (treesit-node-start (treesit-node-parent node))))) (while (re-search-forward (rx (or "\"\"\"" "'''")) end t) (let ((node (treesit-node-at (point)))) ;; The triple quotes surround a non-empty string. -- 2.34.1 --Multipart_Sun_Jan_14_18:15:07_2024-1-- From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 21 09:47:51 2024 Received: (at 68445) by debbugs.gnu.org; 21 Jan 2024 14:47:51 +0000 Received: from localhost ([127.0.0.1]:36869 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRZ70-0003hJ-LI for submit@debbugs.gnu.org; Sun, 21 Jan 2024 09:47:50 -0500 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]:56501) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRZ6w-0003gR-H5 for 68445@debbugs.gnu.org; Sun, 21 Jan 2024 09:47:50 -0500 Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-6dbd146c76cso52818b3a.0 for <68445@debbugs.gnu.org>; Sun, 21 Jan 2024 06:47:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705848457; x=1706453257; darn=debbugs.gnu.org; h=mime-version:user-agent:references:in-reply-to:subject:cc:to:from :message-id:date:from:to:cc:subject:date:message-id:reply-to; bh=zfNT3bPr7baTScT0CFRAPq983vSt6Yqe5SXiRBtv/UU=; b=ZR+4laNRl/+KjRtzU6HK6ATK9LSNR2O1JkV1WEiJ9DwglRXX/Sn59dYBnTS3CybSDi jRRoV8vev99skY2IplNexIM0a2MpoLjTNLHVXRN9XKCRYOn2lxeZ6uHmgCfQpYPCfqou FWoam6Vtn/+sCL4H4y02GmSxJnTuodcuUXOviVNgXLwv/VHNfuV1z4CLqZbHTj27diWD JUsS6FtfCIpmWfk6ZJRM7lp+AvvQmAzFCLqDcIvYMWs15AtVtrMnV5IEC3skEG8foD2l w0fuUY5+V3F95bRoAwOPXfCfMjSDigNW2OKNfRwTLiIPEcsYiS7EwPtxQ4ew+DF4eyeU 4kLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705848457; x=1706453257; h=mime-version:user-agent:references:in-reply-to:subject:cc:to:from :message-id:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=zfNT3bPr7baTScT0CFRAPq983vSt6Yqe5SXiRBtv/UU=; b=JFSw0yiK/d54ZLqI/IX1fheEZjCwiRgEYcRTxXcsfle0jxpwD4kiaSa/mxPPSzc2LN XHxKRlfhYVNQP7nPTWUoQTJCnYA0vZjd57WrhkHnIcWWQbbsfU1N2tbO43ucE6H1pQO4 OT/KH+GJ+nqEPCjBPWd1xwqsT4BqX+T4PCJxtOcxJD2PHst0NGxOp5Vp/z3GupvSFfbC sNhT0aX0g326cC6ZqB9qUz/rezKkC9P6L1x0cpVtNdenR9UVow5DITUI51vr8+hke2Ay RNU5Y8DLAslbA3DA0+KeVZGLiedcefk0G4n/NcaCysUdWnRIHK6xLRszLUKUYcwRXkuv zMgA== X-Gm-Message-State: AOJu0Yy//RBWwtOQxUXiIVUEAGWxy466xOlYvlAnwU9RY53k4FlYeIUU th+1WLBaPZzMlh9KONHz5TdsCHq1Foozb2dMkLIIdpJpIPqS63lA X-Google-Smtp-Source: AGHT+IHgX/ymikiRe0bHr4ZNMX5TtmV3Olp61bj+41a/9YKkvCadXXAdJj1/FzaVz/4Fk8DSNHDd7w== X-Received: by 2002:a05:6a20:144a:b0:19a:2bda:7273 with SMTP id a10-20020a056a20144a00b0019a2bda7273mr962766pzi.84.1705848456909; Sun, 21 Jan 2024 06:47:36 -0800 (PST) Received: from localhost (58x12x133x161.ap58.ftth.ucom.ne.jp. [58.12.133.161]) by smtp.gmail.com with ESMTPSA id az2-20020a170902a58200b001d73adf350asm1616251plb.56.2024.01.21.06.47.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Jan 2024 06:47:36 -0800 (PST) Date: Sun, 21 Jan 2024 23:47:39 +0900 Message-ID: From: kobarity To: Yuan Fu , Dmitry Gutov Subject: Re: [PATCH] Problem with python--treesit-syntax-propertize In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.0.50 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 68445 Cc: 68445@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) I am resending my mail, as I made a mistake in X-Debbugs-CC. I wrote: > Hi, > > I found a problem with python--treesit-syntax-propertize recently > introduced by the Bug#67977 patch. > > 1. emacs -Q > 2. Open a file in python-ts-mode with the following contents: > > #+begin_src python > """Docstring. > > test. > """ > S = """string.""" > #+end_src > > 3. Locate the point on the third line. > 4. M-q > 5. An empty line will be inserted. > 6. M-q > 7. The string literal on the last line will be split as follows: > > S = "" > > "string.""" > > This problem does not occur in python-mode. > > The direct cause of this problem is that the string-delimiter property > set in the docstring is removed. python--treesit-syntax-propertize is > called to set the property, but it fails to set it properly. Here is > the trace of python--treesit-syntax-propertize from step 4 above. > > ====================================================================== > 1 -> (python--treesit-syntax-propertize 1 45) > 1 <- python--treesit-syntax-propertize: nil > ====================================================================== > 1 -> (python--treesit-syntax-propertize 16 45) > 1 <- python--treesit-syntax-propertize: nil > > python--treesit-syntax-propertize is called with argument START 16. > This is the position inside the docstring. > > It seems to me that python--treesit-syntax-propertize assumes that the > START argument is outside the triple-quoted string. So one solution > might be to change START to the start of the string if it is within a > string, as in the attached patch. However, I'm not sure this is the > right approach. Should we use > syntax-propertize-extend-region-functions? > > -- > In GNU Emacs 30.0.50 (build 5, x86_64-pc-linux-gnu, X toolkit, cairo > version 1.16.0, Xaw scroll bars) of 2024-01-13 built on ubuntu > Repository revision: 106cd9aafe8248ef91d7e89161adc5f912ea54eb > Repository branch: master > System Description: Ubuntu 22.04.3 LTS I appreciate your comments. From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 21 13:09:18 2024 Received: (at 68445) by debbugs.gnu.org; 21 Jan 2024 18:09:18 +0000 Received: from localhost ([127.0.0.1]:38969 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRcFy-0001Ov-8P for submit@debbugs.gnu.org; Sun, 21 Jan 2024 13:09:18 -0500 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:38165) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRcFv-0001Od-BR for 68445@debbugs.gnu.org; Sun, 21 Jan 2024 13:09:17 -0500 Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 19DEE5C00CB; Sun, 21 Jan 2024 13:09:06 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Sun, 21 Jan 2024 13:09:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm1; t=1705860546; x=1705946946; bh=0ZzQVz6m7LjBx6uSmTqUfhae0tUB9c/AutImxxDi0xY=; b= MIqZBK5FOpfpNFlsW5EliEBE/pNcz7YfK2s0kzo5OZ/4fBOQi6yP5HFc4rUZoycC 7P4BPno+Dfba+wpNG8QM95BPEnvmGBEbtasLQrxCQEANqAxW2heLJPWTSb9IpZQ2 VfS5yN+Vc1FOvs6JRISn7BmQ9LZonVameB6LkuqwW2wbFX/SQB/xubANzkyw4G6S 3jyOCkaiPVKyAUB3iZGCMNZKqirvQFW1odWyqM97vV/bA3h59mfEWv9Q5+Me+eDd VpL4Z4BvslfxMea1pjZ0VvidUD+EhVnHYeX0Z3raYAcUbTIy8FFMqhRsnu2Gpkcr UwzMkXrGBx2Pd9xbw+b3Nw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1705860546; x= 1705946946; bh=0ZzQVz6m7LjBx6uSmTqUfhae0tUB9c/AutImxxDi0xY=; b=C WWCTcKlL+moMI/wOX0Yo1R7RHGxW9PUwOmDLcymRpos8tJJL7qpmf8caG0CyVYzr FH/ibV8WDFkoh4MCAijXc4NKyu/65ZBK/DP6cQPDhP+tNVQnXKMAScye5YicOCR3 8pwTZ2HllgcpXl3mCkS9tnhF3RGsh+txwWDCAVdzP3NDlXcsLLsuOnSi4PPCgrPG J94eJflyK/4wh/FpQFcBRwO6eEOimP6auOJCXYcCyNcMLOzIjnSP5ueXDd7bMP3D YG5zzYMl/ihOH6OI9HAOfi32Fyo1nCHnix5m3VhnnSA8CdjVap4nLKJ4j1bXvGdh 8NR21QcReLDBru/kZOmWQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdekgedguddtkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepkfffgggfuffvvehfhfgjtgfgse htjeertddtvdejnecuhfhrohhmpeffmhhithhrhicuifhuthhovhcuoegumhhithhrhies ghhuthhovhdruggvvheqnecuggftrfgrthhtvghrnhepteduleejgeehtefgheegjeekue ehvdevieekueeftddvtdevfefhvdevgedujeehnecuvehluhhsthgvrhfuihiivgeptden ucfrrghrrghmpehmrghilhhfrhhomhepughmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 21 Jan 2024 13:09:04 -0500 (EST) Message-ID: <5135ab69-3704-459f-b6c7-b8ea738e8d31@gutov.dev> Date: Sun, 21 Jan 2024 20:09:03 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] Problem with python--treesit-syntax-propertize Content-Language: en-US To: kobarity , Yuan Fu References: From: Dmitry Gutov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 68445 Cc: 68445@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi! On 21/01/2024 16:47, kobarity wrote: > > I am resending my mail, as I made a mistake in X-Debbugs-CC. Was it supposed to appear in the bug's thread? I don't see it anywhere. > I wrote: >> Hi, >> >> I found a problem with python--treesit-syntax-propertize recently >> introduced by the Bug#67977 patch. >> >> 1. emacs -Q >> 2. Open a file in python-ts-mode with the following contents: >> >> #+begin_src python >> """Docstring. >> >> test. >> """ >> S = """string.""" >> #+end_src >> >> 3. Locate the point on the third line. >> 4. M-q >> 5. An empty line will be inserted. >> 6. M-q >> 7. The string literal on the last line will be split as follows: >> >> S = "" >> >> "string.""" >> >> This problem does not occur in python-mode. >> >> The direct cause of this problem is that the string-delimiter property >> set in the docstring is removed. python--treesit-syntax-propertize is >> called to set the property, but it fails to set it properly. Here is >> the trace of python--treesit-syntax-propertize from step 4 above. >> >> ====================================================================== >> 1 -> (python--treesit-syntax-propertize 1 45) >> 1 <- python--treesit-syntax-propertize: nil >> ====================================================================== >> 1 -> (python--treesit-syntax-propertize 16 45) >> 1 <- python--treesit-syntax-propertize: nil >> >> python--treesit-syntax-propertize is called with argument START 16. >> This is the position inside the docstring. >> >> It seems to me that python--treesit-syntax-propertize assumes that the >> START argument is outside the triple-quoted string. So one solution >> might be to change START to the start of the string if it is within a >> string, as in the attached patch. However, I'm not sure this is the >> right approach. Sounds good to me. I don't see the patch, though, or where to read it. >> Should we use >> syntax-propertize-extend-region-functions? That's another option, but it shouldn't be necessary. After all, the absence of a notification from the parser (which would extend the range) should mean that the node before position 16 is untouched, so there's no real need to clear the properties there. I think there is also another approach--handle two different types of nodes separately, instead of just string_content, so we don't have to start from the beginning of the literal. Like this: diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index e2f614f52c2..4f8b0cb9473 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize (while (re-search-forward (rx (or "\"\"\"" "'''")) end t) (let ((node (treesit-node-at (point)))) ;; The triple quotes surround a non-empty string. - (when (equal (treesit-node-type node) "string_content") - (let ((start (treesit-node-start node)) - (end (treesit-node-end node))) - (put-text-property (1- start) start - 'syntax-table (string-to-syntax "|")) - (put-text-property end (min (1+ end) (point-max)) - 'syntax-table (string-to-syntax "|")))))))) + (cond + ((equal (treesit-node-type node) "string_content") + (put-text-property (1- (treesit-node-start node)) + (treesit-node-start node) + 'syntax-table (string-to-syntax "|"))) + ((and (equal (treesit-node-type node) "string_end") + (= (treesit-node-start node) (- (point) 3))) + (put-text-property (- (point) 3) (- (point) 2) + 'syntax-table (string-to-syntax "|")))))))) ;;; Indentation From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 22 10:44:29 2024 Received: (at 68445) by debbugs.gnu.org; 22 Jan 2024 15:44:30 +0000 Received: from localhost ([127.0.0.1]:41575 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRwTN-0007Z1-Df for submit@debbugs.gnu.org; Mon, 22 Jan 2024 10:44:29 -0500 Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]:60603) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRwTI-0007Yh-LN for 68445@debbugs.gnu.org; Mon, 22 Jan 2024 10:44:28 -0500 Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-28e8c930c01so1296692a91.0 for <68445@debbugs.gnu.org>; Mon, 22 Jan 2024 07:44:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705938254; x=1706543054; darn=debbugs.gnu.org; h=mime-version:user-agent:references:in-reply-to:subject:cc:to:from :message-id:date:from:to:cc:subject:date:message-id:reply-to; bh=39sBZwQYwFCSKHsvhLc5nUjd6lao5D+twW1QsF1vB5g=; b=az40mzQfPQwn8Nna+3089QeDkU45W1DT9x7l+hsaoTcL4lqdo+1KsaSgB0CGV+7NbR hDtoIbYpAJNbUF2DbiDkS7ifu05U9FCquLSzhHoqXmaRbydiFOZB/dlm06MAB72Qsf1/ PaNLXA5IXUnMlZx+oGZHEh1atVewlSBWQSFTklBwEYV5HNXsLf/gfN1g7pPWn/B4NUpk MZaR8ReJ9HZIY6kMjXLU5GuJ77ufAVsnIP6kulhlXE22sSJ1UQDWg/3I1LDhkLKcqtGZ ErjjNsWYaYlA+5+LVvYy0FB4oNyRBBfjvESqw5TichETo+yHtxVfRy9ec6Ob6+hmNEbe 5HvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705938254; x=1706543054; h=mime-version:user-agent:references:in-reply-to:subject:cc:to:from :message-id:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=39sBZwQYwFCSKHsvhLc5nUjd6lao5D+twW1QsF1vB5g=; b=VTxIQOSGXlP/UXkyaZ2Ej7Tm0LVMtxW3HH7YSjYCafIGeDf5eEnzGrlisz+98BfBui m5DxmavUtBfuFrLJAbFsuDx7PYgpuX+AsEHPtix7PFL3WHpYkJ4nGxZ4BOrxa43cX21C W4PRVRnsKimDyA6/vrbuqc0Q452Tqut2F+TbI9Q2LMiUZyuUo0lfWzzARz7f79Szg4qi nEqPgKPAzXgwPFerm5Hypt1HNilJSrAlEYfzGpKZy1B3epndiPvPvxspUbvV18LLcm8X 0b1gzauduFEHgH8sG/6vt6TbhvQ1JOprH+uvlnc3DOGZPDmIHVCVr/3fS2A8R148Kxd6 1nLQ== X-Gm-Message-State: AOJu0YznaoRheCC4yX4gZVtWe1O8Y9gg01dIdKYmym4AgrbnpziNlL/A 0J0jzHA2XOmyqPZKdXHfmz+tTjbvUUHUxJR9saDdM/Kj+RJwCab3 X-Google-Smtp-Source: AGHT+IHU6qG6Ta6yHtZgWa+tBQeNgDAJ7pKkuMswGfp9RywRL6Rwx6xFJxh+O+QJztPcNwjyfwWkMA== X-Received: by 2002:a17:90a:df02:b0:28e:8e7b:1edf with SMTP id gp2-20020a17090adf0200b0028e8e7b1edfmr1464744pjb.75.1705938254589; Mon, 22 Jan 2024 07:44:14 -0800 (PST) Received: from localhost (58x12x133x161.ap58.ftth.ucom.ne.jp. [58.12.133.161]) by smtp.gmail.com with ESMTPSA id d14-20020a17090b004e00b0029020be4298sm9712815pjt.0.2024.01.22.07.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 07:44:14 -0800 (PST) Date: Tue, 23 Jan 2024 00:44:16 +0900 Message-ID: From: kobarity To: Dmitry Gutov Subject: Re: [PATCH] Problem with python--treesit-syntax-propertize In-Reply-To: <5135ab69-3704-459f-b6c7-b8ea738e8d31@gutov.dev> References: <5135ab69-3704-459f-b6c7-b8ea738e8d31@gutov.dev> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.0.50 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 68445 Cc: Yuan Fu , 68445@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, Dmitry Gutov wrote: > On 21/01/2024 16:47, kobarity wrote: > > I am resending my mail, as I made a mistake in X-Debbugs-CC. > Was it supposed to appear in the bug's thread? I don't see it anywhere. My first mail was registered as Bug#68445, and my patch is there. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68445 It says: Report forwarded to casouri gmail.com, dmitry@.gutov.dev, bug-gnu-emacs gnu.org: The extra period is my mistake and it may have caused the problem. I'm sorry for the confusion. > I think there is also another approach--handle two different types of > nodes separately, instead of just string_content, so we don't have to > start from the beginning of the literal. Like this: > > diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el > index e2f614f52c2..4f8b0cb9473 100644 > --- a/lisp/progmodes/python.el > +++ b/lisp/progmodes/python.el > @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize > (while (re-search-forward (rx (or "\"\"\"" "'''")) end t) > (let ((node (treesit-node-at (point)))) > ;; The triple quotes surround a non-empty string. > - (when (equal (treesit-node-type node) "string_content") > - (let ((start (treesit-node-start node)) > - (end (treesit-node-end node))) > - (put-text-property (1- start) start > - 'syntax-table (string-to-syntax "|")) > - (put-text-property end (min (1+ end) (point-max)) > - 'syntax-table (string-to-syntax "|")))))))) > + (cond > + ((equal (treesit-node-type node) "string_content") > + (put-text-property (1- (treesit-node-start node)) > + (treesit-node-start node) > + 'syntax-table (string-to-syntax "|"))) > + ((and (equal (treesit-node-type node) "string_end") > + (= (treesit-node-start node) (- (point) 3))) > + (put-text-property (- (point) 3) (- (point) 2) > + 'syntax-table (string-to-syntax "|")))))))) > > > ;;; Indentation > This approach seems better than my patch, but it does not seem to address the following special case. #+begin_src python """a""""""b""" #+end_src From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 22 13:53:13 2024 Received: (at 68445) by debbugs.gnu.org; 22 Jan 2024 18:53:13 +0000 Received: from localhost ([127.0.0.1]:41805 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRzQ0-0004HN-S0 for submit@debbugs.gnu.org; Mon, 22 Jan 2024 13:53:13 -0500 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:48011) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRzPy-0004H9-Gd for 68445@debbugs.gnu.org; Mon, 22 Jan 2024 13:53:12 -0500 Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id D92455C014E; Mon, 22 Jan 2024 13:53:00 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Mon, 22 Jan 2024 13:53:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm1; t=1705949580; x=1706035980; bh=FSX9OAEp1e6FSHD19ZNzPBM+D6vXzzdeWp76F2w23VY=; b= mCBvLEaREPYKy5ORtIcQfdarT4iNAGlLOfQrHf0NLgQPS4ttRI90t7J6PYJpAVJu IbqJtpbxMv40lXpPsjEsOMEaJVNE+95b/OfIcznb9tx60OnIQ2S9U2r5/rTCDWSv TT9GrjaIFsiStcenMqTaAxF7RkQHa/h2lTsmWIBTuVtVvnVuZyDC9ANDZwDfF7Ff AFD7iQzAXSLXX15JK2LWHcHeBorvahRTj3xs4S1/FPLVtA1Gw89Byuk0N9ULJDzv Da3nPv0czayiH8k/4KJNT61OaL+ngfTudwF1uk1OwSJHyrPWW7Wx+PJZdVT3Mwft DjMaIGh6+hhxnWFn3XQZXQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1705949580; x= 1706035980; bh=FSX9OAEp1e6FSHD19ZNzPBM+D6vXzzdeWp76F2w23VY=; b=G BNwfxInkdzohQv4u7pW9AaP2/2kJT3ebdK0blX1xCz3/70KnpA90YTMfzKZqXb8G p1EfGCINO41jnuTeWmWJK5R/Szn9UH7tHZlV8hk20ztGnFtoMsPfOzUv56B2k6CA wQI8rYFAnqtJbsXbHTuEU3/YAeurKTGgYu89J4u4CB38CS6XHI0LJdx4C+wH4yzq aqHpkVfUlWEMTS5+RBCtAPGWKaker053tpBVWJJUGDYRYuWU3ZGyXujGxxZATBZj oq1jZFvmB/m+uuI3j5PN/yoWe4QwNHrqR/i1RB/c/Xa4Mr9cI/ZZd0FS0d+I1hJ3 xv+JNDrjAA/0nh3SMCMkg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdekiedgudduiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepkfffgggfuffvvehfhfgjtgfgse htjeertddtvdejnecuhfhrohhmpeffmhhithhrhicuifhuthhovhcuoegumhhithhrhies ghhuthhovhdruggvvheqnecuggftrfgrthhtvghrnhepffeifedvleeukedtgfelieegud fgveekfeejveejffetffeuueeugefhveeiuddvnecuffhomhgrihhnpehgnhhurdhorhhg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepughmih htrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 22 Jan 2024 13:52:59 -0500 (EST) Message-ID: <707dc3b4-c7dd-45b5-9764-a9738a9142b6@gutov.dev> Date: Mon, 22 Jan 2024 20:52:58 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] Problem with python--treesit-syntax-propertize Content-Language: en-US To: kobarity References: <5135ab69-3704-459f-b6c7-b8ea738e8d31@gutov.dev> From: Dmitry Gutov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 68445 Cc: Yuan Fu , 68445@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 22/01/2024 17:44, kobarity wrote: > Hi, > > Dmitry Gutov wrote: >> On 21/01/2024 16:47, kobarity wrote: >>> I am resending my mail, as I made a mistake in X-Debbugs-CC. >> Was it supposed to appear in the bug's thread? I don't see it anywhere. > > My first mail was registered as Bug#68445, and my patch is there. > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68445 > > It says: > > Report forwarded to casouri gmail.com, dmitry@.gutov.dev, bug-gnu-emacs gnu.org: > > The extra period is my mistake and it may have caused the problem. > I'm sorry for the confusion. Yeah, but even so that's odd: I'm subscribed to the bug tracker, so the email should have at least arrived in my inbox, but it did not. >> I think there is also another approach--handle two different types of >> nodes separately, instead of just string_content, so we don't have to >> start from the beginning of the literal. Like this: >> >> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el >> index e2f614f52c2..4f8b0cb9473 100644 >> --- a/lisp/progmodes/python.el >> +++ b/lisp/progmodes/python.el >> @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize >> (while (re-search-forward (rx (or "\"\"\"" "'''")) end t) >> (let ((node (treesit-node-at (point)))) >> ;; The triple quotes surround a non-empty string. >> - (when (equal (treesit-node-type node) "string_content") >> - (let ((start (treesit-node-start node)) >> - (end (treesit-node-end node))) >> - (put-text-property (1- start) start >> - 'syntax-table (string-to-syntax "|")) >> - (put-text-property end (min (1+ end) (point-max)) >> - 'syntax-table (string-to-syntax "|")))))))) >> + (cond >> + ((equal (treesit-node-type node) "string_content") >> + (put-text-property (1- (treesit-node-start node)) >> + (treesit-node-start node) >> + 'syntax-table (string-to-syntax "|"))) >> + ((and (equal (treesit-node-type node) "string_end") >> + (= (treesit-node-start node) (- (point) 3))) >> + (put-text-property (- (point) 3) (- (point) 2) >> + 'syntax-table (string-to-syntax "|")))))))) >> >> >> ;;; Indentation >> > > This approach seems better than my patch, but it does not seem to > address the following special case. > > #+begin_src python > """a""""""b""" > #+end_src All right, try the patch below, please. It also covers the case of the empty literal. I've tried to find a case where it would behave poorly (e.g. by misdetecting three quotes from a combination of some other string literals), but couldn't. E.g., s = '''asdasd' is not a concatenation. It's always an error, at least according to the TS grammar. diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index e2f614f52c2..41f612c8b1c 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -1359,15 +1359,15 @@ python--treesit-syntax-propertize (save-excursion (goto-char start) (while (re-search-forward (rx (or "\"\"\"" "'''")) end t) - (let ((node (treesit-node-at (point)))) - ;; The triple quotes surround a non-empty string. - (when (equal (treesit-node-type node) "string_content") - (let ((start (treesit-node-start node)) - (end (treesit-node-end node))) - (put-text-property (1- start) start - 'syntax-table (string-to-syntax "|")) - (put-text-property end (min (1+ end) (point-max)) - 'syntax-table (string-to-syntax "|")))))))) + (let ((node (treesit-node-at (- (point) 3)))) + ;; Handle triple-quoted strings. + (pcase (treesit-node-type node) + ("string_start" + (put-text-property (1- (point)) (point) + 'syntax-table (string-to-syntax "|"))) + ("string_end" + (put-text-property (- (point) 3) (- (point) 2) + 'syntax-table (string-to-syntax "|")))))))) ;;; Indentation From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 23 09:14:53 2024 Received: (at 68445) by debbugs.gnu.org; 23 Jan 2024 14:14:53 +0000 Received: from localhost ([127.0.0.1]:42503 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rSHYD-0007mr-0j for submit@debbugs.gnu.org; Tue, 23 Jan 2024 09:14:53 -0500 Received: from mail-pf1-x433.google.com ([2607:f8b0:4864:20::433]:55457) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rSHYB-0007md-LB for 68445@debbugs.gnu.org; Tue, 23 Jan 2024 09:14:52 -0500 Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-6da202aa138so3349569b3a.2 for <68445@debbugs.gnu.org>; Tue, 23 Jan 2024 06:14:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706019281; x=1706624081; darn=debbugs.gnu.org; h=mime-version:user-agent:references:in-reply-to:subject:cc:to:from :message-id:date:from:to:cc:subject:date:message-id:reply-to; bh=2CN5eyeY11WAievFir91viimgHJC9MDwHjfftuaDmwE=; b=FQ7+sppB3wgGd1uEkikS35gBlBRocKoZCi25O019qLS1f4cbf8kUy1V1GBF7JObTzs zUeRmNcmcRgte2PBeAHQ564aCppDlwAEnHrpeH2MUPSZUTdIVhsAd0tBTaiD5liqpu7f c5cBujiOgNOrqTK5GR1PGMyL5iI8XseixKamWsa6gLDuO4k8rElrB0FKljXLfXy8Dldt l/RDsq9236BAnlw2n1hW3Ph3q8/Pxqghi280ne//rzoBxUh9tL8Bf0yf4itNIGNdtDKi bElaBNj2ozmXXCnZ745M5dTdpdLfn3pnybKMxXCjKB/CBJt2L1r+MOtL4eM07O4vNif9 hPIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706019281; x=1706624081; h=mime-version:user-agent:references:in-reply-to:subject:cc:to:from :message-id:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2CN5eyeY11WAievFir91viimgHJC9MDwHjfftuaDmwE=; b=vDMvM7qm8BhhJ+AhPV5zSrYLS3tyHR+jHSDfiU+QZoMREz5nwnDcssEkPfjEP/NyoK Dj7fBSK/TLVnd9RdBKc2anh5TzmrWFnYpcuTXjlxGCtw703VjUG+wrf84LiOn43C7chg uKc1dbmRWVbGM9rbOvyd6i/NXPakp0+4ZVnfHV4mtaXJKThPlsSAF9P0Wfyzaz0pcPx/ lfM6qOpeIGN8PPiOGivsk91O1T/+YUhCIGN2hGzBlALtfrlBAHaTteJM4oJHAtOtt+eC 5/JkiTm/L+jt2fymist8XLmHyMg/liSyMZnaDdz3mNSQb7M6nm8K7ba3yyHdx/eD8Fyk UbDQ== X-Gm-Message-State: AOJu0YxmuBXuqAX4aAiJVh74v2EoUszf82/al/a+POrFCiOeV08a0jgp KCkd5dOOsV0NRWVbrHrvKk2J/AHL9457Wo4jRG64FP4rGNOblS+Y X-Google-Smtp-Source: AGHT+IHUgXnnCAm24lFwrRa1rw+NKa0bf6BzNxsX4jyLR4zLDvl67OxVDHpLR/OQp++3/7j+6J2dSQ== X-Received: by 2002:a05:6a00:2450:b0:6db:c5e6:8a46 with SMTP id d16-20020a056a00245000b006dbc5e68a46mr8196900pfj.58.1706019280956; Tue, 23 Jan 2024 06:14:40 -0800 (PST) Received: from localhost (58x12x133x161.ap58.ftth.ucom.ne.jp. [58.12.133.161]) by smtp.gmail.com with ESMTPSA id fj1-20020a056a003a0100b006cbe1bb5e3asm11663440pfb.138.2024.01.23.06.14.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 06:14:40 -0800 (PST) Date: Tue, 23 Jan 2024 23:14:42 +0900 Message-ID: From: kobarity To: Dmitry Gutov Subject: Re: [PATCH] Problem with python--treesit-syntax-propertize In-Reply-To: <707dc3b4-c7dd-45b5-9764-a9738a9142b6@gutov.dev> References: <5135ab69-3704-459f-b6c7-b8ea738e8d31@gutov.dev> <707dc3b4-c7dd-45b5-9764-a9738a9142b6@gutov.dev> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.0.50 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 68445 Cc: Yuan Fu , 68445@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Dmitry Gutov wrote: > > On 22/01/2024 17:44, kobarity wrote: > > Hi, > > > > Dmitry Gutov wrote: > >> On 21/01/2024 16:47, kobarity wrote: > >>> I am resending my mail, as I made a mistake in X-Debbugs-CC. > >> Was it supposed to appear in the bug's thread? I don't see it anywhere. > > > > My first mail was registered as Bug#68445, and my patch is there. > > > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68445 > > > > It says: > > > > Report forwarded to casouri gmail.com, dmitry@.gutov.dev, bug-gnu-emacs gnu.org: > > > > The extra period is my mistake and it may have caused the problem. > > I'm sorry for the confusion. > > Yeah, but even so that's odd: I'm subscribed to the bug tracker, so > the email should have at least arrived in my inbox, but it did not. I agree. I can't find my first mail in the bug-gnu-emacs archive. > >> I think there is also another approach--handle two different types of > >> nodes separately, instead of just string_content, so we don't have to > >> start from the beginning of the literal. Like this: > >> > >> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el > >> index e2f614f52c2..4f8b0cb9473 100644 > >> --- a/lisp/progmodes/python.el > >> +++ b/lisp/progmodes/python.el > >> @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize > >> (while (re-search-forward (rx (or "\"\"\"" "'''")) end t) > >> (let ((node (treesit-node-at (point)))) > >> ;; The triple quotes surround a non-empty string. > >> - (when (equal (treesit-node-type node) "string_content") > >> - (let ((start (treesit-node-start node)) > >> - (end (treesit-node-end node))) > >> - (put-text-property (1- start) start > >> - 'syntax-table (string-to-syntax "|")) > >> - (put-text-property end (min (1+ end) (point-max)) > >> - 'syntax-table (string-to-syntax "|")))))))) > >> + (cond > >> + ((equal (treesit-node-type node) "string_content") > >> + (put-text-property (1- (treesit-node-start node)) > >> + (treesit-node-start node) > >> + 'syntax-table (string-to-syntax "|"))) > >> + ((and (equal (treesit-node-type node) "string_end") > >> + (= (treesit-node-start node) (- (point) 3))) > >> + (put-text-property (- (point) 3) (- (point) 2) > >> + 'syntax-table (string-to-syntax "|")))))))) > >> > >> > >> ;;; Indentation > >> > > > > This approach seems better than my patch, but it does not seem to > > address the following special case. > > > > #+begin_src python > > """a""""""b""" > > #+end_src > > All right, try the patch below, please. It also covers the case of the > empty literal. Thanks, it looks good to me. > I've tried to find a case where it would behave poorly (e.g. by > misdetecting three quotes from a combination of some other string > literals), but couldn't. E.g., > > s = '''asdasd' > > is not a concatenation. It's always an error, at least according to > the TS grammar. I think the TS grammar is correct, because this example is also an error according to the Python interpreter. From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 25 20:15:41 2024 Received: (at 68445-done) by debbugs.gnu.org; 26 Jan 2024 01:15:41 +0000 Received: from localhost ([127.0.0.1]:49704 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rTAom-0002an-Ua for submit@debbugs.gnu.org; Thu, 25 Jan 2024 20:15:41 -0500 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:54371) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rTAol-0002aW-2X for 68445-done@debbugs.gnu.org; Thu, 25 Jan 2024 20:15:40 -0500 Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 3CB375C00DB; Thu, 25 Jan 2024 20:15:27 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Thu, 25 Jan 2024 20:15:27 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1706231727; x=1706318127; bh=j3r4kHund9inyvmy2AAuy3XoX3XWl1CVmIxqo1miAXU=; b= Hv2FmrNnTeTj/nHXIMWTsJeQT/iF2q3wWuMnZt6YgWdNXDOMtlu7RlPZb6xmlk0H gJikQsZxknPg9rmklZ6+zDNNgTijyUOjijAfSSjlXUAs33E2Dw/dewtVDOTqJfnN SbO2IE3l/JpQ6MwNS652VQKgl2yZySNDqBCX+AowneYQ1iPVhQLqD9EellXSu3RK XMN6FtarLdJ3jxD27YkNnNpEmpbidzCuW0muWwDEcSdci+mFIhliHH4RN2P65Bzb I2RkO1eXb5XLELpgLsoluHyPyGKaNhmCiM4GITThAnBut0vtkriC4zI/sr7dPytj 7fGNtIRXL7FILIrcpwoV8Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1706231727; x= 1706318127; bh=j3r4kHund9inyvmy2AAuy3XoX3XWl1CVmIxqo1miAXU=; b=q D9mOYBIpylyQ7S/mO0tm7xrQFWUu5GEvUCEvrkgDFAzvrMfCpflcDyNAm2F/mjuX xc1kyUY1kwr8Z+/yrDzf0Pb4k6Y/JjWC3Ge2Zj7kFXnCMLPKaIuI9LOaClMcGx7Y /73NVsQrwUMxgX6E2IdpgSotOsxc9Cl3DQ59pI+PugpMF9+TUXi0cKCgOV3GB0Zf 7WO4zogkVid+jJ5n82/BabeSykHE8gYtojGBv0Z+oBH2+2bF6aCBJuJ91ud9ix8Q EMCozQ7BfzHdArsrmWpP1vHWZu9zBpGLxvWFSEYleJmfrgfcEGQ6qeTDXeWUdwn4 Ib/JyW6JZGFXBRn17wLFg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdeliedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefkffggfgfuvfevfhfhjggtgfesth ejredttddvjeenucfhrhhomhepffhmihhtrhihucfiuhhtohhvuceoughmihhtrhihsehg uhhtohhvrdguvghvqeenucggtffrrghtthgvrhhnpeffieefvdelueektdfgleeigedugf evkeefjeevjeffteffueeuueeghfevieduvdenucffohhmrghinhepghhnuhdrohhrghen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumhhith hrhiesghhuthhovhdruggvvh X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 25 Jan 2024 20:15:25 -0500 (EST) Message-ID: Date: Fri, 26 Jan 2024 03:15:24 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] Problem with python--treesit-syntax-propertize Content-Language: en-US To: kobarity References: <5135ab69-3704-459f-b6c7-b8ea738e8d31@gutov.dev> <707dc3b4-c7dd-45b5-9764-a9738a9142b6@gutov.dev> From: Dmitry Gutov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 68445-done Cc: Yuan Fu , 68445-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 23/01/2024 16:14, kobarity wrote: > > Dmitry Gutov wrote: >> >> On 22/01/2024 17:44, kobarity wrote: >>> Hi, >>> >>> Dmitry Gutov wrote: >>>> On 21/01/2024 16:47, kobarity wrote: >>>>> I am resending my mail, as I made a mistake in X-Debbugs-CC. >>>> Was it supposed to appear in the bug's thread? I don't see it anywhere. >>> >>> My first mail was registered as Bug#68445, and my patch is there. >>> >>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68445 >>> >>> It says: >>> >>> Report forwarded to casouri gmail.com, dmitry@.gutov.dev, bug-gnu-emacs gnu.org: >>> >>> The extra period is my mistake and it may have caused the problem. >>> I'm sorry for the confusion. >> >> Yeah, but even so that's odd: I'm subscribed to the bug tracker, so >> the email should have at least arrived in my inbox, but it did not. > > I agree. I can't find my first mail in the bug-gnu-emacs archive. > >>>> I think there is also another approach--handle two different types of >>>> nodes separately, instead of just string_content, so we don't have to >>>> start from the beginning of the literal. Like this: >>>> >>>> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el >>>> index e2f614f52c2..4f8b0cb9473 100644 >>>> --- a/lisp/progmodes/python.el >>>> +++ b/lisp/progmodes/python.el >>>> @@ -1361,13 +1361,15 @@ python--treesit-syntax-propertize >>>> (while (re-search-forward (rx (or "\"\"\"" "'''")) end t) >>>> (let ((node (treesit-node-at (point)))) >>>> ;; The triple quotes surround a non-empty string. >>>> - (when (equal (treesit-node-type node) "string_content") >>>> - (let ((start (treesit-node-start node)) >>>> - (end (treesit-node-end node))) >>>> - (put-text-property (1- start) start >>>> - 'syntax-table (string-to-syntax "|")) >>>> - (put-text-property end (min (1+ end) (point-max)) >>>> - 'syntax-table (string-to-syntax "|")))))))) >>>> + (cond >>>> + ((equal (treesit-node-type node) "string_content") >>>> + (put-text-property (1- (treesit-node-start node)) >>>> + (treesit-node-start node) >>>> + 'syntax-table (string-to-syntax "|"))) >>>> + ((and (equal (treesit-node-type node) "string_end") >>>> + (= (treesit-node-start node) (- (point) 3))) >>>> + (put-text-property (- (point) 3) (- (point) 2) >>>> + 'syntax-table (string-to-syntax "|")))))))) >>>> >>>> >>>> ;;; Indentation >>>> >>> >>> This approach seems better than my patch, but it does not seem to >>> address the following special case. >>> >>> #+begin_src python >>> """a""""""b""" >>> #+end_src >> >> All right, try the patch below, please. It also covers the case of the >> empty literal. > > Thanks, it looks good to me. > >> I've tried to find a case where it would behave poorly (e.g. by >> misdetecting three quotes from a combination of some other string >> literals), but couldn't. E.g., >> >> s = '''asdasd' >> >> is not a concatenation. It's always an error, at least according to >> the TS grammar. > > I think the TS grammar is correct, because this example is also an > error according to the Python interpreter. Thanks for testing! Installed, and closing. From unknown Fri Aug 15 12:51:18 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 23 Feb 2024 12:24:11 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator