From unknown Thu Jun 19 14:03:23 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#60691 <60691@debbugs.gnu.org> To: bug#60691 <60691@debbugs.gnu.org> Subject: Status: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Reply-To: bug#60691 <60691@debbugs.gnu.org> Date: Thu, 19 Jun 2025 21:03:23 +0000 retitle 60691 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode reassign 60691 emacs submitter 60691 Juri Linkov severity 60691 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 09 12:35:20 2023 Received: (at submit) by debbugs.gnu.org; 9 Jan 2023 17:35:20 +0000 Received: from localhost ([127.0.0.1]:38091 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pEw3M-0004DE-CP for submit@debbugs.gnu.org; Mon, 09 Jan 2023 12:35:20 -0500 Received: from lists.gnu.org ([209.51.188.17]:33332) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pEw3K-0004D6-TW for submit@debbugs.gnu.org; Mon, 09 Jan 2023 12:35:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pEw3J-0007G0-8U for bug-gnu-emacs@gnu.org; Mon, 09 Jan 2023 12:35:17 -0500 Received: from relay6-d.mail.gandi.net ([217.70.183.198]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pEw3H-0005Eh-OE for bug-gnu-emacs@gnu.org; Mon, 09 Jan 2023 12:35:17 -0500 Received: (Authenticated sender: juri@linkov.net) by mail.gandi.net (Postfix) with ESMTPSA id 3DADEC0009 for ; Mon, 9 Jan 2023 17:35:09 +0000 (UTC) From: Juri Linkov To: bug-gnu-emacs@gnu.org Subject: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Organization: LINKOV.NET Date: Mon, 09 Jan 2023 19:16:12 +0200 Message-ID: <867cxv3dnn.fsf@mail.linkov.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/30.0.50 (x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=217.70.183.198; envelope-from=juri@linkov.net; helo=relay6-d.mail.gandi.net X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) X-Debbugs-Cc: Dmitry Gutov After more rules were added recently to ruby-ts--font-lock-settings, font-lock became slow even on very small files. Some measurements: M-: (benchmark-run 1000 (progn (font-lock-mode -1) (font-lock-mode 1) (font-lock-ensure))) M-x ruby-mode (1.3564674989999999 0 0.0) M-x ruby-ts-mode (8.349582391999999 2 6.489918534000001) This is not a problem when files are visited infrequently, but becomes a problem for diff-syntax fontification that wants to highlight simultaneously many files from git logs. So a temporary measure would be not to enable ruby-ts-mode in internal buffers: (add-hook 'find-file-hook (lambda () (when (and (eq major-mode 'ruby-mode) ;; Only when not internal as from diff-syntax (not (string-prefix-p " " (buffer-name)))) (ruby-ts-mode)))) From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 09 17:33:22 2023 Received: (at 60691) by debbugs.gnu.org; 9 Jan 2023 22:33:23 +0000 Received: from localhost ([127.0.0.1]:38348 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pF0hm-0003tH-KA for submit@debbugs.gnu.org; Mon, 09 Jan 2023 17:33:22 -0500 Received: from mail-wm1-f51.google.com ([209.85.128.51]:43611) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pF0hk-0003sx-HX for 60691@debbugs.gnu.org; Mon, 09 Jan 2023 17:33:21 -0500 Received: by mail-wm1-f51.google.com with SMTP id k22-20020a05600c1c9600b003d1ee3a6289so8425690wms.2 for <60691@debbugs.gnu.org>; Mon, 09 Jan 2023 14:33:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=WYRubfQD14nka8tVqf3xnXSCiwbxr3yjYi5GLCe7sTw=; b=ij17FLxTXlpXUnY5J8+Sioh9IjSFeYAPR+atDsUeR0BWYw2F6qpjQWBBrQkUgi3t/3 Pfz3DRzIMdRe6PcEuBE6wbWrpKd+uk7/08O318TDKfhAEeXW4xtNLi6d5gvICpsTWeB3 8Hh/3WF+uJ7UULXKbfbRsIOwV8ijYkXWUFfe9RcUXJ2Q0S8FNYvwV2g1+SwKG3woi21S ohrOqHTrmegct/uqT7PiDHLoD9Hkwg9/ZHOR9jPT5yGlpYMa/S9NlhH0vYuUc2hn5xJS DklzrdabSToE2HZtG+8FiJaUBMbX1tIDsbrZ8wH44QbI8EXu9oVUSYQAaK1AZVjCBh5H WgXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WYRubfQD14nka8tVqf3xnXSCiwbxr3yjYi5GLCe7sTw=; b=Fq8eL8l3M92G7CjY+LI2jQLzMd992jmnmKE0FigHz0VKEN/lKjh3bTBmmT1rIO36cX 9LvFbymgFu8Na5pdPm/eWUfqr9xmXbNhh94yQqQNLNlrTTh5GBNvuskandvKcGd39ePy yB5vlkYQiqUL1aB92OUAIvOLBDe0D7mheHXE8aqEuvXkuwyDKjRWR64eCgKKbh7hhUVO 5gklnbYOSduEVRsf+KN37hmU6O0BrPSkg73f2+Igq1qCV7mUMH09kGCILm5RnGlu4cm/ SnVKzaf1R8hMtb5JoVzUecRWPYhalRwGSmGg1e1BkyL52BPIkQGsAk4VzPhzSH4WNRfR bBpw== X-Gm-Message-State: AFqh2koRyLKs5cRcQIy+OeYgWS6T/74fF+9YO+Q7VJRprxxBFSuPLgjb kgsrPIXpPJdQJE8SWM/R+I0= X-Google-Smtp-Source: AMrXdXsGYKRrlHNV0Y6gPFwFPASJC3AIAEVFKgNP5cK/yVbpSCpOnZjAn472t/hlRDEJrg1IoCrywQ== X-Received: by 2002:a7b:ca51:0:b0:3d2:7a7:5cc6 with SMTP id m17-20020a7bca51000000b003d207a75cc6mr51171463wml.18.1673303594410; Mon, 09 Jan 2023 14:33:14 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id o5-20020a05600c510500b003b4ff30e566sm45901wms.3.2023.01.09.14.33.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 09 Jan 2023 14:33:13 -0800 (PST) Message-ID: <51ee2f6f-6e1d-eccd-f536-461d916cc94d@yandex.ru> Date: Tue, 10 Jan 2023 00:33:12 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Juri Linkov , 60691@debbugs.gnu.org References: <867cxv3dnn.fsf@mail.linkov.net> From: Dmitry Gutov In-Reply-To: <867cxv3dnn.fsf@mail.linkov.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) Hi! On 09/01/2023 19:16, Juri Linkov wrote: > X-Debbugs-Cc: Dmitry Gutov > > After more rules were added recently to ruby-ts--font-lock-settings, > font-lock became slow even on very small files. Some measurements: If you saw a particular commit that made things slower, did you try reverting it? What was the performance after? > M-: (benchmark-run 1000 (progn (font-lock-mode -1) (font-lock-mode 1) (font-lock-ensure))) > > M-x ruby-mode > (1.3564674989999999 0 0.0) > > M-x ruby-ts-mode > (8.349582391999999 2 6.489918534000001) I have tried this scenario (which, to be frank, is pretty artificial, given that fontification is usually performed in chunks, not over the whole buffer). Perhaps the results depend on a particular file. The ones I have tried (ruby.rb and ruby-after-operator-indent.rb) show only 2x difference (or less). The difference was in favor of ruby-mode, but given the difference in approaches I wouldn't be surprised if ruby-ts-mode incurs a fixed overhead somewhere. > This is not a problem when files are visited infrequently, but > becomes a problem for diff-syntax fontification that wants to > highlight simultaneously many files from git logs. > So a temporary measure would be not to enable ruby-ts-mode > in internal buffers: Is it common to try to highlight 1000 or even 100 files in one diff? > (add-hook 'find-file-hook > (lambda () > (when (and (eq major-mode 'ruby-mode) > ;; Only when not internal as from diff-syntax > (not (string-prefix-p " " (buffer-name)))) > (ruby-ts-mode)))) Have you tried similar tests with other -ts- modes? Ones with complex font-lock rules in particular. I've tried commenting out different rules in ruby-ts--font-lock-settings, but none of them seem to have particularly outsides impact. Performance seems, roughly, inversely proportional to the number of separate "features". And if all ts modes turn out to have this problem, perhaps the place to improve this is inside some common code. From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 10 03:25:43 2023 Received: (at 60691) by debbugs.gnu.org; 10 Jan 2023 08:25:43 +0000 Received: from localhost ([127.0.0.1]:38704 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pF9x1-0004dY-6d for submit@debbugs.gnu.org; Tue, 10 Jan 2023 03:25:43 -0500 Received: from relay10.mail.gandi.net ([217.70.178.230]:54479) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pF9wz-0004dI-3d for 60691@debbugs.gnu.org; Tue, 10 Jan 2023 03:25:42 -0500 Received: (Authenticated sender: juri@linkov.net) by mail.gandi.net (Postfix) with ESMTPSA id B02AD240004; Tue, 10 Jan 2023 08:25:31 +0000 (UTC) From: Juri Linkov To: Dmitry Gutov Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode In-Reply-To: <51ee2f6f-6e1d-eccd-f536-461d916cc94d@yandex.ru> (Dmitry Gutov's message of "Tue, 10 Jan 2023 00:33:12 +0200") Organization: LINKOV.NET References: <867cxv3dnn.fsf@mail.linkov.net> <51ee2f6f-6e1d-eccd-f536-461d916cc94d@yandex.ru> Date: Tue, 10 Jan 2023 10:10:53 +0200 Message-ID: <86leman71u.fsf@mail.linkov.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/30.0.50 (x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) >> After more rules were added recently to ruby-ts--font-lock-settings, >> font-lock became slow even on very small files. Some measurements: > > If you saw a particular commit that made things slower, did you try > reverting it? What was the performance after? No particular commit, just adding more rules degrades performance gradually. >> M-: (benchmark-run 1000 (progn (font-lock-mode -1) (font-lock-mode 1) (font-lock-ensure))) >> M-x ruby-mode >> (1.3564674989999999 0 0.0) >> M-x ruby-ts-mode >> (8.349582391999999 2 6.489918534000001) > > I have tried this scenario (which, to be frank, is pretty artificial, given > that fontification is usually performed in chunks, not over the whole > buffer). > > Perhaps the results depend on a particular file. The ones I have tried > (ruby.rb and ruby-after-operator-indent.rb) show only 2x difference (or > less). The difference was in favor of ruby-mode, but given the difference > in approaches I wouldn't be surprised if ruby-ts-mode incurs a fixed > overhead somewhere. On test/lisp/progmodes/ruby-mode-resources/ruby.rb I see these numbers: ruby-mode (8.701560543000001 95 1.045961102) ruby-ts-mode (34.653148898000005 1464 16.904981779) >> This is not a problem when files are visited infrequently, but >> becomes a problem for diff-syntax fontification that wants to >> highlight simultaneously many files from git logs. >> So a temporary measure would be not to enable ruby-ts-mode >> in internal buffers: > > Is it common to try to highlight 1000 or even 100 files in one diff? 100 is rare, but tens is pretty common, so this problem affects only this specific case. >> (add-hook 'find-file-hook >> (lambda () >> (when (and (eq major-mode 'ruby-mode) >> ;; Only when not internal as from diff-syntax >> (not (string-prefix-p " " (buffer-name)))) >> (ruby-ts-mode)))) > > Have you tried similar tests with other -ts- modes? Ones with complex > font-lock rules in particular. I tried with c-ts-mode, and it's very fast. > I've tried commenting out different rules in ruby-ts--font-lock-settings, > but none of them seem to have particularly outsides impact. Performance > seems, roughly, inversely proportional to the number of separate > "features". Indeed, this is what I see - no particular rule, only their number affects performance. > And if all ts modes turn out to have this problem, perhaps the place to > improve this is inside some common code. I noticed that while most library files are small, e.g. libtree-sitter-c.so is 401,528 bytes, libtree-sitter-ruby.so is 2,130,616 bytes that means that it has more complex logic that might explain its performance. In this case, when nothing could be done to improve performance, please close this request. From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 10 09:11:00 2023 Received: (at 60691) by debbugs.gnu.org; 10 Jan 2023 14:11:00 +0000 Received: from localhost ([127.0.0.1]:39163 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFFLA-0006c8-8e for submit@debbugs.gnu.org; Tue, 10 Jan 2023 09:11:00 -0500 Received: from mail-wm1-f46.google.com ([209.85.128.46]:52084) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFFL8-0006bs-47 for 60691@debbugs.gnu.org; Tue, 10 Jan 2023 09:10:59 -0500 Received: by mail-wm1-f46.google.com with SMTP id g10so8886810wmo.1 for <60691@debbugs.gnu.org>; Tue, 10 Jan 2023 06:10:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=TjVO7Iu2PTtRuEs848MOngu4IUBVHaOl2a8o366V1ms=; b=mrNBiJoScwiSRm/Lhc5+jcHxvT/Vdwmt6EVFVRtCfCxCxKCX4Iybb06RjBS+Ae87jl mncde8aZGREUgKaagbs8TnpZ1HX9gBR5KWrWZ7Y2aBBMo6vnCpM44Sj//0ttVlwzMMCd YF760PI+PIlkdhbKUGKao3uQuURe6rANQOnLDEc5+Yb8YkO+s9s/V7db/HfsFoz27Of/ QJzcy39gwJlgs9jeV4DiQ3bPo8vklzazgObzLybOhZ49zROZ/3Zs0nZs1jUoUbB6gOtr ZjedvEu9g1VV4VwOzCXQcO8ybTFWXVhNSnH0G72xWgu4IZC35bzhe+8uuKuDnytTozQR 1Bdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TjVO7Iu2PTtRuEs848MOngu4IUBVHaOl2a8o366V1ms=; b=2EtT6sutPxHqezg8hl7tHubZqrzA3mksm5tr/QxUU62nb4Aw9UTxWXmjNWoPA3DQ2y wAeOAPDPqGWxH0oB1EfwlYOaObJ2cWkdJpTRGzWkSComeor8zCNmB+obVzRTL8JCUrQa 2u5aYN+IhNI6h0T0RICZDl46AF9D91dHqfpNFFiEKj39zw57wiPWsbZI38198a8ADf6A rl6ZzmPfFxV2av+GzzMa0J209T2HARGMLsgGpNKM/3FF/l8prCkygBMKFVcDg9mPK64c LHgjpSeikTDVkxK1ttK6OZlUnw7IAAZvureCgBh6Ig3Gtd8ZZJa+/1BRplbqenpcIXH/ kb2Q== X-Gm-Message-State: AFqh2krljc7xaIfA0DSMbsL/HveqmiyqAUQOQDyGnixwDPtlcAh1tuYD MrCFqy049GMnAeltKO6w/3k= X-Google-Smtp-Source: AMrXdXvMhePPVrJP0Vk6j6S4D0bW5vHAxMAn40mHnz+tstN2UpV1JNYObp/GlBJMFWQkXFRIjfy2Ww== X-Received: by 2002:a05:600c:4da2:b0:3d2:39dc:f50e with SMTP id v34-20020a05600c4da200b003d239dcf50emr49123608wmp.7.1673359852028; Tue, 10 Jan 2023 06:10:52 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id i14-20020a05600c354e00b003d1d5a83b2esm21678073wmq.35.2023.01.10.06.10.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Jan 2023 06:10:51 -0800 (PST) Message-ID: <4ed5051c-ea5a-91b1-6b8c-5349a3495a16@yandex.ru> Date: Tue, 10 Jan 2023 16:10:49 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Juri Linkov , Yuan Fu References: <867cxv3dnn.fsf@mail.linkov.net> <51ee2f6f-6e1d-eccd-f536-461d916cc94d@yandex.ru> <86leman71u.fsf@mail.linkov.net> From: Dmitry Gutov In-Reply-To: <86leman71u.fsf@mail.linkov.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) On 10/01/2023 10:10, Juri Linkov wrote: >>> After more rules were added recently to ruby-ts--font-lock-settings, >>> font-lock became slow even on very small files. Some measurements: >> >> If you saw a particular commit that made things slower, did you try >> reverting it? What was the performance after? > > No particular commit, just adding more rules degrades performance > gradually. But I don't think I added that many rules recently. No more than a quarter anyway. >>> M-: (benchmark-run 1000 (progn (font-lock-mode -1) (font-lock-mode 1) (font-lock-ensure))) >>> M-x ruby-mode >>> (1.3564674989999999 0 0.0) >>> M-x ruby-ts-mode >>> (8.349582391999999 2 6.489918534000001) >> >> I have tried this scenario (which, to be frank, is pretty artificial, given >> that fontification is usually performed in chunks, not over the whole >> buffer). >> >> Perhaps the results depend on a particular file. The ones I have tried >> (ruby.rb and ruby-after-operator-indent.rb) show only 2x difference (or >> less). The difference was in favor of ruby-mode, but given the difference >> in approaches I wouldn't be surprised if ruby-ts-mode incurs a fixed >> overhead somewhere. > > On test/lisp/progmodes/ruby-mode-resources/ruby.rb I see these numbers: > > ruby-mode > (8.701560543000001 95 1.045961102) > > ruby-ts-mode > (34.653148898000005 1464 16.904981779) Interesting. It's 12s vs 36s for me, as I've retested now. >>> This is not a problem when files are visited infrequently, but >>> becomes a problem for diff-syntax fontification that wants to >>> highlight simultaneously many files from git logs. >>> So a temporary measure would be not to enable ruby-ts-mode >>> in internal buffers: >> >> Is it common to try to highlight 1000 or even 100 files in one diff? > > 100 is rare, but tens is pretty common, so this problem affects > only this specific case. So it's a 0,8-3s delay in those cases? That's not ideal. >>> (add-hook 'find-file-hook >>> (lambda () >>> (when (and (eq major-mode 'ruby-mode) >>> ;; Only when not internal as from diff-syntax >>> (not (string-prefix-p " " (buffer-name)))) >>> (ruby-ts-mode)))) >> >> Have you tried similar tests with other -ts- modes? Ones with complex >> font-lock rules in particular. > > I tried with c-ts-mode, and it's very fast. Just how fast is it? The number of font-lock features is has is comparable (though a little smaller). I've tried the same benchmark for it in admin/alloc-colors.c, and it comes out to (3.2004193190000003 30 0.9609690980000067) Which seems comparable. Not sure how to directly test the modes against each other, but if I enable ruby-ts-mode in the same file, the benchmark comes to 1s. Or if I enable c-ts-mode in ruby.rb -- 16s. >> I've tried commenting out different rules in ruby-ts--font-lock-settings, >> but none of them seem to have particularly outsides impact. Performance >> seems, roughly, inversely proportional to the number of separate >> "features". > > Indeed, this is what I see - no particular rule, only their number > affects performance. > >> And if all ts modes turn out to have this problem, perhaps the place to >> improve this is inside some common code. > > I noticed that while most library files are small, e.g. > libtree-sitter-c.so is 401,528 bytes, > libtree-sitter-ruby.so is 2,130,616 bytes > that means that it has more complex logic > that might explain its performance. ruby is indeed one of the larger ones. Among the ones I have here compiled, it's exceeded only by cpp. 2.29 MB vs 2.12 MB. But testing admin/alloc-colors.c with c++-ts-mode vs c-ts-mode gives very similar performance, so it's unlikely that the complexity of the grammar is directly responsible. > In this case, when nothing could be done to improve performance, > please close this request. Perhaps Yuan has some further ideas. There are some strong oddities here: - Some time into debugging and repeating the benchmark again and again, I get the "Pure Lisp storage overflowed" message. Just once per Emacs session. It doesn't seem to change much, so it might be unimportant. - The profiler output looks like this: 18050 75% - font-lock-fontify-syntactically-region 15686 65% - treesit-font-lock-fontify-region 3738 15% treesit--children-covering-range-recurse 188 0% treesit-fontify-with-override - When running the benchmark for the first time in a buffer (such as ruby.rb), the variable treesit--font-lock-fast-mode is usually changed to t. In one Emacs session, after I changed it to nil and re-ran the benchmark, the variable stayed nil, and the benchmark ran much faster (like 10s vs 36s). In the next session, after I restarted Emacs, that didn't happen: it always stayed at t, even if I reset it to nil between runs. But if I comment out the block in treesit-font-lock-fontify-region that uses it ;; (when treesit--font-lock-fast-mode ;; (setq nodes (treesit--children-covering-range-recurse ;; (car nodes) start end (* 4 jit-lock-chunk-size)))) and evaluate the defun, the benchmark runs much faster again: 11s. (But then I brought it all back, and re-ran the tests, and the variable stayed nil that time around; to sum up: the way it's turned on is unstable.) Should treesit--font-lock-fast-mode be locally bound inside that function, so that it's reset between chunks? Or maybe the condition for its enabling should be tweaked? E.g. I don't think there are any particularly large or deep nodes in ruby.rb's parse tree. It's a very shallow file. From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 10 12:57:53 2023 Received: (at 60691) by debbugs.gnu.org; 10 Jan 2023 17:57:53 +0000 Received: from localhost ([127.0.0.1]:41068 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFIsj-00073F-KI for submit@debbugs.gnu.org; Tue, 10 Jan 2023 12:57:53 -0500 Received: from relay3-d.mail.gandi.net ([217.70.183.195]:42941) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFIsi-00072o-79 for 60691@debbugs.gnu.org; Tue, 10 Jan 2023 12:57:52 -0500 Received: (Authenticated sender: juri@linkov.net) by mail.gandi.net (Postfix) with ESMTPSA id E634B60008; Tue, 10 Jan 2023 17:57:44 +0000 (UTC) From: Juri Linkov To: Dmitry Gutov Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode In-Reply-To: <4ed5051c-ea5a-91b1-6b8c-5349a3495a16@yandex.ru> (Dmitry Gutov's message of "Tue, 10 Jan 2023 16:10:49 +0200") Organization: LINKOV.NET References: <867cxv3dnn.fsf@mail.linkov.net> <51ee2f6f-6e1d-eccd-f536-461d916cc94d@yandex.ru> <86leman71u.fsf@mail.linkov.net> <4ed5051c-ea5a-91b1-6b8c-5349a3495a16@yandex.ru> Date: Tue, 10 Jan 2023 19:50:44 +0200 Message-ID: <86tu0yqnwr.fsf@mail.linkov.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/30.0.50 (x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 60691 Cc: Yuan Fu , 60691@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) >>> Is it common to try to highlight 1000 or even 100 files in one diff? >> 100 is rare, but tens is pretty common, so this problem affects >> only this specific case. > > So it's a 0,8-3s delay in those cases? That's not ideal. The delay is noticeable, alas. >> I noticed that while most library files are small, e.g. >> libtree-sitter-c.so is 401,528 bytes, >> libtree-sitter-ruby.so is 2,130,616 bytes >> that means that it has more complex logic >> that might explain its performance. > > ruby is indeed one of the larger ones. Among the ones I have here compiled, > it's exceeded only by cpp. 2.29 MB vs 2.12 MB. The winner is libtree-sitter-julia.so with 7.25 MB. But regarding libtree-sitter-cpp.so I confirm it's 2.3 MB. And c++-ts-mode is even faster than c-ts-mode. On the same admin/alloc-colors.c: c-mode (33.378821569 1500 17.632000617) c-ts-mode (2.1949608069999997 34 0.4119784769999981) c++-ts-mode (2.0979403910000003 34 0.39749122499999956) So size doesn't matter. From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 11 07:12:20 2023 Received: (at 60691) by debbugs.gnu.org; 11 Jan 2023 12:12:20 +0000 Received: from localhost ([127.0.0.1]:41850 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFZxr-0002hD-UR for submit@debbugs.gnu.org; Wed, 11 Jan 2023 07:12:20 -0500 Received: from mail-wm1-f47.google.com ([209.85.128.47]:56299) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFZxo-0002gv-6u for 60691@debbugs.gnu.org; Wed, 11 Jan 2023 07:12:18 -0500 Received: by mail-wm1-f47.google.com with SMTP id l26so10940024wme.5 for <60691@debbugs.gnu.org>; Wed, 11 Jan 2023 04:12:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=r/dqXjaqPACUM8T6HzTmldIJgBC3gYahWG0LRqnobaE=; b=E1NuJ7q9CT91Z5wPrmAGiQYwFVMrXE3FCF9rH/ty3V8nBu3aaGPbd4slClsF2WLMxU JAnDJ/6YR/+W3DJ+1CMdWhq3+wl6yUd2SkDUGrT/oEisCM+JjGqr6Vje73vFpCO/nirx IvcqkRnSCjGPD4Tl5+Y/CADiBiscwHuM/mOtcnyixNjw7xxaOw+ax07xYH6h8S4075ey dGubsj4IsOSGGWQ1KKoKJkAHMw4s8WqY1dr4VN4PQ9858nkhpmjz7uCn+Fs7ld1r6YvG QrbBcUOQXNxVkbe+LVayPElKm54E/KN/x/kX6aGEYWnpS9JQpp+e1rZlFXFhmg7y1hwH 3P9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=r/dqXjaqPACUM8T6HzTmldIJgBC3gYahWG0LRqnobaE=; b=jVzrMFEbpLDp03pnJDZmfiA95j7zr+Clzc5ic15W4fZSFM0cMtDrKsb27Iu4OLG7Sl 9PJMcAbRap5vswhPjztES1BXta43ETlZT2kGOLhXSkUIqOBFfN9hX5a6je+Cd5x6JHUD frIaqMtLcow5fEvRLo/OyInoixpm7VW1/JcEci3U93lvb8WDC97DAFY6XOfYqLziBrNC Aujaz11KNCh4YR34EcL1fM/kzOG8molU61kD8t5FCB44FsGCLA0ULIB6NqqSRNwwuABd +p6BXeTQA0bCAGnelpXQ/bXXNJV2CFe17ivANYsPCahWO7JLz44Zgawgs6bNO+LSdda1 rKug== X-Gm-Message-State: AFqh2kq69iy+F0qzBM2mWHYKRhqFPtIKoW2d+N8mtmqgdl0HSGxLy8Vd qu0JrHEYgn+r1e3UT1pxEs4= X-Google-Smtp-Source: AMrXdXuO5Z0p/z3o7HMYwqaVdRD/nYqwEHixlgivPZSyu6BqB1biVzoSO7OxSjc/p2z6+KqzAtUyKg== X-Received: by 2002:a05:600c:4e48:b0:3cf:69f4:bfd4 with SMTP id e8-20020a05600c4e4800b003cf69f4bfd4mr53607359wmq.7.1673439129168; Wed, 11 Jan 2023 04:12:09 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id l36-20020a05600c1d2400b003d9fb59c16fsm5362760wms.11.2023.01.11.04.12.07 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 11 Jan 2023 04:12:08 -0800 (PST) Message-ID: Date: Wed, 11 Jan 2023 14:12:06 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Juri Linkov References: <867cxv3dnn.fsf@mail.linkov.net> <51ee2f6f-6e1d-eccd-f536-461d916cc94d@yandex.ru> <86leman71u.fsf@mail.linkov.net> <4ed5051c-ea5a-91b1-6b8c-5349a3495a16@yandex.ru> <86tu0yqnwr.fsf@mail.linkov.net> From: Dmitry Gutov In-Reply-To: <86tu0yqnwr.fsf@mail.linkov.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691 Cc: Yuan Fu , 60691@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) On 10/01/2023 19:50, Juri Linkov wrote: >>>> Is it common to try to highlight 1000 or even 100 files in one diff? >>> 100 is rare, but tens is pretty common, so this problem affects >>> only this specific case. >> >> So it's a 0,8-3s delay in those cases? That's not ideal. > > The delay is noticeable, alas. Right. I'm somewhat worried for the processing speed xref--collect-matches too. But that's probably only going to be noticeable after we add syntax-propertize-function to ruby-ts-mode. >>> I noticed that while most library files are small, e.g. >>> libtree-sitter-c.so is 401,528 bytes, >>> libtree-sitter-ruby.so is 2,130,616 bytes >>> that means that it has more complex logic >>> that might explain its performance. >> >> ruby is indeed one of the larger ones. Among the ones I have here compiled, >> it's exceeded only by cpp. 2.29 MB vs 2.12 MB. > > The winner is libtree-sitter-julia.so with 7.25 MB. > But regarding libtree-sitter-cpp.so I confirm it's 2.3 MB. > And c++-ts-mode is even faster than c-ts-mode. Yep. > On the same admin/alloc-colors.c: > > c-mode > (33.378821569 1500 17.632000617) > > c-ts-mode > (2.1949608069999997 34 0.4119784769999981) > > c++-ts-mode > (2.0979403910000003 34 0.39749122499999956) > > So size doesn't matter. From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 11 07:12:37 2023 Received: (at 60691) by debbugs.gnu.org; 11 Jan 2023 12:12:37 +0000 Received: from localhost ([127.0.0.1]:41853 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFZy9-0002hq-8X for submit@debbugs.gnu.org; Wed, 11 Jan 2023 07:12:37 -0500 Received: from mail-wm1-f47.google.com ([209.85.128.47]:56299) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFZy7-0002gv-W1 for 60691@debbugs.gnu.org; Wed, 11 Jan 2023 07:12:36 -0500 Received: by mail-wm1-f47.google.com with SMTP id l26so10940815wme.5 for <60691@debbugs.gnu.org>; Wed, 11 Jan 2023 04:12:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:references:cc:to:from :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=vC6n3tsfqPm2aUlkwOCfZYpy83jGbKoAY0Ti3HvVIVw=; b=BLpKty+bgWmjtBIcDYATth9XVabHbrnCaZOaXujQG7sDLX8QWjBMmj3dwQrTqL4zCH 432aahievevUnNb4zuxrrIYnDiOT5eWwcwh5d7SZzLbzOroB7DPKhIfZ5b5LVWClhQAJ p5i/qK2SqLa/mmUCYM5iPdY28e6JxYQJno/GSleq/gNdWQJAWaufTZ9QE1HXhkH3F7i2 D1/S0QJrh/ej2mhm7iLQEhE2CK2+LzZlr6VW2/NduwhqeNOOeis3z6/9rpky/ADyr+Uq DzFRC/qSxBXPc6OzFzKjW1XaAOuMxhZXoHrbX7aPGJyMsiycUG17YAboYFUmJs44EIux tK2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:references:cc:to:from :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vC6n3tsfqPm2aUlkwOCfZYpy83jGbKoAY0Ti3HvVIVw=; b=QWVqvzAMNmyZDRPePg1omhREAa15lUzQ315TZia7c5XCBBLcGf0vIywYi3n3dRVqOP Poy1Ad/tzhyI0txsgCQ92SOXJ0Rqp37anf3DTbEbs6Ou1BruyLg3SWbKnuZVr5bclyTc HrQMjVU+IfnsRh3JJsJgjM2SZMl5eD6DaVBgJjmVzuzn5E6drHHaflvrijLW/2jpnOqc 4Lij9pXYQ5O/NLf/5OuR8emKJxVSkhN9/rS5Bk0SmAfvDjbh2jfUS3e2Y69DBo89ZytS 7JIHWrxmeWP3/PF8rqKo4jigTGrTuomD1XQqk4DWbDFjfiDbiaAfsW+ZlJhrrEo091px cd+Q== X-Gm-Message-State: AFqh2kqCx3nI1vLUDCnJr7mgHiGP6b6zpe2QdYL1AlpVaYOXJYiAQ0uQ qu/b410y4MU5dTa/+ozWKpAMA5VTeCo= X-Google-Smtp-Source: AMrXdXsb/M8YjD+tYmzVQ7OvS3SwAPae2g8YhFgFJHUBEfHBYX7hXdHwo7QdG8kScjSbkU53MDvoUw== X-Received: by 2002:a05:600c:4e51:b0:3d1:e1f4:21d1 with SMTP id e17-20020a05600c4e5100b003d1e1f421d1mr63557064wmq.26.1673439155546; Wed, 11 Jan 2023 04:12:35 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id g12-20020a05600c310c00b003c70191f267sm24680009wmo.39.2023.01.11.04.12.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 11 Jan 2023 04:12:35 -0800 (PST) Message-ID: <122d12c9-9b7f-d8ff-9679-f2af0e8e2a93@yandex.ru> Date: Wed, 11 Jan 2023 14:12:33 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US From: Dmitry Gutov To: Juri Linkov , Yuan Fu References: <867cxv3dnn.fsf@mail.linkov.net> <51ee2f6f-6e1d-eccd-f536-461d916cc94d@yandex.ru> <86leman71u.fsf@mail.linkov.net> <4ed5051c-ea5a-91b1-6b8c-5349a3495a16@yandex.ru> In-Reply-To: <4ed5051c-ea5a-91b1-6b8c-5349a3495a16@yandex.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) Yuan? Just making sure you got this message. On 10/01/2023 16:10, Dmitry Gutov wrote: > Perhaps Yuan has some further ideas. There are some strong oddities here: > > - Some time into debugging and repeating the benchmark again and again, > I get the "Pure Lisp storage overflowed" message. Just once per Emacs > session. It doesn't seem to change much, so it might be unimportant. > > - The profiler output looks like this: > >   18050  75%                    - font-lock-fontify-syntactically-region >   15686  65%                     - treesit-font-lock-fontify-region >    3738  15% treesit--children-covering-range-recurse >     188   0%                        treesit-fontify-with-override > > - When running the benchmark for the first time in a buffer (such as > ruby.rb), the variable treesit--font-lock-fast-mode is usually changed > to t. In one Emacs session, after I changed it to nil and re-ran the > benchmark, the variable stayed nil, and the benchmark ran much faster > (like 10s vs 36s). > > In the next session, after I restarted Emacs, that didn't happen: it > always stayed at t, even if I reset it to nil between runs. But if I > comment out the block in treesit-font-lock-fontify-region that uses it > >     ;; (when treesit--font-lock-fast-mode >     ;;   (setq nodes (treesit--children-covering-range-recurse >     ;;                (car nodes) start end (* 4 jit-lock-chunk-size)))) > > and evaluate the defun, the benchmark runs much faster again: 11s. > > (But then I brought it all back, and re-ran the tests, and the variable > stayed nil that time around; to sum up: the way it's turned on is > unstable.) > > Should treesit--font-lock-fast-mode be locally bound inside that > function, so that it's reset between chunks? Or maybe the condition for > its enabling should be tweaked? E.g. I don't think there are any > particularly large or deep nodes in ruby.rb's parse tree. It's a very > shallow file. From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 12 16:58:58 2023 Received: (at 60691) by debbugs.gnu.org; 12 Jan 2023 21:58:58 +0000 Received: from localhost ([127.0.0.1]:48724 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pG5b8-0003mj-1I for submit@debbugs.gnu.org; Thu, 12 Jan 2023 16:58:58 -0500 Received: from mail-pj1-f43.google.com ([209.85.216.43]:45003) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pG5b5-0003mU-Kw for 60691@debbugs.gnu.org; Thu, 12 Jan 2023 16:58:56 -0500 Received: by mail-pj1-f43.google.com with SMTP id o7-20020a17090a0a0700b00226c9b82c3aso22453651pjo.3 for <60691@debbugs.gnu.org>; Thu, 12 Jan 2023 13:58:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:cc:date:message-id:subject:mime-version :content-transfer-encoding:from:from:to:cc:subject:date:message-id :reply-to; bh=/9woX2rdgAbZJ8bomT/4I+Ne9ZTerZyswRjNWI7oGkE=; b=TGu/eHxMPdXqGQDlTUsZu7WQJLAba4FGFePSyKV6LC76RBcKAT5zdbAxXbHTCRUH46 CF6lDPMcINyjQ1upETykht8ptQ0Mtb0QZmasa1C5z7g10lZu1+ztA3bgqus7+ZkKuo4m snnsxr/guH8E7nH9EXCx5Z7uJ/D3gIBlqEPINjyiTzMS3mhxcOIzvrvgenos0uOCPCdC qW4+JzeDnwwY5+EWiShCKQ4+xeDLnmgDw7khpvgnHTxjsC4w0eDeZXuN1nYd0ZFF7e/t J21tBp67jZFNSfWpSVGMnVu2oTP3/Rqy2f3KpCnlu1wW5YBQFCo4QeKXssNBQ+T3I/dr GGWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:cc:date:message-id:subject:mime-version :content-transfer-encoding:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/9woX2rdgAbZJ8bomT/4I+Ne9ZTerZyswRjNWI7oGkE=; b=FvcxUX2tl0f9wgaX4c9EPRFFC2YJzpS2WOUHLGedqHyeOUsRQ9EJZePO2wtEYpevGU 45uSKetyLIyUjxykEuRNjAZ6UAIrAN7SsW5mf4vPtZ3aYnRcWDTmF/vxEzaeFgu0ORnV m0+4I299oooEJlSZj03dOgDpxi5E+J6jb3Cabnjv79OjGKrWuz55kwhEU9LRPOTWyLkq JzhspLkDp6FvbGnystfZ/AjTYMW3gvrOOMOr6NgTTPPIsI8VBLtOPHIUrGYM9QMZ0qzn FMpkiVsbGph4kUS5dt+ZxxBb2FLZz+4dOImXBMLDv5kKaCiuaYR3gxIFnSQU53TfYSlk Pz4w== X-Gm-Message-State: AFqh2kpkWeqPt1Yl4HPsRkprfZVJH6h86gVKSg6vYwTcDpiUdbroaX3R 615TccQ+PFHOr0PRauDUogw= X-Google-Smtp-Source: AMrXdXuz6obcOM3zRLdOo2rr7B+Y3MayZlhGofsxOm3zsumwAPCbGueVNgdmvrnpF8mf14Xqo+Q/gA== X-Received: by 2002:a17:903:2093:b0:194:4285:dfef with SMTP id d19-20020a170903209300b001944285dfefmr9954462plc.49.1673560729810; Thu, 12 Jan 2023 13:58:49 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id a10-20020a170902ecca00b00189a7fbfd44sm12612249plh.211.2023.01.12.13.58.49 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Jan 2023 13:58:49 -0800 (PST) From: Yuan Fu Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Message-Id: <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> Date: Thu, 12 Jan 2023 13:58:48 -0800 To: Dmitry Gutov X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Dmitry Gutov writes: > Yuan? Just making sure you got this message. Sorry for the delay :-) > On 10/01/2023 16:10, Dmitry Gutov wrote: >> Perhaps Yuan has some further ideas. There are some strong oddities = here: >> - Some time into debugging and repeating the benchmark again and >> again, I get the "Pure Lisp storage overflowed" message. Just once >> per Emacs session. It doesn't seem to change much, so it might be >> unimportant. That sounds like 60653. The next time you encounter it, could you record the output of M-x memory-usage and M-x memory-report?=20 >> - The profiler output looks like this: >> 18050 75% - >> font-lock-fontify-syntactically-region >> 15686 65% - treesit-font-lock-fontify-region >> 3738 15% treesit--children-covering-range-recurse >> 188 0% treesit-fontify-with-override >> - When running the benchmark for the first time in a buffer (such as >> ruby.rb), the variable treesit--font-lock-fast-mode is usually >> changed to t. In one Emacs session, after I changed it to nil and >> re-ran the benchmark, the variable stayed nil, and the benchmark ran >> much faster (like 10s vs 36s). >> In the next session, after I restarted Emacs, that didn't happen: it >> always stayed at t, even if I reset it to nil between runs. But if I >> comment out the block in treesit-font-lock-fontify-region that uses >> it >> ;; (when treesit--font-lock-fast-mode >> ;; (setq nodes (treesit--children-covering-range-recurse >> ;; (car nodes) start end (* 4 = jit-lock-chunk-size)))) >> and evaluate the defun, the benchmark runs much faster again: 11s. >> (But then I brought it all back, and re-ran the tests, and the >> variable stayed nil that time around; to sum up: the way it's turned >> on is unstable.) >> Should treesit--font-lock-fast-mode be locally bound inside that >> function, so that it's reset between chunks? Or maybe the condition >> for its enabling should be tweaked? E.g. I don't think there are any >> particularly large or deep nodes in ruby.rb's parse tree. It's a >> very shallow file. Yeah that is a not-very-clever hack. I=E2=80=99ve got an idea: I can add = a C function that checks the maximum depth of a parse tree and the maximum node span, and turn on the fast-mode if the depth is too large or a node is too wide. And we do that check once before doing any fontification. I=E2=80=99ll report back once I add it. Yuan From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 12 18:41:07 2023 Received: (at 60691) by debbugs.gnu.org; 12 Jan 2023 23:41:07 +0000 Received: from localhost ([127.0.0.1]:48861 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pG7Bz-0006rz-8u for submit@debbugs.gnu.org; Thu, 12 Jan 2023 18:41:07 -0500 Received: from mail-ej1-f47.google.com ([209.85.218.47]:43983) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pG7Bx-0006rO-7J for 60691@debbugs.gnu.org; Thu, 12 Jan 2023 18:41:06 -0500 Received: by mail-ej1-f47.google.com with SMTP id hw16so36701651ejc.10 for <60691@debbugs.gnu.org>; Thu, 12 Jan 2023 15:41:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=G9PWF+FHoTG5EJvb7lvyvAbPqIQaMs0RZYu1eiR7/s0=; b=aiQ7w+/uAGckaH2duHMDgKSvzwZ5C9XZ55W331sgtGM1UThoYa6nO+R2q0+KjQhjO3 Eq3a4zS3JFVf263Y+adKKDr6DCw8oBzlr2Qr+LIqqdLHhscy4HcCHNcrdfNVdhoG/OdK YgAZAJMmyDKldeKwQR4EjPrlMKHO7Ly4/YyvW0x26QoVRgtaDRslIaEgKxd7VwulmYq0 L5XoSBqC+IYeAT8ENhxe60k6kPXFr7Vq2DzNAPM1s/CA1Z0cozmdAVKEyu43+Qe5Jq8K 1kvHtN6WGGSA19yr14utPDPjCZqx9hYGSKltxQ6D4fWD7Z304NP4lncwdndteMXmoP9R 9G7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=G9PWF+FHoTG5EJvb7lvyvAbPqIQaMs0RZYu1eiR7/s0=; b=v6MRcz4GjazDZKa8sl3ihcD+GG/iBhmvSU5uK2HGBUCrsvvWZ6qPfr6ECSnwhGw6qr fF3+VXIDXLhXYVWAyVyhu+YvTcn2CSzxTFpoC1I6cqjllJCkGEnvM7M6gbtyoyz9bGGH ndQSRIg3HpyM8OrKnKe3QvxQOi2Z50l84N0m50kxxBJ3jkjowv3DcwAPZ7myMUZ8nrY6 rRUVgGHEtjZMMHQ3Zd6uR3EINsCzyw+duDFf984QrcEwFdySye6enqbfx4X90Byxm09U h8Bf5XvRGSBInXlXdYPnOHK91dsuov0i1CBQDco1nRSsqReylxNJciVR2D1hBM/ok0G8 PU7Q== X-Gm-Message-State: AFqh2kpiwdNAdhhXlidj6kaBEjUvCoDKo2FIE5WfohFrpF7MWWYSXE6o l1MzT6uQWQfAMBZnAmUlhBc= X-Google-Smtp-Source: AMrXdXsYW9Hxb41UNyatlRxxpijViRNQ6fbwiVaFzCthONJEMsK+HVANLJhLuB8SC1IWjVd6H1SKgQ== X-Received: by 2002:a17:907:a70b:b0:7c1:98d:a8a3 with SMTP id vw11-20020a170907a70b00b007c1098da8a3mr60271563ejc.7.1673566859131; Thu, 12 Jan 2023 15:40:59 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id d9-20020a1709063ec900b0084c2065b388sm7853479ejj.128.2023.01.12.15.40.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 12 Jan 2023 15:40:58 -0800 (PST) Message-ID: <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> Date: Fri, 13 Jan 2023 01:40:56 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Yuan Fu References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> From: Dmitry Gutov In-Reply-To: <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) On 12/01/2023 23:58, Yuan Fu wrote: > > Dmitry Gutov writes: > >> Yuan? Just making sure you got this message. > > Sorry for the delay :-) > >> On 10/01/2023 16:10, Dmitry Gutov wrote: >>> Perhaps Yuan has some further ideas. There are some strong oddities here: >>> - Some time into debugging and repeating the benchmark again and >>> again, I get the "Pure Lisp storage overflowed" message. Just once >>> per Emacs session. It doesn't seem to change much, so it might be >>> unimportant. > > That sounds like 60653. The next time you encounter it, could you record > the output of M-x memory-usage and M-x memory-report? Managed to reproduce this after running the test in a couple of different files. But 'M-x memory-usage' says no such command, and 'M-x memory-report' ends up with this error: Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil) memory-report--gc-elem(nil strings) memory-report--garbage-collect() memory-report() funcall-interactively(memory-report) #(memory-report record nil) apply(# memory-report (record nil)) call-interactively@ido-cr+-record-current-command(# memory-report record nil) apply(call-interactively@ido-cr+-record-current-command # (memory-report record nil)) call-interactively(memory-report record nil) command-execute(memory-report record) execute-extended-command(nil "memory-report" nil) funcall-interactively(execute-extended-command nil "memory-report" nil) #(execute-extended-command nil nil) apply(# execute-extended-command (nil nil)) call-interactively@ido-cr+-record-current-command(# execute-extended-command nil nil) apply(call-interactively@ido-cr+-record-current-command # (execute-extended-command nil nil)) call-interactively(execute-extended-command nil nil) command-execute(execute-extended-command) garbage-collect's docstring says: However, if there was overflow in pure space, and Emacs was dumped using the "unexec" method, ‘garbage-collect’ returns nil, because real GC can’t be done. I don't know if my Emacs was dumped using "unexec", though. ./configure says I'm using pdumper. In case that matters, I'm testing the emacs-29 branch. >>> - The profiler output looks like this: >>> 18050 75% - >>> font-lock-fontify-syntactically-region >>> 15686 65% - treesit-font-lock-fontify-region >>> 3738 15% treesit--children-covering-range-recurse >>> 188 0% treesit-fontify-with-override >>> - When running the benchmark for the first time in a buffer (such as >>> ruby.rb), the variable treesit--font-lock-fast-mode is usually >>> changed to t. In one Emacs session, after I changed it to nil and >>> re-ran the benchmark, the variable stayed nil, and the benchmark ran >>> much faster (like 10s vs 36s). >>> In the next session, after I restarted Emacs, that didn't happen: it >>> always stayed at t, even if I reset it to nil between runs. But if I >>> comment out the block in treesit-font-lock-fontify-region that uses >>> it >>> ;; (when treesit--font-lock-fast-mode >>> ;; (setq nodes (treesit--children-covering-range-recurse >>> ;; (car nodes) start end (* 4 jit-lock-chunk-size)))) >>> and evaluate the defun, the benchmark runs much faster again: 11s. >>> (But then I brought it all back, and re-ran the tests, and the >>> variable stayed nil that time around; to sum up: the way it's turned >>> on is unstable.) >>> Should treesit--font-lock-fast-mode be locally bound inside that >>> function, so that it's reset between chunks? Or maybe the condition >>> for its enabling should be tweaked? E.g. I don't think there are any >>> particularly large or deep nodes in ruby.rb's parse tree. It's a >>> very shallow file. > > Yeah that is a not-very-clever hack. I’ve got an idea: I can add a C > function that checks the maximum depth of a parse tree and the maximum > node span, and turn on the fast-mode if the depth is too large or a node > is too wide. And we do that check once before doing any fontification. > > I’ll report back once I add it. Thanks! And if the check can be fast enough, we could probably do it in the beginning of fontifying every chunk. From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 13 02:58:11 2023 Received: (at 60691) by debbugs.gnu.org; 13 Jan 2023 07:58:11 +0000 Received: from localhost ([127.0.0.1]:49444 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGEx1-0000bJ-52 for submit@debbugs.gnu.org; Fri, 13 Jan 2023 02:58:11 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56654) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGEwz-0000b7-95 for 60691@debbugs.gnu.org; Fri, 13 Jan 2023 02:58:09 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pGEwl-00043c-Ue; Fri, 13 Jan 2023 02:58:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=7SqaeqM7TefYeYdaQoiAgZ8yRV2f2LhqjeZu/WeQesA=; b=TAYP8vHh4M1fFMkMZ+AJ /W4Yij1VZlOeA29vsX13ksDTIHjcpvpNMxbhg1b0vv57CRo+gx/j8WPcnY44pKYTjifyNAcfql71r OYHFJnKlmKNWbbp44gR6K/oJX4vNaogJ6gEBKek8YmCykOi279FIX1m1xgSnKMzz2w0fxaGOUc3bR XhbQRDK/XWQ0jg15D6NcPhO3sEMhliQoPFQ4nT6YxwnIQFRV0mUnOMXfU7sbn884VVdOe8dugBeZS W93u8ktK989JbDseVn7YN1YACG18soPus+wjp9l04halMOq932XHMVXMyleJfnf/GMSCh5s8CVh+Q 0lRSjgDqxXEE0g==; Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pGEwk-0008Bi-4R; Fri, 13 Jan 2023 02:57:55 -0500 Date: Fri, 13 Jan 2023 09:57:52 +0200 Message-Id: <83r0vyamtb.fsf@gnu.org> From: Eli Zaretskii To: casouri@gmail.com, Dmitry Gutov In-Reply-To: <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> (message from Dmitry Gutov on Fri, 13 Jan 2023 01:40:56 +0200) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, Stefan Monnier , juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > Cc: 60691@debbugs.gnu.org, juri@linkov.net > Date: Fri, 13 Jan 2023 01:40:56 +0200 > From: Dmitry Gutov > > Managed to reproduce this after running the test in a couple of > different files. > > But 'M-x memory-usage' says no such command, and 'M-x memory-report' > ends up with this error: > > Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil) > memory-report--gc-elem(nil strings) > memory-report--garbage-collect() > memory-report() This means GC is disabled in this session at the time you invoke memory-report. Which shouldn't happen, of course. It sounds like your pure Lisp storage overflowed, and that disabled GC. And I think I see the problem: we use build_pure_c_string in treesit.c in places that we shouldn't. Yuan, build_pure_c_string should only be used in places such as syms_of_treesit, which are called just once, during dumping. Look at all the other calls to this function in the sources, and you will see it. In all other cases, you should do one of the following: . for strings whose text is fixed, define a variable, give it the value in syms_of_treesit using build_pure_c_string, then use that variable elsewhere in the source . for strings whose text depends on run-time information, use AUTO_STRING or build_string This is a serious problem, and we should fix it ASAP. > garbage-collect's docstring says: > > However, if there was overflow in pure space, and Emacs was dumped > using the "unexec" method, ‘garbage-collect’ returns nil, because > real GC can’t be done. > > I don't know if my Emacs was dumped using "unexec", though. ./configure > says I'm using pdumper. The above text doesn't account for bugs ;-) Functions that produce objects in pure space are supposed to be called only during the build, a.k.a. "when dumping", and for that the text is correct. From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 13 04:15:22 2023 Received: (at 60691) by debbugs.gnu.org; 13 Jan 2023 09:15:22 +0000 Received: from localhost ([127.0.0.1]:49570 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGG9i-0004xB-6E for submit@debbugs.gnu.org; Fri, 13 Jan 2023 04:15:22 -0500 Received: from mail-pl1-f181.google.com ([209.85.214.181]:46730) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGG9g-0004wv-Db for 60691@debbugs.gnu.org; Fri, 13 Jan 2023 04:15:21 -0500 Received: by mail-pl1-f181.google.com with SMTP id jn22so22830061plb.13 for <60691@debbugs.gnu.org>; Fri, 13 Jan 2023 01:15:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=grdT2JPcyMNP8d/LqeBAZEhUtyUNSyPdL5gl1vq0JwQ=; b=bThSs0rp4hQV/OnYaq7tmwI6hjfM+5E/b3PJpDr3WBhRNi/viUR+Q4kfYQKWIjIwyU EItgHVlGzCZKVgU3E8bl9Npl7/WlaFEUQOsGmcr9HiAWY6n4JqTZg5VilQ23esOxXmBZ grBJW+kgWJSmEuhDW3BNDD452UiGwqev2PZkmpC33l7UCkBKVBWvJvBvovtYYM49xi2s Qwka22Z2s7nMP4YZQEeK356+o15ydGsPtVZ6IADbbFs/Sc3ABUor+vGmtXEBJ7AnO61d Yc2jDxqXwSQEagkdhbEtIV86tP7aV/XW6TNsPt8AAYphLwWI7kUN2SpPHkQLJiMgiqIK VHKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=grdT2JPcyMNP8d/LqeBAZEhUtyUNSyPdL5gl1vq0JwQ=; b=rmzSb2gpRZm3SqfJcWcHRTPdrznlxr7SQMx+CJTFLMk6TBMYy6G0QYdTCvXKmR9hcQ TRbm/H9j6U7PTOazcjznFSweonydJYtRgEVsr9SHsIgCbsi+wcALT59f4DFimEQFPXhO TnXWkOqZ914Rr8MqX74ds6Q9eJ8ESSfFubtaOi63s5+PAlZfo1jMiBwGRC5S47R9ulHR PYYwtXwBdNBd3fP7NVxwqINj+Bhymb0epgjkWztERcx8v86SKB03ecXnSDYyCYBqNqWs XaQrmXbV/vdMn7wG1WvnKLZ30emlIqhGYFSZtU188ym1M1bwntP3ZUPa2mt2snzaGlXY 4F/w== X-Gm-Message-State: AFqh2kof7hcb5GlE+bYYSxi4pBFALRz0HeOp6pFS6C+qoB9p7VJCJtGx KsrDtAVhY7YoeEuRTBHHzr8= X-Google-Smtp-Source: AMrXdXs491aEfVqGOkzMleIhIQp8QY8tXpp+MsNSsDX8VGQ815yIxeLK+jJIW3RfH8xVZHwrN6axhw== X-Received: by 2002:a17:902:edc5:b0:192:c882:703e with SMTP id q5-20020a170902edc500b00192c882703emr39929936plk.43.1673601314502; Fri, 13 Jan 2023 01:15:14 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id u11-20020a6540cb000000b0046ff3634a78sm11320328pgp.71.2023.01.13.01.15.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Jan 2023 01:15:14 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode From: Yuan Fu In-Reply-To: <83r0vyamtb.fsf@gnu.org> Date: Fri, 13 Jan 2023 01:15:09 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> <83r0vyamtb.fsf@gnu.org> To: Eli Zaretskii X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 60691 Cc: Juri Linkov , 60691@debbugs.gnu.org, Stefan Monnier , Dmitry Gutov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) > On Jan 12, 2023, at 11:57 PM, Eli Zaretskii wrote: >=20 >> Cc: 60691@debbugs.gnu.org, juri@linkov.net >> Date: Fri, 13 Jan 2023 01:40:56 +0200 >> From: Dmitry Gutov >>=20 >> Managed to reproduce this after running the test in a couple of=20 >> different files. >>=20 >> But 'M-x memory-usage' says no such command, and 'M-x memory-report'=20= >> ends up with this error: >>=20 >> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p = nil) >> memory-report--gc-elem(nil strings) >> memory-report--garbage-collect() >> memory-report() >=20 > This means GC is disabled in this session at the time you invoke > memory-report. Which shouldn't happen, of course. It sounds like > your pure Lisp storage overflowed, and that disabled GC. >=20 > And I think I see the problem: we use build_pure_c_string in treesit.c > in places that we shouldn't. >=20 > Yuan, build_pure_c_string should only be used in places such as > syms_of_treesit, which are called just once, during dumping. Look at > all the other calls to this function in the sources, and you will see > it. In all other cases, you should do one of the following: >=20 > . for strings whose text is fixed, define a variable, give it the > value in syms_of_treesit using build_pure_c_string, then use that > variable elsewhere in the source Can I define a bunch of static C variables and initialize them in = syms_of_treesit, or they have to be all Lisp variables? Eg, static Lisp_Object TREESIT_STAR; ... void syms_of_treesit (void) { ... TREESIT_STAR =3D build_pure_c_string ("*"); ... } Yuan= From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 13 06:51:27 2023 Received: (at 60691) by debbugs.gnu.org; 13 Jan 2023 11:51:27 +0000 Received: from localhost ([127.0.0.1]:49768 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGIal-0000kU-4m for submit@debbugs.gnu.org; Fri, 13 Jan 2023 06:51:27 -0500 Received: from eggs.gnu.org ([209.51.188.92]:44674) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGIaj-0000kI-93 for 60691@debbugs.gnu.org; Fri, 13 Jan 2023 06:51:25 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pGIab-0004gp-Of; Fri, 13 Jan 2023 06:51:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=v8L5KErd7/OHhGFb/Qc5qHRtsMnHLsT41EQx85iBh5k=; b=lt3mc3XnkpMi wKgin12cfjd39wdkwFMBf7/au6rEBHLG6Z2GNLyrBxIbnFe+TLAE6iMNuJ0VIXMEeAoljP6oX0cio KS8iIebJl1fSH9Fyd2ze698oCq85Fg2NzCQNEWKQxuJ6jv6ulKQKKcVy4WRks7Pr/Sa0om6kqKP61 y7Pw8RSU1qkUTwezMoSUkQ9KOG7pJSRIpFUTfRBXcac0caVHsI+VBhIorKFlAFT5KzEY7ROnLb6SA 0i8l9NKpyfUFtIjI1GgIgCa6frZ7TJMXPPP2UPB9WNJeJY3cdnK4NbNnppMTZuWBLscWzfK36txLC EWQqAJPDl98sULV/BPzd6Q==; Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pGIaa-0007gG-Vg; Fri, 13 Jan 2023 06:51:17 -0500 Date: Fri, 13 Jan 2023 13:51:15 +0200 Message-Id: <83mt6mac0c.fsf@gnu.org> From: Eli Zaretskii To: Yuan Fu In-Reply-To: (message from Yuan Fu on Fri, 13 Jan 2023 01:15:09 -0800) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> <83r0vyamtb.fsf@gnu.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 60691 Cc: juri@linkov.net, 60691@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Yuan Fu > Date: Fri, 13 Jan 2023 01:15:09 -0800 > Cc: Dmitry Gutov , > 60691@debbugs.gnu.org, > Juri Linkov , > Stefan Monnier > > > On Jan 12, 2023, at 11:57 PM, Eli Zaretskii wrote: > > > >> Cc: 60691@debbugs.gnu.org, juri@linkov.net > >> Date: Fri, 13 Jan 2023 01:40:56 +0200 > >> From: Dmitry Gutov > >> > >> Managed to reproduce this after running the test in a couple of > >> different files. > >> > >> But 'M-x memory-usage' says no such command, and 'M-x memory-report' > >> ends up with this error: > >> > >> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil) > >> memory-report--gc-elem(nil strings) > >> memory-report--garbage-collect() > >> memory-report() > > > > This means GC is disabled in this session at the time you invoke > > memory-report. Which shouldn't happen, of course. It sounds like > > your pure Lisp storage overflowed, and that disabled GC. > > > > And I think I see the problem: we use build_pure_c_string in treesit.c > > in places that we shouldn't. > > > > Yuan, build_pure_c_string should only be used in places such as > > syms_of_treesit, which are called just once, during dumping. Look at > > all the other calls to this function in the sources, and you will see > > it. In all other cases, you should do one of the following: > > > > . for strings whose text is fixed, define a variable, give it the > > value in syms_of_treesit using build_pure_c_string, then use that > > variable elsewhere in the source > > Can I define a bunch of static C variables and initialize them in syms_of_treesit, or they have to be all Lisp variables? Eg, > > static Lisp_Object TREESIT_STAR; > > ... > > void > syms_of_treesit (void) > { > ... > TREESIT_STAR = build_pure_c_string ("*"); > ... > } Yes, of course. Look, for example, how coding.c does that: /* A string that serves as name of the reusable work buffer, and as base name of temporary work buffers used for code-conversion operations. */ static Lisp_Object Vcode_conversion_workbuf_name; [...] void syms_of_coding (void) { [...] staticpro (&Vcode_conversion_workbuf_name); Vcode_conversion_workbuf_name = build_pure_c_string (" *code-conversion-work*"); But please keep the convention of naming such variables Vsome_thing, both regarding the "V" and the fact that the name is otherwise lower-case. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 13 22:48:50 2023 Received: (at 60691) by debbugs.gnu.org; 14 Jan 2023 03:48:50 +0000 Received: from localhost ([127.0.0.1]:52926 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGXXG-000886-86 for submit@debbugs.gnu.org; Fri, 13 Jan 2023 22:48:50 -0500 Received: from mail-pj1-f54.google.com ([209.85.216.54]:37558) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGXXE-00087t-1X for 60691@debbugs.gnu.org; Fri, 13 Jan 2023 22:48:49 -0500 Received: by mail-pj1-f54.google.com with SMTP id o1-20020a17090a678100b00219cf69e5f0so28848528pjj.2 for <60691@debbugs.gnu.org>; Fri, 13 Jan 2023 19:48:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=mJwoFOhxZip+RMp6eVcrydfjakDfE+zcJXQfnL9xL+g=; b=bllNYqR/56URhI50lwJTXe6eKVKS2ElYzc5rrfDx2vGMMTLe1n/fiiitn9X0FvvaLR 7K1Ybw+pmxe8AK0fUlz43OHNIl7zvHb6aBqmSBAWnLTxcldLkw8RTkfjgnoYG9odVpgW Mklkr+98W0lZbNpmSuatBVZotqPdcLqXWkqs2PnOJsYCVp5p5JykGa1tlHMftagCazZ9 QYeKAC5ze4pQsZCWzA0PEwEEwqK+NMeCG6RALFC7yboItpJICrP3XBi3/XovSRN8oaxu 9RJtIRzMCCSftQ2lgiajhpmL55SIhJwyAuO/xJ+sSqbYG9zmNcfBi5jMElESxcWVbvVK FHog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mJwoFOhxZip+RMp6eVcrydfjakDfE+zcJXQfnL9xL+g=; b=yJV44p43g5WwanplNpYTdEAKLSJHGeiiuEFxnfTqZlDIyrsC1vTUaK4IeR8HVsuiHN YVY4HVB4vsX438u8cvc4V0wWYrWRxumC457wMZsxXO+yQVTQ8Y4IcPN9AJQEkmfNWwQN tc1kDjBceafkD2WDIlaMP84egRRTMxFuRIxEtOcjKJU9REwX8HP7s4jI70wnDaxzCrNm TATQ+ykh3UrjKBHPmFYu/VFRrt0BZah0wj8vADwxvuJ8MOIgGbhiqPm40aOyPfrarIDK l2n1ninYLqCcrJe/g9OXSASVFgYbCfjOjGF4EztLzmknJDLYIYLnruf/PQlNcq18qz1c P79Q== X-Gm-Message-State: AFqh2konV5y19veQUnVCYRVFF32Ov0Awc4EQYJ29nLwtRN+k7kCBU/rb o+s5PghJJgsmEmIcFKjyK5g= X-Google-Smtp-Source: AMrXdXtpSk23kJnzTR+bw/wfRcdIt8T3jDpdSKU5es3l/FqspbfkVSE3CUGlUYNrbDd1t0kotkb7bA== X-Received: by 2002:a17:902:6a87:b0:194:6d39:5911 with SMTP id n7-20020a1709026a8700b001946d395911mr4941697plk.40.1673668122011; Fri, 13 Jan 2023 19:48:42 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id h18-20020a170902f55200b001929dff50a9sm14950031plf.87.2023.01.13.19.48.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Jan 2023 19:48:41 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode From: Yuan Fu In-Reply-To: <83mt6mac0c.fsf@gnu.org> Date: Fri, 13 Jan 2023 19:48:40 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <3FBD8C40-2C2B-4B10-BDB3-8CDD93B495A9@gmail.com> References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> <83r0vyamtb.fsf@gnu.org> <83mt6mac0c.fsf@gnu.org> To: Eli Zaretskii X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 60691 Cc: juri@linkov.net, 60691@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) > On Jan 13, 2023, at 3:51 AM, Eli Zaretskii wrote: >=20 >> From: Yuan Fu >> Date: Fri, 13 Jan 2023 01:15:09 -0800 >> Cc: Dmitry Gutov , >> 60691@debbugs.gnu.org, >> Juri Linkov , >> Stefan Monnier >>=20 >>> On Jan 12, 2023, at 11:57 PM, Eli Zaretskii wrote: >>>=20 >>>> Cc: 60691@debbugs.gnu.org, juri@linkov.net >>>> Date: Fri, 13 Jan 2023 01:40:56 +0200 >>>> From: Dmitry Gutov >>>>=20 >>>> Managed to reproduce this after running the test in a couple of=20 >>>> different files. >>>>=20 >>>> But 'M-x memory-usage' says no such command, and 'M-x = memory-report'=20 >>>> ends up with this error: >>>>=20 >>>> Debugger entered--Lisp error: (wrong-type-argument = number-or-marker-p nil) >>>> memory-report--gc-elem(nil strings) >>>> memory-report--garbage-collect() >>>> memory-report() >>>=20 >>> This means GC is disabled in this session at the time you invoke >>> memory-report. Which shouldn't happen, of course. It sounds like >>> your pure Lisp storage overflowed, and that disabled GC. >>>=20 >>> And I think I see the problem: we use build_pure_c_string in = treesit.c >>> in places that we shouldn't. >>>=20 >>> Yuan, build_pure_c_string should only be used in places such as >>> syms_of_treesit, which are called just once, during dumping. Look = at >>> all the other calls to this function in the sources, and you will = see >>> it. In all other cases, you should do one of the following: >>>=20 >>> . for strings whose text is fixed, define a variable, give it the >>> value in syms_of_treesit using build_pure_c_string, then use that >>> variable elsewhere in the source >>=20 >> Can I define a bunch of static C variables and initialize them in = syms_of_treesit, or they have to be all Lisp variables? Eg, >>=20 >> static Lisp_Object TREESIT_STAR; >>=20 >> ... >>=20 >> void >> syms_of_treesit (void) >> { >> ... >> TREESIT_STAR =3D build_pure_c_string ("*"); >> ... >> } >=20 > Yes, of course. Look, for example, how coding.c does that: >=20 > /* A string that serves as name of the reusable work buffer, and as = base > name of temporary work buffers used for code-conversion = operations. */ > static Lisp_Object Vcode_conversion_workbuf_name; > [...] > void > syms_of_coding (void) > { > [...] > staticpro (&Vcode_conversion_workbuf_name); > Vcode_conversion_workbuf_name =3D build_pure_c_string (" = *code-conversion-work*"); >=20 > But please keep the convention of naming such variables Vsome_thing, > both regarding the "V" and the fact that the name is otherwise > lower-case. Thanks, I pushed a fix for it. I also used intern_c_string in some = places like these: intern_c_string (":?=E2=80=9D) intern_c_string (":*") I want to change them to use DEFSYM, but what should be the c name for = them? Yuan From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 14 02:29:15 2023 Received: (at 60691) by debbugs.gnu.org; 14 Jan 2023 07:29:16 +0000 Received: from localhost ([127.0.0.1]:53121 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGayZ-00088D-LB for submit@debbugs.gnu.org; Sat, 14 Jan 2023 02:29:15 -0500 Received: from eggs.gnu.org ([209.51.188.92]:59966) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGayX-000881-IC for 60691@debbugs.gnu.org; Sat, 14 Jan 2023 02:29:13 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pGayQ-0006yC-WA; Sat, 14 Jan 2023 02:29:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=1AlmdwoHsiVSqHOb9EH33puDmiL/W0ey+HQq6y/Jpyg=; b=j+2vNrHy0Fkk+1nIOnWg fIO+/66MG703M3UWCwwpdTNHg8gU0HPS5qWmQYMHak4f3zg8R1nhnAYR25f6Zs3ttvjunXpbVyJWN iqC8JyHmrnVxasay3lftfOpi0TYnWbqs/LdP0vAiAI/qZV0eRrT709sSEptU+rYE0j8gpDVCGMHdJ nSdgydJK8i2AQq3aEZUIpSz9cweapAdT5h+s60UpACvrilvpAU7jBWwLikfygW830QNqpMNEQP5mm SjDdr/Nc/yinR/v1roLiwGAFlHWInONZu/kY9WoPG2ANOh8HEuLVNUy78zPwpUF7KmywVljcSOObE nF2mqARb2J2cog==; Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pGayQ-0004QO-7m; Sat, 14 Jan 2023 02:29:06 -0500 Date: Sat, 14 Jan 2023 09:29:08 +0200 Message-Id: <83mt6l8th7.fsf@gnu.org> From: Eli Zaretskii To: Yuan Fu In-Reply-To: <3FBD8C40-2C2B-4B10-BDB3-8CDD93B495A9@gmail.com> (message from Yuan Fu on Fri, 13 Jan 2023 19:48:40 -0800) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> <83r0vyamtb.fsf@gnu.org> <83mt6mac0c.fsf@gnu.org> <3FBD8C40-2C2B-4B10-BDB3-8CDD93B495A9@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 60691 Cc: juri@linkov.net, 60691@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Yuan Fu > Date: Fri, 13 Jan 2023 19:48:40 -0800 > Cc: dgutov@yandex.ru, > 60691@debbugs.gnu.org, > juri@linkov.net, > monnier@iro.umontreal.ca > > Thanks, I pushed a fix for it. I also used intern_c_string in some places like these: > > intern_c_string (":?”) > intern_c_string (":*") > > I want to change them to use DEFSYM, but what should be the c name for them? Yes, DEFSYM is better in such cases. The C name can be QCquestion and QCasterix, for example. From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 14 02:51:14 2023 Received: (at 60691) by debbugs.gnu.org; 14 Jan 2023 07:51:15 +0000 Received: from localhost ([127.0.0.1]:53181 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGbJq-0000OC-Ee for submit@debbugs.gnu.org; Sat, 14 Jan 2023 02:51:14 -0500 Received: from mail-pj1-f50.google.com ([209.85.216.50]:53092) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGbJo-0000Nx-U6 for 60691@debbugs.gnu.org; Sat, 14 Jan 2023 02:51:13 -0500 Received: by mail-pj1-f50.google.com with SMTP id o13so21047464pjg.2 for <60691@debbugs.gnu.org>; Fri, 13 Jan 2023 23:51:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=tlwGvZ6qBr3cwq3HB43bzz+uAXUoxC54NqVO7H8qBDQ=; b=gC6QEerdtly6ThSbVLgPj300mYvlo6rNAJNTIZRaZFla71m1OQYvcjcE2c72MZnCKT Dx4QOLjT5Gdd8nyxVCHXbQcMIoUd+WyFUEKbIMdMiNf0X7+w9AEVLoH8kxDOODlez35o tmFkFD3MaP5Im9Q+ImN6qVNv0s2rq/V9e1ojbHDV9zG4gPCF+i2XoPrDsP27kYJkojT7 wY5QulssJfCZaKZ39tXHPb2M9TX06E2A7Cqfa73RASTtcfPDi60cDfKRrkxkp+YLaBk/ daS050u17I9jv4HH6qIK50a8YwP4uWkk51rAscL9fNI5wrqWAMJmnzFZZllbu7q5osm5 oJTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tlwGvZ6qBr3cwq3HB43bzz+uAXUoxC54NqVO7H8qBDQ=; b=e3eq3VFAX/kJA6DW5xX+2VZWV2xq1oums5m3jwcz0yuzsfKa67Sa8cUQomGSoDl08/ VAEkzX2JyqLNuhFoCE/l1jvCmoO6Aup4gRISvx8O0flyeSZ6eQTKPDVjco32ZVOHPqdg nVuYrJiW8zbSIkZp4fjvwyOrc/sp7gOydFhIqNsJ9uv+w4PZRVvh5ecfKkzIr57iu9Km T3zTQIpP5KLhIbWmIl71/BI6aRQai9esZp2U984LP5wYUFlIv7Bq2LIVxLbHMfxyraqg u8RwKIhunHMi1OU8fAlT8UvOfD2uzjnrSvlyQ2WVlB0oqqKlw4unXd+G11ynQ4LI/+65 xPAw== X-Gm-Message-State: AFqh2kquch3cPEwJYz9QuUa5pb8c2Z05GGzmr44kRamZ9mBO74zIDMF0 ZZBIiszlqx5+5O3f5ChvQ1c= X-Google-Smtp-Source: AMrXdXsOaG8BsiuvSK7mo9yUGxzMSefoZmlyNv0xOcOUYTk+r7C+6fXbRfZ7QGv18ksB0NvXI6Fvxw== X-Received: by 2002:a17:903:2c5:b0:192:cf35:3ff8 with SMTP id s5-20020a17090302c500b00192cf353ff8mr51947643plk.21.1673682666821; Fri, 13 Jan 2023 23:51:06 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id f5-20020a170902ce8500b001870dc3b4c0sm15428772plg.74.2023.01.13.23.51.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Jan 2023 23:51:06 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode From: Yuan Fu In-Reply-To: <83mt6l8th7.fsf@gnu.org> Date: Fri, 13 Jan 2023 23:51:05 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <09B9914D-2B6D-4F73-89E7-1B9A02A1C44E@gmail.com> References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> <83r0vyamtb.fsf@gnu.org> <83mt6mac0c.fsf@gnu.org> <3FBD8C40-2C2B-4B10-BDB3-8CDD93B495A9@gmail.com> <83mt6l8th7.fsf@gnu.org> To: Eli Zaretskii X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 60691 Cc: juri@linkov.net, 60691@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) > On Jan 13, 2023, at 11:29 PM, Eli Zaretskii wrote: >=20 >> From: Yuan Fu >> Date: Fri, 13 Jan 2023 19:48:40 -0800 >> Cc: dgutov@yandex.ru, >> 60691@debbugs.gnu.org, >> juri@linkov.net, >> monnier@iro.umontreal.ca >>=20 >> Thanks, I pushed a fix for it. I also used intern_c_string in some = places like these: >>=20 >> intern_c_string (":?=E2=80=9D) >> intern_c_string (":*") >>=20 >> I want to change them to use DEFSYM, but what should be the c name = for them? >=20 > Yes, DEFSYM is better in such cases. The C name can be QCquestion and > QCasterix, for example. My worry is that they will conflict with, eg, symbol `question=E2=80=99 = and `asterix=E2=80=99, if someone ever defines them in the C codebase. = Is that not possible? Yuan= From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 14 03:01:21 2023 Received: (at 60691) by debbugs.gnu.org; 14 Jan 2023 08:01:21 +0000 Received: from localhost ([127.0.0.1]:53204 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGbTc-00030K-VT for submit@debbugs.gnu.org; Sat, 14 Jan 2023 03:01:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41390) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGbTa-000304-Cj for 60691@debbugs.gnu.org; Sat, 14 Jan 2023 03:01:18 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pGbTT-0003jy-QC; Sat, 14 Jan 2023 03:01:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=N9UwvxEadXRzsCXGqNPimCSEfr0rK7JFM3H0qgefWYk=; b=TM/GOCnBZMbdWyfhaj1X p7FkYOu2vTK6+9DK6qwUTHoMWD+h6jbwoEOxWDwp76uGPMbk5tG5H7ti7hme0BbGujWHQUxCKNHSZ 6ikMEk6MKrMegmn50bUqjWHbD1Dr6R1NGDYf2Kjq27PevdsmzKigDd2OJ/Awz1ocxExtzGnMMKLGy Zyx3YDApFiMFkZb6rtb0NtwkhBgZwMgXIOo4tferrjOv9TjsXjeIoFfP8xzaeobBToDDlOkmuyNbv 3WSPjtdYt63OBD4HWbu8JbCO45MuVAx5Wtw4S3VkMp3ZeL8Cf5nAEtG3iB+HzzCTrPUhLtvFbgImW +Wk+3TGIZlsMYA==; Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pGbTS-0003Fv-Ci; Sat, 14 Jan 2023 03:01:10 -0500 Date: Sat, 14 Jan 2023 10:01:12 +0200 Message-Id: <83fscd8rzr.fsf@gnu.org> From: Eli Zaretskii To: Yuan Fu In-Reply-To: <09B9914D-2B6D-4F73-89E7-1B9A02A1C44E@gmail.com> (message from Yuan Fu on Fri, 13 Jan 2023 23:51:05 -0800) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> <83r0vyamtb.fsf@gnu.org> <83mt6mac0c.fsf@gnu.org> <3FBD8C40-2C2B-4B10-BDB3-8CDD93B495A9@gmail.com> <83mt6l8th7.fsf@gnu.org> <09B9914D-2B6D-4F73-89E7-1B9A02A1C44E@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 60691 Cc: juri@linkov.net, 60691@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Yuan Fu > Date: Fri, 13 Jan 2023 23:51:05 -0800 > Cc: dgutov@yandex.ru, > 60691@debbugs.gnu.org, > juri@linkov.net, > monnier@iro.umontreal.ca > > > Yes, DEFSYM is better in such cases. The C name can be QCquestion and > > QCasterix, for example. > > My worry is that they will conflict with, eg, symbol `question’ and `asterix’, if someone ever defines them in the C codebase. Is that not possible? It's possible, but how can a symbol conflict in that case? it will just be reused. But if you want to have treesit-specific symbols, you can use names like QCasterix_treesit. From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 14 03:46:51 2023 Received: (at 60691) by debbugs.gnu.org; 14 Jan 2023 08:46:51 +0000 Received: from localhost ([127.0.0.1]:53315 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGcBf-0004B2-9H for submit@debbugs.gnu.org; Sat, 14 Jan 2023 03:46:51 -0500 Received: from mail-out.m-online.net ([212.18.0.9]:48654) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGcBd-0004As-9b for 60691@debbugs.gnu.org; Sat, 14 Jan 2023 03:46:50 -0500 Received: from frontend03.mail.m-online.net (unknown [192.168.6.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4NvBjM1Z3qz1r0PY; Sat, 14 Jan 2023 09:46:46 +0100 (CET) Received: from localhost (dynscan3.mnet-online.de [192.168.6.84]) by mail.m-online.net (Postfix) with ESMTP id 4NvBjL63pWz1qqlR; Sat, 14 Jan 2023 09:46:46 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan3.mail.m-online.net [192.168.6.84]) (amavisd-new, port 10024) with ESMTP id Mo-eJJuFWYka; Sat, 14 Jan 2023 09:46:45 +0100 (CET) X-Auth-Info: jteDJ3Xj92DxZ3+cUV4ADUq73qBcgcStHT1b3AWy9zsAPU8Yri0FOPq9MQxfAnqQ Received: from tiger.home (aftr-62-216-205-42.dynamic.mnet-online.de [62.216.205.42]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Sat, 14 Jan 2023 09:46:45 +0100 (CET) Received: by tiger.home (Postfix, from userid 1000) id 44C2B158951; Sat, 14 Jan 2023 09:46:45 +0100 (CET) From: Andreas Schwab To: Yuan Fu Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> <83r0vyamtb.fsf@gnu.org> <83mt6mac0c.fsf@gnu.org> <3FBD8C40-2C2B-4B10-BDB3-8CDD93B495A9@gmail.com> <83mt6l8th7.fsf@gnu.org> <09B9914D-2B6D-4F73-89E7-1B9A02A1C44E@gmail.com> X-Yow: Yow! Am I JOGGING yet?? Date: Sat, 14 Jan 2023 09:46:45 +0100 In-Reply-To: <09B9914D-2B6D-4F73-89E7-1B9A02A1C44E@gmail.com> (Yuan Fu's message of "Fri, 13 Jan 2023 23:51:05 -0800") Message-ID: <87tu0t5wqy.fsf@linux-m68k.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.4 (/) X-Debbugs-Envelope-To: 60691 Cc: Eli Zaretskii , dgutov@yandex.ru, 60691@debbugs.gnu.org, monnier@iro.umontreal.ca, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.4 (-) On Jan 13 2023, Yuan Fu wrote: >> On Jan 13, 2023, at 11:29 PM, Eli Zaretskii wrote: >> >>> From: Yuan Fu >>> Date: Fri, 13 Jan 2023 19:48:40 -0800 >>> Cc: dgutov@yandex.ru, >>> 60691@debbugs.gnu.org, >>> juri@linkov.net, >>> monnier@iro.umontreal.ca >>> >>> Thanks, I pushed a fix for it. I also used intern_c_string in some places like these: >>> >>> intern_c_string (":?”) >>> intern_c_string (":*") >>> >>> I want to change them to use DEFSYM, but what should be the c name for them? >> >> Yes, DEFSYM is better in such cases. The C name can be QCquestion and >> QCasterix, for example. > > My worry is that they will conflict with, eg, symbol `question’ and `asterix’, if someone ever defines them in the C codebase. Is that not possible? The C name of the symbol `question' would be Qquestion, without the C (which stands for the `:' prefix). -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 14 18:04:11 2023 Received: (at 60691) by debbugs.gnu.org; 14 Jan 2023 23:04:11 +0000 Received: from localhost ([127.0.0.1]:55747 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGpZK-0004DR-K9 for submit@debbugs.gnu.org; Sat, 14 Jan 2023 18:04:10 -0500 Received: from mail-pl1-f177.google.com ([209.85.214.177]:41512) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pGpZG-0004Cw-Bo for 60691@debbugs.gnu.org; Sat, 14 Jan 2023 18:04:09 -0500 Received: by mail-pl1-f177.google.com with SMTP id jl4so26860901plb.8 for <60691@debbugs.gnu.org>; Sat, 14 Jan 2023 15:04:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=+Ree4OIIMcvzLK2NyLmPHBOsXxPT49S4KtNauvoGq7I=; b=pjH7MtqX3t8AoYJsooPxIHb/NpmPng0eUPcK+FnDkKXnvBrEfiI5lFfsnMfHSIWQ3P daqKr+y7pvQ/4owBzsVNCqiWaf4jF8o1WMsR0DyCwBA/c2fSd/FnkHyPGjqkn4JdI/Jj Cq2W6XLwS2vj7zrln30hDtZRG4bzP5VN9eXBXOCHYpxQOSkwDerapFuDZzvl7NhGfu7T YDz4n27zq+PKRZ0OBIgIRtFddRUw3QDMcF9ys9eZffu7mEcq8vbjXjvMTi5+Vh3bMWU8 itXMTqiK9Je3nyjfZ7I/NvVjlV5b/P2grvNcGugmaJwBhP4bKBm/9mt3O5Kx/r2GkmbV cSJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+Ree4OIIMcvzLK2NyLmPHBOsXxPT49S4KtNauvoGq7I=; b=GrDUFIummgTgjGrTAT+yVvgfDuQfFYKZDjZxtB4aQaEXeqEpRgVuZBZ+AlJl26SQ+8 sNa90leJr5O1EybfybXQPkV8qNGaqBWh7x1QFZu8425cGudMlCRyc/0/v1w3YwEGiEiO mIB+RSAVYyu1ZjfUHFkkpooKnAW4m78NCg377F+mSoEYfzJKAnVWfWTtPxkRC26zd0BY 9uw/MdTbrO0cCDS1nw3dPJF8u+uubd6AQs3aS6EGqhIUTVmXOWBcEKm5n9tq5rzU3POW kFgi14fTJ65lPIwBl3c6TDlgzjo9+h/aoprfkpI14FtQQTbxN8YPArT573bQdARzMFi4 DeSw== X-Gm-Message-State: AFqh2kqjTMApQBnBv3/nbmaczSDeHwoYZrCPqs+tBppsiclxqceQtwfJ yi5ku9nL6amCsqTMIC0fso8= X-Google-Smtp-Source: AMrXdXt95DSx08o1DXzaGernUbs6TwkNwFWh0utvyVslIeL9YdhsNkwzNpZl2dTkq9OsK0OYJ6RdvQ== X-Received: by 2002:a05:6a21:3394:b0:b8:5238:ea7 with SMTP id yy20-20020a056a21339400b000b852380ea7mr2476935pzb.7.1673737440498; Sat, 14 Jan 2023 15:04:00 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id z18-20020a63c052000000b0049f2c7e59f5sm13233413pgi.27.2023.01.14.15.03.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 Jan 2023 15:04:00 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode From: Yuan Fu In-Reply-To: <87tu0t5wqy.fsf@linux-m68k.org> Date: Sat, 14 Jan 2023 15:03:58 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <867cxv3dnn.fsf@mail.linkov.net> <6F1CC7E3-E5B2-4E51-93F6-455A2D0C771E@gmail.com> <0ba1ca9c-78e3-f961-787e-4758beaa3c5b@yandex.ru> <83r0vyamtb.fsf@gnu.org> <83mt6mac0c.fsf@gnu.org> <3FBD8C40-2C2B-4B10-BDB3-8CDD93B495A9@gmail.com> <83mt6l8th7.fsf@gnu.org> <09B9914D-2B6D-4F73-89E7-1B9A02A1C44E@gmail.com> <87tu0t5wqy.fsf@linux-m68k.org> To: Andreas Schwab X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 60691 Cc: Eli Zaretskii , dgutov@yandex.ru, 60691@debbugs.gnu.org, Stefan Monnier , Juri Linkov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) > On Jan 14, 2023, at 12:46 AM, Andreas Schwab = wrote: >=20 > On Jan 13 2023, Yuan Fu wrote: >=20 >>> On Jan 13, 2023, at 11:29 PM, Eli Zaretskii wrote: >>>=20 >>>> From: Yuan Fu >>>> Date: Fri, 13 Jan 2023 19:48:40 -0800 >>>> Cc: dgutov@yandex.ru, >>>> 60691@debbugs.gnu.org, >>>> juri@linkov.net, >>>> monnier@iro.umontreal.ca >>>>=20 >>>> Thanks, I pushed a fix for it. I also used intern_c_string in some = places like these: >>>>=20 >>>> intern_c_string (":?=E2=80=9D) >>>> intern_c_string (":*") >>>>=20 >>>> I want to change them to use DEFSYM, but what should be the c name = for them? >>>=20 >>> Yes, DEFSYM is better in such cases. The C name can be QCquestion = and >>> QCasterix, for example. >>=20 >> My worry is that they will conflict with, eg, symbol `question=E2=80=99= and `asterix=E2=80=99, if someone ever defines them in the C codebase. = Is that not possible? >=20 > The C name of the symbol `question' would be Qquestion, without the C > (which stands for the `:' prefix). Sorry, I meant `:question=E2=80=99, and `:asterix=E2=80=99.=20 Yuan= From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 18 01:50:25 2023 Received: (at 60691) by debbugs.gnu.org; 18 Jan 2023 06:50:25 +0000 Received: from localhost ([127.0.0.1]:38936 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pI2HA-0007Wj-TP for submit@debbugs.gnu.org; Wed, 18 Jan 2023 01:50:25 -0500 Received: from mail-pl1-f178.google.com ([209.85.214.178]:38528) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pI2H6-0007WU-UX for 60691@debbugs.gnu.org; Wed, 18 Jan 2023 01:50:23 -0500 Received: by mail-pl1-f178.google.com with SMTP id k18so11924056pll.5 for <60691@debbugs.gnu.org>; Tue, 17 Jan 2023 22:50:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:cc:date:message-id:subject:mime-version :content-transfer-encoding:from:from:to:cc:subject:date:message-id :reply-to; bh=lKP4z0KCgsW+4Ea/EbCIRO7pfNTLOpRxmqx/d8HG7SI=; b=I0uTlsn8XzHNZsh6HFJ22idUaLe16gqNL6rU9jH1BCkmPWpzv/J9AdasXqhxIaczF6 HKouOG1qu+xYteec7ITKRtvOqMrM4NgYgzCdoI7Xtu2A5WuYoqDFT1JC3DvcE9/Rt4r5 k8f2SXEjyXahh4I2aAFTx9OA0dVMI8Zq8tYRx6JkpdVvrKL1j0Rqo9e6U67mbs66fY3k SgUpXR/86MkuXrgsZz2+jwWZ9e3EbJ80tpssja4mF6u5niB17joPEzYo/SzqA9Dwr/lA p8ImqbtxxjC6ssNxtmoLyUCFYYJT5284/WXDkdJa4SrM/9ju/ZylIHSL7Sy+J0nBbQL5 KfLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:cc:date:message-id:subject:mime-version :content-transfer-encoding:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lKP4z0KCgsW+4Ea/EbCIRO7pfNTLOpRxmqx/d8HG7SI=; b=28IM3rfybCFFSt8QD24XjcNQaaWSDl7EJBl/VETMUf2xh4WVPp9mFc6BNKZUUl1wS6 zfZ7l3JxXi4DE6dV+exmV1Yz1qWkEggey7/+nylS1/hldKtk+C2j1jjYNGafX4Bw09mv pKGLPyAGUQlCnM9pzE5df54ZtjZgHPTwgofhTHcfVbiCjAzCSaMjIy+Sm/TbFyJN1GFI Tj0oa/5L0r+9u7G20ERHJ9J70HEUx91o9y0vDr5xjxO8QUajX9YuFbDWI5LM/voLksio /KVoFKnhPzcIn5HdDHBWUkcU+eDRwX0LYwbhfcDXWdVp2zY1L3tj7QP6ZVCrzdtzmmOE 6LMA== X-Gm-Message-State: AFqh2ko8E2ukYXMD55AMx4f4TB6eVdr76vOel2Pc6OUf+viXggD397kF TClBJRay91HJdl0oL+uufJU= X-Google-Smtp-Source: AMrXdXsHALJjHxZxIpApyWuupj+VBPEikvrJSyUHm/ngHMHYpvweMmZu1HtixKW/5DcxNI5JJjUMsA== X-Received: by 2002:a05:6a20:c213:b0:b8:c5b3:bd7 with SMTP id bt19-20020a056a20c21300b000b8c5b30bd7mr4972262pzb.59.1674024615005; Tue, 17 Jan 2023 22:50:15 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id l67-20020a622546000000b00581fddb7495sm7983905pfl.58.2023.01.17.22.50.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 17 Jan 2023 22:50:14 -0800 (PST) From: Yuan Fu Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Message-Id: Date: Tue, 17 Jan 2023 22:50:12 -0800 To: Dmitry Gutov X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Yuan Fu writes: > Dmitry Gutov writes: > >> Yuan? Just making sure you got this message. > > Sorry for the delay :-) > >> On 10/01/2023 16:10, Dmitry Gutov wrote: >>> Perhaps Yuan has some further ideas. There are some strong oddities = here: >>> - Some time into debugging and repeating the benchmark again and >>> again, I get the "Pure Lisp storage overflowed" message. Just once >>> per Emacs session. It doesn't seem to change much, so it might be >>> unimportant. > > That sounds like 60653. The next time you encounter it, could you = record > the output of M-x memory-usage and M-x memory-report?=20 > >>> - The profiler output looks like this: >>> 18050 75% - >>> font-lock-fontify-syntactically-region >>> 15686 65% - treesit-font-lock-fontify-region >>> 3738 15% treesit--children-covering-range-recurse >>> 188 0% treesit-fontify-with-override >>> - When running the benchmark for the first time in a buffer (such as >>> ruby.rb), the variable treesit--font-lock-fast-mode is usually >>> changed to t. In one Emacs session, after I changed it to nil and >>> re-ran the benchmark, the variable stayed nil, and the benchmark ran >>> much faster (like 10s vs 36s). >>> In the next session, after I restarted Emacs, that didn't happen: it >>> always stayed at t, even if I reset it to nil between runs. But if I >>> comment out the block in treesit-font-lock-fontify-region that uses >>> it >>> ;; (when treesit--font-lock-fast-mode >>> ;; (setq nodes (treesit--children-covering-range-recurse >>> ;; (car nodes) start end (* 4 = jit-lock-chunk-size)))) >>> and evaluate the defun, the benchmark runs much faster again: 11s. >>> (But then I brought it all back, and re-ran the tests, and the >>> variable stayed nil that time around; to sum up: the way it's turned >>> on is unstable.) >>> Should treesit--font-lock-fast-mode be locally bound inside that >>> function, so that it's reset between chunks? Or maybe the condition >>> for its enabling should be tweaked? E.g. I don't think there are any >>> particularly large or deep nodes in ruby.rb's parse tree. It's a >>> very shallow file. > > Yeah that is a not-very-clever hack. I=E2=80=99ve got an idea: I can = add a C > function that checks the maximum depth of a parse tree and the maximum > node span, and turn on the fast-mode if the depth is too large or a = node > is too wide. And we do that check once before doing any fontification. > > I=E2=80=99ll report back once I add it. I wrote that function. But I didn=E2=80=99t end up using it. Instead I = added a "grace count", so that the query time has to be longer than the threshold 5 times before we switch on the fast mode instead of 1. My main worry is that simply looking at the parse tree would not catch all the case where there will be expensive queries. Could you try the latest commit and see if the fast mode still switches on when it shouldn=E2=80=99t? Yuan From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 19 13:28:51 2023 Received: (at 60691) by debbugs.gnu.org; 19 Jan 2023 18:28:51 +0000 Received: from localhost ([127.0.0.1]:44727 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pIZec-0004rE-Sx for submit@debbugs.gnu.org; Thu, 19 Jan 2023 13:28:51 -0500 Received: from mail-ed1-f50.google.com ([209.85.208.50]:35610) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pIZeb-0004r2-CT for 60691@debbugs.gnu.org; Thu, 19 Jan 2023 13:28:50 -0500 Received: by mail-ed1-f50.google.com with SMTP id y19so3998660edc.2 for <60691@debbugs.gnu.org>; Thu, 19 Jan 2023 10:28:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=IEtOXFc3EtCB2Fm2g8Vnr3oxuso2rd41z//ttnErIxk=; b=flnvTt75gLO+EUDU7nxPZIxHSAkGbRBwQ9Bc1llqjjz8hrgoBdkIzVsbwvEOrODtT5 ep3aqRhH23NBb310P1NPev5tlSiIazdcWNhegqqnek8dOCLhW6CPe9qkzMsLlrH7tQyk DekQOLoHOhGV1P2/3gx/YqemAML+I9jk6HVt/fO4Futwu5Mmd8C1svtSduXYxFlngttp qRCHJ1BFWDPtSpwaadVHRMhhTEEshhJs/dcAKRQ5gkHYu3zCf5jNpLcu7IQFUd0Yu7fZ DgB42QaAodMAcZllbzUiqY/AbqFuPHZISDOp3Hs9FJcT1O2qYmdJi/HaxXA/Vg9GdEhS cIFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IEtOXFc3EtCB2Fm2g8Vnr3oxuso2rd41z//ttnErIxk=; b=8CoCWn3ndFC46B7fu5h3J10GhudLQw77aEOUTVg1HO1HzL2Rs0fijXLxG60fVyRxGA r+hbMtXI7nyn3n64yx6TsWdUdRdNuBSB+zJXGi4ZbikG+dk3WSaMPqoCkDA8Bc0WpOBo EpciITHa+WYGaIH58hA2ukCv/6dSS1nW1p+9i2Z8iJ033S03yxO04Pq0uGdEXakgLDld vYR5htnsm40m3w3cvBP5q1AjbICxGdWRxaHa4CbZeGT+ut4wO81Jyhqpo7uYHNShA3eG v3YrqaJx4pOhraxLS9T/MwjPYAYKyvFpM4nKMo7JBzxkHEcmYDVZX3I3JFf7aFrx7Qg0 3BVA== X-Gm-Message-State: AFqh2koeCXzgZhVH208D+5ijmBUD2/FbPdNgdku6Z+p6sjDpwxO4jjA9 NAAkpfYClvgbQHGNy8g+eGM= X-Google-Smtp-Source: AMrXdXuA1B0/qpCv4A2WttN0GVH9WzblPHIYwI2BaH5xBk6/ZVpg5KMsxHWj6EFIjSKesjffYF9jVg== X-Received: by 2002:a05:6402:4486:b0:48f:a9a2:29f4 with SMTP id er6-20020a056402448600b0048fa9a229f4mr12527694edb.1.1674152923429; Thu, 19 Jan 2023 10:28:43 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id m2-20020a509302000000b0046892e493dcsm16188924eda.26.2023.01.19.10.28.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 19 Jan 2023 10:28:43 -0800 (PST) Message-ID: <4b75a1e5-ace9-2312-54f7-c1de9c798d9c@yandex.ru> Date: Thu, 19 Jan 2023 20:28:41 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Yuan Fu References: <867cxv3dnn.fsf@mail.linkov.net> From: Dmitry Gutov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) Hi Yuan, On 18/01/2023 08:50, Yuan Fu wrote: >>>> Should treesit--font-lock-fast-mode be locally bound inside that >>>> function, so that it's reset between chunks? Or maybe the condition >>>> for its enabling should be tweaked? E.g. I don't think there are any >>>> particularly large or deep nodes in ruby.rb's parse tree. It's a >>>> very shallow file. >> >> Yeah that is a not-very-clever hack. I’ve got an idea: I can add a C >> function that checks the maximum depth of a parse tree and the maximum >> node span, and turn on the fast-mode if the depth is too large or a node >> is too wide. And we do that check once before doing any fontification. >> >> I’ll report back once I add it. > > I wrote that function. But I didn’t end up using it. Instead I added a > "grace count", so that the query time has to be longer than the > threshold 5 times before we switch on the fast mode instead of 1. > > My main worry is that simply looking at the parse tree would not catch > all the case where there will be expensive queries. That might be true, but a criterion that doesn't specify conditions exactly can give no guarantee against false positives. > Could you try the latest commit and see if the fast mode still switches > on when it shouldn’t? At first it seemed to help, but then I switched the major mode a couple more times, and ran the benchmark twice more, and the "fast mode" switched on again. Which seems to make sense: there is no resetting the counter, right? So if previously it happened once somehow during a certain scenario, now I have to repeat the same scenario 4 times, and the condition is met. From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 20 17:24:47 2023 Received: (at 60691) by debbugs.gnu.org; 20 Jan 2023 22:24:47 +0000 Received: from localhost ([127.0.0.1]:47424 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pIzoV-0003Oo-Df for submit@debbugs.gnu.org; Fri, 20 Jan 2023 17:24:47 -0500 Received: from mail-pj1-f42.google.com ([209.85.216.42]:43982) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pIzoT-0003OU-Pc for 60691@debbugs.gnu.org; Fri, 20 Jan 2023 17:24:46 -0500 Received: by mail-pj1-f42.google.com with SMTP id m3-20020a17090a414300b00229ef93c5b0so5535524pjg.2 for <60691@debbugs.gnu.org>; Fri, 20 Jan 2023 14:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=yBRL9SkyjI0Jh0dvzLjzRWSjHDz+ERBBGBnWrhbSQQU=; b=qh7kIcMM//Jpe6K4HABzbQg70334yvaWMsQoTJ415s3VWxQfMSQyK6rBz6gjaw0dV7 N58yki7m3Lp0h0LSlf7aH/H/blrEeaqNMEBw+0V4iRVwK0sus92XMr7boZI1XxNqgA2m tCS2/W5Go1f5+dWU7dUAJ7A1cA/ckpy2X6Ui3oKiZVdnfhV2FWVFuzbOVjruiDNFyKVn 0310T0NYJtV4JgWryrmr5er/LSjRZJQi5V6hVh4oxXTvMLHeaul6gIYJYmkE0yZdsk+v wp7AUTA2SETN/aqgiZRqC4Jld4ItHoH6T7v+IBpwflsX4vJ0yna1oMzh78YSD4XIhRv8 CW2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yBRL9SkyjI0Jh0dvzLjzRWSjHDz+ERBBGBnWrhbSQQU=; b=wKJ9+rSJ2J+eGSDhSzcRHRR2WoqO9DT27ijSe1LPcBeDWSLwFBzbVRS3gN+psexoWa y66H1qEQ5FzY/0RqQ0nT8bZ3/gH4kXCxjSwos7jO2JjGMQNAb80RwsdliYcekUuDihiC DXNVhZhUL9Kb7ra4FYzqfmBtQrZ0e/ixUafwokIWFxWm8IWWPc2uIn5vwDyjPZUO+yen dQ8XLHq8U53vqaK+VjFk+eTD8jF/eL0FovHmqiLyVJayCX/1uvHJLHp/vbfEGGeA8Pj1 JsxNBs5Cv/Npsm5Y6NS+FIAi7KJmMRHK2HE5TX9iPrvJCf0yHWvHkY31Qcc+R1WqubBq PmXQ== X-Gm-Message-State: AFqh2krNwzNfnaGuNqQuiVYAnfn+2d90Znf1ikLYlyYOqXOC4uWC5Det gS3p/G9vAgxr4vPIVsIM4UE= X-Google-Smtp-Source: AMrXdXvQHBQ20KolP43LmU0diXci8PctW0EwUNX6kedU1HwHooS2U6Jfevy31brXPC1nOSreutdB3Q== X-Received: by 2002:a17:903:40cc:b0:194:3f40:69b7 with SMTP id t12-20020a17090340cc00b001943f4069b7mr16455709pld.63.1674253480148; Fri, 20 Jan 2023 14:24:40 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id z20-20020a170903409400b001933b4b1a49sm22014213plc.183.2023.01.20.14.24.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 20 Jan 2023 14:24:39 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode From: Yuan Fu In-Reply-To: <4b75a1e5-ace9-2312-54f7-c1de9c798d9c@yandex.ru> Date: Fri, 20 Jan 2023 14:24:28 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <90D70081-4A27-4219-93C3-240D35F29711@gmail.com> References: <867cxv3dnn.fsf@mail.linkov.net> <4b75a1e5-ace9-2312-54f7-c1de9c798d9c@yandex.ru> To: Dmitry Gutov X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) > On Jan 19, 2023, at 10:28 AM, Dmitry Gutov wrote: >=20 > Hi Yuan, >=20 > On 18/01/2023 08:50, Yuan Fu wrote: >>>>> Should treesit--font-lock-fast-mode be locally bound inside that >>>>> function, so that it's reset between chunks? Or maybe the = condition >>>>> for its enabling should be tweaked? E.g. I don't think there are = any >>>>> particularly large or deep nodes in ruby.rb's parse tree. It's a >>>>> very shallow file. >>>=20 >>> Yeah that is a not-very-clever hack. I=E2=80=99ve got an idea: I can = add a C >>> function that checks the maximum depth of a parse tree and the = maximum >>> node span, and turn on the fast-mode if the depth is too large or a = node >>> is too wide. And we do that check once before doing any = fontification. >>>=20 >>> I=E2=80=99ll report back once I add it. >> I wrote that function. But I didn=E2=80=99t end up using it. Instead = I added a >> "grace count", so that the query time has to be longer than the >> threshold 5 times before we switch on the fast mode instead of 1. >> My main worry is that simply looking at the parse tree would not = catch >> all the case where there will be expensive queries. >=20 > That might be true, but a criterion that doesn't specify conditions = exactly can give no guarantee against false positives. The condition is =E2=80=9Cquery is (consistently) slow=E2=80=9D, = that=E2=80=99s why I thought measuring the time is the most direct way. >=20 >> Could you try the latest commit and see if the fast mode still = switches >> on when it shouldn=E2=80=99t? >=20 > At first it seemed to help, but then I switched the major mode a = couple more times, and ran the benchmark twice more, and the "fast mode" = switched on again. >=20 > Which seems to make sense: there is no resetting the counter, right? >=20 > So if previously it happened once somehow during a certain scenario, = now I have to repeat the same scenario 4 times, and the condition is = met. I was hoping that the scenario only happen once, oh well :-) I=E2=80=99ll = change the decision based on analyzing the tree=E2=80=99s dimension: too = deep or too wide activates the fast mode. Let=E2=80=99s see how it = works. Yuan From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 21 21:01:49 2023 Received: (at 60691) by debbugs.gnu.org; 22 Jan 2023 02:01:49 +0000 Received: from localhost ([127.0.0.1]:50143 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pJPg5-0002Rg-0q for submit@debbugs.gnu.org; Sat, 21 Jan 2023 21:01:49 -0500 Received: from mail-ed1-f46.google.com ([209.85.208.46]:38411) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pJPg2-0002RQ-N0 for 60691@debbugs.gnu.org; Sat, 21 Jan 2023 21:01:47 -0500 Received: by mail-ed1-f46.google.com with SMTP id w14so10869310edi.5 for <60691@debbugs.gnu.org>; Sat, 21 Jan 2023 18:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=mBPwKCs9yt08/M0LVqu84LIsYbWS6LrGx9QbglXHzQA=; b=YiXM6kgthn5MVQFiIrkCnITsHiD7v94AcqhclpuzHbJroMnnHdnKx30hBcBuxdc/A0 OVfrik9HmXMr55aGs4kVp/qtyCDThtX99GNrJUN8zpWjPHJ3h+ly1m5iBy7buFyFCpvx n0iPaArtOCDUzm7dYqKZ0EcnowKI8RuwLYprCTHEGOL2FIKXwOtvOc+M/2cGDcQCB+IL jmWN1+nrBSTJQZiCCw1xS76kENGf7wkWqrR6nOfW4kGjefqzBqB2KlAnzj13KexUCk+5 tELJTAHXbM3xUibFSAqmxcsBNPvX7YY32w/KheIbAMCy6y6PjQ1ySOdZ6LKVwBtza+vU n0ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mBPwKCs9yt08/M0LVqu84LIsYbWS6LrGx9QbglXHzQA=; b=U1qXMpR4C25HVcAt6Hgsi56G7s5u7Xs7aCVM3gTT81/1q9Vt/zkoEQcTNfs9ee8nV4 TdVO6iTFZ5/mU/0rmvxMcDYK3AK/vg53QKyLXNKOX0YPbni1KTc7H3w3JWcjgJl21nWj v/SPxIPBSdpRcz7ijgUFYzpm2QueOQMvpWuxVcWyGH51IKrMGyEJwY1sJnAO/Bbi2eFZ kIqlDy1M8T+B/k5O5pdhLXh6+/4SBK8bgNWqlACT43fKtGYQItEs+Qpp1I1MRKyuFMeb WobYKcj8JN7Y4llIflsSNdieE0KCUMA/eGO5FEAKZE3/WjI22Ad9HAEH6dWjUsPKSxNX mLFw== X-Gm-Message-State: AFqh2kqZimWqlZxgk/009mP1ie9JvNtoVP4NGyAKhM3gpu7S4Vop/Dpe hFqV91Nc1MmwVJAze8bn1Bc= X-Google-Smtp-Source: AMrXdXtdo9bFIDxU2Uts2RJne77nstH2gXciqcavWP0LX8RHf5D9Wrn0Rx9oAR6xYm/RylgVMYK+Hw== X-Received: by 2002:a05:6402:2499:b0:499:1ed2:645d with SMTP id q25-20020a056402249900b004991ed2645dmr21912553eda.17.1674352900719; Sat, 21 Jan 2023 18:01:40 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id f6-20020a056402150600b0049622a61f8fsm19573434edw.30.2023.01.21.18.01.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 21 Jan 2023 18:01:40 -0800 (PST) Message-ID: <3308a010-b1aa-8c99-887e-7dc1e0e94709@yandex.ru> Date: Sun, 22 Jan 2023 04:01:38 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Yuan Fu References: <867cxv3dnn.fsf@mail.linkov.net> <4b75a1e5-ace9-2312-54f7-c1de9c798d9c@yandex.ru> <90D70081-4A27-4219-93C3-240D35F29711@gmail.com> From: Dmitry Gutov In-Reply-To: <90D70081-4A27-4219-93C3-240D35F29711@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) On 21/01/2023 00:24, Yuan Fu wrote: > > >> On Jan 19, 2023, at 10:28 AM, Dmitry Gutov wrote: >> >> Hi Yuan, >> >> On 18/01/2023 08:50, Yuan Fu wrote: >>>>>> Should treesit--font-lock-fast-mode be locally bound inside that >>>>>> function, so that it's reset between chunks? Or maybe the condition >>>>>> for its enabling should be tweaked? E.g. I don't think there are any >>>>>> particularly large or deep nodes in ruby.rb's parse tree. It's a >>>>>> very shallow file. >>>> >>>> Yeah that is a not-very-clever hack. I’ve got an idea: I can add a C >>>> function that checks the maximum depth of a parse tree and the maximum >>>> node span, and turn on the fast-mode if the depth is too large or a node >>>> is too wide. And we do that check once before doing any fontification. >>>> >>>> I’ll report back once I add it. >>> I wrote that function. But I didn’t end up using it. Instead I added a >>> "grace count", so that the query time has to be longer than the >>> threshold 5 times before we switch on the fast mode instead of 1. >>> My main worry is that simply looking at the parse tree would not catch >>> all the case where there will be expensive queries. >> >> That might be true, but a criterion that doesn't specify conditions exactly can give no guarantee against false positives. > > The condition is “query is (consistently) slow”, that’s why I thought measuring the time is the most direct way. The benchmark itself might be artificial, in that it's measuring the font-lock of a specific buffer, in whole, for 1000 iterations. But Juri must have come up with the original report based on real usage scenario. OTOH, the scenario which it might correspond to, is used typing in the same buffer for a long time (triggering thousands of refontifications, possibly partial ones). I don't know if it's feasible to try to reproduce it specifically. But, again, anything that can happen once can happen 4 more times. >>> Could you try the latest commit and see if the fast mode still switches >>> on when it shouldn’t? >> >> At first it seemed to help, but then I switched the major mode a couple more times, and ran the benchmark twice more, and the "fast mode" switched on again. >> >> Which seems to make sense: there is no resetting the counter, right? >> >> So if previously it happened once somehow during a certain scenario, now I have to repeat the same scenario 4 times, and the condition is met. > > I was hoping that the scenario only happen once, oh well :-) I’ll change the decision based on analyzing the tree’s dimension: too deep or too wide activates the fast mode. Let’s see how it works. Thank you, let me know when it's time to test again. From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 29 03:25:41 2023 Received: (at 60691) by debbugs.gnu.org; 29 Jan 2023 08:25:41 +0000 Received: from localhost ([127.0.0.1]:42239 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pM30O-0001fQ-Vi for submit@debbugs.gnu.org; Sun, 29 Jan 2023 03:25:41 -0500 Received: from mail-pj1-f41.google.com ([209.85.216.41]:45717) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pM30M-0001f6-81 for 60691@debbugs.gnu.org; Sun, 29 Jan 2023 03:25:40 -0500 Received: by mail-pj1-f41.google.com with SMTP id nm12-20020a17090b19cc00b0022c2155cc0bso8433016pjb.4 for <60691@debbugs.gnu.org>; Sun, 29 Jan 2023 00:25:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:cc:date:message-id:subject:mime-version :content-transfer-encoding:from:from:to:cc:subject:date:message-id :reply-to; bh=UPbzADtHQ8/4BAGuJ9oO2Evj3IM+Czlfa1U92qLTR9o=; b=HokYOUp31EjDlD/L+ZgnB7FB6J8C2+mE/OKxI+HGwp1WaovNxuh2Z+6MADjJSFWyTZ JZ6slj5X3URYQS0lme+1vGkjN29qdfcM9pSg8sCo8dmRT050KUUf6J3AZP2M5vhwa7Ci zYzQJ2wxFt00guf5JW/0AWKMAQdmwmQv3B8+gt0hswpF3geRJ+0FEU5uNn/Mb/5+xNLi KbTS1nKMGAbZFm20A1jT2P3MA6joYDQpEiZJw/h+gQ7Hf1aWoZwxyvXoOKdWjY8vnhqn 6FH/1DsL2QMmotSB9WFCPwfg47OT2Lk8gX6b2Xg/Jlhk39jvYvN51uYhInYWB2s8IxBy 81wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:cc:date:message-id:subject:mime-version :content-transfer-encoding:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UPbzADtHQ8/4BAGuJ9oO2Evj3IM+Czlfa1U92qLTR9o=; b=pdgGF45Fz93emXMfjsfSIn3DPvmxeRP70Wqj8hgVh8TODTUFvqVJGTGslfdWT3NimX e9VsQJD1W/TyTNg4tIjORYFcNpx1T2xswCVBpLdZJ3mexTwtyx0ZelxmxZsGXJ7mf2qi edwFksjC5YmhYC/ES0rtqLQr3u0lNq6m/tsdUY/WwX2YApFkJvO2uDT2iEN7spVPLIUl IkrTDluNwqIrzOXe6BNVPQet+ONYGD07JhtmMfDRajuSedC83W0W+h3Orc01k7CnYLZL qp80FCDYvR/3QxH6ycPIs8rbzmpDQ02/nacQVR83ITT0/M+6b4npp7ivX1a2pGlxgN4A 56aA== X-Gm-Message-State: AO0yUKWn8o3mOGRVTJT9/dmclnS9TVpQlAOXh60ng9QZ/6iHp9D3f2Rg bwmhBG5YwCUbE+A3EAYmBk4= X-Google-Smtp-Source: AK7set+fyfk1b8AivjQ6l/LkJLINWtTjGtyE9sn1UDLuJfp3i+Koh0YhAu0Gy6N/wy/srFeUwr/JXw== X-Received: by 2002:a05:6a20:7da3:b0:bc:cc66:60b7 with SMTP id v35-20020a056a207da300b000bccc6660b7mr2232823pzj.51.1674980732339; Sun, 29 Jan 2023 00:25:32 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id t16-20020a639550000000b0046fefb18a09sm4707596pgn.91.2023.01.29.00.25.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 29 Jan 2023 00:25:32 -0800 (PST) From: Yuan Fu Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Message-Id: <90D8581D-C543-43EE-8BEB-51B5AFBCBAEE@gmail.com> Date: Sun, 29 Jan 2023 00:25:20 -0800 To: Dmitry Gutov X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Dmitry Gutov writes: > On 21/01/2023 00:24, Yuan Fu wrote: >>=20 >>> On Jan 19, 2023, at 10:28 AM, Dmitry Gutov wrote: >>> >>> Hi Yuan, >>> >>> On 18/01/2023 08:50, Yuan Fu wrote: >>>>>>> Should treesit--font-lock-fast-mode be locally bound inside that >>>>>>> function, so that it's reset between chunks? Or maybe the = condition >>>>>>> for its enabling should be tweaked? E.g. I don't think there are = any >>>>>>> particularly large or deep nodes in ruby.rb's parse tree. It's a >>>>>>> very shallow file. >>>>> >>>>> Yeah that is a not-very-clever hack. I=E2=80=99ve got an idea: I = can add a C >>>>> function that checks the maximum depth of a parse tree and the = maximum >>>>> node span, and turn on the fast-mode if the depth is too large or = a node >>>>> is too wide. And we do that check once before doing any = fontification. >>>>> >>>>> I=E2=80=99ll report back once I add it. >>>> I wrote that function. But I didn=E2=80=99t end up using it. = Instead I added a >>>> "grace count", so that the query time has to be longer than the >>>> threshold 5 times before we switch on the fast mode instead of 1. >>>> My main worry is that simply looking at the parse tree would not = catch >>>> all the case where there will be expensive queries. >>> >>> That might be true, but a criterion that doesn't specify conditions = exactly can give no guarantee against false positives. >> The condition is =E2=80=9Cquery is (consistently) slow=E2=80=9D, = that=E2=80=99s why I >> thought measuring the time is the most direct way. > > The benchmark itself might be artificial, in that it's measuring the > font-lock of a specific buffer, in whole, for 1000 iterations. But > Juri must have come up with the original report based on real usage > scenario. > > OTOH, the scenario which it might correspond to, is used typing in the > same buffer for a long time (triggering thousands of refontifications, > possibly partial ones). I don't know if it's feasible to try to > reproduce it specifically. But, again, anything that can happen once > can happen 4 more times. > >>>> Could you try the latest commit and see if the fast mode still = switches >>>> on when it shouldn=E2=80=99t? >>> >>> At first it seemed to help, but then I switched the major mode a >>> couple more times, and ran the benchmark twice more, and the "fast >>> mode" switched on again. >>> >>> Which seems to make sense: there is no resetting the counter, right? >>> >>> So if previously it happened once somehow during a certain scenario, = now I have to repeat the same scenario 4 times, and the condition is = met. >> I was hoping that the scenario only happen once, oh well :-) I=E2=80=99= ll >> change the decision based on analyzing the tree=E2=80=99s dimension: = too >> deep or too wide activates the fast mode. Let=E2=80=99s see how it = works. > > Thank you, let me know when it's time to test again. Sorry for the delay. Now treesit-font-lock-fontify-region uses treesit-subtree-stat to determine whether to enable the "fast mode". Now it should be impossible to activate the fast mode on moderately sized buffers. Yuan From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 29 18:07:21 2023 Received: (at 60691) by debbugs.gnu.org; 29 Jan 2023 23:07:22 +0000 Received: from localhost ([127.0.0.1]:45563 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pMGld-0005N0-Fv for submit@debbugs.gnu.org; Sun, 29 Jan 2023 18:07:21 -0500 Received: from mail-ed1-f48.google.com ([209.85.208.48]:33711) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pMGlb-0005Mn-Kt for 60691@debbugs.gnu.org; Sun, 29 Jan 2023 18:07:20 -0500 Received: by mail-ed1-f48.google.com with SMTP id x7so6138561edr.0 for <60691@debbugs.gnu.org>; Sun, 29 Jan 2023 15:07:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=tVI5ZqFpxXK1//xIYEcRpZrjtZawj0OZt0MXOlQlPZM=; b=TsZDjH/dneM+L5NiOgo9fnVB1eqCWjPxqObcNtDcM5honPtwcOt2CQyR/0nG/oFcYS 8VQbqMZIKEgF2VB0AYOBsC2ekhMfrfpOwLxYdwALQF86dT17eA/OKeGI27fTrce3YLfR 34jw6BLtDkNbX/SRM8g/aswZ/6ildq0r1YjhDfDLGmHN0EynUBVAEXSA9VuW4Evfntgy ZGF8PLIlsQnbnvX1ZezvxNU3C1FPN5UcJqY8hsTrT/u+1/YWnDvn2+4QREY/k8gc6PsS 2ZAWkyIxYIbofRaF+Jwabf4ZKch2h/H/kij+qrQQGoYevROx551tEY8WIJNSVrg0Qt2Y XiBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tVI5ZqFpxXK1//xIYEcRpZrjtZawj0OZt0MXOlQlPZM=; b=11OGInyf5YGrxm3FDflRkvYH38dgGygWqGKqFcyBirO4qYM8gQzDlWzIQxwcl3vKrP RPygQUoixmzu8tTVnUsI7ySoQoXu05MZSvqaXYTFzV+b2x8qVbh3e8a5mAvn9GA9PX7a x5t8p5Bd9CRnu7HT9tDAN5vvovexR8B1qtxpRgwdaTTDrqdUPsJKMdt826EqziXZtIV2 8bb8V7azy1o2SBiorrY1jJ+g+MTq8uL4O+W/aNo8JzQ6x0NEEualjqxbc8dV8O3wMoei 29K2z66vbb+j0OkOkyBUqL/3Xs/eBD1yxWUEHMo4jNdJGR8v2kMo8rXvGDyuZLONxsph q8Jg== X-Gm-Message-State: AFqh2krTpCSsObtOZ6JWLn6C9pP8TwiW3XaTUwzmwhTDlhbuaxm9gEjV OOXKAIFLHi7xZCnC9oxXYpE= X-Google-Smtp-Source: AMrXdXsG/eJcH/M21fyFDVFbzWCYGpAJpyMiQq1S7lds7QLSKAVa36gdzIu8j0nxHU27Ztj8Z6Pnjw== X-Received: by 2002:a05:6402:2b92:b0:45a:7d2:9b35 with SMTP id fj18-20020a0564022b9200b0045a07d29b35mr50810466edb.0.1675033633640; Sun, 29 Jan 2023 15:07:13 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id y8-20020a056402134800b004610899742asm5869860edw.13.2023.01.29.15.07.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 29 Jan 2023 15:07:12 -0800 (PST) Message-ID: <0e552ada-081f-ad90-19c2-645a64ef50ac@yandex.ru> Date: Mon, 30 Jan 2023 01:07:10 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Yuan Fu References: <867cxv3dnn.fsf@mail.linkov.net> <90D8581D-C543-43EE-8BEB-51B5AFBCBAEE@gmail.com> From: Dmitry Gutov In-Reply-To: <90D8581D-C543-43EE-8BEB-51B5AFBCBAEE@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) Hi Yuan, On 29/01/2023 10:25, Yuan Fu wrote: >>>> So if previously it happened once somehow during a certain scenario, now I have to repeat the same scenario 4 times, and the condition is met. >>> I was hoping that the scenario only happen once, oh well :-) I’ll >>> change the decision based on analyzing the tree’s dimension: too >>> deep or too wide activates the fast mode. Let’s see how it works. >> >> Thank you, let me know when it's time to test again. > > Sorry for the delay. Now treesit-font-lock-fontify-region uses > treesit-subtree-stat to determine whether to enable the "fast mode". Now > it should be impossible to activate the fast mode on moderately sized > buffers. Thank you, it seems to work just fine in my scenario. And treesit-subtree-stat makes sense. I have a few more questions about the current strategy, though. IIUC, we only do the treesit--font-lock-fast-mode test once in treesit-font-lock-fontify-region, and then use the detected value for the whole later life of the buffer. Is that right? What if the buffer didn't originally have the problematic error nodes we are guarding from, and then later the user wrote enough code to have at least one of them? If they didn't close Emacs, or revert the buffer, our logic still wouldn't use the "fast node", would it? Or vice versa: if the buffer started out with error nodes, and consequently, "fast mode", but then the user has edited it so that those error nodes disappeared, shouldn't the buffer stop using the "fast mode"? From my measurements, in ruby-mode, at least treesit-subtree-stat is 20-40x faster than refontifying the whole buffer. So one possible strategy would be to repeat the test every time. I'm not sure it's fast enough in the "problem" buffers, though, and I don't have any to test. In those I did test, though, it takes ~1 ms. But we could repeat the test only once every couple of seconds and/or after the buffer has changed again. That would hopefully make it a non-bottleneck in all cases. From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 29 18:24:39 2023 Received: (at 60691) by debbugs.gnu.org; 29 Jan 2023 23:24:39 +0000 Received: from localhost ([127.0.0.1]:45572 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pMH2M-0005lV-W9 for submit@debbugs.gnu.org; Sun, 29 Jan 2023 18:24:39 -0500 Received: from mail-pj1-f41.google.com ([209.85.216.41]:41529) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pMH2L-0005lI-LN for 60691@debbugs.gnu.org; Sun, 29 Jan 2023 18:24:38 -0500 Received: by mail-pj1-f41.google.com with SMTP id z1-20020a17090a66c100b00226f05b9595so9531317pjl.0 for <60691@debbugs.gnu.org>; Sun, 29 Jan 2023 15:24:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=S+O5xk0WxjqbnURzVnqCS0FuG59/bZrjU85yOrksg7U=; b=XwOam6OkjOYpn9ErqoVS1Z9gBHzFUSmaSHfsDlDzj/kqVdp+PRTg5YvkT3xnt7N2ui 8+aguUQL97LthjfMNC2R1Ms+oE4mVk1QDQq2TgJmMvaYbXq3vKWRrS3pwlAHDjisisZb C4aMQnbXNGPCwxrbYZrUdJ+tuoYLbQTmPT4NzkBAIqBVSQJP9wOZIydywdcO0AkLedDd 5Igs+DDbk5OvMutJxdQmjzo+eodsFcrfaYHbN49gF+yle3N0HLu9mEd3A72IlgSERpjP lWoNJHRSLK3QXdz8/IikHS25STGCX0DygDV7duoG4y6MqZ/gDOtV2IJBc/zZf6DXHbdI cbxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S+O5xk0WxjqbnURzVnqCS0FuG59/bZrjU85yOrksg7U=; b=y34SjkNluVwAAC0FmOkxKcTi1VRSENEgwQpqB3R0Bz+SUpS8pFNSzqLIjls60BLars jenR3Hz9+0b4yx+0890gRu46Eo5wNJpu6tZ1QAsyTe3ztBArYFKsDgikrQM1RW6BxWD/ dL+97rAhkRCmwX2vFiRDADBljwWyVlXNlqtKm9/NuqaOtPLSV/q9KRcVatwlXBobU16V SLvpyl1f4Q/4AVlGqpm7QRNyv0QMYax7LgF+qxjl5ATLjjNb9digMAOnvjfL7OTggbQi IDTs+b2P7BNoW1+Rb8zvyPtyq9KqLkPqzhDlWUnn3BBlRDor0LpFSR1ph7JbB5JU1AN+ eIRA== X-Gm-Message-State: AO0yUKVL4KFLBEZi8/fj4mxAsPbGZzH6p/JPecICnXgGWpNlOpXzZY27 ZVV4ntNc+Ld2nex/lEHyYGI= X-Google-Smtp-Source: AK7set899EM4dbxz6E86gsdg392jUvvGnccGM1P1hroVYKyzmJqIa5OaKda1Aq26H7vAWaCOsnui3w== X-Received: by 2002:a05:6a20:8f06:b0:bc:d738:592e with SMTP id b6-20020a056a208f0600b000bcd738592emr4860024pzk.14.1675034671668; Sun, 29 Jan 2023 15:24:31 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id v25-20020a63ac19000000b0047685ed724dsm5510278pge.40.2023.01.29.15.23.46 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 29 Jan 2023 15:23:59 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode From: Yuan Fu In-Reply-To: <0e552ada-081f-ad90-19c2-645a64ef50ac@yandex.ru> Date: Sun, 29 Jan 2023 15:23:34 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <867cxv3dnn.fsf@mail.linkov.net> <90D8581D-C543-43EE-8BEB-51B5AFBCBAEE@gmail.com> <0e552ada-081f-ad90-19c2-645a64ef50ac@yandex.ru> To: Dmitry Gutov X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 60691 Cc: 60691@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) > On Jan 29, 2023, at 3:07 PM, Dmitry Gutov wrote: >=20 > Hi Yuan, >=20 > On 29/01/2023 10:25, Yuan Fu wrote: >=20 >>>>> So if previously it happened once somehow during a certain = scenario, now I have to repeat the same scenario 4 times, and the = condition is met. >>>> I was hoping that the scenario only happen once, oh well :-) I=E2=80=99= ll >>>> change the decision based on analyzing the tree=E2=80=99s = dimension: too >>>> deep or too wide activates the fast mode. Let=E2=80=99s see how it = works. >>>=20 >>> Thank you, let me know when it's time to test again. >> Sorry for the delay. Now treesit-font-lock-fontify-region uses >> treesit-subtree-stat to determine whether to enable the "fast mode". = Now >> it should be impossible to activate the fast mode on moderately sized >> buffers. >=20 > Thank you, it seems to work just fine in my scenario. And = treesit-subtree-stat makes sense. >=20 > I have a few more questions about the current strategy, though. >=20 > IIUC, we only do the treesit--font-lock-fast-mode test once in = treesit-font-lock-fontify-region, and then use the detected value for = the whole later life of the buffer. Is that right? >=20 > What if the buffer didn't originally have the problematic error nodes = we are guarding from, and then later the user wrote enough code to have = at least one of them? If they didn't close Emacs, or revert the buffer, = our logic still wouldn't use the "fast node", would it? >=20 > Or vice versa: if the buffer started out with error nodes, and = consequently, "fast mode", but then the user has edited it so that those = error nodes disappeared, shouldn't the buffer stop using the "fast = mode"? >=20 > =46rom my measurements, in ruby-mode, at least treesit-subtree-stat is = 20-40x faster than refontifying the whole buffer. So one possible = strategy would be to repeat the test every time. I'm not sure it's fast = enough in the "problem" buffers, though, and I don't have any to test. >=20 > In those I did test, though, it takes ~1 ms. >=20 > But we could repeat the test only once every couple of seconds and/or = after the buffer has changed again. That would hopefully make it a = non-bottleneck in all cases. I should mention this in the comments, but the fast mode is only for = very rare cases, where the file is mechanically generated and has some = peculiarities that causes tree-sitter to work poorly. If the file is = hand-written and =E2=80=9Cnormal=E2=80=9D, even huge files like xdisp.c = is well below the bar. Therefore I don=E2=80=99t think =E2=80=9Ccrossing = the line=E2=80=9D will realistically happen when editing source files. Here is the stats of two =E2=80=9Cproblematic files=E2=80=9D, named = packet and dec_mask, comparing to xdisp.c: ;; max-depth max-width count ;; cut-off 100 4000 ;; packet (98159 46581 1895137) ;; dec mask (3 64301 283995) ;; xdisp.c (29 985 218971) I=E2=80=99d say that any regular source file, even mechanically = generated, wouldn=E2=80=99t go beyond ~50 levels in depth, and = hand-written files should never has a node that has 4000+ direct = children in the parse tree. Yuan= From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 29 19:15:55 2023 Received: (at 60691-done) by debbugs.gnu.org; 30 Jan 2023 00:15:55 +0000 Received: from localhost ([127.0.0.1]:45607 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pMHpy-00071u-TM for submit@debbugs.gnu.org; Sun, 29 Jan 2023 19:15:55 -0500 Received: from mail-wm1-f53.google.com ([209.85.128.53]:51136) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pMHpw-00071g-SC for 60691-done@debbugs.gnu.org; Sun, 29 Jan 2023 19:15:53 -0500 Received: by mail-wm1-f53.google.com with SMTP id bg26so1119331wmb.0 for <60691-done@debbugs.gnu.org>; Sun, 29 Jan 2023 16:15:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=yAWpgZLe2L13sJGMg0oxfJqkKN/wH/FJe6Py2cJQQH8=; b=ALQ5OEp6jXRy4tmnNI6QsZVHw92xfxVCopiVFRTAFYhnxpohraQDuLPhWHMZD70Hq8 co7Eb8MDLGcJn1jTqbtdHkZue0HitufeMU82JBV57lBl5NA9x7yMW4KpGYmtfA6HjqrL 1/+GxlLkoblyUCkSKWMfcxMGKLF5FRvZsOAmyakBG4y7En3Idul5h650h4yxGxhhrE2M ONrXabH35Kj3cn6BubVgkJrHdxEV6FFse2xhVPwz2LsBiI+zIzgw14ZQHvlqnOW9fPva LUB6U7SZu6fpBRRSC7U7Z9rKt43bqRbsDL32w3G8IWvlihAZFsfUPSFalnHtu10jVhUv PxhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yAWpgZLe2L13sJGMg0oxfJqkKN/wH/FJe6Py2cJQQH8=; b=JH9MqONsX02SPJa9lqAKxCDIneV1gyuxnIGKsfw9WJM0BUljYQCKDjiDZld8QaLjtI MV+LAWoAn42DQQj+pAhXTqN0gTo5lLdwYqJjWFa/CRbBTMjdsilYOppNdACSSG/Dos67 c4MkxK6mO9HuLOgY6FGYNkHYifV1cdZAXTXuhvcbxx/qZDerj7Z1L2AHt4/TNaa2lGBi QzU8P0z/u+ax6GMTnYkjLJHK4nFCVZ2JW725lVPpfenBl3F3gwMYkdo0qE60aw6S6lmL TNuSWm5pbOqGq3K/dHwjJYmh8NwdBzRX29tyofUsFOuME8ORrsg0OJY0enLPmMqoCrAz fZYw== X-Gm-Message-State: AO0yUKWe2jsrsyXsUG4GG+sjgIN+JEUX/BtQ670OQ+lwWDZuFE9gszT9 uh45AgfiwsI37NP5IkvvV/A= X-Google-Smtp-Source: AK7set+4DiPFItomTG73yZLfgp+rDdjx/O38fqLWpdLvkB58Ru/o3YTBB6F6SnhN/Thd8EX5jMjFXQ== X-Received: by 2002:a05:600c:825:b0:3dc:5823:d6be with SMTP id k37-20020a05600c082500b003dc5823d6bemr2392941wmp.25.1675037746767; Sun, 29 Jan 2023 16:15:46 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id o3-20020a05600c4fc300b003db1d9553e7sm16870749wmq.32.2023.01.29.16.15.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 29 Jan 2023 16:15:46 -0800 (PST) Message-ID: <152c2d15-ab5c-ff35-f79b-71691fc223f8@yandex.ru> Date: Mon, 30 Jan 2023 02:15:44 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Yuan Fu References: <867cxv3dnn.fsf@mail.linkov.net> <90D8581D-C543-43EE-8BEB-51B5AFBCBAEE@gmail.com> <0e552ada-081f-ad90-19c2-645a64ef50ac@yandex.ru> From: Dmitry Gutov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691-done Cc: juri@linkov.net, 60691-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) On 30/01/2023 01:23, Yuan Fu wrote: > >> On Jan 29, 2023, at 3:07 PM, Dmitry Gutov wrote: >> >> Hi Yuan, >> >> On 29/01/2023 10:25, Yuan Fu wrote: >> >>>>>> So if previously it happened once somehow during a certain scenario, now I have to repeat the same scenario 4 times, and the condition is met. >>>>> I was hoping that the scenario only happen once, oh well 😄 I’ll >>>>> change the decision based on analyzing the tree’s dimension: too >>>>> deep or too wide activates the fast mode. Let’s see how it works. >>>> Thank you, let me know when it's time to test again. >>> Sorry for the delay. Now treesit-font-lock-fontify-region uses >>> treesit-subtree-stat to determine whether to enable the "fast mode". Now >>> it should be impossible to activate the fast mode on moderately sized >>> buffers. >> Thank you, it seems to work just fine in my scenario. And treesit-subtree-stat makes sense. >> >> I have a few more questions about the current strategy, though. >> >> IIUC, we only do the treesit--font-lock-fast-mode test once in treesit-font-lock-fontify-region, and then use the detected value for the whole later life of the buffer. Is that right? >> >> What if the buffer didn't originally have the problematic error nodes we are guarding from, and then later the user wrote enough code to have at least one of them? If they didn't close Emacs, or revert the buffer, our logic still wouldn't use the "fast node", would it? >> >> Or vice versa: if the buffer started out with error nodes, and consequently, "fast mode", but then the user has edited it so that those error nodes disappeared, shouldn't the buffer stop using the "fast mode"? >> >> From my measurements, in ruby-mode, at least treesit-subtree-stat is 20-40x faster than refontifying the whole buffer. So one possible strategy would be to repeat the test every time. I'm not sure it's fast enough in the "problem" buffers, though, and I don't have any to test. >> >> In those I did test, though, it takes ~1 ms. >> >> But we could repeat the test only once every couple of seconds and/or after the buffer has changed again. That would hopefully make it a non-bottleneck in all cases. > I should mention this in the comments, but the fast mode is only for very rare cases, where the file is mechanically generated and has some peculiarities that causes tree-sitter to work poorly. If the file is hand-written and “normal”, even huge files like xdisp.c is well below the bar. Therefore I don’t think “crossing the line” will realistically happen when editing source files. > > Here is the stats of two “problematic files”, named packet and dec_mask, comparing to xdisp.c: > > ;; max-depth max-width count > ;; cut-off 100 4000 > ;; packet (98159 46581 1895137) > ;; dec mask (3 64301 283995) > ;; xdisp.c (29 985 218971) > > I’d say that any regular source file, even mechanically generated, wouldn’t go beyond ~50 levels in depth, and hand-written files should never has a node that has 4000+ direct children in the parse tree. Oh, thanks for the explanation. Then the current strategy makes sense. Is xdisp.c absolutely the largest C file in your experience? According to the above numbers, a file that's only 4x as large could hit our current cutoff. Though, TBH, maybe some extreme files do, and they have font-lock performance reduced somewhat. That's not the end of the world, and it shouldn't make a difference for the original scenario (diff-syntax fontification). Either way, I'm closing this report. Thank you for your help. From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 01 01:54:51 2023 Received: (at 60691-done) by debbugs.gnu.org; 1 Feb 2023 06:54:51 +0000 Received: from localhost ([127.0.0.1]:56254 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pN719-0007HH-0l for submit@debbugs.gnu.org; Wed, 01 Feb 2023 01:54:51 -0500 Received: from mail-oi1-f177.google.com ([209.85.167.177]:40912) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pN716-0007Gw-Cv for 60691-done@debbugs.gnu.org; Wed, 01 Feb 2023 01:54:49 -0500 Received: by mail-oi1-f177.google.com with SMTP id s66so14936973oib.7 for <60691-done@debbugs.gnu.org>; Tue, 31 Jan 2023 22:54:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=yghn100hqnq+asIhDyug5gwPGWU4Dt5pLb8rRSr1U8U=; b=khyPJ+eN+pF0rKE1G+tD4Xa0Fyj+8K8hPKi91m4UYu24s0m7hf0k+zg4bSR8D1ceVN gqoEbBQ6cipYmiqiXHjTT4Q8CRQ1wsmdt8U+Ak1qzG9vkLLok9WznKhZT1ABujIDY9KV lex827u1zL85mIcW+z9O2S06sY5WP0N/AXeWzZuLDoq35B1FBePUJDhc7dfrREANlIw8 4COpkQqaLp2+351jqrpK0tNYBXHvzPEgv9JUjezpzFeKrL1TujzXBvEqpKEZJWZYpCmc LLU53Fo+UWOL9jdcmGTJQB9qfCJ/HbF/fxTIOFz6ZzOTTpAFzQ4E6jGvZyX1VP5xxst7 JTvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yghn100hqnq+asIhDyug5gwPGWU4Dt5pLb8rRSr1U8U=; b=XuBx0NsyCNWEZYXV8S/2qpfyZbB5uSh5JZny6pSQb9tudHDI+wu+cw7B6Q4pYlKHOM 01TYg+H8x3kR63id0CwOXFT+YTS8mASQQ0VPpRsx4VAJypG8bUUz+rtA5xcWTAF6YXNY vfQjRhtQy4RESla+4BZ3t1llJRKGdjDQ9O+z2tNEJGlTdun2NBV2IZr3xUwHDu6xbXLV m5Vwz6q2l7CvYOwMAeGNk9NOib7Ssy8hzFCbNcEaPh2tOimDS0FAWtYTnnBEM6LP8ng8 NipT3OJzd4DJQDKR0q+pOImTa4mwn8j5iY7S5XF7Vx2aDGyUPaasESj8Q+ilj9TC1sxW 7Lsg== X-Gm-Message-State: AO0yUKVWcjf1X53rOSDqf7cMDTDUIf04NaarK6MW4JD1bp7ymIfH5LX8 xxFDtrJ/hqryvv96c5gYkMxN1CnML8M= X-Google-Smtp-Source: AK7set80z01PHYwEcDV2fg0X62CXL4EYXmhvlef2z/oUtDrHmcukinKkgMQXbOeazvozlNI0Xg+9xQ== X-Received: by 2002:a17:90b:1651:b0:22c:3544:8cb0 with SMTP id il17-20020a17090b165100b0022c35448cb0mr1062348pjb.8.1675230069869; Tue, 31 Jan 2023 21:41:09 -0800 (PST) Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id q8-20020a17090a178800b0022badab0dbasm434903pja.3.2023.01.31.21.41.01 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jan 2023 21:41:02 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode From: Yuan Fu In-Reply-To: <152c2d15-ab5c-ff35-f79b-71691fc223f8@yandex.ru> Date: Tue, 31 Jan 2023 21:26:11 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <867cxv3dnn.fsf@mail.linkov.net> <90D8581D-C543-43EE-8BEB-51B5AFBCBAEE@gmail.com> <0e552ada-081f-ad90-19c2-645a64ef50ac@yandex.ru> <152c2d15-ab5c-ff35-f79b-71691fc223f8@yandex.ru> To: Dmitry Gutov X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 60691-done Cc: juri@linkov.net, 60691-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) >> I should mention this in the comments, but the fast mode is only for = very rare cases, where the file is mechanically generated and has some = peculiarities that causes tree-sitter to work poorly. If the file is = hand-written and =E2=80=9Cnormal=E2=80=9D, even huge files like xdisp.c = is well below the bar. Therefore I don=E2=80=99t think =E2=80=9Ccrossing = the line=E2=80=9D will realistically happen when editing source files. >> Here is the stats of two =E2=80=9Cproblematic files=E2=80=9D, named = packet and dec_mask, comparing to xdisp.c: >> ;; max-depth max-width count >> ;; cut-off 100 4000 >> ;; packet (98159 46581 1895137) >> ;; dec mask (3 64301 283995) >> ;; xdisp.c (29 985 218971) >> I=E2=80=99d say that any regular source file, even mechanically = generated, wouldn=E2=80=99t go beyond ~50 levels in depth, and = hand-written files should never has a node that has 4000+ direct = children in the parse tree. >=20 > Oh, thanks for the explanation. Then the current strategy makes sense. >=20 > Is xdisp.c absolutely the largest C file in your experience? >=20 > According to the above numbers, a file that's only 4x as large could = hit our current cutoff. I don=E2=80=99t think these stats increase linearly as the file size = increases. Even if there is a file that has a node with 3999 direct = children, and the developer adds another one, I=E2=80=99d say it=E2=80=99s= better not to turn on =E2=80=9Cfast mode=E2=80=9D immediately. Yuan= From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 01 10:11:27 2023 Received: (at 60691-done) by debbugs.gnu.org; 1 Feb 2023 15:11:27 +0000 Received: from localhost ([127.0.0.1]:59615 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pNElj-00015T-9p for submit@debbugs.gnu.org; Wed, 01 Feb 2023 10:11:27 -0500 Received: from mail-ed1-f45.google.com ([209.85.208.45]:44917) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pNElh-00015G-Co for 60691-done@debbugs.gnu.org; Wed, 01 Feb 2023 10:11:26 -0500 Received: by mail-ed1-f45.google.com with SMTP id v13so17972411eda.11 for <60691-done@debbugs.gnu.org>; Wed, 01 Feb 2023 07:11:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=WWAMMVpCzF/LjSUlkyr2+hsmrrPbH9+6RQ+pVqtxCdM=; b=otTSuOgavT/QIwmqmIqf5ZszfKckl/QO6/n9OZ5ja5SEMpPROI2AbVipDrriffNO3S 8OIoErrtU8thiGRua95PasPHbujIbcdfCawnzojxIWhpURndzmWE90ZiHQj13SxW1OI4 IYkXA9k1FTwkybc/OgpVkWE7LvJxUVNKKIs1y9ZCU1KMGerb1h++ex2TBNZTNNiS7GIx U8e/8BH80nWfLIdyhtdmZHyrTbbGQ0Qja/rstP2alxrZSkmv7VAdc+VxwLCK3aXS+zZx MJ6RXS6QL2ftlSCIHOY8zpMjInDocOobB2TNAGwxKs0n0CEJ91WfliXmLJRk6o/YNXYp NYYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WWAMMVpCzF/LjSUlkyr2+hsmrrPbH9+6RQ+pVqtxCdM=; b=iSH3EFJTkIYMB/gAx09WpKGZwCXG3diDppaPqz7AzxsjPUBLV6pRR3l2r3z+++KciY mSuQY61EnAmX2IXtOZb8/oAHQxXLjgGsTBdDIfh2OQJS185P6tHdiHCbme3d1z0pi/bm 4JZge4Spi8A2EDhDnxNfL0RByUqQNv847vWk/zt+j7a9yUVrxnptfo+PiCPkDOcF7XBv 7q1Jfo3cAyx3yUqd795/vRxIu1NVsBMUGX5I/rK/IIIuqJEKP34B5LDuZ8JQ+VYerJUh W0x7ncXU2LSxPmsxcNBumxmCp+tTTF8wLc6xN7GZBdMUnYQ1x84oztvV7M4b11KTDpQb nU5Q== X-Gm-Message-State: AO0yUKWabWlbZMpBAupy7/uJiDEB7H+zxMaXQkfGAQn3iZTjpzAj04Oy xMYC7FPQPajZqIwEIDXW9C8= X-Google-Smtp-Source: AK7set81F8Lpj9QhquRkDpuSrGPxbiMcVTD+UO1XqfKgfkNoFv4UKvkGYRVhHancA3Aj+jbUD69X4A== X-Received: by 2002:a05:6402:2742:b0:4a2:3371:cb82 with SMTP id z2-20020a056402274200b004a23371cb82mr3344088edd.18.1675264279414; Wed, 01 Feb 2023 07:11:19 -0800 (PST) Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id u20-20020a50a414000000b004a08c52a2f0sm10070376edb.76.2023.02.01.07.11.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 01 Feb 2023 07:11:18 -0800 (PST) Message-ID: Date: Wed, 1 Feb 2023 17:11:16 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: bug#60691: 29.0.60; Slow tree-sitter font-lock in ruby-ts-mode Content-Language: en-US To: Yuan Fu References: <867cxv3dnn.fsf@mail.linkov.net> <90D8581D-C543-43EE-8BEB-51B5AFBCBAEE@gmail.com> <0e552ada-081f-ad90-19c2-645a64ef50ac@yandex.ru> <152c2d15-ab5c-ff35-f79b-71691fc223f8@yandex.ru> From: Dmitry Gutov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -0.9 (/) X-Debbugs-Envelope-To: 60691-done Cc: 60691-done@debbugs.gnu.org, juri@linkov.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.9 (-) On 01/02/2023 07:26, Yuan Fu wrote: >>> I should mention this in the comments, but the fast mode is only for very rare cases, where the file is mechanically generated and has some peculiarities that causes tree-sitter to work poorly. If the file is hand-written and “normal”, even huge files like xdisp.c is well below the bar. Therefore I don’t think “crossing the line” will realistically happen when editing source files. >>> Here is the stats of two “problematic files”, named packet and dec_mask, comparing to xdisp.c: >>> ;; max-depth max-width count >>> ;; cut-off 100 4000 >>> ;; packet (98159 46581 1895137) >>> ;; dec mask (3 64301 283995) >>> ;; xdisp.c (29 985 218971) >>> I’d say that any regular source file, even mechanically generated, wouldn’t go beyond ~50 levels in depth, and hand-written files should never has a node that has 4000+ direct children in the parse tree. >> Oh, thanks for the explanation. Then the current strategy makes sense. >> >> Is xdisp.c absolutely the largest C file in your experience? >> >> According to the above numbers, a file that's only 4x as large could hit our current cutoff. > I don’t think these stats increase linearly as the file size increases. Even if there is a file that has a node with 3999 direct children, and the developer adds another one, I’d say it’s better not to turn on “fast mode” immediately. I see your point. In the previous message I was talking about a different scenario: when a project has a file 4x the size of xdisp.c, and the user just opens it. I suspect it's not great to have "fast mode" enabled in that case? Like, false positive. Anyway, this is a very theoretical concern on my part. From unknown Thu Jun 19 14:03:23 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 02 Mar 2023 12:24:08 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator