Package: emacs;
Reported by: Juri Linkov <juri <at> linkov.net>
Date: Tue, 25 Mar 2025 18:44:02 UTC
Severity: normal
Fixed in version 31.0.50
Done: Juri Linkov <juri <at> linkov.net>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 77256 in the body.
You can then email your comments to 77256 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
casouri <at> gmail.com, v.pupillo <at> gmail.com, bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 25 Mar 2025 18:44:02 GMT) Full text and rfc822 format available.Juri Linkov <juri <at> linkov.net>:casouri <at> gmail.com, v.pupillo <at> gmail.com, bug-gnu-emacs <at> gnu.org.
(Tue, 25 Mar 2025 18:44:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: bug-gnu-emacs <at> gnu.org Subject: Treesit language-at-point Date: Tue, 25 Mar 2025 20:42:02 +0200
Do we still need such complicated functions as
mhtml-ts-mode--language-at-point, js--treesit-language-at-point, etc.
that duplicate the rules from 'treesit-range-rules'
when now the default language-at-point function could be implemented
just as
(treesit-parser-language
(or (seq-some (lambda (o) (overlay-get o 'treesit-parser))
(overlays-at (point) t))
treesit-primary-parser))
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Sat, 29 Mar 2025 08:27:01 GMT) Full text and rfc822 format available.Message #8 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Yuan Fu <casouri <at> gmail.com> To: Juri Linkov <juri <at> linkov.net> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Sat, 29 Mar 2025 01:25:44 -0700
> On Mar 25, 2025, at 11:42 AM, Juri Linkov <juri <at> linkov.net> wrote: > > Do we still need such complicated functions as > mhtml-ts-mode--language-at-point, js--treesit-language-at-point, etc. > that duplicate the rules from 'treesit-range-rules' > when now the default language-at-point function could be implemented > just as > > (treesit-parser-language > (or (seq-some (lambda (o) (overlay-get o 'treesit-parser)) > (overlays-at (point) t)) > treesit-primary-parser)) Yeah, we can provide a default language-at-point function now that determines the “most relevant parser” by embed level. But we should keep treesit-language-at-point-function because a) it’s already in a release version and b) we want major modes to be able to customize what parser to pick at any given point. Yuan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Mon, 31 Mar 2025 17:05:01 GMT) Full text and rfc822 format available.Message #11 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Mon, 31 Mar 2025 19:53:14 +0300
[Message part 1 (text/plain, inline)]
>> Do we still need such complicated functions as >> mhtml-ts-mode--language-at-point, js--treesit-language-at-point, etc. >> that duplicate the rules from 'treesit-range-rules' >> when now the default language-at-point function could be implemented >> just as >> >> (treesit-parser-language >> (or (seq-some (lambda (o) (overlay-get o 'treesit-parser)) >> (overlays-at (point) t)) >> treesit-primary-parser)) > > Yeah, we can provide a default language-at-point function now that > determines the “most relevant parser” by embed level. But we should keep > treesit-language-at-point-function because a) it’s already in a release > version and b) we want major modes to be able to customize what parser to > pick at any given point. Ok, so here is a complete patch:
[treesit-parsers-at.patch (text/x-diff, inline)]
diff --git a/lisp/textmodes/mhtml-ts-mode.el b/lisp/textmodes/mhtml-ts-mode.el
index 22c0455a4ee..fdb286fc8c4 100644
--- a/lisp/textmodes/mhtml-ts-mode.el
+++ b/lisp/textmodes/mhtml-ts-mode.el
@@ -221,21 +222,6 @@ mhtml-ts-mode-menu
"Menu bar for `mhtml-ts-mode'."
css-mode--menu)
-;; To enable some basic treesiter functionality, you should define
-;; a function that recognizes which grammar is used at-point.
-;; This function should be assigned to `treesit-language-at-point-function'
-(defun mhtml-ts-mode--language-at-point (point)
- "Return the language at POINT assuming the point is within a HTML buffer."
- (let* ((node (treesit-node-at point 'html))
- (parent (treesit-node-parent node))
- (node-query (format "(%s (%s))"
- (treesit-node-type parent)
- (treesit-node-type node))))
- (cond
- ((equal "(script_element (raw_text))" node-query) (js--treesit-language-at-point point))
- ((equal "(style_element (raw_text))" node-query) 'css)
- (t 'html))))
-
;; Custom font-lock function that's used to apply color to css color
;; The signature of the function should be conforming to signature
;; QUERY-SPEC required by `treesit-font-lock-rules'.
@@ -448,7 +434,7 @@ mhtml-ts-mode-flymake-mhtml
;;;###autoload
(define-derived-mode mhtml-ts-mode html-ts-mode
- '("HTML+" (:eval (let ((lang (mhtml-ts-mode--language-at-point (point))))
+ '("HTML+" (:eval (let ((lang (treesit-language-at (point))))
(cond ((eq lang 'html) "")
((eq lang 'javascript) "JS")
((eq lang 'css) "CSS")))))
@@ -527,10 +520,6 @@ mhtml-ts-mode
(setq-local c-ts-common--comment-regexp
js--treesit-jsdoc-comment-regexp))
-
- ;; Many treesit functions need to know the language at-point.
- ;; So you should define such a function.
- (setq-local treesit-language-at-point-function #'mhtml-ts-mode--language-at-point)
(setq-local prettify-symbols-alist mhtml-ts-mode--prettify-symbols-alist)
;; Indent.
diff --git a/lisp/treesit.el b/lisp/treesit.el
index 54c29326df2..75c04912216 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -185,15 +185,15 @@ treesit-language-at
This function assumes that parser ranges are up-to-date. It
returns the return value of `treesit-language-at-point-function'
if it's non-nil, otherwise it returns the language of the first
-parser in `treesit-parser-list', or nil if there is no parser.
+parser from `treesit-parsers-at', or the primary parser.
-In a multi-language buffer, make sure
-`treesit-language-at-point-function' is implemented! Otherwise
-`treesit-language-at' wouldn't return the correct result."
+In a multi-language buffer, optionally you can implement
+`treesit-language-at-point-function' to return more correct result."
(if treesit-language-at-point-function
(funcall treesit-language-at-point-function position)
- (when-let* ((parser (car (treesit-parser-list))))
- (treesit-parser-language parser))))
+ (treesit-parser-language
+ (or (car (treesit-parsers-at position))
+ treesit-primary-parser))))
;;; Node API supplement
@@ -247,8 +247,9 @@ treesit-node-at
(parser-or-lang
(let* ((local-parser (car (treesit-local-parsers-at
pos parser-or-lang)))
- (global-parser (car (treesit-parser-list
- nil parser-or-lang)))
+ (global-parser (or (car (treesit-parsers-at
+ pos parser-or-lang))
+ treesit-primary-parser))
(parser (or local-parser global-parser)))
(when parser
(treesit-parser-root-node parser))))
@@ -267,13 +268,10 @@ treesit-node-at
(local-parser
;; Find the local parser with highest
;; embed-level at point.
- (car (seq-sort-by #'treesit-parser-embed-level
- (lambda (a b)
- (> (or a 0) (or b 0)))
- (treesit-local-parsers-at
- pos lang))))
- (global-parser (car (treesit-parser-list
- nil lang)))
+ (car (treesit-local-parsers-at pos lang)))
+ (global-parser (or (car (treesit-parsers-at
+ pos lang))
+ treesit-primary-parser))
(parser (or local-parser global-parser)))
(when parser
(treesit-parser-root-node parser))))))
@@ -851,6 +849,38 @@ treesit--clip-ranges
if (<= start (car range) (cdr range) end)
collect range))
+(defun treesit-parsers-at (&optional pos language with-host local-only)
+ "Return all the non-primary parsers at POS.
+
+POS defaults to point.
+If LANGUAGE is non-nil, only return parsers for LANGUAGE.
+
+If WITH-HOST is non-nil, return a list of (PARSER . HOST-PARSER)
+instead. HOST-PARSER is the host parser which created the PARSER.
+
+If LOCAL-ONLY is non-nil, return only local parsers.
+Local parsers are those which only parse a limited region marked
+by an overlay with non-nil `treesit-parser' property."
+ (let ((res nil))
+ ;; Refer to (ref:local-parser-overlay) for more explanation of local
+ ;; parser overlays.
+ (dolist (ov (overlays-at (or pos (point))))
+ (when-let* ((parser (overlay-get ov 'treesit-parser))
+ (host-parser (or (null with-host)
+ (overlay-get ov 'treesit-host-parser)))
+ (_ (or (null local-only)
+ (overlay-get ov 'treesit-parser-local-p)))
+ (_ (or (null language)
+ (eq (treesit-parser-language parser)
+ language))))
+ (push (if with-host (cons parser host-parser) parser) res)))
+ (seq-sort-by (lambda (p)
+ (treesit-parser-embed-level
+ (or (car-safe p) p)))
+ (lambda (a b)
+ (> (or a 0) (or b 0)))
+ res)))
+
(defun treesit-local-parsers-at (&optional pos language with-host)
"Return all the local parsers at POS.
@@ -862,19 +892,7 @@ treesit-local-parsers-at
If WITH-HOST is non-nil, return a list of (PARSER . HOST-PARSER)
instead. HOST-PARSER is the host parser which created the local
PARSER."
- (let ((res nil))
- ;; Refer to (ref:local-parser-overlay) for more explanation of local
- ;; parser overlays.
- (dolist (ov (overlays-at (or pos (point))))
- (let ((parser (overlay-get ov 'treesit-parser))
- (host-parser (overlay-get ov 'treesit-host-parser))
- (local-p (overlay-get ov 'treesit-parser-local-p)))
- (when (and parser host-parser local-p
- (or (null language)
- (eq (treesit-parser-language parser)
- language)))
- (push (if with-host (cons parser host-parser) parser) res))))
- (nreverse res)))
+ (treesit-parsers-at pos language with-host t))
(defun treesit-local-parsers-on (&optional beg end language with-host)
"Return the list of local parsers that cover the region between BEG and END.
@@ -3135,9 +3159,7 @@ treesit-up-list
(setq parent (treesit-parent-until parent pred)))
(unless parent
- (let ((parsers (seq-keep (lambda (o)
- (overlay-get o 'treesit-host-parser))
- (overlays-at (point) t))))
+ (let ((parsers (mapcar #'cdr (treesit-parsers-at (point) nil t))))
(while (and (not parent) parsers)
(setq parent (treesit-parent-until
(treesit-node-at (point) (car parsers)) pred)
@@ -3887,9 +3929,9 @@ treesit-simple-imenu
(lambda (entry)
(let* ((lang (car entry))
(settings (cdr entry))
- (global-parser (car (treesit-parser-list nil lang)))
- (local-parsers
- (treesit-parser-list nil lang 'embedded)))
+ (global-parser (or (car (treesit-parsers-at nil lang))
+ treesit-primary-parser))
+ (local-parsers (treesit-local-parsers-at nil lang)))
(cons (treesit-language-display-name lang)
;; No one says you can't have both global and local
;; parsers for the same language. E.g., Rust uses
@@ -4029,9 +4074,7 @@ treesit-outline-level
(setq level (1+ level)))
;; Continue counting the host nodes.
- (dolist (parser (seq-keep (lambda (o)
- (overlay-get o 'treesit-host-parser))
- (overlays-at (point) t)))
+ (dolist (parser (mapcar #'cdr (treesit-parsers-at (point) nil t)))
(let* ((node (treesit-node-at (point) parser))
(lang (treesit-parser-language parser))
(pred (alist-get lang treesit-aggregated-outline-predicate)))
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 01 Apr 2025 00:28:03 GMT) Full text and rfc822 format available.Message #14 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Yuan Fu <casouri <at> gmail.com> To: Juri Linkov <juri <at> linkov.net> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Mon, 31 Mar 2025 17:27:20 -0700
> On Mar 31, 2025, at 9:53 AM, Juri Linkov <juri <at> linkov.net> wrote:
>
>>> Do we still need such complicated functions as
>>> mhtml-ts-mode--language-at-point, js--treesit-language-at-point, etc.
>>> that duplicate the rules from 'treesit-range-rules'
>>> when now the default language-at-point function could be implemented
>>> just as
>>>
>>> (treesit-parser-language
>>> (or (seq-some (lambda (o) (overlay-get o 'treesit-parser))
>>> (overlays-at (point) t))
>>> treesit-primary-parser))
>>
>> Yeah, we can provide a default language-at-point function now that
>> determines the “most relevant parser” by embed level. But we should keep
>> treesit-language-at-point-function because a) it’s already in a release
>> version and b) we want major modes to be able to customize what parser to
>> pick at any given point.
>
> Ok, so here is a complete patch:
Thanks for working on this!
>
> diff --git a/lisp/textmodes/mhtml-ts-mode.el b/lisp/textmodes/mhtml-ts-mode.el
> index 22c0455a4ee..fdb286fc8c4 100644
> --- a/lisp/textmodes/mhtml-ts-mode.el
> +++ b/lisp/textmodes/mhtml-ts-mode.el
> @@ -221,21 +222,6 @@ mhtml-ts-mode-menu
> "Menu bar for `mhtml-ts-mode'."
> css-mode--menu)
>
> -;; To enable some basic treesiter functionality, you should define
> -;; a function that recognizes which grammar is used at-point.
> -;; This function should be assigned to `treesit-language-at-point-function'
> -(defun mhtml-ts-mode--language-at-point (point)
> - "Return the language at POINT assuming the point is within a HTML buffer."
> - (let* ((node (treesit-node-at point 'html))
> - (parent (treesit-node-parent node))
> - (node-query (format "(%s (%s))"
> - (treesit-node-type parent)
> - (treesit-node-type node))))
> - (cond
> - ((equal "(script_element (raw_text))" node-query) (js--treesit-language-at-point point))
> - ((equal "(style_element (raw_text))" node-query) 'css)
> - (t 'html))))
> -
> ;; Custom font-lock function that's used to apply color to css color
> ;; The signature of the function should be conforming to signature
> ;; QUERY-SPEC required by `treesit-font-lock-rules'.
> @@ -448,7 +434,7 @@ mhtml-ts-mode-flymake-mhtml
>
> ;;;###autoload
> (define-derived-mode mhtml-ts-mode html-ts-mode
> - '("HTML+" (:eval (let ((lang (mhtml-ts-mode--language-at-point (point))))
> + '("HTML+" (:eval (let ((lang (treesit-language-at (point))))
> (cond ((eq lang 'html) "")
> ((eq lang 'javascript) "JS")
> ((eq lang 'css) "CSS")))))
> @@ -527,10 +520,6 @@ mhtml-ts-mode
> (setq-local c-ts-common--comment-regexp
> js--treesit-jsdoc-comment-regexp))
>
> -
> - ;; Many treesit functions need to know the language at-point.
> - ;; So you should define such a function.
> - (setq-local treesit-language-at-point-function #'mhtml-ts-mode--language-at-point)
> (setq-local prettify-symbols-alist mhtml-ts-mode--prettify-symbols-alist)
>
> ;; Indent.
> diff --git a/lisp/treesit.el b/lisp/treesit.el
> index 54c29326df2..75c04912216 100644
> --- a/lisp/treesit.el
> +++ b/lisp/treesit.el
> @@ -185,15 +185,15 @@ treesit-language-at
> This function assumes that parser ranges are up-to-date. It
> returns the return value of `treesit-language-at-point-function'
> if it's non-nil, otherwise it returns the language of the first
> -parser in `treesit-parser-list', or nil if there is no parser.
> +parser from `treesit-parsers-at', or the primary parser.
If we handle the fallback case in treesit-language-at directly, rather than defining a separate default function, IMO we should describe the fallback behavior as the default. So something like “Return the language at POS. When there are multiple parsers that covers POS, determine the most relevant parser (hence language) by their embed level. If treesit-language-at-point-function is non-nil, return the return value of that function instead.” (Not saying we should use this exact docstring but to illustrate the point.)
>
> -In a multi-language buffer, make sure
> -`treesit-language-at-point-function' is implemented! Otherwise
> -`treesit-language-at' wouldn't return the correct result."
> +In a multi-language buffer, optionally you can implement
> +`treesit-language-at-point-function' to return more correct result."
> (if treesit-language-at-point-function
> (funcall treesit-language-at-point-function position)
> - (when-let* ((parser (car (treesit-parser-list))))
> - (treesit-parser-language parser))))
> + (treesit-parser-language
> + (or (car (treesit-parsers-at position))
> + treesit-primary-parser))))
>
> ;;; Node API supplement
>
> @@ -247,8 +247,9 @@ treesit-node-at
> (parser-or-lang
> (let* ((local-parser (car (treesit-local-parsers-at
> pos parser-or-lang)))
> - (global-parser (car (treesit-parser-list
> - nil parser-or-lang)))
> + (global-parser (or (car (treesit-parsers-at
> + pos parser-or-lang))
> + treesit-primary-parser))
> (parser (or local-parser global-parser)))
> (when parser
> (treesit-parser-root-node parser))))
> @@ -267,13 +268,10 @@ treesit-node-at
> (local-parser
> ;; Find the local parser with highest
> ;; embed-level at point.
> - (car (seq-sort-by #'treesit-parser-embed-level
> - (lambda (a b)
> - (> (or a 0) (or b 0)))
> - (treesit-local-parsers-at
> - pos lang))))
> - (global-parser (car (treesit-parser-list
> - nil lang)))
> + (car (treesit-local-parsers-at pos lang)))
> + (global-parser (or (car (treesit-parsers-at
> + pos lang))
> + treesit-primary-parser))
> (parser (or local-parser global-parser)))
> (when parser
> (treesit-parser-root-node parser))))))
> @@ -851,6 +849,38 @@ treesit--clip-ranges
> if (<= start (car range) (cdr range) end)
> collect range))
>
> +(defun treesit-parsers-at (&optional pos language with-host local-only)
> + "Return all the non-primary parsers at POS.
I get why you used treesit-parsers-at rather than a more technically correct name like treesit-non-primary-parsers-at, or treesit-embeded-parsers-at. But this is confusing for not much benefit IMO. I suggest either use treesit-parsers-at, and add an optional argument exclude-primary; or use a more correct name and don’t include the primary parser. Also, instead of using LOCAL-ONLY, we might be more future-prove to use a ONLY parameter, and let user pass ‘local to mean local-only. This way we can add the option of returning only non-local non-primary parsers in the future, should the need arises (it already kinda does in treesit-node-at and treesit-simple-imenu, technically global-parser should be picked from non-local parsers).
> +
> +POS defaults to point.
> +If LANGUAGE is non-nil, only return parsers for LANGUAGE.
> +
> +If WITH-HOST is non-nil, return a list of (PARSER . HOST-PARSER)
> +instead. HOST-PARSER is the host parser which created the PARSER.
> +
> +If LOCAL-ONLY is non-nil, return only local parsers.
> +Local parsers are those which only parse a limited region marked
> +by an overlay with non-nil `treesit-parser' property."
> + (let ((res nil))
> + ;; Refer to (ref:local-parser-overlay) for more explanation of local
> + ;; parser overlays.
> + (dolist (ov (overlays-at (or pos (point))))
> + (when-let* ((parser (overlay-get ov 'treesit-parser))
> + (host-parser (or (null with-host)
> + (overlay-get ov 'treesit-host-parser)))
> + (_ (or (null local-only)
> + (overlay-get ov 'treesit-parser-local-p)))
> + (_ (or (null language)
> + (eq (treesit-parser-language parser)
> + language))))
> + (push (if with-host (cons parser host-parser) parser) res)))
> + (seq-sort-by (lambda (p)
> + (treesit-parser-embed-level
> + (or (car-safe p) p)))
> + (lambda (a b)
> + (> (or a 0) (or b 0)))
> + res)))
> +
> (defun treesit-local-parsers-at (&optional pos language with-host)
> "Return all the local parsers at POS.
>
> @@ -862,19 +892,7 @@ treesit-local-parsers-at
> If WITH-HOST is non-nil, return a list of (PARSER . HOST-PARSER)
> instead. HOST-PARSER is the host parser which created the local
> PARSER."
> - (let ((res nil))
> - ;; Refer to (ref:local-parser-overlay) for more explanation of local
> - ;; parser overlays.
> - (dolist (ov (overlays-at (or pos (point))))
> - (let ((parser (overlay-get ov 'treesit-parser))
> - (host-parser (overlay-get ov 'treesit-host-parser))
> - (local-p (overlay-get ov 'treesit-parser-local-p)))
> - (when (and parser host-parser local-p
> - (or (null language)
> - (eq (treesit-parser-language parser)
> - language)))
> - (push (if with-host (cons parser host-parser) parser) res))))
> - (nreverse res)))
> + (treesit-parsers-at pos language with-host t))
>
> (defun treesit-local-parsers-on (&optional beg end language with-host)
> "Return the list of local parsers that cover the region between BEG and END.
> @@ -3135,9 +3159,7 @@ treesit-up-list
> (setq parent (treesit-parent-until parent pred)))
>
> (unless parent
> - (let ((parsers (seq-keep (lambda (o)
> - (overlay-get o 'treesit-host-parser))
> - (overlays-at (point) t))))
> + (let ((parsers (mapcar #'cdr (treesit-parsers-at (point) nil t))))
> (while (and (not parent) parsers)
> (setq parent (treesit-parent-until
> (treesit-node-at (point) (car parsers)) pred)
> @@ -3887,9 +3929,9 @@ treesit-simple-imenu
> (lambda (entry)
> (let* ((lang (car entry))
> (settings (cdr entry))
> - (global-parser (car (treesit-parser-list nil lang)))
> - (local-parsers
> - (treesit-parser-list nil lang 'embedded)))
> + (global-parser (or (car (treesit-parsers-at nil lang))
> + treesit-primary-parser))
> + (local-parsers (treesit-local-parsers-at nil lang)))
> (cons (treesit-language-display-name lang)
> ;; No one says you can't have both global and local
> ;; parsers for the same language. E.g., Rust uses
> @@ -4029,9 +4074,7 @@ treesit-outline-level
> (setq level (1+ level)))
>
> ;; Continue counting the host nodes.
> - (dolist (parser (seq-keep (lambda (o)
> - (overlay-get o 'treesit-host-parser))
> - (overlays-at (point) t)))
> + (dolist (parser (mapcar #'cdr (treesit-parsers-at (point) nil t)))
> (let* ((node (treesit-node-at (point) parser))
> (lang (treesit-parser-language parser))
> (pred (alist-get lang treesit-aggregated-outline-predicate)))
We should also update manual sections that mentions treesit-language-at.
Yuan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 01 Apr 2025 16:56:02 GMT) Full text and rfc822 format available.Message #17 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Tue, 01 Apr 2025 19:53:37 +0300
[Message part 1 (text/plain, inline)]
>> @@ -185,15 +185,15 @@ treesit-language-at >> This function assumes that parser ranges are up-to-date. It >> returns the return value of `treesit-language-at-point-function' >> if it's non-nil, otherwise it returns the language of the first >> -parser in `treesit-parser-list', or nil if there is no parser. >> +parser from `treesit-parsers-at', or the primary parser. > > If we handle the fallback case in treesit-language-at directly, rather than > defining a separate default function, IMO we should describe the fallback > behavior as the default. So something like “Return the language at > POS. When there are multiple parsers that covers POS, determine the most > relevant parser (hence language) by their embed level. If > treesit-language-at-point-function is non-nil, return the return value of > that function instead.” (Not saying we should use this exact docstring but > to illustrate the point.) Now fixed in the next patch. >> +(defun treesit-parsers-at (&optional pos language with-host local-only) >> + "Return all the non-primary parsers at POS. > > I get why you used treesit-parsers-at rather than a more technically > correct name like treesit-non-primary-parsers-at, or > treesit-embeded-parsers-at. But this is confusing for not much benefit > IMO. I suggest either use treesit-parsers-at, and add an optional argument > exclude-primary; or use a more correct name and don’t include the primary > parser. Also, instead of using LOCAL-ONLY, we might be more future-prove to > use a ONLY parameter, and let user pass ‘local to mean local-only. This way > we can add the option of returning only non-local non-primary parsers in > the future, should the need arises (it already kinda does in > treesit-node-at and treesit-simple-imenu, technically global-parser should > be picked from non-local parsers). I can't find a better function signature than to add the argument ONLY with the following combinations of symbols for its list, wrt existing uses of treesit-parsers-at: nil treesit-language-at (nil returns all parsers) (local) treesit-local-parsers-at (primary global) treesit-node-at, treesit-simple-imenu (global local) treesit-outline-level, treesit-up-list > We should also update manual sections that mentions treesit-language-at. Will do when everything is ok with this patch:
[treesit-parsers-at.patch (text/x-diff, inline)]
diff --git a/lisp/treesit.el b/lisp/treesit.el
index 54c29326df2..4ba46e5b7ce 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -182,18 +182,14 @@ treesit-language-at-point-function
(defun treesit-language-at (position)
"Return the language at POSITION.
-This function assumes that parser ranges are up-to-date. It
-returns the return value of `treesit-language-at-point-function'
-if it's non-nil, otherwise it returns the language of the first
-parser in `treesit-parser-list', or nil if there is no parser.
-
-In a multi-language buffer, make sure
-`treesit-language-at-point-function' is implemented! Otherwise
-`treesit-language-at' wouldn't return the correct result."
+When there are multiple parsers that covers POSITION, determine
+the most relevant parser (hence language) by their embed level.
+If `treesit-language-at-point-function' is non-nil, return
+the return value of that function instead."
(if treesit-language-at-point-function
(funcall treesit-language-at-point-function position)
- (when-let* ((parser (car (treesit-parser-list))))
- (treesit-parser-language parser))))
+ (treesit-parser-language
+ (car (treesit-parsers-at position)))))
;;; Node API supplement
@@ -247,8 +243,9 @@ treesit-node-at
(parser-or-lang
(let* ((local-parser (car (treesit-local-parsers-at
pos parser-or-lang)))
- (global-parser (car (treesit-parser-list
- nil parser-or-lang)))
+ (global-parser (car (treesit-parsers-at
+ pos parser-or-lang nil
+ '(primary global))))
(parser (or local-parser global-parser)))
(when parser
(treesit-parser-root-node parser))))
@@ -267,13 +264,10 @@ treesit-node-at
(local-parser
;; Find the local parser with highest
;; embed-level at point.
- (car (seq-sort-by #'treesit-parser-embed-level
- (lambda (a b)
- (> (or a 0) (or b 0)))
- (treesit-local-parsers-at
- pos lang))))
- (global-parser (car (treesit-parser-list
- nil lang)))
+ (car (treesit-local-parsers-at pos lang)))
+ (global-parser (car (treesit-parsers-at
+ pos lang nil
+ '(primary global))))
(parser (or local-parser global-parser)))
(when parser
(treesit-parser-root-node parser))))))
@@ -851,6 +845,56 @@ treesit--clip-ranges
if (<= start (car range) (cdr range) end)
collect range))
+(defun treesit-parsers-at (&optional pos language with-host only)
+ "Return all parsers at POS.
+
+POS defaults to point. The returned parsers are sorted by
+the decreasing embed level.
+
+If LANGUAGE is non-nil, only return parsers for LANGUAGE.
+
+If WITH-HOST is non-nil, return a list of (PARSER . HOST-PARSER)
+instead. HOST-PARSER is the host parser which created the PARSER.
+
+If ONLY is nil, return all parsers including the primary parser.
+
+The argument ONLY can be a list of symbols that specify what
+parsers to include in the return value.
+
+If ONLY contains the symbol `local', include local parsers.
+Local parsers are those which only parse a limited region marked
+by an overlay with non-nil `treesit-parser' property.
+
+If ONLY contains the symbol `global', include non-local parsers
+excluding the primary parser.
+
+If ONLY contains the symbol `primary', include the primary parser."
+ (let ((res nil))
+ ;; Refer to (ref:local-parser-overlay) for more explanation of local
+ ;; parser overlays.
+ (dolist (ov (overlays-at (or pos (point))))
+ (when-let* ((parser (overlay-get ov 'treesit-parser))
+ (host-parser (or (null with-host)
+ (overlay-get ov 'treesit-host-parser)))
+ (_ (or (null language)
+ (eq (treesit-parser-language parser)
+ language)))
+ (_ (or (null only)
+ (and (memq 'local only) (memq 'global only))
+ (and (memq 'local only)
+ (overlay-get ov 'treesit-parser-local-p))
+ (and (memq 'global only)
+ (not (overlay-get ov 'treesit-parser-local-p))))))
+ (push (if with-host (cons parser host-parser) parser) res)))
+ (when (or (null only) (memq 'primary only))
+ (setq res (cons treesit-primary-parser res)))
+ (seq-sort-by (lambda (p)
+ (treesit-parser-embed-level
+ (or (car-safe p) p)))
+ (lambda (a b)
+ (> (or a 0) (or b 0)))
+ res)))
+
(defun treesit-local-parsers-at (&optional pos language with-host)
"Return all the local parsers at POS.
@@ -862,19 +906,7 @@ treesit-local-parsers-at
If WITH-HOST is non-nil, return a list of (PARSER . HOST-PARSER)
instead. HOST-PARSER is the host parser which created the local
PARSER."
- (let ((res nil))
- ;; Refer to (ref:local-parser-overlay) for more explanation of local
- ;; parser overlays.
- (dolist (ov (overlays-at (or pos (point))))
- (let ((parser (overlay-get ov 'treesit-parser))
- (host-parser (overlay-get ov 'treesit-host-parser))
- (local-p (overlay-get ov 'treesit-parser-local-p)))
- (when (and parser host-parser local-p
- (or (null language)
- (eq (treesit-parser-language parser)
- language)))
- (push (if with-host (cons parser host-parser) parser) res))))
- (nreverse res)))
+ (treesit-parsers-at pos language with-host '(local)))
(defun treesit-local-parsers-on (&optional beg end language with-host)
"Return the list of local parsers that cover the region between BEG and END.
@@ -3135,9 +3173,7 @@ treesit-up-list
(setq parent (treesit-parent-until parent pred)))
(unless parent
- (let ((parsers (seq-keep (lambda (o)
- (overlay-get o 'treesit-host-parser))
- (overlays-at (point) t))))
+ (let ((parsers (mapcar #'cdr (treesit-parsers-at (point) nil t '(global local)))))
(while (and (not parent) parsers)
(setq parent (treesit-parent-until
(treesit-node-at (point) (car parsers)) pred)
@@ -3887,9 +3943,8 @@ treesit-simple-imenu
(lambda (entry)
(let* ((lang (car entry))
(settings (cdr entry))
- (global-parser (car (treesit-parser-list nil lang)))
- (local-parsers
- (treesit-parser-list nil lang 'embedded)))
+ (global-parser (car (treesit-parsers-at nil lang nil '(primary global))))
+ (local-parsers (treesit-local-parsers-at nil lang)))
(cons (treesit-language-display-name lang)
;; No one says you can't have both global and local
;; parsers for the same language. E.g., Rust uses
@@ -4029,9 +4087,7 @@ treesit-outline-level
(setq level (1+ level)))
;; Continue counting the host nodes.
- (dolist (parser (seq-keep (lambda (o)
- (overlay-get o 'treesit-host-parser))
- (overlays-at (point) t)))
+ (dolist (parser (mapcar #'cdr (treesit-parsers-at (point) nil t '(global local))))
(let* ((node (treesit-node-at (point) parser))
(lang (treesit-parser-language parser))
(pred (alist-get lang treesit-aggregated-outline-predicate)))
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Wed, 02 Apr 2025 00:00:03 GMT) Full text and rfc822 format available.Message #20 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Yuan Fu <casouri <at> gmail.com> To: Juri Linkov <juri <at> linkov.net> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Tue, 1 Apr 2025 16:59:03 -0700
> On Apr 1, 2025, at 9:53 AM, Juri Linkov <juri <at> linkov.net> wrote: > >>> @@ -185,15 +185,15 @@ treesit-language-at >>> This function assumes that parser ranges are up-to-date. It >>> returns the return value of `treesit-language-at-point-function' >>> if it's non-nil, otherwise it returns the language of the first >>> -parser in `treesit-parser-list', or nil if there is no parser. >>> +parser from `treesit-parsers-at', or the primary parser. >> >> If we handle the fallback case in treesit-language-at directly, rather than >> defining a separate default function, IMO we should describe the fallback >> behavior as the default. So something like “Return the language at >> POS. When there are multiple parsers that covers POS, determine the most >> relevant parser (hence language) by their embed level. If >> treesit-language-at-point-function is non-nil, return the return value of >> that function instead.” (Not saying we should use this exact docstring but >> to illustrate the point.) > > Now fixed in the next patch. > >>> +(defun treesit-parsers-at (&optional pos language with-host local-only) >>> + "Return all the non-primary parsers at POS. >> >> I get why you used treesit-parsers-at rather than a more technically >> correct name like treesit-non-primary-parsers-at, or >> treesit-embeded-parsers-at. But this is confusing for not much benefit >> IMO. I suggest either use treesit-parsers-at, and add an optional argument >> exclude-primary; or use a more correct name and don’t include the primary >> parser. Also, instead of using LOCAL-ONLY, we might be more future-prove to >> use a ONLY parameter, and let user pass ‘local to mean local-only. This way >> we can add the option of returning only non-local non-primary parsers in >> the future, should the need arises (it already kinda does in >> treesit-node-at and treesit-simple-imenu, technically global-parser should >> be picked from non-local parsers). > > I can't find a better function signature than to add the argument ONLY > with the following combinations of symbols for its list, wrt existing uses > of treesit-parsers-at: > > nil treesit-language-at (nil returns all parsers) > (local) treesit-local-parsers-at > (primary global) treesit-node-at, treesit-simple-imenu > (global local) treesit-outline-level, treesit-up-list > >> We should also update manual sections that mentions treesit-language-at. > > Will do when everything is ok with this patch: > Thanks! LGTM. Yuan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Fri, 04 Apr 2025 16:16:02 GMT) Full text and rfc822 format available.Message #23 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Fri, 04 Apr 2025 19:14:11 +0300
>>> We should also update manual sections that mentions treesit-language-at. >> >> Will do when everything is ok with this patch: > > Thanks! LGTM. Unfortunately, I discovered that it causes treesit-node-outdated errors. This is because overlays are updated by 'pre-redisplay-functions' later than outline-minor-mode uses them by 'after-change-functions', since 'after-change-functions' runs before 'pre-redisplay-functions'. The reproducible test case: with the latest patch for treesit.el, and another patch that removes 'mhtml-ts-mode--language-at-point' from mhtml-ts-mode.el, open a simple html file like this: <!DOCTYPE html> <html> <head> <title>test</title> </head> <body> </body> </html> then type: 1. M-x load-library RET treesit-x RET 2. M-x liquid-generic-ts-mode RET 3. M-x outline-minor-mode RET 4. move point before </html> 5. type SPC then with non-nil debug-on-error: Debugger entered--Lisp error: (treesit-node-outdated #<treesit-node-outdated>) treesit-node-enclosed-p(#<treesit-node-outdated> #<treesit-node element in 17-82>) treesit-navigate-thing(73 1 beg html-ts-mode--outline-predicate) treesit-outline-search(#<marker at 82 in test.html>) outline-map-region(#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_46> 74 82) outline--fix-up-all-buttons(74 82) outline--fix-buttons-after-change(74 75 0) self-insert-command(1 32) funcall-interactively(self-insert-command 1 32) command-execute(self-insert-command)
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Sat, 05 Apr 2025 18:39:02 GMT) Full text and rfc822 format available.Message #26 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Sat, 05 Apr 2025 21:35:54 +0300
> Unfortunately, I discovered that it causes treesit-node-outdated errors.
>
> This is because overlays are updated by 'pre-redisplay-functions'
> later than outline-minor-mode uses them by 'after-change-functions',
> since 'after-change-functions' runs before 'pre-redisplay-functions'.
This fixes the problem:
(defun outline--fix-buttons-after-change (beg end _len)
+ (when (fboundp 'treesit-update-ranges)
+ (treesit-update-ranges beg end))
;; Handle whole lines
(save-excursion (goto-char beg) (setq beg (pos-bol)))
(save-excursion (goto-char end) (setq end (pos-eol)))
But can we do better? I see that 'treesit-major-mode-setup'
adds the notifier
(treesit-parser-add-notifier
treesit-primary-parser #'treesit--font-lock-mark-ranges-to-fontify)
to 'after_change_functions'. But why the treesit
'after_change_functions' notifier is called after the
'after-change-functions' hook? Can we change their order?
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Sun, 06 Apr 2025 04:52:02 GMT) Full text and rfc822 format available.Message #29 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Yuan Fu <casouri <at> gmail.com> To: Juri Linkov <juri <at> linkov.net> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Sat, 5 Apr 2025 21:51:29 -0700
> On Apr 5, 2025, at 11:35 AM, Juri Linkov <juri <at> linkov.net> wrote: > >> Unfortunately, I discovered that it causes treesit-node-outdated errors. >> >> This is because overlays are updated by 'pre-redisplay-functions' >> later than outline-minor-mode uses them by 'after-change-functions', >> since 'after-change-functions' runs before 'pre-redisplay-functions'. > > This fixes the problem: > > (defun outline--fix-buttons-after-change (beg end _len) > + (when (fboundp 'treesit-update-ranges) > + (treesit-update-ranges beg end)) > ;; Handle whole lines > (save-excursion (goto-char beg) (setq beg (pos-bol))) > (save-excursion (goto-char end) (setq end (pos-eol))) Using treesit-update-ranges is the right approach. But is there no better place to put it? I assume there should be a function that involves tree-sitter, in front of which you can put this call. > But can we do better? I see that 'treesit-major-mode-setup' > adds the notifier > > (treesit-parser-add-notifier > treesit-primary-parser #'treesit--font-lock-mark-ranges-to-fontify) > > to 'after_change_functions'. But why the treesit > 'after_change_functions' notifier is called after the > 'after-change-functions' hook? Can we change their order? Because we parse lazily. And I should’ve called them after-parse functions in the docstring (good thing I named them notifiers, not after-change-functions)—they get called after the parser re-parses, but the parser doesn’t have to re-parse immediately after a buffer change. It only re-parses when some Lisp asks for a node from the parse tree. In practice, that usually happens in pre-redisplay-function where we set ranges for embedded parsers, which causes the primary parser to re-parse and return a root node for querying. As a (intended) side-effect, the notifiers get called and mark relevant regions to be re-fontified. Yuan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Sun, 06 Apr 2025 06:57:01 GMT) Full text and rfc822 format available.Message #32 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Sun, 06 Apr 2025 09:51:27 +0300
>>> Unfortunately, I discovered that it causes treesit-node-outdated errors.
>>>
>>> This is because overlays are updated by 'pre-redisplay-functions'
>>> later than outline-minor-mode uses them by 'after-change-functions',
>>> since 'after-change-functions' runs before 'pre-redisplay-functions'.
>>
>> This fixes the problem:
>>
>> (defun outline--fix-buttons-after-change (beg end _len)
>> + (when (fboundp 'treesit-update-ranges)
>> + (treesit-update-ranges beg end))
>> ;; Handle whole lines
>> (save-excursion (goto-char beg) (setq beg (pos-bol)))
>> (save-excursion (goto-char end) (setq end (pos-eol)))
>
> Using treesit-update-ranges is the right approach. But is there no
> better place to put it? I assume there should be a function that
> involves tree-sitter, in front of which you can put this call.
'treesit-outline-search' has no information about the positions of
the beginning and end of the range of changed text. Only
'after-change-functions' has access to this information.
So what we could do is this, and I've tested that this approach works:
#+begin_src diff
diff --git a/lisp/treesit.el b/lisp/treesit.el
index 07861603244..c77b28c4647 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -4044,6 +4112,10 @@ treesit-outline-level
level))
+(defun treesit--after-change (beg end _len)
+ "Force updating the ranges after each text change."
+ (treesit-update-ranges beg end))
+
;;; Hideshow mode
(defun treesit-hs-block-end ()
@@ -4338,7 +4410,16 @@ treesit-major-mode-setup
(setq treesit-outline-predicate
#'treesit-outline-predicate--from-imenu))
(setq-local outline-search-function #'treesit-outline-search
- outline-level #'treesit-outline-level))
+ outline-level #'treesit-outline-level)
+ (add-hook 'outline-minor-mode-hook
+ (lambda ()
+ (if (bound-and-true-p outline-minor-mode)
+ (add-hook 'after-change-functions
+ #'treesit--after-change
+ 0 t)
+ (remove-hook 'after-change-functions
+ #'treesit--after-change t)))
+ nil t))
;; Remove existing local parsers.
(dolist (ov (overlays-in (point-min) (point-max)))
#+end_src
>> But can we do better? I see that 'treesit-major-mode-setup'
>> adds the notifier
>>
>> (treesit-parser-add-notifier
>> treesit-primary-parser #'treesit--font-lock-mark-ranges-to-fontify)
>>
>> to 'after_change_functions'. But why the treesit
>> 'after_change_functions' notifier is called after the
>> 'after-change-functions' hook? Can we change their order?
>
> Because we parse lazily. And I should’ve called them after-parse functions
> in the docstring (good thing I named them notifiers, not
> after-change-functions)—they get called after the parser re-parses, but the
> parser doesn’t have to re-parse immediately after a buffer change. It only
> re-parses when some Lisp asks for a node from the parse tree.
>
> In practice, that usually happens in pre-redisplay-function where we set
> ranges for embedded parsers, which causes the primary parser to re-parse
> and return a root node for querying. As a (intended) side-effect, the
> notifiers get called and mark relevant regions to be re-fontified.
Would it be possible on the first attempt of Lisp code to access a node
to detect such a situation that there are some outdated nodes and
to update their ranges automatically?
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 08 Apr 2025 17:46:01 GMT) Full text and rfc822 format available.Message #35 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Tue, 08 Apr 2025 20:40:44 +0300
close 77256 31.0.50 thanks >>> We should also update manual sections that mentions treesit-language-at. >> >> Will do when everything is ok with this patch: > > Thanks! LGTM. Now pushed together with updates in the manual. Please check the commit 5e0daa1ef77.
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 08 Apr 2025 17:48:02 GMT) Full text and rfc822 format available.Message #38 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Tue, 08 Apr 2025 20:44:05 +0300
>> Using treesit-update-ranges is the right approach. But is there no >> better place to put it? I assume there should be a function that >> involves tree-sitter, in front of which you can put this call. > > 'treesit-outline-search' has no information about the positions of > the beginning and end of the range of changed text. Only > 'after-change-functions' has access to this information. > > So what we could do is this, and I've tested that this approach works: This is pushed as well.
Juri Linkov <juri <at> linkov.net>
to control <at> debbugs.gnu.org.
(Tue, 08 Apr 2025 17:48:02 GMT) Full text and rfc822 format available.bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 08 Apr 2025 18:23:02 GMT) Full text and rfc822 format available.Message #43 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Juri Linkov <juri <at> linkov.net> Subject: Re: bug#77256: Treesit language-at-point Date: Tue, 08 Apr 2025 14:22:01 -0400
>> (if treesit-language-at-point-function
>> (funcall treesit-language-at-point-function position)
BTW, can we please avoid this anti-pattern?
Instead, we should unconditionally call
`treesit-language-at-point-function` and give a non-nil default value to
this variable.
This would make sure the `treesit-language-at-point-function` is good
enough to implement the default behavior and it makes it possible to use
`add-function` on `treesit-language-at-point-function`.
Stefan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 08 Apr 2025 18:49:01 GMT) Full text and rfc822 format available.Message #46 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: Yuan Fu <casouri <at> gmail.com>, Vincenzo Pupillo <v.pupillo <at> gmail.com>, 77256 <at> debbugs.gnu.org Subject: Re: bug#77256: Treesit language-at-point Date: Tue, 08 Apr 2025 21:43:20 +0300
>>> (if treesit-language-at-point-function >>> (funcall treesit-language-at-point-function position) > > BTW, can we please avoid this anti-pattern? > Instead, we should unconditionally call > `treesit-language-at-point-function` and give a non-nil default value to > this variable. > > This would make sure the `treesit-language-at-point-function` is good > enough to implement the default behavior and it makes it possible to use > `add-function` on `treesit-language-at-point-function`. Unfortunately, it's too late to add a non-nil default value. I already tried to do the same by adding 'forward-sexp-default-function' to 'forward-sexp-function', but failed spectacularly (bug#70426), because there are existing uses that expect the default value to be nil.
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 08 Apr 2025 19:53:01 GMT) Full text and rfc822 format available.Message #49 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Juri Linkov <juri <at> linkov.net> Cc: Yuan Fu <casouri <at> gmail.com>, Vincenzo Pupillo <v.pupillo <at> gmail.com>, 77256 <at> debbugs.gnu.org Subject: Re: bug#77256: Treesit language-at-point Date: Tue, 08 Apr 2025 15:52:08 -0400
>> This would make sure the `treesit-language-at-point-function` is good
>> enough to implement the default behavior and it makes it possible to use
>> `add-function` on `treesit-language-at-point-function`.
>
> Unfortunately, it's too late to add a non-nil default value.
Is it?
I can't find any code out there that seems to care what is its default
value (all users set it but doesn't look at it, AFAICT).
> I already tried to do the same by adding 'forward-sexp-default-function'
> to 'forward-sexp-function', but failed spectacularly (bug#70426),
Yes, that one is much harder (it's one of the first I tried to change),
Stefan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Wed, 09 Apr 2025 06:26:02 GMT) Full text and rfc822 format available.Message #52 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: Yuan Fu <casouri <at> gmail.com>, Vincenzo Pupillo <v.pupillo <at> gmail.com>, 77256 <at> debbugs.gnu.org Subject: Re: bug#77256: Treesit language-at-point Date: Wed, 09 Apr 2025 09:18:52 +0300
[Message part 1 (text/plain, inline)]
>>> This would make sure the `treesit-language-at-point-function` is good
>>> enough to implement the default behavior and it makes it possible to use
>>> `add-function` on `treesit-language-at-point-function`.
>>
>> Unfortunately, it's too late to add a non-nil default value.
>
> Is it?
> I can't find any code out there that seems to care what is its default
> value (all users set it but doesn't look at it, AFAICT).
This case looks problematic:
;; LANG can be nil. We don't want to use the fallback
;; in `treesit-language-at', so here we call
;; `treesit-language-at-point-function' directly.
(let* ((lang (and treesit-language-at-point-function
(funcall treesit-language-at-point-function
pos)))
But I believe this whole 'let*' can be replaced by just
the 'treesit-parsers-at' call. I hope Yuan could confirm this.
>> I already tried to do the same by adding 'forward-sexp-default-function'
>> to 'forward-sexp-function', but failed spectacularly (bug#70426),
>
> Yes, that one is much harder (it's one of the first I tried to change),
If no more code outside relies on the default value,
the default function could be added like this:
[treesit-language-at-point-default.patch (text/x-diff, inline)]
diff --git a/lisp/treesit.el b/lisp/treesit.el index 8e57a6dae14..a010ff24afb 100644 --- a/lisp/treesit.el +++ b/lisp/treesit.el @@ -165,7 +165,8 @@ treesit-primary-parser The primary parser should be a parser that parses the entire buffer, as opposed to embedded parsers which parses only part of the buffer.") -(defvar-local treesit-language-at-point-function nil +(defvar-local treesit-language-at-point-function + #'treesit-language-at-point-default "A function that returns the language at point. This is used by `treesit-language-at', which is used by various functions to determine which parser to use at point. @@ -183,10 +184,13 @@ treesit-language-at the most relevant parser (hence language) by their embed level. If `treesit-language-at-point-function' is non-nil, return the return value of that function instead." - (if treesit-language-at-point-function - (funcall treesit-language-at-point-function position) - (treesit-parser-language - (car (treesit-parsers-at position))))) + (funcall treesit-language-at-point-function position)) + +(defun treesit-language-at-point-default (position) + "Default function for `treesit-language-at-point-function'. +Return the deepest parser by embed level." + (treesit-parser-language + (car (treesit-parsers-at position)))) ;;; Node API supplement
[Message part 3 (text/plain, inline)]
BTW, I noticed now that indentation of the default value
in defvar-local is incorrect and differs from defvar and defcustom.
Here is the fix:
diff --git a/lisp/subr.el b/lisp/subr.el
index be847aab28a..164ed2a6f63 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -197,7 +197,7 @@ defvar-local
Like `defvar' but additionally marks the variable as being automatically
buffer-local wherever it is set.
\n(fn SYMBOL &optional VALUE DOCSTRING)"
- (declare (debug defvar) (doc-string 3) (indent 2))
+ (declare (debug defvar) (doc-string 3) (indent defun))
;; Can't use backquote here, it's too early in the bootstrap.
(let ((value (car-safe args))
(docstring (car-safe (cdr-safe args))))
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Wed, 09 Apr 2025 07:27:01 GMT) Full text and rfc822 format available.Message #55 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Yuan Fu <casouri <at> gmail.com> To: Juri Linkov <juri <at> linkov.net> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com> Subject: Re: bug#77256: Treesit language-at-point Date: Wed, 9 Apr 2025 00:26:04 -0700
> On Apr 5, 2025, at 11:51 PM, Juri Linkov <juri <at> linkov.net> wrote: > >>>> Unfortunately, I discovered that it causes treesit-node-outdated errors. >>>> >>>> This is because overlays are updated by 'pre-redisplay-functions' >>>> later than outline-minor-mode uses them by 'after-change-functions', >>>> since 'after-change-functions' runs before 'pre-redisplay-functions'. >>> >>> This fixes the problem: >>> >>> (defun outline--fix-buttons-after-change (beg end _len) >>> + (when (fboundp 'treesit-update-ranges) >>> + (treesit-update-ranges beg end)) >>> ;; Handle whole lines >>> (save-excursion (goto-char beg) (setq beg (pos-bol))) >>> (save-excursion (goto-char end) (setq end (pos-eol))) >> >> Using treesit-update-ranges is the right approach. But is there no >> better place to put it? I assume there should be a function that >> involves tree-sitter, in front of which you can put this call. > > 'treesit-outline-search' has no information about the positions of > the beginning and end of the range of changed text. Only > 'after-change-functions' has access to this information. > > So what we could do is this, and I've tested that this approach works: > > #+begin_src diff > diff --git a/lisp/treesit.el b/lisp/treesit.el > index 07861603244..c77b28c4647 100644 > --- a/lisp/treesit.el > +++ b/lisp/treesit.el > @@ -4044,6 +4112,10 @@ treesit-outline-level > > level)) > > +(defun treesit--after-change (beg end _len) > + "Force updating the ranges after each text change." > + (treesit-update-ranges beg end)) > + > ;;; Hideshow mode > > (defun treesit-hs-block-end () > @@ -4338,7 +4410,16 @@ treesit-major-mode-setup > (setq treesit-outline-predicate > #'treesit-outline-predicate--from-imenu)) > (setq-local outline-search-function #'treesit-outline-search > - outline-level #'treesit-outline-level)) > + outline-level #'treesit-outline-level) > + (add-hook 'outline-minor-mode-hook > + (lambda () > + (if (bound-and-true-p outline-minor-mode) > + (add-hook 'after-change-functions > + #'treesit--after-change > + 0 t) > + (remove-hook 'after-change-functions > + #'treesit--after-change t))) > + nil t)) > > ;; Remove existing local parsers. > (dolist (ov (overlays-in (point-min) (point-max))) > #+end_src This seems a bit convoluted to me. What do you think about adding some hook or generic function to outline-minor-mode? >>> But can we do better? I see that 'treesit-major-mode-setup' >>> adds the notifier >>> >>> (treesit-parser-add-notifier >>> treesit-primary-parser #'treesit--font-lock-mark-ranges-to-fontify) >>> >>> to 'after_change_functions'. But why the treesit >>> 'after_change_functions' notifier is called after the >>> 'after-change-functions' hook? Can we change their order? >> >> Because we parse lazily. And I should’ve called them after-parse functions >> in the docstring (good thing I named them notifiers, not >> after-change-functions)—they get called after the parser re-parses, but the >> parser doesn’t have to re-parse immediately after a buffer change. It only >> re-parses when some Lisp asks for a node from the parse tree. >> >> In practice, that usually happens in pre-redisplay-function where we set >> ranges for embedded parsers, which causes the primary parser to re-parse >> and return a root node for querying. As a (intended) side-effect, the >> notifiers get called and mark relevant regions to be re-fontified. > > Would it be possible on the first attempt of Lisp code to access a node > to detect such a situation that there are some outdated nodes and > to update their ranges automatically? Probably not, since you need to have correct ranges to know which parser to use to get nodes in the first place. Yuan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Wed, 09 Apr 2025 15:03:02 GMT) Full text and rfc822 format available.Message #58 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Juri Linkov <juri <at> linkov.net> Cc: Yuan Fu <casouri <at> gmail.com>, Vincenzo Pupillo <v.pupillo <at> gmail.com>, 77256 <at> debbugs.gnu.org Subject: Re: bug#77256: Treesit language-at-point Date: Wed, 09 Apr 2025 11:02:28 -0400
>> Is it?
>> I can't find any code out there that seems to care what is its default
>> value (all users set it but doesn't look at it, AFAICT).
>
> This case looks problematic:
>
> ;; LANG can be nil. We don't want to use the fallback
> ;; in `treesit-language-at', so here we call
> ;; `treesit-language-at-point-function' directly.
> (let* ((lang (and treesit-language-at-point-function
> (funcall treesit-language-at-point-function
> pos)))
Ah, indeed you're right.
> But I believe this whole 'let*' can be replaced by just
> the 'treesit-parsers-at' call. I hope Yuan could confirm this.
[ Not familiar enough with this code, so I'll let Yuan chime in. ]
> If no more code outside relies on the default value,
> the default function could be added like this:
[...]
> - (if treesit-language-at-point-function
> - (funcall treesit-language-at-point-function position)
> - (treesit-parser-language
> - (car (treesit-parsers-at position)))))
> + (funcall treesit-language-at-point-function position))
Usually, I use something like:
(funcall (or treesit-language-at-point-function
#'treesit-language-at-point-default)
position)
when I make such a change, so as to preserve some compatibility with
code that sets the var to nil.
> BTW, I noticed now that indentation of the default value
> in defvar-local is incorrect and differs from defvar and defcustom.
> Here is the fix:
>
> diff --git a/lisp/subr.el b/lisp/subr.el
> index be847aab28a..164ed2a6f63 100644
> --- a/lisp/subr.el
> +++ b/lisp/subr.el
> @@ -197,7 +197,7 @@ defvar-local
> Like `defvar' but additionally marks the variable as being automatically
> buffer-local wherever it is set.
> \n(fn SYMBOL &optional VALUE DOCSTRING)"
> - (declare (debug defvar) (doc-string 3) (indent 2))
> + (declare (debug defvar) (doc-string 3) (indent defun))
> ;; Can't use backquote here, it's too early in the bootstrap.
> (let ((value (car-safe args))
> (docstring (car-safe (cdr-safe args))))
Good call, feel free to push, thanks.
Stefan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Wed, 09 Apr 2025 17:25:02 GMT) Full text and rfc822 format available.Message #61 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#77256: Treesit language-at-point Date: Wed, 09 Apr 2025 20:16:19 +0300
[Message part 1 (text/plain, inline)]
>>>>> Unfortunately, I discovered that it causes treesit-node-outdated errors. >>>>> >>>>> This is because overlays are updated by 'pre-redisplay-functions' >>>>> later than outline-minor-mode uses them by 'after-change-functions', >>>>> since 'after-change-functions' runs before 'pre-redisplay-functions'. >>>> >>>> This fixes the problem: >>>> >>>> (defun outline--fix-buttons-after-change (beg end _len) >>>> + (when (fboundp 'treesit-update-ranges) >>>> + (treesit-update-ranges beg end)) >>>> ;; Handle whole lines >>>> (save-excursion (goto-char beg) (setq beg (pos-bol))) >>>> (save-excursion (goto-char end) (setq end (pos-eol))) >>> >>> Using treesit-update-ranges is the right approach. But is there no >>> better place to put it? I assume there should be a function that >>> involves tree-sitter, in front of which you can put this call. >> >> 'treesit-outline-search' has no information about the positions of >> the beginning and end of the range of changed text. Only >> 'after-change-functions' has access to this information. >> >> So what we could do is this, and I've tested that this approach works: >> >> #+begin_src diff >> diff --git a/lisp/treesit.el b/lisp/treesit.el >> index 07861603244..c77b28c4647 100644 >> --- a/lisp/treesit.el >> +++ b/lisp/treesit.el >> @@ -4044,6 +4112,10 @@ treesit-outline-level >> >> level)) >> >> +(defun treesit--after-change (beg end _len) >> + "Force updating the ranges after each text change." >> + (treesit-update-ranges beg end)) >> + >> ;;; Hideshow mode >> >> (defun treesit-hs-block-end () >> @@ -4338,7 +4410,16 @@ treesit-major-mode-setup >> (setq treesit-outline-predicate >> #'treesit-outline-predicate--from-imenu)) >> (setq-local outline-search-function #'treesit-outline-search >> - outline-level #'treesit-outline-level)) >> + outline-level #'treesit-outline-level) >> + (add-hook 'outline-minor-mode-hook >> + (lambda () >> + (if (bound-and-true-p outline-minor-mode) >> + (add-hook 'after-change-functions >> + #'treesit--after-change >> + 0 t) >> + (remove-hook 'after-change-functions >> + #'treesit--after-change t))) >> + nil t)) >> >> ;; Remove existing local parsers. >> (dolist (ov (overlays-in (point-min) (point-max))) >> #+end_src > > This seems a bit convoluted to me. What do you think about adding > some hook or generic function to outline-minor-mode? Is this better?
[outline-after-change.patch (text/x-diff, inline)]
diff --git a/lisp/outline.el b/lisp/outline.el
index 1209b1ce766..ff1322b1e93 100644
--- a/lisp/outline.el
+++ b/lisp/outline.el
@@ -1980,7 +1980,11 @@ outline--fix-up-all-buttons
(outline--insert-button (if close-p 'close 'open))))
(or from (point-min)) (or to (point-max)))))
-(defun outline--fix-buttons-after-change (beg end _len)
+(defvar outline-after-change nil
+ "List of functions to call after each text change in outline-mode.")
+
+(defun outline--fix-buttons-after-change (beg end len)
+ (run-hook-with-args 'outline-after-change beg end len)
;; Handle whole lines
(save-excursion (goto-char beg) (setq beg (pos-bol)))
(save-excursion (goto-char end) (setq end (pos-eol)))
diff --git a/lisp/treesit.el b/lisp/treesit.el
index 8e57a6dae14..05523bc857c 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -4069,7 +4069,8 @@ treesit-outline-level
level))
(defun treesit--after-change (beg end _len)
- "Force updating the ranges after each text change."
+ "Force updating the ranges in BEG...END.
+Expected to be called after each text change."
(treesit-update-ranges beg end))
;;; Hideshow mode
@@ -4367,15 +4368,7 @@ treesit-major-mode-setup
#'treesit-outline-predicate--from-imenu))
(setq-local outline-search-function #'treesit-outline-search
outline-level #'treesit-outline-level)
- (add-hook 'outline-minor-mode-hook
- (lambda ()
- (if (bound-and-true-p outline-minor-mode)
- (add-hook 'after-change-functions
- #'treesit--after-change
- 0 t)
- (remove-hook 'after-change-functions
- #'treesit--after-change t)))
- nil t))
+ (add-hook 'outline-after-change #'treesit--after-change nil t))
;; Remove existing local parsers.
(dolist (ov (overlays-in (point-min) (point-max)))
[Message part 3 (text/plain, inline)]
>>>> But can we do better? I see that 'treesit-major-mode-setup' >>>> adds the notifier >>>> >>>> (treesit-parser-add-notifier >>>> treesit-primary-parser #'treesit--font-lock-mark-ranges-to-fontify) >>>> >>>> to 'after_change_functions'. But why the treesit >>>> 'after_change_functions' notifier is called after the >>>> 'after-change-functions' hook? Can we change their order? >>> >>> Because we parse lazily. And I should’ve called them after-parse functions >>> in the docstring (good thing I named them notifiers, not >>> after-change-functions)—they get called after the parser re-parses, but the >>> parser doesn’t have to re-parse immediately after a buffer change. It only >>> re-parses when some Lisp asks for a node from the parse tree. >>> >>> In practice, that usually happens in pre-redisplay-function where we set >>> ranges for embedded parsers, which causes the primary parser to re-parse >>> and return a root node for querying. As a (intended) side-effect, the >>> notifiers get called and mark relevant regions to be re-fontified. >> >> Would it be possible on the first attempt of Lisp code to access a node >> to detect such a situation that there are some outdated nodes and >> to update their ranges automatically? > > Probably not, since you need to have correct ranges to know > which parser to use to get nodes in the first place.
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Wed, 09 Apr 2025 17:25:02 GMT) Full text and rfc822 format available.Message #64 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#77256: Treesit language-at-point Date: Wed, 09 Apr 2025 20:20:47 +0300
[Message part 1 (text/plain, inline)]
>>> I can't find any code out there that seems to care what is its default >>> value (all users set it but doesn't look at it, AFAICT). >> >> This case looks problematic: >> >> ;; LANG can be nil. We don't want to use the fallback >> ;; in `treesit-language-at', so here we call >> ;; `treesit-language-at-point-function' directly. >> (let* ((lang (and treesit-language-at-point-function >> (funcall treesit-language-at-point-function >> pos))) > > Ah, indeed you're right. > >> But I believe this whole 'let*' can be replaced by just >> the 'treesit-parsers-at' call. I hope Yuan could confirm this. > > [ Not familiar enough with this code, so I'll let Yuan chime in. ] Yuan, please confirm whether the reference to 'treesit-language-at-point-function' can be removed from 'treesit-node-at' this way:
[treesit-language-at-point-function.patch (text/x-diff, inline)]
diff --git a/lisp/treesit.el b/lisp/treesit.el
index 8e57a6dae14..5a2721cdda4 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -165,7 +165,8 @@ treesit-primary-parser
The primary parser should be a parser that parses the entire buffer, as
opposed to embedded parsers which parses only part of the buffer.")
-(defvar-local treesit-language-at-point-function nil
+(defvar-local treesit-language-at-point-function
+ #'treesit-language-at-point-default
"A function that returns the language at point.
This is used by `treesit-language-at', which is used by various
functions to determine which parser to use at point.
@@ -183,10 +184,15 @@ treesit-language-at
the most relevant parser (hence language) by their embed level.
If `treesit-language-at-point-function' is non-nil, return
the return value of that function instead."
- (if treesit-language-at-point-function
- (funcall treesit-language-at-point-function position)
- (treesit-parser-language
- (car (treesit-parsers-at position)))))
+ (funcall (or treesit-language-at-point-function
+ #'treesit-language-at-point-default)
+ position))
+
+(defun treesit-language-at-point-default (position)
+ "Default function for `treesit-language-at-point-function'.
+Return the deepest parser by embed level."
+ (treesit-parser-language
+ (car (treesit-parsers-at position))))
;;; Node API supplement
@@ -238,12 +244,8 @@ treesit-node-at
;; 2. Given a language, try local parser, then global
;; parser.
(parser-or-lang
- (let* ((local-parser (car (treesit-local-parsers-at
- pos parser-or-lang)))
- (global-parser (car (treesit-parsers-at
- pos parser-or-lang nil
- '(primary global))))
- (parser (or local-parser global-parser)))
+ (let ((parser (car (treesit-parsers-at
+ pos parser-or-lang))))
(when parser
(treesit-parser-root-node parser))))
;; 3. No given language, try to get a language at point.
@@ -252,20 +254,8 @@ treesit-node-at
;; finding parser, try local parser first, then global
;; parser.
(t
- ;; LANG can be nil. We don't want to use the fallback
- ;; in `treesit-language-at', so here we call
- ;; `treesit-language-at-point-function' directly.
- (let* ((lang (and treesit-language-at-point-function
- (funcall treesit-language-at-point-function
- pos)))
- (local-parser
- ;; Find the local parser with highest
- ;; embed-level at point.
- (car (treesit-local-parsers-at pos lang)))
- (global-parser (car (treesit-parsers-at
- pos lang nil
- '(primary global))))
- (parser (or local-parser global-parser)))
+ ;; LANG can be nil. Use the parser deepest by embed level.
+ (let ((parser (car (treesit-parsers-at pos))))
(when parser
(treesit-parser-root-node parser))))))
(node root)
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Thu, 10 Apr 2025 08:08:02 GMT) Full text and rfc822 format available.Message #67 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Yuan Fu <casouri <at> gmail.com> To: Juri Linkov <juri <at> linkov.net> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#77256: Treesit language-at-point Date: Thu, 10 Apr 2025 01:06:47 -0700
> On Apr 8, 2025, at 11:18 PM, Juri Linkov <juri <at> linkov.net> wrote: > >>>> This would make sure the `treesit-language-at-point-function` is good >>>> enough to implement the default behavior and it makes it possible to use >>>> `add-function` on `treesit-language-at-point-function`. >>> >>> Unfortunately, it's too late to add a non-nil default value. >> >> Is it? >> I can't find any code out there that seems to care what is its default >> value (all users set it but doesn't look at it, AFAICT). The default value shouldn't matter. What’s the difference between a major-mode-set non-nil value and a default non-nil value? > > This case looks problematic: > > ;; LANG can be nil. We don't want to use the fallback > ;; in `treesit-language-at', so here we call > ;; `treesit-language-at-point-function' directly. > (let* ((lang (and treesit-language-at-point-function > (funcall treesit-language-at-point-function > pos))) > > But I believe this whole 'let*' can be replaced by just > the 'treesit-parsers-at' call. I hope Yuan could confirm this. Why is this problematic? And yes, treesit-parsers-at is a good fit. > >>> I already tried to do the same by adding 'forward-sexp-default-function' >>> to 'forward-sexp-function', but failed spectacularly (bug#70426), >> >> Yes, that one is much harder (it's one of the first I tried to change), > > If no more code outside relies on the default value, > the default function could be added like this: > diff --git a/lisp/treesit.el b/lisp/treesit.el > index 8e57a6dae14..a010ff24afb 100644 > --- a/lisp/treesit.el > +++ b/lisp/treesit.el > @@ -165,7 +165,8 @@ treesit-primary-parser > The primary parser should be a parser that parses the entire buffer, as > opposed to embedded parsers which parses only part of the buffer.") > > -(defvar-local treesit-language-at-point-function nil > +(defvar-local treesit-language-at-point-function > + #'treesit-language-at-point-default > "A function that returns the language at point. > This is used by `treesit-language-at', which is used by various > functions to determine which parser to use at point. > @@ -183,10 +184,13 @@ treesit-language-at > the most relevant parser (hence language) by their embed level. > If `treesit-language-at-point-function' is non-nil, return > the return value of that function instead." > - (if treesit-language-at-point-function > - (funcall treesit-language-at-point-function position) > - (treesit-parser-language > - (car (treesit-parsers-at position))))) > + (funcall treesit-language-at-point-function position)) > + > +(defun treesit-language-at-point-default (position) > + "Default function for `treesit-language-at-point-function'. > +Return the deepest parser by embed level." > + (treesit-parser-language > + (car (treesit-parsers-at position)))) > > ;;; Node API supplement > LGTM. > BTW, I noticed now that indentation of the default value > in defvar-local is incorrect and differs from defvar and defcustom. > Here is the fix: > > diff --git a/lisp/subr.el b/lisp/subr.el > index be847aab28a..164ed2a6f63 100644 > --- a/lisp/subr.el > +++ b/lisp/subr.el > @@ -197,7 +197,7 @@ defvar-local > Like `defvar' but additionally marks the variable as being automatically > buffer-local wherever it is set. > \n(fn SYMBOL &optional VALUE DOCSTRING)" > - (declare (debug defvar) (doc-string 3) (indent 2)) > + (declare (debug defvar) (doc-string 3) (indent defun)) > ;; Can't use backquote here, it's too early in the bootstrap. > (let ((value (car-safe args)) > (docstring (car-safe (cdr-safe args))))
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Thu, 10 Apr 2025 14:48:01 GMT) Full text and rfc822 format available.Message #70 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Juri Linkov <juri <at> linkov.net> Subject: Re: bug#77256: Treesit language-at-point Date: Thu, 10 Apr 2025 10:47:50 -0400
>>> Is it?
>>> I can't find any code out there that seems to care what is its default
>>> value (all users set it but doesn't look at it, AFAICT).
> The default value shouldn't matter. What’s the difference between
> a major-mode-set non-nil value and a default non-nil value?
No difference, in theory, indeed.
The problem is when code cares about the nil -vs- non-nil value of the variable.
>> This case looks problematic:
>>
>> ;; LANG can be nil. We don't want to use the fallback
>> ;; in `treesit-language-at', so here we call
>> ;; `treesit-language-at-point-function' directly.
>> (let* ((lang (and treesit-language-at-point-function
>> (funcall treesit-language-at-point-function
>> pos)))
>>
>> But I believe this whole 'let*' can be replaced by just
>> the 'treesit-parsers-at' call. I hope Yuan could confirm this.
>
> Why is this problematic?
Because the `and` treats `treesit-language-at-point-function` as
a boolean rather than as a function (and according to the comment, it
does so to avoid the default behavior, so if we set the var to a non-nil
value that provides the default behavior, it defeats the purpose of the
test).
But apparently, Juri's patch gets rid of this problem. 🙂
Stefan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Tue, 15 Apr 2025 23:30:02 GMT) Full text and rfc822 format available.Message #73 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Yuan Fu <casouri <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Juri Linkov <juri <at> linkov.net> Subject: Re: bug#77256: Treesit language-at-point Date: Tue, 15 Apr 2025 16:28:53 -0700
> On Apr 10, 2025, at 7:47 AM, Stefan Monnier <monnier <at> iro.umontreal.ca> wrote: > >>>> Is it? >>>> I can't find any code out there that seems to care what is its default >>>> value (all users set it but doesn't look at it, AFAICT). >> The default value shouldn't matter. What’s the difference between >> a major-mode-set non-nil value and a default non-nil value? > > No difference, in theory, indeed. > The problem is when code cares about the nil -vs- non-nil value of the variable. > >>> This case looks problematic: >>> >>> ;; LANG can be nil. We don't want to use the fallback >>> ;; in `treesit-language-at', so here we call >>> ;; `treesit-language-at-point-function' directly. >>> (let* ((lang (and treesit-language-at-point-function >>> (funcall treesit-language-at-point-function >>> pos))) >>> >>> But I believe this whole 'let*' can be replaced by just >>> the 'treesit-parsers-at' call. I hope Yuan could confirm this. >> >> Why is this problematic? > > Because the `and` treats `treesit-language-at-point-function` as > a boolean rather than as a function (and according to the comment, it > does so to avoid the default behavior, so if we set the var to a non-nil > value that provides the default behavior, it defeats the purpose of the > test). Oops, that was bad code on my part—it wasn’t intentional. > > But apparently, Juri's patch gets rid of this problem. 🙂 Yeah, feel free to apply the patch, Juri :) Yuan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Wed, 16 Apr 2025 17:14:02 GMT) Full text and rfc822 format available.Message #76 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#77256: Treesit language-at-point Date: Wed, 16 Apr 2025 20:01:47 +0300
> Yeah, feel free to apply the patch, Juri :) Now the patch is pushed.
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Wed, 16 Apr 2025 17:17:11 GMT) Full text and rfc822 format available.Message #79 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Juri Linkov <juri <at> linkov.net> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#77256: Treesit language-at-point Date: Wed, 16 Apr 2025 20:12:21 +0300
>> This seems a bit convoluted to me. What do you think about adding >> some hook or generic function to outline-minor-mode? > > Is this better? > > diff --git a/lisp/outline.el b/lisp/outline.el > -(defun outline--fix-buttons-after-change (beg end _len) > +(defvar outline-after-change nil > + "List of functions to call after each text change in outline-mode.") > + > +(defun outline--fix-buttons-after-change (beg end len) > + (run-hook-with-args 'outline-after-change beg end len) > ;; Handle whole lines > (save-excursion (goto-char beg) (setq beg (pos-bol))) > (save-excursion (goto-char end) (setq end (pos-eol))) This is pushed as well.
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Thu, 17 Apr 2025 05:22:03 GMT) Full text and rfc822 format available.Message #82 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Yuan Fu <casouri <at> gmail.com> To: Juri Linkov <juri <at> linkov.net> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#77256: Treesit language-at-point Date: Wed, 16 Apr 2025 22:21:34 -0700
> On Apr 16, 2025, at 10:12 AM, Juri Linkov <juri <at> linkov.net> wrote: > >>> This seems a bit convoluted to me. What do you think about adding >>> some hook or generic function to outline-minor-mode? >> >> Is this better? >> >> diff --git a/lisp/outline.el b/lisp/outline.el >> -(defun outline--fix-buttons-after-change (beg end _len) >> +(defvar outline-after-change nil >> + "List of functions to call after each text change in outline-mode.") >> + >> +(defun outline--fix-buttons-after-change (beg end len) >> + (run-hook-with-args 'outline-after-change beg end len) >> ;; Handle whole lines >> (save-excursion (goto-char beg) (setq beg (pos-bol))) >> (save-excursion (goto-char end) (setq end (pos-eol))) > > This is pushed as well. FTR I’m not super satisfied with how we are adding after-change hook to update ranges. But I couldn’t came up with anything in the past few days so I didn’t suggest any alternatives :-) Ultimately I think you are right that we want guarantee to Lisp programs that parser ranges are always up-to-date, so they don’t have to call treesit-update-ranges themselves every where. But the exact implementation to achieve that is still not super clear in my head. After thinking about this more and rewriting this reply three times, I think the best way might be to call treesit--pre-redisplay in after-change-functions. Treesit--pre-redisplay should make sure all the parser ranges are updated. The only thing I don’t like about this is that there’re many cases where a series of buffer edits are made, and we don’t want to update ranges after each single buffer edit. Post-command-hook should prevent that, but post-command-hook feels really user-facing and high-level to me. I’d like to know what do you and Stefan think about this. Yuan
bug-gnu-emacs <at> gnu.org:bug#77256; Package emacs.
(Thu, 17 Apr 2025 15:02:02 GMT) Full text and rfc822 format available.Message #85 received at 77256 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Yuan Fu <casouri <at> gmail.com> Cc: 77256 <at> debbugs.gnu.org, Vincenzo Pupillo <v.pupillo <at> gmail.com>, Juri Linkov <juri <at> linkov.net> Subject: Re: bug#77256: Treesit language-at-point Date: Thu, 17 Apr 2025 11:01:41 -0400
> Ultimately I think you are right that we want guarantee to Lisp programs
> that parser ranges are always up-to-date, so they don’t have to call
> treesit-update-ranges themselves every where. But the exact implementation
> to achieve that is still not super clear in my head. After thinking about
> this more and rewriting this reply three times, I think the best way might
> be to call treesit--pre-redisplay in
> after-change-functions. Treesit--pre-redisplay should make sure all the
> parser ranges are updated. The only thing I don’t like about this is that
> there’re many cases where a series of buffer edits are made, and we don’t
> want to update ranges after each single buffer edit. Post-command-hook
> should prevent that, but post-command-hook feels really user-facing and
> high-level to me.
I don't know how ranges are managed/computed, but the usual way to
handle this kind of problem is to separate the "mark as out of date"
from the actual update which you do only lazily when a request
is made.
Stefan
Debbugs Internal Request <help-debbugs <at> gnu.org>
to internal_control <at> debbugs.gnu.org.
(Fri, 16 May 2025 11:24:10 GMT) Full text and rfc822 format available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.