Subject: Re: bug#74610: 31.0.50; Submitting mhtml-ts-mode, treesitter alternative to mhtml-mode

From: Vincenzo Pupillo <[email-masked]>
Date: Tue, 14 Jan 2025 22:41:01 +0100

Ciao Yuan and Juri, 
this is an updated version of mhtml-ts-mode. 
I have tried to reduce as much as possible copies of parts of the major modes 
from which it is derived.
To do this, I had to move some values that were assigned directly to treesit's 
own variables  (in ccs-mode.el, in js.el, and in html-ts-mode.el) into new 
variables. 
I also added three new functions to treesit.el to make it easier to combine 
parts derived from the other major modes. So now any changes to these new 
variables are directly reflected in the behavior of mhtml-ts-mode.
There are a few things I would like to highlight:
1. treesit-font-lock-feature-lists are not defined per parser, so simply 
merging the different lists will cause display differences from the original 
major-modes; for example “function” is defined at level 3 in css-ts-mode but 
at level 4 in js-ts-mode.
2. treesit-defun-type-regexp has the same problem as treesit-font-lock-
feature-list, so I had to define it myself.

But other than that, it works pretty well.

IMHO, a global "list" where you can define "font-lock", "indent-list", "font-
lock-feature", etc. by language (perhaps with getter and setter methods) might 
make it easier to define new multilingual major-modes. It could improve the 
decoupling between multilingual major-modes and the major-modes they are 
derived from. It could also better decouple the internal implementation of 
treesit.el from the treesitter-based major-modes.

Let me know what you think.

Thanks.

Vincenzo

p.s. In this version I also added support for wich-function-mode, for flymake 
and a pretty-print function.

In data martedì 24 dicembre 2024 23:18:39 Ora standard dell’Europa centrale, 
Yuan Fu ha scritto:
> > On Dec 24, 2024, at 12:37 AM, Yuan Fu <[email-masked]> wrote:
> >> On Dec 14, 2024, at 2:37 AM, Vincenzo Pupillo <[email-masked]>
> >> wrote:
> >> 
> >> In data mercoledì 11 dicembre 2024 05:54:09 Ora standard dell’Europa
> >> centrale,>> 
> >> Yuan Fu ha scritto:
> >>>> On Dec 3, 2024, at 6:29 AM, Vincenzo Pupillo <[email-masked]>
> >>>> wrote:
> >>>> 
> >>>> In data domenica 1 dicembre 2024 07:01:21 Ora standard dell’Europa
> >>>> centrale,>
> >>>> 
> >>>> Yuan Fu ha scritto:
> >>>>> It's not uncommon to see different indent offset for CSS and
> >>>>> Javascript, so it's a good idea to have separate control for them.
> >>>> 
> >>>> Is the behavior the same as mhtml-mode, or would you like something
> >>>> like
> >>>> this?>
> >>>> 
> >>>>  <style>
> >>>>  
> >>>>                                z {
> >>>>                                
> >>>>                                    color: red;
> >>>>                                
> >>>>                                }
> >>>>  
> >>>>  </style>
> >>>>  <script>
> >>>>  
> >>>>      function myFunction(p1, p2) {
> >>>>      
> >>>>          return p1 * p2;
> >>>>      
> >>>>      }
> >>>>  
> >>>>  </script>
> >>>> 
> >>>> The mhtml-ts-mode-js-css-indent-offset variable controls only the
> >>>> indentation relative to the <style> and <script> tags.
> >>> 
> >>> Ah, I see, it’s the offset from the enclosing tag. In that case it
> >>> should be fine to use a common variable.
> >>> 
> >>> Yuan
> >> 
> >> Thank you Yuan.
> >> Attached is the revised patch following your previous comments.
> >> As I already wrote to Dmitry, I am doing some tests to see if
> >> html-ts-mode can be extended and if there is a way to integrate one
> >> multi-language mode into another multi-language mode.
> >> 
> >> Vincenzo
> >> <0001-Add-mhtml-ts-mode.patch>
> > 
> > Btw, mhtml-ts-mode--defun-name seems to contain some debugging code? And
> > also I think you should use treesit-node-language.
> > 
> > Yuan
> 
> I just added treesit-aggregated-simple-imenu-settings to master. This
> variable will allow you to setup Imenu for multiple languages. Please give
> it a try. I tested locally with mhtml-ts-mode and works well.
> 
> Yuan

>From 58480feff12b9cf61a91e8be665d5a453ffc415d Mon Sep 17 00:00:00 2001
From: Vincenzo Pupillo <[email-masked]>
Date: Tue, 14 Jan 2025 15:40:32 +0100
Subject: [PATCH] Add mhtml-ts-mode.

New major-mode alternative to mhtml-mode, based on treesitter, for
editing files containing html, javascript and css.

* etc/NEWS: Mention the new mode and new functions.
* lisp/textmodes/mhtml-ts-mode.el: New file.
* lisp/progmodes/js.el (js--treesit-thing-settings): New variable.
(js--treesit-font-lock-feature-list); New variable.
(js--treesit-simple-imenu-settings): New variable.
(js--treesit-defun-type-regexp): New variable.
(js--treesit-jsdoc-comment-regexp): New variable.
(js-ts-mode): Use of new variables instead of direct assignment of
values.
* lisp/textmodes/css-mode.el (css-mode--menu): New variable.
(css-mode-map): Use new variable.
(css--treesit-font-lock-feature-list): New variable.
(css--treesit-simple-imenu-settings): New variable.
(css--treesit-defun-type-regexp): New variable.
(cs-ts-mode): Use of new variables instead of direct assignment of
values.
* lisp/textmodes/html-ts-mode.el
(html-ts-mode--treesit-things-settings): New variable.
(html-ts-mode--treesit-font-lock-feature-list): New variable.
(html-ts-mode--treesit-simple-imenu-settings): New variable.
(html-ts-mode--treesit-defun-type-regexp): New variable.
(html-ts-mode): Use of new variables instead of direct assignment of
values.
* lisp/treesit.el
(treesit-merge-font-lock-feature-list): New fuction.
(treesit-replace-font-lock-feature-settings): New fuction.
(treesit-modify-indent-rules): New function.
---
 etc/NEWS                       | 29 ++++++++++++++
 lisp/progmodes/js.el           | 72 +++++++++++++++++++++-------------
 lisp/textmodes/css-mode.el     | 47 ++++++++++++++--------
 lisp/textmodes/html-ts-mode.el | 46 ++++++++++++++--------
 lisp/treesit.el                | 65 ++++++++++++++++++++++++++++++
 5 files changed, 200 insertions(+), 59 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 2f04204ad94..8ce8ccae581 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -948,6 +948,13 @@ destination window is chosen using 'display-buffer-alist'.  Example:
 
 * New Modes and Packages in Emacs 31.1
 
+** New major modes based on the tree-sitter library
+
+*** New major mode 'mhtml-ts-mode'.
+An optional major mode based on the tree-sitter library for editing html
+files. This mode handles indentation, fontification, and commenting for
+embedded JavaScript and CSS.
+
 
 * Incompatible Lisp Changes in Emacs 31.1
 
@@ -1080,6 +1087,28 @@ language symbol.  For example, 'cpp' is translated to "C++".  A new
 variable 'treesit-language-display-name-alist' holds the translations of
 language symbols where that translation is not trivial.
 
++++
+*** New function 'treesit-merge-font-lock-feature-list'.
+This function the merge two tree-sitter font lock feature lists.
+Returns a new font lock feature list with no duplicates in the same level.
+It can be used to merge font lock feature lists in a multi-language major mode.
+
++++
+*** New function 'treesit-replace-font-lock-feature-settings'.
+Given two treesit-font-lock-settings replaces the feature in the second
+font-lock-settings with the same feature in the first
+font-lock-settings. In a multi-linguage major mode it is sometimes
+necessary to replace features from one of the major modes, with others
+that are better suited to the new multilingual context.
+
++++
+*** New function 'treesit-modify-indent-rules'.
+Given two treesit ident rules, it replaces, adds, or prepends the new
+rules to the old ones, then returns a new treesit indent rules.
+In a multi-linguage major mode it is sometimes necessary to modify rules
+from one of the major modes, with others that are better suited to the
+new multilingual context.
+
 +++
 *** New command 'treesit-explore'
 This command replaces 'treesit-explore-mode'.  It turns on
diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 101b882c718..c7b301c100c 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3901,6 +3901,44 @@ js--treesit-list-nodes
 (defvar js--treesit-jsdoc-beginning-regexp (rx bos "/**")
   "Regular expression matching the beginning of a jsdoc block comment.")
 
+(defvar js--treesit-thing-settings
+  `((javascript
+     (sexp ,(js--regexp-opt-symbol js--treesit-sexp-nodes))
+     (list ,(js--regexp-opt-symbol js--treesit-list-nodes))
+     (sentence ,(js--regexp-opt-symbol js--treesit-sentence-nodes))
+     (text ,(js--regexp-opt-symbol '("comment"
+                                     "string_fragment")))))
+  "Settings for `treesit-thing-settings'.")
+
+(defvar js--treesit-font-lock-feature-list
+  '(( comment document definition)
+                  ( keyword string)
+                  ( assignment constant escape-sequence jsx number
+                    pattern string-interpolation)
+                  ( bracket delimiter function operator property))
+  "Settings for `treesit-font-lock-feature-list'.")
+
+(defvar js--treesit-simple-imenu-settings
+  `(("Function" "\\`function_declaration\\'" nil nil)
+    ("Variable" "\\`lexical_declaration\\'"
+     js--treesit-valid-imenu-entry nil)
+    ("Class" ,(rx bos (or "class_declaration"
+                          "method_definition")
+                  eos)
+     nil nil))
+  "Settings for `treesit-simple-imenu'.")
+
+(defvar js--treesit-defun-type-regexp
+  (rx (or "class_declaration"
+          "method_definition"
+          "function_declaration"
+          "lexical_declaration"))
+  "Settings for `treesit-defun-type-regexp'.")
+
+(defvar js--treesit-jsdoc-comment-regexp
+  (rx (or "comment" "line_comment" "block_comment" "description"))
+  "Regexp for `c-ts-common--comment-regexp'.")
+
 ;;;###autoload
 (define-derived-mode js-ts-mode js-base-mode "JavaScript"
   "Major mode for editing JavaScript.
@@ -3931,29 +3969,15 @@ js-ts-mode
     ;; Indent.
     (setq-local treesit-simple-indent-rules js--treesit-indent-rules)
     ;; Navigation.
-    (setq-local treesit-defun-type-regexp
-                (rx (or "class_declaration"
-                        "method_definition"
-                        "function_declaration"
-                        "lexical_declaration")))
+    (setq-local treesit-defun-type-regexp js--treesit-defun-type-regexp)
+
     (setq-local treesit-defun-name-function #'js--treesit-defun-name)
 
-    (setq-local treesit-thing-settings
-                `((javascript
-                   (sexp ,(js--regexp-opt-symbol js--treesit-sexp-nodes))
-                   (list ,(js--regexp-opt-symbol js--treesit-list-nodes))
-                   (sentence ,(js--regexp-opt-symbol js--treesit-sentence-nodes))
-                   (text ,(js--regexp-opt-symbol '("comment"
-                                                   "string_fragment"))))))
+    (setq-local treesit-thing-settings js--treesit-thing-settings)
 
     ;; Fontification.
     (setq-local treesit-font-lock-settings js--treesit-font-lock-settings)
-    (setq-local treesit-font-lock-feature-list
-                '(( comment document definition)
-                  ( keyword string)
-                  ( assignment constant escape-sequence jsx number
-                    pattern string-interpolation)
-                  ( bracket delimiter function operator property)))
+    (setq-local treesit-font-lock-feature-list js--treesit-font-lock-feature-list)
 
     (when (treesit-ready-p 'jsdoc t)
       (setq-local treesit-range-settings
@@ -3963,17 +3987,11 @@ js-ts-mode
                    :local t
                    `(((comment) @capture (:match ,js--treesit-jsdoc-beginning-regexp @capture)))))
 
-      (setq c-ts-common--comment-regexp (rx (or "comment" "line_comment" "block_comment" "description"))))
+      (setq c-ts-common--comment-regexp js--treesit-jsdoc-comment-regexp))
 
     ;; Imenu
-    (setq-local treesit-simple-imenu-settings
-                `(("Function" "\\`function_declaration\\'" nil nil)
-                  ("Variable" "\\`lexical_declaration\\'"
-                   js--treesit-valid-imenu-entry nil)
-                  ("Class" ,(rx bos (or "class_declaration"
-                                        "method_definition")
-                                eos)
-                   nil nil)))
+    (setq-local treesit-simple-imenu-settings js--treesit-simple-imenu-settings)
+
     (treesit-major-mode-setup)
 
     (add-to-list 'auto-mode-alist
diff --git a/lisp/textmodes/css-mode.el b/lisp/textmodes/css-mode.el
index 53340195386..35c61e4f66d 100644
--- a/lisp/textmodes/css-mode.el
+++ b/lisp/textmodes/css-mode.el
@@ -893,13 +893,7 @@ css-mode-syntax-table
     (modify-syntax-entry ?? "." st)
     st))
 
-(defvar-keymap css-mode-map
-  :doc "Keymap used in `css-mode'."
-  "<remap> <info-lookup-symbol>" #'css-lookup-symbol
-  ;; `info-complete-symbol' is not used.
-  "<remap> <complete-symbol>" #'completion-at-point
-  "C-c C-f" #'css-cycle-color-format
-  :menu
+(defvar css-mode--menu
   '("CSS"
     :help "CSS-specific features"
     ["Reformat block" fill-paragraph
@@ -910,7 +904,17 @@ css-mode-map
     ["Describe symbol" css-lookup-symbol
      :help "Display documentation for a CSS symbol"]
     ["Complete symbol" completion-at-point
-     :help "Complete symbol before point"]))
+     :help "Complete symbol before point"])
+    "Menu bar for `css-mode'")
+
+(defvar-keymap css-mode-map
+  :doc "Keymap used in `css-mode'."
+  "<remap> <info-lookup-symbol>" #'css-lookup-symbol
+  ;; `info-complete-symbol' is not used.
+  "<remap> <complete-symbol>" #'completion-at-point
+  "C-c C-f" #'css-cycle-color-format
+  :menu
+  css-mode--menu)
 
 (eval-and-compile
   (defconst css--uri-re
@@ -1771,6 +1775,21 @@ css--extract-index-name
               (replace-regexp-in-string "[\n ]+" " " s)))
            res)))))))
 
+(defvar css--treesit-font-lock-feature-list
+  '((selector comment query keyword)
+    (property constant string)
+    (error variable function operator bracket))
+  "Settings for `treesit-font-lock-feature-list'.")
+
+(defvar css--treesit-simple-imenu-settings
+  `(( nil ,(rx bos (or "rule_set" "media_statement") eos)
+      nil nil))
+  "Settings for `treesit-simple-imenu'.")
+
+(defvar css--treesit-defun-type-regexp
+  "rule_set"
+  "Settings for `treesit-defun-type-regexp'.")
+
 (define-derived-mode css-base-mode prog-mode "CSS"
   "Generic mode to edit Cascading Style Sheets (CSS).
 
@@ -1825,16 +1844,12 @@ css-ts-mode
     ;; Tree-sitter specific setup.
     (setq treesit-primary-parser (treesit-parser-create 'css))
     (setq-local treesit-simple-indent-rules css--treesit-indent-rules)
-    (setq-local treesit-defun-type-regexp "rule_set")
+    (setq-local treesit-defun-type-regexp css--treesit-defun-type-regexp)
     (setq-local treesit-defun-name-function #'css--treesit-defun-name)
     (setq-local treesit-font-lock-settings css--treesit-settings)
-    (setq-local treesit-font-lock-feature-list
-                '((selector comment query keyword)
-                  (property constant string)
-                  (error variable function operator bracket)))
-    (setq-local treesit-simple-imenu-settings
-                `(( nil ,(rx bos (or "rule_set" "media_statement") eos)
-                    nil nil)))
+    (setq-local treesit-font-lock-feature-list css--treesit-font-lock-feature-list)
+    (setq-local treesit-simple-imenu-settings css--treesit-simple-imenu-settings)
+
     (treesit-major-mode-setup)
 
     (add-to-list 'auto-mode-alist '("\\.css\\'" . css-ts-mode))))
diff --git a/lisp/textmodes/html-ts-mode.el b/lisp/textmodes/html-ts-mode.el
index dad49b7ed4c..de945d89ffb 100644
--- a/lisp/textmodes/html-ts-mode.el
+++ b/lisp/textmodes/html-ts-mode.el
@@ -87,6 +87,32 @@ html-ts-mode--font-lock-settings
    `((attribute_name) @font-lock-variable-name-face))
   "Tree-sitter font-lock settings for `html-ts-mode'.")
 
+(defvar html-ts-mode--treesit-things-settings
+  `((html
+     (sexp ,(regexp-opt '("element"
+                          "text"
+                          "attribute"
+                          "value")))
+     (sexp-list ,(regexp-opt '("element")) 'symbols)
+     (sentence "tag")
+     ;; (sentence ,(regex-opt '("start_tag" "end_tag")))
+     (text ,(regexp-opt '("comment" "text")))))
+  "Settings for `treesit-thing-settings'.")
+
+(defvar html-ts-mode--treesit-font-lock-feature-list
+  '((comment keyword definition)
+    (property string)
+    () ())
+  "Settings for `treesit-font-lock-feature-list'.")
+
+(defvar html-ts-mode--treesit-simple-imenu-settings
+  '(("Element" "\\`tag_name\\'" nil nil))
+  "Settings for `treesit-simple-imenu'.")
+
+(defvar html-ts-mode--treesit-defun-type-regexp
+  "element"
+  "Settings for `treesit-defun-type-regexp'.")
+
 (defun html-ts-mode--defun-name (node)
   "Return the defun name of NODE.
 Return nil if there is no name or if NODE is not a defun node."
@@ -107,30 +133,18 @@ html-ts-mode
   (setq-local treesit-simple-indent-rules html-ts-mode--indent-rules)
 
   ;; Navigation.
-  (setq-local treesit-defun-type-regexp "element")
+  (setq-local treesit-defun-type-regexp html-ts-mode--treesit-defun-type-regexp)
 
   (setq-local treesit-defun-name-function #'html-ts-mode--defun-name)
 
-  (setq-local treesit-thing-settings
-              `((html
-                 (sexp ,(regexp-opt '("element"
-                                      "text"
-                                      "attribute"
-                                      "value")))
-                 (list ,(regexp-opt '("element")) 'symbols)
-                 (sentence "tag")
-                 (text ,(regexp-opt '("comment" "text"))))))
+  (setq-local treesit-thing-settings html-ts-mode--treesit-things-settings)
 
   ;; Font-lock.
   (setq-local treesit-font-lock-settings html-ts-mode--font-lock-settings)
-  (setq-local treesit-font-lock-feature-list
-              '((comment keyword definition)
-                (property string)
-                () ()))
+  (setq-local treesit-font-lock-feature-list html-ts-mode--treesit-font-lock-feature-list)
 
   ;; Imenu.
-  (setq-local treesit-simple-imenu-settings
-              '(("Element" "\\`tag_name\\'" nil nil)))
+  (setq-local treesit-simple-imenu-settings html-ts-mode--treesit-simple-imenu-settings)
 
   ;; Outline minor mode.
   (setq-local treesit-outline-predicate "\\`element\\'")
diff --git a/lisp/treesit.el b/lisp/treesit.el
index ac34edaf84d..597090c8543 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -1259,6 +1259,40 @@ treesit-font-lock-recompute-features
                        ((memq feature remove-list) nil)
                        (t current-value))))))
 
+(defun treesit-merge-font-lock-feature-list (features-list-1 features-list-2)
+  "Merge two tree-sitter font lock feature lists.
+Returns a new font lock feature list with no duplicates in the same level.
+It can be used to merge font lock feature lists in a multi-language major mode.
+FEATURES-LIST-1 and FEATURES-LIST-2 are list of lists of feature symbols."
+    (let ((result nil)
+	(features-1 (car features-list-1))
+	(features-2 (car features-list-2)))
+    (while (or features-1 features-2)
+      (cond
+       ((and features-1 (not features-2)) (push features-1 result))
+       ((and (not features-1) features-2) (push features-2 result))
+       ((and features-1 features-2) (push (cl-union features-1 features-2) result)))
+      (setq features-list-1 (cdr features-list-1)
+	    features-list-2 (cdr features-list-2)
+	    features-1 (car features-list-1)
+            features-2 (car features-list-2)))
+    (nreverse result)))
+
+(defun treesit-replace-font-lock-feature-settings (new-settings settings)
+  "Replaces :feature in SETTINGS with :feature from NEW-SETTINGS.
+Both SETTINGS and NEW-SETTINGS must be a value suitable for
+`treesit-font-lock-settings'.
+Return a value suitable for `treesit-font-lock-settings'"
+  (let ((result nil))
+    (dolist (new-setting new-settings)
+      (let ((new-feature (treesit-font-lock-setting-feature new-setting)))
+	(dolist (setting settings)
+	  (let ((feature (treesit-font-lock-setting-feature setting)))
+	    (if (eq new-feature feature)
+		(push new-setting result)
+	      (push setting result))))))
+    (nreverse result)))
+
 (defun treesit-add-font-lock-rules (rules &optional how feature)
   "Add font-lock RULES to the current buffer.
 
@@ -2357,6 +2391,37 @@ treesit--indent-rules-optimize
                             offset)))))
              (cons lang (mapcar #'optimize-rule indent-rules)))))
 
+(defun treesit-modify-indent-rules (new-rules rules &optional how)
+  "Modify RULES using NEW-RULES.
+As default replace rules with the same anchor.
+When HOW is :prepend NEW-RULES are prepend to RULES, when
+:append NEW-RULES are appended to RULES, when :replace (the default)
+NEW-RULES replace rule in RULES which the same anchor."
+  (let ((n-lang (car (car new-rules)))
+	(lang (car (car rules))))
+    (when (not (eq n-lang lang))
+      (error "The language must be the same"))
+    (let* ((nr (cdr (car new-rules)))
+           (r (cdr (car rules)))
+           (tmp nil)
+           (result
+            (cond
+             ((eq how :prepend)
+	      (append nr r))
+             ((eq how :append)
+              (append r nr))
+             ((or (eq how :replace) t)
+              (nreverse
+               (progn
+                 (dolist (new-rule nr)
+	           (dolist (rule r)
+	             (if (equal (nth 0 new-rule) (nth 0 rule))
+		         (push new-rule tmp)
+		       (push rule tmp))))
+                 tmp))))))
+      (push lang result)
+      `(,result))))
+
 ;;; Search
 
 (defun treesit-search-forward-goto
-- 
2.47.1