Subject: [PATCH] Support a local repo as URL in treesit-language-source-alist
From: Konstantin Kharlamov <[email-masked]>
To: [email-masked]
Date: Fri, 19 Jan 2024 11:08:00 +0300
List-Id: <debbugs-submit.debbugs.gnu.org>

TL;DR: this just to simplify debugging/rebuilding treesitter grammars,
because they are often broken (both the build system and the infamous
"unversioning" part) and only Emacs knows what to do with them. So
gotta have some way to point Emacs to a local grammar repo.

-----------

Long story:

I've been recently stumbling upon the infamous "Debug the query with
treesit-query-validate" problem, and found that it's not the only
problem in that context. The workaround to this problem currently is
finding the commit in the offending treesitter-grammar that works and
re-building the grammar from it.

Well, that "re-building" part of the advice turns out to be very
convoluted. Let's take for example typescript¹. It has no "building"
docs, but I guess they are not needed because it's obvious by
`Cargo.toml` presence that it's written in Rust, and hence building is
just `cargo bulid --release`, right? Well, no. Doing that does not
produce the shared libs Emacs expects. Now a user starts reverse-
engineering, trying to find what's wrong. At some point they realize
that Emacs builds code from the specific directory `tsx/src`, but…
There's no build system! What do you do with these files…? Well, the
answer is that Emacs somehow knows what to do, so now you have to
modify `treesit-language-source-alist` to point at the local repo and
make Emacs do the job of building the grammar.

However, Emacs does not support local paths as part of that grammar.
What it does instead is it clones FROM the local path (which is
confusing on itself, because everything completes successfully, but it
didn't do what you expected it to).

So what this patch does is it adds detection of a full path to the
treesit-language-source-alist processing to make sure we avoid cloning
the repo in that case and just proceed building it.

1: https://github.com/tree-sitter/tree-sitter-typescript
From c9de59a39566a5df46aa39ef93348487bb2948aa Mon Sep 17 00:00:00 2001
From: Konstantin Kharlamov <[email-masked]>
Date: Fri, 19 Jan 2024 10:33:47 +0300
Subject: [PATCH] Support a local repo as URL in treesit-language-source-alist

Sometimes people may need to bisect to find specific revision in a
grammar repo. In this case they'd want to point the URL to the local
repo to avoid cloning it on every rebuild. So add support for full
path in treesit-language-source-alist.

* lisp/treesit.el (treesit--install-language-grammar-1): test if URL
starts with / meaning that the URL is a local path. Then if it is,
avoid cloning the repo and removing the path on success.
(treesit--git-clone-repo): factor out the code for cloning to a separate
function.
(treesit--git-checkout-branch): a helper to checkout the revision for
cases where we didn't clone the repo but want it to point the
revision.
---
 lisp/treesit.el | 38 ++++++++++++++++++++++++++------------
 1 file changed, 26 insertions(+), 12 deletions(-)

diff --git a/lisp/treesit.el b/lisp/treesit.el
index c8b473c7bb8..11631e68793 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -3417,7 +3417,7 @@ treesit-language-source-alist
     (LANG . (URL REVISION SOURCE-DIR CC C++))
 
 Only LANG and URL are mandatory.  LANG is the language symbol.
-URL is the Git repository URL for the grammar.
+URL is the Git repository URL or full path for the grammar.
 
 REVISION is the Git tag or branch of the desired version,
 defaulting to the latest default branch.
@@ -3551,6 +3551,23 @@ treesit--call-process-signal
                                  (buffer-string)))
     (erase-buffer)))
 
+(defun treesit--git-checkout-branch (repo-path revision)
+  "Checkout `revision' in a repo located in `repo-path'"
+  (treesit--call-process-signal
+   "git" nil t nil "-C" repo-path "checkout" revision))
+
+(defun treesit--git-clone-repo (url revision workdir)
+  "Clone repo pointed by `url' at commit `revision' to `workdir'"
+  (message "Cloning repository")
+  ;; git clone xxx --depth 1 --quiet [-b yyy] workdir
+  (if revision
+      (treesit--call-process-signal
+       "git" nil t nil "clone" url "--depth" "1" "--quiet"
+       "-b" revision workdir)
+    (treesit--call-process-signal
+     "git" nil t nil "clone" url "--depth" "1" "--quiet"
+     workdir)))
+
 (defun treesit--install-language-grammar-1
     (out-dir lang url &optional revision source-dir cc c++)
   "Install and compile a tree-sitter language grammar library.
@@ -3564,8 +3581,10 @@ treesit--install-language-grammar-1
 `treesit-language-source-alist'.  If anything goes wrong, this
 function signals an error."
   (let* ((lang (symbol-name lang))
+         ;; don't clone if url is a local path
+         (url-is-path (string-prefix-p "/" url))
          (default-directory (make-temp-file "treesit-workdir" t))
-         (workdir (expand-file-name "repo"))
+         (workdir (if url-is-path url (expand-file-name "repo")))
          (source-dir (expand-file-name (or source-dir "src") workdir))
          (cc (or cc (seq-find #'executable-find '("cc" "gcc" "c99"))
                  ;; If no C compiler found, just use cc and let
@@ -3580,15 +3599,10 @@ treesit--install-language-grammar-1
          (lib-name (concat "libtree-sitter-" lang soext)))
     (unwind-protect
         (with-temp-buffer
-          (message "Cloning repository")
-          ;; git clone xxx --depth 1 --quiet [-b yyy] workdir
-          (if revision
-              (treesit--call-process-signal
-               "git" nil t nil "clone" url "--depth" "1" "--quiet"
-               "-b" revision workdir)
-            (treesit--call-process-signal
-             "git" nil t nil "clone" url "--depth" "1" "--quiet"
-             workdir))
+          (if url-is-path
+              (when revision
+                (treesit--git-checkout-branch url revision))
+            (treesit--git-clone-repo url revision workdir))
           ;; We need to go into the source directory because some
           ;; header files use relative path (#include "../xxx").
           ;; cd "${sourcedir}"
@@ -3635,7 +3649,7 @@ treesit--install-language-grammar-1
             ;; Ignore errors, in case the old version is still used.
             (ignore-errors (delete-file old-fname)))
           (message "Library installed to %s/%s" out-dir lib-name))
-      (when (file-exists-p workdir)
+      (when (and (not url-is-path) (file-exists-p workdir))
         (delete-directory workdir t)))))
 
 ;;; Etc
-- 
2.43.0