Subject: Wrong conversion from Emacs to Tree-sitter S-expression syntax
From: Mattias Engdegård <[email-masked]>
To: Emacs Bug Report <[email-masked]>
Cc: Basil Contovounesios <[email-masked]>, Yuan Fu <[email-masked]>
Date: Mon, 12 Jun 2023 16:14:01 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

`treesit-pattern-expand` converts a query pattern into tree-sitter S-expression syntax, as a string. The conversion mainly converts certain keywords but the main problem is that it prints strings in Emacs syntax which differs from that of tree-sitter.

As a consequence, :match regexps cannot contain newlines:

(treesit-query-capture
 'java
 '(((identifier) @font-lock-constant-face
    (:match "hello\n" @font-lock-constant-face))))

signals a syntax error.

As far as I can tell the tree-sitter string syntax allows for the escape sequences:

\n = LF
\r = CR
\t = TAB
\0 = NUL  (only a single 0 -- no octal escapes!)
\X = the character X itself

Unescape newlines result in a syntax error as seen in the example above. NULs don't seem to go well either.

At the very least, the conversion should avoid literal newlines and NULs in the result (and probably CR and TAB). This cannot be done with a straight prin1-to-string.

(By the way, why is the conversion written in C? Was Lisp too slow?)

Ideally we should not need to expose the tree-sitter s-exp query syntax at all. Surely Emacs s-exps should be preferable in every case?