Subject: Re: bug#35140: 12.1; Scan errors in doctex mode with ^^A-comments after braces

From: Ikumi Keita <[email-masked]>
Date: Tue, 01 Mar 2022 18:34:51 +0900

Hello Markus and all developpers,

I'd like to discuss this issue again.

After reconsideration, I came to think that there is little point in
discriminating two kinds of comments, i.e. those after "%" and those
after "^^A", as long as AUCTeX is concerned. It requires complicated
operation in syntax propertization (`font-latex-doctex-^^A') whereas it
calls for unwanted troubles like this bug#35140 and comments written in
`font-latex-find-matching-close':
,----
|         ;; XXX: Do not look up syntax-table properties since they may
|         ;; be misleading, e.g. in the case of "{foo}^^A" where the
|         ;; closing brace gets a comment end syntax.
`----

Thus I'm inclined to an idea just to fontify "^^A" type comments with
normal comment face, not bothering with changing its syntax. Attached is
a patch that implements this idea.

What do others think about this?

Regards,
Ikumi Keita
#StandWithUkraine #StopWarInUkraine

diff --git a/font-latex.el b/font-latex.el
index 8b1fec60..1a83db08 100644
--- a/font-latex.el
+++ b/font-latex.el
@@ -1091,11 +1091,12 @@ have changed."
          (1 "|") (2 "|")))))
   (when font-latex-syntactic-keywords-extra
     (nconc font-latex-syntactic-keywords font-latex-syntactic-keywords-extra))
-  ;; Cater for docTeX mode.
-  (setq font-latex-doctex-syntactic-keywords
-        (append font-latex-syntactic-keywords
-                ;; For docTeX comment-in-doc.
-                '(("\\(\\^\\)\\^A" (1 (font-latex-doctex-^^A)))))))
+  ;; ;; Cater for docTeX mode.
+  ;; (setq font-latex-doctex-syntactic-keywords
+  ;;       (append font-latex-syntactic-keywords
+  ;;               ;; For docTeX comment-in-doc.
+  ;;               '(("\\(\\^\\)\\^A" (1 (font-latex-doctex-^^A))))))
+  )

 ;;; Syntactic fontification
@@ -1262,9 +1263,9 @@ triggers Font Lock to recognize the change."
   "The `syntax-propertize-function' for (La)TeX documents."
   (with-no-warnings
     (let ((font-lock-syntactic-keywords
-           (if (derived-mode-p 'doctex-mode)
-               font-latex-doctex-syntactic-keywords
-             font-latex-syntactic-keywords)))
+           ;; (if (derived-mode-p 'doctex-mode)
+           ;;     font-latex-doctex-syntactic-keywords
+             font-latex-syntactic-keywords));)
       (font-lock-fontify-syntactic-keywords-region start end))))
 
 ;;;###autoload
@@ -2251,32 +2252,26 @@ set to french, and >>german<< (and 8-bit) are used if set to german."
   :group 'font-latex-highlighting-faces)
 
 (defvar font-latex-doctex-keywords
-  (append font-latex-keywords-2
+  (append '((font-latex-doctex-match-^^A 0 font-lock-comment-face t))
+          font-latex-keywords-2
           '(("^%<[^>]*>" (0 font-latex-doctex-preprocessor-face t)))))
 
-;; Copy and adaptation of `doctex-font-lock-^^A' in `tex-mode.el' of
-;; CVS Emacs (March 2004)
-(defun font-latex-doctex-^^A ()
-  (if (eq (char-after (line-beginning-position)) ?\%)
-      (progn
-        (put-text-property
-         (1- (match-beginning 1)) (match-beginning 1) 'syntax-table
-         (if (= (1+ (line-beginning-position)) (match-beginning 1))
-             ;; The `%' is a single-char comment, which Emacs
-             ;; syntax-table can't deal with.  We could turn it
-             ;; into a non-comment, or use `\n%' or `%^' as the comment.
-             ;; Instead, we include it in the ^^A comment.
-             (eval-when-compile (string-to-syntax "< b"))
-           ;; FIXME: Those `eval-when-compile' shouldn't be needed any
-           ;; more since the byte-compiler will precompute those calls
-           ;; anyway (because `string-to-syntax'  is marked as pure).
-           (eval-when-compile (string-to-syntax ">"))))
-        (let ((end (line-end-position)))
-          (if (< end (point-max))
-              (put-text-property end (1+ end) 'syntax-table
-                                 (eval-when-compile
-                                   (string-to-syntax "> b")))))
-        (eval-when-compile (string-to-syntax "< b")))))
+(defun font-latex-doctex-match-^^A (limit)
+  "In docTeX mode, match comment started by ^^A before LIMIT."
+  (catch 'found
+    (while (TeX-search-forward-unescaped "^^A" limit t)
+      (when (eq (char-after (line-beginning-position)) ?\%)
+        (forward-line 1)
+        ;; Adjust `font-latex--updated-region-end' if necessary.
+        (let ((p (point)))
+          (if (< font-latex--updated-region-end limit)
+              (setq font-latex--updated-region-end limit))
+          (when (< font-latex--updated-region-end p)
+            (font-lock-unfontify-region
+             font-latex--updated-region-end p)
+            (setq font-latex--updated-region-end p))
+          (set-match-data (list (match-beginning 0) p)))
+        (throw 'found t)))))
 
 ;; Copy and adaptation of `doctex-font-lock-syntactic-face-function'
 ;; in `tex-mode.el' of CVS Emacs (March 2004)

>>>>> Ikumi Keita <[email-masked]> writes:
>>>>> Markus Kohm <[email-masked]> writes:

>> The LaTeX package doc, that is usually used via class ltxdoc for the
>> documentation part of dtx-files, declares ^^A as comment character.
>> But auctex's doctex mode fails scanning for balanced {...} pairs, if
>> } is immediately followed by ^^A. You can test it, with this example file:

> The reason of this error is the syntax-table property put on the closing
> curly brace.  In order to achieve different fontification on the docTeX
> documentation and the comment after ^^A, the function
> `font-latex-doctex-^^A' puts the "comment ender" syntax on the closing
> "}", which hides the proper syntax "closing paren paired with {" and
> thus leads to the reported error.

> Actually, `font-latex-doctex-^^A' is essentially just a copy of
> `doctex-font-lock-^^A' in `tex-mode.el', so the emacs built-in
> doctex-mode shares exactly the same problem with respect to the given
> test file at double-clicking.

> It was hard to me to resolve this problem keeping the closing "}" having
> "comment ender" syntax, so my proposal is to borrow an idea used at
> fontifying the verbatim-like environments.  I.e, to shift the position
> of "comment ender" syntax 1 character rightward and put another "comment
> starter" syntax on the second, not the first, "^" of ^^A.  The wrong
> fontification which the first "^" receives is overwrote afterwards by an
> additional entry in `font-latex-doctex-keywords'.  See the attached
> patch.

> Strictly speaking, this patch sets the boundary of two different types
> of comments in "SOMETHING^^A" at wrong position, not between "G" and the
> first "^", but between the first and second "^".  However, the treatment
> of comments of AUCTeX is rather regexp centric and slight in syntax, I
> expect this change doesn't bring serious bad side effects at editing
> docTeX documents such as filling and indenting.

> How do you think about this idea?  Any comments are appreciated.

> Regards,
> Ikumi Keita

>> --- test.dtx ---
>> % \begin{macro}{\test}^^A This is an official dtx-comment
>> % This does not work. Try to double-click to the closing brace before the
>> % comment and you will get: |Scan error: "Unbalanced parantheses", 22, 22|
>> % \end{macro}
>> %
>> % \begin{macro}{\test}%^^A This is a workaround for an official dtx-comment
>> % This does work, but the \% after the closing brace is not wanted.
>> % \end{macro}
>> % \endinput
>> % Local Variables:
>> % mode: doctex
>> % TeX-master: t
>> % End:
>> --- end of file ---