Subject: Re: bug#35140: 12.1; Scan errors in doctex mode with ^^A-comments after braces

From: Ikumi Keita <[email-masked]>
Date: Tue, 07 May 2019 12:39:35 +0900

Hi Markus and all developpers,

>>>>> Markus Kohm <[email-masked]> writes:

> The LaTeX package doc, that is usually used via class ltxdoc for the
> documentation part of dtx-files, declares ^^A as comment character.
> But auctex's doctex mode fails scanning for balanced {...} pairs, if
> } is immediately followed by ^^A. You can test it, with this example file:

The reason of this error is the syntax-table property put on the closing
curly brace.  In order to achieve different fontification on the docTeX
documentation and the comment after ^^A, the function
`font-latex-doctex-^^A' puts the "comment ender" syntax on the closing
"}", which hides the proper syntax "closing paren paired with {" and
thus leads to the reported error.

Actually, `font-latex-doctex-^^A' is essentially just a copy of
`doctex-font-lock-^^A' in `tex-mode.el', so the emacs built-in
doctex-mode shares exactly the same problem with respect to the given
test file at double-clicking.

It was hard to me to resolve this problem keeping the closing "}" having
"comment ender" syntax, so my proposal is to borrow an idea used at
fontifying the verbatim-like environments.  I.e, to shift the position
of "comment ender" syntax 1 character rightward and put another "comment
starter" syntax on the second, not the first, "^" of ^^A.  The wrong
fontification which the first "^" receives is overwrote afterwards by an
additional entry in `font-latex-doctex-keywords'.  See the attached
patch.

Strictly speaking, this patch sets the boundary of two different types
of comments in "SOMETHING^^A" at wrong position, not between "G" and the
first "^", but between the first and second "^".  However, the treatment
of comments of AUCTeX is rather regexp centric and slight in syntax, I
expect this change doesn't bring serious bad side effects at editing
docTeX documents such as filling and indenting.

How do you think about this idea?  Any comments are appreciated.

Regards,
Ikumi Keita

> --- test.dtx ---
> % \begin{macro}{\test}^^A This is an official dtx-comment
> % This does not work. Try to double-click to the closing brace before the
> % comment and you will get: |Scan error: "Unbalanced parantheses", 22, 22|
> % \end{macro}
> %
> % \begin{macro}{\test}%^^A This is a workaround for an official dtx-comment
> % This does work, but the \% after the closing brace is not wanted.
> % \end{macro}
> % \endinput
> % Local Variables:
> % mode: doctex
> % TeX-master: t
> % End:
> --- end of file ---

>From af36beb67d385dcb9afbca107248d16e3be3e2b7 Mon Sep 17 00:00:00 2001
From: Ikumi Keita <[email-masked]>
Date: Fri, 3 May 2019 14:59:50 +0900
Subject: [PATCH] Improve comment syntax handling in docTeX mode

* font-latex.el (font-latex-set-syntactic-keywords):
(font-latex-doctex-^^A):
Assign comment ender syntax to the first "^" of ^^A, not its preceding
character.  Begin style b comment with the second "^".  These fix
bug#35140.
(font-latex-doctex-keywords): Add an entry so as to give the correct
fontification to the first "^" above.
(font-latex-find-matching-close): Add comments.
---
 font-latex.el | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git a/font-latex.el b/font-latex.el
index 486f39fe..cf84f261 100644
--- a/font-latex.el
+++ b/font-latex.el
@@ -1053,7 +1053,7 @@ have changed."
   (setq font-latex-doctex-syntactic-keywords
 	(append font-latex-syntactic-keywords
 		;; For docTeX comment-in-doc.
-		`(("\\(\\^\\)\\^A" (1 (font-latex-doctex-^^A)))))))
+		'(("\\^\\(\\^\\)A" (1 (font-latex-doctex-^^A)))))))

 ;;; Syntactic fontification
@@ -1464,6 +1464,10 @@ ignored during the search."
 	;; XXX: Do not look up syntax-table properties since they may
 	;; be misleading, e.g. in the case of "{foo}^^A" where the
 	;; closing brace gets a comment end syntax.
+	;; (Comment added on May 2019): This particular example is no
+	;; longer a problem with the modification on
+	;; `font-latex-doctex-^^A'.  Not sure there are other cases
+	;; that we can not rely on syntax property.
 	(parse-sexp-lookup-properties nil))
     (or
      (condition-case nil
@@ -2173,27 +2177,32 @@ END marks boundaries for searching for quotation ends."
 
 (defvar font-latex-doctex-keywords
   (append font-latex-keywords-2
-	  '(("^%<[^>]*>" (0 font-latex-doctex-preprocessor-face t)))))
+	  '(("^%<[^>]*>" (0 font-latex-doctex-preprocessor-face t))
+	    ("\\(\\^\\)\\^A" (1 (get-text-property (match-end 1) 'face) t)))))
 
 ;; Copy and adaptation of `doctex-font-lock-^^A' in `tex-mode.el' of
 ;; CVS Emacs (March 2004)
 (defun font-latex-doctex-^^A ()
   (if (eq (char-after (line-beginning-position)) ?\%)
       (progn
+	;; Assign comment ender syntax to the first "^" of ^^A.  It
+	;; used to be assigned to the character just before the "^",
+	;; but that sometimes caused trouble because the closing brace
+	;; in "{foo}^^A" didn't get the close paren syntax.  See
+	;; comment in `font-latex-find-matching-close' and bug#35140.
+	;; There is a hack in `font-latex-doctex-keywords' to restore
+	;; the correct fontification of this "^".
 	(put-text-property
-	 (1- (match-beginning 1)) (match-beginning 1) 'syntax-table
-	 (if (= (1+ (line-beginning-position)) (match-beginning 1))
-	     ;; The `%' is a single-char comment, which Emacs
-	     ;; syntax-table can't deal with.  We could turn it
-	     ;; into a non-comment, or use `\n%' or `%^' as the comment.
-	     ;; Instead, we include it in the ^^A comment.
-	     (eval-when-compile (string-to-syntax "< b"))
-	   (eval-when-compile (string-to-syntax ">"))))
+	 (match-beginning 0) (match-beginning 1) 'syntax-table
+	 (eval-when-compile (string-to-syntax ">")))
+	;; Make the EOL as the boundary of the style b comment which
+	;; begins with ^^A.
 	(let ((end (line-end-position)))
 	  (if (< end (point-max))
 	      (put-text-property end (1+ end) 'syntax-table
 				    (eval-when-compile
 				      (string-to-syntax "> b")))))
+	;; Begin style b comment with the second "^" of ^^A.
 	(eval-when-compile (string-to-syntax "< b")))))
 
 ;; Copy and adaptation of `doctex-font-lock-syntactic-face-function'
-- 
2.21.0