Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros

From: "J.D. Smith" <[email-masked]>
Date: Tue, 29 Jul 2025 21:41:12 -0400

Stefan Monnier <[email-masked]> writes:

>> One big category of nil car's resulted from all the :autoload-end's around.
>
> Oh, what you mean is that nil is when the element is not a cons cell?
> Indeed that makes sense.
> [ Cons cells with a nil element would be surprising, OTOH.  ]

Yep, like :autoload-end :). 

>> That leaves a number with explicit nil cars in their expansion, for
>> whatever reason:
>
> I think these have `nil` in the list of forms (that follows a `progn`),
> but not in the `car` of one of the forms, right?  So again, it's when
> `form` is not a cons-cell.
> In any case, indeed those non-cons situations justify the "not nil" check.

Yes, but I just wonder what such non-conses are doing in the expansion.
Maybe just doc strings or some such.

>> No, it's a good chance to learn.  Will want to iterate on docs first, if
>> you think its ready (I think it is).
>
> I was thinking to do it in the single-hunk bug fix, which I do think is
> ready (it just needs a commit message).

Is the commit message in need of expansion?  I worked up some quick docs
and included in the patch; see attached.

>> Have you had a chance to test it "in the wild" with a custom macro?
>
> No, not yet.  But I think the transient macros are a good test case.

Good idea.  Michael, if you can try this along with a patch like the one
Stefan proposed for tramp (correcting the declare form name) that would
be useful.  Assuming it all works as expected, you'll notice
`tramp-loaddefs.el' no longer contains the macro and the calls to it,
but rather only the expanded forms it generates.

diff --git a/doc/lispref/functions.texi b/doc/lispref/functions.texi
index be3c1e78889..25a01d942b6 100644
--- a/doc/lispref/functions.texi
+++ b/doc/lispref/functions.texi
@@ -2633,6 +2633,11 @@ Declare Form
 This is typically used for macros, though it works for functions too.
 @xref{Indenting Macros}.
 
+@item (autoload-macro @var{value})
+This is used when defining a macro.  If @var{value} is @samp{expand},
+any calls to the macro which follow an autoload comment will first be
+expanded during generation of the autoloads.  @xref{Autoload}.
+
 @item (interactive-only @var{value})
 Set the function's @code{interactive-only} property to @var{value}.
 @xref{The interactive-only property}.
diff --git a/doc/lispref/loading.texi b/doc/lispref/loading.texi
index 750f6e76eff..65a55547a9f 100644
--- a/doc/lispref/loading.texi
+++ b/doc/lispref/loading.texi
@@ -710,8 +710,31 @@ Autoload
   If you write a function definition with an unusual macro that is not
 one of the known and recognized function definition methods, use of an
 ordinary magic autoload comment would copy the whole definition into
-@code{loaddefs.el}.  That is not desirable.  You can put the desired
-@code{autoload} call into @code{loaddefs.el} instead by writing this:
+@code{loaddefs.el}.  That is often not desirable.  In this case, you can
+use the special declare form @code{(autoload-macro expand)} in your
+macro definition (@pxref{Declare Form}).  This indicates to the autoload
+system that calls to your macro following @samp{;;;###autoload} should
+be expanded, similar to how the special functions listed above are
+handled.  For example, a macro which wraps @code{define-minor-mode} can
+request expansion, so that proper @code{autoload} calls for the function
+it defines are generated.
+
+The symbol @code{:autoload-end} can be used in the expansion to prevent
+including unwanted forms in the autoload output.  Its presence causes
+any further elements within the list where it appears to be silently
+skipped.
+
+Note that, if a symbol in the car of an autoloaded form is not defined
+during autoload generation, the file in which the associated
+@samp{;;;###autoload} appears will itself be loaded, to give the macro a
+chance to be defined.  Packages which use special loading mechanisms,
+including requiring their own @samp{foo-autoload.el} file, should
+therefore gracefully handle load errors, since these files may not yet
+exist during autoload generation.  This can be done, e.g., by setting
+the @var{no-error} argument of @code{require} non-@code{nil}).
+
+Alternatively, instead of expansion, you can put the desired
+@code{autoload} call into @code{loaddefs.el} by writing this:
 
 @example
 ;;;###autoload (autoload 'foo "myfile")
diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 366423904db..3e521e18567 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -271,6 +271,11 @@ 'byte-run--set-debug
 	    (list 'put (list 'quote name)
 		  ''edebug-form-spec (list 'quote spec)))))
 
+(defalias 'byte-run--set-autoload-macro
+  #'(lambda (name _args spec)
+      (list 'function-put (list 'quote name)
+	    ''autoload-macro (list 'quote spec))))
+
 (defalias 'byte-run--set-no-font-lock-keyword
   #'(lambda (name _args val)
       (list 'function-put (list 'quote name)
@@ -350,8 +355,10 @@ macro-declarations-alist
   (cons
    (list 'debug #'byte-run--set-debug)
    (cons
-    (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
-    defun-declarations-alist))
+    (list 'autoload-macro #'byte-run--set-autoload-macro)
+    (cons
+     (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+     defun-declarations-alist)))
   "List associating properties of macros to their macro expansion.
 Each element of the list takes the form (PROP FUN) where FUN is a function.
 For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
@@ -397,6 +404,7 @@ 'defmacro
            (if declarations
 	       (cons 'prog1 (cons def (car declarations)))
 	     def))))))
+(function-put 'defmacro 'autoload-macro 'expand) ; Since we cannot `declare' it
 
 ;; Now that we defined defmacro we can use it!
 (defmacro defun (name arglist &rest body)
@@ -409,7 +417,7 @@ defun
 The return value is undefined.
 
 \(fn NAME ARGLIST [DOCSTRING] [DECL] [INTERACTIVE] BODY...)"
-  (declare (doc-string 3) (indent 2))
+  (declare (doc-string 3) (indent 2) (autoload-macro expand))
   (or name (error "Cannot define '%s' as a function" name))
   (if (null
        (and (listp arglist)
diff --git a/lisp/emacs-lisp/cl-generic.el b/lisp/emacs-lisp/cl-generic.el
index 3aa26fba3c3..4a85d89d5fa 100644
--- a/lisp/emacs-lisp/cl-generic.el
+++ b/lisp/emacs-lisp/cl-generic.el
@@ -258,7 +258,8 @@ cl-defgeneric
                                cl--generic-edebug-make-name in:method]
                               lambda-doc
                               def-body)]]
-             def-body)))
+             def-body))
+           (autoload-macro expand))
   (let* ((doc (if (stringp (car-safe options-and-methods))
                   (pop options-and-methods)))
          (declarations nil)
diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 2a0a9e5c6de..1b64f106a1b 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -398,7 +398,8 @@ cl-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))
   `(defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
 ;;;###autoload
@@ -416,7 +417,8 @@ cl-iter-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))
   (require 'generator)
   `(iter-defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
@@ -475,7 +477,8 @@ cl-defmacro
   (declare (debug
             (&define name cl-macro-list cl-declarations-or-string def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))
   `(defmacro ,name ,@(cl--transform-lambda (cons args body) name)))
 
 (def-edebug-elem-spec 'cl-lambda-expr
@@ -3011,7 +3014,8 @@ cl-defstruct
                         sexp])]
              [&optional stringp]
              ;; All the above is for the following def-form.
-             &rest &or symbolp (symbolp &optional def-form &rest sexp))))
+             &rest &or symbolp (symbolp &optional def-form &rest sexp)))
+           (autoload-macro expand))
   (let* ((name (if (consp struct) (car struct) struct))
 	 (warning nil)
 	 (opts (cdr-safe struct))
diff --git a/lisp/emacs-lisp/easy-mmode.el b/lisp/emacs-lisp/easy-mmode.el
index f53db48f0b7..0e428abd0b7 100644
--- a/lisp/emacs-lisp/easy-mmode.el
+++ b/lisp/emacs-lisp/easy-mmode.el
@@ -222,10 +222,11 @@ define-minor-mode
            (indent defun)
            (debug (&define name string-or-null-p
 			   [&optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp]
+			              &optional [&not keywordp] sexp
+			              &optional [&not keywordp] sexp]
 			   [&rest [keywordp sexp]]
-			   def-body)))
+			   def-body))
+           (autoload-macro expand))
 
   (let* ((last-message (make-symbol "last-message"))
          (mode-name (symbol-name mode))
@@ -487,7 +488,7 @@ define-globalized-minor-mode
 on if the hook has explicitly disabled it.
 
 \(fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)"
-  (declare (doc-string 2) (indent defun))
+  (declare (doc-string 2) (indent defun) (autoload-macro expand))
   (let* ((global-mode-name (symbol-name global-mode))
 	 (mode-name (symbol-name mode))
 	 (pretty-name (easy-mmode-pretty-mode-name mode))
diff --git a/lisp/emacs-lisp/generator.el b/lisp/emacs-lisp/generator.el
index 344e11e245e..59784ff51e8 100644
--- a/lisp/emacs-lisp/generator.el
+++ b/lisp/emacs-lisp/generator.el
@@ -675,7 +675,8 @@ iter-defun
 of values.  Callers can retrieve each value using `iter-next'."
   (declare (indent defun)
            (debug (&define name lambda-list lambda-doc &rest sexp))
-           (doc-string 3))
+           (doc-string 3)
+           (autoload-macro expand))
   (cl-assert lexical-binding)
   (let* ((parsed-body (macroexp-parse-body body))
          (declarations (car parsed-body))
diff --git a/lisp/emacs-lisp/inline.el b/lisp/emacs-lisp/inline.el
index c015e2b57d0..5acd6f17ee2 100644
--- a/lisp/emacs-lisp/inline.el
+++ b/lisp/emacs-lisp/inline.el
@@ -135,7 +135,7 @@ define-inline
 This is like `defmacro', but has several advantages.
 See Info node `(elisp)Defining Functions' for more details."
   ;; FIXME: How can this work with CL arglists?
-  (declare (indent defun) (debug defun) (doc-string 3))
+  (declare (indent defun) (debug defun) (doc-string 3) (autoload-macro expand))
   (let ((doc (if (stringp (car-safe body)) (list (pop body))))
         (declares (if (eq (car-safe (car-safe body)) 'declare) (pop body)))
         (cm-name (intern (format "%s--inliner" name)))
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..f9c6f82c31f 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -142,12 +142,29 @@ loaddefs-generate--shorten-autoload
              3)
         form))
 
+;; TODO: some macros can be removed from this list once they
+;;  (declare ((autoload-macro expand) ...)
+(defconst loaddefs--defining-macros
+  '(define-skeleton define-derived-mode
+     define-compilation-mode define-generic-mode
+     easy-mmode-define-global-mode define-global-minor-mode
+     define-globalized-minor-mode
+     easy-mmode-define-minor-mode define-minor-mode
+     cl-defun defun* cl-defmacro defmacro*
+     define-overloadable-function
+     transient-define-prefix transient-define-suffix
+     transient-define-infix transient-define-argument))
+
+(defvar loaddefs--load-error-files nil)
 (defun loaddefs-generate--make-autoload (form file &optional expansion)
   "Turn FORM into an autoload or defvar for source file FILE.
 Returns nil if FORM is not a special autoload form (i.e. a function definition
 or macro definition or a defcustom).
 If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently.
+
+Note that macros can request expansion by including `(autoload-macro
+expand)' among their `declare' forms."
   (let ((car (car-safe form)) expand)
     (cond
      ((and expansion (eq car 'defalias))
@@ -191,35 +208,40 @@ loaddefs-generate--make-autoload
           (setq form (copy-sequence form))
           (setcdr (memq :autoload-end form) nil))
         (let ((exps (delq nil (mapcar (lambda (form)
-                                        (loaddefs-generate--make-autoload
-                                         form file expansion))
+                                        (unless (eq form :autoload-end)
+                                          (loaddefs-generate--make-autoload
+                                           form file expansion)))
                                       (cdr form)))))
           (when exps (cons 'progn exps)))))
 
-     ;; For complex cases, try again on the macro-expansion.
-     ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
-                       define-globalized-minor-mode defun defmacro
-		       easy-mmode-define-minor-mode define-minor-mode
-                       define-inline cl-defun cl-defmacro cl-defgeneric
-                       cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
-           (macrop car)
-	   (setq expand (let ((load-true-file-name file)
-                              (load-file-name file))
-                          (macroexpand form)))
-	   (memq (car expand) '(progn prog1 defalias)))
+     ;; For macros which request it, try again on their expansion.
+     ((progn
+        ;; If the car is an unknown symbol, we load the file first to
+        ;; give packages a chance to define their macros.
+        (unless (or (not (symbolp car)) (fboundp car)
+                    ;; Special cases handled below
+                    (memq car loaddefs--defining-macros)
+                    (memq car '(defclass defcustom deftheme defgroup nil))
+                    (assoc file load-history)
+                    (member file loaddefs--load-error-files))
+          (let ((load-path (cons (file-name-directory file) load-path)))
+            (message "loaddefs-gen: loading file %s (for %s)" file car)
+            (condition-case e (load file)
+              (error
+               (push file loaddefs--load-error-files) ; do not attempt again
+               (warn "loaddefs-gen: load error\n\t%s" e)))))
+        (and (macrop car)
+	     (eq 'expand (function-get car 'autoload-macro))
+	     (setq expand (let ((load-true-file-name file)
+				(load-file-name file))
+			    (macroexpand form)))
+	     (not (eq car (car expand)))))
       ;; Recurse on the expansion.
       (loaddefs-generate--make-autoload expand file 'expansion))
 
-     ;; For special function-like operators, use the `autoload' function.
-     ((memq car '(define-skeleton define-derived-mode
-                   define-compilation-mode define-generic-mode
-		   easy-mmode-define-global-mode define-global-minor-mode
-		   define-globalized-minor-mode
-		   easy-mmode-define-minor-mode define-minor-mode
-		   cl-defun defun* cl-defmacro defmacro*
-                   define-overloadable-function
-                   transient-define-prefix transient-define-suffix
-                   transient-define-infix transient-define-argument))
+     ;; For known special macros which define functions, use `autoload'
+     ;; directly.
+     ((memq car loaddefs--defining-macros)
       (let* ((macrop (memq car '(defmacro cl-defmacro defmacro*)))
 	     (name (nth 1 form))
 	     (args (pcase car
diff --git a/lisp/emacs-lisp/pcase.el b/lisp/emacs-lisp/pcase.el
index 73554fd66fd..ab8edf1f569 100644
--- a/lisp/emacs-lisp/pcase.el
+++ b/lisp/emacs-lisp/pcase.el
@@ -537,7 +537,7 @@ pcase-defmacro
 By convention, DOC should use \"EXPVAL\" to stand
 for the result of evaluating EXP (first arg to `pcase').
 \n(fn NAME ARGS [DOC] &rest BODY...)"
-  (declare (indent 2) (debug defun) (doc-string 3))
+  (declare (indent 2) (debug defun) (doc-string 3) (autoload-macro expand))
   ;; Add the function via `fsym', so that an autoload cookie placed
   ;; on a pcase-defmacro will cause the macro to be loaded on demand.
   (let ((fsym (intern (format "%s--pcase-macroexpander" name)))