Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros

From: "J.D. Smith" <[email-masked]>
Date: Mon, 04 Aug 2025 11:21:39 -0400

Eli Zaretskii <[email-masked]> writes:

>> From: "J.D. Smith" <[email-masked]>
>> Cc: [email-masked],  [email-masked],  [email-masked]
>> Date: Sun, 03 Aug 2025 18:53:34 -0400
>> 
>> > No, I mean _why_ we want to request that in those places and not the
>> > others.
>> 
>> The short but unsatisfying answer is to preserve the hard-coded list of
>> macros to expand, which this functionality replaces and extends.  The
>> former explanation in `loaddefs-generate--make-autoload' was quite
>> terse:
>> 
>>  ;; For complex cases, try again on the macro-expansion.
>> 
>> I've tried to inject a short explanation everywhere this declaration is
>> used.  I left the `byte-run--set-autoload-macro' alone, since that's
>> just a generic property setter, and values other than `expand' may be
>> used in the future. Latest patch below.
>
> This is much better (thanks!), but I think we need to improve it some
> more by saying in the manual that this declaration is an alternative
> to hard-coding macros in loaddefs--defining-macros in loaddefs-gen.el.

OK, I have implemented these doc suggestions and a few other minor
improvements in the attached.  I've also tried to make a somewhat
unclear situation w.r.t. the hard-coded macro lists more clear in the
comments.

`loaddefs--defining-macros' is the "shortcut" list for turning a
function-defining macro directly into an (autoload ..) form, whereas
(autoload-macro expand) replaced a /different/ list of macros: those to
be expanded.  So there where 2 main hard-coded lists, and now there is
1.  The lists are closely related, because usually a macro expands into
one or more of the shortcut forms.

I conjecture (but have not verified), that ALL hard-coded lists of
macros can eventually be removed, save some very basic primitives like
defalias (and defmacro?).  Whether that's desirable is less clear.

diff --git a/doc/lispref/functions.texi b/doc/lispref/functions.texi
index be3c1e78889..be6f9507dbe 100644
--- a/doc/lispref/functions.texi
+++ b/doc/lispref/functions.texi
@@ -2633,6 +2633,14 @@ Declare Form
 This is typically used for macros, though it works for functions too.
 @xref{Indenting Macros}.
 
+@cindex @code{autoload-macro} (@code{declare} spec)
+@item (autoload-macro @var{value})
+This is used when defining a macro.  If @var{value} is @samp{expand},
+any calls to the macro which follow an autoload comment will first be
+expanded during generation of the autoloads.  This declaration is used
+as an alternative to hard-coding lists of macros to expand in
+@code{loaddefs-generate--make-autoload}.  @xref{Autoload}.
+
 @item (interactive-only @var{value})
 Set the function's @code{interactive-only} property to @var{value}.
 @xref{The interactive-only property}.
diff --git a/doc/lispref/loading.texi b/doc/lispref/loading.texi
index 750f6e76eff..a6e1c4385d1 100644
--- a/doc/lispref/loading.texi
+++ b/doc/lispref/loading.texi
@@ -645,10 +645,12 @@ Autoload
 
   The same magic comment can copy any kind of form into
 @file{loaddefs.el}.  The form following the magic comment is copied
-verbatim, @emph{except} if it is one of the forms which the autoload
-facility handles specially (e.g., by conversion into an
-@code{autoload} call).  The forms which are not copied verbatim are
-the following:
+verbatim, unless it is a form which the autoload facility handles
+specially, by conversion directly into an @code{autoload} call, or by
+recursively expanding the macro.  Any macro can request expansion of
+autoloads which call it during generation; @xref{autoload-macro expand}.
+
+The following forms are handled specially:
 
 @table @asis
 @item Definitions for function or function-like objects:
@@ -664,8 +666,11 @@ Autoload
 
 @item Other definition types:
 @code{defcustom}, @code{defgroup}, @code{deftheme}, @code{defclass}
-(@pxref{Top,EIEIO,,eieio,EIEIO}), and @code{define-skeleton}
-(@pxref{Top,Autotyping,,autotype,Autotyping}).
+(@pxref{Top,EIEIO,,eieio,EIEIO}), @code{define-skeleton}
+(@pxref{Top,Autotyping,,autotype,Autotyping}),
+@code{transient-define-prefix}, @code{transient-define-suffix},
+@code{transient-define-infix}, and @code{transient-define-argument}
+(@pxref{TOP,Transient,,transient,Transient User and Developer Manual}).
 @end table
 
   You can also use a magic comment to execute a form at build time
@@ -707,11 +712,50 @@ Autoload
 documentation string is replaced with the function's name when the
 various help functions (@pxref{Help Functions}) display it.
 
+@anchor{autoload-macro expand}
   If you write a function definition with an unusual macro that is not
-one of the known and recognized function definition methods, use of an
-ordinary magic autoload comment would copy the whole definition into
-@code{loaddefs.el}.  That is not desirable.  You can put the desired
-@code{autoload} call into @code{loaddefs.el} instead by writing this:
+one of the known and recognized function definition methods, using an
+ordinary magic autoload comment with a call to the macro would require
+autoloading the macro definition itself to work.  Doing so copies the
+whole macro definition into the autoload file.  If that is not desired,
+you can use the special declare form @code{(autoload-macro expand)} in
+your macro definition (@pxref{Declare Form}), instead of autoloading it.
+This indicates to the autoload system that calls to your macro following
+@samp{;;;###autoload} should be expanded, similar to how the special
+functions listed above are handled.  For example, a macro which wraps
+@code{define-minor-mode} can request expansion, so that proper
+@code{autoload} calls for the function it defines are generated.
+
+@cindex @code{:autoload-end}
+The keyword symbol @code{:autoload-end} can be used in the expansion of
+a macro to prevent including unwanted forms in the autoload output.  Its
+presence causes any further elements within the form where it appears to
+be silently skipped.  For example, if during autoload generation, a
+macro's expansion includes:
+
+@example
+(progn
+  (put my-mode 'visible-prop t)
+  :autoload-end
+  (put my-mode 'hidden-prop nil))
+@end example
+
+@noindent
+the final form (@w{@code{(put my-mode 'hidden-prop nil)}}) will not be
+copied into the autoload file.
+
+Note that, if a symbol in the @code{car} of an autoloaded form is found
+to be undefined during autoload generation, the file in which the
+associated @samp{;;;###autoload} appears will itself be loaded, to give
+the macro a chance to be defined.  Packages which use special loading
+mechanisms, including loading their own @file{@var{package}-loaddefs.el}
+file, should therefore gracefully handle load errors, since these files
+may not yet exist during autoload generation.  This can be done, e.g.,
+by setting the @var{no-error} argument of @code{require} non-@code{nil}
+(@pxref{Named Features})).
+
+Alternatively, instead of expansion, you can put the desired
+@code{autoload} call into @file{loaddefs.el} by writing this:
 
 @example
 ;;;###autoload (autoload 'foo "myfile")
diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 366423904db..48562b8a36b 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -271,6 +271,12 @@ 'byte-run--set-debug
 	    (list 'put (list 'quote name)
 		  ''edebug-form-spec (list 'quote spec)))))
 
+(defalias 'byte-run--set-autoload-macro
+  #'(lambda (name _args spec)
+      (list 'function-put (list 'quote name)
+	    ''autoload-macro (list 'quote spec)))
+  "Handle autoload-macro declarations")
+
 (defalias 'byte-run--set-no-font-lock-keyword
   #'(lambda (name _args val)
       (list 'function-put (list 'quote name)
@@ -350,8 +356,13 @@ macro-declarations-alist
   (cons
    (list 'debug #'byte-run--set-debug)
    (cons
-    (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
-    defun-declarations-alist))
+    ;; macros can declare (autoload-macro expand) to request expansion
+    ;; during autoload generation of forms calling them.  See
+    ;; `loaddefs-generate--make-autoload'.
+    (list 'autoload-macro #'byte-run--set-autoload-macro)
+    (cons
+     (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+     defun-declarations-alist)))
   "List associating properties of macros to their macro expansion.
 Each element of the list takes the form (PROP FUN) where FUN is a function.
 For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
@@ -397,6 +408,8 @@ 'defmacro
            (if declarations
 	       (cons 'prog1 (cons def (car declarations)))
 	     def))))))
+;; Expand to defalias and related forms on autoload gen
+(function-put 'defmacro 'autoload-macro 'expand) ; Since we cannot `declare' it
 
 ;; Now that we defined defmacro we can use it!
 (defmacro defun (name arglist &rest body)
@@ -409,7 +422,9 @@ defun
 The return value is undefined.
 
 \(fn NAME ARGLIST [DOCSTRING] [DECL] [INTERACTIVE] BODY...)"
-  (declare (doc-string 3) (indent 2))
+  (declare (doc-string 3) (indent 2)
+           ;; Expand to defalias on autoload gen
+           (autoload-macro expand))
   (or name (error "Cannot define '%s' as a function" name))
   (if (null
        (and (listp arglist)
diff --git a/lisp/emacs-lisp/cl-generic.el b/lisp/emacs-lisp/cl-generic.el
index 3aa26fba3c3..b5da24be2fe 100644
--- a/lisp/emacs-lisp/cl-generic.el
+++ b/lisp/emacs-lisp/cl-generic.el
@@ -258,7 +258,9 @@ cl-defgeneric
                                cl--generic-edebug-make-name in:method]
                               lambda-doc
                               def-body)]]
-             def-body)))
+             def-body))
+           ;; Expand to defun and related forms on autoload definition
+           (autoload-macro expand))
   (let* ((doc (if (stringp (car-safe options-and-methods))
                   (pop options-and-methods)))
          (declarations nil)
diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 2a0a9e5c6de..617991308b6 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -398,7 +398,9 @@ cl-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           ;; expand to function definition on autoload gen
+           (autoload-macro expand))
   `(defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
 ;;;###autoload
@@ -416,7 +418,9 @@ cl-iter-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           ;; expand (eventually) to function definition on autoload gen
+           (autoload-macro expand))
   (require 'generator)
   `(iter-defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
@@ -475,7 +479,8 @@ cl-defmacro
   (declare (debug
             (&define name cl-macro-list cl-declarations-or-string def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))   ; expand to defmacro on autoload gen
   `(defmacro ,name ,@(cl--transform-lambda (cons args body) name)))
 
 (def-edebug-elem-spec 'cl-lambda-expr
@@ -3011,7 +3016,9 @@ cl-defstruct
                         sexp])]
              [&optional stringp]
              ;; All the above is for the following def-form.
-             &rest &or symbolp (symbolp &optional def-form &rest sexp))))
+             &rest &or symbolp (symbolp &optional def-form &rest sexp)))
+           ;; expand to function definitions and related forms on autoload gen
+           (autoload-macro expand))
   (let* ((name (if (consp struct) (car struct) struct))
 	 (warning nil)
 	 (opts (cdr-safe struct))
diff --git a/lisp/emacs-lisp/easy-mmode.el b/lisp/emacs-lisp/easy-mmode.el
index f53db48f0b7..ff96a7afac6 100644
--- a/lisp/emacs-lisp/easy-mmode.el
+++ b/lisp/emacs-lisp/easy-mmode.el
@@ -222,10 +222,12 @@ define-minor-mode
            (indent defun)
            (debug (&define name string-or-null-p
 			   [&optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp]
+			              &optional [&not keywordp] sexp
+			              &optional [&not keywordp] sexp]
 			   [&rest [keywordp sexp]]
-			   def-body)))
+			   def-body))
+           ;; expand to the command definition on autoload gen
+           (autoload-macro expand))
 
   (let* ((last-message (make-symbol "last-message"))
          (mode-name (symbol-name mode))
@@ -487,7 +489,9 @@ define-globalized-minor-mode
 on if the hook has explicitly disabled it.
 
 \(fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)"
-  (declare (doc-string 2) (indent defun))
+  (declare (doc-string 2) (indent defun)
+           ;; expand to the minor-mode definition on autoload gen
+           (autoload-macro expand))
   (let* ((global-mode-name (symbol-name global-mode))
 	 (mode-name (symbol-name mode))
 	 (pretty-name (easy-mmode-pretty-mode-name mode))
diff --git a/lisp/emacs-lisp/generator.el b/lisp/emacs-lisp/generator.el
index 344e11e245e..c01e559fd5c 100644
--- a/lisp/emacs-lisp/generator.el
+++ b/lisp/emacs-lisp/generator.el
@@ -675,7 +675,8 @@ iter-defun
 of values.  Callers can retrieve each value using `iter-next'."
   (declare (indent defun)
            (debug (&define name lambda-list lambda-doc &rest sexp))
-           (doc-string 3))
+           (doc-string 3)
+           (autoload-macro expand)) ; expand to the defun on autoload gen
   (cl-assert lexical-binding)
   (let* ((parsed-body (macroexp-parse-body body))
          (declarations (car parsed-body))
diff --git a/lisp/emacs-lisp/inline.el b/lisp/emacs-lisp/inline.el
index c015e2b57d0..027c24c465b 100644
--- a/lisp/emacs-lisp/inline.el
+++ b/lisp/emacs-lisp/inline.el
@@ -135,7 +135,8 @@ define-inline
 This is like `defmacro', but has several advantages.
 See Info node `(elisp)Defining Functions' for more details."
   ;; FIXME: How can this work with CL arglists?
-  (declare (indent defun) (debug defun) (doc-string 3))
+  (declare (indent defun) (debug defun) (doc-string 3)
+           (autoload-macro expand))     ; expand to the defun on autoload gen
   (let ((doc (if (stringp (car-safe body)) (list (pop body))))
         (declares (if (eq (car-safe (car-safe body)) 'declare) (pop body)))
         (cm-name (intern (format "%s--inliner" name)))
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..6ec600fea91 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -142,12 +142,37 @@ loaddefs-generate--shorten-autoload
              3)
         form))
 
+;; The following macros are known to define functions, and are treated
+;; specially when encountered during autoload generation, translating
+;; calls to them directly into appropriate (autoload function ...)
+;; forms.
+;;
+;; An alternative to appearing on this list is for a macro to declare
+;; (autoload-macro expand), so calls to it get expanded into more basic
+;; forms during generation.  Macros may be removed from this list once
+;; they request such expansion and produce suitable output (e.g. by
+;; employing :autoload-end to omit unneeded forms).
+(defconst loaddefs--defining-macros
+  '(define-skeleton define-derived-mode
+     define-compilation-mode define-generic-mode
+     easy-mmode-define-global-mode define-global-minor-mode
+     define-globalized-minor-mode
+     easy-mmode-define-minor-mode define-minor-mode
+     cl-defun defun* cl-defmacro defmacro*
+     define-overloadable-function
+     transient-define-prefix transient-define-suffix
+     transient-define-infix transient-define-argument))
+
+(defvar loaddefs--load-error-files nil)
 (defun loaddefs-generate--make-autoload (form file &optional expansion)
   "Turn FORM into an autoload or defvar for source file FILE.
 Returns nil if FORM is not a special autoload form (i.e. a function definition
 or macro definition or a defcustom).
 If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently.
+
+Note that macros can request expansion by including `(autoload-macro
+expand)' among their `declare' forms."
   (let ((car (car-safe form)) expand)
     (cond
      ((and expansion (eq car 'defalias))
@@ -191,35 +216,40 @@ loaddefs-generate--make-autoload
           (setq form (copy-sequence form))
           (setcdr (memq :autoload-end form) nil))
         (let ((exps (delq nil (mapcar (lambda (form)
-                                        (loaddefs-generate--make-autoload
-                                         form file expansion))
+                                        (unless (eq form :autoload-end)
+                                          (loaddefs-generate--make-autoload
+                                           form file expansion)))
                                       (cdr form)))))
           (when exps (cons 'progn exps)))))
 
-     ;; For complex cases, try again on the macro-expansion.
-     ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
-                       define-globalized-minor-mode defun defmacro
-		       easy-mmode-define-minor-mode define-minor-mode
-                       define-inline cl-defun cl-defmacro cl-defgeneric
-                       cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
-           (macrop car)
-	   (setq expand (let ((load-true-file-name file)
-                              (load-file-name file))
-                          (macroexpand form)))
-	   (memq (car expand) '(progn prog1 defalias)))
+     ;; For macros which request it, try again on their expansion.
+     ((progn
+        ;; If the car is an unknown symbol, we load the file first to
+        ;; give packages a chance to define their macros.
+        (unless (or (not (symbolp car)) (fboundp car)
+                    ;; Special cases handled below
+                    (memq car loaddefs--defining-macros)
+                    (memq car '(defclass defcustom deftheme defgroup nil))
+                    (assoc file load-history)
+                    (member file loaddefs--load-error-files))
+          (let ((load-path (cons (file-name-directory file) load-path)))
+            (message "loaddefs-gen: loading file %s (for %s)" file car)
+            (condition-case e (load file)
+              (error
+               (push file loaddefs--load-error-files) ; do not attempt again
+               (warn "loaddefs-gen: load error\n\t%s" e)))))
+        (and (macrop car)
+	     (eq 'expand (function-get car 'autoload-macro))
+	     (setq expand (let ((load-true-file-name file)
+				(load-file-name file))
+			    (macroexpand form)))
+	     (not (eq car (car expand)))))
       ;; Recurse on the expansion.
       (loaddefs-generate--make-autoload expand file 'expansion))
 
-     ;; For special function-like operators, use the `autoload' function.
-     ((memq car '(define-skeleton define-derived-mode
-                   define-compilation-mode define-generic-mode
-		   easy-mmode-define-global-mode define-global-minor-mode
-		   define-globalized-minor-mode
-		   easy-mmode-define-minor-mode define-minor-mode
-		   cl-defun defun* cl-defmacro defmacro*
-                   define-overloadable-function
-                   transient-define-prefix transient-define-suffix
-                   transient-define-infix transient-define-argument))
+     ;; For known special macros which define functions, use `autoload'
+     ;; directly.
+     ((memq car loaddefs--defining-macros)
       (let* ((macrop (memq car '(defmacro cl-defmacro defmacro*)))
 	     (name (nth 1 form))
 	     (args (pcase car
diff --git a/lisp/emacs-lisp/pcase.el b/lisp/emacs-lisp/pcase.el
index 73554fd66fd..5e85ab4f075 100644
--- a/lisp/emacs-lisp/pcase.el
+++ b/lisp/emacs-lisp/pcase.el
@@ -537,7 +537,9 @@ pcase-defmacro
 By convention, DOC should use \"EXPVAL\" to stand
 for the result of evaluating EXP (first arg to `pcase').
 \n(fn NAME ARGS [DOC] &rest BODY...)"
-  (declare (indent 2) (debug defun) (doc-string 3))
+  (declare (indent 2) (debug defun) (doc-string 3)
+           ;; Expand to defun and related forms on autoload gen
+           (autoload-macro expand))
   ;; Add the function via `fsym', so that an autoload cookie placed
   ;; on a pcase-defmacro will cause the macro to be loaded on demand.
   (let ((fsym (intern (format "%s--pcase-macroexpander" name)))