Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros

From: "J.D. Smith" <[email-masked]>
Date: Sun, 03 Aug 2025 18:53:34 -0400

Eli Zaretskii <[email-masked]> writes:

>> From: "J.D. Smith" <[email-masked]>
>> Cc: [email-masked],  [email-masked],  [email-masked]
>> Date: Sun, 03 Aug 2025 14:16:05 -0400
>> 
>> > Can we have a comment here explaining the reason for the use of
>> > autoload-macro?
>> > And maybe here as well?
>> > And here. x 4
>> 
>> I'm not sure what kind of comment you mean.
>
> To explain why do we use this declaration there.
>
>> Are you thinking of something simple, like "request macro-expansion
>> during autoload generation"?
>
> No, I mean _why_ we want to request that in those places and not the
> others.

The short but unsatisfying answer is to preserve the hard-coded list of
macros to expand, which this functionality replaces and extends.  The
former explanation in `loaddefs-generate--make-autoload' was quite
terse:

 ;; For complex cases, try again on the macro-expansion.

I've tried to inject a short explanation everywhere this declaration is
used.  I left the `byte-run--set-autoload-macro' alone, since that's
just a generic property setter, and values other than `expand' may be
used in the future. Latest patch below.

Meta-comments:
==============

Though outside of the scope of this patch, a few mildly bothersome
attributes remain, which I leave here for posterity.

1. It's somewhat arbitrary which macros request expansion

Eventually, I think for all builtin macros, the question should be
asked: "Will calls to this macro likely be marked ;;;###autoload"?  If
the answer is yes, the macro should declare `autoload-macro=expand', so
that these calls get recursively expanded down to their most basic forms
— `defalias', `put', and so on.

2. Some autoloaded macro expansions are not "autoload-hygienic".

Some macros currently /rely/ on skipping directly to `(autoload name
...)' generation via the hard-coded "short-cut" list (see
`loaddefs--defining-macros').  I.e. they do not expect to be expanded,
and are therefore not autoload-hygenic in their expansion
(e.g. neglecting to use `:autoload-end' to elide unwanted forms, for
example).  This is confusing and prone to bugs (e.g. the bug discovered
in this thread related to the MODES arg).

3. Some macros' autoload treatment depend on whether they are defined at
autoload-generation time.

Such macros are both on the short-cut list /and/ request expansion.  So
they are expanded, but only if they are defined at the time autoload
generation occurs.  Most are, but some, e.g. cl-macs, are not.  So they
go through the "short-cut" instead.

IMO, these macros should either have their expansion cleaned up for
autoload hygiene (if necessary) and be pre-loaded during autoload
generation, or they should just not request expansion at all.

diff --git a/doc/lispref/functions.texi b/doc/lispref/functions.texi
index be3c1e78889..269c0ad0026 100644
--- a/doc/lispref/functions.texi
+++ b/doc/lispref/functions.texi
@@ -2633,6 +2633,12 @@ Declare Form
 This is typically used for macros, though it works for functions too.
 @xref{Indenting Macros}.
 
+@cindex @code{autoload-macro} (@code{declare} spec)
+@item (autoload-macro @var{value})
+This is used when defining a macro.  If @var{value} is @samp{expand},
+any calls to the macro which follow an autoload comment will first be
+expanded during generation of the autoloads.  @xref{Autoload}.
+
 @item (interactive-only @var{value})
 Set the function's @code{interactive-only} property to @var{value}.
 @xref{The interactive-only property}.
diff --git a/doc/lispref/loading.texi b/doc/lispref/loading.texi
index 750f6e76eff..038aae74caa 100644
--- a/doc/lispref/loading.texi
+++ b/doc/lispref/loading.texi
@@ -708,10 +708,47 @@ Autoload
 various help functions (@pxref{Help Functions}) display it.
 
   If you write a function definition with an unusual macro that is not
-one of the known and recognized function definition methods, use of an
-ordinary magic autoload comment would copy the whole definition into
-@code{loaddefs.el}.  That is not desirable.  You can put the desired
-@code{autoload} call into @code{loaddefs.el} instead by writing this:
+one of the known and recognized function definition methods, using an
+ordinary magic autoload comment with a call to the macro would require
+autoloading the macro definition itself to work.  Doing so copies the
+whole macro definition into the autoload file.  If that is not desired,
+you can use the special declare form @code{(autoload-macro expand)} in
+your macro definition (@pxref{Declare Form}), instead of autoloading it.
+This indicates to the autoload system that calls to your macro following
+@samp{;;;###autoload} should be expanded, similar to how the special
+functions listed above are handled.  For example, a macro which wraps
+@code{define-minor-mode} can request expansion, so that proper
+@code{autoload} calls for the function it defines are generated.
+
+@cindex @code{:autoload-end}
+The keyword symbol @code{:autoload-end} can be used in the expansion of
+a macro to prevent including unwanted forms in the autoload output.  Its
+presence causes any further elements within the form where it appears to
+be silently skipped.  For example, if during autoload generation, a
+macro's expansion includes:
+
+@example
+(progn
+  (put my-mode 'visible-prop t)
+  :autoload-end
+  (put my-mode 'hidden-prop nil))}
+@end example
+
+the final form (@code{(put my-mode 'hidden-prop nil)}) will not be
+copied into the autoload file.
+
+Note that, if a symbol in the @code{car} of an autoloaded form is found
+to be undefined during autoload generation, the file in which the
+associated @samp{;;;###autoload} appears will itself be loaded, to give
+the macro a chance to be defined.  Packages which use special loading
+mechanisms, including loading their own @samp{foo-autoload.el} file,
+should therefore gracefully handle load errors, since these files may
+not yet exist during autoload generation.  This can be done, e.g., by
+setting the @var{no-error} argument of @code{require} non-@code{nil}
+(@pxref{Named Features})).
+
+Alternatively, instead of expansion, you can put the desired
+@code{autoload} call into @file{loaddefs.el} by writing this:
 
 @example
 ;;;###autoload (autoload 'foo "myfile")
diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 366423904db..48562b8a36b 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -271,6 +271,12 @@ 'byte-run--set-debug
 	    (list 'put (list 'quote name)
 		  ''edebug-form-spec (list 'quote spec)))))
 
+(defalias 'byte-run--set-autoload-macro
+  #'(lambda (name _args spec)
+      (list 'function-put (list 'quote name)
+	    ''autoload-macro (list 'quote spec)))
+  "Handle autoload-macro declarations")
+
 (defalias 'byte-run--set-no-font-lock-keyword
   #'(lambda (name _args val)
       (list 'function-put (list 'quote name)
@@ -350,8 +356,13 @@ macro-declarations-alist
   (cons
    (list 'debug #'byte-run--set-debug)
    (cons
-    (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
-    defun-declarations-alist))
+    ;; macros can declare (autoload-macro expand) to request expansion
+    ;; during autoload generation of forms calling them.  See
+    ;; `loaddefs-generate--make-autoload'.
+    (list 'autoload-macro #'byte-run--set-autoload-macro)
+    (cons
+     (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+     defun-declarations-alist)))
   "List associating properties of macros to their macro expansion.
 Each element of the list takes the form (PROP FUN) where FUN is a function.
 For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
@@ -397,6 +408,8 @@ 'defmacro
            (if declarations
 	       (cons 'prog1 (cons def (car declarations)))
 	     def))))))
+;; Expand to defalias and related forms on autoload gen
+(function-put 'defmacro 'autoload-macro 'expand) ; Since we cannot `declare' it
 
 ;; Now that we defined defmacro we can use it!
 (defmacro defun (name arglist &rest body)
@@ -409,7 +422,9 @@ defun
 The return value is undefined.
 
 \(fn NAME ARGLIST [DOCSTRING] [DECL] [INTERACTIVE] BODY...)"
-  (declare (doc-string 3) (indent 2))
+  (declare (doc-string 3) (indent 2)
+           ;; Expand to defalias on autoload gen
+           (autoload-macro expand))
   (or name (error "Cannot define '%s' as a function" name))
   (if (null
        (and (listp arglist)
diff --git a/lisp/emacs-lisp/cl-generic.el b/lisp/emacs-lisp/cl-generic.el
index 3aa26fba3c3..b5da24be2fe 100644
--- a/lisp/emacs-lisp/cl-generic.el
+++ b/lisp/emacs-lisp/cl-generic.el
@@ -258,7 +258,9 @@ cl-defgeneric
                                cl--generic-edebug-make-name in:method]
                               lambda-doc
                               def-body)]]
-             def-body)))
+             def-body))
+           ;; Expand to defun and related forms on autoload definition
+           (autoload-macro expand))
   (let* ((doc (if (stringp (car-safe options-and-methods))
                   (pop options-and-methods)))
          (declarations nil)
diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 2a0a9e5c6de..617991308b6 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -398,7 +398,9 @@ cl-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           ;; expand to function definition on autoload gen
+           (autoload-macro expand))
   `(defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
 ;;;###autoload
@@ -416,7 +418,9 @@ cl-iter-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           ;; expand (eventually) to function definition on autoload gen
+           (autoload-macro expand))
   (require 'generator)
   `(iter-defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
@@ -475,7 +479,8 @@ cl-defmacro
   (declare (debug
             (&define name cl-macro-list cl-declarations-or-string def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))   ; expand to defmacro on autoload gen
   `(defmacro ,name ,@(cl--transform-lambda (cons args body) name)))
 
 (def-edebug-elem-spec 'cl-lambda-expr
@@ -3011,7 +3016,9 @@ cl-defstruct
                         sexp])]
              [&optional stringp]
              ;; All the above is for the following def-form.
-             &rest &or symbolp (symbolp &optional def-form &rest sexp))))
+             &rest &or symbolp (symbolp &optional def-form &rest sexp)))
+           ;; expand to function definitions and related forms on autoload gen
+           (autoload-macro expand))
   (let* ((name (if (consp struct) (car struct) struct))
 	 (warning nil)
 	 (opts (cdr-safe struct))
diff --git a/lisp/emacs-lisp/easy-mmode.el b/lisp/emacs-lisp/easy-mmode.el
index f53db48f0b7..ff96a7afac6 100644
--- a/lisp/emacs-lisp/easy-mmode.el
+++ b/lisp/emacs-lisp/easy-mmode.el
@@ -222,10 +222,12 @@ define-minor-mode
            (indent defun)
            (debug (&define name string-or-null-p
 			   [&optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp]
+			              &optional [&not keywordp] sexp
+			              &optional [&not keywordp] sexp]
 			   [&rest [keywordp sexp]]
-			   def-body)))
+			   def-body))
+           ;; expand to the command definition on autoload gen
+           (autoload-macro expand))
 
   (let* ((last-message (make-symbol "last-message"))
          (mode-name (symbol-name mode))
@@ -487,7 +489,9 @@ define-globalized-minor-mode
 on if the hook has explicitly disabled it.
 
 \(fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)"
-  (declare (doc-string 2) (indent defun))
+  (declare (doc-string 2) (indent defun)
+           ;; expand to the minor-mode definition on autoload gen
+           (autoload-macro expand))
   (let* ((global-mode-name (symbol-name global-mode))
 	 (mode-name (symbol-name mode))
 	 (pretty-name (easy-mmode-pretty-mode-name mode))
diff --git a/lisp/emacs-lisp/generator.el b/lisp/emacs-lisp/generator.el
index 344e11e245e..c01e559fd5c 100644
--- a/lisp/emacs-lisp/generator.el
+++ b/lisp/emacs-lisp/generator.el
@@ -675,7 +675,8 @@ iter-defun
 of values.  Callers can retrieve each value using `iter-next'."
   (declare (indent defun)
            (debug (&define name lambda-list lambda-doc &rest sexp))
-           (doc-string 3))
+           (doc-string 3)
+           (autoload-macro expand)) ; expand to the defun on autoload gen
   (cl-assert lexical-binding)
   (let* ((parsed-body (macroexp-parse-body body))
          (declarations (car parsed-body))
diff --git a/lisp/emacs-lisp/inline.el b/lisp/emacs-lisp/inline.el
index c015e2b57d0..027c24c465b 100644
--- a/lisp/emacs-lisp/inline.el
+++ b/lisp/emacs-lisp/inline.el
@@ -135,7 +135,8 @@ define-inline
 This is like `defmacro', but has several advantages.
 See Info node `(elisp)Defining Functions' for more details."
   ;; FIXME: How can this work with CL arglists?
-  (declare (indent defun) (debug defun) (doc-string 3))
+  (declare (indent defun) (debug defun) (doc-string 3)
+           (autoload-macro expand))     ; expand to the defun on autoload gen
   (let ((doc (if (stringp (car-safe body)) (list (pop body))))
         (declares (if (eq (car-safe (car-safe body)) 'declare) (pop body)))
         (cm-name (intern (format "%s--inliner" name)))
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..6a7bbce3908 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -142,12 +142,32 @@ loaddefs-generate--shorten-autoload
              3)
         form))
 
+;; The following macros are known to define functions, and are treated
+;; specially when encountered during autoload generation, translating
+;; calls to them into appropriate (autoload function ...) forms.
+;; TODO: some macros can be removed from this list once they
+;;  (declare ((autoload-macro expand) ...)
+(defconst loaddefs--defining-macros
+  '(define-skeleton define-derived-mode
+     define-compilation-mode define-generic-mode
+     easy-mmode-define-global-mode define-global-minor-mode
+     define-globalized-minor-mode
+     easy-mmode-define-minor-mode define-minor-mode
+     cl-defun defun* cl-defmacro defmacro*
+     define-overloadable-function
+     transient-define-prefix transient-define-suffix
+     transient-define-infix transient-define-argument))
+
+(defvar loaddefs--load-error-files nil)
 (defun loaddefs-generate--make-autoload (form file &optional expansion)
   "Turn FORM into an autoload or defvar for source file FILE.
 Returns nil if FORM is not a special autoload form (i.e. a function definition
 or macro definition or a defcustom).
 If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently.
+
+Note that macros can request expansion by including `(autoload-macro
+expand)' among their `declare' forms."
   (let ((car (car-safe form)) expand)
     (cond
      ((and expansion (eq car 'defalias))
@@ -191,35 +211,40 @@ loaddefs-generate--make-autoload
           (setq form (copy-sequence form))
           (setcdr (memq :autoload-end form) nil))
         (let ((exps (delq nil (mapcar (lambda (form)
-                                        (loaddefs-generate--make-autoload
-                                         form file expansion))
+                                        (unless (eq form :autoload-end)
+                                          (loaddefs-generate--make-autoload
+                                           form file expansion)))
                                       (cdr form)))))
           (when exps (cons 'progn exps)))))
 
-     ;; For complex cases, try again on the macro-expansion.
-     ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
-                       define-globalized-minor-mode defun defmacro
-		       easy-mmode-define-minor-mode define-minor-mode
-                       define-inline cl-defun cl-defmacro cl-defgeneric
-                       cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
-           (macrop car)
-	   (setq expand (let ((load-true-file-name file)
-                              (load-file-name file))
-                          (macroexpand form)))
-	   (memq (car expand) '(progn prog1 defalias)))
+     ;; For macros which request it, try again on their expansion.
+     ((progn
+        ;; If the car is an unknown symbol, we load the file first to
+        ;; give packages a chance to define their macros.
+        (unless (or (not (symbolp car)) (fboundp car)
+                    ;; Special cases handled below
+                    (memq car loaddefs--defining-macros)
+                    (memq car '(defclass defcustom deftheme defgroup nil))
+                    (assoc file load-history)
+                    (member file loaddefs--load-error-files))
+          (let ((load-path (cons (file-name-directory file) load-path)))
+            (message "loaddefs-gen: loading file %s (for %s)" file car)
+            (condition-case e (load file)
+              (error
+               (push file loaddefs--load-error-files) ; do not attempt again
+               (warn "loaddefs-gen: load error\n\t%s" e)))))
+        (and (macrop car)
+	     (eq 'expand (function-get car 'autoload-macro))
+	     (setq expand (let ((load-true-file-name file)
+				(load-file-name file))
+			    (macroexpand form)))
+	     (not (eq car (car expand)))))
       ;; Recurse on the expansion.
       (loaddefs-generate--make-autoload expand file 'expansion))
 
-     ;; For special function-like operators, use the `autoload' function.
-     ((memq car '(define-skeleton define-derived-mode
-                   define-compilation-mode define-generic-mode
-		   easy-mmode-define-global-mode define-global-minor-mode
-		   define-globalized-minor-mode
-		   easy-mmode-define-minor-mode define-minor-mode
-		   cl-defun defun* cl-defmacro defmacro*
-                   define-overloadable-function
-                   transient-define-prefix transient-define-suffix
-                   transient-define-infix transient-define-argument))
+     ;; For known special macros which define functions, use `autoload'
+     ;; directly.
+     ((memq car loaddefs--defining-macros)
       (let* ((macrop (memq car '(defmacro cl-defmacro defmacro*)))
 	     (name (nth 1 form))
 	     (args (pcase car
diff --git a/lisp/emacs-lisp/pcase.el b/lisp/emacs-lisp/pcase.el
index 73554fd66fd..5e85ab4f075 100644
--- a/lisp/emacs-lisp/pcase.el
+++ b/lisp/emacs-lisp/pcase.el
@@ -537,7 +537,9 @@ pcase-defmacro
 By convention, DOC should use \"EXPVAL\" to stand
 for the result of evaluating EXP (first arg to `pcase').
 \n(fn NAME ARGS [DOC] &rest BODY...)"
-  (declare (indent 2) (debug defun) (doc-string 3))
+  (declare (indent 2) (debug defun) (doc-string 3)
+           ;; Expand to defun and related forms on autoload gen
+           (autoload-macro expand))
   ;; Add the function via `fsym', so that an autoload cookie placed
   ;; on a pcase-defmacro will cause the macro to be loaded on demand.
   (let ((fsym (intern (format "%s--pcase-macroexpander" name)))