Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros

From: "J.D. Smith" <[email-masked]>
Date: Sat, 26 Jul 2025 21:46:46 -0400

Stefan Monnier <[email-masked]> writes:

Thanks for your feedback.

>>> I went with (declare ((autoload . macro-expand))) on the notion that
>>> there might be other autoload-related declarations in the future,
>>> but open to suggestion.
>
> [ I don't have an opinion on the naming here but I don't understand your
>   argument: wouldn't a declaration name of `autoload` be *more* likely
>   to conflict with another future autoload-related declaration?  ]

I suppose I had imagined multiple flags, à la:

  (autoload (macro-expand something-else))

but maybe that's frowned on.  In which case we should just go with
`(autoload-macro . expand)'.

> One way to look at it is that the crux of the matter is that we use
> `;;;###autoload` for 2 different purpose:
>
> A. To copy the code into the loaddefs file.
> B. To arrange for the file to be autoloaded when the corresponding thingy
>    is used.
>
> (B) is the one that deserves the name "autoload" while (A) might be
> better named "preload".

That's a sensible way of looking at it.

> The distinction between the two is not that clear when you consider that
> a `;;;###autoload` cookie placed on a `defun` may arrange to autoload
> that file when calling that function but it may *also* preload some
> `put`s extracted from the `declare`.
>
> Maybe a "general" solution would be to introduce a `;;;###preload`.
> Not sure it's worth the trouble, tho.

Could be considered radical.  I guess the contract would be that all
`;;;###preload's would happen before all `;;;autoload's?  I can imagine
people confusing these two (similar to use-package's :init vs. :config).

> Also, I'm not sure how "general" this would be, since the same problem
> will tend to happen with any package which `;;;###autoload` a call to
> a macro that's not predefined if that package uses its own
> `FOO-loaddefs.el`, even if that macro is defined in some other library.

It would be general if it were universally adopted, and autoload
scanning proceeded across all files, first generating "stub" loaddefs
for preloads, then, in a subsequent pass, doing the autoloads.

The main issue as I see it is we are now (for the first time) possibly
/loading/ files, vs. just scanning them passively.  I think the
situation where this distinction matters is quite rare.  Note that the
only file(s) where a load are attempted now are for TRAMP.

> So, the general problem is: your patch will sometimes `eval-buffer` for
> a buffer which doesn't expect to be evaluated before its package's
> `FOO-loaddefs.el` has been created.   I suspect that in most cases this
> can be fixed by replacing
>
>     (require 'foo-loaddefs)
> by
>     (require 'foo-loaddefs nil t)
>
> It's not ideal, but I think it's acceptable.

What about big packages (like tramp) that maintain and load their own
autoloads, and would like to request macro expansion for some of their
package-defined macros, and then also /use/ that macro in autoloads?
E.g. imagine if `tramp--with-startup' wanted to be expanded instead of
included as-is ("preloaded").  That's a pretty specific case, obviously.

> BTW, in the specific case of Tramp, I see:
>
>     (defmacro tramp--with-startup (&rest body)
>       "Schedule BODY to be executed at the end of tramp.el."
>       `(add-hook 'tramp--startup-hook (lambda nil ,@body)))
>
> so I think `tramp--with-startup` *is* one of the macros that would
> benefit from your new `(autoload macro-expand)` declaration.

Ah right, my imagining above was not so misguided then.  Maybe the tramp
maintainer will want to expand that simple macro away now.  I guess
package authors have to decide:

- Do you want to include ("preload") your full macro and any autoloaded
  uses of it (as is) in *-loaddefs.el?  You should be sure to require
  your package's custom autoload file with suppressed errors, in case it
  doesn't yet exist during build.

- Would you prefer your package's macro is expanded at autoload
  generation time instead?  You should require it "for real", and
  declare it (autoload-macro expand).

- You can't get both behaviors.

>> [1] Note that the order of declare entries already matters, 
>
> Really?  Sounds like a bug.  Are you sure?

You're right, it was a transitory bug.  I've moved them all back to the
end (since they are most recently added that seems to make sense).

>> (defconst loaddefs--function-like-operators
>
> "operators" doesn't sound quite right.
> Maybe "loaddefs--defining-macros"?

Yeah I agree, I was picking up the comment below that:

     ;; For special function-like operators, use the `autoload'
     ;; function.

But I like your version better (and have changed the comment).

>> +Note that macros can request expansion by including `(autoload
>> macro-expand)' +among their `declare' forms."
>
> Nitpick: I'd start "Note..." on its own line.  We're not trying to
> justify-fill the text.  🙂

Done (even an extra blank to call it out). 

>> +        ;; If the car is entirely unknown, we load the file first to
>> +        ;; give packages a chance to define their macros.
>> +        (unless (or (null car) (macrop car) (functionp car) (special-form-p car)
>
> I'd test (and (symbolp car) (not (fboundp car)))

For whatever reason, I get lots of nil cars.  So how about:

          (unless (or (not (symbolp car)) (fboundp car) (null car)
                      ... ;; try to load the file

Is

(eq (fboundp x) (or (macrop x) (functionp x) (special-form-p x))) ∀x∈ (symbolp x)?

>> +                    (memq car '(defclass defcustom deftheme defgroup))
> This deserves a comment in the code.

Done.

> I'd go with `assoc`.

Thanks.

>> +        (and (macrop car)
>> +	     (eq 'macro-expand ; a macro can declare (autoload macro-expand)
>> +		 (or (get car 'autoload)
>> +		     (get (car-safe (last (function-alias-p car))) 'autoload)))
>
> I'd have expected `function-get` here.

I was honestly vague on the difference.  Seems like `function-get'
should maybe chase aliases for you...

> If we swap this case with the preceding one we can remove the
>
>     (memq car loaddefs--function-like-operators)
>
> from the `unless` of that case.
> [ And it probably results in a diff that's harder to read, so thanks for
>   not doing it this time.  🙂 ]

Good idea. I only recently decided this was necessary.  Though it isn't
really needed anymore with this change, I think keeping a defconst is a
good idea.

>> @@ -257,8 +278,8 @@ loaddefs-generate--make-autoload
>>                        t)
>>                   (and (eq (car-safe (car body)) 'interactive)
>>                        ;; List of modes or just t.
>> -                      (or (if (nthcdr 1 (car body))
>> -                              (list 'quote (nthcdr 1 (car body)))
>> +                      (or (if (nthcdr 2 (car body))
>> +                              (list 'quote (nthcdr 2 (car body)))
>>                              t))))
>>              ,(if macrop ''macro nil)))))
>
> I think you can just push this to `master` as  separate patch, thank you.

I've attached a separate patch here (I don't have commit privileges).

>> @@ -19835,7 +19836,7 @@
>>  loaddefs.el output file, and the rest are the directories to
>>  use.")
>>   (load "theme-loaddefs.el" t)
>> -(register-definition-prefixes "loaddefs-gen" '("autoload-" "generated-autoload-" "loaddefs-generate--" "no-update-autoloads"))
>> +(register-definition-prefixes "loaddefs-gen" '("autoload-" "generated-autoload-" "loaddefs-" "my/tmp-lddg-list2" "no-update-autoloads"))
>
> Maybe better use `autoload-` rather than `loaddefs-` as prefix for the
> new definitions.
> [ I don't see `my/tmp-lddg-list2` in the patch you sent, so I assume it
>   was something temporary.  ]

This last file was an inadvertent inclusion.

Please take a look at the attached patches, and give a try with your own
user macro declaring autoload-expand if you can.  If this all looks good
I can add to the docs and NEWS.

diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index b43947d5b2b..20394442167 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -254,8 +254,8 @@ loaddefs-generate--make-autoload
                       t)
                  (and (eq (car-safe (car body)) 'interactive)
                       ;; List of modes or just t.
-                      (or (if (nthcdr 1 (car body))
-                              (list 'quote (nthcdr 1 (car body)))
+                      (or (if (nthcdr 2 (car body))
+                              (list 'quote (nthcdr 2 (car body)))
                             t))))
             ,(if macrop ''macro nil)))))
 
diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 366423904db..3e521e18567 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -271,6 +271,11 @@ 'byte-run--set-debug
 	    (list 'put (list 'quote name)
 		  ''edebug-form-spec (list 'quote spec)))))
 
+(defalias 'byte-run--set-autoload-macro
+  #'(lambda (name _args spec)
+      (list 'function-put (list 'quote name)
+	    ''autoload-macro (list 'quote spec))))
+
 (defalias 'byte-run--set-no-font-lock-keyword
   #'(lambda (name _args val)
       (list 'function-put (list 'quote name)
@@ -350,8 +355,10 @@ macro-declarations-alist
   (cons
    (list 'debug #'byte-run--set-debug)
    (cons
-    (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
-    defun-declarations-alist))
+    (list 'autoload-macro #'byte-run--set-autoload-macro)
+    (cons
+     (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+     defun-declarations-alist)))
   "List associating properties of macros to their macro expansion.
 Each element of the list takes the form (PROP FUN) where FUN is a function.
 For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
@@ -397,6 +404,7 @@ 'defmacro
            (if declarations
 	       (cons 'prog1 (cons def (car declarations)))
 	     def))))))
+(function-put 'defmacro 'autoload-macro 'expand) ; Since we cannot `declare' it
 
 ;; Now that we defined defmacro we can use it!
 (defmacro defun (name arglist &rest body)
@@ -409,7 +417,7 @@ defun
 The return value is undefined.
 
 \(fn NAME ARGLIST [DOCSTRING] [DECL] [INTERACTIVE] BODY...)"
-  (declare (doc-string 3) (indent 2))
+  (declare (doc-string 3) (indent 2) (autoload-macro expand))
   (or name (error "Cannot define '%s' as a function" name))
   (if (null
        (and (listp arglist)
diff --git a/lisp/emacs-lisp/cl-generic.el b/lisp/emacs-lisp/cl-generic.el
index 3aa26fba3c3..86f71234657 100644
--- a/lisp/emacs-lisp/cl-generic.el
+++ b/lisp/emacs-lisp/cl-generic.el
@@ -241,8 +241,7 @@ cl-defgeneric
 DEFAULT-BODY, if present, is used as the body of a default method.
 
 \(fn NAME ARGS [DOC-STRING] [OPTIONS-AND-METHODS...] &rest DEFAULT-BODY)"
-  (declare (indent 2) (doc-string 3)
-           (debug
+  (declare (indent 2) (doc-string 3) (debug
             (&define
              &interpose
              [&name sexp] ;Allow (setf ...) additionally to symbols.
@@ -258,7 +257,8 @@ cl-defgeneric
                                cl--generic-edebug-make-name in:method]
                               lambda-doc
                               def-body)]]
-             def-body)))
+             def-body))
+           (autoload-macro expand))
   (let* ((doc (if (stringp (car-safe options-and-methods))
                   (pop options-and-methods)))
          (declarations nil)
diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 2a0a9e5c6de..2b008f5c708 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -398,7 +398,8 @@ cl-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))
   `(defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
 ;;;###autoload
@@ -408,7 +409,8 @@ cl-iter-defun
 and BODY is implicitly surrounded by (cl-block NAME ...).
 
 \(fn NAME ARGLIST [DOCSTRING] BODY...)"
-  (declare (debug
+  (declare (indent 2)
+           (debug
             ;; Same as iter-defun but use cl-lambda-list.
             (&define [&name sexp]   ;Allow (setf ...) additionally to symbols.
                      cl-lambda-list
@@ -416,7 +418,7 @@ cl-iter-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (autoload-macro expand))
   (require 'generator)
   `(iter-defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
@@ -475,7 +477,8 @@ cl-defmacro
   (declare (debug
             (&define name cl-macro-list cl-declarations-or-string def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))
   `(defmacro ,name ,@(cl--transform-lambda (cons args body) name)))
 
 (def-edebug-elem-spec 'cl-lambda-expr
@@ -3001,8 +3004,7 @@ cl-defstruct
 \\[describe-symbol] or \\[cl-describe-type].
 
 \(fn NAME &optional DOCSTRING &rest SLOTS)"
-  (declare (doc-string 2) (indent 1)
-           (debug
+  (declare (debug
             (&define                    ;Makes top-level form not be wrapped.
              [&or symbolp
                   (gate ;; FIXME: Why?
@@ -3011,7 +3013,8 @@ cl-defstruct
                         sexp])]
              [&optional stringp]
              ;; All the above is for the following def-form.
-             &rest &or symbolp (symbolp &optional def-form &rest sexp))))
+             &rest &or symbolp (symbolp &optional def-form &rest sexp))) (doc-string 2) (indent 1)
+           (autoload-macro expand))
   (let* ((name (if (consp struct) (car struct) struct))
 	 (warning nil)
 	 (opts (cdr-safe struct))
diff --git a/lisp/emacs-lisp/easy-mmode.el b/lisp/emacs-lisp/easy-mmode.el
index f53db48f0b7..cfd71a69b04 100644
--- a/lisp/emacs-lisp/easy-mmode.el
+++ b/lisp/emacs-lisp/easy-mmode.el
@@ -218,14 +218,15 @@ define-minor-mode
 INIT-VALUE LIGHTER KEYMAP.
 
 \(fn MODE DOC [KEYWORD VAL ... &rest BODY])"
-  (declare (doc-string 2)
-           (indent defun)
-           (debug (&define name string-or-null-p
+  (declare (debug (&define name string-or-null-p
 			   [&optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp]
+			              &optional [&not keywordp] sexp
+			              &optional [&not keywordp] sexp]
 			   [&rest [keywordp sexp]]
-			   def-body)))
+			   def-body))
+           (doc-string 2)
+           (indent defun)
+           (autoload-macro expand))
 
   (let* ((last-message (make-symbol "last-message"))
          (mode-name (symbol-name mode))
@@ -487,7 +488,7 @@ define-globalized-minor-mode
 on if the hook has explicitly disabled it.
 
 \(fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)"
-  (declare (doc-string 2) (indent defun))
+  (declare (doc-string 2) (indent defun) (autoload-macro expand))
   (let* ((global-mode-name (symbol-name global-mode))
 	 (mode-name (symbol-name mode))
 	 (pretty-name (easy-mmode-pretty-mode-name mode))
diff --git a/lisp/emacs-lisp/generator.el b/lisp/emacs-lisp/generator.el
index 344e11e245e..59784ff51e8 100644
--- a/lisp/emacs-lisp/generator.el
+++ b/lisp/emacs-lisp/generator.el
@@ -675,7 +675,8 @@ iter-defun
 of values.  Callers can retrieve each value using `iter-next'."
   (declare (indent defun)
            (debug (&define name lambda-list lambda-doc &rest sexp))
-           (doc-string 3))
+           (doc-string 3)
+           (autoload-macro expand))
   (cl-assert lexical-binding)
   (let* ((parsed-body (macroexp-parse-body body))
          (declarations (car parsed-body))
diff --git a/lisp/emacs-lisp/inline.el b/lisp/emacs-lisp/inline.el
index c015e2b57d0..5acd6f17ee2 100644
--- a/lisp/emacs-lisp/inline.el
+++ b/lisp/emacs-lisp/inline.el
@@ -135,7 +135,7 @@ define-inline
 This is like `defmacro', but has several advantages.
 See Info node `(elisp)Defining Functions' for more details."
   ;; FIXME: How can this work with CL arglists?
-  (declare (indent defun) (debug defun) (doc-string 3))
+  (declare (indent defun) (debug defun) (doc-string 3) (autoload-macro expand))
   (let ((doc (if (stringp (car-safe body)) (list (pop body))))
         (declares (if (eq (car-safe (car-safe body)) 'declare) (pop body)))
         (cm-name (intern (format "%s--inliner" name)))
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..b43947d5b2b 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -142,12 +142,29 @@ loaddefs-generate--shorten-autoload
              3)
         form))
 
+;; TODO: some macros can be removed from this list once they
+;;  (declare ((autoload-macro expand) ...)
+(defconst loaddefs--defining-macros
+  '(define-skeleton define-derived-mode
+     define-compilation-mode define-generic-mode
+     easy-mmode-define-global-mode define-global-minor-mode
+     define-globalized-minor-mode
+     easy-mmode-define-minor-mode define-minor-mode
+     cl-defun defun* cl-defmacro defmacro*
+     define-overloadable-function
+     transient-define-prefix transient-define-suffix
+     transient-define-infix transient-define-argument))
+
+(defvar loaddefs--load-error-files nil)
 (defun loaddefs-generate--make-autoload (form file &optional expansion)
   "Turn FORM into an autoload or defvar for source file FILE.
 Returns nil if FORM is not a special autoload form (i.e. a function definition
 or macro definition or a defcustom).
 If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently.
+
+Note that macros can request expansion by including `(autoload-macro
+expand)' among their `declare' forms."
   (let ((car (car-safe form)) expand)
     (cond
      ((and expansion (eq car 'defalias))
@@ -196,30 +213,10 @@ loaddefs-generate--make-autoload
                                       (cdr form)))))
           (when exps (cons 'progn exps)))))
 
-     ;; For complex cases, try again on the macro-expansion.
-     ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
-                       define-globalized-minor-mode defun defmacro
-		       easy-mmode-define-minor-mode define-minor-mode
-                       define-inline cl-defun cl-defmacro cl-defgeneric
-                       cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
-           (macrop car)
-	   (setq expand (let ((load-true-file-name file)
-                              (load-file-name file))
-                          (macroexpand form)))
-	   (memq (car expand) '(progn prog1 defalias)))
-      ;; Recurse on the expansion.
-      (loaddefs-generate--make-autoload expand file 'expansion))
-
-     ;; For special function-like operators, use the `autoload' function.
-     ((memq car '(define-skeleton define-derived-mode
-                   define-compilation-mode define-generic-mode
-		   easy-mmode-define-global-mode define-global-minor-mode
-		   define-globalized-minor-mode
-		   easy-mmode-define-minor-mode define-minor-mode
-		   cl-defun defun* cl-defmacro defmacro*
-                   define-overloadable-function
-                   transient-define-prefix transient-define-suffix
-                   transient-define-infix transient-define-argument))
+     ;; For macros which request it, try again on their expansion.
+     ;; For known special macros which define functions, use `autoload'
+     ;; directly.
+     ((memq car loaddefs--defining-macros)
       (let* ((macrop (memq car '(defmacro cl-defmacro defmacro*)))
 	     (name (nth 1 form))
 	     (args (pcase car
@@ -262,6 +259,32 @@ loaddefs-generate--make-autoload
                             t))))
             ,(if macrop ''macro nil)))))
 
+     ((progn
+        ;; If the car is an unknown symbol, we load the file first to
+        ;; give packages a chance to define their macros.
+        (unless (or (not (symbolp car)) (fboundp car) (null car)
+                    ;; Special cases handled below
+                    (memq car '(defclass defcustom deftheme defgroup))
+                    (assoc file load-history)
+                    (member file loaddefs--load-error-files))
+          (let ((load-path (cons (file-name-directory file) load-path)))
+            (message "loaddefs-gen: loading file %s (for %s)" file car)
+            (condition-case e (load file)
+              (error
+               (push file loaddefs--load-error-files) ; do not attempt again
+               (warn "loaddefs-gen: load error\n\t%s" e)))))
+        (and (macrop car)
+	     (eq 'expand ; a macro can declare (autoload-macro expand)
+		 (or (function-get car 'autoload-macro)
+		     (function-get (car-safe (last (function-alias-p car)))
+                          'autoload-macro)))
+	     (setq expand (let ((load-true-file-name file)
+				(load-file-name file))
+			    (macroexpand form)))
+	     (not (eq car (car expand)))))
+      ;; Recurse on the expansion.
+      (loaddefs-generate--make-autoload expand file 'expansion))
+
      ;; For defclass forms, use `eieio-defclass-autoload'.
      ((eq car 'defclass)
       (let ((name (nth 1 form))
diff --git a/lisp/emacs-lisp/pcase.el b/lisp/emacs-lisp/pcase.el
index 73554fd66fd..ab8edf1f569 100644
--- a/lisp/emacs-lisp/pcase.el
+++ b/lisp/emacs-lisp/pcase.el
@@ -537,7 +537,7 @@ pcase-defmacro
 By convention, DOC should use \"EXPVAL\" to stand
 for the result of evaluating EXP (first arg to `pcase').
 \n(fn NAME ARGS [DOC] &rest BODY...)"
-  (declare (indent 2) (debug defun) (doc-string 3))
+  (declare (indent 2) (debug defun) (doc-string 3) (autoload-macro expand))
   ;; Add the function via `fsym', so that an autoload cookie placed
   ;; on a pcase-defmacro will cause the macro to be loaded on demand.
   (let ((fsym (intern (format "%s--pcase-macroexpander" name)))