Subject: bug#78995: [PATCH] ;;;autoload-expand for special macros

From: "J.D. Smith" <[email-masked]>
Date: Thu, 24 Jul 2025 10:09:07 -0400

"J.D. Smith" <[email-masked]> writes:

> Here is an update on progress.  The attached is working reasonably well,
> reproducing lisp/loaddefs.el with a few changes (see attached).  I went
> with (declare ((autoload . macro-expand))) on the notion that there
> might be other autoload-related declarations in the future, but open to
> suggestion.

Please have a look at this latest patch, which I believe solves the
mentioned problems.

Changes:

1. I implemented the list 2 bug fix for autoload's INTERACTIVE argument
   described above, correctly keeping interactive arg specs out of the
   loaddefs files.

2. I moved the autoload macro-expand declare up front everywhere it's
   used[1], to ensure the macro-expand request will be included in
   *-loaddefs.el files.  This is important for packages which manage
   their own *-loaddefs.el files (e.g. `cl-macs').
   
3. I have kept list 2 around[2], and count presence on this list, or the
   list of 4 specially-handled macros (defgroup, defcustom, deftheme,
   defclass) as "we know about this car".  No file load will be
   attempted for such special macros, so only proper "user-defined
   UNKNOWN macros" will result in a load attempt.

With these, the loaddefs.el file changes only modestly (see attached),
and (almost) no files have load attempted.

About that almost.  There is one small order-of-operation issue yet to
be solved, for packages which do clever things.  Take any of the
`net/tramp-*.el` files.  They all require `tramp-compat`, which requires
`tramp-loaddefs', which is itself an autoload file the build process is
generating.  For example, tramp's autoload file eventually includes the
macro `tramp--with-startup'.  When one of tramp's autoloads use an
unknown macro like this, there will be an attempt to load the file,
implicitly requiring that yet-to-exist `tramp-loaddefs.el' file,
resulting in a (handled) error.  Classic chicken and egg.

    INFO     Scraping 1543 files for loaddefs...70% 
  loaddefs-gen: loading file tramp-adb (for tramp--with-startup)
  Loading /Users/jdsmith/code/emacs/emacs-mac/lisp/net/tramp-adb.el (source)...
  Warning (emacs): loaddefs-gen: load error
  	(file-missing Cannot open load file No such file or directory tramp-loaddefs)
  loaddefs-gen: loading file tramp-androidsu (for tramp--with-startup)
  Loading /Users/jdsmith/code/emacs/emacs-mac/lisp/net/tramp-androidsu.el (source)...
  Warning (emacs): loaddefs-gen: load error
  	(file-missing Cannot open load file No such file or directory tramp-loaddefs)
  ...

In this case these are harmless warnings, since in fact
`tramp--with-startup' is /not/ requesting macro-expand, so failing to
define it doesn't matter.  Subsequent builds do not show this error.
But this could be alarming to people, so we probably need a solution.

[1] Note that the order of declare entries already matters, because some
    of them generate forms which include `:autoload-end', and this
    token, once encountered _anywhere_, puts an absolute stop to
    emitting additional declare-generated forms into loaddefs.
    E.g. `(declare (debug ...))' does this.  This, by the way, is why
    /some/ macros have their (e.g.) `indent' declares passed through to
    the loaddefs files, and others do not.

[2] Some of these macros aren't yet suited for full expansion anyway.
    For example `define-derived-mode' includes a bunch of unwanted forms
    if it gets the expansion treatment.  So for now, leaving the
    shortcut list 2 intact seems to be the lesser evil.

diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 366423904db..4d3cc96db7d 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -271,6 +271,11 @@ 'byte-run--set-debug
 	    (list 'put (list 'quote name)
 		  ''edebug-form-spec (list 'quote spec)))))
 
+(defalias 'byte-run--set-autoload
+  #'(lambda (name _args spec)
+      (list 'function-put (list 'quote name)
+	    ''autoload (list 'quote spec))))
+
 (defalias 'byte-run--set-no-font-lock-keyword
   #'(lambda (name _args val)
       (list 'function-put (list 'quote name)
@@ -350,8 +355,10 @@ macro-declarations-alist
   (cons
    (list 'debug #'byte-run--set-debug)
    (cons
-    (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
-    defun-declarations-alist))
+    (list 'autoload #'byte-run--set-autoload)
+    (cons
+     (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+     defun-declarations-alist)))
   "List associating properties of macros to their macro expansion.
 Each element of the list takes the form (PROP FUN) where FUN is a function.
 For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
@@ -397,6 +404,7 @@ 'defmacro
            (if declarations
 	       (cons 'prog1 (cons def (car declarations)))
 	     def))))))
+(function-put 'defmacro 'autoload 'macro-expand) ; Since we cannot `declare' it
 
 ;; Now that we defined defmacro we can use it!
 (defmacro defun (name arglist &rest body)
@@ -409,7 +417,7 @@ defun
 The return value is undefined.
 
 \(fn NAME ARGLIST [DOCSTRING] [DECL] [INTERACTIVE] BODY...)"
-  (declare (doc-string 3) (indent 2))
+  (declare (autoload macro-expand) (doc-string 3) (indent 2))
   (or name (error "Cannot define '%s' as a function" name))
   (if (null
        (and (listp arglist)
diff --git a/lisp/emacs-lisp/cl-generic.el b/lisp/emacs-lisp/cl-generic.el
index 3aa26fba3c3..8867b83164f 100644
--- a/lisp/emacs-lisp/cl-generic.el
+++ b/lisp/emacs-lisp/cl-generic.el
@@ -241,7 +241,7 @@ cl-defgeneric
 DEFAULT-BODY, if present, is used as the body of a default method.
 
 \(fn NAME ARGS [DOC-STRING] [OPTIONS-AND-METHODS...] &rest DEFAULT-BODY)"
-  (declare (indent 2) (doc-string 3)
+  (declare (autoload macro-expand) (indent 2) (doc-string 3)
            (debug
             (&define
              &interpose
diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 2a0a9e5c6de..fd24424820d 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -390,7 +390,8 @@ cl-defun
 more details.
 
 \(fn NAME ARGLIST [DOCSTRING] BODY...)"
-  (declare (debug
+  (declare (autoload macro-expand)
+           (debug
             ;; Same as defun but use cl-lambda-list.
             (&define [&name symbolp]
                      cl-lambda-list
@@ -408,7 +409,8 @@ cl-iter-defun
 and BODY is implicitly surrounded by (cl-block NAME ...).
 
 \(fn NAME ARGLIST [DOCSTRING] BODY...)"
-  (declare (debug
+  (declare (autoload macro-expand)
+           (debug
             ;; Same as iter-defun but use cl-lambda-list.
             (&define [&name sexp]   ;Allow (setf ...) additionally to symbols.
                      cl-lambda-list
@@ -472,7 +474,8 @@ cl-defmacro
 more details.
 
 \(fn NAME ARGLIST [DOCSTRING] BODY...)"
-  (declare (debug
+  (declare (autoload macro-expand)
+           (debug
             (&define name cl-macro-list cl-declarations-or-string def-body))
            (doc-string 3)
            (indent 2))
@@ -3001,7 +3004,7 @@ cl-defstruct
 \\[describe-symbol] or \\[cl-describe-type].
 
 \(fn NAME &optional DOCSTRING &rest SLOTS)"
-  (declare (doc-string 2) (indent 1)
+  (declare (autoload macro-expand) (doc-string 2) (indent 1)
            (debug
             (&define                    ;Makes top-level form not be wrapped.
              [&or symbolp
diff --git a/lisp/emacs-lisp/easy-mmode.el b/lisp/emacs-lisp/easy-mmode.el
index f53db48f0b7..e618a76ef4e 100644
--- a/lisp/emacs-lisp/easy-mmode.el
+++ b/lisp/emacs-lisp/easy-mmode.el
@@ -218,12 +218,13 @@ define-minor-mode
 INIT-VALUE LIGHTER KEYMAP.
 
 \(fn MODE DOC [KEYWORD VAL ... &rest BODY])"
-  (declare (doc-string 2)
+  (declare (autoload macro-expand)
+           (doc-string 2)
            (indent defun)
            (debug (&define name string-or-null-p
 			   [&optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp]
+			              &optional [&not keywordp] sexp
+			              &optional [&not keywordp] sexp]
 			   [&rest [keywordp sexp]]
 			   def-body)))
 
@@ -487,7 +488,7 @@ define-globalized-minor-mode
 on if the hook has explicitly disabled it.
 
 \(fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)"
-  (declare (doc-string 2) (indent defun))
+  (declare (autoload macro-expand) (doc-string 2) (indent defun))
   (let* ((global-mode-name (symbol-name global-mode))
 	 (mode-name (symbol-name mode))
 	 (pretty-name (easy-mmode-pretty-mode-name mode))
diff --git a/lisp/emacs-lisp/generator.el b/lisp/emacs-lisp/generator.el
index 344e11e245e..ccf4efffdc6 100644
--- a/lisp/emacs-lisp/generator.el
+++ b/lisp/emacs-lisp/generator.el
@@ -673,7 +673,8 @@ iter-defun
 When called as a function, NAME returns an iterator value that
 encapsulates the state of a computation that produces a sequence
 of values.  Callers can retrieve each value using `iter-next'."
-  (declare (indent defun)
+  (declare (autoload macro-expand)
+           (indent defun)
            (debug (&define name lambda-list lambda-doc &rest sexp))
            (doc-string 3))
   (cl-assert lexical-binding)
diff --git a/lisp/emacs-lisp/inline.el b/lisp/emacs-lisp/inline.el
index c015e2b57d0..a062f88946d 100644
--- a/lisp/emacs-lisp/inline.el
+++ b/lisp/emacs-lisp/inline.el
@@ -135,7 +135,7 @@ define-inline
 This is like `defmacro', but has several advantages.
 See Info node `(elisp)Defining Functions' for more details."
   ;; FIXME: How can this work with CL arglists?
-  (declare (indent defun) (debug defun) (doc-string 3))
+  (declare (autoload macro-expand) (indent defun) (debug defun) (doc-string 3))
   (let ((doc (if (stringp (car-safe body)) (list (pop body))))
         (declares (if (eq (car-safe (car-safe body)) 'declare) (pop body)))
         (cm-name (intern (format "%s--inliner" name)))
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..86abdeb24a1 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -142,12 +142,28 @@ loaddefs-generate--shorten-autoload
              3)
         form))
 
+;; TODO: some macros can be removed from this list once they
+;;  (declare ((autoload macro-expand) ...)
+(defconst loaddefs--function-like-operators
+  '(define-skeleton define-derived-mode
+     define-compilation-mode define-generic-mode
+     easy-mmode-define-global-mode define-global-minor-mode
+     define-globalized-minor-mode
+     easy-mmode-define-minor-mode define-minor-mode
+     cl-defun defun* cl-defmacro defmacro*
+     define-overloadable-function
+     transient-define-prefix transient-define-suffix
+     transient-define-infix transient-define-argument))
+
+(defvar loaddefs--load-error-files nil)
 (defun loaddefs-generate--make-autoload (form file &optional expansion)
   "Turn FORM into an autoload or defvar for source file FILE.
 Returns nil if FORM is not a special autoload form (i.e. a function definition
 or macro definition or a defcustom).
 If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently.  Note
+that macros can request expansion by including `(autoload macro-expand)'
+among their `declare' forms."
   (let ((car (car-safe form)) expand)
     (cond
      ((and expansion (eq car 'defalias))
@@ -196,30 +212,35 @@ loaddefs-generate--make-autoload
                                       (cdr form)))))
           (when exps (cons 'progn exps)))))
 
-     ;; For complex cases, try again on the macro-expansion.
-     ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
-                       define-globalized-minor-mode defun defmacro
-		       easy-mmode-define-minor-mode define-minor-mode
-                       define-inline cl-defun cl-defmacro cl-defgeneric
-                       cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
-           (macrop car)
-	   (setq expand (let ((load-true-file-name file)
-                              (load-file-name file))
-                          (macroexpand form)))
-	   (memq (car expand) '(progn prog1 defalias)))
+     ;; For macros which request it, try again on their expansion.
+     ((progn
+        ;; If the car is entirely unknown, we load the file first to
+        ;; give packages a chance to define their macros.
+        (unless (or (null car) (macrop car) (functionp car) (special-form-p car)
+                    (memq car loaddefs--function-like-operators)
+                    (memq car '(defclass defcustom deftheme defgroup))
+                    (alist-get file load-history)
+                    (member file loaddefs--load-error-files))
+          (let ((load-path (cons (file-name-directory file) load-path)))
+            (message "loaddefs-gen: loading file %s (for %s)" file car)
+            (condition-case e (load file)
+              (error
+               (push file loaddefs--load-error-files) ; do not attempt again
+               (warn "loaddefs-gen: load error\n\t%s" e)))))
+        (and (macrop car)
+	     (eq 'macro-expand ; a macro can declare (autoload macro-expand)
+		 (or (get car 'autoload)
+		     (get (car-safe (last (function-alias-p car))) 'autoload)))
+	     (setq expand (let ((load-true-file-name file)
+				(load-file-name file))
+			    (macroexpand form)))
+	     (not (eq car (car expand)))))
       ;; Recurse on the expansion.
       (loaddefs-generate--make-autoload expand file 'expansion))
 
-     ;; For special function-like operators, use the `autoload' function.
-     ((memq car '(define-skeleton define-derived-mode
-                   define-compilation-mode define-generic-mode
-		   easy-mmode-define-global-mode define-global-minor-mode
-		   define-globalized-minor-mode
-		   easy-mmode-define-minor-mode define-minor-mode
-		   cl-defun defun* cl-defmacro defmacro*
-                   define-overloadable-function
-                   transient-define-prefix transient-define-suffix
-                   transient-define-infix transient-define-argument))
+     ;; For special function-like operators, use the `autoload'
+     ;; function.
+     ((memq car loaddefs--function-like-operators)
       (let* ((macrop (memq car '(defmacro cl-defmacro defmacro*)))
 	     (name (nth 1 form))
 	     (args (pcase car
@@ -257,8 +278,8 @@ loaddefs-generate--make-autoload
                       t)
                  (and (eq (car-safe (car body)) 'interactive)
                       ;; List of modes or just t.
-                      (or (if (nthcdr 1 (car body))
-                              (list 'quote (nthcdr 1 (car body)))
+                      (or (if (nthcdr 2 (car body))
+                              (list 'quote (nthcdr 2 (car body)))
                             t))))
             ,(if macrop ''macro nil)))))
 
diff --git a/lisp/emacs-lisp/pcase.el b/lisp/emacs-lisp/pcase.el
index 73554fd66fd..893e3db2b59 100644
--- a/lisp/emacs-lisp/pcase.el
+++ b/lisp/emacs-lisp/pcase.el
@@ -537,7 +537,7 @@ pcase-defmacro
 By convention, DOC should use \"EXPVAL\" to stand
 for the result of evaluating EXP (first arg to `pcase').
 \n(fn NAME ARGS [DOC] &rest BODY...)"
-  (declare (indent 2) (debug defun) (doc-string 3))
+  (declare (autoload macro-expand) (indent 2) (debug defun) (doc-string 3))
   ;; Add the function via `fsym', so that an autoload cookie placed
   ;; on a pcase-defmacro will cause the macro to be loaded on demand.
   (let ((fsym (intern (format "%s--pcase-macroexpander" name)))
--- loaddefs_old.el	2025-07-15 17:56:40
+++ loaddefs.el	2025-07-24 09:26:22
@@ -8420,6 +8420,7 @@
 INIT-VALUE LIGHTER KEYMAP.
 
 (fn MODE DOC [KEYWORD VAL ... &rest BODY])" nil t)
+(function-put 'define-minor-mode 'autoload 'macro-expand)
 (function-put 'define-minor-mode 'doc-string-elt 2)
 (function-put 'define-minor-mode 'lisp-indent-function 'defun)
 (defalias 'define-global-minor-mode #'define-globalized-minor-mode)
@@ -8464,6 +8465,7 @@
 on if the hook has explicitly disabled it.
 
 (fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)" nil t)
+(function-put 'define-globalized-minor-mode 'autoload 'macro-expand)
 (function-put 'define-globalized-minor-mode 'doc-string-elt 2)
 (function-put 'define-globalized-minor-mode 'lisp-indent-function 'defun)
 (autoload 'easy-mmode-define-keymap "easy-mmode" "\
@@ -8813,7 +8815,7 @@
 A call with prefix PREFIX reads the symbol to insert from the minibuffer with
 completion.
 
-(fn PREFIX)" '("P"))
+(fn PREFIX)" t)
 (autoload 'ebrowse-tags-loop-continue "ebrowse" "\
 Repeat last operation on files in tree.
 FIRST-TIME non-nil means this is not a repetition, but the first time.
@@ -10549,7 +10551,7 @@
 and defers to `erc-compute-port', `erc-compute-user', and
 `erc-compute-nick' for those respective parameters.
 
-(fn &key SERVER PORT NICK USER PASSWORD FULL-NAME ID)" '((let ((erc--display-context `((erc-interactive-display . erc) ,@erc--display-context))) (erc-select-read-args))))
+(fn &key SERVER PORT NICK USER PASSWORD FULL-NAME ID)" t)
 (defalias 'erc-select #'erc)
 (autoload 'erc-tls "erc" "\
 Connect to an IRC server over a TLS-encrypted connection.
@@ -10572,7 +10574,7 @@
 node `(erc) Connecting' for a fuller description of the various
 parameters, like ID.
 
-(fn &key SERVER PORT NICK USER PASSWORD FULL-NAME CLIENT-CERTIFICATE ID)" '((let ((erc-default-port erc-default-port-tls) (erc--display-context `((erc-interactive-display . erc-tls) ,@erc--display-context))) (erc-select-read-args))))
+(fn &key SERVER PORT NICK USER PASSWORD FULL-NAME CLIENT-CERTIFICATE ID)" t)
 (autoload 'erc-handle-irc-url "erc" "\
 Use ERC to IRC on HOST:PORT in CHANNEL.
 If ERC is already connected to HOST:PORT, simply /join CHANNEL.
@@ -18809,6 +18811,7 @@
 See Info node `(elisp)Defining Functions' for more details.
 
 (fn NAME ARGS &rest BODY)" nil t)
+(function-put 'define-inline 'autoload 'macro-expand)
 (function-put 'define-inline 'lisp-indent-function 'defun)
 (function-put 'define-inline 'doc-string-elt 3)
 (register-definition-prefixes "inline" '("inline-"))
@@ -19797,8 +19800,6 @@
 ;;; Generated autoloads from emacs-lisp/loaddefs-gen.el
 
 (put 'autoload-compute-prefixes 'safe-local-variable #'booleanp)
-(put 'autoload-expand-macros 'safe-local-variable #'listp)
-(put 'autoload-expand-macros 'safe-local-variable #'listp)
 (put 'generated-autoload-file 'safe-local-variable 'stringp)
 (put 'generated-autoload-load-name 'safe-local-variable 'stringp)
 (autoload 'loaddefs-generate "loaddefs-gen" "\
@@ -19835,7 +19836,7 @@
 loaddefs.el output file, and the rest are the directories to
 use.")
  (load "theme-loaddefs.el" t)
-(register-definition-prefixes "loaddefs-gen" '("autoload-" "generated-autoload-" "loaddefs-generate--" "no-update-autoloads"))
+(register-definition-prefixes "loaddefs-gen" '("autoload-" "generated-autoload-" "loaddefs-" "my/tmp-lddg-list2" "no-update-autoloads"))
 
 
 ;;; Generated autoloads from loadhist.el
@@ -24547,6 +24548,7 @@
 for the result of evaluating EXP (first arg to `pcase').
 
 (fn NAME ARGS [DOC] &rest BODY...)" nil t)
+(function-put 'pcase-defmacro 'autoload 'macro-expand)
 (function-put 'pcase-defmacro 'lisp-indent-function 2)
 (function-put 'pcase-defmacro 'doc-string-elt 3)
 (register-definition-prefixes "pcase" '("pcase-"))