Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros

From: "J.D. Smith" <[email-masked]>
Date: Sun, 27 Jul 2025 15:14:21 -0400

Stefan Monnier <[email-masked]> writes:

>> For whatever reason, I get lots of nil cars.
>
> Hmm... then there's something I'm missing.
> I wonder in which case(s) `car` is nil here.

One big category of nil car's resulted from all the :autoload-end's
around.  Because only the tail /after/ that keyword is trimmed (a
classic singly linked list issue), the keyword itself remains in the
form, and was still being (pointlessly) recursed on.  Easy fix.

That leaves a number with explicit nil cars in their expansion, for
whatever reason:

   Got nil car:
   nil "display-line-numbers" expansion
   Got nil car:
   nil "elec-pair" expansion
   Got nil car:
   nil "emacs-lock" expansion
   Got nil car:
   nil "hi-lock" expansion
   Got nil car:
   nil "hilit-chg" expansion
   Got nil car:
   nil "ruler-mode" expansion
   Got nil car:
   nil "saveplace" expansion
   Got nil car:
   nil "tab-line" expansion
     INFO     Scraping 1566 files for loaddefs...17% 
   Got nil car:
   nil "visual-wrap" expansion
   Got nil car:
   nil "whitespace" expansion
   Got nil car:
   nil "window-tool-bar" expansion
     INFO     Scraping 1566 files for loaddefs...39% 
     INFO     Scraping 1566 files for loaddefs...64% 
   Got nil car:
   nil "goto-addr" expansion
   loaddefs-gen: loading file tramp-adb (for tramp--with-startup)
   Loading tramp-adb...
   Source file ‘code/emacs/emacs-mac/lisp/net/tramp-loaddefs.el’ newer than byte-compiled file; using older file
   Source file ‘code/emacs/emacs-mac/lisp/emacs-lisp/cl-loaddefs.el’ newer than byte-compiled file; using older file
     INFO     Scraping 1566 files for loaddefs...80% 
   Got nil car:
   nil "cwarn" expansion
   Got nil car:
   nil "subword" expansion
   Got nil car:
   nil "subword" expansion
     INFO     Scraping 1566 files for loaddefs...92% 
   Got nil car:
   nil "word-wrap-mode" expansion

Probably define-minor-mode includes some bare symbols in the expansion?
Are there good reasons for including non-forms in an autoload file?
Bare strings or symbols?

BTW, I also realized there actually /was/ a small reason to put the
`autoload-macro' cond form first, before the special cases: all presence
on that list does is avoid /loading/ the file, if any symbol-car's
happen to be undefined (which is of course unlikely for such a
hard-coded list of special macros).  The recursing on expansion still
happens, if requested.  Putting ours first is a form of eating our own
dog food.  If all the defining-macros are cleaned up so that expansions
are sensible, the special case can be entirely omitted in favor of
direct expansion.  Letting the macros advertise their own interactive
forms = fewer bugs (of the nthcdr sort I had to correct).

Right now, autoloads proceed through the special case for:

 - define-derived-mode
 - cl-defun (I guess because cl-macs.el is not loaded?)
 - define-overloadable-function
 - define-skeleton
 - define-compilation-mode

>>> I'd have expected `function-get` here.
>> I was honestly vague on the difference.  Seems like `function-get'
>> should maybe chase aliases for you...
>
> It does.

Oh nice.  Not obvious from its code.  I've simplified accordingly. 

>>> I think you can just push this to `master` as  separate patch, thank you.
>> I've attached a separate patch here (I don't have commit privileges).
>
> Hmm...  According to
> https://savannah.gnu.org/project/memberlist.php?group=emacs,
> John-David T. Smith does have commit privileges.  I know "Smith" is not
> exactly rare, but...

Huh, maybe I've just never exercised them.  Or maybe I assumed there
would be a ceremony of some kind :).

> I'll gladly push it, if you prefer, of course.

No, it's a good chance to learn.  Will want to iterate on docs first, if
you think its ready (I think it is).  I took the chance to clean up the
declare forms to simplify the diff as well.

Have you had a chance to test it "in the wild" with a custom macro?

diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 366423904db..3e521e18567 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -271,6 +271,11 @@ 'byte-run--set-debug
 	    (list 'put (list 'quote name)
 		  ''edebug-form-spec (list 'quote spec)))))
 
+(defalias 'byte-run--set-autoload-macro
+  #'(lambda (name _args spec)
+      (list 'function-put (list 'quote name)
+	    ''autoload-macro (list 'quote spec))))
+
 (defalias 'byte-run--set-no-font-lock-keyword
   #'(lambda (name _args val)
       (list 'function-put (list 'quote name)
@@ -350,8 +355,10 @@ macro-declarations-alist
   (cons
    (list 'debug #'byte-run--set-debug)
    (cons
-    (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
-    defun-declarations-alist))
+    (list 'autoload-macro #'byte-run--set-autoload-macro)
+    (cons
+     (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+     defun-declarations-alist)))
   "List associating properties of macros to their macro expansion.
 Each element of the list takes the form (PROP FUN) where FUN is a function.
 For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
@@ -397,6 +404,7 @@ 'defmacro
            (if declarations
 	       (cons 'prog1 (cons def (car declarations)))
 	     def))))))
+(function-put 'defmacro 'autoload-macro 'expand) ; Since we cannot `declare' it
 
 ;; Now that we defined defmacro we can use it!
 (defmacro defun (name arglist &rest body)
@@ -409,7 +417,7 @@ defun
 The return value is undefined.
 
 \(fn NAME ARGLIST [DOCSTRING] [DECL] [INTERACTIVE] BODY...)"
-  (declare (doc-string 3) (indent 2))
+  (declare (doc-string 3) (indent 2) (autoload-macro expand))
   (or name (error "Cannot define '%s' as a function" name))
   (if (null
        (and (listp arglist)
diff --git a/lisp/emacs-lisp/cl-generic.el b/lisp/emacs-lisp/cl-generic.el
index 3aa26fba3c3..4a85d89d5fa 100644
--- a/lisp/emacs-lisp/cl-generic.el
+++ b/lisp/emacs-lisp/cl-generic.el
@@ -258,7 +258,8 @@ cl-defgeneric
                                cl--generic-edebug-make-name in:method]
                               lambda-doc
                               def-body)]]
-             def-body)))
+             def-body))
+           (autoload-macro expand))
   (let* ((doc (if (stringp (car-safe options-and-methods))
                   (pop options-and-methods)))
          (declarations nil)
diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 2a0a9e5c6de..1b64f106a1b 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -398,7 +398,8 @@ cl-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))
   `(defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
 ;;;###autoload
@@ -416,7 +417,8 @@ cl-iter-defun
                      [&optional ("interactive" interactive)]
                      def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))
   (require 'generator)
   `(iter-defun ,name ,@(cl--transform-lambda (cons args body) name)))
 
@@ -475,7 +477,8 @@ cl-defmacro
   (declare (debug
             (&define name cl-macro-list cl-declarations-or-string def-body))
            (doc-string 3)
-           (indent 2))
+           (indent 2)
+           (autoload-macro expand))
   `(defmacro ,name ,@(cl--transform-lambda (cons args body) name)))
 
 (def-edebug-elem-spec 'cl-lambda-expr
@@ -3011,7 +3014,8 @@ cl-defstruct
                         sexp])]
              [&optional stringp]
              ;; All the above is for the following def-form.
-             &rest &or symbolp (symbolp &optional def-form &rest sexp))))
+             &rest &or symbolp (symbolp &optional def-form &rest sexp)))
+           (autoload-macro expand))
   (let* ((name (if (consp struct) (car struct) struct))
 	 (warning nil)
 	 (opts (cdr-safe struct))
diff --git a/lisp/emacs-lisp/easy-mmode.el b/lisp/emacs-lisp/easy-mmode.el
index f53db48f0b7..0e428abd0b7 100644
--- a/lisp/emacs-lisp/easy-mmode.el
+++ b/lisp/emacs-lisp/easy-mmode.el
@@ -222,10 +222,11 @@ define-minor-mode
            (indent defun)
            (debug (&define name string-or-null-p
 			   [&optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp
-			    &optional [&not keywordp] sexp]
+			              &optional [&not keywordp] sexp
+			              &optional [&not keywordp] sexp]
 			   [&rest [keywordp sexp]]
-			   def-body)))
+			   def-body))
+           (autoload-macro expand))
 
   (let* ((last-message (make-symbol "last-message"))
          (mode-name (symbol-name mode))
@@ -487,7 +488,7 @@ define-globalized-minor-mode
 on if the hook has explicitly disabled it.
 
 \(fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)"
-  (declare (doc-string 2) (indent defun))
+  (declare (doc-string 2) (indent defun) (autoload-macro expand))
   (let* ((global-mode-name (symbol-name global-mode))
 	 (mode-name (symbol-name mode))
 	 (pretty-name (easy-mmode-pretty-mode-name mode))
diff --git a/lisp/emacs-lisp/generator.el b/lisp/emacs-lisp/generator.el
index 344e11e245e..59784ff51e8 100644
--- a/lisp/emacs-lisp/generator.el
+++ b/lisp/emacs-lisp/generator.el
@@ -675,7 +675,8 @@ iter-defun
 of values.  Callers can retrieve each value using `iter-next'."
   (declare (indent defun)
            (debug (&define name lambda-list lambda-doc &rest sexp))
-           (doc-string 3))
+           (doc-string 3)
+           (autoload-macro expand))
   (cl-assert lexical-binding)
   (let* ((parsed-body (macroexp-parse-body body))
          (declarations (car parsed-body))
diff --git a/lisp/emacs-lisp/inline.el b/lisp/emacs-lisp/inline.el
index c015e2b57d0..5acd6f17ee2 100644
--- a/lisp/emacs-lisp/inline.el
+++ b/lisp/emacs-lisp/inline.el
@@ -135,7 +135,7 @@ define-inline
 This is like `defmacro', but has several advantages.
 See Info node `(elisp)Defining Functions' for more details."
   ;; FIXME: How can this work with CL arglists?
-  (declare (indent defun) (debug defun) (doc-string 3))
+  (declare (indent defun) (debug defun) (doc-string 3) (autoload-macro expand))
   (let ((doc (if (stringp (car-safe body)) (list (pop body))))
         (declares (if (eq (car-safe (car-safe body)) 'declare) (pop body)))
         (cm-name (intern (format "%s--inliner" name)))
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..f9c6f82c31f 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -142,12 +142,29 @@ loaddefs-generate--shorten-autoload
              3)
         form))
 
+;; TODO: some macros can be removed from this list once they
+;;  (declare ((autoload-macro expand) ...)
+(defconst loaddefs--defining-macros
+  '(define-skeleton define-derived-mode
+     define-compilation-mode define-generic-mode
+     easy-mmode-define-global-mode define-global-minor-mode
+     define-globalized-minor-mode
+     easy-mmode-define-minor-mode define-minor-mode
+     cl-defun defun* cl-defmacro defmacro*
+     define-overloadable-function
+     transient-define-prefix transient-define-suffix
+     transient-define-infix transient-define-argument))
+
+(defvar loaddefs--load-error-files nil)
 (defun loaddefs-generate--make-autoload (form file &optional expansion)
   "Turn FORM into an autoload or defvar for source file FILE.
 Returns nil if FORM is not a special autoload form (i.e. a function definition
 or macro definition or a defcustom).
 If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently.
+
+Note that macros can request expansion by including `(autoload-macro
+expand)' among their `declare' forms."
   (let ((car (car-safe form)) expand)
     (cond
      ((and expansion (eq car 'defalias))
@@ -191,35 +208,40 @@ loaddefs-generate--make-autoload
           (setq form (copy-sequence form))
           (setcdr (memq :autoload-end form) nil))
         (let ((exps (delq nil (mapcar (lambda (form)
-                                        (loaddefs-generate--make-autoload
-                                         form file expansion))
+                                        (unless (eq form :autoload-end)
+                                          (loaddefs-generate--make-autoload
+                                           form file expansion)))
                                       (cdr form)))))
           (when exps (cons 'progn exps)))))
 
-     ;; For complex cases, try again on the macro-expansion.
-     ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
-                       define-globalized-minor-mode defun defmacro
-		       easy-mmode-define-minor-mode define-minor-mode
-                       define-inline cl-defun cl-defmacro cl-defgeneric
-                       cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
-           (macrop car)
-	   (setq expand (let ((load-true-file-name file)
-                              (load-file-name file))
-                          (macroexpand form)))
-	   (memq (car expand) '(progn prog1 defalias)))
+     ;; For macros which request it, try again on their expansion.
+     ((progn
+        ;; If the car is an unknown symbol, we load the file first to
+        ;; give packages a chance to define their macros.
+        (unless (or (not (symbolp car)) (fboundp car)
+                    ;; Special cases handled below
+                    (memq car loaddefs--defining-macros)
+                    (memq car '(defclass defcustom deftheme defgroup nil))
+                    (assoc file load-history)
+                    (member file loaddefs--load-error-files))
+          (let ((load-path (cons (file-name-directory file) load-path)))
+            (message "loaddefs-gen: loading file %s (for %s)" file car)
+            (condition-case e (load file)
+              (error
+               (push file loaddefs--load-error-files) ; do not attempt again
+               (warn "loaddefs-gen: load error\n\t%s" e)))))
+        (and (macrop car)
+	     (eq 'expand (function-get car 'autoload-macro))
+	     (setq expand (let ((load-true-file-name file)
+				(load-file-name file))
+			    (macroexpand form)))
+	     (not (eq car (car expand)))))
       ;; Recurse on the expansion.
       (loaddefs-generate--make-autoload expand file 'expansion))
 
-     ;; For special function-like operators, use the `autoload' function.
-     ((memq car '(define-skeleton define-derived-mode
-                   define-compilation-mode define-generic-mode
-		   easy-mmode-define-global-mode define-global-minor-mode
-		   define-globalized-minor-mode
-		   easy-mmode-define-minor-mode define-minor-mode
-		   cl-defun defun* cl-defmacro defmacro*
-                   define-overloadable-function
-                   transient-define-prefix transient-define-suffix
-                   transient-define-infix transient-define-argument))
+     ;; For known special macros which define functions, use `autoload'
+     ;; directly.
+     ((memq car loaddefs--defining-macros)
       (let* ((macrop (memq car '(defmacro cl-defmacro defmacro*)))
 	     (name (nth 1 form))
 	     (args (pcase car
diff --git a/lisp/emacs-lisp/pcase.el b/lisp/emacs-lisp/pcase.el
index 73554fd66fd..ab8edf1f569 100644
--- a/lisp/emacs-lisp/pcase.el
+++ b/lisp/emacs-lisp/pcase.el
@@ -537,7 +537,7 @@ pcase-defmacro
 By convention, DOC should use \"EXPVAL\" to stand
 for the result of evaluating EXP (first arg to `pcase').
 \n(fn NAME ARGS [DOC] &rest BODY...)"
-  (declare (indent 2) (debug defun) (doc-string 3))
+  (declare (indent 2) (debug defun) (doc-string 3) (autoload-macro expand))
   ;; Add the function via `fsym', so that an autoload cookie placed
   ;; on a pcase-defmacro will cause the macro to be loaded on demand.
   (let ((fsym (intern (format "%s--pcase-macroexpander" name)))