Package: emacs;
Reported by: JD Smith <jdtsmith <at> gmail.com>
Date: Fri, 11 Jul 2025 19:29:02 UTC
Severity: normal
Tags: patch
Fixed in version 31
Done: "J.D. Smith" <jdtsmith <at> gmail.com>
To reply to this bug, email your comments to 78995 AT debbugs.gnu.org.
There is no need to reopen the bug first.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Fri, 11 Jul 2025 19:29:02 GMT) Full text and rfc822 format available.JD Smith <jdtsmith <at> gmail.com>:bug-gnu-emacs <at> gnu.org.
(Fri, 11 Jul 2025 19:29:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: JD Smith <jdtsmith <at> gmail.com> To: bug-gnu-emacs <at> gnu.org Subject: [PATCH] ;;;autoload-expand for special macros Date: Fri, 11 Jul 2025 15:28:03 -0400
[Message part 1 (text/plain, inline)]
As mentioned over on emacs-devel, it's sometimes a problem that `loaddefs-gen' hard-codes the list of special macros it is willing to expand: (memq car '(easy-mmode-define-global-mode define-global-minor-mode define-globalized-minor-mode defun defmacro easy-mmode-define-minor-mode define-minor-mode define-inline cl-defun cl-defmacro cl-defgeneric cl-defstruct pcase-defmacro iter-defun cl-iter-defun)) This makes it challenging to wrap things like `define-minor-mode' into a new macro, and autoload the resulting forms. This small patch adds a new variant of the autoload cookie: ;;;autoload-expand (my-autoloadable-macro-to-expand ...) which allows package authors to "opt-in" to macro expansion of the following form during autoload generation. It would be useful primarily for special macros that are not in the hard-coded list above, but which expand to other special macros that are. Docs can be added if this looks good.
[autoload-expand.patch (application/octet-stream, attachment)]
[Message part 3 (text/plain, inline)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 12 Jul 2025 06:11:02 GMT) Full text and rfc822 format available.Message #8 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: JD Smith <jdtsmith <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>, Philip Kaludercic <philipk <at> posteo.net> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 09:10:05 +0300
> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Fri, 11 Jul 2025 15:28:03 -0400
>
> (defun loaddefs-generate--make-autoload (form file &optional expansion)
> "Turn FORM into an autoload or defvar for source file FILE.
> -Returns nil if FORM is not a special autoload form (i.e. a function definition
> -or macro definition or a defcustom).
> -If EXPANSION is non-nil, we're processing the macro expansion of an
> -expression, in which case we want to handle forms differently."
> +Returns nil if FORM is not a special autoload form (i.e. a function
> +definition or macro definition or a defcustom). If EXPANSION is
> +non-nil, we're processing the macro expansion of an expression, in which
> +case we want to handle forms differently. If it the symbol `force',
^^^^^^^^^^^^^
"it is the symbol".
Also, I wonder why you decided to refill the existing text. A doc
string is easier readable if each argument's description starts on a
new line, because then it's easier to scan the doc string if you are
looking for the description of a specific argument.
And finally, the last sentence, which you added, doesn't really
explain the effect of the special value 'force' -- can we do a better
job describing that?
> - (or local-outfile main-outfile))))
> + (or local-outfile main-outfile)))
> + (expansion (and (looking-at "-expand[ \t]*") 'force)))
And this part seems to be completely undocumented?
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 12 Jul 2025 15:01:01 GMT) Full text and rfc822 format available.Message #11 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: JD Smith <jdtsmith <at> gmail.com> To: Eli Zaretskii <eliz <at> gnu.org> Cc: Philip Kaludercic <philipk <at> posteo.net>, 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 10:59:45 -0400
[Message part 1 (text/plain, inline)]
Thanks, improved doc-strings in the attached. If this is looking good I can also update the `Autoload' section of the elisp manual, near "If you write a function definition with an unusual macro...". I also noticed that `:autoload-end' is not mentioned there, and probably should be (this keyword marker halts further processing of forms in the macro expansion). One question on nesting of macros: I'm wondering whether an ;;;###autoload-expand should result in using `EXPANSION=force' in all recursive calls to `loaddefs-generate--make-autoload'. I.e. should all macros be _recursively_ expanded with `force'? This would allow nesting, e.g. a macro wrapping a macro which itself wraps `define-minor-mode'. But it would also expand all other macros encountered in the expansion, which could be problematic. My instinct is that one level of such forced macro expansion is all that can be safely supported.
[autoload-expand.patch (application/octet-stream, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 12 Jul 2025 16:11:01 GMT) Full text and rfc822 format available.Message #14 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: JD Smith <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 12:09:55 -0400
> This makes it challenging to wrap things like `define-minor-mode' into a new
> macro, and autoload the resulting forms. This small patch adds a new
> variant of the autoload cookie:
>
> ;;;autoload-expand
> (my-autoloadable-macro-to-expand ...)
I suspect this will often fail to work because autoloads are often
generated by a fresh new session where `my-autoloadable-macro-to-expand`
will not yet be defined.
So at the very least, your patch should make sure to signal an error
when the cookie is of the form `;;;autoload-expand` by the head is not
currently defined as a macro.
I've been wanting to add the feature you propose, but this "macro is not
defined" has gotten in the way until now. I think we need to add
something like a file-local `autoload-requires-for-macros`.
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 12 Jul 2025 16:13:02 GMT) Full text and rfc822 format available.Message #17 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: JD Smith <jdtsmith <at> gmail.com> Cc: Eli Zaretskii <eliz <at> gnu.org>, 78995 <at> debbugs.gnu.org, Philip Kaludercic <philipk <at> posteo.net> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 12:12:42 -0400
> One question on nesting of macros: I'm wondering whether
> an ;;;###autoload-expand should result in using `EXPANSION=force' in all
> recursive calls to `loaddefs-generate--make-autoload'. I.e. should all
> macros be _recursively_ expanded with `force'? This would allow nesting,
> e.g. a macro wrapping a macro which itself wraps `define-minor-mode'.
> But it would also expand all other macros encountered in the expansion,
> which could be problematic. My instinct is that one level of such forced
> macro expansion is all that can be safely supported.
Another way might be to replace the `;;;###autoload-expand` marker with
an list of of macros which should be expanded (i.e. a way to extent the
hardcoded list).
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 12 Jul 2025 17:27:01 GMT) Full text and rfc822 format available.Message #20 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 13:26:04 -0400
[Message part 1 (text/plain, inline)]
Stefan Monnier <monnier <at> iro.umontreal.ca> writes: >> This makes it challenging to wrap things like `define-minor-mode' into a new >> macro, and autoload the resulting forms. This small patch adds a new >> variant of the autoload cookie: >> >> ;;;autoload-expand >> (my-autoloadable-macro-to-expand ...) > > I suspect this will often fail to work because autoloads are often > generated by a fresh new session where `my-autoloadable-macro-to-expand` > will not yet be defined. Good point. That chicken-and-egg problem was definitely in the back of my mind, and is why I didn't attempt to create an expandable list to replace the list of hard-coded special macros. > I've been wanting to add the feature you propose, but this "macro is not > defined" has gotten in the way until now. I think we need to add > something like a file-local `autoload-requires-for-macros`. How about this solution? If the car of the following form is not a macro (and not a function), attempt to load the file itself (temporarily expanding `load-path' to include the file's containing directory). If that fails to define the macro, warn the user, and simply proceed without expansion.
[autoload-expand.patch (text/x-patch, inline)]
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..f7919a7a377 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -147,7 +147,10 @@ loaddefs-generate--make-autoload
Returns nil if FORM is not a special autoload form (i.e. a function definition
or macro definition or a defcustom).
If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently. If
+EXPANSION is the symbol `force' and the following form is a macro call,
+it will be expanded prior to generating autoload forms, similar to what
+is done automatically for special macros like `define-minor-mode'."
(let ((car (car-safe form)) expand)
(cond
((and expansion (eq car 'defalias))
@@ -196,12 +199,13 @@ loaddefs-generate--make-autoload
(cdr form)))))
(when exps (cons 'progn exps)))))
- ;; For complex cases, try again on the macro-expansion.
- ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
- define-globalized-minor-mode defun defmacro
- easy-mmode-define-minor-mode define-minor-mode
- define-inline cl-defun cl-defmacro cl-defgeneric
- cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
+ ;; For complex cases and forced expansions, try again on the macro-expansion.
+ ((and (or (eq expansion 'force)
+ (memq car '(easy-mmode-define-global-mode define-global-minor-mode
+ define-globalized-minor-mode defun defmacro
+ easy-mmode-define-minor-mode define-minor-mode
+ define-inline cl-defun cl-defmacro cl-defgeneric
+ cl-defstruct pcase-defmacro iter-defun cl-iter-defun)))
(macrop car)
(setq expand (let ((load-true-file-name file)
(load-file-name file))
@@ -381,6 +385,10 @@ loaddefs-generate--parse-file
setting of `generated-autoload-file' in FILE, and
by ;;;###foo-autoload statements.
+The special statement ;;;###[foo-]autoload-expand causes the following
+form to be macro-expanded prior to generating autoloads; see
+`loaddefs-generate--make-autoload'.
+
If PACKAGE-DATA is `only', return only the package data. If t,
include the package data with the rest of the data. Otherwise,
don't include."
@@ -465,16 +473,38 @@ loaddefs-generate--parse-file
(expand-file-name
(concat aname "-loaddefs.el")
(file-name-directory file))
- (or local-outfile main-outfile))))
+ (or local-outfile main-outfile)))
+ (expansion (and (looking-at "-expand[ \t]*") 'force)))
+ (when expansion
+ (goto-char (match-end 0)))
(if (eolp)
;; We have a form following.
(let* ((form (prog1
(read (current-buffer))
(unless (bolp)
(forward-line 1))))
- (autoload (or (loaddefs-generate--make-autoload
- form load-name)
- form)))
+ (car (car form))
+ autoload)
+ (when expansion
+ ;; We encountered ;;;###autoload-expand, so the
+ ;; following form should be a macro. Check that
+ ;; it is defined. If it is undefined (and not a
+ ;; function), load the file to attempt to define
+ ;; it. If it remains undefined, warn and simply
+ ;; ignore the expand request.
+ (unless (or (macrop car) (functionp car))
+ (let ((load-path (cons (file-name-directory file) load-path)))
+ (message "loaddefs-gen: loading file %s" file)
+ (condition-case e (load file)
+ (error
+ (warn "loaddefs-gen: load error\n\t%s" e)))))
+ (unless (macrop car)
+ (warn "loaddefs-gen: macro undefined, skipping expansion\n\t%s"
+ car)
+ (setq expansion nil)))
+ (setq autoload (or (loaddefs-generate--make-autoload
+ form load-name expansion)
+ form))
;; We get back either an autoload form, or a tree
;; structure of `(progn ...)' things, so unravel that.
(let ((forms (if (eq (car autoload) 'progn)
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 12 Jul 2025 17:52:01 GMT) Full text and rfc822 format available.Message #23 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 13:51:09 -0400
>> I've been wanting to add the feature you propose, but this "macro is not
>> defined" has gotten in the way until now. I think we need to add
>> something like a file-local `autoload-requires-for-macros`.
> How about this solution? If the car of the following form is not a
> macro (and not a function), attempt to load the file itself (temporarily
> expanding `load-path' to include the file's containing directory). If
> that fails to define the macro, warn the user, and simply proceed
> without expansion.
I don't think it's good enough. Many of the use-cases I've bumped into
use macros defined in another file (or even another package).
I think we should aim to provide a mechanism that lets us reduce the
size of the hardcoded list (ideally to nothing).
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 12 Jul 2025 20:00:03 GMT) Full text and rfc822 format available.Message #26 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 15:59:44 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes: >>> I've been wanting to add the feature you propose, but this "macro is not >>> defined" has gotten in the way until now. I think we need to add >>> something like a file-local `autoload-requires-for-macros`. >> How about this solution? If the car of the following form is not a >> macro (and not a function), attempt to load the file itself (temporarily >> expanding `load-path' to include the file's containing directory). If >> that fails to define the macro, warn the user, and simply proceed >> without expansion. > > I don't think it's good enough. Many of the use-cases I've bumped into > use macros defined in another file (or even another package). Doesn't a file usually need to require such files/packages in order to make use of their macros? Or is the concern that those other files/packages would not be in the current directory or on the load-path? I think package-install installs all requirements first, so that shouldn't happen for it. Or perhaps the issue arises in cases like multi-file packages where `declare-function' is used because you know the load order. I.e. if: 1. A.el always requires B.el and is its entry point, 2. B.el uses a macro from A.el with an ;;;###autoload-expand, and 3. B.el does _not_ require A.el, only (possibly) mentioning it via `declare-function`. Then loading B.el is, as you say, not good enough. This is tricky to work around. > I think we should aim to provide a mechanism that lets us reduce the > size of the hardcoded list (ideally to nothing). There are actually _multiple_ hard-coded lists in `loaddefs-generate--make-autoload': 1. Macros to expand (by name) and recurse on the expansion. 2. "Function-like operators", which get an `(autoload ...)' form created for them. 3. Macros which are known to produce interactive function definitions (to set the INTERACTIVE arg for the generated autoload). What about just mentioning in the docs that using ;;;###autoload-expand successfully requires that the macro to be expanded is defined upon loading the containing file? There is a prominent warning if loading the file fails to define the macro, so developers should be aware (or made aware by their users). They can always, e.g., factor out their macro to a common file they can require. The "fallback" is equivalent to the current behavior: copy the full form into the autoload file. Otherwise, if a documentation solution is not good enough, we could create an additional file-local variable that could be consulted if the user of ;;;###autoload-expand needs to give it some help, e.g.: - `autoload-macro-expansion-files': a file-local list of additional file(s) to load prior to macro expansion, if any macros in calls directly preceded by ;;;###autoload-expand are undefined. A symbol `file' in this list indicates the current file. Defaults to `(file)'.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 12 Jul 2025 22:45:02 GMT) Full text and rfc822 format available.Message #29 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 18:43:56 -0400
>> I don't think it's good enough. Many of the use-cases I've bumped into
>> use macros defined in another file (or even another package).
> Doesn't a file usually need to require such files/packages in order to
> make use of their macros?
Duh! You're right.
> Or perhaps the issue arises in cases like multi-file packages where
> `declare-function' is used because you know the load order. I.e. if:
No, no, these are "evil" and they get what they deserve.
>> I think we should aim to provide a mechanism that lets us reduce the
>> size of the hardcoded list (ideally to nothing).
>
> There are actually _multiple_ hard-coded lists in
> `loaddefs-generate--make-autoload':
>
> 1. Macros to expand (by name) and recurse on the expansion.
> 2. "Function-like operators", which get an `(autoload ...)' form
> created for them.
> 3. Macros which are known to produce interactive function definitions
> (to set the INTERACTIVE arg for the generated autoload).
The only list that I've found a need to extend is the (1) above, so
I wouldn't worry too much about the other two.
[ Declaring specific macros as "to be expanded for autoload" (as opposed
to marking specific autoloaded macro calls), would still be overall
better, since it would solve the question of "what about macros that
expand to something that also needs expanding". ]
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 13 Jul 2025 02:26:01 GMT) Full text and rfc822 format available.Message #32 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 22:25:05 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
> The only list that I've found a need to extend is the (1) above, so
> I wouldn't worry too much about the other two.
>
> [ Declaring specific macros as "to be expanded for autoload" (as opposed
> to marking specific autoloaded macro calls), would still be overall
> better, since it would solve the question of "what about macros that
> expand to something that also needs expanding". ]
That's a good point. It would relieve us of the binary choice of
"expand just this form" vs. "expand any and all macros this form expands
to", which isn't ideal. But I wonder what the best approach is? A
file-local variable, which extends the (1) list only for that file?
We _could_ do both: an ;;;###autoload-expand magic comment for "expand
this one", and a optional file-local variable, something like
`autoload-extra-expand-macros', naming additional macros to expand at
any depth[1] (if needed). We could even allow elements of the list to
look like (MACRO . FILE) for an additional FILE to define MACRO load
(for the evil people), defaulting to `(file)' (meaning: this file).
That's some duplication of purpose, and you don't technically need the
magic form, but formatting file-local variables is a pain if you don't
need them.
[1] Would we possibly have to guard against recursive macro definitions?
An advantage of "just this next form" expansion is you don't have to
worry about that.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 13 Jul 2025 02:49:01 GMT) Full text and rfc822 format available.Message #35 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 12 Jul 2025 22:47:44 -0400
>> The only list that I've found a need to extend is the (1) above, so
>> I wouldn't worry too much about the other two.
>>
>> [ Declaring specific macros as "to be expanded for autoload" (as opposed
>> to marking specific autoloaded macro calls), would still be overall
>> better, since it would solve the question of "what about macros that
>> expand to something that also needs expanding". ]
>
> That's a good point. It would relieve us of the binary choice of
> "expand just this form" vs. "expand any and all macros this form expands
> to", which isn't ideal. But I wonder what the best approach is?
FWIW, I've come to the opinion that what you propose and what I want
match different use-cases. The one I care about is when a package
provides a (typically autoloaded) macro like `define-minor-mode`, for
use by other packages. But what you propose matches the use-case where
a package defines its own macro for it own internal use, usually with
no intention to autoload it.
> A file-local variable, which extends the (1) list only for that file?
I'm thinking we should replace the hardcoded list with a symbol
property. When defining macros like `define-minor-mode` we'd set this
`autoload-macroexpand` property to a non-nil value, and `autoload.el`
would simply check the property instead of looking inside the hardcoded
list with `memq`.
This would work only for predefined or autoloaded macros (where the
`autoload-macroexpand` setting would be also pre-defined/loaded), so
it wouldn't work conveniently for a locally-defined and locally-used
macro, since you'd then need to somehow set the property and define or
autoload the macro before `autoload.el` gets to the macro calls.
So I think your approach might be a good solution for the case of local
macros. I don't know how often we need such a thing, tho. I can't
remember bumping into it, but it might be because I'd intuitively avoid
it, knowing we don't have support for it.
I assume there's a concrete use-case which motivated your patch.
Do you mind sharing it?
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 13 Jul 2025 13:48:02 GMT) Full text and rfc822 format available.Message #38 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 13 Jul 2025 09:47:48 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
>> A file-local variable, which extends the (1) list only for that file?
>
> I'm thinking we should replace the hardcoded list with a symbol
> property. When defining macros like `define-minor-mode` we'd set this
> `autoload-macroexpand` property to a non-nil value, and `autoload.el`
> would simply check the property instead of looking inside the hardcoded
> list with `memq`.
Oh, this is much better! It relieves the _user_ of the macro the burden
of remembering to "auto-load it correctly." For macros which are
defined in every session, you're done. No special autoload-expand magic
comment needed. It also makes it possible to entirely eliminate
hard-coded list #1. You'd just need to insert, e.g.:
(put 'defun 'autoload-macroexpand t)
where the special macros are _defined_. This would be a a bit more
self-documenting too.
We'd leave the "function-like operator" hard-coded list, but it could
probably be trimmed down with the new `autoload-macroexpand' feature.
For example, `transient-define-prefix' boils down to a defalias, which
is already supported.
> This would work only for predefined or autoloaded macros (where the
> `autoload-macroexpand` setting would be also pre-defined/loaded), so
> it wouldn't work conveniently for a locally-defined and locally-used
> macro, since you'd then need to somehow set the property and define or
> autoload the macro before `autoload.el` gets to the macro calls.
We could be aggressive about it. If the car of some form encountered
during file scanning (including expansion) is a macro, we check its
`autoload-macroexpand' property and act accordingly. But if it not
already a known special-form, function, or macro, we load the file to
see if that defines it. If it _remains_ undefined, we just include the
whole form in the autoload file, just as we do now (generating a
warning). But if loading the file _did_ define it as a macro, we check
its `autoload-macroexpand' property, and proceed as before.
I was curious how many existing ;;;###autoloads this would affect, since
I almost always see it used with `defun' or `define-minor-mode'. I
looked through the entire emacs-30 lisp codebase[1] and could only find
`define-erc-module' as the car of a top-level autoload form which wasn't
already a known function, macro, or special form. I repeated this
exercise among my 171 installed packages, and found only:
Unknown autoload car: cape-char--define
Unknown autoload car: define-erc-module
Importantly, in all of these cases, they are actually using adorned
magic comments, like:
;;;###autoload (autoload 'cape-tex "cape-char" nil t)
(cape-char--define tex "TeX" ?\\ ?^ ?_)
so not relevant to our proposal (except that they could be improved by
it.)
> I assume there's a concrete use-case which motivated your patch.
> Do you mind sharing it?
I have a large package in development which has a flexible extension
system. I'd like to enable "3rd-party" extensions with a convenience
`define-extension' macro. In addition to defining a minor mode, this
macro will generate various `(put some-extension-mode ...)' statements,
which themselves need to be auto-loaded, as they are used to determine
when to load the extension (including, possibly, at package load time).
[1] Here's the distribution of pure ;;;###autoload forms in the emacs-30
codebase:
defun: 3226
defcustom: 181
define-minor-mode: 173
define-derived-mode: 157
defmacro: 130
defalias: 101
put: 88
defvar: 74
define-obsolete-function-alias: 27
defvar-local: 20
define-overloadable-function: 19
cl-defgeneric: 19
defconst: 15
define-globalized-minor-mode: 15
progn: 12
defclass: 8
add-to-list: 8
cl-defun: 8
defsubst: 7
cl-defmacro: 5
define-mail-user-agent: 4
or: 3
pcase-defmacro: 3
define-global-minor-mode: 2
make-variable-buffer-local: 2
eval-after-load: 2
dolist: 2
define-key: 2
unless: 1
autoload: 1
def-edebug-elem-spec: 1
defvar-keymap: 1
define-inline: 1
when: 1
define-compilation-mode: 1
define-skeleton: 1
let: 1
fset: 1
setq: 1
Created with:
rg -IA 1 -t elisp '^;;;###autoload *$' | perl -ne '$AL{$1}++ if /^\(([^ ]+) /; END { foreach $k (reverse sort { $AL{$a} <=> $AL{$b} } keys %AL) {print "$k: $AL{$k}\n"}}'
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 13 Jul 2025 15:20:01 GMT) Full text and rfc822 format available.Message #41 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 13 Jul 2025 11:19:08 -0400
>>> A file-local variable, which extends the (1) list only for that file?
>> I'm thinking we should replace the hardcoded list with a symbol
>> property. When defining macros like `define-minor-mode` we'd set this
>> `autoload-macroexpand` property to a non-nil value, and `autoload.el`
>> would simply check the property instead of looking inside the hardcoded
>> list with `memq`.
>
> Oh, this is much better! It relieves the _user_ of the macro the burden
> of remembering to "auto-load it correctly." For macros which are
> defined in every session, you're done. No special autoload-expand magic
> comment needed. It also makes it possible to entirely eliminate
> hard-coded list #1. You'd just need to insert, e.g.:
>
> (put 'defun 'autoload-macroexpand t)
>
> where the special macros are _defined_. This would be a a bit more
> self-documenting too.
Yes, tho we'd probably define a `declare` thingy so you wouldn't write
the `put` but you'd write
(defmacro define-minor-mode ...
(declare (...
(autoload macroexpand)
...))
...)
> We'd leave the "function-like operator" hard-coded list, but it could
> probably be trimmed down with the new `autoload-macroexpand' feature.
> For example, `transient-define-prefix' boils down to a defalias, which
> is already supported.
Ah, yes, I had not noticed that `transient-define-prefix` had been added
there, and now that I see it, it does make it seem like my impression
that it was unimportant may have been wrong.
Usually macroexpanding those things is enough (the macro can be defined
appropriately). We could still refine the above with something like:
(defmacro transient-define-prefix ...
(declare (...
(autoload function)
...))
...)
tho this probably needs more thought/design because we need more info
(where's the name of the defined thing, whether it's a macro or
a function, what's the list of arguments, ...).
>> This would work only for predefined or autoloaded macros (where the
>> `autoload-macroexpand` setting would be also pre-defined/loaded), so
>> it wouldn't work conveniently for a locally-defined and locally-used
>> macro, since you'd then need to somehow set the property and define or
>> autoload the macro before `autoload.el` gets to the macro calls.
>
> We could be aggressive about it. If the car of some form encountered
> during file scanning (including expansion) is a macro, we check its
> `autoload-macroexpand' property and act accordingly. But if it not
> already a known special-form, function, or macro, we load the file to
> see if that defines it.
Oh, yeah, neat.
> Importantly, in all of these cases, they are actually using adorned
> magic comments, like:
>
> ;;;###autoload (autoload 'cape-tex "cape-char" nil t)
> (cape-char--define tex "TeX" ?\\ ?^ ?_)
>
> so not relevant to our proposal (except that they could be improved by
> it.)
Yes, these are the ones that "document" a need for the new feature.
Stefan
diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 6412c8cde22..654b67ada1b 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -286,6 +286,11 @@
(list 'put (list 'quote name)
''edebug-form-spec (list 'quote spec)))))
+(defalias 'byte-run--set-autoload-macro
+ #'(lambda (name _args spec)
+ (list 'put (list 'quote name)
+ ''autoload-macro (list 'quote spec))))
+
(defalias 'byte-run--set-no-font-lock-keyword
#'(lambda (name _args val)
(list 'function-put (list 'quote name)
@@ -365,8 +370,10 @@ macro-declarations-alist
(cons
(list 'debug #'byte-run--set-debug)
(cons
- (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
- defun-declarations-alist))
+ (list 'autoload-macro #'byte-run--set-autoload-macro)
+ (cons
+ (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+ defun-declarations-alist)))
"List associating properties of macros to their macro expansion.
Each element of the list takes the form (PROP FUN) where FUN is a function.
For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 8a131bf885f..b8dfe095589 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -198,19 +198,20 @@
(when exps (cons 'progn exps)))))
;; For complex cases, try again on the macro-expansion.
- ((and (memq car '( define-globalized-minor-mode defun defmacro
+ ((and (or (eq 'expand (get car 'autoload-macro))
+ (memq car '( define-globalized-minor-mode defun defmacro
define-minor-mode define-inline
cl-defun cl-defmacro cl-defgeneric
cl-defstruct pcase-defmacro iter-defun cl-iter-defun
;; Obsolete; keep until the alias is removed.
easy-mmode-define-global-mode
easy-mmode-define-minor-mode
- define-global-minor-mode))
+ define-global-minor-mode)))
(macrop car)
(setq expand (let ((load-true-file-name file)
(load-file-name file))
(macroexpand form)))
- (memq (car expand) '(progn prog1 defalias)))
+ (not (eq car (car expand))))
;; Recurse on the expansion.
(loaddefs-generate--make-autoload expand file 'expansion))
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 13 Jul 2025 19:02:02 GMT) Full text and rfc822 format available.Message #44 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 13 Jul 2025 15:01:07 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
>> You'd just need to insert, e.g.:
>>
>> (put 'defun 'autoload-macroexpand t)
>>
>> where the special macros are _defined_. This would be a a bit more
>> self-documenting too.
>
> Yes, tho we'd probably define a `declare` thingy so you wouldn't write
> the `put` but you'd write
>
> (defmacro define-minor-mode ...
> (declare (...
> (autoload macroexpand)
Wow, this is even nicer! Never knew those `delare's were so simple.
>> We'd leave the "function-like operator" hard-coded list, but it could
>> probably be trimmed down with the new `autoload-macroexpand' feature.
>> For example, `transient-define-prefix' boils down to a defalias, which
>> is already supported.
>
> Ah, yes, I had not noticed that `transient-define-prefix` had been added
> there, and now that I see it, it does make it seem like my impression
> that it was unimportant may have been wrong.
> Usually macroexpanding those things is enough (the macro can be defined
> appropriately). We could still refine the above with something like:
>
> (defmacro transient-define-prefix ...
> (declare (...
> (autoload function)
I mean `defalias' is supported because `defun' expands to it. Isn't
that enough? I guess some macros are complex enough you want to take
the quick short cut and turn them right into an `(autoload ...)'
statement, without having to design them carefully?
> tho this probably needs more thought/design because we need more info
> (where's the name of the defined thing, whether it's a macro or
> a function, what's the list of arguments, ...).
The code in `--make-autoload' seems to think it knows how to handle all
those "function-like operators". But list 1 and list 2 seem to be
disjoint. E.g. `transient-define-prefix' is on list 2, but _not_ on
list 1. So it would like to be treated as a function operator, but not
expanded.
Actually I'm confused about that logic. Take `define-minor-mode'. It's
on list 1 (expand the macro) and list 2 (treat it as a function
operator). I don't see how it could ever get list 2 treatment, since it
always would have been expanded first by the list 1 logic above. There
are many macros in the 1 & 2 category:
;; Expand but don't treat as a special function (1, not 2):
(defun defmacro define-inline
cl-defgeneric
cl-defstruct
pcase-defmacro
iter-defun
cl-iter-defun)
;; Only treat as a special function op (2, not 1):
(define-skeleton
define-derived-mode
define-compilation-mode
define-generic-mode
defun*
defmacro*
define-overloadable-function
transient-define-prefix
transient-define-suffix
transient-define-infix
transient-define-argument)
;; Expand _and_ treat as a special function op (1 & 2):
(define-skeleton
define-derived-mode
define-compilation-mode
define-generic-mode
easy-mmode-define-global-mode
define-global-minor-mode
define-globalized-minor-mode
easy-mmode-define-minor-mode
define-minor-mode cl-defun
defun*
cl-defmacro
defmacro*
define-overloadable-function
transient-define-prefix
transient-define-suffix
transient-define-infix
transient-define-argument
defun defmacro define-inline
cl-defgeneric
cl-defstruct
pcase-defmacro
iter-defun
cl-iter-defun)
What am I missing?
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 13 Jul 2025 22:50:02 GMT) Full text and rfc822 format available.Message #47 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 13 Jul 2025 18:49:30 -0400
>> (defmacro define-minor-mode ...
>> (declare (...
>> (autoload macroexpand)
>
> Wow, this is even nicer! Never knew those `delare's were so simple.
🙂
[ Of course, third-party code would have to use `put` in the short term
for backward compatibility reasons. 🙁 ]
>> (defmacro transient-define-prefix ...
>> (declare (...
>> (autoload function)
>
> I mean `defalias' is supported because `defun' expands to it. Isn't
> that enough?
Yes, I think we could mark `transient-define-prefix` as `expand`.
My understanding is that currently putting it in "list 1" wouldn't work
simply because it's not autoloaded.
> The code in `--make-autoload' seems to think it knows how to handle all
> those "function-like operators". But list 1 and list 2 seem to be
> disjoint. E.g. `transient-define-prefix' is on list 2, but _not_ on
> list 1. So it would like to be treated as a function operator, but not
> expanded.
As mentioned above, I think it's just because of a lack of autoloading.
> ;; Expand _and_ treat as a special function op (1 & 2):
> (define-skeleton
> define-derived-mode
> define-compilation-mode
> define-generic-mode
> easy-mmode-define-global-mode
> define-global-minor-mode
> define-globalized-minor-mode
> easy-mmode-define-minor-mode
> define-minor-mode cl-defun
> defun*
> cl-defmacro
> defmacro*
> define-overloadable-function
> transient-define-prefix
> transient-define-suffix
> transient-define-infix
> transient-define-argument
> defun defmacro define-inline
> cl-defgeneric
> cl-defstruct
> pcase-defmacro
> iter-defun
> cl-iter-defun)
I don't see `cl-defstruct` in list 2 (same for several others), so
I think the above list isn't right.
But yes, some are in both lists and it's mostly "historical accidents".
IIRC `easy-mmode-define-minor-mode` started in list 2 and was later
added to list 1 so as to get a better result (e.g. more complete
docstring) in those cases where the macro happens to be defined.
But I think your trick of evaluating the file (to cause the macros
to be defined) should allow us to focus on list 1 and leave only the
strict minimum in list 2.
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Mon, 14 Jul 2025 00:06:02 GMT) Full text and rfc822 format available.Message #50 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 13 Jul 2025 20:05:07 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
> I don't see `cl-defstruct` in list 2 (same for several others), so
> I think the above list isn't right.
Oops, wrong set function. It's:
;; Expand and treat as function op:
(easy-mmode-define-global-mode
define-global-minor-mode
define-globalized-minor-mode
easy-mmode-define-minor-mode
define-minor-mode cl-defun
cl-defmacro)
> some are in both lists and it's mostly "historical accidents".
Makes sense.
> But I think your trick of evaluating the file (to cause the macros
> to be defined) should allow us to focus on list 1 and leave only the
> strict minimum in list 2.
I will work up a patch using this approach unless you'd like to (will be
a few days min). I think we are in agreement that a simple (autoload
macroexpand) will suffice. We could leave list 2 as-is for now, with a
note that it can be trimmed once `declare' for list 1 is fully
operational.
Probably would also need to loop in package authors, e.g. for transient,
to ensure they are aware of the change.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Mon, 14 Jul 2025 02:33:02 GMT) Full text and rfc822 format available.Message #53 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 13 Jul 2025 22:31:58 -0400
> I will work up a patch using this approach
Sounds good. 🙂
> I think we are in agreement that a simple (autoload macroexpand) will
> suffice. We could leave list 2 as-is for now, with a note that it can
> be trimmed once `declare' for list 1 is fully operational.
>
> Probably would also need to loop in package authors, e.g. for transient,
> to ensure they are aware of the change.
I think we can start by removing those entries that we can turn into
`declare` directly in Emacs (e.g. not those for Transient).
And then wait for the relevant package authors to start using the new
functionality before we remove the rest, bit by bit.
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 16 Jul 2025 03:37:01 GMT) Full text and rfc822 format available.Message #56 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Tue, 15 Jul 2025 23:35:54 -0400
[Message part 1 (text/plain, inline)]
Here is an update on progress. The attached is working reasonably well,
reproducing lisp/loaddefs.el with a few changes (see attached). I went
with (declare ((autoload . macro-expand))) on the notion that there
might be other autoload-related declarations in the future, but open to
suggestion.
I've now fully eliminated list 1 in favor of declare statements, but
also had to handle the case of _aliased_ macros, e.g. all the
[easy-mmode]-define-global[ized]-minor-mode yada yada.
There are a couple things I need to check on.
1. Non-repeatable loaddefs:
When you re-run make and it needs to regenerate lisp/loaddefs.el, it
appears easy-mmode is _not_ loaded by whatever emacs process conducts
that subsequent autoload-generation round. Consequently,
`define-minor-mode' isn't loaded, so calls to it produce only partial
autoload entries (via, I believe, list 2). Maybe that's why list 2
duplicates entries from list 1, as a backup against this non-repeatable
build. But it seems strange to have different packages loaded on
different passes of the same Makefile target. You can confirm this by:
% mv lisp/loaddefs.el lisp/loaddefs_orig.el
% make
% diff -u lisp/loaddefs_orig.el lisp/loaddefs.el
I'm not clear if this has anything to do with this patch, or could even
be a pre-existing build bug. Once you are in this situation, you
basically have to `make bootstrap' to get the full and complete autoload
statements to appear again in loaddefs.
2. Some files are now being loaded during autoloads generation:
I noticed a few files among bundled lisp packages that now get loaded,
because they autoload undefined macros:
loaddefs-gen: loading file frameset (cl-defun)
Loading frameset...
INFO Scraping 1565 files for loaddefs...15%
loaddefs-gen: loading file ede/cpp-root (defclass)
Loading ede/cpp-root...
INFO Scraping 1565 files for loaddefs...25%
INFO Scraping 1565 files for loaddefs...47%
INFO Scraping 1565 files for loaddefs...64%
loaddefs-gen: loading file tramp-adb (tramp--with-startup)
Loading tramp-adb...
INFO Scraping 1565 files for loaddefs...80%
loaddefs-gen: loading file grep (define-compilation-mode)
Loading grep...
These mostly seem to be innocuous in that after loading them, it becomes
clear that these particular autoloaded macros don't request
macro-expansion, so nothing different happens. But it does represent a
few new "internal" load calls during autoload generation. I think list
1 is now "doing more work" because of this pre-load mechanism.
Thoughts and suggestions very welcome.
[autoload-expand_2.patch (text/x-patch, attachment)]
[loaddefs.diff (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 23 Jul 2025 21:21:02 GMT) Full text and rfc822 format available.Message #59 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: 78995 <at> debbugs.gnu.org Cc: Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 23 Jul 2025 17:20:13 -0400
[Message part 1 (text/plain, inline)]
Stefan, can you take a look at this? I think I now understand why there are slight differences in the loaddefs.el file generated by my declare-based, eager-loading patch. There is a /bug/ in the fallback "list 3" generation of autoloads, leading to incorrect autoload statements being written out. The `INTERACTIVE' arg to `autoload' is supposed to be formed as follows: If INTERACTIVE is a list, it is interpreted as a list of modes the function is applicable for. The current code mistakenly injects the actual ARG-DESCRIPTOR list into the autoload file, for those (few) macros which are not expanded but fall through to list 2, possibly just `cl-defun's. You might see for example the full `(let...` descriptors for `erc` in your loaddefs.el file. That's a bug I believe, and will presumably mess with the mode-specific M-S-x type functionality. So by expanding more eagerly with `(declare (autoload macro-expand))', this bug was "eliminated". The attached fixes it for any cases which still use fall-through to list 2. Thoughts?
[autoload-bug.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Thu, 24 Jul 2025 14:10:02 GMT) Full text and rfc822 format available.Message #62 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: 78995 <at> debbugs.gnu.org Cc: Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Thu, 24 Jul 2025 10:09:07 -0400
[Message part 1 (text/plain, inline)]
"J.D. Smith" <jdtsmith <at> gmail.com> writes:
> Here is an update on progress. The attached is working reasonably well,
> reproducing lisp/loaddefs.el with a few changes (see attached). I went
> with (declare ((autoload . macro-expand))) on the notion that there
> might be other autoload-related declarations in the future, but open to
> suggestion.
Please have a look at this latest patch, which I believe solves the
mentioned problems.
Changes:
1. I implemented the list 2 bug fix for autoload's INTERACTIVE argument
described above, correctly keeping interactive arg specs out of the
loaddefs files.
2. I moved the autoload macro-expand declare up front everywhere it's
used[1], to ensure the macro-expand request will be included in
*-loaddefs.el files. This is important for packages which manage
their own *-loaddefs.el files (e.g. `cl-macs').
3. I have kept list 2 around[2], and count presence on this list, or the
list of 4 specially-handled macros (defgroup, defcustom, deftheme,
defclass) as "we know about this car". No file load will be
attempted for such special macros, so only proper "user-defined
UNKNOWN macros" will result in a load attempt.
With these, the loaddefs.el file changes only modestly (see attached),
and (almost) no files have load attempted.
About that almost. There is one small order-of-operation issue yet to
be solved, for packages which do clever things. Take any of the
`net/tramp-*.el` files. They all require `tramp-compat`, which requires
`tramp-loaddefs', which is itself an autoload file the build process is
generating. For example, tramp's autoload file eventually includes the
macro `tramp--with-startup'. When one of tramp's autoloads use an
unknown macro like this, there will be an attempt to load the file,
implicitly requiring that yet-to-exist `tramp-loaddefs.el' file,
resulting in a (handled) error. Classic chicken and egg.
INFO Scraping 1543 files for loaddefs...70%
loaddefs-gen: loading file tramp-adb (for tramp--with-startup)
Loading /Users/jdsmith/code/emacs/emacs-mac/lisp/net/tramp-adb.el (source)...
Warning (emacs): loaddefs-gen: load error
(file-missing Cannot open load file No such file or directory tramp-loaddefs)
loaddefs-gen: loading file tramp-androidsu (for tramp--with-startup)
Loading /Users/jdsmith/code/emacs/emacs-mac/lisp/net/tramp-androidsu.el (source)...
Warning (emacs): loaddefs-gen: load error
(file-missing Cannot open load file No such file or directory tramp-loaddefs)
...
In this case these are harmless warnings, since in fact
`tramp--with-startup' is /not/ requesting macro-expand, so failing to
define it doesn't matter. Subsequent builds do not show this error.
But this could be alarming to people, so we probably need a solution.
[1] Note that the order of declare entries already matters, because some
of them generate forms which include `:autoload-end', and this
token, once encountered _anywhere_, puts an absolute stop to
emitting additional declare-generated forms into loaddefs.
E.g. `(declare (debug ...))' does this. This, by the way, is why
/some/ macros have their (e.g.) `indent' declares passed through to
the loaddefs files, and others do not.
[2] Some of these macros aren't yet suited for full expansion anyway.
For example `define-derived-mode' includes a bunch of unwanted forms
if it gets the expansion treatment. So for now, leaving the
shortcut list 2 intact seems to be the lesser evil.
[autoload-expand_3.patch (text/x-patch, attachment)]
[loaddefs_2.diff (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 26 Jul 2025 17:14:02 GMT) Full text and rfc822 format available.Message #65 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 26 Jul 2025 13:13:40 -0400
[ Sorry for the long delay. ]
>> Here is an update on progress. The attached is working reasonably well,
>> reproducing lisp/loaddefs.el with a few changes (see attached). I went
>> with (declare ((autoload . macro-expand))) on the notion that there
>> might be other autoload-related declarations in the future, but open to
>> suggestion.
[ I don't have an opinion on the naming here but I don't understand your
argument: wouldn't a declaration name of `autoload` be *more* likely
to conflict with another future autoload-related declaration? ]
> Warning (emacs): loaddefs-gen: load error
> (file-missing Cannot open load file No such file or directory tramp-loaddefs)
> ...
>
> In this case these are harmless warnings, since in fact
> `tramp--with-startup' is /not/ requesting macro-expand, so failing to
> define it doesn't matter. Subsequent builds do not show this error.
> But this could be alarming to people, so we probably need a solution.
Yeah, it's one of those nasty issues. One way to look at it is that the
crux of the matter is that we use `;;;###autoload` for 2 different
purpose:
A. To copy the code into the loaddefs file.
B. To arrange for the file to be autoloaded when the corresponding thingy
is used.
(B) is the one that deserves the name "autoload" while (A) might be
better named "preload".
Your `;;;###autoload-expand` was a fix for that core problem, in a sense.
The distinction between the two is not that clear when you consider that
a `;;;###autoload` cookie placed on a `defun` may arrange to autoload
that file when calling that function but it may *also* preload some
`put`s extracted from the `declare`.
Maybe a "general" solution would be to introduce a `;;;###preload`.
Not sure it's worth the trouble, tho.
Also, I'm not sure how "general" this would be, since the same problem
will tend to happen with any package which `;;;###autoload` a call to
a macro that's not predefined if that package uses its own
`FOO-loaddefs.el`, even if that macro is defined in some other library.
So, the general problem is: your patch will sometimes `eval-buffer` for
a buffer which doesn't expect to be evaluated before its package's
`FOO-loaddefs.el` has been created. I suspect that in most cases this
can be fixed by replacing
(require 'foo-loaddefs)
by
(require 'foo-loaddefs nil t)
It's not ideal, but I think it's acceptable. Maybe it won't always be
sufficient either. E.g. in `cl-lib` we solved this problem (for
a different reason) with:
(unless (load "cl-loaddefs" 'noerror 'quiet)
;; When bootstrapping, cl-loaddefs hasn't been built yet!
(require 'cl-macs)
(require 'cl-seq)
;; FIXME: Arguably we should also load `cl-extra', except that this
;; currently causes more bootstrap troubles, and `cl-extra' is
;; rarely used, so instead we explicitly (require 'cl-extra) at
;; those rare places where we do need it.
)
BTW, in the specific case of Tramp, I see:
(defmacro tramp--with-startup (&rest body)
"Schedule BODY to be executed at the end of tramp.el."
`(add-hook 'tramp--startup-hook (lambda nil ,@body)))
so I think `tramp--with-startup` *is* one of the macros that would
benefit from your new `(autoload macro-expand)` declaration.
> [1] Note that the order of declare entries already matters, because some
> of them generate forms which include `:autoload-end', and this
> token, once encountered _anywhere_, puts an absolute stop to
> emitting additional declare-generated forms into loaddefs.
> E.g. `(declare (debug ...))' does this. This, by the way, is why
> /some/ macros have their (e.g.) `indent' declares passed through to
> the loaddefs files, and others do not.
Really? Sounds like a bug. Are you sure?
(macroexpand '(defmacro my-foo () (declare (debug t) (indent 1)) (help)))
gives me:
(prog1 (defalias 'my-foo (cons 'macro #'(lambda nil (help))))
(progn :autoload-end (put 'my-foo 'edebug-form-spec 't))
(function-put 'my-foo 'lisp-indent-function 1))
and AFAIK the `:autoload-end` truncation is done with
(let ((end (memq :autoload-end form)))
(when end ;Cut-off anything after the :autoload-end marker.
(setq form (copy-sequence form))
(setcdr (memq :autoload-end form) nil))
so it should only truncate to:
(prog1 (defalias 'my-foo (cons 'macro #'(lambda nil (help))))
(progn :autoload-end)
(function-put 'my-foo 'lisp-indent-function 1))
and thus keep the subsequent `indent` declaration intact.
> [2] Some of these macros aren't yet suited for full expansion anyway.
> For example `define-derived-mode' includes a bunch of unwanted forms
> if it gets the expansion treatment. So for now, leaving the
> shortcut list 2 intact seems to be the lesser evil.
Interesting.
> +(defalias 'byte-run--set-autoload
> + #'(lambda (name _args spec)
> + (list 'function-put (list 'quote name)
> + ''autoload (list 'quote spec))))
I think the choice of `autoload` as name of symbol property is too
generic and runs too high a risk of clashing with other uses.
> (defconst loaddefs--function-like-operators
"operators" doesn't sound quite right.
Maybe "loaddefs--defining-macros"?
> If EXPANSION is non-nil, we're processing the macro expansion of an
> -expression, in which case we want to handle forms differently."
> +expression, in which case we want to handle forms differently. Note
> +that macros can request expansion by including `(autoload macro-expand)'
> +among their `declare' forms."
Nitpick: I'd start "Note..." on its own line. We're not trying to
justify-fill the text. 🙂
> @@ -196,30 +212,35 @@ loaddefs-generate--make-autoload
> (cdr form)))))
> (when exps (cons 'progn exps)))))
>
> - ;; For complex cases, try again on the macro-expansion.
> - ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
> - define-globalized-minor-mode defun defmacro
> - easy-mmode-define-minor-mode define-minor-mode
> - define-inline cl-defun cl-defmacro cl-defgeneric
> - cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
> - (macrop car)
> - (setq expand (let ((load-true-file-name file)
> - (load-file-name file))
> - (macroexpand form)))
> - (memq (car expand) '(progn prog1 defalias)))
> + ;; For macros which request it, try again on their expansion.
> + ((progn
> + ;; If the car is entirely unknown, we load the file first to
> + ;; give packages a chance to define their macros.
> + (unless (or (null car) (macrop car) (functionp car) (special-form-p car)
I'd test (and (symbolp car) (not (fboundp car)))
> + (memq car '(defclass defcustom deftheme defgroup))
This deserves a comment in the code.
> + (alist-get file load-history)
`alist-get` defaults to testing with `eq` so it won't work well with
a string key. 🙁
I'd go with `assoc`.
> + (and (macrop car)
> + (eq 'macro-expand ; a macro can declare (autoload macro-expand)
> + (or (get car 'autoload)
> + (get (car-safe (last (function-alias-p car))) 'autoload)))
I'd have expected `function-get` here.
> - ;; For special function-like operators, use the `autoload' function.
> - ((memq car '(define-skeleton define-derived-mode
> - define-compilation-mode define-generic-mode
> - easy-mmode-define-global-mode define-global-minor-mode
> - define-globalized-minor-mode
> - easy-mmode-define-minor-mode define-minor-mode
> - cl-defun defun* cl-defmacro defmacro*
> - define-overloadable-function
> - transient-define-prefix transient-define-suffix
> - transient-define-infix transient-define-argument))
> + ;; For special function-like operators, use the `autoload'
> + ;; function.
> + ((memq car loaddefs--function-like-operators)
> (let* ((macrop (memq car '(defmacro cl-defmacro defmacro*)))
> (name (nth 1 form))
> (args (pcase car
If we swap this case with the preceding one we can remove the
(memq car loaddefs--function-like-operators)
from the `unless` of that case.
[ And it probably results in a diff that's harder to read, so thanks for
not doing it this time. 🙂 ]
> @@ -257,8 +278,8 @@ loaddefs-generate--make-autoload
> t)
> (and (eq (car-safe (car body)) 'interactive)
> ;; List of modes or just t.
> - (or (if (nthcdr 1 (car body))
> - (list 'quote (nthcdr 1 (car body)))
> + (or (if (nthcdr 2 (car body))
> + (list 'quote (nthcdr 2 (car body)))
> t))))
> ,(if macrop ''macro nil)))))
I think you can just push this to `master` as separate patch, thank you.
> @@ -19835,7 +19836,7 @@
> loaddefs.el output file, and the rest are the directories to
> use.")
> (load "theme-loaddefs.el" t)
> -(register-definition-prefixes "loaddefs-gen" '("autoload-" "generated-autoload-" "loaddefs-generate--" "no-update-autoloads"))
> +(register-definition-prefixes "loaddefs-gen" '("autoload-" "generated-autoload-" "loaddefs-" "my/tmp-lddg-list2" "no-update-autoloads"))
Maybe better use `autoload-` rather than `loaddefs-` as prefix for the
new definitions.
[ I don't see `my/tmp-lddg-list2` in the patch you sent, so I assume it
was something temporary. ]
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 27 Jul 2025 01:47:02 GMT) Full text and rfc822 format available.Message #68 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 26 Jul 2025 21:46:46 -0400
[Message part 1 (text/plain, inline)]
Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
Thanks for your feedback.
>>> I went with (declare ((autoload . macro-expand))) on the notion that
>>> there might be other autoload-related declarations in the future,
>>> but open to suggestion.
>
> [ I don't have an opinion on the naming here but I don't understand your
> argument: wouldn't a declaration name of `autoload` be *more* likely
> to conflict with another future autoload-related declaration? ]
I suppose I had imagined multiple flags, à la:
(autoload (macro-expand something-else))
but maybe that's frowned on. In which case we should just go with
`(autoload-macro . expand)'.
> One way to look at it is that the crux of the matter is that we use
> `;;;###autoload` for 2 different purpose:
>
> A. To copy the code into the loaddefs file.
> B. To arrange for the file to be autoloaded when the corresponding thingy
> is used.
>
> (B) is the one that deserves the name "autoload" while (A) might be
> better named "preload".
That's a sensible way of looking at it.
> The distinction between the two is not that clear when you consider that
> a `;;;###autoload` cookie placed on a `defun` may arrange to autoload
> that file when calling that function but it may *also* preload some
> `put`s extracted from the `declare`.
>
> Maybe a "general" solution would be to introduce a `;;;###preload`.
> Not sure it's worth the trouble, tho.
Could be considered radical. I guess the contract would be that all
`;;;###preload's would happen before all `;;;autoload's? I can imagine
people confusing these two (similar to use-package's :init vs. :config).
> Also, I'm not sure how "general" this would be, since the same problem
> will tend to happen with any package which `;;;###autoload` a call to
> a macro that's not predefined if that package uses its own
> `FOO-loaddefs.el`, even if that macro is defined in some other library.
It would be general if it were universally adopted, and autoload
scanning proceeded across all files, first generating "stub" loaddefs
for preloads, then, in a subsequent pass, doing the autoloads.
The main issue as I see it is we are now (for the first time) possibly
/loading/ files, vs. just scanning them passively. I think the
situation where this distinction matters is quite rare. Note that the
only file(s) where a load are attempted now are for TRAMP.
> So, the general problem is: your patch will sometimes `eval-buffer` for
> a buffer which doesn't expect to be evaluated before its package's
> `FOO-loaddefs.el` has been created. I suspect that in most cases this
> can be fixed by replacing
>
> (require 'foo-loaddefs)
> by
> (require 'foo-loaddefs nil t)
>
> It's not ideal, but I think it's acceptable.
What about big packages (like tramp) that maintain and load their own
autoloads, and would like to request macro expansion for some of their
package-defined macros, and then also /use/ that macro in autoloads?
E.g. imagine if `tramp--with-startup' wanted to be expanded instead of
included as-is ("preloaded"). That's a pretty specific case, obviously.
> BTW, in the specific case of Tramp, I see:
>
> (defmacro tramp--with-startup (&rest body)
> "Schedule BODY to be executed at the end of tramp.el."
> `(add-hook 'tramp--startup-hook (lambda nil ,@body)))
>
> so I think `tramp--with-startup` *is* one of the macros that would
> benefit from your new `(autoload macro-expand)` declaration.
Ah right, my imagining above was not so misguided then. Maybe the tramp
maintainer will want to expand that simple macro away now. I guess
package authors have to decide:
- Do you want to include ("preload") your full macro and any autoloaded
uses of it (as is) in *-loaddefs.el? You should be sure to require
your package's custom autoload file with suppressed errors, in case it
doesn't yet exist during build.
- Would you prefer your package's macro is expanded at autoload
generation time instead? You should require it "for real", and
declare it (autoload-macro expand).
- You can't get both behaviors.
>> [1] Note that the order of declare entries already matters,
>
> Really? Sounds like a bug. Are you sure?
You're right, it was a transitory bug. I've moved them all back to the
end (since they are most recently added that seems to make sense).
>> (defconst loaddefs--function-like-operators
>
> "operators" doesn't sound quite right.
> Maybe "loaddefs--defining-macros"?
Yeah I agree, I was picking up the comment below that:
;; For special function-like operators, use the `autoload'
;; function.
But I like your version better (and have changed the comment).
>> +Note that macros can request expansion by including `(autoload
>> macro-expand)' +among their `declare' forms."
>
> Nitpick: I'd start "Note..." on its own line. We're not trying to
> justify-fill the text. 🙂
Done (even an extra blank to call it out).
>> + ;; If the car is entirely unknown, we load the file first to
>> + ;; give packages a chance to define their macros.
>> + (unless (or (null car) (macrop car) (functionp car) (special-form-p car)
>
> I'd test (and (symbolp car) (not (fboundp car)))
For whatever reason, I get lots of nil cars. So how about:
(unless (or (not (symbolp car)) (fboundp car) (null car)
... ;; try to load the file
Is
(eq (fboundp x) (or (macrop x) (functionp x) (special-form-p x))) ∀x∈ (symbolp x)?
>> + (memq car '(defclass defcustom deftheme defgroup))
> This deserves a comment in the code.
Done.
> I'd go with `assoc`.
Thanks.
>> + (and (macrop car)
>> + (eq 'macro-expand ; a macro can declare (autoload macro-expand)
>> + (or (get car 'autoload)
>> + (get (car-safe (last (function-alias-p car))) 'autoload)))
>
> I'd have expected `function-get` here.
I was honestly vague on the difference. Seems like `function-get'
should maybe chase aliases for you...
> If we swap this case with the preceding one we can remove the
>
> (memq car loaddefs--function-like-operators)
>
> from the `unless` of that case.
> [ And it probably results in a diff that's harder to read, so thanks for
> not doing it this time. 🙂 ]
Good idea. I only recently decided this was necessary. Though it isn't
really needed anymore with this change, I think keeping a defconst is a
good idea.
>> @@ -257,8 +278,8 @@ loaddefs-generate--make-autoload
>> t)
>> (and (eq (car-safe (car body)) 'interactive)
>> ;; List of modes or just t.
>> - (or (if (nthcdr 1 (car body))
>> - (list 'quote (nthcdr 1 (car body)))
>> + (or (if (nthcdr 2 (car body))
>> + (list 'quote (nthcdr 2 (car body)))
>> t))))
>> ,(if macrop ''macro nil)))))
>
> I think you can just push this to `master` as separate patch, thank you.
I've attached a separate patch here (I don't have commit privileges).
>> @@ -19835,7 +19836,7 @@
>> loaddefs.el output file, and the rest are the directories to
>> use.")
>> (load "theme-loaddefs.el" t)
>> -(register-definition-prefixes "loaddefs-gen" '("autoload-" "generated-autoload-" "loaddefs-generate--" "no-update-autoloads"))
>> +(register-definition-prefixes "loaddefs-gen" '("autoload-" "generated-autoload-" "loaddefs-" "my/tmp-lddg-list2" "no-update-autoloads"))
>
> Maybe better use `autoload-` rather than `loaddefs-` as prefix for the
> new definitions.
> [ I don't see `my/tmp-lddg-list2` in the patch you sent, so I assume it
> was something temporary. ]
This last file was an inadvertent inclusion.
Please take a look at the attached patches, and give a try with your own
user macro declaring autoload-expand if you can. If this all looks good
I can add to the docs and NEWS.
[no-arg-descriptor-in-autoload.patch (text/x-patch, attachment)]
[autoload-expand_4.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 27 Jul 2025 07:13:02 GMT) Full text and rfc822 format available.Message #71 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 27 Jul 2025 09:12:24 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes:
Hi,
>> BTW, in the specific case of Tramp, I see:
>>
>> (defmacro tramp--with-startup (&rest body)
>> "Schedule BODY to be executed at the end of tramp.el."
>> `(add-hook 'tramp--startup-hook (lambda nil ,@body)))
>>
>> so I think `tramp--with-startup` *is* one of the macros that would
>> benefit from your new `(autoload macro-expand)` declaration.
>
> Ah right, my imagining above was not so misguided then. Maybe the tramp
> maintainer will want to expand that simple macro away now.
I give this explanation to Stefan, he added this macro to Tramp years
ago (commit 07d8827946f9) :-)
--8<---------------cut here---------------start------------->8---
Functionally split tramp-loaddefs.el into two parts: one part run
while loading it at the very beginning of loading tramp.el (holding plain
autoloads and declarations), and another run at the very end of loading
tramp.el (holding setup code).
This should reduce infinite-recursion problems while loading.
--8<---------------cut here---------------end--------------->8---
See also bug#59194, which waits for me understanding what to do.
Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 27 Jul 2025 13:13:02 GMT) Full text and rfc822 format available.Message #74 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 27 Jul 2025 09:11:47 -0400
> I suppose I had imagined multiple flags, à la:
>
> (autoload (macro-expand something-else))
>
> but maybe that's frowned on.
It's not frowned upon no. You could also have
(autoload macro-expand something-else)
>> Maybe a "general" solution would be to introduce a `;;;###preload`.
>> Not sure it's worth the trouble, tho.
> Could be considered radical. I guess the contract would be that all
> `;;;###preload's would happen before all `;;;autoload's? I can imagine
> people confusing these two (similar to use-package's :init vs. :config).
Yeah, I'd rather not go there (I think it might have made sense to
distinguish the two at the beginning, but now that it's been conflated
for so many years it'd be much harder to introduce the distinction).
>>> + ;; If the car is entirely unknown, we load the file first to
>>> + ;; give packages a chance to define their macros.
>>> + (unless (or (null car) (macrop car) (functionp car) (special-form-p car)
>>
>> I'd test (and (symbolp car) (not (fboundp car)))
>
> For whatever reason, I get lots of nil cars.
Hmm... then there's something I'm missing.
I wonder in which case(s) `car` is nil here.
> So how about:
>
> (unless (or (not (symbolp car)) (fboundp car) (null car)
> ... ;; try to load the file
Fine by me, tho I'd still like to understand why we need to care about `nil`.
We could also add `nil` to the (defclass defcustom deftheme defgroup) list.
> Is
>
> (eq (fboundp x) (or (macrop x) (functionp x) (special-form-p x))) ∀x∈ (symbolp x)?
No. E.g. `x` could be bound to a keymap, or to a string (old-style
keyboard macro), or potentially to anything else (integer, hash-table,
you name it: that's not useful in the `symbol-function` cell but that
doesn't mean it can't happen).
>> I'd have expected `function-get` here.
> I was honestly vague on the difference. Seems like `function-get'
> should maybe chase aliases for you...
It does.
>> I think you can just push this to `master` as separate patch, thank you.
> I've attached a separate patch here (I don't have commit privileges).
Hmm... According to
https://savannah.gnu.org/project/memberlist.php?group=emacs,
John-David T. Smith does have commit privileges. I know "Smith" is not
exactly rare, but...
I'll gladly push it, if you prefer, of course.
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 27 Jul 2025 13:15:02 GMT) Full text and rfc822 format available.Message #77 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: 78995 <at> debbugs.gnu.org, "J.D. Smith" <jdtsmith <at> gmail.com> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 27 Jul 2025 09:14:33 -0400
>> Ah right, my imagining above was not so misguided then. Maybe the tramp
>> maintainer will want to expand that simple macro away now.
>
> I give this explanation to Stefan, he added this macro to Tramp years
> ago (commit 07d8827946f9) :-)
>
> --8<---------------cut here---------------start------------->8---
> Functionally split tramp-loaddefs.el into two parts: one part run
> while loading it at the very beginning of loading tramp.el (holding plain
> autoloads and declarations), and another run at the very end of loading
> tramp.el (holding setup code).
> This should reduce infinite-recursion problems while loading.
> --8<---------------cut here---------------end--------------->8---
>
> See also bug#59194, which waits for me understanding what to do.
[ It did look like something I'd seen before but I had no recollection
of actually writing it. ]
Damn! So now I'm the one who needs to figure out how to keep it working
in Emacs<31 while trying to make it behave with the new
"autoload-expand".
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 27 Jul 2025 18:00:02 GMT) Full text and rfc822 format available.Message #80 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 27 Jul 2025 13:59:05 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes: >>> Ah right, my imagining above was not so misguided then. Maybe the tramp >>> maintainer will want to expand that simple macro away now. >> >> I give this explanation to Stefan, he added this macro to Tramp years >> ago (commit 07d8827946f9) :-) >> >> --8<---------------cut here---------------start------------->8--- >> Functionally split tramp-loaddefs.el into two parts: one part run >> while loading it at the very beginning of loading tramp.el (holding plain >> autoloads and declarations), and another run at the very end of loading >> tramp.el (holding setup code). >> This should reduce infinite-recursion problems while loading. >> --8<---------------cut here---------------end--------------->8--- >> >> See also bug#59194, which waits for me understanding what to do. > > [ It did look like something I'd seen before but I had no recollection > of actually writing it. ] > > Damn! So now I'm the one who needs to figure out how to keep it working > in Emacs<31 while trying to make it behave with the new > "autoload-expand". Maybe it is as simple as requiring a (new?) file that defines the macro directly, and whenever you require tramp-loaddefs, using NOERROR=t? Then in Emacs 31, the first file with an ;;;###autoload followed by usage of TRAMP's macro will be loaded, causing the macro to be defined. If you want, you can declare `(autoload-macro expand)' when you define the macro (I think old Emacs version will just ignore it?). This will mean: - In Emacs 31: the macro itself is never mentioned, but the add-hook and any other forms are instead copied into tramp-loaddefs. - In earlier Emacs, the macro and then calls to it are both included in tramp-loaddefs, just as they are now. I think either setup works (though on-demand expansion may be "safer", since it doesn't depend on load order any more.)
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 27 Jul 2025 19:15:01 GMT) Full text and rfc822 format available.Message #83 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 27 Jul 2025 15:14:21 -0400
[Message part 1 (text/plain, inline)]
Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
>> For whatever reason, I get lots of nil cars.
>
> Hmm... then there's something I'm missing.
> I wonder in which case(s) `car` is nil here.
One big category of nil car's resulted from all the :autoload-end's
around. Because only the tail /after/ that keyword is trimmed (a
classic singly linked list issue), the keyword itself remains in the
form, and was still being (pointlessly) recursed on. Easy fix.
That leaves a number with explicit nil cars in their expansion, for
whatever reason:
Got nil car:
nil "display-line-numbers" expansion
Got nil car:
nil "elec-pair" expansion
Got nil car:
nil "emacs-lock" expansion
Got nil car:
nil "hi-lock" expansion
Got nil car:
nil "hilit-chg" expansion
Got nil car:
nil "ruler-mode" expansion
Got nil car:
nil "saveplace" expansion
Got nil car:
nil "tab-line" expansion
INFO Scraping 1566 files for loaddefs...17%
Got nil car:
nil "visual-wrap" expansion
Got nil car:
nil "whitespace" expansion
Got nil car:
nil "window-tool-bar" expansion
INFO Scraping 1566 files for loaddefs...39%
INFO Scraping 1566 files for loaddefs...64%
Got nil car:
nil "goto-addr" expansion
loaddefs-gen: loading file tramp-adb (for tramp--with-startup)
Loading tramp-adb...
Source file ‘code/emacs/emacs-mac/lisp/net/tramp-loaddefs.el’ newer than byte-compiled file; using older file
Source file ‘code/emacs/emacs-mac/lisp/emacs-lisp/cl-loaddefs.el’ newer than byte-compiled file; using older file
INFO Scraping 1566 files for loaddefs...80%
Got nil car:
nil "cwarn" expansion
Got nil car:
nil "subword" expansion
Got nil car:
nil "subword" expansion
INFO Scraping 1566 files for loaddefs...92%
Got nil car:
nil "word-wrap-mode" expansion
Probably define-minor-mode includes some bare symbols in the expansion?
Are there good reasons for including non-forms in an autoload file?
Bare strings or symbols?
BTW, I also realized there actually /was/ a small reason to put the
`autoload-macro' cond form first, before the special cases: all presence
on that list does is avoid /loading/ the file, if any symbol-car's
happen to be undefined (which is of course unlikely for such a
hard-coded list of special macros). The recursing on expansion still
happens, if requested. Putting ours first is a form of eating our own
dog food. If all the defining-macros are cleaned up so that expansions
are sensible, the special case can be entirely omitted in favor of
direct expansion. Letting the macros advertise their own interactive
forms = fewer bugs (of the nthcdr sort I had to correct).
Right now, autoloads proceed through the special case for:
- define-derived-mode
- cl-defun (I guess because cl-macs.el is not loaded?)
- define-overloadable-function
- define-skeleton
- define-compilation-mode
>>> I'd have expected `function-get` here.
>> I was honestly vague on the difference. Seems like `function-get'
>> should maybe chase aliases for you...
>
> It does.
Oh nice. Not obvious from its code. I've simplified accordingly.
>>> I think you can just push this to `master` as separate patch, thank you.
>> I've attached a separate patch here (I don't have commit privileges).
>
> Hmm... According to
> https://savannah.gnu.org/project/memberlist.php?group=emacs,
> John-David T. Smith does have commit privileges. I know "Smith" is not
> exactly rare, but...
Huh, maybe I've just never exercised them. Or maybe I assumed there
would be a ceremony of some kind :).
> I'll gladly push it, if you prefer, of course.
No, it's a good chance to learn. Will want to iterate on docs first, if
you think its ready (I think it is). I took the chance to clean up the
declare forms to simplify the diff as well.
Have you had a chance to test it "in the wild" with a custom macro?
[autoload-expand_5.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Tue, 29 Jul 2025 21:19:02 GMT) Full text and rfc822 format available.Message #86 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Tue, 29 Jul 2025 17:18:33 -0400
>> Damn! So now I'm the one who needs to figure out how to keep it working
>> in Emacs<31 while trying to make it behave with the new
>> "autoload-expand".
> Maybe it is as simple as requiring a (new?) file that defines the macro
> directly, and whenever you require tramp-loaddefs, using NOERROR=t?
I *think* it's simpler than that: just add the `declare` thingy and the
NOERROR=t.
Stefan
diff --git a/lisp/net/tramp-compat.el b/lisp/net/tramp-compat.el
index 9787e3a6553..f003bbebf31 100644
--- a/lisp/net/tramp-compat.el
+++ b/lisp/net/tramp-compat.el
@@ -29,7 +29,7 @@
;;; Code:
-(require 'tramp-loaddefs)
+(require 'tramp-loaddefs nil t) ;; Don't error while building the autoloads.
(require 'ansi-color)
(require 'auth-source)
(require 'format-spec)
diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 2208ce880d7..485c18fc4a2 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -105,6 +105,7 @@ tramp-completion-file-name-regexp
(defmacro tramp--with-startup (&rest body)
"Schedule BODY to be executed at the end of tramp.el."
+ (declare (autoload macro-expand))
`(add-hook 'tramp--startup-hook (lambda () ,@body)))
(eval-and-compile
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Tue, 29 Jul 2025 21:31:01 GMT) Full text and rfc822 format available.Message #89 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Tue, 29 Jul 2025 17:29:49 -0400
> One big category of nil car's resulted from all the :autoload-end's around.
Oh, what you mean is that nil is when the element is not a cons cell?
Indeed that makes sense.
[ Cons cells with a nil element would be surprising, OTOH. ]
> That leaves a number with explicit nil cars in their expansion, for
> whatever reason:
I think these have `nil` in the list of forms (that follows a `progn`),
but not in the `car` of one of the forms, right? So again, it's when
`form` is not a cons-cell.
In any case, indeed those non-cons situations justify the "not nil" check.
> Probably define-minor-mode includes some bare symbols in the expansion?
> Are there good reasons for including non-forms in an autoload file?
> Bare strings or symbols?
No, they can be removed, but if we compile the `.el`, then the
compiler takes care of it for us.
>> I'll gladly push it, if you prefer, of course.
> No, it's a good chance to learn. Will want to iterate on docs first, if
> you think its ready (I think it is).
I was thinking to do it in the single-hunk bug fix, which I do think is
ready (it just needs a commit message).
> Have you had a chance to test it "in the wild" with a custom macro?
No, not yet. But I think the transient macros are a good test case.
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 00:37:02 GMT) Full text and rfc822 format available.Message #92 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Tue, 29 Jul 2025 20:36:11 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes: >>> Damn! So now I'm the one who needs to figure out how to keep it working >>> in Emacs<31 while trying to make it behave with the new >>> "autoload-expand". >> Maybe it is as simple as requiring a (new?) file that defines the macro >> directly, and whenever you require tramp-loaddefs, using NOERROR=t? > > I *think* it's simpler than that: just add the `declare` thingy and the > NOERROR=t. Ahah, I see that tramp-compat is required, so this should work. It's `(autoload-macro expand)' btw.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 01:42:02 GMT) Full text and rfc822 format available.Message #95 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Tue, 29 Jul 2025 21:41:12 -0400
[Message part 1 (text/plain, inline)]
Stefan Monnier <monnier <at> iro.umontreal.ca> writes: >> One big category of nil car's resulted from all the :autoload-end's around. > > Oh, what you mean is that nil is when the element is not a cons cell? > Indeed that makes sense. > [ Cons cells with a nil element would be surprising, OTOH. ] Yep, like :autoload-end :). >> That leaves a number with explicit nil cars in their expansion, for >> whatever reason: > > I think these have `nil` in the list of forms (that follows a `progn`), > but not in the `car` of one of the forms, right? So again, it's when > `form` is not a cons-cell. > In any case, indeed those non-cons situations justify the "not nil" check. Yes, but I just wonder what such non-conses are doing in the expansion. Maybe just doc strings or some such. >> No, it's a good chance to learn. Will want to iterate on docs first, if >> you think its ready (I think it is). > > I was thinking to do it in the single-hunk bug fix, which I do think is > ready (it just needs a commit message). Is the commit message in need of expansion? I worked up some quick docs and included in the patch; see attached. >> Have you had a chance to test it "in the wild" with a custom macro? > > No, not yet. But I think the transient macros are a good test case. Good idea. Michael, if you can try this along with a patch like the one Stefan proposed for tramp (correcting the declare form name) that would be useful. Assuming it all works as expected, you'll notice `tramp-loaddefs.el' no longer contains the macro and the calls to it, but rather only the expanded forms it generates.
[autoload-expand_6.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 08:01:02 GMT) Full text and rfc822 format available.Message #98 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 30 Jul 2025 10:00:00 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes:
Hi,
> Good idea. Michael, if you can try this along with a patch like the one
> Stefan proposed for tramp (correcting the declare form name) that would
> be useful. Assuming it all works as expected, you'll notice
> `tramp-loaddefs.el' no longer contains the macro and the calls to it,
> but rather only the expanded forms it generates.
Before doing so, I've tried Stefan's patch with Emacs 28. There are
compilation warnings.
--8<---------------cut here---------------start------------->8---
# gmake -C lisp distclean ; gmake -C lisp EMACS=/usr/local/src/emacs-28/src/emacs
gmake: Entering directory '/home/albinus/src/tramp/lisp'
rm -f .\#* \#* .*~ *~ *.aux *.cp *.cps *.diff *.dvi *.elc *.fn *.fns *.html *.info *.ky *.kys *.log *.pdf *.pg *.tmp *.toc *.tp *.vr *.vrs
rm -f tramp-loaddefs.el
gmake: Leaving directory '/home/albinus/src/tramp/lisp'
gmake: Entering directory '/home/albinus/src/tramp/lisp'
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l autoload \
--eval "(setq generate-autoload-cookie \";;;###tramp-autoload\")" \
--eval "(setq generated-autoload-file \
(expand-file-name \"tramp-loaddefs.el\"))" \
--eval "(setq make-backup-files nil)" \
-f batch-update-autoloads .
SCRAPE .
INFO Scraping files for tramp-loaddefs.el...
INFO Scraping files for tramp-loaddefs.el...done
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-loaddefs.el
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile trampver.el
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-compat.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
In toplevel form:
tramp.el:109:50: Warning: Unknown macro property autoload-macro in
tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-adb.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-androidsu.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-cache.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-cmds.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-container.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-crypt.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-ftp.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-fuse.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-integration.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-message.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-rclone.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-sh.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-smb.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-sshfs.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-sudoedit.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-uu.el
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-archive.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
/usr/local/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-gvfs.el
tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
gmake: Leaving directory '/home/albinus/src/tramp/lisp'
--8<---------------cut here---------------end--------------->8---
And also at runtime.
--8<---------------cut here---------------start------------->8---
# /usr/local/src/emacs-28/src/emacs -Q -L /home/albinus/src/tramp/lisp /ssh::
For information about GNU Emacs and the GNU system, type C-h C-a.
../home/albinus/src/tramp/lisp/tramp-loaddefs.el: Warning: Unknown macro property autoload-macro in tramp--with-startup
Waiting for git... [2 times]
Tramp: Opening connection for gandalf using ssh...done
Tramp: Setup connection for gandalf using ssh...done
--8<---------------cut here---------------end--------------->8---
Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 12:35:02 GMT) Full text and rfc822 format available.Message #101 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 30 Jul 2025 08:34:13 -0400
Michael Albinus <michael.albinus <at> gmx.de> writes: > "J.D. Smith" <jdtsmith <at> gmail.com> writes: > > Hi, > >> Good idea. Michael, if you can try this along with a patch like the one >> Stefan proposed for tramp (correcting the declare form name) that would >> be useful. Assuming it all works as expected, you'll notice >> `tramp-loaddefs.el' no longer contains the macro and the calls to it, >> but rather only the expanded forms it generates. > > Before doing so, I've tried Stefan's patch with Emacs 28. There are > compilation warnings. > ... > And also at runtime. That would seem to make it difficult to add new declare forms and simultaneously support older versions of Emacs. How have such issues been dealt with in the past? Do you have a version or branch of TRAMP that is only deployed on master?
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 15:40:01 GMT) Full text and rfc822 format available.Message #104 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 30 Jul 2025 11:38:55 -0400
>>> Good idea. Michael, if you can try this along with a patch like the one
>>> Stefan proposed for tramp (correcting the declare form name) that would
>>> be useful. Assuming it all works as expected, you'll notice
>>> `tramp-loaddefs.el' no longer contains the macro and the calls to it,
>>> but rather only the expanded forms it generates.
>>
>> Before doing so, I've tried Stefan's patch with Emacs 28. There are
>> compilation warnings.
>> ...
>> And also at runtime.
>
> That would seem to make it difficult to add new declare forms and
> simultaneously support older versions of Emacs. How have such issues
> been dealt with in the past? Do you have a version or branch of TRAMP
> that is only deployed on master?
I think it's not that bad: just adding the ERROR=t is sufficient.
The `autoload-macro` declaration can be kept for later, once Tramp drops
support for Emacs<31.
I suspect the same will apply for Transient: we'll want to make sure the
new functionality works well for it, but we'll have to keep the old
ad-hoc solution until it drops support for Emacs<31.
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 15:45:02 GMT) Full text and rfc822 format available.Message #107 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 30 Jul 2025 17:44:15 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes:
Hi,
>> Before doing so, I've tried Stefan's patch with Emacs 28. There are
>> compilation warnings.
>> ...
>> And also at runtime.
>
> That would seem to make it difficult to add new declare forms and
> simultaneously support older versions of Emacs. How have such issues
> been dealt with in the past? Do you have a version or branch of TRAMP
> that is only deployed on master?
No, there's only one Tramp branch, which is deployed on master and on
GNU ELPA.
Tramp uses an own declare form `tramp-suppress-trace'. In order to make
it happen, there's the code
--8<---------------cut here---------------start------------->8---
(eval-and-compile
(defalias 'tramp-byte-run--set-suppress-trace
#'(lambda (f _args val)
(list 'function-put (list 'quote f)
''tramp-suppress-trace val)))
(add-to-list
'defun-declarations-alist
(list 'tramp-suppress-trace #'tramp-byte-run--set-suppress-trace))))
--8<---------------cut here---------------end--------------->8---
Perhaps this trick could be used?
Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 15:55:03 GMT) Full text and rfc822 format available.Message #110 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 30 Jul 2025 11:54:18 -0400
Michael Albinus <michael.albinus <at> gmx.de> writes: > > Tramp uses an own declare form `tramp-suppress-trace'. In order to make > it happen, there's the code > We are already adding the new declare form to `macro-declarations-alist', but the problem is, this code will not exist in Emacs <31. I guess we could recommend: (add-to-list macro-declarations-alist (list autoload-macro ignore)) in the meantime. But just skipping the declare (for now) as Stefan suggests is I guess the simpler approach. People who want to use it will need to require v>=31.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 17:52:02 GMT) Full text and rfc822 format available.Message #113 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 30 Jul 2025 19:51:02 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes: Hi, >> Tramp uses an own declare form `tramp-suppress-trace'. In order to make >> it happen, there's the code > > We are already adding the new declare form to > `macro-declarations-alist', but the problem is, this code will not > exist in Emacs <31. I meant to do this in tramp.el or tramp-compat.el Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 30 Jul 2025 18:26:01 GMT) Full text and rfc822 format available.Message #116 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 30 Jul 2025 14:25:35 -0400
Michael Albinus <michael.albinus <at> gmx.de> writes:
> "J.D. Smith" <jdtsmith <at> gmail.com> writes:
>
> Hi,
>
>>> Tramp uses an own declare form `tramp-suppress-trace'. In order to make
>>> it happen, there's the code
>>
>> We are already adding the new declare form to
>> `macro-declarations-alist', but the problem is, this code will not
>> exist in Emacs <31.
>
> I meant to do this in tramp.el or tramp-compat.el
Aha. One idea would be for tramp-compat to including something like:
;; Remove when Emacs >=31 is required
(require 'loaddefs-gen)
(cl-symbol-macrolet
((amac (alist-get 'autoload-macro macro-declarations-alist)))
(unless amac (setq amac #'ignore)))
to silence the warning, yet remain harmless once Emacs 31 is in the
wild.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Thu, 31 Jul 2025 05:25:02 GMT) Full text and rfc822 format available.Message #119 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Thu, 31 Jul 2025 07:23:51 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes: Hi, > Aha. One idea would be for tramp-compat to including something like: > > ;; Remove when Emacs >=31 is required > (require 'loaddefs-gen) > (cl-symbol-macrolet > ((amac (alist-get 'autoload-macro macro-declarations-alist))) > (unless amac (setq amac #'ignore))) > > to silence the warning, yet remain harmless once Emacs 31 is in the > wild. Feel free to add this to Tramp in master. I'm busy these days; will check over the weekend. Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Fri, 01 Aug 2025 14:40:02 GMT) Full text and rfc822 format available.Message #122 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Fri, 01 Aug 2025 16:39:37 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes: Hi, >>>> Tramp uses an own declare form `tramp-suppress-trace'. In order to make >>>> it happen, there's the code >>> >>> We are already adding the new declare form to >>> `macro-declarations-alist', but the problem is, this code will not >>> exist in Emacs <31. >> >> I meant to do this in tramp.el or tramp-compat.el > > Aha. One idea would be for tramp-compat to including something like: > > ;; Remove when Emacs >=31 is required > (require 'loaddefs-gen) > (cl-symbol-macrolet > ((amac (alist-get 'autoload-macro macro-declarations-alist))) > (unless amac (setq amac #'ignore))) > > to silence the warning, yet remain harmless once Emacs 31 is in the > wild. That doesn't work with Emacs 28: --8<---------------cut here---------------start------------->8--- # make -C lisp EMACS=/home/albinus/src/emacs-28/src/emacs make: Entering directory '/home/albinus/src/tramp/lisp' /home/albinus/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l autoload \ --eval "(setq generate-autoload-cookie \";;;###tramp-autoload\")" \ --eval "(setq generated-autoload-file \ (expand-file-name \"tramp-loaddefs.el\"))" \ --eval "(setq make-backup-files nil)" \ -f batch-update-autoloads . SCRAPE . INFO Scraping files for tramp-loaddefs.el... INFO Scraping files for tramp-loaddefs.el...done /home/albinus/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-loaddefs.el /home/albinus/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile trampver.el /home/albinus/src/emacs-28/src/emacs -Q -batch -L /home/albinus/src/tramp/lisp -l bytecomp -f batch-byte-compile tramp-compat.el In toplevel form: tramp-compat.el:33:1: Error: Cannot open load file: No such file or directory, loaddefs-gen make: *** [Makefile:77: tramp-compat.elc] Error 1 make: Leaving directory '/home/albinus/src/tramp/lisp' --8<---------------cut here---------------end--------------->8--- Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Fri, 01 Aug 2025 18:49:03 GMT) Full text and rfc822 format available.Message #125 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Fri, 01 Aug 2025 14:48:06 -0400
Michael Albinus <michael.albinus <at> gmx.de> writes:
> "J.D. Smith" <jdtsmith <at> gmail.com> writes:
>
> Hi,
>
>>>>> Tramp uses an own declare form `tramp-suppress-trace'. In order to make
>>>>> it happen, there's the code
>>>>
>>>> We are already adding the new declare form to
>>>> `macro-declarations-alist', but the problem is, this code will not
>>>> exist in Emacs <31.
>>>
>>> I meant to do this in tramp.el or tramp-compat.el
>>
>> Aha. One idea would be for tramp-compat to including something like:
>>
>> ;; Remove when Emacs >=31 is required
>> (require 'loaddefs-gen)
>> (cl-symbol-macrolet
>> ((amac (alist-get 'autoload-macro macro-declarations-alist)))
>> (unless amac (setq amac #'ignore)))
>>
>> to silence the warning, yet remain harmless once Emacs 31 is in the
>> wild.
>
> That doesn't work with Emacs 28:
>
Hmm, I guess `loaddefs-gen' was introduced in emacs 29. But I think you
don't actually need that `require' at all. Maybe try a simple:
(unless (alist-get 'autoload-macro macro-declarations-alist)
(push '(autoload-macro ignore) macro-declarations-alist))
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Fri, 01 Aug 2025 22:38:02 GMT) Full text and rfc822 format available.Message #128 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>, Eli Zaretskii <eliz <at> gnu.org>, Philip Kaludercic <philipk <at> posteo.net> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Fri, 01 Aug 2025 18:37:28 -0400
[Message part 1 (text/plain, inline)]
JD Smith <jdtsmith <at> gmail.com> writes: Stefan, Eli, any comments on the documentation (or other) parts of this patch? Stefan, do you think the proposed TRAMP solution (add an `ignore' `macro-declarations-alist' entry for `autoload-macro' for earlier versions of emacs) makes sense?
[autoload-expand_6.patch (text/x-patch, inline)]
diff --git a/doc/lispref/functions.texi b/doc/lispref/functions.texi
index be3c1e78889..25a01d942b6 100644
--- a/doc/lispref/functions.texi
+++ b/doc/lispref/functions.texi
@@ -2633,6 +2633,11 @@ Declare Form
This is typically used for macros, though it works for functions too.
@xref{Indenting Macros}.
+@item (autoload-macro @var{value})
+This is used when defining a macro. If @var{value} is @samp{expand},
+any calls to the macro which follow an autoload comment will first be
+expanded during generation of the autoloads. @xref{Autoload}.
+
@item (interactive-only @var{value})
Set the function's @code{interactive-only} property to @var{value}.
@xref{The interactive-only property}.
diff --git a/doc/lispref/loading.texi b/doc/lispref/loading.texi
index 750f6e76eff..65a55547a9f 100644
--- a/doc/lispref/loading.texi
+++ b/doc/lispref/loading.texi
@@ -710,8 +710,31 @@ Autoload
If you write a function definition with an unusual macro that is not
one of the known and recognized function definition methods, use of an
ordinary magic autoload comment would copy the whole definition into
-@code{loaddefs.el}. That is not desirable. You can put the desired
-@code{autoload} call into @code{loaddefs.el} instead by writing this:
+@code{loaddefs.el}. That is often not desirable. In this case, you can
+use the special declare form @code{(autoload-macro expand)} in your
+macro definition (@pxref{Declare Form}). This indicates to the autoload
+system that calls to your macro following @samp{;;;###autoload} should
+be expanded, similar to how the special functions listed above are
+handled. For example, a macro which wraps @code{define-minor-mode} can
+request expansion, so that proper @code{autoload} calls for the function
+it defines are generated.
+
+The symbol @code{:autoload-end} can be used in the expansion to prevent
+including unwanted forms in the autoload output. Its presence causes
+any further elements within the list where it appears to be silently
+skipped.
+
+Note that, if a symbol in the car of an autoloaded form is not defined
+during autoload generation, the file in which the associated
+@samp{;;;###autoload} appears will itself be loaded, to give the macro a
+chance to be defined. Packages which use special loading mechanisms,
+including requiring their own @samp{foo-autoload.el} file, should
+therefore gracefully handle load errors, since these files may not yet
+exist during autoload generation. This can be done, e.g., by setting
+the @var{no-error} argument of @code{require} non-@code{nil}).
+
+Alternatively, instead of expansion, you can put the desired
+@code{autoload} call into @code{loaddefs.el} by writing this:
@example
;;;###autoload (autoload 'foo "myfile")
diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 366423904db..3e521e18567 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -271,6 +271,11 @@ 'byte-run--set-debug
(list 'put (list 'quote name)
''edebug-form-spec (list 'quote spec)))))
+(defalias 'byte-run--set-autoload-macro
+ #'(lambda (name _args spec)
+ (list 'function-put (list 'quote name)
+ ''autoload-macro (list 'quote spec))))
+
(defalias 'byte-run--set-no-font-lock-keyword
#'(lambda (name _args val)
(list 'function-put (list 'quote name)
@@ -350,8 +355,10 @@ macro-declarations-alist
(cons
(list 'debug #'byte-run--set-debug)
(cons
- (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
- defun-declarations-alist))
+ (list 'autoload-macro #'byte-run--set-autoload-macro)
+ (cons
+ (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+ defun-declarations-alist)))
"List associating properties of macros to their macro expansion.
Each element of the list takes the form (PROP FUN) where FUN is a function.
For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
@@ -397,6 +404,7 @@ 'defmacro
(if declarations
(cons 'prog1 (cons def (car declarations)))
def))))))
+(function-put 'defmacro 'autoload-macro 'expand) ; Since we cannot `declare' it
;; Now that we defined defmacro we can use it!
(defmacro defun (name arglist &rest body)
@@ -409,7 +417,7 @@ defun
The return value is undefined.
\(fn NAME ARGLIST [DOCSTRING] [DECL] [INTERACTIVE] BODY...)"
- (declare (doc-string 3) (indent 2))
+ (declare (doc-string 3) (indent 2) (autoload-macro expand))
(or name (error "Cannot define '%s' as a function" name))
(if (null
(and (listp arglist)
diff --git a/lisp/emacs-lisp/cl-generic.el b/lisp/emacs-lisp/cl-generic.el
index 3aa26fba3c3..4a85d89d5fa 100644
--- a/lisp/emacs-lisp/cl-generic.el
+++ b/lisp/emacs-lisp/cl-generic.el
@@ -258,7 +258,8 @@ cl-defgeneric
cl--generic-edebug-make-name in:method]
lambda-doc
def-body)]]
- def-body)))
+ def-body))
+ (autoload-macro expand))
(let* ((doc (if (stringp (car-safe options-and-methods))
(pop options-and-methods)))
(declarations nil)
diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 2a0a9e5c6de..1b64f106a1b 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -398,7 +398,8 @@ cl-defun
[&optional ("interactive" interactive)]
def-body))
(doc-string 3)
- (indent 2))
+ (indent 2)
+ (autoload-macro expand))
`(defun ,name ,@(cl--transform-lambda (cons args body) name)))
;;;###autoload
@@ -416,7 +417,8 @@ cl-iter-defun
[&optional ("interactive" interactive)]
def-body))
(doc-string 3)
- (indent 2))
+ (indent 2)
+ (autoload-macro expand))
(require 'generator)
`(iter-defun ,name ,@(cl--transform-lambda (cons args body) name)))
@@ -475,7 +477,8 @@ cl-defmacro
(declare (debug
(&define name cl-macro-list cl-declarations-or-string def-body))
(doc-string 3)
- (indent 2))
+ (indent 2)
+ (autoload-macro expand))
`(defmacro ,name ,@(cl--transform-lambda (cons args body) name)))
(def-edebug-elem-spec 'cl-lambda-expr
@@ -3011,7 +3014,8 @@ cl-defstruct
sexp])]
[&optional stringp]
;; All the above is for the following def-form.
- &rest &or symbolp (symbolp &optional def-form &rest sexp))))
+ &rest &or symbolp (symbolp &optional def-form &rest sexp)))
+ (autoload-macro expand))
(let* ((name (if (consp struct) (car struct) struct))
(warning nil)
(opts (cdr-safe struct))
diff --git a/lisp/emacs-lisp/easy-mmode.el b/lisp/emacs-lisp/easy-mmode.el
index f53db48f0b7..0e428abd0b7 100644
--- a/lisp/emacs-lisp/easy-mmode.el
+++ b/lisp/emacs-lisp/easy-mmode.el
@@ -222,10 +222,11 @@ define-minor-mode
(indent defun)
(debug (&define name string-or-null-p
[&optional [¬ keywordp] sexp
- &optional [¬ keywordp] sexp
- &optional [¬ keywordp] sexp]
+ &optional [¬ keywordp] sexp
+ &optional [¬ keywordp] sexp]
[&rest [keywordp sexp]]
- def-body)))
+ def-body))
+ (autoload-macro expand))
(let* ((last-message (make-symbol "last-message"))
(mode-name (symbol-name mode))
@@ -487,7 +488,7 @@ define-globalized-minor-mode
on if the hook has explicitly disabled it.
\(fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)"
- (declare (doc-string 2) (indent defun))
+ (declare (doc-string 2) (indent defun) (autoload-macro expand))
(let* ((global-mode-name (symbol-name global-mode))
(mode-name (symbol-name mode))
(pretty-name (easy-mmode-pretty-mode-name mode))
diff --git a/lisp/emacs-lisp/generator.el b/lisp/emacs-lisp/generator.el
index 344e11e245e..59784ff51e8 100644
--- a/lisp/emacs-lisp/generator.el
+++ b/lisp/emacs-lisp/generator.el
@@ -675,7 +675,8 @@ iter-defun
of values. Callers can retrieve each value using `iter-next'."
(declare (indent defun)
(debug (&define name lambda-list lambda-doc &rest sexp))
- (doc-string 3))
+ (doc-string 3)
+ (autoload-macro expand))
(cl-assert lexical-binding)
(let* ((parsed-body (macroexp-parse-body body))
(declarations (car parsed-body))
diff --git a/lisp/emacs-lisp/inline.el b/lisp/emacs-lisp/inline.el
index c015e2b57d0..5acd6f17ee2 100644
--- a/lisp/emacs-lisp/inline.el
+++ b/lisp/emacs-lisp/inline.el
@@ -135,7 +135,7 @@ define-inline
This is like `defmacro', but has several advantages.
See Info node `(elisp)Defining Functions' for more details."
;; FIXME: How can this work with CL arglists?
- (declare (indent defun) (debug defun) (doc-string 3))
+ (declare (indent defun) (debug defun) (doc-string 3) (autoload-macro expand))
(let ((doc (if (stringp (car-safe body)) (list (pop body))))
(declares (if (eq (car-safe (car-safe body)) 'declare) (pop body)))
(cm-name (intern (format "%s--inliner" name)))
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..f9c6f82c31f 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -142,12 +142,29 @@ loaddefs-generate--shorten-autoload
3)
form))
+;; TODO: some macros can be removed from this list once they
+;; (declare ((autoload-macro expand) ...)
+(defconst loaddefs--defining-macros
+ '(define-skeleton define-derived-mode
+ define-compilation-mode define-generic-mode
+ easy-mmode-define-global-mode define-global-minor-mode
+ define-globalized-minor-mode
+ easy-mmode-define-minor-mode define-minor-mode
+ cl-defun defun* cl-defmacro defmacro*
+ define-overloadable-function
+ transient-define-prefix transient-define-suffix
+ transient-define-infix transient-define-argument))
+
+(defvar loaddefs--load-error-files nil)
(defun loaddefs-generate--make-autoload (form file &optional expansion)
"Turn FORM into an autoload or defvar for source file FILE.
Returns nil if FORM is not a special autoload form (i.e. a function definition
or macro definition or a defcustom).
If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently.
+
+Note that macros can request expansion by including `(autoload-macro
+expand)' among their `declare' forms."
(let ((car (car-safe form)) expand)
(cond
((and expansion (eq car 'defalias))
@@ -191,35 +208,40 @@ loaddefs-generate--make-autoload
(setq form (copy-sequence form))
(setcdr (memq :autoload-end form) nil))
(let ((exps (delq nil (mapcar (lambda (form)
- (loaddefs-generate--make-autoload
- form file expansion))
+ (unless (eq form :autoload-end)
+ (loaddefs-generate--make-autoload
+ form file expansion)))
(cdr form)))))
(when exps (cons 'progn exps)))))
- ;; For complex cases, try again on the macro-expansion.
- ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
- define-globalized-minor-mode defun defmacro
- easy-mmode-define-minor-mode define-minor-mode
- define-inline cl-defun cl-defmacro cl-defgeneric
- cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
- (macrop car)
- (setq expand (let ((load-true-file-name file)
- (load-file-name file))
- (macroexpand form)))
- (memq (car expand) '(progn prog1 defalias)))
+ ;; For macros which request it, try again on their expansion.
+ ((progn
+ ;; If the car is an unknown symbol, we load the file first to
+ ;; give packages a chance to define their macros.
+ (unless (or (not (symbolp car)) (fboundp car)
+ ;; Special cases handled below
+ (memq car loaddefs--defining-macros)
+ (memq car '(defclass defcustom deftheme defgroup nil))
+ (assoc file load-history)
+ (member file loaddefs--load-error-files))
+ (let ((load-path (cons (file-name-directory file) load-path)))
+ (message "loaddefs-gen: loading file %s (for %s)" file car)
+ (condition-case e (load file)
+ (error
+ (push file loaddefs--load-error-files) ; do not attempt again
+ (warn "loaddefs-gen: load error\n\t%s" e)))))
+ (and (macrop car)
+ (eq 'expand (function-get car 'autoload-macro))
+ (setq expand (let ((load-true-file-name file)
+ (load-file-name file))
+ (macroexpand form)))
+ (not (eq car (car expand)))))
;; Recurse on the expansion.
(loaddefs-generate--make-autoload expand file 'expansion))
- ;; For special function-like operators, use the `autoload' function.
- ((memq car '(define-skeleton define-derived-mode
- define-compilation-mode define-generic-mode
- easy-mmode-define-global-mode define-global-minor-mode
- define-globalized-minor-mode
- easy-mmode-define-minor-mode define-minor-mode
- cl-defun defun* cl-defmacro defmacro*
- define-overloadable-function
- transient-define-prefix transient-define-suffix
- transient-define-infix transient-define-argument))
+ ;; For known special macros which define functions, use `autoload'
+ ;; directly.
+ ((memq car loaddefs--defining-macros)
(let* ((macrop (memq car '(defmacro cl-defmacro defmacro*)))
(name (nth 1 form))
(args (pcase car
diff --git a/lisp/emacs-lisp/pcase.el b/lisp/emacs-lisp/pcase.el
index 73554fd66fd..ab8edf1f569 100644
--- a/lisp/emacs-lisp/pcase.el
+++ b/lisp/emacs-lisp/pcase.el
@@ -537,7 +537,7 @@ pcase-defmacro
By convention, DOC should use \"EXPVAL\" to stand
for the result of evaluating EXP (first arg to `pcase').
\n(fn NAME ARGS [DOC] &rest BODY...)"
- (declare (indent 2) (debug defun) (doc-string 3))
+ (declare (indent 2) (debug defun) (doc-string 3) (autoload-macro expand))
;; Add the function via `fsym', so that an autoload cookie placed
;; on a pcase-defmacro will cause the macro to be loaded on demand.
(let ((fsym (intern (format "%s--pcase-macroexpander" name)))
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 02 Aug 2025 06:53:02 GMT) Full text and rfc822 format available.Message #131 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 02 Aug 2025 09:52:32 +0300
> From: "J.D. Smith" <jdtsmith <at> gmail.com>
> Date: Fri, 01 Aug 2025 18:37:28 -0400
>
> JD Smith <jdtsmith <at> gmail.com> writes:
>
> Stefan, Eli, any comments on the documentation (or other) parts of this
> patch?
Some, see below.
> --- a/doc/lispref/functions.texi
> +++ b/doc/lispref/functions.texi
> @@ -2633,6 +2633,11 @@ Declare Form
> This is typically used for macros, though it works for functions too.
> @xref{Indenting Macros}.
>
> +@item (autoload-macro @var{value})
> +This is used when defining a macro. If @var{value} is @samp{expand},
> +any calls to the macro which follow an autoload comment will first be
> +expanded during generation of the autoloads. @xref{Autoload}.
This can benefit from an index entry.
> --- a/doc/lispref/loading.texi
> +++ b/doc/lispref/loading.texi
> @@ -710,8 +710,31 @@ Autoload
> If you write a function definition with an unusual macro that is not
> one of the known and recognized function definition methods, use of an
> ordinary magic autoload comment would copy the whole definition into
> -@code{loaddefs.el}. That is not desirable. You can put the desired
> -@code{autoload} call into @code{loaddefs.el} instead by writing this:
> +@code{loaddefs.el}. That is often not desirable. In this case, you can
^^^^^^^^^^^^
It is not clear what case is alluded here as "this case". Suggest
instead to say something like "If you don't want the macro copied to
@file{loaddefs.el}, you can ..." Or to make a simple sentence like
"If this is not desirable, you can ...".
Btw, please use @file{loaddefs.el}, as @code is wrong when referring
to files.
> +use the special declare form @code{(autoload-macro expand)} in your
> +macro definition (@pxref{Declare Form}). This indicates to the autoload
> +system that calls to your macro following @samp{;;;###autoload} should
> +be expanded, similar to how the special functions listed above are
> +handled. For example, a macro which wraps @code{define-minor-mode} can
> +request expansion, so that proper @code{autoload} calls for the function
> +it defines are generated.
This text left me wondering how this is different from not using
autoload-macro. When you describe some measure to avoid an
undesirable outcome, please always be sure to say what will happen
under if that measure is taken, as opposed to when it isn't. Here,
you say what autoload-macro does, but saying that the macro is
expanded doesn't help, because it is expanded even if autoload-macro
is not used (right?).
> +The symbol @code{:autoload-end} can be used in the expansion to prevent
> +including unwanted forms in the autoload output. Its presence causes
> +any further elements within the list where it appears to be silently
> +skipped.
An index entry is missing here. Also, the description is complex
enough to call for an example as a means to clarify it.
> +Note that, if a symbol in the car of an autoloaded form is not defined
^^^
"car" should be in @code.
> +during autoload generation, the file in which the associated
> +@samp{;;;###autoload} appears will itself be loaded, to give the macro a
> +chance to be defined. Packages which use special loading mechanisms,
> +including requiring their own @samp{foo-autoload.el} file, should
> +therefore gracefully handle load errors, since these files may not yet
> +exist during autoload generation. This can be done, e.g., by setting
> +the @var{no-error} argument of @code{require} non-@code{nil}).
Please add here a cross-reference to where 'require' is defined.
> --- a/lisp/emacs-lisp/byte-run.el
> +++ b/lisp/emacs-lisp/byte-run.el
> @@ -271,6 +271,11 @@ 'byte-run--set-debug
> (list 'put (list 'quote name)
> ''edebug-form-spec (list 'quote spec)))))
>
> +(defalias 'byte-run--set-autoload-macro
> + #'(lambda (name _args spec)
> + (list 'function-put (list 'quote name)
> + ''autoload-macro (list 'quote spec))))
Can we have a comment here explaining the reason for the use of
autoload-macro?
> (defalias 'byte-run--set-no-font-lock-keyword
> #'(lambda (name _args val)
> (list 'function-put (list 'quote name)
> @@ -350,8 +355,10 @@ macro-declarations-alist
> (cons
> (list 'debug #'byte-run--set-debug)
> (cons
> - (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
> - defun-declarations-alist))
> + (list 'autoload-macro #'byte-run--set-autoload-macro)
And maybe here as well?
> --- a/lisp/emacs-lisp/easy-mmode.el
> +++ b/lisp/emacs-lisp/easy-mmode.el
> @@ -222,10 +222,11 @@ define-minor-mode
> (indent defun)
> (debug (&define name string-or-null-p
> [&optional [¬ keywordp] sexp
> - &optional [¬ keywordp] sexp
> - &optional [¬ keywordp] sexp]
> + &optional [¬ keywordp] sexp
> + &optional [¬ keywordp] sexp]
> [&rest [keywordp sexp]]
> - def-body)))
> + def-body))
> + (autoload-macro expand))
And here.
> @@ -487,7 +488,7 @@ define-globalized-minor-mode
> on if the hook has explicitly disabled it.
>
> \(fn GLOBAL-MODE MODE TURN-ON [KEY VALUE]... BODY...)"
> - (declare (doc-string 2) (indent defun))
> + (declare (doc-string 2) (indent defun) (autoload-macro expand))
And here.
> --- a/lisp/emacs-lisp/inline.el
> +++ b/lisp/emacs-lisp/inline.el
> @@ -135,7 +135,7 @@ define-inline
> This is like `defmacro', but has several advantages.
> See Info node `(elisp)Defining Functions' for more details."
> ;; FIXME: How can this work with CL arglists?
> - (declare (indent defun) (debug defun) (doc-string 3))
> + (declare (indent defun) (debug defun) (doc-string 3) (autoload-macro expand))
And here.
> --- a/lisp/emacs-lisp/pcase.el
> +++ b/lisp/emacs-lisp/pcase.el
> @@ -537,7 +537,7 @@ pcase-defmacro
> By convention, DOC should use \"EXPVAL\" to stand
> for the result of evaluating EXP (first arg to `pcase').
> \n(fn NAME ARGS [DOC] &rest BODY...)"
> - (declare (indent 2) (debug defun) (doc-string 3))
> + (declare (indent 2) (debug defun) (doc-string 3) (autoload-macro expand))
And here.
> --- a/lisp/emacs-lisp/loaddefs-gen.el
> +++ b/lisp/emacs-lisp/loaddefs-gen.el
> @@ -142,12 +142,29 @@ loaddefs-generate--shorten-autoload
> 3)
> form))
>
> +;; TODO: some macros can be removed from this list once they
> +;; (declare ((autoload-macro expand) ...)
> +(defconst loaddefs--defining-macros
> + '(define-skeleton define-derived-mode
> + define-compilation-mode define-generic-mode
> + easy-mmode-define-global-mode define-global-minor-mode
> + define-globalized-minor-mode
> + easy-mmode-define-minor-mode define-minor-mode
> + cl-defun defun* cl-defmacro defmacro*
> + define-overloadable-function
> + transient-define-prefix transient-define-suffix
> + transient-define-infix transient-define-argument))
Please explain in a comment which macros should appear the above list,
and why.
Thanks.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 02 Aug 2025 17:21:02 GMT) Full text and rfc822 format available.Message #134 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 02 Aug 2025 19:20:00 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes: Hi, > Hmm, I guess `loaddefs-gen' was introduced in emacs 29. But I think you > don't actually need that `require' at all. Maybe try a simple: > > (unless (alist-get 'autoload-macro macro-declarations-alist) > (push '(autoload-macro ignore) macro-declarations-alist)) That works with Emacs 28, if I put these lines before the tramp--with-startup macro declaration. I cannot test with the master branch, because the autoload-macro patch doesn't seem to be there yet. Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 03 Aug 2025 18:17:02 GMT) Full text and rfc822 format available.Message #137 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Eli Zaretskii <eliz <at> gnu.org> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 03 Aug 2025 14:16:05 -0400
[Message part 1 (text/plain, inline)]
Eli Zaretskii <eliz <at> gnu.org> writes: > JD Smith <jdtsmith <at> gmail.com> writes: > > Stefan, Eli, any comments on the documentation (or other) parts of this > patch? > >> Some, see below. Thanks. > Can we have a comment here explaining the reason for the use of > autoload-macro? > And maybe here as well? > And here. x 4 I'm not sure what kind of comment you mean. None of the other declare forms are commented, either on definition or on use. Are you thinking of something simple, like "request macro-expansion during autoload generation"? Current patch attached.
[autoload-expand_7.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 03 Aug 2025 18:19:01 GMT) Full text and rfc822 format available.Message #140 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 03 Aug 2025 14:18:22 -0400
[Message part 1 (text/plain, inline)]
Michael Albinus <michael.albinus <at> gmx.de> writes: > "J.D. Smith" <jdtsmith <at> gmail.com> writes: > > Hi, > >> Hmm, I guess `loaddefs-gen' was introduced in emacs 29. But I think you >> don't actually need that `require' at all. Maybe try a simple: >> >> (unless (alist-get 'autoload-macro macro-declarations-alist) >> (push '(autoload-macro ignore) macro-declarations-alist)) > > That works with Emacs 28, if I put these lines before the > tramp--with-startup macro declaration. > > I cannot test with the master branch, because the autoload-macro patch > doesn't seem to be there yet. Great. I will push it soon and then you can test, or feel free to try this patch in the meantime. Thanks.
[autoload-expand_7.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 03 Aug 2025 19:04:01 GMT) Full text and rfc822 format available.Message #143 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 03 Aug 2025 22:03:00 +0300
> From: "J.D. Smith" <jdtsmith <at> gmail.com> > Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca > Date: Sun, 03 Aug 2025 14:16:05 -0400 > > > Can we have a comment here explaining the reason for the use of > > autoload-macro? > > And maybe here as well? > > And here. x 4 > > I'm not sure what kind of comment you mean. To explain why do we use this declaration there. > None of the other declare > forms are commented, either on definition or on use. This one is much more subtle than the others. > Are you thinking of something simple, like "request macro-expansion > during autoload generation"? No, I mean _why_ we want to request that in those places and not the others.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 03 Aug 2025 22:54:02 GMT) Full text and rfc822 format available.Message #146 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Eli Zaretskii <eliz <at> gnu.org> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 03 Aug 2025 18:53:34 -0400
[Message part 1 (text/plain, inline)]
Eli Zaretskii <eliz <at> gnu.org> writes: >> From: "J.D. Smith" <jdtsmith <at> gmail.com> >> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca >> Date: Sun, 03 Aug 2025 14:16:05 -0400 >> >> > Can we have a comment here explaining the reason for the use of >> > autoload-macro? >> > And maybe here as well? >> > And here. x 4 >> >> I'm not sure what kind of comment you mean. > > To explain why do we use this declaration there. > >> Are you thinking of something simple, like "request macro-expansion >> during autoload generation"? > > No, I mean _why_ we want to request that in those places and not the > others. The short but unsatisfying answer is to preserve the hard-coded list of macros to expand, which this functionality replaces and extends. The former explanation in `loaddefs-generate--make-autoload' was quite terse: ;; For complex cases, try again on the macro-expansion. I've tried to inject a short explanation everywhere this declaration is used. I left the `byte-run--set-autoload-macro' alone, since that's just a generic property setter, and values other than `expand' may be used in the future. Latest patch below. Meta-comments: ============== Though outside of the scope of this patch, a few mildly bothersome attributes remain, which I leave here for posterity. 1. It's somewhat arbitrary which macros request expansion Eventually, I think for all builtin macros, the question should be asked: "Will calls to this macro likely be marked ;;;###autoload"? If the answer is yes, the macro should declare `autoload-macro=expand', so that these calls get recursively expanded down to their most basic forms — `defalias', `put', and so on. 2. Some autoloaded macro expansions are not "autoload-hygienic". Some macros currently /rely/ on skipping directly to `(autoload name ...)' generation via the hard-coded "short-cut" list (see `loaddefs--defining-macros'). I.e. they do not expect to be expanded, and are therefore not autoload-hygenic in their expansion (e.g. neglecting to use `:autoload-end' to elide unwanted forms, for example). This is confusing and prone to bugs (e.g. the bug discovered in this thread related to the MODES arg). 3. Some macros' autoload treatment depend on whether they are defined at autoload-generation time. Such macros are both on the short-cut list /and/ request expansion. So they are expanded, but only if they are defined at the time autoload generation occurs. Most are, but some, e.g. cl-macs, are not. So they go through the "short-cut" instead. IMO, these macros should either have their expansion cleaned up for autoload hygiene (if necessary) and be pre-loaded during autoload generation, or they should just not request expansion at all.
[autoload-expand_8.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 03 Aug 2025 23:18:02 GMT) Full text and rfc822 format available.Message #149 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: Eli Zaretskii <eliz <at> gnu.org>, 78995 <at> debbugs.gnu.org, philipk <at> posteo.net Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 03 Aug 2025 19:17:12 -0400
> The short but unsatisfying answer is to preserve the hard-coded list
> of macros to expand, which this functionality replaces and extends.
> The former explanation in `loaddefs-generate--make-autoload' was quite
> terse:
>
> ;; For complex cases, try again on the macro-expansion.
The reason is always that calls to these macros don't want to be
*preloaded* but only to be *autoloaded*. By expanding the macro,
`autoload.el` (aka `loaddefs-gen.el`) gets to see the underlying
`defalias` and friends necessary to figure out *how* to autoload it.
In practice this means it's used for macros which define functions (or
macros).
> I've tried to inject a short explanation everywhere this declaration
> is used. I left the `byte-run--set-autoload-macro' alone, since
> that's just a generic property setter, and values other than `expand'
> may be used in the future. Latest patch below.
>
> Meta-comments:
> ==============
>
> Though outside of the scope of this patch, a few mildly bothersome
> attributes remain, which I leave here for posterity.
>
> 1. It's somewhat arbitrary which macros request expansion
>
> Eventually, I think for all builtin macros, the question should be
> asked: "Will calls to this macro likely be marked ;;;###autoload"?
> If the answer is yes, the macro should declare
> `autoload-macro=expand', so that these calls get recursively expanded
> down to their most basic forms — `defalias', `put', and so on.
It's a judgment call: will it be more common to *preload* or to
*autoload*?
> 2. Some autoloaded macro expansions are not "autoload-hygienic".
> 3. Some macros' autoload treatment depend on whether they are defined
> at autoload-generation time.
I hope your patch will help us fix those issues.
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Mon, 04 Aug 2025 12:53:02 GMT) Full text and rfc822 format available.Message #152 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Mon, 04 Aug 2025 15:51:47 +0300
> From: "J.D. Smith" <jdtsmith <at> gmail.com>
> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca
> Date: Sun, 03 Aug 2025 18:53:34 -0400
>
> > No, I mean _why_ we want to request that in those places and not the
> > others.
>
> The short but unsatisfying answer is to preserve the hard-coded list of
> macros to expand, which this functionality replaces and extends. The
> former explanation in `loaddefs-generate--make-autoload' was quite
> terse:
>
> ;; For complex cases, try again on the macro-expansion.
>
> I've tried to inject a short explanation everywhere this declaration is
> used. I left the `byte-run--set-autoload-macro' alone, since that's
> just a generic property setter, and values other than `expand' may be
> used in the future. Latest patch below.
This is much better (thanks!), but I think we need to improve it some
more by saying in the manual that this declaration is an alternative
to hard-coding macros in loaddefs--defining-macros in loaddefs-gen.el.
> +the final form (@code{(put my-mode 'hidden-prop nil)}) will not be
Long forms like the above should be wrapped in @w{..}, to prevent them
from being broken between two lines.
> +copied into the autoload file.
"autoloads file", I guess?
> +Note that, if a symbol in the @code{car} of an autoloaded form is found
> +to be undefined during autoload generation, the file in which the
> +associated @samp{;;;###autoload} appears will itself be loaded, to give
> +the macro a chance to be defined. Packages which use special loading
> +mechanisms, including loading their own @samp{foo-autoload.el} file,
"foo" should be in @var, since it isn't a literal "foo". I'd even use
@file{@var{package}-loaddefs.el} instead (note the use of @file).
> +;; The following macros are known to define functions, and are treated
> +;; specially when encountered during autoload generation, translating
> +;; calls to them into appropriate (autoload function ...) forms.
> +;; TODO: some macros can be removed from this list once they
> +;; (declare ((autoload-macro expand) ...)
I'd prefer to say explicitly that autoload-macro is the alternative
used so as to keep this list as short as possible.
Thanks.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Mon, 04 Aug 2025 15:22:01 GMT) Full text and rfc822 format available.Message #155 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Eli Zaretskii <eliz <at> gnu.org> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Mon, 04 Aug 2025 11:21:39 -0400
[Message part 1 (text/plain, inline)]
Eli Zaretskii <eliz <at> gnu.org> writes: >> From: "J.D. Smith" <jdtsmith <at> gmail.com> >> Cc: philipk <at> posteo.net, 78995 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca >> Date: Sun, 03 Aug 2025 18:53:34 -0400 >> >> > No, I mean _why_ we want to request that in those places and not the >> > others. >> >> The short but unsatisfying answer is to preserve the hard-coded list of >> macros to expand, which this functionality replaces and extends. The >> former explanation in `loaddefs-generate--make-autoload' was quite >> terse: >> >> ;; For complex cases, try again on the macro-expansion. >> >> I've tried to inject a short explanation everywhere this declaration is >> used. I left the `byte-run--set-autoload-macro' alone, since that's >> just a generic property setter, and values other than `expand' may be >> used in the future. Latest patch below. > > This is much better (thanks!), but I think we need to improve it some > more by saying in the manual that this declaration is an alternative > to hard-coding macros in loaddefs--defining-macros in loaddefs-gen.el. OK, I have implemented these doc suggestions and a few other minor improvements in the attached. I've also tried to make a somewhat unclear situation w.r.t. the hard-coded macro lists more clear in the comments. `loaddefs--defining-macros' is the "shortcut" list for turning a function-defining macro directly into an (autoload ..) form, whereas (autoload-macro expand) replaced a /different/ list of macros: those to be expanded. So there where 2 main hard-coded lists, and now there is 1. The lists are closely related, because usually a macro expands into one or more of the shortcut forms. I conjecture (but have not verified), that ALL hard-coded lists of macros can eventually be removed, save some very basic primitives like defalias (and defmacro?). Whether that's desirable is less clear.
[autoload-expand_9.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Tue, 05 Aug 2025 07:13:02 GMT) Full text and rfc822 format available.Message #158 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: Eli Zaretskii <eliz <at> gnu.org>, 78995 <at> debbugs.gnu.org, philipk <at> posteo.net, monnier <at> iro.umontreal.ca Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Tue, 05 Aug 2025 09:12:42 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes: Hi, > OK, I have implemented these doc suggestions and a few other minor > improvements in the attached. I've also tried to make a somewhat > unclear situation w.r.t. the hard-coded macro lists more clear in the > comments. Pls don't forget the Tramp change in your patch. Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Tue, 05 Aug 2025 13:14:02 GMT) Full text and rfc822 format available.Message #161 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: Eli Zaretskii <eliz <at> gnu.org>, 78995 <at> debbugs.gnu.org, philipk <at> posteo.net, monnier <at> iro.umontreal.ca Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Tue, 05 Aug 2025 09:13:37 -0400
Michael Albinus <michael.albinus <at> gmx.de> writes:
>
> Pls don't forget the Tramp change in your patch.
There are definitely builtin packages that can/should be updated as a
result of this new functionality, e.g. transient. But I don't think
those changes should be part of this patch, because it's sensible to get
the basic capability in place for people to test before adapting other
packages.
That said, TRAMP may be a special case because of the "chicken-and-egg"
problem represented by tramp-loaddefs.el, and the error generated by
attempting to load that file before it exists. For now just setting
NOERROR when requiring tramp-loaddefs might be the right approach.
Stefan, what do you think a good approach is for packages which want to
start declaring autoload-macro expand in some of their macros, but
/also/ want to support earlier versions of emacs? I can't think of an
approach other than each such package including:
(unless (alist-get 'autoload-macro macro-declarations-alist)
(push '(autoload-macro ignore) macro-declarations-alist))
somewhere.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 10 Aug 2025 21:24:02 GMT) Full text and rfc822 format available.Message #164 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: Eli Zaretskii <eliz <at> gnu.org>, 78995 <at> debbugs.gnu.org, philipk <at> posteo.net Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 10 Aug 2025 17:23:29 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes: >> 1. It's somewhat arbitrary which macros request expansion >> >> Eventually, I think for all builtin macros, the question should be >> asked: "Will calls to this macro likely be marked ;;;###autoload"? >> If the answer is yes, the macro should declare >> `autoload-macro=expand', so that these calls get recursively expanded >> down to their most basic forms — `defalias', `put', and so on. > > It's a judgment call: will it be more common to *preload* or to > *autoload*? I suppose an argument for preloading a macro's definition and copying in ;;;###autoload calls to it is if it 1) is very long and involved and cannot be reduced to a small number of expanded forms, and 2) is autoloaded many times, thus bloating the autoload file. The only other scenario I can imagine is if it relies on information that is not yet known at compile time, but which is known reliably at load time (network address on emacs start?). But I'm not aware of any such builtin macros. What other preload uses am I missing? Unless there are any more thoughts on this patch I'll give it a last look over and apply to master. I'll patch TRAMP in a separate commit.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 10 Aug 2025 21:42:02 GMT) Full text and rfc822 format available.Message #167 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 10 Aug 2025 17:41:37 -0400
[Message part 1 (text/plain, inline)]
"J.D. Smith" <jdtsmith <at> gmail.com> writes:
> Michael Albinus <michael.albinus <at> gmx.de> writes:
>
>> "J.D. Smith" <jdtsmith <at> gmail.com> writes:
>>
>> Hi,
>>
>>> Hmm, I guess `loaddefs-gen' was introduced in emacs 29. But I think you
>>> don't actually need that `require' at all. Maybe try a simple:
>>>
>>> (unless (alist-get 'autoload-macro macro-declarations-alist)
>>> (push '(autoload-macro ignore) macro-declarations-alist))
>>
>> That works with Emacs 28, if I put these lines before the
>> tramp--with-startup macro declaration.
>>
>> I cannot test with the master branch, because the autoload-macro patch
>> doesn't seem to be there yet.
>
> Great. I will push it soon and then you can test, or feel free to try
> this patch in the meantime.
Here's an updated patch to try with TRAMP, to see if it works as
expected in both old and new Emacs versions. Note that there are many
places where TRAMP does something more advanced, like:
;;;###tramp-autoload
(when (featurep 'dbusbind)
(tramp--with-startup
(dolist (method tramp-gvfs-methods)
...
so I think you'll need to keep autoloading the macro definition too.
[tramp-autoload-macro-expand.patch (text/x-patch, attachment)]
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 17 Aug 2025 09:51:01 GMT) Full text and rfc822 format available.Message #170 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Michael Albinus <michael.albinus <at> gmx.de> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 17 Aug 2025 11:50:10 +0200
"J.D. Smith" <jdtsmith <at> gmail.com> writes: Hi, > Here's an updated patch to try with TRAMP, to see if it works as > expected in both old and new Emacs versions. Note that there are many > places where TRAMP does something more advanced, like: > > ;;;###tramp-autoload > (when (featurep 'dbusbind) > (tramp--with-startup > (dolist (method tramp-gvfs-methods) > ... > > so I think you'll need to keep autoloading the macro definition too. I've checked it with both Emacs 28 and 31, and it still works w/o problems. So pls push it to master. Best regards, Michael.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sun, 17 Aug 2025 15:22:01 GMT) Full text and rfc822 format available.Message #173 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Michael Albinus <michael.albinus <at> gmx.de> Cc: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sun, 17 Aug 2025 11:21:48 -0400
Michael Albinus <michael.albinus <at> gmx.de> writes: > "J.D. Smith" <jdtsmith <at> gmail.com> writes: > > Hi, > >> Here's an updated patch to try with TRAMP, to see if it works as >> expected in both old and new Emacs versions. Note that there are many >> places where TRAMP does something more advanced, like: >> >> ;;;###tramp-autoload >> (when (featurep 'dbusbind) >> (tramp--with-startup >> (dolist (method tramp-gvfs-methods) >> ... >> >> so I think you'll need to keep autoloading the macro definition too. > > I've checked it with both Emacs 28 and 31, and it still works w/o > problems. So pls push it to master. Great, thanks. Done, and closing.
"J.D. Smith" <jdtsmith <at> gmail.com>
to control <at> debbugs.gnu.org.
(Sun, 17 Aug 2025 15:23:01 GMT) Full text and rfc822 format available.bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 20 Aug 2025 11:19:01 GMT) Full text and rfc822 format available.Message #178 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Changes to `loaddefs.el` for define-minor-mode Date: Wed, 20 Aug 2025 07:17:45 -0400
AFAICT, the new code seems to change the contents of `loaddefs.el` for
`define-minor-mode`, e.g.:
% grep '(.*global-auto-revert-mode' **/*defs*.el
lisp/ldefs-boot.el:(defvar global-auto-revert-mode nil "\
lisp/ldefs-boot.el:(custom-autoload 'global-auto-revert-mode "autorevert" nil)
lisp/ldefs-boot.el:(autoload 'global-auto-revert-mode "autorevert" "\
lisp/ldefs-boot.el:evaluate `(default-value \\='global-auto-revert-mode)'.
lisp/loaddefs.el:(autoload 'global-auto-revert-mode "autorevert" "\
%
As you can see the `global-auto-revert-mode` variable ends up
not preloaded. I'm generally opposed to preloading variables, but the
change introduces a backward incompatibility with packages which presume
they can take the value of `global-auto-revert-mode` without resorting
to any `boundp` check, such as Magit.
So I suggest the patch below,
Stefan
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 9e91a11204d..e0b3859136c 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -155,7 +155,7 @@ loaddefs-generate--shorten-autoload
;; employing :autoload-end to omit unneeded forms).
(defconst loaddefs--defining-macros
'( define-skeleton define-derived-mode define-compilation-mode
- define-generic-mode define-globalized-minor-mode define-minor-mode
+ define-generic-mode define-globalized-minor-mode
cl-defun defun* cl-defmacro defmacro* define-overloadable-function
transient-define-prefix transient-define-suffix transient-define-infix
transient-define-argument transient-define-group
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 20 Aug 2025 14:20:02 GMT) Full text and rfc822 format available.Message #181 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 20 Aug 2025 10:19:33 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
> AFAICT, the new code seems to change the contents of `loaddefs.el` for
> `define-minor-mode`, e.g.:
>
> % grep '(.*global-auto-revert-mode' **/*defs*.el
> lisp/ldefs-boot.el:(defvar global-auto-revert-mode nil "\
> lisp/ldefs-boot.el:(custom-autoload 'global-auto-revert-mode "autorevert" nil)
> lisp/ldefs-boot.el:(autoload 'global-auto-revert-mode "autorevert" "\
> lisp/ldefs-boot.el:evaluate `(default-value \\='global-auto-revert-mode)'.
> lisp/loaddefs.el:(autoload 'global-auto-revert-mode "autorevert" "\
> %
>
> As you can see the `global-auto-revert-mode` variable ends up
> not preloaded. I'm generally opposed to preloading variables, but the
> change introduces a backward incompatibility with packages which presume
> they can take the value of `global-auto-revert-mode` without resorting
> to any `boundp` check, such as Magit.
I don't see this:
% grep '(.*global-auto-revert-mode' **/*defs*.el
ldefs-boot.el:(defvar global-auto-revert-mode nil "\
ldefs-boot.el:(custom-autoload 'global-auto-revert-mode "autorevert" nil)
ldefs-boot.el:(autoload 'global-auto-revert-mode "autorevert" "\
ldefs-boot.el:evaluate `(default-value \\='global-auto-revert-mode)'.
loaddefs_old.el:(defvar global-auto-revert-mode nil "\
loaddefs_old.el:(custom-autoload 'global-auto-revert-mode "autorevert" nil)
loaddefs_old.el:(autoload 'global-auto-revert-mode "autorevert" "\
loaddefs_old.el:evaluate `(default-value \\='global-auto-revert-mode)'.
loaddefs.el:(defvar global-auto-revert-mode nil "\
loaddefs.el:(custom-autoload 'global-auto-revert-mode "autorevert" nil)
loaddefs.el:(autoload 'global-auto-revert-mode "autorevert" "\
loaddefs.el:evaluate `(default-value \\='global-auto-revert-mode)'.
(N.B.: `loaddefs_old.el' was a backup taken prior to the improvements we
made here.)
To me, this looks like you don't have autoload-macro declarations
exposed while building loaddefs (recall this used to be a hard-coded
list), so the `define-minor-mode' does not get expanded correctly.
What I'd expect/hope is that `define-minor-mode' is already defined via
autoload when autoload generation occurs, and has declared itself
`(autoload-macro expand)`, in `loaddefs.el'.
If you `make bootstrap', do you see:
% grep '(.*define-minor-mode.*autoload-macro' **/*defs*.el
lisp/loaddefs.el:(function-put 'define-minor-mode 'autoload-macro 'expand)
This is essential, because the emacs which is used to generate
loaddefs.el incorporates this file (right? see below).
Summarizing, during generation of loaddefs.el, we need either:
1. easy-mmode to have been actually loaded, so its macros are properly
defined along with their declarations, or
2. the autoload file used in the version of emacs which is building the
autoload file to contain autoload-macro expand.
Chicken-and-eggy again, though it does work for me. I just don't have a
clear enough picture of the build sequence. Questions I have:
1. What does `ldefs-boot.el' do? Is it just a "backup" of
`loaddefs.el', or is it ever actually loaded?
2. Which version of emacs is used to generate loaddefs.el? I think the
"final version" is.
3. Is loaddefs.el compiled into the version of emacs that generates
loaddefs.el?
3. If so, how is that circular dependency broken? Iteration? Is that
what `ldefs-boot' is about.
4. Is there a list somewhere of packages that are always explicitly
loaded for every emacs? Is easy-mmode on that list?
JD
>
> So I suggest the patch below,
>
>
> Stefan
>
>
> diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
> index 9e91a11204d..e0b3859136c 100644
> --- a/lisp/emacs-lisp/loaddefs-gen.el
> +++ b/lisp/emacs-lisp/loaddefs-gen.el
> @@ -155,7 +155,7 @@ loaddefs-generate--shorten-autoload
> ;; employing :autoload-end to omit unneeded forms).
> (defconst loaddefs--defining-macros
> '( define-skeleton define-derived-mode define-compilation-mode
> - define-generic-mode define-globalized-minor-mode define-minor-mode
> + define-generic-mode define-globalized-minor-mode
> cl-defun defun* cl-defmacro defmacro* define-overloadable-function
> transient-define-prefix transient-define-suffix transient-define-infix
> transient-define-argument transient-define-group
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Wed, 20 Aug 2025 16:25:02 GMT) Full text and rfc822 format available.Message #184 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Wed, 20 Aug 2025 12:24:39 -0400
> I don't see this:
Hmm... I'll have to investigate further, then.
Might be some issue on my side.
Stefan
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Fri, 22 Aug 2025 14:46:01 GMT) Full text and rfc822 format available.Message #187 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: "J.D. Smith" <jdtsmith <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Fri, 22 Aug 2025 10:44:57 -0400
Stefan Monnier <monnier <at> iro.umontreal.ca> writes: >> I don't see this: > > Hmm... I'll have to investigate further, then. > Might be some issue on my side. Did the problem go away with a `make bootstrap'? I'd feel more confident if I understood better how the Emacs which produces loaddefs.el itself includes autoloads.
bug-gnu-emacs <at> gnu.org:bug#78995; Package emacs.
(Sat, 23 Aug 2025 06:45:02 GMT) Full text and rfc822 format available.Message #190 received at 78995 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: "J.D. Smith" <jdtsmith <at> gmail.com> Cc: 78995 <at> debbugs.gnu.org Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros Date: Sat, 23 Aug 2025 02:44:19 -0400
>> Hmm... I'll have to investigate further, then.
>> Might be some issue on my side.
> Did the problem go away with a `make bootstrap'?
Yup. And I wasn't able to reproduce the problem afterwards, so I don't
know where it came from.
> I'd feel more confident if I understood better how the Emacs which
> produces loaddefs.el itself includes autoloads.
🙂
Stefan
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.