GNU bug report logs - #75812
29.2.50; Support native-compiling package-install from batch Emacs

Previous Next

To reply to this bug, email your comments to 75812 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Spencer Baugh <sbaugh <at> janestreet.com>
To: bug-gnu-emacs <at> gnu.org
Cc: app-emacs-dev <at> janestreet.com, Stefan Kangas <stefankangas <at> gmail.com>
Subject: 29.2.50; Support native-compiling package-install from batch Emacs
Date: Fri, 24 Jan 2025 12:54:08 -0500

Currently, setting package-native-compile to t will make package-install
asynchronously native-compile the package.  This works great for
interactive usage.

However, this is unsuitable for a batch Emacs running package-install.
"emacs -batch --eval (package-install something)" will exit once the
package is installed, but before the asynchronous native compilation
completes; the native compilation will just be abandoned.  So,
currently, there's no straightforward way to native compile a package
from a batch Emacs.

Running package-install from a batch Emacs is useful as a way to update
or pre-install packages in scripts, without requiring user interaction.
This is widely used by package managers like Nix or dpkg to install
Emacs packages as OS packages.  In my specific case, it's used to
install and byte-compile common packages into site-lisp.  In all these
cases, it would be nice to be able to native-compile the package at
installation time and include the .eln files in the package directory.

A straightforward solution would be for package-install-file to support
synchronously native-compiling the installed package, such that after
package-install finishes running, the native compilation is complete.
Perhaps with a new value of package-native-compile?

Message #8 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Spencer Baugh <sbaugh <at> janestreet.com>
Cc: app-emacs-dev <at> janestreet.com, 75812 <at> debbugs.gnu.org
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install from
 batch Emacs
Date: Fri, 24 Jan 2025 16:03:35 -0600

severity 75812 wishlist
tags 75812 + confirmed
thanks

Spencer Baugh <sbaugh <at> janestreet.com> writes:

> Currently, setting package-native-compile to t will make package-install
> asynchronously native-compile the package.  This works great for
> interactive usage.
>
> However, this is unsuitable for a batch Emacs running package-install.
> "emacs -batch --eval (package-install something)" will exit once the
> package is installed, but before the asynchronous native compilation
> completes; the native compilation will just be abandoned.  So,
> currently, there's no straightforward way to native compile a package
> from a batch Emacs.
>
> Running package-install from a batch Emacs is useful as a way to update
> or pre-install packages in scripts, without requiring user interaction.
> This is widely used by package managers like Nix or dpkg to install
> Emacs packages as OS packages.  In my specific case, it's used to
> install and byte-compile common packages into site-lisp.  In all these
> cases, it would be nice to be able to native-compile the package at
> installation time and include the .eln files in the package directory.
>
> A straightforward solution would be for package-install-file to support
> synchronously native-compiling the installed package, such that after
> package-install finishes running, the native compilation is complete.
> Perhaps with a new value of package-native-compile?

Sounds good to me, thanks.  We'll take patches implementing this.

Message #15 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Spencer Baugh <sbaugh <at> janestreet.com>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: app-emacs-dev <at> janestreet.com, 75812 <at> debbugs.gnu.org
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Fri, 24 Jan 2025 17:35:50 -0500

Stefan Kangas <stefankangas <at> gmail.com> writes:
> Spencer Baugh <sbaugh <at> janestreet.com> writes:
>
>> Currently, setting package-native-compile to t will make package-install
>> asynchronously native-compile the package.  This works great for
>> interactive usage.
>>
>> However, this is unsuitable for a batch Emacs running package-install.
>> "emacs -batch --eval (package-install something)" will exit once the
>> package is installed, but before the asynchronous native compilation
>> completes; the native compilation will just be abandoned.  So,
>> currently, there's no straightforward way to native compile a package
>> from a batch Emacs.
>>
>> Running package-install from a batch Emacs is useful as a way to update
>> or pre-install packages in scripts, without requiring user interaction.
>> This is widely used by package managers like Nix or dpkg to install
>> Emacs packages as OS packages.  In my specific case, it's used to
>> install and byte-compile common packages into site-lisp.  In all these
>> cases, it would be nice to be able to native-compile the package at
>> installation time and include the .eln files in the package directory.
>>
>> A straightforward solution would be for package-install-file to support
>> synchronously native-compiling the installed package, such that after
>> package-install finishes running, the native compilation is complete.
>> Perhaps with a new value of package-native-compile?
>
> Sounds good to me, thanks.  We'll take patches implementing this.

I'm happy to implement this, I'm just a bit lost about where to start.
I tried the obvious "iterate over *.el files in the package directory,
calling native-compile", but that failed on various files in various
packages.

There's a lot of code in package--compile and byte-recompile-directory,
which does a bunch of stuff like ignoring certain files.  I guess I need
that code.  Ultimately, package--compile calls byte-compile-file, so if
I could get byte-compile-file to also emit native-compiled files, that
would work.  Is there a way to do that?

byte+native-compile looks like a relevant variable, but is that the
right direction to head in?

Message #18 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Spencer Baugh <sbaugh <at> janestreet.com>,
 Andrea Corallo <acorallo <at> gnu.org>
Cc: app-emacs-dev <at> janestreet.com, stefankangas <at> gmail.com, 75812 <at> debbugs.gnu.org
Subject: Re: bug#75812: 29.2.50;
 Support native-compiling package-install from batch Emacs
Date: Sat, 25 Jan 2025 09:57:22 +0200

> Cc: app-emacs-dev <at> janestreet.com, 75812 <at> debbugs.gnu.org
> Date: Fri, 24 Jan 2025 17:35:50 -0500
> From:  Spencer Baugh via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> > Sounds good to me, thanks.  We'll take patches implementing this.
> 
> I'm happy to implement this, I'm just a bit lost about where to start.
> I tried the obvious "iterate over *.el files in the package directory,
> calling native-compile", but that failed on various files in various
> packages.

Why not simply wait till all the async subprocesses compiling to
native code finish and exit?  Bonus points for displaying some kind of
status or progress messages while waiting, so the user is informed why
the wait.

Andrea, is there a better way you can suggest?

Message #21 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <acorallo <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Spencer Baugh <sbaugh <at> janestreet.com>, app-emacs-dev <at> janestreet.com,
 stefankangas <at> gmail.com, 75812 <at> debbugs.gnu.org
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Thu, 30 Jan 2025 10:29:23 -0500

Eli Zaretskii <eliz <at> gnu.org> writes:

>> Cc: app-emacs-dev <at> janestreet.com, 75812 <at> debbugs.gnu.org
>> Date: Fri, 24 Jan 2025 17:35:50 -0500
>> From:  Spencer Baugh via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>> 
>> > Sounds good to me, thanks.  We'll take patches implementing this.
>> 
>> I'm happy to implement this, I'm just a bit lost about where to start.
>> I tried the obvious "iterate over *.el files in the package directory,
>> calling native-compile", but that failed on various files in various
>> packages.
>
> Why not simply wait till all the async subprocesses compiling to
> native code finish and exit?  Bonus points for displaying some kind of
> status or progress messages while waiting, so the user is informed why
> the wait.
>
> Andrea, is there a better way you can suggest?

Not ATM, I agree having some way to visualize/report the status of the
compilation queue would be nice.

Message #24 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Ship Mints <shipmints <at> gmail.com>
To: Andrea Corallo <acorallo <at> gnu.org>
Cc: Spencer Baugh <sbaugh <at> janestreet.com>, Eli Zaretskii <eliz <at> gnu.org>,
 app-emacs-dev <at> janestreet.com, stefankangas <at> gmail.com, 75812 <at> debbugs.gnu.org
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install from
 batch Emacs
Date: Thu, 30 Jan 2025 10:37:39 -0500

[Message part 1 (text/plain, inline)]

If it is of interest, I wrote the below to visualize the compilation queue.
It appears when something is enqueued for compilation, and disappears when
the queue is empty.

(when (native-comp-available-p)
  (defvar my:comp--async-running-func)
  (if (version< emacs-version "30")
      (progn
        (declare-function comp--async-runnings nil)
        (require 'comp)
        (setq my:comp--async-running-func #'comp-async-runnings))
    (require 'comp-run)
    (setq my:comp--async-running-func #'comp--async-runnings))
  (defun my/native-comp-mode-line-lighter ()
    "mode-line lighter for `my/native-comp-mode-line-mode'."
    (when comp-files-queue
      (propertize
       (format " [◴%s:%d/%d]" comp-native-version-dir (length
comp-files-queue) (funcall my:comp--async-running-func))
       'face (list :height 0.75))))

  (define-minor-mode my/native-comp-mode-line-mode
    "Display the native compilation queue depth during compiles."
    :global t
    :group 'mode-line
    :lighter (:eval (my/native-comp-mode-line-lighter)))
  (when my:mode-line-native-comp
    (my/native-comp-mode-line-mode)))

On Thu, Jan 30, 2025 at 10:30 AM Andrea Corallo <acorallo <at> gnu.org> wrote:

> Eli Zaretskii <eliz <at> gnu.org> writes:
>
> >> Cc: app-emacs-dev <at> janestreet.com, 75812 <at> debbugs.gnu.org
> >> Date: Fri, 24 Jan 2025 17:35:50 -0500
> >> From:  Spencer Baugh via "Bug reports for GNU Emacs,
> >>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> >>
> >> > Sounds good to me, thanks.  We'll take patches implementing this.
> >>
> >> I'm happy to implement this, I'm just a bit lost about where to start.
> >> I tried the obvious "iterate over *.el files in the package directory,
> >> calling native-compile", but that failed on various files in various
> >> packages.
> >
> > Why not simply wait till all the async subprocesses compiling to
> > native code finish and exit?  Bonus points for displaying some kind of
> > status or progress messages while waiting, so the user is informed why
> > the wait.
> >
> > Andrea, is there a better way you can suggest?
>
> Not ATM, I agree having some way to visualize/report the status of the
> compilation queue would be nice.
>
>
>
>

[Message part 2 (text/html, inline)]

Message #27 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Spencer Baugh <sbaugh <at> janestreet.com>
To: Ship Mints <shipmints <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Andrea Corallo <acorallo <at> gnu.org>,
 stefankangas <at> gmail.com, 75812 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Wed, 23 Apr 2025 12:26:44 -0400

Conveniently there's already a hook, native-comp-async-all-done-hook,
which gets called when the async native compilation is done.  I just
need to synchronously wait for that hook to be called.

So what about something like this?

(defun native-comp-async-wait-for-all-done ()
  (unless (zerop (comp--async-runnings))
    (let* ((native-comp-finished-pipe (make-pipe-process :name "native-comp-finished"))
           (native-comp-async-all-done-hook (list (lambda () (delete-process native-comp-finished-pipe)))))
      (accept-process-output native-comp-finished-pipe))))

The use of a pipe process is weird, but I don't know of a better way to
wait for a hook to be called.

---------------------

There's another related issue for the packaging use case I mentioned in
my original email: The .eln files are placed in ~/.emacs.d/eln-cache/ by
default when doing this native compilation.  That complicates building
and installing these packages independently.  Is there a way to put the
.eln files for a package in that individual package directory instead?
The same directory as the .el and .elc files.  The packages are
recompiled for each new Emacs version anyway, so I think this should be
safe from a versioning perspective.

Message #30 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Spencer Baugh <sbaugh <at> janestreet.com>
Cc: acorallo <at> gnu.org, stefankangas <at> gmail.com, shipmints <at> gmail.com,
 75812 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Thu, 24 Apr 2025 08:28:43 +0300

> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Cc: Andrea Corallo <acorallo <at> gnu.org>,  Eli Zaretskii <eliz <at> gnu.org>,
>    app-emacs-dev <at> janestreet.com,  stefankangas <at> gmail.com,
>    75812 <at> debbugs.gnu.org
> Date: Wed, 23 Apr 2025 12:26:44 -0400
> 
> 
> Conveniently there's already a hook, native-comp-async-all-done-hook,
> which gets called when the async native compilation is done.  I just
> need to synchronously wait for that hook to be called.
> 
> So what about something like this?
> 
> (defun native-comp-async-wait-for-all-done ()
>   (unless (zerop (comp--async-runnings))
>     (let* ((native-comp-finished-pipe (make-pipe-process :name "native-comp-finished"))
>            (native-comp-async-all-done-hook (list (lambda () (delete-process native-comp-finished-pipe)))))
>       (accept-process-output native-comp-finished-pipe))))
> 
> The use of a pipe process is weird, but I don't know of a better way to
> wait for a hook to be called.

That will not work on Windows, so let's not.

I don't understand why you need all that complexity.  Why not loop
calling comp--async-runnings as long as it returns non-nil, each time
sleeping for, say, 0.5 sec?  Or why not install a
native-comp-async-all-done-hook that will notice when compilations are
done and allow the batch job to exit?  All you need is a means to wait
until all the async compilations are finished, no?

Or what am I missing?

> There's another related issue for the packaging use case I mentioned in
> my original email: The .eln files are placed in ~/.emacs.d/eln-cache/ by
> default when doing this native compilation.  That complicates building
> and installing these packages independently.  Is there a way to put the
> .eln files for a package in that individual package directory instead?

You need to tweak native-comp-eln-load-path, e.g. prepend to the
default-value list the directory where you want the *.eln files to be
written.  But you are well advised to leave the other directories in
the list alone.

Message #33 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Spencer Baugh <sbaugh <at> janestreet.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: app-emacs-dev <at> janestreet.com, acorallo <at> gnu.org, stefankangas <at> gmail.com,
 75812 <at> debbugs.gnu.org, shipmints <at> gmail.com
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Thu, 24 Apr 2025 17:25:47 -0400

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Spencer Baugh <sbaugh <at> janestreet.com>
>> Cc: Andrea Corallo <acorallo <at> gnu.org>,  Eli Zaretskii <eliz <at> gnu.org>,
>>    app-emacs-dev <at> janestreet.com,  stefankangas <at> gmail.com,
>>    75812 <at> debbugs.gnu.org
>> Date: Wed, 23 Apr 2025 12:26:44 -0400
>> 
>> 
>> Conveniently there's already a hook, native-comp-async-all-done-hook,
>> which gets called when the async native compilation is done.  I just
>> need to synchronously wait for that hook to be called.
>> 
>> So what about something like this?
>> 
>> (defun native-comp-async-wait-for-all-done ()
>>   (unless (zerop (comp--async-runnings))
>>     (let* ((native-comp-finished-pipe (make-pipe-process :name "native-comp-finished"))
>>            (native-comp-async-all-done-hook (list (lambda () (delete-process native-comp-finished-pipe)))))
>>       (accept-process-output native-comp-finished-pipe))))
>> 
>> The use of a pipe process is weird, but I don't know of a better way to
>> wait for a hook to be called.
>
> That will not work on Windows, so let's not.

Fair.

> I don't understand why you need all that complexity.  Why not loop
> calling comp--async-runnings as long as it returns non-nil, each time
> sleeping for, say, 0.5 sec?

Sleeping for 0.5 sec means that building a package will take an extra
.25 seconds on average, which adds up over hundreds of packages.
Sleeping for 0.0001 sec wastes CPU, which also adds up over hundreds of
packages.

> Or why not install a native-comp-async-all-done-hook that will notice
> when compilations are done and allow the batch job to exit?  All you
> need is a means to wait until all the async compilations are finished,
> no?

How do you propose to do that?  I guess I could call kill-emacs from the
hook, but I have more code I want to run after the native compilation is
finished, and I can't move it all into the hook.

>> There's another related issue for the packaging use case I mentioned in
>> my original email: The .eln files are placed in ~/.emacs.d/eln-cache/ by
>> default when doing this native compilation.  That complicates building
>> and installing these packages independently.  Is there a way to put the
>> .eln files for a package in that individual package directory instead?
>
> You need to tweak native-comp-eln-load-path, e.g. prepend to the
> default-value list the directory where you want the *.eln files to be
> written.  But you are well advised to leave the other directories in
> the list alone.

Makes sense.

Does that mean that, if I want the .eln files to be loaded from the
individual package directory, I have to put every single package
directory on native-comp-eln-load-path at runtime?  That seems annoying,
and probably slow.

Is there a way to get an .eln file to be loaded from the directory the
.elc file was found in?  Maybe that just happens by default before
searching native-comp-eln-load-path?

Message #36 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Spencer Baugh <sbaugh <at> janestreet.com>
Cc: app-emacs-dev <at> janestreet.com, acorallo <at> gnu.org, stefankangas <at> gmail.com,
 75812 <at> debbugs.gnu.org, shipmints <at> gmail.com
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Fri, 25 Apr 2025 10:12:27 +0300

> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Cc: acorallo <at> gnu.org,  stefankangas <at> gmail.com,  shipmints <at> gmail.com,
>    75812 <at> debbugs.gnu.org,  app-emacs-dev <at> janestreet.com
> Date: Thu, 24 Apr 2025 17:25:47 -0400
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > I don't understand why you need all that complexity.  Why not loop
> > calling comp--async-runnings as long as it returns non-nil, each time
> > sleeping for, say, 0.5 sec?
> 
> Sleeping for 0.5 sec means that building a package will take an extra
> .25 seconds on average, which adds up over hundreds of packages.
> Sleeping for 0.0001 sec wastes CPU, which also adds up over hundreds of
> packages.

So make it 0.02 sec instead.  0.5 sec was just an example; no need to
take a suggestion "ad absurdum", you know.

> > Or why not install a native-comp-async-all-done-hook that will notice
> > when compilations are done and allow the batch job to exit?  All you
> > need is a means to wait until all the async compilations are finished,
> > no?
> 
> How do you propose to do that?  I guess I could call kill-emacs from the
> hook, but I have more code I want to run after the native compilation is
> finished, and I can't move it all into the hook.

Then don't call kill-emacs from the hook; instead, use 'throw', or run
the rest of the code from the hook directly.

> >> There's another related issue for the packaging use case I mentioned in
> >> my original email: The .eln files are placed in ~/.emacs.d/eln-cache/ by
> >> default when doing this native compilation.  That complicates building
> >> and installing these packages independently.  Is there a way to put the
> >> .eln files for a package in that individual package directory instead?
> >
> > You need to tweak native-comp-eln-load-path, e.g. prepend to the
> > default-value list the directory where you want the *.eln files to be
> > written.  But you are well advised to leave the other directories in
> > the list alone.
> 
> Makes sense.
> 
> Does that mean that, if I want the .eln files to be loaded from the
> individual package directory, I have to put every single package
> directory on native-comp-eln-load-path at runtime?  That seems annoying,
> and probably slow.

I'm confused.  It was you who asked how to avoid placing the *.eln
files in ~/.emacs.d/eln-cache/, but now you say it causes problems?
"If it hurts, don't do that."  I don't understand why the default
location is problematic in the first place, and you haven't provided
any reasons.

And why are you talking about a separate directory for each package?
One could envision using a single directory for the *.eln files of all
the packages, for example.  Whether that is reasonable again depends
on your goals and specific cases, which you haven't described.

> Is there a way to get an .eln file to be loaded from the directory the
> .elc file was found in?

No, and it doesn't make sense to do that, because the *.eln files are
not in the directories mentioned in native-comp-eln-load-path, they
are in versioned subdirectories of those directories, and those
subdirectories must be verified by computing the hashes of the
corresponding *.el files, to make sure we are not by chance loading
incompatible machine code.

Message #39 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Spencer Baugh <sbaugh <at> janestreet.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: acorallo <at> gnu.org, app-emacs-dev <at> janestreet.com, stefankangas <at> gmail.com,
 75812 <at> debbugs.gnu.org, shipmints <at> gmail.com
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Tue, 29 Apr 2025 14:05:38 -0400

Eli Zaretskii <eliz <at> gnu.org> writes:
>> > Or why not install a native-comp-async-all-done-hook that will notice
>> > when compilations are done and allow the batch job to exit?  All you
>> > need is a means to wait until all the async compilations are finished,
>> > no?
>> 
>> How do you propose to do that?  I guess I could call kill-emacs from the
>> hook, but I have more code I want to run after the native compilation is
>> finished, and I can't move it all into the hook.
>
> Then don't call kill-emacs from the hook; instead, use 'throw', or run
> the rest of the code from the hook directly.

So a function like this?

(defun native-comp-async-wait-for-all-done ()
  (unless (zerop (comp--async-runnings))
    (catch 'native-comp-done
      (let ((native-comp-async-all-done-hook (list (lambda () (throw 'native-comp-done)))))
        (while (sit-for 999))))))

Could we add that?

>> >> There's another related issue for the packaging use case I mentioned in
>> >> my original email: The .eln files are placed in ~/.emacs.d/eln-cache/ by
>> >> default when doing this native compilation.  That complicates building
>> >> and installing these packages independently.  Is there a way to put the
>> >> .eln files for a package in that individual package directory instead?
>> >
>> > You need to tweak native-comp-eln-load-path, e.g. prepend to the
>> > default-value list the directory where you want the *.eln files to be
>> > written.  But you are well advised to leave the other directories in
>> > the list alone.
>> 
>> Makes sense.
>> 
>> Does that mean that, if I want the .eln files to be loaded from the
>> individual package directory, I have to put every single package
>> directory on native-comp-eln-load-path at runtime?  That seems annoying,
>> and probably slow.
>
> I'm confused.  It was you who asked how to avoid placing the *.eln
> files in ~/.emacs.d/eln-cache/, but now you say it causes problems?
> "If it hurts, don't do that."  I don't understand why the default
> location is problematic in the first place, and you haven't provided
> any reasons.
>
> And why are you talking about a separate directory for each package?
> One could envision using a single directory for the *.eln files of all
> the packages, for example.  Whether that is reasonable again depends
> on your goals and specific cases, which you haven't described.

Yes, it's a matter of compiling each package separately in parallel,
which is annoying to do if they all write files to a single directory as
a side-effect.

But, I think what I could do is compile each package with a different
directory prepended to native-comp-eln-load-path, and then once all the
packages are compiled, merge those directories into one omnibus
directory, which is bundled with my installed Emacs.

Would it be reasonable to merge those directories into the last entry of
native-comp-eln-load-path?  (e.g. /usr/lib/emacs/30.1.50/native-lisp)

Or will it break something for there to be more than just bundled elisp
in that directory?

>> Is there a way to get an .eln file to be loaded from the directory the
>> .elc file was found in?
>
> No, and it doesn't make sense to do that, because the *.eln files are
> not in the directories mentioned in native-comp-eln-load-path, they
> are in versioned subdirectories of those directories, and those
> subdirectories must be verified by computing the hashes of the
> corresponding *.el files, to make sure we are not by chance loading
> incompatible machine code.

That's fair, makes sense.

Message #42 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Spencer Baugh <sbaugh <at> janestreet.com>
Cc: acorallo <at> gnu.org, app-emacs-dev <at> janestreet.com, stefankangas <at> gmail.com,
 75812 <at> debbugs.gnu.org, shipmints <at> gmail.com
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Tue, 29 Apr 2025 21:45:44 +0300

> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Cc: app-emacs-dev <at> janestreet.com,  acorallo <at> gnu.org,
>    stefankangas <at> gmail.com,  75812 <at> debbugs.gnu.org,  shipmints <at> gmail.com
> Date: Tue, 29 Apr 2025 14:05:38 -0400
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> >> > Or why not install a native-comp-async-all-done-hook that will notice
> >> > when compilations are done and allow the batch job to exit?  All you
> >> > need is a means to wait until all the async compilations are finished,
> >> > no?
> >> 
> >> How do you propose to do that?  I guess I could call kill-emacs from the
> >> hook, but I have more code I want to run after the native compilation is
> >> finished, and I can't move it all into the hook.
> >
> > Then don't call kill-emacs from the hook; instead, use 'throw', or run
> > the rest of the code from the hook directly.
> 
> So a function like this?
> 
> (defun native-comp-async-wait-for-all-done ()
>   (unless (zerop (comp--async-runnings))
>     (catch 'native-comp-done
>       (let ((native-comp-async-all-done-hook (list (lambda () (throw 'native-comp-done)))))
>         (while (sit-for 999))))))
> 
> Could we add that?

Why do we need to add it?  Can't you just use it in your code?

> Yes, it's a matter of compiling each package separately in parallel,
> which is annoying to do if they all write files to a single directory as
> a side-effect.
> 
> But, I think what I could do is compile each package with a different
> directory prepended to native-comp-eln-load-path, and then once all the
> packages are compiled, merge those directories into one omnibus
> directory, which is bundled with my installed Emacs.

I still don't understand what you are trying to achieve, but the above
should work.

> Would it be reasonable to merge those directories into the last entry of
> native-comp-eln-load-path?  (e.g. /usr/lib/emacs/30.1.50/native-lisp)

No!  The last directory is in many cases not writable by users, and it
includes preloaded files, see the doc string of the variable.  The
last directory should be left alone.  Instead, add another directory
to the default value, between the user's eln-cache and the last
member, and have your local packages' *.eln files there.

Message #45 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Spencer Baugh <sbaugh <at> janestreet.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: shipmints <at> gmail.com, acorallo <at> gnu.org, stefankangas <at> gmail.com,
 75812 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Tue, 29 Apr 2025 15:28:11 -0400

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Spencer Baugh <sbaugh <at> janestreet.com>
>> Cc: app-emacs-dev <at> janestreet.com,  acorallo <at> gnu.org,
>>    stefankangas <at> gmail.com,  75812 <at> debbugs.gnu.org,  shipmints <at> gmail.com
>> Date: Tue, 29 Apr 2025 14:05:38 -0400
>> 
>> Eli Zaretskii <eliz <at> gnu.org> writes:
>> >> > Or why not install a native-comp-async-all-done-hook that will notice
>> >> > when compilations are done and allow the batch job to exit?  All you
>> >> > need is a means to wait until all the async compilations are finished,
>> >> > no?
>> >> 
>> >> How do you propose to do that?  I guess I could call kill-emacs from the
>> >> hook, but I have more code I want to run after the native compilation is
>> >> finished, and I can't move it all into the hook.
>> >
>> > Then don't call kill-emacs from the hook; instead, use 'throw', or run
>> > the rest of the code from the hook directly.
>> 
>> So a function like this?
>> 
>> (defun native-comp-async-wait-for-all-done ()
>>   (unless (zerop (comp--async-runnings))
>>     (catch 'native-comp-done
>>       (let ((native-comp-async-all-done-hook (list (lambda () (throw 'native-comp-done)))))
>>         (while (sit-for 999))))))
>> 
>> Could we add that?
>
> Why do we need to add it?  Can't you just use it in your code?

Since others in this thread have said they would like to be able to wait
monitor the completion of native compilation.  It seems like it would be
good to have a high quality implementation be part of Emacs.  (and
frankly, I don't think throwing out of a hook as a means of control flow
is a good idea, and I'd like to do something better than that)

>> Yes, it's a matter of compiling each package separately in parallel,
>> which is annoying to do if they all write files to a single directory as
>> a side-effect.
>> 
>> But, I think what I could do is compile each package with a different
>> directory prepended to native-comp-eln-load-path, and then once all the
>> packages are compiled, merge those directories into one omnibus
>> directory, which is bundled with my installed Emacs.
>
> I still don't understand what you are trying to achieve, but the above
> should work.

Great, thank you for confirming!

>> Would it be reasonable to merge those directories into the last entry of
>> native-comp-eln-load-path?  (e.g. /usr/lib/emacs/30.1.50/native-lisp)
>
> No!  The last directory is in many cases not writable by users, and it
> includes preloaded files, see the doc string of the variable.  The
> last directory should be left alone.  Instead, add another directory
> to the default value, between the user's eln-cache and the last
> member, and have your local packages' *.eln files there.

I'm doing this package precompilation at Emacs build time, and the last
directory is writable at that time.  After installation, of course, it
is not writable, but that doesn't matter since only these precompiled
package .elns are going to be written there.

In general, I have site-lisp that is byte-compiled and native-compiled,
and it seems like the right place to write .eln files for site-lisp is
the last directory on native-comp-eln-load-path.  After all, site-lisp
is effectively bundled with Emacs, set up at install time just like core
Elisp.  Is there some issue with doing that?

Message #48 received at 75812 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Spencer Baugh <sbaugh <at> janestreet.com>
Cc: shipmints <at> gmail.com, acorallo <at> gnu.org, stefankangas <at> gmail.com,
 75812 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
Subject: Re: bug#75812: 29.2.50; Support native-compiling package-install
 from batch Emacs
Date: Wed, 30 Apr 2025 08:49:21 +0300

> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Cc: acorallo <at> gnu.org,  app-emacs-dev <at> janestreet.com,
>    stefankangas <at> gmail.com,  75812 <at> debbugs.gnu.org,  shipmints <at> gmail.com
> Date: Tue, 29 Apr 2025 15:28:11 -0400
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> (defun native-comp-async-wait-for-all-done ()
> >>   (unless (zerop (comp--async-runnings))
> >>     (catch 'native-comp-done
> >>       (let ((native-comp-async-all-done-hook (list (lambda () (throw 'native-comp-done)))))
> >>         (while (sit-for 999))))))
> >> 
> >> Could we add that?
> >
> > Why do we need to add it?  Can't you just use it in your code?
> 
> Since others in this thread have said they would like to be able to wait
> monitor the completion of native compilation.  It seems like it would be
> good to have a high quality implementation be part of Emacs.  (and
> frankly, I don't think throwing out of a hook as a means of control flow
> is a good idea, and I'd like to do something better than that)

I'm not sure we can come up with a general enough function for this
purpose, because what the caller wants to do while waiting will differ
from application to application.  In the above case the caller just
waits, but that's not necessarily what everyone will want to do.

So if you want to propose a general-purpose function, we need more
thorough design and a better abstraction, IMO.  Without that, the
above is just a thin wrapper around an existing hook.

> >> Would it be reasonable to merge those directories into the last entry of
> >> native-comp-eln-load-path?  (e.g. /usr/lib/emacs/30.1.50/native-lisp)
> >
> > No!  The last directory is in many cases not writable by users, and it
> > includes preloaded files, see the doc string of the variable.  The
> > last directory should be left alone.  Instead, add another directory
> > to the default value, between the user's eln-cache and the last
> > member, and have your local packages' *.eln files there.
> 
> I'm doing this package precompilation at Emacs build time, and the last
> directory is writable at that time.  After installation, of course, it
> is not writable, but that doesn't matter since only these precompiled
> package .elns are going to be written there.
> 
> In general, I have site-lisp that is byte-compiled and native-compiled,
> and it seems like the right place to write .eln files for site-lisp is
> the last directory on native-comp-eln-load-path.  After all, site-lisp
> is effectively bundled with Emacs, set up at install time just like core
> Elisp.  Is there some issue with doing that?

You are invalidating the assumption that the last directory is special
in that it holds the results of compiling the Lisp files that are
bundled with Emacs.  You asked me whether this can be done, and my
warm recommendation is not to do that.  You can do whatever you want
regardless, of course, but then please don't come back reporting what
you think is a bug which was caused by that, because placing other
*.eln files in that directory is not supported.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.