GNU bug report logs -
#78783
30.1.50; AOT-compiling site-lisp during the Emacs build doesn't work
Previous Next
To reply to this bug, email your comments to 78783 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Fri, 13 Jun 2025 13:35:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Spencer Baugh <sbaugh <at> janestreet.com>:
New bug report received and forwarded. Copy sent to
bug-gnu-emacs <at> gnu.org.
(Fri, 13 Jun 2025 13:35:03 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Native compilation of site-lisp directories during the Emacs build
(e.g. --with-native-compilation=aot) produces .eln files which are
unused when installed.
The reason is that the absolute file name of the site-lisp directory
differs between build-time and install-time, and therefore
comp-el-to-eln-rel-filename computes a different path hash for the files
contained in site-lisp at build-time and at install-time.
The same issue would affect AOT native compilation of all Lisp in Emacs,
except that there's a workaround built into comp-el-to-eln-rel-filename,
as explained by this comment:
As installing .eln files compiled during the build changes their
absolute path we need an hashing mechanism that is not sensitive
to that. For this we replace if match PATH_DUMPLOADSEARCH or
*PATH_REL_LOADSEARCH with '//' before computing the hash.
In other words, file names containing e.g. "30.1.50/lisp", like
"/usr/local/share/emacs/30.1.50/lisp/foo.el", are transformed to
"//foo.el" before the path hash is computed.
I suggest the same should be done for file names containing
"30.1.50/site-lisp". That would resolve the problem and allow site-lisp
to actually be AOT-compiled.
In GNU Emacs 30.1.50 (build 19, x86_64-pc-linux-gnu, X toolkit, cairo
version 1.15.12, Xaw scroll bars) of 2025-05-26 built on
igm-qws-u22796a
Repository revision: 1743ab3d48bedae367ca6a1deb3faf8f902e7d5c
Repository branch: emacs-30
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Rocky Linux 8.10 (Green Obsidian)
Configured using:
'configure --config-cache --with-x-toolkit=lucid --without-gpm
--without-gconf --without-selinux --without-imagemagick --with-modules
--with-gif=no --with-cairo --with-rsvg --without-compress-install
--with-tree-sitter --with-native-compilation=aot
PKG_CONFIG_PATH=/usr/local/home/garnish/libtree-sitter/0.22.6-1/lib/pkgconfig/'
Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSYSTEMD
LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP
SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11 XDBE XIM
XINPUT2 XPM LUCID ZLIB
Important settings:
value of $LANG: en_US.UTF-8
locale-coding-system: utf-8-unix
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Fri, 13 Jun 2025 15:00:03 GMT)
Full text and
rfc822 format available.
Message #8 received at 78783 <at> debbugs.gnu.org (full text, mbox):
severity 78783 wishlist
thanks
> Cc: Andrea Corallo <acorallo <at> gnu.org>, app-emacs-dev <at> janestreet.com
> Date: Fri, 13 Jun 2025 09:34:10 -0400
> From: Spencer Baugh via "Bug reports for GNU Emacs,
> the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>
>
> Native compilation of site-lisp directories during the Emacs build
> (e.g. --with-native-compilation=aot) produces .eln files which are
> unused when installed.
Why are you compiling files in site-lisp during the build? This is
not supported, and I see no reason why we should extend our build
processes to support it.
Why cannot you compile site-lisp files after Emacs is built and
installed?
Severity set to 'wishlist' from 'normal'
Request was from
Eli Zaretskii <eliz <at> gnu.org>
to
control <at> debbugs.gnu.org.
(Fri, 13 Jun 2025 15:00:03 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Fri, 13 Jun 2025 15:42:04 GMT)
Full text and
rfc822 format available.
Message #13 received at 78783 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
> severity 78783 wishlist
> thanks
>
>> Cc: Andrea Corallo <acorallo <at> gnu.org>, app-emacs-dev <at> janestreet.com
>> Date: Fri, 13 Jun 2025 09:34:10 -0400
>> From: Spencer Baugh via "Bug reports for GNU Emacs,
>> the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>>
>>
>> Native compilation of site-lisp directories during the Emacs build
>> (e.g. --with-native-compilation=aot) produces .eln files which are
>> unused when installed.
>
> Why are you compiling files in site-lisp during the build? This is
> not supported, and I see no reason why we should extend our build
> processes to support it.
>
> Why cannot you compile site-lisp files after Emacs is built and
> installed?
If a site-lisp file is built (whether as part of Emacs or not) and
subsequently installed into the site-lisp directory, its file name will
change. That changes the path_hash computed by
comp-el-to-eln-rel-filename which will cause this issue.
This is a general problem with AOT-compiling files which are
subsequently installed somewhere else. It applies both for core Emacs
files and for site-lisp.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Fri, 13 Jun 2025 17:43:02 GMT)
Full text and
rfc822 format available.
Message #16 received at 78783 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
> Date: Fri, 13 Jun 2025 11:41:27 -0400
>
> Eli Zaretskii <eliz <at> gnu.org> writes:
> > severity 78783 wishlist
> > thanks
> >
> >> Cc: Andrea Corallo <acorallo <at> gnu.org>, app-emacs-dev <at> janestreet.com
> >> Date: Fri, 13 Jun 2025 09:34:10 -0400
> >> From: Spencer Baugh via "Bug reports for GNU Emacs,
> >> the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> >>
> >>
> >> Native compilation of site-lisp directories during the Emacs build
> >> (e.g. --with-native-compilation=aot) produces .eln files which are
> >> unused when installed.
> >
> > Why are you compiling files in site-lisp during the build? This is
> > not supported, and I see no reason why we should extend our build
> > processes to support it.
> >
> > Why cannot you compile site-lisp files after Emacs is built and
> > installed?
>
> If a site-lisp file is built (whether as part of Emacs or not) and
> subsequently installed into the site-lisp directory, its file name will
> change. That changes the path_hash computed by
> comp-el-to-eln-rel-filename which will cause this issue.
>
> This is a general problem with AOT-compiling files which are
> subsequently installed somewhere else. It applies both for core Emacs
> files and for site-lisp.
No, core Emacs files are okay and supported.
It's compiling site-lisp files in this way is not supported. And you
still haven't answered my question: why do you need that?
The supported way of compiling site-lisp files is after they are
installed in site-lisp.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Tue, 08 Jul 2025 18:52:03 GMT)
Full text and
rfc822 format available.
Message #19 received at 78783 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
>> From: Spencer Baugh <sbaugh <at> janestreet.com>
>> Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
>> Date: Fri, 13 Jun 2025 11:41:27 -0400
>>
>> Eli Zaretskii <eliz <at> gnu.org> writes:
>> > severity 78783 wishlist
>> > thanks
>> >
>> >> Cc: Andrea Corallo <acorallo <at> gnu.org>, app-emacs-dev <at> janestreet.com
>> >> Date: Fri, 13 Jun 2025 09:34:10 -0400
>> >> From: Spencer Baugh via "Bug reports for GNU Emacs,
>> >> the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>> >>
>> >>
>> >> Native compilation of site-lisp directories during the Emacs build
>> >> (e.g. --with-native-compilation=aot) produces .eln files which are
>> >> unused when installed.
>> >
>> > Why are you compiling files in site-lisp during the build? This is
>> > not supported, and I see no reason why we should extend our build
>> > processes to support it.
>> >
>> > Why cannot you compile site-lisp files after Emacs is built and
>> > installed?
>>
>> If a site-lisp file is built (whether as part of Emacs or not) and
>> subsequently installed into the site-lisp directory, its file name will
>> change. That changes the path_hash computed by
>> comp-el-to-eln-rel-filename which will cause this issue.
>>
>> This is a general problem with AOT-compiling files which are
>> subsequently installed somewhere else. It applies both for core Emacs
>> files and for site-lisp.
>
> No, core Emacs files are okay and supported.
>
> It's compiling site-lisp files in this way is not supported. And you
> still haven't answered my question: why do you need that?
>
> The supported way of compiling site-lisp files is after they are
> installed in site-lisp.
This makes it difficult to develop code that will be installed in
site-lisp, because code which is not installed cannot be tested with the
native-compilation that will be used when that code is actually
installed. Since there are sometimes bugs in packages when used with
native compilation, it would be better to be able to run the code before
it is installed with the same native-compilation artifacts that will be
used when it is installed.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Wed, 09 Jul 2025 12:02:02 GMT)
Full text and
rfc822 format available.
Message #22 received at 78783 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
> Date: Tue, 08 Jul 2025 14:25:03 -0400
>
> Eli Zaretskii <eliz <at> gnu.org> writes:
> >> This is a general problem with AOT-compiling files which are
> >> subsequently installed somewhere else. It applies both for core Emacs
> >> files and for site-lisp.
> >
> > No, core Emacs files are okay and supported.
> >
> > It's compiling site-lisp files in this way is not supported. And you
> > still haven't answered my question: why do you need that?
> >
> > The supported way of compiling site-lisp files is after they are
> > installed in site-lisp.
>
> This makes it difficult to develop code that will be installed in
> site-lisp, because code which is not installed cannot be tested with the
> native-compilation that will be used when that code is actually
> installed. Since there are sometimes bugs in packages when used with
> native compilation, it would be better to be able to run the code before
> it is installed with the same native-compilation artifacts that will be
> used when it is installed.
Emacs 30 and later has the emacs-lisp-native-compile command, which
can be used to compile the file visited in the current buffer. The
companion command emacs-lisp-native-compile-and-load will also load
the resulting .eln file. I think these two commands can be used to
test the results of natively-compiling any Lisp file, anywhere.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Wed, 09 Jul 2025 13:19:01 GMT)
Full text and
rfc822 format available.
Message #25 received at 78783 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Wed, Jul 9, 2025, 8:01 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > From: Spencer Baugh <sbaugh <at> janestreet.com>
> > Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org,
> app-emacs-dev <at> janestreet.com
> > Date: Tue, 08 Jul 2025 14:25:03 -0400
> >
> > Eli Zaretskii <eliz <at> gnu.org> writes:
> > >> This is a general problem with AOT-compiling files which are
> > >> subsequently installed somewhere else. It applies both for core Emacs
> > >> files and for site-lisp.
> > >
> > > No, core Emacs files are okay and supported.
> > >
> > > It's compiling site-lisp files in this way is not supported. And you
> > > still haven't answered my question: why do you need that?
> > >
> > > The supported way of compiling site-lisp files is after they are
> > > installed in site-lisp.
> >
> > This makes it difficult to develop code that will be installed in
> > site-lisp, because code which is not installed cannot be tested with the
> > native-compilation that will be used when that code is actually
> > installed. Since there are sometimes bugs in packages when used with
> > native compilation, it would be better to be able to run the code before
> > it is installed with the same native-compilation artifacts that will be
> > used when it is installed.
>
> Emacs 30 and later has the emacs-lisp-native-compile command, which
> can be used to compile the file visited in the current buffer. The
> companion command emacs-lisp-native-compile-and-load will also load
> the resulting .eln file. I think these two commands can be used to
> test the results of natively-compiling any Lisp file, anywhere.
>
Yes, but then those native compilation artifacts aren't the ones which will
actually be used when Emacs is installed. So there is the possibility of
differences between them, which can cause bugs which aren't caught by
testing only the uninstalled version.
>
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Wed, 09 Jul 2025 13:55:02 GMT)
Full text and
rfc822 format available.
Message #28 received at 78783 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Date: Wed, 9 Jul 2025 09:17:41 -0400
> Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org,
> app-emacs-dev <app-emacs-dev <at> janestreet.com>
>
> > > The supported way of compiling site-lisp files is after they are
> > > installed in site-lisp.
> >
> > This makes it difficult to develop code that will be installed in
> > site-lisp, because code which is not installed cannot be tested with the
> > native-compilation that will be used when that code is actually
> > installed. Since there are sometimes bugs in packages when used with
> > native compilation, it would be better to be able to run the code before
> > it is installed with the same native-compilation artifacts that will be
> > used when it is installed.
>
> Emacs 30 and later has the emacs-lisp-native-compile command, which
> can be used to compile the file visited in the current buffer. The
> companion command emacs-lisp-native-compile-and-load will also load
> the resulting .eln file. I think these two commands can be used to
> test the results of natively-compiling any Lisp file, anywhere.
>
> Yes, but then those native compilation artifacts aren't the ones which will actually be used when Emacs is
> installed. So there is the possibility of differences between them, which can cause bugs which aren't caught
> by testing only the uninstalled version.
Every file you install on the end-user machine is never the same as
the one you tested on your development machine. And yet we don't say
they cannot be tested.
So I don't see the difference here, and don't understand why we are
splitting hair. The problems you raise are barely tangible at best.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Wed, 09 Jul 2025 14:15:02 GMT)
Full text and
rfc822 format available.
Message #31 received at 78783 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Wed, Jul 9, 2025, 9:53 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > From: Spencer Baugh <sbaugh <at> janestreet.com>
> > Date: Wed, 9 Jul 2025 09:17:41 -0400
> > Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org,
> > app-emacs-dev <app-emacs-dev <at> janestreet.com>
> >
> > > > The supported way of compiling site-lisp files is after they are
> > > > installed in site-lisp.
> > >
> > > This makes it difficult to develop code that will be installed in
> > > site-lisp, because code which is not installed cannot be tested with
> the
> > > native-compilation that will be used when that code is actually
> > > installed. Since there are sometimes bugs in packages when used with
> > > native compilation, it would be better to be able to run the code
> before
> > > it is installed with the same native-compilation artifacts that will
> be
> > > used when it is installed.
> >
> > Emacs 30 and later has the emacs-lisp-native-compile command, which
> > can be used to compile the file visited in the current buffer. The
> > companion command emacs-lisp-native-compile-and-load will also load
> > the resulting .eln file. I think these two commands can be used to
> > test the results of natively-compiling any Lisp file, anywhere.
> >
> > Yes, but then those native compilation artifacts aren't the ones which
> will actually be used when Emacs is
> > installed. So there is the possibility of differences between them,
> which can cause bugs which aren't caught
> > by testing only the uninstalled version.
>
> Every file you install on the end-user machine is never the same as
> the one you tested on your development machine. And yet we don't say
> they cannot be tested.
>
When I compile a .o on my machine and then copy it to a user machine and
run it there, it is the same file, because it has the same contents and the
same name.
So I don't see the difference here, and don't understand why we are
> splitting hair. The problems you raise are barely tangible at best.
>
The other issue is that in my build system, when automated tests are run
they are run in a temporary directory with a randomized name. This means
that the native compilation has to happen separately in every test. This
is much too slow; it would be better to be able to native compile once and
share that native compilation between all tests, like can be done with byte
compilation.
>
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Wed, 09 Jul 2025 15:54:02 GMT)
Full text and
rfc822 format available.
Message #34 received at 78783 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Date: Wed, 9 Jul 2025 10:13:48 -0400
> Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org,
> app-emacs-dev <app-emacs-dev <at> janestreet.com>
>
> On Wed, Jul 9, 2025, 9:53 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > Emacs 30 and later has the emacs-lisp-native-compile command, which
> > can be used to compile the file visited in the current buffer. The
> > companion command emacs-lisp-native-compile-and-load will also load
> > the resulting .eln file. I think these two commands can be used to
> > test the results of natively-compiling any Lisp file, anywhere.
> >
> > Yes, but then those native compilation artifacts aren't the ones which will actually be used when
> Emacs is
> > installed. So there is the possibility of differences between them, which can cause bugs which aren't
> caught
> > by testing only the uninstalled version.
>
> Every file you install on the end-user machine is never the same as
> the one you tested on your development machine. And yet we don't say
> they cannot be tested.
>
> When I compile a .o on my machine and then copy it to a user machine and run it there, it is the same file,
> because it has the same contents and the same name.
You can only do that if the other machine has exactly the same
architecture and the same versions of system libraries. See bug#78907
for what happens otherwise.
> So I don't see the difference here, and don't understand why we are
> splitting hair. The problems you raise are barely tangible at best.
>
> The other issue is that in my build system, when automated tests are run they are run in a temporary
> directory with a randomized name. This means that the native compilation has to happen separately in
> every test. This is much too slow; it would be better to be able to native compile once and share that native
> compilation between all tests, like can be done with byte compilation.
If you are trying to argue that we should give up the strict testing
which is precondition for loading a .eln file, then it's not going to
happen, sorry. Crashing an Emacs session due to some
incompatibilities we avoid this way is a much higher price to pay than
what you describe, even if there's no better solutions within the
current restrictions.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Wed, 09 Jul 2025 16:18:02 GMT)
Full text and
rfc822 format available.
Message #37 received at 78783 <at> debbugs.gnu.org (full text, mbox):
On Wed, Jul 9, 2025 at 11:52 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Spencer Baugh <sbaugh <at> janestreet.com>
> > Date: Wed, 9 Jul 2025 10:13:48 -0400
> > Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org,
> > app-emacs-dev <app-emacs-dev <at> janestreet.com>
> >
> > On Wed, Jul 9, 2025, 9:53 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > > Emacs 30 and later has the emacs-lisp-native-compile command, which
> > > can be used to compile the file visited in the current buffer. The
> > > companion command emacs-lisp-native-compile-and-load will also load
> > > the resulting .eln file. I think these two commands can be used to
> > > test the results of natively-compiling any Lisp file, anywhere.
> > >
> > > Yes, but then those native compilation artifacts aren't the ones which will actually be used when
> > Emacs is
> > > installed. So there is the possibility of differences between them, which can cause bugs which aren't
> > caught
> > > by testing only the uninstalled version.
> >
> > Every file you install on the end-user machine is never the same as
> > the one you tested on your development machine. And yet we don't say
> > they cannot be tested.
> >
> > When I compile a .o on my machine and then copy it to a user machine and run it there, it is the same file,
> > because it has the same contents and the same name.
>
> You can only do that if the other machine has exactly the same
> architecture and the same versions of system libraries. See bug#78907
> for what happens otherwise.
Yes, of course. But this is how package managers work: they provide
binaries that are compiled on one machine and then installed on many
user machines, and this works because they are careful to use binaries
on the same architecture and with the same system libraries.
> > So I don't see the difference here, and don't understand why we are
> > splitting hair. The problems you raise are barely tangible at best.
> >
> > The other issue is that in my build system, when automated tests are run they are run in a temporary
> > directory with a randomized name. This means that the native compilation has to happen separately in
> > every test. This is much too slow; it would be better to be able to native compile once and share that native
> > compilation between all tests, like can be done with byte compilation.
>
> If you are trying to argue that we should give up the strict testing
> which is precondition for loading a .eln file, then it's not going to
> happen, sorry. Crashing an Emacs session due to some
> incompatibilities we avoid this way is a much higher price to pay than
> what you describe, even if there's no better solutions within the
> current restrictions.
The strict testing done by comp-el-to-eln-rel-filename is deliberately
weakened for files under PATH_REL_LOADSEARCH or PATH_DUMPLOADSEARCH:
part of the filename is ignored and not checked. I am simply
suggesting that we should do the same exact thing for files under
PATH_SITELOADSEARCH.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Wed, 09 Jul 2025 17:37:03 GMT)
Full text and
rfc822 format available.
Message #40 received at 78783 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Date: Wed, 9 Jul 2025 12:17:26 -0400
> Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
>
> > > When I compile a .o on my machine and then copy it to a user machine and run it there, it is the same file,
> > > because it has the same contents and the same name.
> >
> > You can only do that if the other machine has exactly the same
> > architecture and the same versions of system libraries. See bug#78907
> > for what happens otherwise.
>
> Yes, of course. But this is how package managers work: they provide
> binaries that are compiled on one machine and then installed on many
> user machines, and this works because they are careful to use binaries
> on the same architecture and with the same system libraries.
If the architecture is the same, you can take the *.eln files you
compiled on your system and install them on the end-user's one. So I
don't see a problem to begin with -- you should be able to solve your
problem with a local solution.
> > If you are trying to argue that we should give up the strict testing
> > which is precondition for loading a .eln file, then it's not going to
> > happen, sorry. Crashing an Emacs session due to some
> > incompatibilities we avoid this way is a much higher price to pay than
> > what you describe, even if there's no better solutions within the
> > current restrictions.
>
> The strict testing done by comp-el-to-eln-rel-filename is deliberately
> weakened for files under PATH_REL_LOADSEARCH or PATH_DUMPLOADSEARCH:
> part of the filename is ignored and not checked. I am simply
> suggesting that we should do the same exact thing for files under
> PATH_SITELOADSEARCH.
I don't see the need. The Emacs build doesn't compile any files
inside directories on that path, so there's no reason for the upstream
project to complicate the code on behalf of situations that don't
happen when Emacs is built. There should be no problem for you to do
it with your local arrangements.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Wed, 09 Jul 2025 17:44:02 GMT)
Full text and
rfc822 format available.
Message #43 received at 78783 <at> debbugs.gnu.org (full text, mbox):
On Wed, Jul 9, 2025 at 1:36 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Spencer Baugh <sbaugh <at> janestreet.com>
> > Date: Wed, 9 Jul 2025 12:17:26 -0400
> > Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
> >
> > > > When I compile a .o on my machine and then copy it to a user machine and run it there, it is the same file,
> > > > because it has the same contents and the same name.
> > >
> > > You can only do that if the other machine has exactly the same
> > > architecture and the same versions of system libraries. See bug#78907
> > > for what happens otherwise.
> >
> > Yes, of course. But this is how package managers work: they provide
> > binaries that are compiled on one machine and then installed on many
> > user machines, and this works because they are careful to use binaries
> > on the same architecture and with the same system libraries.
>
> If the architecture is the same, you can take the *.eln files you
> compiled on your system and install them on the end-user's one. So I
> don't see a problem to begin with -- you should be able to solve your
> problem with a local solution.
Not if those *.eln files were compiled before installing Emacs, e.g.
with emacs-lisp-native-compile as you suggested.
> > > If you are trying to argue that we should give up the strict testing
> > > which is precondition for loading a .eln file, then it's not going to
> > > happen, sorry. Crashing an Emacs session due to some
> > > incompatibilities we avoid this way is a much higher price to pay than
> > > what you describe, even if there's no better solutions within the
> > > current restrictions.
> >
> > The strict testing done by comp-el-to-eln-rel-filename is deliberately
> > weakened for files under PATH_REL_LOADSEARCH or PATH_DUMPLOADSEARCH:
> > part of the filename is ignored and not checked. I am simply
> > suggesting that we should do the same exact thing for files under
> > PATH_SITELOADSEARCH.
>
> I don't see the need. The Emacs build doesn't compile any files
> inside directories on that path, so there's no reason for the upstream
> project to complicate the code on behalf of situations that don't
> happen when Emacs is built. There should be no problem for you to do
> it with your local arrangements.
What local arrangement are you suggesting? I can indeed locally patch
comp-el-to-eln-rel-filename to ignore part of the filename for files
under PATH_SITELOADSEARCH, and that's my current plan, if you think
this is reasonable.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Thu, 10 Jul 2025 04:51:02 GMT)
Full text and
rfc822 format available.
Message #46 received at 78783 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Date: Wed, 9 Jul 2025 13:43:11 -0400
> Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.com
>
> > If the architecture is the same, you can take the *.eln files you
> > compiled on your system and install them on the end-user's one. So I
> > don't see a problem to begin with -- you should be able to solve your
> > problem with a local solution.
>
> Not if those *.eln files were compiled before installing Emacs, e.g.
> with emacs-lisp-native-compile as you suggested.
Then do it after installing Emacs on the machine which produces the
packages, and then add the resulting *.eln files to the package.
Or do something else; the tools are all there at your disposal.
> > > The strict testing done by comp-el-to-eln-rel-filename is deliberately
> > > weakened for files under PATH_REL_LOADSEARCH or PATH_DUMPLOADSEARCH:
> > > part of the filename is ignored and not checked. I am simply
> > > suggesting that we should do the same exact thing for files under
> > > PATH_SITELOADSEARCH.
> >
> > I don't see the need. The Emacs build doesn't compile any files
> > inside directories on that path, so there's no reason for the upstream
> > project to complicate the code on behalf of situations that don't
> > happen when Emacs is built. There should be no problem for you to do
> > it with your local arrangements.
>
> What local arrangement are you suggesting?
I don't know! You haven't explained the rationale for the urge to
AOT-compile the files in site-lisp, nor even why they need to be in
site-lisp (and not in your Emacs tree, where you build) in the first
place. You haven't explained why you cannot natively-compile the
files after the build is complete, directing the *.eln files to some
shared directory, and package those *.eln files together with the rest
of the built Emacs. And you haven't explained many other aspects.
Which you don't need to do if you implement your own local solution
for your local problem, but without which you cannot expect us to
suggest the solution of the problem for you.
My point is that I don't see a need for the Emacs upstream project to
support AOT native compilation of site-lisp files, because files in
site-lisp are by definition the responsibility of a site. The Emacs
project is not obliged to accommodate every single quirk of every
downstream distribution. We provide the tools and the source code for
you to use and change as you see fit, and I see several possible
avenues of approach for this issue that you could have taken for
solving your problem locally. But for some reason, you reject all the
proposals, and insist on having something in upstream Emacs.
> I can indeed locally patch comp-el-to-eln-rel-filename to ignore
> part of the filename for files under PATH_SITELOADSEARCH, and that's
> my current plan, if you think this is reasonable.
I don't think it's reasonable, but it's your distribution and your
users you are putting at risk.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Thu, 10 Jul 2025 10:10:03 GMT)
Full text and
rfc822 format available.
Message #49 received at 78783 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Thu, Jul 10, 2025, 12:50 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > From: Spencer Baugh <sbaugh <at> janestreet.com>
> > Date: Wed, 9 Jul 2025 13:43:11 -0400
> > Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org, app-emacs-dev <at> janestreet.co
> <app-emacs-dev <at> janestreet.com>m
> > > The strict testing done by comp-el-to-eln-rel-filename is deliberately
> > > > weakened for files under PATH_REL_LOADSEARCH or PATH_DUMPLOADSEARCH:
> > > > part of the filename is ignored and not checked. I am simply
> > > > suggesting that we should do the same exact thing for files under
> > > > PATH_SITELOADSEARCH.
> > >
> > > I don't see the need. The Emacs build doesn't compile any files
> > > inside directories on that path, so there's no reason for the upstream
> > > project to complicate the code on behalf of situations that don't
> > > happen when Emacs is built. There should be no problem for you to do
> > > it with your local arrangements.
> >
> > What local arrangement are you suggesting?
>
> I don't know! You haven't explained the rationale for the urge to
> AOT-compile the files in site-lisp,
To save compute and storage space on the user's machines by avoiding them
needing to native compile thousands of files every time the Emacs binary
changes.
nor even why they need to be in
> site-lisp (and not in your Emacs tree, where you build) in the first
> place.
No reason. They can move to the Emacs tree instead of site-lisp if that's
the correct place to put hundreds of ELPA packages that are bundled with
Emacs at my site. I'll do that if that's what you suggest.
You haven't explained why you cannot natively-compile the
> files after the build is complete, directing the *.eln files to some
> shared directory, and package those *.eln files together with the rest
> of the built Emacs.
Because the native compilation doesn't just need to happen after the build
is complete, but also after Emacs is installed, because
comp-el-to-eln-rel-filename refuses to use native compilation artifacts
that were produced before Emacs was installed, because of its strict
filename checking. Except that it has a special exception for .el files
shipped with Emacs, which could be extended to site-lisp.
And you haven't explained many other aspects.
> Which you don't need to do if you implement your own local solution
> for your local problem, but without which you cannot expect us to
> suggest the solution of the problem for you.
>
> My point is that I don't see a need for the Emacs upstream project to
> support AOT native compilation of site-lisp files, because files in
> site-lisp are by definition the responsibility of a site. The Emacs
> project is not obliged to accommodate every single quirk of every
> downstream distribution. We provide the tools and the source code for
> you to use and change as you see fit, and I see several possible
> avenues of approach for this issue that you could have taken for
> solving your problem locally. But for some reason, you reject all the
> proposals, and insist on having something in upstream Emacs.
>
> > I can indeed locally patch comp-el-to-eln-rel-filename to ignore
> > part of the filename for files under PATH_SITELOADSEARCH, and that's
> > my current plan, if you think this is reasonable.
>
> I don't think it's reasonable, but it's your distribution and your
> users you are putting at risk.
>
Why isn't it reasonable? Andrea has previously commented in other bugs
that the file name check is not very important in the first place; it's not
about correctness, it's just a convenience for avoiding conflicts during
parallel compilation. Honestly the file name check could probably be
removed entirely, as it has already been weakened in other bugs.
>
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Thu, 10 Jul 2025 13:05:03 GMT)
Full text and
rfc822 format available.
Message #52 received at 78783 <at> debbugs.gnu.org (full text, mbox):
> From: Spencer Baugh <sbaugh <at> janestreet.com>
> Date: Thu, 10 Jul 2025 06:09:20 -0400
> Cc: acorallo <at> gnu.org, 78783 <at> debbugs.gnu.org,
> app-emacs-dev <app-emacs-dev <at> janestreet.com>
>
> You haven't explained why you cannot natively-compile the
> files after the build is complete, directing the *.eln files to some
> shared directory, and package those *.eln files together with the rest
> of the built Emacs.
>
> Because the native compilation doesn't just need to happen after the build is complete, but also after Emacs
> is installed, because comp-el-to-eln-rel-filename refuses to use native compilation artifacts that were
> produced before Emacs was installed, because of its strict filename checking. Except that it has a special
> exception for .el files shipped with Emacs, which could be extended to site-lisp.
I don't understand what you are saying. I feel there's a
misunderstanding here.
My suggestion above was to natively-compile those files _after_ you
run "make install", directing the *.eln files to the same directory
where the Emacs installation places the*.eln files produced by the
build.
> > I can indeed locally patch comp-el-to-eln-rel-filename to ignore
> > part of the filename for files under PATH_SITELOADSEARCH, and that's
> > my current plan, if you think this is reasonable.
>
> I don't think it's reasonable, but it's your distribution and your
> users you are putting at risk.
>
> Why isn't it reasonable? Andrea has previously commented in other bugs that the file name check is not
> very important in the first place; it's not about correctness, it's just a convenience for avoiding conflicts
> during parallel compilation. Honestly the file name check could probably be removed entirely, as it has
> already been weakened in other bugs.
That check is to minimize the risk of loading incompatible *.eln
files.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Thu, 10 Jul 2025 21:13:04 GMT)
Full text and
rfc822 format available.
Message #55 received at 78783 <at> debbugs.gnu.org (full text, mbox):
Spencer Baugh <sbaugh <at> janestreet.com> writes:
> Native compilation of site-lisp directories during the Emacs build
> (e.g. --with-native-compilation=aot) produces .eln files which are
> unused when installed.
>
> The reason is that the absolute file name of the site-lisp directory
> differs between build-time and install-time, and therefore
> comp-el-to-eln-rel-filename computes a different path hash for the files
> contained in site-lisp at build-time and at install-time.
>
> The same issue would affect AOT native compilation of all Lisp in Emacs,
> except that there's a workaround built into comp-el-to-eln-rel-filename,
> as explained by this comment:
>
> As installing .eln files compiled during the build changes their
> absolute path we need an hashing mechanism that is not sensitive
> to that. For this we replace if match PATH_DUMPLOADSEARCH or
> *PATH_REL_LOADSEARCH with '//' before computing the hash.
>
> In other words, file names containing e.g. "30.1.50/lisp", like
> "/usr/local/share/emacs/30.1.50/lisp/foo.el", are transformed to
> "//foo.el" before the path hash is computed.
>
> I suggest the same should be done for file names containing
> "30.1.50/site-lisp". That would resolve the problem and allow site-lisp
> to actually be AOT-compiled.
Mmmmh I don't think it would work as AFAIU not all distros embed the
version in the path:
https://fedoraproject.org/wiki/QA%3ATestcase_emacs_filesystem
https://build.opensuse.org/projects/editors/packages/emacs/files/macros.emacs?expand=0&
Andrea
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Fri, 11 Jul 2025 21:42:01 GMT)
Full text and
rfc822 format available.
Message #58 received at 78783 <at> debbugs.gnu.org (full text, mbox):
On Thu, Jul 10, 2025 at 5:11 PM Andrea Corallo <acorallo <at> gnu.org> wrote:
>
> Spencer Baugh <sbaugh <at> janestreet.com> writes:
>
> > Native compilation of site-lisp directories during the Emacs build
> > (e.g. --with-native-compilation=aot) produces .eln files which are
> > unused when installed.
> >
> > The reason is that the absolute file name of the site-lisp directory
> > differs between build-time and install-time, and therefore
> > comp-el-to-eln-rel-filename computes a different path hash for the files
> > contained in site-lisp at build-time and at install-time.
> >
> > The same issue would affect AOT native compilation of all Lisp in Emacs,
> > except that there's a workaround built into comp-el-to-eln-rel-filename,
> > as explained by this comment:
> >
> > As installing .eln files compiled during the build changes their
> > absolute path we need an hashing mechanism that is not sensitive
> > to that. For this we replace if match PATH_DUMPLOADSEARCH or
> > *PATH_REL_LOADSEARCH with '//' before computing the hash.
> >
> > In other words, file names containing e.g. "30.1.50/lisp", like
> > "/usr/local/share/emacs/30.1.50/lisp/foo.el", are transformed to
> > "//foo.el" before the path hash is computed.
> >
> > I suggest the same should be done for file names containing
> > "30.1.50/site-lisp". That would resolve the problem and allow site-lisp
> > to actually be AOT-compiled.
>
> Mmmmh I don't think it would work as AFAIU not all distros embed the
> version in the path:
>
> https://fedoraproject.org/wiki/QA%3ATestcase_emacs_filesystem
>
> https://build.opensuse.org/projects/editors/packages/emacs/files/macros.emacs?expand=0&
Yes, I concretely am suggesting we look at PATH_SITELOADSEARCH so we
respect whatever the distro has ./configure'd, not just hardcode
"30.1.50/site-lisp".
That would allow any distro to ship code in site-lisp and also ship
pre-compiled .elns for that code, without having to perform native
compilation *after* this installation, which is generally much easier.
Thanks for looking at this.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#78783; Package
emacs.
(Sat, 12 Jul 2025 06:26:02 GMT)
Full text and
rfc822 format available.
Message #61 received at 78783 <at> debbugs.gnu.org (full text, mbox):
> Cc: app-emacs-dev <at> janestreet.com, 78783 <at> debbugs.gnu.org
> Date: Fri, 11 Jul 2025 17:40:53 -0400
> From: Spencer Baugh via "Bug reports for GNU Emacs,
> the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>
> Yes, I concretely am suggesting we look at PATH_SITELOADSEARCH so we
> respect whatever the distro has ./configure'd, not just hardcode
> "30.1.50/site-lisp".
>
> That would allow any distro to ship code in site-lisp and also ship
> pre-compiled .elns for that code, without having to perform native
> compilation *after* this installation, which is generally much easier.
I'm against such a change, since you could solve your problem without
it. I'm not interested in complicating our startup code and its
subtleties (of which there are already too many) because you have a
local problem that can be solved within the current restrictions,
sorry.
This bug report was last modified 68 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.