Subject: bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

From: Konstantin Kharlamov <[email-masked]>
Date: Sat, 16 Nov 2024 20:45:47 +0300

On Sat, 2024-11-16 at 16:53 +0000, Alan Mackenzie wrote:
> Hello, Konstantin.
>
> On Sat, Nov 16, 2024 at 18:11:01 +0300, Konstantin Kharlamov wrote:
> > CC: Alan Mackenzie, author of the change in
> > 10083e788f7349fa363d100687dc3d94bea88f57
>
> > I've seen for a long time Emacs master builds fail from time to
> > time in spectacular
> > ways after updating the repo, sometimes so badly that `make clean`
> > doesn't help.
>
> > I never dug into that though, but I'm attributing this to the
> > occasional build
> > messages similar to:
>
> >     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-
> > lisp/comp.el’ newer than byte-compiled file; using older file
> >     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-
> > lisp/bytecomp.el’ newer than byte-compiled file; using older file
> >     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-
> > lisp/comp-cstr.el’ newer than byte-compiled file; using older file
>
> > …which makes sense, because if the repo changed `comp.el` API and
> > Emacs during the
> > build of newer files is trying to make use of older `.elc` file and
> > hence the older
> > API, it may result in failure.
>
> The idea is that the byte compiler is first built from .el files,
> giving
> ..elc files.  The .elc byte compiler is then used for all the other
> files, since it's much faster.
>
> > Got some spare time today, dug into one of the messages. From what
> > I understand it's
> > caused by this line `lisp/Makefile.in`:
>
> >     # ... but we must prefer .elc files for those in the early
> > bootstrap.
> >     compile-first: BYTE_COMPILE_FLAGS = $(BYTE_COMPILE_EXTRA_FLAGS)
>
> > > From what I understand, this rewrites BYTE_COMPILE_FLAGS to be an
> > > empty variable,
> > which results in `(setq load-prefer-newer t)` being stripped off of
> > the build.
>
> Yes, this is to prefer the faster .elc byte compiler, whose file
> dates
> have been set back to the epoch (1970-01-01:00:00:00 UTC).  I think
> this
> was to ensure that when we come to native compilation, the .el source
> files of the compiler will be newer than the .elc files, hence
> triggering a native compilation of them.
>
> I honestly don't believe that this mechanism for speeding up the
> early
> build is the cause of the spectacular failures in some of your
> builds,
> but I could be wrong.
>
> > The straightforward solution is to remove this line. But since the
> > line's commentary
> > opposes to such solution, I'm starting up a discussion what exactly
> > should be the
> > behavior here 😊
>
> Perhaps if you could be more specific about the failures you see, we
> might be able to be of more help.

Sure, I just reproduced it after removing all `.elc` files in the repo,
here how:

1. `git checkout f2f13fa630b` (a commit from April)
2. `make -j$(nproc)` to compile. Note: you don't need to wait for build
to finish, I just waited for all files under `lisp/emacs-lisp`
directory to finish compilation, and then ^C'ed it.
3. `git checkout 29098a291f5` (a November commit).
4. `make -j$(nproc)`

## Expected

Build finishes

## Actual

It fails with stacktrace so huge that it goes beyond Konsole scrollback
and with the following message:

    Loading macroexp.elc (compiled; note, source file is newer)...
    Wrong type argument: "../../lisp/emacs-lisp/comp.el", hash-table-p, (#s(byte-to-native-top-level (byte-code "��!���!���!���!���!���!���!���!������%�����DD�����&        �����DD�����&   �����DD���&�����DD�����&������DD�����&������DD�����&������DD�����&�" [require bytecomp cl-lib gv rx subr-x warnings comp-common comp-cstr custom-declare-group ...] 10) t) #s(byte-to-native-top-level (#<symbol defvar at 4425> #<symbol no-native-compile at 4432> nil "Non-nil to prevent native-compiling of Emacs Lisp code.
    Note that when `no-byte-compile' is set to non-nil it overrides the value of
    `no-native-compile'.
    This is normally set in local file variables at the end of the
    Emacs Lisp file:

You can see it fails on the `comp.el`, and I presume the reason is it simply didn't rebuild the necessary .elc files, and instead loaded outdated bytecode.