Subject: Re: Maybe partly undo the patch on Elisp comp-el-to-eln-filename

From: Liliana Marie Prikler <[email-masked]>
Date: Wed, 09 Oct 2024 19:22:55 +0200

Am Mittwoch, dem 09.10.2024 um 17:15 +0200 schrieb Martin Edström:
> Hi, thanks for info! This email will be a bit long because I'm quite
> confused and thinking aloud, so no need to reply to all of it.  But I
> appreciate any attempt to shed clarity!
> 
> 
> On Tue, 08 Oct 2024 19:33:06 +0200, Liliana Marie Prikler
> <[email-masked]> wrote:
> 
> > > It comes as part of the package. I don't want to assume that it
> > > has been compiled, since it's fairly performance-sensitive.
> > > That's why I'll either use a previously existing compiled object
> > > or make a new one.
> > Could you leave that decision to the user?
> 
> I'm considering it, but ran into essentially the same problem.  I
> need a way to tell which one was loaded, of the .elc, .eln or .el.
> 
> This takes the thread a bit off-topic but ... Do you know how?
> 
> Currently I'm leaning towards an algorithm like
> 
> #+begin_src elisp
> (defun guess-loaded-lib (basename)
>   (let* ((el (locate-library (concat basename ".el")))
>          (elc (locate-library (concat basename ".elc")))
>          (eln (comp-el-to-eln-filename el)))
>     (if load-prefer-newer
>         (car (sort (delq 'nil (list el elc eln)) #'file-newer-than-
> file-p))
>       (if (and (file-newer-than-file-p elc el)
>                (file-newer-than-file-p eln el))
>           ;; If elc is newer than el, assume it was compiled from el,
> and
>           ;; therefore is safe to replace with eln
>           eln
>         elc))))
> #+end_src
> 
> I don't know how to "leave it to the user" any better than that.  
The user can just use the output of the help function, it will tell
them whether a function is natively-compiled, byte-compiled, or just
interpreted.  Emacs 30 even gives you `native-comp-function-p'.  See
`gnu/packages/aux-files/emacs/comp-integrity[-next].el' for how we
assert that our Emacs loads natively-compiled libraries.

> On Guix, even in the future when we re-enable JIT compilation, this
> algorithm could never return an .eln, since =file-newer-than-file-p=
> returns nil when both paths have identical timestamps from 1970.
> 
> I found a pretty neat built-in: 
> 
>      (symbol-file 'SOME-FUNCTION-FROM-THE-LIB nil t)
> 
> but its internals also use =file-newer-than-file-p= and =comp-el-to-
> eln-rel-filename=, so not sure it is any more reliable than what I
> have above.
I'm not sure these things work the way you think.  IIUC, `load-prefer-
newer' should guard against the case where the .el is newer than the
.el[cn], not the other way round, so it should still load the
compiled/native-compiled variant if they have the same stamp.

> > There is a separate load path for natively compiled files, called
> > `native-comp-eln-load-path'.
> 
> Good to know! I don't know how Emacs consults it though.  I can't
> e.g. locate an .eln of a given library by calling something like
> 
> #+begin_src elisp
> (locate-eln-file "org-node-parser")
> #+end_src
> 
> so it seems I must simply do:
> 
> #+begin_src elisp
> (let ((el (locate-library "org-node-parser.el")))
>   (when el
>     (locate-eln-file (comp-el-to-eln-rel-filename el))))
> #+end_src
> 
> which is functionally equivalent to:
> 
> #+begin_src elisp
> (let ((eln (comp-el-to-eln-filename (locate-library "org-node-
> parser.el"))))
>   (when (file-exists-p eln)
>     eln))
> #+end_src
I mean, the C code is there to read (along with the patch we've made),
but I can't fault you for not going that deep.  

> > > I infer that Emacs starts with finding a library in load-path,
> > > then converts the path with `comp-el-to-eln-filename`, and checks
> > > if that file exists, then loads it.
> > > 
> > > And crucially, it is not just about the filepath, the function
> > > hashes the file contents as well. That ensures that the output
> > > path is always different if the source file changes.
> > I think relying on such implementation details is perhaps permitted
> > if it's inside of Emacs itself, but even then it clashes with our
> > expectation that Emacs be graftable.
> 
> OK, so if passing "file.el" to =comp-el-to-eln-filename= produces a
> path with a hash of the content, like:
> 
>     .../eln-cache/29.4-HASH/module/file-HASH-BASED-ON-CONTENT.eln
> 
> this would make Emacs non-graftable.  Because the hash would change
> after file.el is upgraded.  I suppose that makes sense, thanks!
> 
> Maybe, as a hack, the graft process could symlink the pre-graft
> filename to the post-graft filename... so we don't have to alter the
> behavior of =comp-el-to-eln-filename=.
The grafting process doesn't look that deep into file names.  On paper,
that could work, since the length of the name is preserved, but it'd
also make the grafting process take longer and not really add all that
much to its reliability.

> But yea... it's less dev effort to just not pre-compile any .eln
> files.
> 
> Out of curiosity: if Guix does no JIT compilation at all anyway, why
> does it not let Emacs do it the usual way into ~/.emacs.d/eln-cache,
> post-installation?  (Aside from the fact it would necessitate
> reverting the behavior of =comp-el-to-eln-filename=.)
There is only one `comp-el-to-eln-filename' to call, you can't (without
a much more ugly hack) have two strategies, and even if you did have
them, it'd be even more error-prone.

> Actually... (with reservation that I may be wasting your time) I'm
> starting to wonder why the filename needs to stay the same for Emacs
> to be graftable?  Realistically, if all Emacs packages get upgraded,
> and if =comp-el-to-eln-filename= works like upstream, we have some
> paths like
> 
>     /gnu/store/emacs-29.4/bin/emacs
>     /gnu/store/emacs-magit-OLD/lisp/magit.el
>     /gnu/store/emacs-magit-NEW/lisp/magit.el
>     .../wherever/magit-OLDHASH.eln
>     .../wherever/magit-NEWHASH.eln
> 
> Then we start Emacs, and run its =comp-el-to-eln-filename= on the new
> magit.el, it would correctly return .../wherever/magit-NEWHASH.eln.
> No need for static filenames to swap out.
I haven't linked at the shared objects themselves, but I have strong
hunch that they use dynamic linking between the libraries.  Since we
only update the /gnu/store/emacs-whatever-HERE-WE-HAVE-A-HASH portion
and nothing post that, we get file names that don't exist :)

If you dig into the patch series, it references the bug it fixes.

> […]
> 
> No, grafting is about avoiding a world re-compile, right?  So like if
> package-A was built with an old dependency package-B, it does not get
> re-built when package-B is upgraded. But does grafting actually ever
> change anything in Emacs?  When you start emacs and it loads package-
> A which in turn loads package-B because there's a =require= statement
> for package-B, Emacs will actually load the fresh package-B from
> /gnu/store/emacs-package-B, won't it?  So grafting would seem to be
> meaningless.
For example, GTK is a package deep in the world, that gets grafted a
lot and causes Emacs to be grafted as well.

> Not sure how to feel if it's like this instead: a compiled function
> from package A, containing a call to some function "package-B-
> frobnicate", will actually call some bytecode originating from
> /gnu/store/emacs-package-A/share/emacs/site-lisp/package-B.elc?
> That's... not what happens, right?  OK, probably yes if it was a
> defsubst or defmacro.
> 
> So we can have package A using a macro that was defined in an old
> version of package-B.  I like that not, but the reason I ask is I'm
> trying to find out where it is that the file path would influence the
> graft.
I haven't looked in the ABI stability of Emacs Lisp packages, but most
of them would actually not need a graft.  It's Emacs itself that needs
the grafting :) 

> > The way our load paths are set up, it is actually the opposite
> > (which still is a bug, just not the one reported).  While `guix
> > upgrade` or a command to the similar effect will swap out the .eln
> > under the hood, the `.el` and `.elc` files stay stable – remember
> > what I wrote in the previous message about that having caused
> > issues with byte compilation?
> 
> I confess I'm puzzled.  Does this just apply to a case like the
> following?
> 
> - Emacs itself is upgraded
> - A package emacs-magit is NOT upgraded
> 
> so that
> 
> - magit.eln is rebuilt
> - magit.el and magit.elc stay the same
> 
> That would make sense.  Do you mean it's a bug because .elc files are
> also not portable between different Emacsen?
We had issues, where a major upgrade between Emacsen would cause
invalid bytecode to be loaded, yes.  The issue in Guix is not so much
that a package is not upgrade, but Emacs being upgraded while Emacs is
running.  You'd get a functioning Emacs once you exit the process and
start a new one, but that feels very dirty.  (Imagine exiting Emacs,
because your package manager tells you so, ugh!)

> Just checking: is it correct to expect that if you actually upgrade
> the emacs-magit package, then its .el, .elc and .eln are all
> upgraded?
Yes, the .el, .elc, and .eln files are built as *one* package.  Any
package manager you use with Emacs should provide you with that
invariant.

Cheers