Subject: bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

From: Maxim Cournoyer <[email-masked]>
Date: Sat, 26 Dec 2020 23:44:32 -0500

Hi Leo,

Leo Prikler <[email-masked]> writes:

[...]

>> If you want to have a clearer idea of how packages from ELPA and the
>> likes are installed, you can have a peek into the 'package.el' file
>> shipped with Emacs (spoiler: it's basically just extracting the
>> contents
>> of the package archive to a single directory -- see the `package-
>> unpack'
>> procedure).
> If we follow that to the letter, should we not install those packages
> directly to site-lisp/$PACKAGE-$VERSION?  Why the guix directory?

I used to wonder the same, but after getting familiar with the 'package'
Emacs library, it defaults (per the default value of the
`package-directory-list' variable) to such a layout:

directory-on-load-path/prefix/name-version/

where 'prefix' defaults to 'elpa'.  It seems a good idea to have the
ELPA-style packages live by themselves, separated from the 'classic'
packages that are installed directly to the load path, to avoid the
package.el loader getting confused or scanning entries it wouldn't know
how to load anyway.

That said, it's shouldn't be strictly required, as old style packages or
resource directories would be lacking a *-pkg.el file and would be
disregarded, and it should be possible to install the package
directories directly to the load path if we wanted to.

> Also, if we do go that route, I think we should install a subdirs.el
> through a profile hook, that adds all these package directories to the
> load path.  See also '(elisp) Startup Summary'.  Alternatively, we can
> let subdirs.el defer to guix-emacs.el, but that'd be very cheeky.
> Either way, we should definitely make -Q work.

If we use a subdirs.el file, it could add the ELPA-style packages more
simply if they are under the guix sub-directory (guix/*).  Otherwise
we'd have to filter the resource directories that potentially find their
ways to the load path for old style packages.  Our custom subdirs.el
file could be shipped with our Emacs packages, installed in their output
at share/emacs/site-lisp/subdirs.el.  When a profile union would be
created this subdirs.el would automatically be made available.

>> Here's an experiment I've done with a profile containing the whole of
>> our current emacs-build-system based packages, with this current
>> change,
>> that checks for collision at the top level of a package installation
>> directory:
>>
>> --8<---------------cut here---------------end--------------->8---
>> $ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-
>> manifest.scm
>>
>> $  find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2
>> -mindepth 2 -not -regex '.*\.elc?$' \
>>           | awk -F '/' '{ if ($9 in packages) \
>>             {printf("%s directory of %s package collides with that of
>> package %s\n", $9, $8, packages[$9])} \
>>             else {packages[$9] = $8} }'
>>
>> doc directory of modus-operandi-theme-1.0.2 package collides with
>> that of package racket-mode-0.0.2-6.5eb31a2
>> tools directory of unidecode-0.2-1.5502ada package collides with that
>> of package company-cabal-0.3.0-1.62112a7
>> snippets directory of minitest-0.8.0-1.1aadb78 package collides with
>> that of package elpy-1.35.0
>> test directory of realgud-1.5.1 package collides with that of package
>> flycheck-haskell-0.8-2.32ddff8
>> snippets directory of yasnippet-snippets-0.23 package collides with
>> that of package elpy-1.35.0
>> snippets directory of haskell-snippets-0.1.0-0.07b0f46 package
>> collides with that of package elpy-1.35.0
>> snippets directory of rspec-1.11-1.66ea7cc package collides with that
>> of package elpy-1.35.0
>> doc directory of modus-themes-1.0.2 package collides with that of
>> package racket-mode-0.0.2-6.5eb31a2
>> data directory of emojify-1.2 package collides with that of package
>> unidecode-0.2-1.5502ada
>> test directory of systemd-mode-1.6 package collides with that of
>> package flycheck-haskell-0.8-2.32ddff8
>> lib directory of slime-2.26.1 package collides with that of package
>> robe-0.8.2
>> contrib directory of sly-1.0.0-7.68561f1 package collides with that
>> of package slime-2.26.1
>> lib directory of sly-1.0.0-7.68561f1 package collides with that of
>> package robe-0.8.2
>> doc directory of modus-vivendi-theme-1.0.2 package collides with that
>> of package racket-mode-0.0.2-6.5eb31a2
>> doc directory of evil-1.14.0 package collides with that of package
>> racket-mode-0.0.2-6.5eb31a2
>> data directory of all-the-icons-4.0.1 package collides with that of
>> package unidecode-0.2-1.5502ada
>> snippets directory of feature-mode-20190801-1.11ae167 package
>> collides with that of package elpy-1.35.0
>> --8<---------------cut here---------------end--------------->8---
>>
>> So 17 Emacs packages in Guix currently conflict, and Guix seems to be
>> silent about it when building a profile with them via 'guix package
>> -p'
>> (a bug?).

> It's a bug in those packages; not in Guix.

I used to think the same, but after reading the Elisp reference manual
mentioning how Emacs packages to be installed with package.el are
expected to live in their own distinct directory with their own
resources; we can't really say that's it's a bug in the packages: they
just weren't designed to be merged in a flat directory with other Elisp
packages.  Users installing these packages manually can simply add their
top level directory to their load path, else use package.el to install
them.

> The whole idea of Guix is, that such directories are merged.  Now
> collisions within files in those directories are a different topic –
> only one can be chosen there and IIRC there should also be a way to
> make them errors.

I'll investigate this separately.

[...]

>> > > We have two schemes to accommodate for our Emacs packages:
>> > >
>> > > 1. Those installed via their own mean, e.g. make install, using
>> > > the
>> > > gnu-build-system for example.  These would still typically
>> > > install
>> > > their
>> > > packages directly under site-lisp, possibly multiple files (that
>> > > could
>> > > still collide).
>> > Why?  Seems inconsistent, does it not?
>>
>> They can be adapted in time; until then it's easy to accommodate
>> both.
> I think we should aim for consistency here, but let's adapt them in
> follow-up commits.

Agreed.

>> > > 2. Those installed via the emacs-build-system.  With the proposed
>> > > changes, those now go to site-lisp/guix/.  The 'guix' sub-
>> > > directory
>> > > makes it unambiguous that anything found under is to be loaded by
>> > > package.el; the `package-directory-list' variable can be pointed
>> > > to
>> > > it
>> > > to have the Emacs' package library discover these self-contained
>> > > packages.
>> > I think you've lost me here.  Basically, instead of being able to
>> > `(require 'my-package)` you first need to load my-package through
>> > package.el and then can `(require 'my-package)`?  I am not sure,
>> > what
>> > the benefit of that would be, if any.
>>
>> Only if you've specified --no-site-start, which would disable
>> user-installed package discovery.  This is the drawback of the
>> trade-off, not the benefit :-).
> It wasn't worded like one.  The word unambiguous typically has a
> positive connotation in my mind.  And package discovery should function
> regardless of site-start.

The word unambiguous was used to describe that using the sub-directory
guix/ would mean anything under it would be strictly packages to be
loaded with package.el and nothing else.

[...]

>> > Being able to load the same libraries as without is vital for
>> > debugging
>> > and scripts.  (Granted, some distros probably break with --no-site-
>> > lisp, but that's nothing to strive for imo.)
>>
>> [...]
>>
>> Thanks for having me think harder about if this is really
>> desirable/needed.  For me the main plus is that we'd be adhering to
>> the
>> current standards used for Emacs packaging: everything is unpacked
>> into
>> a single directory owned by that package, so as long as we include
>> any
>> needed resources in the #:include argument of the emacs-build-system,
>> it
>> works as upstream intended it, with no need to patch variables or
>> anything.  I expect this situation to worsen as Emacs packages tend
>> to
>> get more complicated, and I don't feel strongly enough about it to go
>> arguing about Emacs packaging standards on the Emacs mailing list :-
>> ).
> I agree, that adhering to the standards sounds nice, but this patch
> makes it seem as if we're still cooking our own soup.  I believe we'd
> need to adhere to them harder than that in order for it to be an
> upgrade of the status quo.

I understand this may feel a hack; because for all I know, it is ;-).
I'll try improving it with the subdirs.el discussed above.

[...]

> I think I've already laid that out above, but we really ought to have
> site-lisp/$PACKAGE-$VERSION and a working subdirs.el.  If we do it like
> that, I don't think the multi-directory layout should cause us too many
> troubles.
>
> As far as the build system is concerned, we would probably need to set
> up an environment similar to what will be found in the end.
> Imagine this:
>
> - environment-variables
> - $PACKAGE-$VERSION/
> - $PACKAGE-$VERSION-inputs/
>   - subdirs.el
>   - $INPUT1/
>   - $INPUT2/

In the build environment, there is no profile so each package add their
individual site-lisp/ to the load path (EMACSLOADPATH).  With my
proposed idea to add subdirs.el to Emacs itself, there'd be nothing to
do here.

Thanks again for this discussion; I'll work on a revised patch.  Until
then, happy new year!

Maxim