GNU bug report logs - #45316
[PATCH]: Re-introduce Emacs packages specific installation prefix.

Previous Next

Full log

View this message in rfc822 format

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Leo Prikler <leo.prikler <at> student.tugraz.at>
Cc: 45316 <at> debbugs.gnu.org
Subject: bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.
Date: Sat, 26 Dec 2020 00:01:29 -0500

Hi Leo!

Leo Prikler <leo.prikler <at> student.tugraz.at> writes:

> Hi Maxim,
>
> Am Dienstag, den 22.12.2020, 13:09 -0500 schrieb Maxim Cournoyer:
>> Hi Leo,
>>
>> Leo Prikler <leo.prikler <at> student.tugraz.at> writes:
>>
>> > Hello Maxim,
>> >
>> > As someone, who lobbied for the current status quo, I have some
>> > thoughts to share.
>>
>> I'm happy you're commenting :-).
>>
>> > Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer:
>> > > The Emacs packages built with the Emacs built system used to be
>> > > installed in a sub-directory under the share/emacs/guix.d/
>> > > directory,
>> > > but this was changed in commit
>> > > 65a7dd2950ca13a8b942b2836260a2192351b271
>> > > shortly after having accommodated the site-start.el machinery to
>> > > enable
>> > > loading packages from any profile (via the EMACSLOADPATH search
>> > > path
>> > > specification).
>> > Won't this reintroduce <https://bugs.gnu.org/38309> then?
>>
>> This bug was caused by the EMACSLOADPATH environment variable being
>> too
>> long.  This was because the original search path specification was
>> recursively adding any directories under site-lisp/ so that package
>> installed under a guix.d sub-directory could were directly added to
>> the
>> load path.
>>
>> In the current situation, the EMACSLOADPATH contains only the site-
>> lisp/
>> directory of any profile, plus the versioned lisp/ directory of the
>> installed Emacs package.  This patch series doesn't change that, so
>> no,
>> that bug wouldn't be reintroduced.

> Which is also the reason why our startup code changes and people have
> to accommodate to that, right?

Yes. The startup code is now responsible to activate packages found not
directly in the EMACSLOADPATH (share/emacs/site-lisp), but in
sub-directories under share/emacs/site-lisp/guix.

>> > > While this change allowed to expose simply and directly the
>> > > packages
>> > > found in EMACSLOADPATH, it does introduce the risk of file name
>> > > collisions when multiple Emacs packages are joined in the same
>> > > profile,
>> > > especially with Emacs packages increasing in complexity (e.g.,
>> > > using
>> > > more than a single .el file!) and expecting to have both their
>> > > sources
>> > > and resources extracted under their own nested directory rather
>> > > than
>> > > as
>> > > a flat collection (ELPA, MELPA).
>> > > One recent example I stumbled on was attempting to use the
>> > > emacs-yasnippet-snippets package along with emacs-elpy; both
>> > > wanted
>> > > to
>> > > install a 'snippets' directory to share/emacs/site-lisp/snippets,
>> > > collided and resulted in problems that prove difficult to
>> > > understand.
>> > I believe that to be a problem in those packages.  Data should not
>> > be
>> > installed into share/emacs/site-lisp, but share/emacs/etc.  See for
>> > instance also emacs-telega, which – while not quite adhering to the
>> > above – installs its data in share/emacs/{telega-contrib,telega-
>> > data}.
>> >
>> > Regardless of what you intend to do with site-lisp otherwise, data
>> > files should *not*, I repeat *not* be installed there.  I do
>> > believe
>> > standardizing share/emacs/etc is the way to go, however.
>>
>> While I agree that it would be more elegant the way you propose,
>> that's
>> not the way Emacs packages have been standardized.  So going against
>> the
>> sin--8<---------------cut here---------------start------------->8---
--8<---------------cut here---------------start------------->8---
gle "content directory" (c.f. info "(elisp) Packaging Basics")
>> would
>> break many Elisp library assumptions about where there files are,
>> causing more friction (thus work) to adapt them in Guix.  The content
>> directory of an Elisp package can contain any kind of files (code,
>> images, etc.), according to info "(elisp) Multi-file Packages".
>
> I suppose said manual is perhaps slightly outdated.

The doc/package.texi file of the Emacs git repository is actively
maintained, although cited contents above hasn't changed much for the
last 8 years.  In any case if you look into package.el, you'll see that
package contents are fetched as a single archived, and its content
installed to a single directory.

>> A package is either a “simple package” or a “multi-file package”.  A
>> simple package is stored in a package archive as a single Emacs Lisp
>> file, while a multi-file package is stored as a tar file (containing
>> multiple Lisp files, and possibly non-Lisp files such as a manual).
> When was the last time, you've installed a tar to site-lisp?  I also
> feel as though the expected contents are limited very much to README,
> COPYING and the like:
>
> $ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \
>   -not -name '*.el' \
>   -and -not -name '*.elc' \
>   -and -not -name '*.el.gz'
> ~/.guix-profile/share/emacs/27.1/lisp/term/README
> ~/.guix-profile/share/emacs/27.1/lisp/COPYING
> ~/.guix-profile/share/emacs/27.1/lisp/README

The lisp/ directory is for "the standard Lisp files that come with
Emacs" (from info "(elisp)Library Search").  I don't think we can draw
much comparison between these and user-installable packages from ELPA or
MELPA.

Also, just to be clear, the tar is not installed as-is; it is extracted
and its contents are installed in the "content directory".  Citing again
'(elisp) Multi-file Packages':

    Prior to installation, a multi-file package is stored in a package
    archive as a tar file.  The tar file must be named ‘NAME-VERSION.tar’,
    where NAME is the package name and VERSION is the version number.  Its
    contents, once extracted, must all appear in a directory named
    ‘NAME-VERSION’, the “content directory” (*note Packaging Basics::).
    Files may also extract into subdirectories of the content directory.

       One of the files in the content directory must be named
    ‘NAME-pkg.el’.  It must contain a single Lisp form, consisting of a call
    to the function ‘define-package’, described below.  This defines the
    package’s attributes: version, brief description, and requirements.

       For example, if we distribute version 1.3 of the superfrobnicator as
    a multi-file package, the tar file would be ‘superfrobnicator-1.3.tar’.
    Its contents would extract into the directory ‘superfrobnicator-1.3’,
    and one of these would be the file ‘superfrobnicator-pkg.el’.

If you want to have a clearer idea of how packages from ELPA and the
likes are installed, you can have a peek into the 'package.el' file
shipped with Emacs (spoiler: it's basically just extracting the contents
of the package archive to a single directory -- see the `package-unpack'
procedure).

Here's an experiment I've done with a profile containing the whole of
our current emacs-build-system based packages, with this current change,
that checks for collision at the top level of a package installation
directory:

--8<---------------cut here---------------end--------------->8---
$ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-manifest.scm

$  find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2
-mindepth 2 -not -regex '.*\.elc?$' \
          | awk -F '/' '{ if ($9 in packages) \
            {printf("%s directory of %s package collides with that of package %s\n", $9, $8, packages[$9])} \
            else {packages[$9] = $8} }'

doc directory of modus-operandi-theme-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
tools directory of unidecode-0.2-1.5502ada package collides with that of package company-cabal-0.3.0-1.62112a7
snippets directory of minitest-0.8.0-1.1aadb78 package collides with that of package elpy-1.35.0
test directory of realgud-1.5.1 package collides with that of package flycheck-haskell-0.8-2.32ddff8
snippets directory of yasnippet-snippets-0.23 package collides with that of package elpy-1.35.0
snippets directory of haskell-snippets-0.1.0-0.07b0f46 package collides with that of package elpy-1.35.0
snippets directory of rspec-1.11-1.66ea7cc package collides with that of package elpy-1.35.0
doc directory of modus-themes-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
data directory of emojify-1.2 package collides with that of package unidecode-0.2-1.5502ada
test directory of systemd-mode-1.6 package collides with that of package flycheck-haskell-0.8-2.32ddff8
lib directory of slime-2.26.1 package collides with that of package robe-0.8.2
contrib directory of sly-1.0.0-7.68561f1 package collides with that of package slime-2.26.1
lib directory of sly-1.0.0-7.68561f1 package collides with that of package robe-0.8.2
doc directory of modus-vivendi-theme-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
doc directory of evil-1.14.0 package collides with that of package racket-mode-0.0.2-6.5eb31a2
data directory of all-the-icons-4.0.1 package collides with that of package unidecode-0.2-1.5502ada
snippets directory of feature-mode-20190801-1.11ae167 package collides with that of package elpy-1.35.0
--8<---------------cut here---------------end--------------->8---

So 17 Emacs packages in Guix currently conflict, and Guix seems to be
silent about it when building a profile with them via 'guix package -p'
(a bug?).

> Perhaps we should ask Emacs folks how they feel about separating code
> and data, but bugs have already been caused here and elsewhere by
> stuffing them together.  (I believe yasnippet was once already the
> culprit at some point before it got reorganized.)  In the meantime,
> yes, doing so might cause extra work, but
> 1) it only affects a subset of packages, and
> 2) of this subset, we can prioritize those, that do exhibit this
> behaviour.

The above does output does show that's it's a subset of the packages
that are affected.

[...]

>> We have two schemes to accommodate for our Emacs packages:
>>
>> 1. Those installed via their own mean, e.g. make install, using the
>> gnu-build-system for example.  These would still typically install
>> their
>> packages directly under site-lisp, possibly multiple files (that
>> could
>> still collide).
> Why?  Seems inconsistent, does it not?

They can be adapted in time; until then it's easy to accommodate both.

>> 2. Those installed via the emacs-build-system.  With the proposed
>> changes, those now go to site-lisp/guix/.  The 'guix' sub-directory
>> makes it unambiguous that anything found under is to be loaded by
>> package.el; the `package-directory-list' variable can be pointed to
>> it
>> to have the Emacs' package library discover these self-contained
>> packages.
> I think you've lost me here.  Basically, instead of being able to
> `(require 'my-package)` you first need to load my-package through
> package.el and then can `(require 'my-package)`?  I am not sure, what
> the benefit of that would be, if any.

Only if you've specified --no-site-start, which would disable
user-installed package discovery.  This is the drawback of the
trade-off, not the benefit :-).

>> Currently if you use -Q, the Elisp libraries are in the load-path,
>> but
>> not loaded (you need manually do M-x load-library before you can use
>> it), or call M-x guix-emacs-autoload-packages to load their
>> autoloads.
>> For programs, this mean (require 'some-library) works, but M-x
>> some-library-autoloaded-function doesn't.
>>
>> After this change, if you use -Q the new style Emacs packages are not
>> visible at all until you activate them with 'M-x load-library
>> guix-emacs' then 'M-x guix-package-initialize', or (require 'guix-
>> emacs)
>> then (guix-package-initialize), programmatically.

> I do think, that there are legitimate reasons to not require a full
> initialization here, particularly for the use in batch scripts, but I'm
> not quite sure how much work we currently put into making sure that
> everything is available.

That's a fair point, although I'm not concerned about the overhead of
loading compiled autoloads files.

>> I don't see this as a problem.  -Q is simply an alias for
>> "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources"
>> "--no-site-file".
> But not "--no-load-path" "--no-nothing" ;)

One could argue that the spirit of -Q is to give an Emacs as minimal as
possible (that's what the Emacs manual has to say about it, and the
implicit --no-site-lisp has the meaning of "no user installed
packages").

> Being able to load the same libraries as without is vital for debugging
> and scripts.  (Granted, some distros probably break with --no-site-
> lisp, but that's nothing to strive for imo.)

I think you meant --no-site-file here.  There's not much magic; if you
allow it to run, user installed packages are added to the load path and
autoloaded; if you don't you don't then you only get Emacs standard
libraries.  Nothing else happens in the site-start.el file in Guix
proposed here.  Scripting works well enough that our 900'ish packages
can be rebuilt with the change, many of which come with a test suite.

Thanks for having me think harder about if this is really
desirable/needed.  For me the main plus is that we'd be adhering to the
current standards used for Emacs packaging: everything is unpacked into
a single directory owned by that package, so as long as we include any
needed resources in the #:include argument of the emacs-build-system, it
works as upstream intended it, with no need to patch variables or
anything.  I expect this situation to worsen as Emacs packages tend to
get more complicated, and I don't feel strongly enough about it to go
arguing about Emacs packaging standards on the Emacs mailing list :-).

On the minus side it makes the startup slightly more expensive in terms
of pure scripting, and requires the users to be mindful that
site-start.el evaluation is needed for the user installed packages to be
activated.

Further thoughts?

Thanks again,

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.