Subject: bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

From: Eli Zaretskii <[email-masked]>
Date: Thu, 13 Apr 2023 08:49:46 +0300

> From: Philip Kaludercic <[email-masked]>
> Cc: [email-masked],  [email-masked],  [email-masked],
>   [email-masked]
> Date: Wed, 12 Apr 2023 20:10:50 +0000
> 
> >> > And I thought we all agreed built-in packages need special treatment
> >> > anyway, didn't we?  Then why having a separate command is not a
> >> > natural next step?
> >> 
> >> I don't necessarily agree that "special treatment" requires a separate
> >> command.
> >
> > Even if you don't agree with that in general, having a separate
> > command would allow us to install that command on emacs-29 without any
> > fears.  So that alone is a significant advantage, even if the rest are
> > not yet agreed upon.
> 
> Would this be a permanent thing, or would the command be deprecated by
> emacs-30?  I get the need under the circumstances, but it doesn't seem
> like the best solution if we were to set aside release-concerns.

We could keep the separate command on master as well, at least until
it is obvious that a special command is not needed in any use case.
The time to revisit this is when we take some practical steps towards
removing from emacs.git packages that are on ELPA and figuring out how
to bundle them with a release tarball and how to let users upgrade
them after they install a released Emacs.  It could be that when we
implement all that, there will be some aspects of upgrading core
packages that will benefit from having a separate command.  If we find
this is not needed, we can deprecate the command at that time.

Mind you, the command I have in mind is one that will allow upgrading
_only_ built-in packages, i.e. it will not deal with the packages that
don't come with an Emacs tarball.

> >> I think it is wrong the assume that an built-in package should
> >> automatically be updated to a ELPA package whenever possible.
> >
> > This seems to be an argument in favor of a separate command?  Or what
> > did you mean by that, and how is it related to the issue at hand?
> 
> I don't think package-update should switch from a built-in package to a
> package that was installed from ELPA.  The user should at least once
> commit to the switch (be it with a separate command or package-install).

I tend to agree.  Although, once again, we should revisit this when
core packages are no longer in emacs.git.  Taking a perhaps somewhat
different analogy of updating my smartphone, there's no particular
difference between updating a "built-in" app, one that is part of the
smartphone's system components and came with it OOTB, and updating any
other app, including those I installed myself.  I think the same is
true for updating system components and applications from any modern
distro, isn't it?  So this is yet to be decided, but for now we can
use this more conservative policy.

> >  . install your changes on master only, and leave the problem of
> >    updating a core package unsolved in Emacs 29 (with the workaround
> >    mentioned in the beginning of this bug's discussion available to
> >    alleviate the problem to some extent)
> >  . come up with safer changes for package-install that could be
> >    installed on emacs-29
> >  . add a new command for updating a core package, which can then be
> >    safely installed on emacs-29
> >
> > The last 2 alternatives can be for emacs-29 only, whereas on master we
> > install your proposed change (or something else).
> 
> I would like to investigate option 2, but if nothing is found we can
> fall back to 3.  But even if there are issues in this case, I don't
> consider the matter in general to be that drastic.  If all Joao wants is
> to avoid confusion, we can also improve the error message that
> `package-install' generates when it says that a package ins already
> installed.

Yes, I think if we go with the modified package-install or
package-update, then invoking them in a way that doesn't update
built-in packages should reflect that in the error message, and also
tell how to invoke them to be able to upgrade a built-in package.

> >> One point that might be deliberated is that this means all built-in
> >> dependencies are also installed, even if these are not strictly
> >> necessary.  It shouldn't matter that much, since most users would
> >> upgrade them eventually, but worth mentioning I guess? 
> >
> > That just confirms my fears that we are opening a Pandora's box.  We
> > have no idea what this will do, and no time to test the results.
> > Unintended consequences are abundant.  We must draw any such
> > consequences to the absolute minimum, at least the way the commands
> > work by default.  Even if the result is less than elegant.
> 
> Intuitively I would want to argue that this change has an upper-bound to
> how much harm it could do, but as I cannot prove it in any way I'll
> rather not assert that the point.

There is, of course, an upper bound.  But I don't think we know how
high that is at this time.  Bitter experience has taught me that we
are not very good in predicting that.

> >> +  (and (not (assq (cond
> >> +                   ((package-desc-p package)
> >> +                    (package-desc-name package))
> >> +                   ((stringp package) (intern package))
> >> +                   ((symbolp package) package)
> >> +                   ((error "Unknown package format: %S" package)))
> >> +                  (package--alist)))
> >> +       (package-built-in-p package)))
> >
> > Why do we need all these conditions, where we didn't need them in the
> > current code?
> 
> Practically speaking these conditions are not necessary, I just added it
> in case there was the need to use the function elsewhere at some point.

Then maybe these conditions should be only on master.

> > Also, package-alist is documented as "alist of all packages available
> > for activation", so it is not clear how the fact that assq returns nil
> > is evidence that "the user has not installed a more recent version".
> > Looking at what package--alist and package-load-all-descriptors do, it
> > looks like they just collect packages that were downloaded into the
> > relevant directories?  Is that enough to consider any package not in
> > the list to be "not installed"?  
> 
> The point is that package.el will add all packages it manages (ie. are
> "available to activation") to package-alist.  Built-in/core packages are
> not managed by package.el, but just "acknowledged" via
> `package--builtins'.  So this looks like a safe assumption to me.

These subtleties should ideally be documented somewhere in package.el.
Without being aware of all that, it is hard to read the code of
package.el and understand what it does and why, and the names and doc
strings of the relevant symbols in most cases don't help, as they all
use vague and too-broad words to describe their purpose and effects.

> >                                  And what about the "more recent
> > version" condition -- this doesn't seem to be tested anywhere in
> > package--alist?
> 
> You are right, but this is a misphrasing.

How so? what should be said instead?