GNU bug report logs - #72453
29.4; "use-package" macro does not load all expected related files

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 72453 in the body.
You can then email your comments to 72453 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Derek Upham <derek_upham <at> mailfence.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.4; "use-package" macro does not load all expected related files
Date: Sat, 03 Aug 2024 14:58:08 -0700

This problem exists for the current emacs-30 branch HEAD.

The following "init.el" file contents provide a test case:

  (defun use-package-ensure-trace (name args state)
    (warn "use-package-ensure-trace %s %s %s" name args state)
    t)

  ;; (require 'use-package)

  (setq use-package-ensure-function 'use-package-ensure-trace)
  (setq use-package-always-ensure t)

  (use-package faces)
  (use-package ansi-color)
  (use-package files :ensure t)
  (use-package rx)
  (use-package avoid)

Place it in a scratch directory "use-package-ensure-test" and invoke
Emacs with, for example:

  emacs --init-directory use-package-ensure-test

The init file sets use-package-always-ensure to true, and we expect
trace warnings to show up for each use-package invocation.  In fact, we
only see them for the third and beyond.

 Warning (emacs): use-package-ensure-trace files (t) nil
 Warning (emacs): use-package-ensure-trace rx (t) nil
 Warning (emacs): use-package-ensure-trace avoid (t) nil

Uncomment the "require" line in the init file and we see all of the
traces that we expect:

 Warning (emacs): use-package-ensure-trace faces (t) nil
 Warning (emacs): use-package-ensure-trace ansi-color (t) nil
 Warning (emacs): use-package-ensure-trace files (t) nil
 Warning (emacs): use-package-ensure-trace rx (t) nil
 Warning (emacs): use-package-ensure-trace avoid (t) nil

The root cause is the structure of the "use-package/use-package*.el"
libraries.  The "use-package" macro exists in "use-package-core.el", and
has an autoload.  When Emacs evalutes a use-package form, Emacs loads
"use-package-core.el" only.  It does not load any of the other
"use-package*.el" components.  That means that Emacs doesn't evaluate
any of the `add-to-list' forms at the bottom of "use-package-ensure.el".
As a result, the use-package-defaults mechanism doesn't start out with
the expected :ensure or :pin behaviors.

But the first time we refer to an :ensure property explicitly, that is
enough to trigger the autoload of "use-package-handler/:ensure", which
loads the file and sets up the defaults.  All later invocations run the
:ensure checks as expected.

Moving the "use-package" macro into the top-level "use-package" file
should fix the problem.  The "use-package" file explicitly loads the
other files, which sets up all of the defaults.



In GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.42,
 cairo version 1.18.0) of 2024-06-29, modified by Debian built on sbuild
Windowing system distributor 'The X.Org Foundation', version 11.0.12101011
System Description: Debian GNU/Linux trixie/sid

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --with-cairo --with-x=yes
 --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -Werror=implicit-function-declaration
 -ffile-prefix-map=/build/reproducible-path/emacs-29.4+1=. -fstack-protector-strong
 -fstack-clash-protection -Wformat -Werror=format-security
 -fcf-protection -Wall' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
 LDFLAGS=-Wl,-z,relro'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2
XPM GTK3 ZLIB

-- 
Derek Upham
derek_upham <at> mailfence.com

Message #8 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Derek Upham <derek_upham <at> mailfence.com>,
 John Wiegley <johnw <at> gnu.org>
Cc: 72453 <at> debbugs.gnu.org
Subject: Re: bug#72453: 29.4;
 "use-package" macro does not load all expected related files
Date: Sun, 04 Aug 2024 07:44:43 +0300

> Date: Sat, 03 Aug 2024 14:58:08 -0700
> From:  Derek Upham via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> 
> This problem exists for the current emacs-30 branch HEAD.
> 
> The following "init.el" file contents provide a test case:
> 
>   (defun use-package-ensure-trace (name args state)
>     (warn "use-package-ensure-trace %s %s %s" name args state)
>     t)
> 
>   ;; (require 'use-package)
> 
>   (setq use-package-ensure-function 'use-package-ensure-trace)
>   (setq use-package-always-ensure t)
> 
>   (use-package faces)
>   (use-package ansi-color)
>   (use-package files :ensure t)
>   (use-package rx)
>   (use-package avoid)
> 
> Place it in a scratch directory "use-package-ensure-test" and invoke
> Emacs with, for example:
> 
>   emacs --init-directory use-package-ensure-test

Is this step necessary, i.e. does this problem has anything to do with
the non-default --init-directory setting?

> The init file sets use-package-always-ensure to true, and we expect
> trace warnings to show up for each use-package invocation.  In fact, we
> only see them for the third and beyond.
> 
>  Warning (emacs): use-package-ensure-trace files (t) nil
>  Warning (emacs): use-package-ensure-trace rx (t) nil
>  Warning (emacs): use-package-ensure-trace avoid (t) nil
> 
> Uncomment the "require" line in the init file and we see all of the
> traces that we expect:
> 
>  Warning (emacs): use-package-ensure-trace faces (t) nil
>  Warning (emacs): use-package-ensure-trace ansi-color (t) nil
>  Warning (emacs): use-package-ensure-trace files (t) nil
>  Warning (emacs): use-package-ensure-trace rx (t) nil
>  Warning (emacs): use-package-ensure-trace avoid (t) nil
> 
> The root cause is the structure of the "use-package/use-package*.el"
> libraries.  The "use-package" macro exists in "use-package-core.el", and
> has an autoload.  When Emacs evalutes a use-package form, Emacs loads
> "use-package-core.el" only.  It does not load any of the other
> "use-package*.el" components.  That means that Emacs doesn't evaluate
> any of the `add-to-list' forms at the bottom of "use-package-ensure.el".
> As a result, the use-package-defaults mechanism doesn't start out with
> the expected :ensure or :pin behaviors.
> 
> But the first time we refer to an :ensure property explicitly, that is
> enough to trigger the autoload of "use-package-handler/:ensure", which
> loads the file and sets up the defaults.  All later invocations run the
> :ensure checks as expected.
> 
> Moving the "use-package" macro into the top-level "use-package" file
> should fix the problem.  The "use-package" file explicitly loads the
> other files, which sets up all of the defaults.

John, any comments or suggestions?

Message #11 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: John Wiegley <johnw <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 72453 <at> debbugs.gnu.org, Derek Upham <derek_upham <at> mailfence.com>
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Sat, 03 Aug 2024 23:01:04 -0700

>>>>> Eli Zaretskii <eliz <at> gnu.org> writes:

>> Moving the "use-package" macro into the top-level "use-package" file should
>> fix the problem. The "use-package" file explicitly loads the other files,
>> which sets up all of the defaults.

> John, any comments or suggestions?

I didn’t write any of the :ensure logic, so I have no comments there, but the
suggestion above sounds like a sound and harmless one to me.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Message #14 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: John Wiegley <johnw <at> gnu.org>,
 Stefan Kangas <stefankangas <at> gmail.com>
Cc: 72453 <at> debbugs.gnu.org, derek_upham <at> mailfence.com
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Sun, 04 Aug 2024 11:14:33 +0300

merge 72453 72293
thanks

> From: John Wiegley <johnw <at> gnu.org>
> Cc: Derek Upham <derek_upham <at> mailfence.com>,  72453 <at> debbugs.gnu.org
> Date: Sat, 03 Aug 2024 23:01:04 -0700
> 
> >>>>> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> Moving the "use-package" macro into the top-level "use-package" file should
> >> fix the problem. The "use-package" file explicitly loads the other files,
> >> which sets up all of the defaults.
> 
> > John, any comments or suggestions?
> 
> I didn’t write any of the :ensure logic, so I have no comments there, but the
> suggestion above sounds like a sound and harmless one to me.

Thanks, John.

Stefan, would you please handle this?  Moving use-package from
use-package-core.el to use-package.el would probably mean much more
massive load whenever the macro is used, and I wonder whether there's
a more light-weight solution.  If such a light-weight solution exists,
I think we should install it on the emacs-30 release branch, but if
the only reasonable solution is what the OP suggested, then I think it
should go to master.

Btw, I think this bug and bug#72293 are the same bug, so I'm merging
them.

Message #19 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Derek Upham <derek_upham <at> mailfence.com>
To: Eli Zaretskii <eliz <at> gnu.org>, John Wiegley <johnw <at> gnu.org>
Cc: 72453 <at> debbugs.gnu.org
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Sun, 4 Aug 2024 16:46:57 +0200 (CEST)

[Message part 1 (text/plain, inline)]

On Aug 3, 2024 at 9:44 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:> Place it in a scratch directory "use-package-ensure-test" and invoke
> Emacs with, for example:
> 
>   emacs --init-directory use-package-ensure-test

Is this step necessary, i.e. does this problem has anything to do with
the non-default --init-directory setting?

It's an outcome  of ease-of-reproduction.  This approach sets up a
minimal test case (with "-q", effectively), in a single file, with the
actions happening during the startup as typical for use-package.

-- 
Sent with https://mailfence.com
Secure and private email

[Message part 2 (text/html, inline)]

Message #22 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Derek Upham <derek_upham <at> mailfence.com>
To: Eli Zaretskii <eliz <at> gnu.org>, John Wiegley <johnw <at> gnu.org>,
 Stefan Kangas <stefankangas <at> gmail.com>
Cc: 72453 <at> debbugs.gnu.org
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Sun, 4 Aug 2024 16:56:16 +0200 (CEST)

[Message part 1 (text/plain, inline)]

On Aug 4, 2024 at 1:14 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:Stefan, would you please handle this?  Moving use-package from
use-package-core.el to use-package.el would probably mean much more
massive load whenever the macro is used, and I wonder whether there's
a more light-weight solution.  If such a light-weight solution exists,
I think we should install it on the emacs-30 release branch, but if
the only reasonable solution is what the OP suggested, then I think it
should go to master.

Alternatively, create a new file in the use-package directory that
consolidates the top-level macro and the various add-to-list calls.
(I expect that :delight and :diminish have the same problem as
:ensure and :pin do.)  Maybe "use-package-bootstrap.el".

-- 
Sent with https://mailfence.com
Secure and private email

[Message part 2 (text/html, inline)]

Message #25 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Derek Upham <derek_upham <at> mailfence.com>, stefankangas <at> gmail.com
Cc: johnw <at> gnu.org, 72453 <at> debbugs.gnu.org
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Sat, 17 Aug 2024 11:24:33 +0300

Ping! Stefan, could you please look into this?

> Date: Sun, 4 Aug 2024 16:56:16 +0200 (CEST)
> From: Derek Upham <derek_upham <at> mailfence.com>
> Cc: 72453 <at> debbugs.gnu.org
> 
> On Aug 4, 2024 at 1:14 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>  Stefan, would you please handle this?  Moving use-package from
> use-package-core.el to use-package.el would probably mean much more
> massive load whenever the macro is used, and I wonder whether there's
> a more light-weight solution.  If such a light-weight solution exists,
> I think we should install it on the emacs-30 release branch, but if
> the only reasonable solution is what the OP suggested, then I think it
> should go to master.
> 
> Alternatively, create a new file in the use-package directory that
> consolidates the top-level macro and the various add-to-list calls.
> (I expect that :delight and :diminish have the same problem as
> :ensure and :pin do.)  Maybe "use-package-bootstrap.el".

Message #28 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Derek Upham <derek_upham <at> mailfence.com>, 72453 <at> debbugs.gnu.org
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Sun, 18 Aug 2024 11:21:40 +0000

[Message part 1 (text/plain, inline)]

Derek Upham via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs <at> gnu.org> writes:

>   emacs --init-directory use-package-ensure-test
>
> The init file sets use-package-always-ensure to true, and we expect
> trace warnings to show up for each use-package invocation.  In fact, we
> only see them for the third and beyond.
>
>  Warning (emacs): use-package-ensure-trace files (t) nil
>  Warning (emacs): use-package-ensure-trace rx (t) nil
>  Warning (emacs): use-package-ensure-trace avoid (t) nil

Thanks for the bug report and the clear recipe to reproduce it.

I can reproduce this bug here also.

> Moving the "use-package" macro into the top-level "use-package" file
> should fix the problem.  The "use-package" file explicitly loads the
> other files, which sets up all of the defaults.

The attached less intrusive patch fixes it here.  Could you please test
that it fixes the problem for you?

The patch just moves the relevant definitions to use-package-core.el.
If Eli and Andrea see no problem with this approach, I think we could
install it on emacs-30, after the first pre-release, and otherwise on
master.

Unrelatedly, I also noticed that the defcustoms in use-package-core.el
are all missing :version tags.  I'll fix that on emacs-30 after the
first pre-release is out.

[0001-Fix-early-use-of-use-package-always-ensure.patch (text/x-patch, attachment)]

Message #31 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: 72453 <at> debbugs.gnu.org, derek_upham <at> mailfence.com
Subject: Re: bug#72453: 29.4;
 "use-package" macro does not load all expected related files
Date: Sun, 18 Aug 2024 14:31:24 +0300

> From: Stefan Kangas <stefankangas <at> gmail.com>
> Date: Sun, 18 Aug 2024 11:21:40 +0000
> 
> The attached less intrusive patch fixes it here.  Could you please test
> that it fixes the problem for you?
> 
> The patch just moves the relevant definitions to use-package-core.el.
> If Eli and Andrea see no problem with this approach, I think we could
> install it on emacs-30, after the first pre-release, and otherwise on
> master.

I think this should go to master, since the problem is not terribly
important to fix, and moving stuff between files could cause
unintended consequences.

Thanks.

Message #34 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Derek Upham <derek_upham <at> mailfence.com>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: 72453 <at> debbugs.gnu.org
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Sun, 18 Aug 2024 18:26:22 -0700

Stefan Kangas <stefankangas <at> gmail.com> writes:

>>  Warning (emacs): use-package-ensure-trace files (t) nil
>>  Warning (emacs): use-package-ensure-trace rx (t) nil
>>  Warning (emacs): use-package-ensure-trace avoid (t) nil
>
> Thanks for the bug report and the clear recipe to reproduce it.
>
> I can reproduce this bug here also.
>
>> Moving the "use-package" macro into the top-level "use-package" 
>> file
>> should fix the problem.  The "use-package" file explicitly 
>> loads the
>> other files, which sets up all of the defaults.
>
> The attached less intrusive patch fixes it here.  Could you 
> please test
> that it fixes the problem for you?

Validating on stock 29.4, with this command:

  ~/emacs-29.4/bin/emacs --init-directory use-package-ensure-test

and this test code:

(defun use-package-ensure-trace (name args state)
  (warn "use-package-ensure-trace %s %s %s" name args state)
  t)

;;(require 'use-package)

(setq use-package-ensure-function 'use-package-ensure-trace)
(setq use-package-always-ensure t)

(use-package faces)
(use-package ansi-color)
(use-package files :ensure t)
(use-package rx)
(use-package avoid)

I’m seeing the outputs that we expect:

Warning (emacs): use-package-ensure-trace faces (t) nil
Warning (emacs): use-package-ensure-trace ansi-color (t) nil
Warning (emacs): use-package-ensure-trace files (t) nil
Warning (emacs): use-package-ensure-trace rx (t) nil
Warning (emacs): use-package-ensure-trace avoid (t) nil

Thanks for working on this.

-- 
Derek Upham
derek_upham <at> mailfence.com

Message #39 received at 72453-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Derek Upham <derek_upham <at> mailfence.com>
Cc: 72453-done <at> debbugs.gnu.org
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Mon, 19 Aug 2024 16:58:22 -0700

Version: 31.1

Derek Upham <derek_upham <at> mailfence.com> writes:

> I’m seeing the outputs that we expect:
>
>  Warning (emacs): use-package-ensure-trace faces (t) nil
>  Warning (emacs): use-package-ensure-trace ansi-color (t) nil
>  Warning (emacs): use-package-ensure-trace files (t) nil
>  Warning (emacs): use-package-ensure-trace rx (t) nil
>  Warning (emacs): use-package-ensure-trace avoid (t) nil

Thanks for testing.

So I've pushed this to master as commit 644d2a83316.  I'm consequently
closing this bug report.

Message #47 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: 72453 <at> debbugs.gnu.org, Derek Upham <derek_upham <at> mailfence.com>
Subject: bug#72453: 29.4; "use-package" macro does not load all expected
 related files 
Date: Wed, 21 Aug 2024 14:46:25 +0200

[Message part 1 (text/plain, inline)]

It seems that 644d2a83 and 8f4ad4d7 cause failures in use-package-tests on master (log attached); the tests pass after reverting those commits.

[use-package-tests.log (application/octet-stream, attachment)]

Message #50 received at 72453 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: 72453 <at> debbugs.gnu.org, Derek Upham <derek_upham <at> mailfence.com>
Subject: Re: bug#72453: 29.4; "use-package" macro does not load all expected
 related files
Date: Sun, 1 Sep 2024 02:32:36 -0700

Mattias Engdegård <mattias.engdegard <at> gmail.com> writes:

> It seems that 644d2a83 and 8f4ad4d7 cause failures in use-package-tests on master (log attached); the tests pass after reverting those commits.

Thanks, this should now be fixed.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.