GNU bug report logs - #48117
28.0.50; Update of loaddefs.el during normal build is unreliable

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 48117 in the body.
You can then email your comments to 48117 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50; Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 14:51:11 +0300

If you keep rebuilding Emacs from the Git repository using just the
"make -jN" commands, to track the development branches, after some
time loaddefs.el becomes outdated.  To see the outdated parts, rename
loaddefs.el and then say

  $ make -C lisp autoloads-force

Then compare the old loaddefs.el with the newly created one: you will
see many changes, depending on when was the last time you bootstrapped
or otherwise regenerated loaddefs.el from scratch.

This happens because regeneration of the parts of loaddefs.el affected
by Lisp changes is unreliable and misses some changes, in particular
those where autoloads from some Lisp files are redirected to private
*-loaddefs.el files instead of the common loaddefs.el.

These updates should happen automatically, they should not require
people to "make bootstrap" or manually regenerate loaddefs.el.

In GNU Emacs 28.0.50 (build 122, i686-pc-mingw32)
 of 2021-04-30 built on HOME-C4E4A596F7
Repository revision: ab7a61e0efd0684bc37a556d12f36521f9f61782
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int --with-native-compilation
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs auth-source eieio eieio-core eieio-loaddefs
password-cache json map text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils comp
comp-cstr warnings subr-x rx cl-seq cl-macs cl-extra help-mode seq
byte-opt gv cl-loaddefs cl-lib bytecomp byte-compile cconv iso-transl
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process nativecomp emacs)

Memory information:
((conses 16 84237 12365)
 (symbols 48 8956 1)
 (strings 16 25167 3939)
 (string-bytes 1 768361)
 (vectors 16 16937)
 (vector-slots 8 295589 16641)
 (floats 8 28 124)
 (intervals 40 267 89)
 (buffers 888 11))

Message #8 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 15:01:15 +0000

>
> If you keep rebuilding Emacs from the Git repository using just the 
> "make -jN" commands, to track the development branches, after some time 
> loaddefs.el becomes outdated.  To see the outdated parts, rename 
> loaddefs.el and then say
>
>  $ make -C lisp autoloads-force
>
> Then compare the old loaddefs.el with the newly created one: you will 
> see many changes, depending on when was the last time you bootstrapped 
> or otherwise regenerated loaddefs.el from scratch.
>
> This happens because regeneration of the parts of loaddefs.el affected 
> by Lisp changes is unreliable and misses some changes, in particular 
> those where autoloads from some Lisp files are redirected to private 
> *-loaddefs.el files instead of the common loaddefs.el.
>
> These updates should happen automatically, they should not require 
> people to "make bootstrap" or manually regenerate loaddefs.el.
>

Would it not make sense to include these changes in their respective 
commits?

Another solution would be to have a cron job running somewhere and 
automatically pushing these changes once a day or so.

Message #11 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 18:03:39 +0300

> Date: Fri, 30 Apr 2021 15:01:15 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: 48117 <at> debbugs.gnu.org
> 
> > These updates should happen automatically, they should not require 
> > people to "make bootstrap" or manually regenerate loaddefs.el.
> >
> 
> Would it not make sense to include these changes in their respective 
> commits?

loaddefs.el is a generated file (and so are the other *-loaddefs.el
files).

> Another solution would be to have a cron job running somewhere and 
> automatically pushing these changes once a day or so.

How can we do that for files that aren't versioned?

Message #14 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 11:22:00 -0400

This issue has been present forever.
There are some comments in the Makefiles about it.
There are several issues, eg:

1) autoload generation is slow.

2) the dependencies of the loaddefs files are unknown to make,
and are basically "all lisp files". (You can't even say "just those
files with autoload statements", because removing a previously existing
autoload statement changes the output.)

3) Traditionally, re-making loaddefs files could make trivial changes
to the output that weren't important (eg ordering of the "no
autoloads" section, timestamping), but would still trigger re-dumping emacs.
Which could then trigger regeneration of the autoloads, and
re-dumping, etc.  This may be better nowadays, since there is no
longer timestamp information in the loaddefs files (see autoload-timestamps).

Message #17 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 11:38:43 -0400

One idea, don't know if workable:
always regenerate loaddefs, but using a temporary file name for the
main loaddefs.
use build-aux/move-if-change to only replace the real loaddefs file if
there have been changes. Although again, it's possible these would not
be "real" (ie significant) changes.

This would mean every invocation of make would generate a loaddefs file,
often for no need, which seems ugly.
It's a trade off between doing things properly (which is a bootstrap)
and efficiently, as has always been the case with Emacs's build.

Message #20 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 15:47:51 +0000

>>> These updates should happen automatically, they should not require 
>>> people to "make bootstrap" or manually regenerate loaddefs.el.
>>
>> Would it not make sense to include these changes in their respective 
>> commits?
>
> loaddefs.el is a generated file (and so are the other *-loaddefs.el 
> files).
>

Ah, yes, indeed.

>> Another solution would be to have a cron job running somewhere and 
>> automatically pushing these changes once a day or so.
>
> How can we do that for files that aren't versioned?
>

I have two questions:

Why are these files not versioned?

The generated lisp/loaddefs.el file is AFAICS identical to the 
lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
versioned.  Why is the lisp/loaddefs.el generated?

Message #23 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 18:48:34 +0300

> From: Glenn Morris <rgm <at> gnu.org>
> Cc: 48117 <at> debbugs.gnu.org
> Date: Fri, 30 Apr 2021 11:22:00 -0400
> 
> This issue has been present forever.

Yes, I know.  This isn't trivial, or else it would have been solved
long ago.

> 1) autoload generation is slow.

Based on my latest experience, I think this is somewhat exaggerated,
especially given that our builds became slower lately.

> 2) the dependencies of the loaddefs files are unknown to make,
> and are basically "all lisp files". (You can't even say "just those
> files with autoload statements", because removing a previously existing
> autoload statement changes the output.)
> 
> 3) Traditionally, re-making loaddefs files could make trivial changes
> to the output that weren't important (eg ordering of the "no
> autoloads" section, timestamping), but would still trigger re-dumping emacs.
> Which could then trigger regeneration of the autoloads, and
> re-dumping, etc.  This may be better nowadays, since there is no
> longer timestamp information in the loaddefs files (see autoload-timestamps).

What worries me the most is that when 'autoloads' is run (and it is,
from time to time), we still end up with outdated loaddefs.el.  I
think we could live with outdated loaddefs.el for short periods of
time, but it looks like running 'autoloads' only updates the part(s)
of the file for Lisp files that the build thinks to be responsible for
the update.  Or something like that, because how else to explain that
some parts remain outdated?

Message #26 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 18:51:28 +0300

> From: Glenn Morris <rgm <at> gnu.org>
> Cc: 48117 <at> debbugs.gnu.org
> Date: Fri, 30 Apr 2021 11:38:43 -0400
> 
> One idea, don't know if workable:
> always regenerate loaddefs, but using a temporary file name for the
> main loaddefs.
> use build-aux/move-if-change to only replace the real loaddefs file if
> there have been changes. Although again, it's possible these would not
> be "real" (ie significant) changes.

Thanks, I think it's worth trying.

Message #29 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 17:59:45 +0200

On Apr 30 2021, Glenn Morris wrote:

> always regenerate loaddefs, but using a temporary file name for the
> main loaddefs.
> use build-aux/move-if-change to only replace the real loaddefs file if
> there have been changes.

That's already what batch-update-autoloads does.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #32 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 19:00:55 +0300

> Date: Fri, 30 Apr 2021 15:47:51 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: 48117 <at> debbugs.gnu.org
> 
> Why are these files not versioned?

Because we try not to keep in Git any files that are generated by the
build.  It's redundant to have them there, and also causes frequent
unnecessary conflicts.

> The generated lisp/loaddefs.el file is AFAICS identical to the 
> lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
> versioned.  Why is the lisp/loaddefs.el generated?

ldefs-boot.el is versioned because it is needed for the initial build
of a fresh clone, and it is not identical to loaddefs.el, especially
not when you are developing.

Message #35 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 19:03:53 +0300

> From: Andreas Schwab <schwab <at> linux-m68k.org>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  48117 <at> debbugs.gnu.org
> Date: Fri, 30 Apr 2021 17:59:45 +0200
> 
> On Apr 30 2021, Glenn Morris wrote:
> 
> > always regenerate loaddefs, but using a temporary file name for the
> > main loaddefs.
> > use build-aux/move-if-change to only replace the real loaddefs file if
> > there have been changes.
> 
> That's already what batch-update-autoloads does.

Judging by the results, that's not so.  Specifically,
batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
only some of its part(s).

Message #38 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 18:21:07 +0200

On Apr 30 2021, Eli Zaretskii wrote:

> Judging by the results, that's not so.  Specifically,
> batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
> only some of its part(s).

Yes, because it calls update-directory-autoloads for each directory.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #41 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 13:10:02 -0400

Andreas Schwab wrote:

> That's already what batch-update-autoloads does.

Shows how much my memory is worth! :)

Message #44 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 13:25:36 -0400

Another idea, FWIW:
In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
older than ldefs-boot.el, start by copying the latter to the former.
This ought to limit how outdated loaddefs can get?

Message #47 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 17:32:13 +0000

>> The generated lisp/loaddefs.el file is AFAICS identical to the 
>> lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
>> versioned.  Why is the lisp/loaddefs.el generated?
>
> ldefs-boot.el is versioned because it is needed for the initial build of 
> a fresh clone, and it is not identical to loaddefs.el, especially not 
> when you are developing.
>

Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
regenerated?  ISTM that if one assumes that ldefs-boot.el is kept up to 
date, it is necessary to regenerate the loaddefs file only when 
ldefs-boot.el becomes more recent than the loaddefs file.  This would 
avoid rebuilding the loaddefs file on each make invocation.

Message #50 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 20:54:04 +0300

> From: Andreas Schwab <schwab <at> linux-m68k.org>
> Cc: rgm <at> gnu.org,  48117 <at> debbugs.gnu.org
> Date: Fri, 30 Apr 2021 18:21:07 +0200
> 
> On Apr 30 2021, Eli Zaretskii wrote:
> 
> > Judging by the results, that's not so.  Specifically,
> > batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
> > only some of its part(s).
> 
> Yes, because it calls update-directory-autoloads for each directory.

AFAIU, Glenn's proposal was to process all the directories.

Message #53 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 20:58:55 +0300

> From: Glenn Morris <rgm <at> gnu.org>
> Cc: 48117 <at> debbugs.gnu.org
> Date: Fri, 30 Apr 2021 13:25:36 -0400
> 
> Another idea, FWIW:
> In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> older than ldefs-boot.el, start by copying the latter to the former.
> This ought to limit how outdated loaddefs can get?

Yes, I think this is a good idea.

Message #56 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 21:57:02 +0300

> Date: Fri, 30 Apr 2021 17:32:13 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: 48117 <at> debbugs.gnu.org
> 
> Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
> regenerated?

I believe that's what Glenn suggested a few minutes ago.

Message #59 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 19:08:48 +0000

>> Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
>> regenerated?
>
> I believe that's what Glenn suggested a few minutes ago.
>

Yes, it's more or less the same idea, our mails crossed each other because 
of the lists.gnu.org delivery time.

Note that, as I said, this won't work without keeping ldefs-boot.el up to 
date.

Message #62 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 22:10:15 +0300

> Date: Fri, 30 Apr 2021 19:08:48 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: 48117 <at> debbugs.gnu.org
> 
> Note that, as I said, this won't work without keeping ldefs-boot.el up to 
> date.

We (read: Glenn) already do.  You can see that in "git log".

Message #65 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Fri, 30 Apr 2021 19:20:08 +0000

>> Note that, as I said, this won't work without keeping ldefs-boot.el up 
>> to date.
>
> We (read: Glenn) already do.  You can see that in "git log".
>

I think this should preferably be automated.  The current ldefs-boot.el 
file in the trunk is 25 days old, running admin/update_autogen -L adds 247 
lines and modifies 100 lines.

Message #68 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Fri, 30 Apr 2021 20:16:52 -0400

Gregory Heytings wrote:

> I think this should preferably be automated.  The current
> ldefs-boot.el file in the trunk is 25 days old, running
> admin/update_autogen -L adds 247 lines and modifies 100 lines.

It gets automaticallly updated on the first of the month.
It could be more frequent, but it would seem like churn for no real
reason most of the time.
(It wouldn't make any difference to the issue that prompted this report.)

Message #71 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sat, 01 May 2021 08:09:42 +0000

>> I think this should preferably be automated.  The current ldefs-boot.el 
>> file in the trunk is 25 days old, running admin/update_autogen -L adds 
>> 247 lines and modifies 100 lines.
>
> It gets automaticallly updated on the first of the month.  It could be 
> more frequent, but it would seem like churn for no real reason most of 
> the time.
>

IMO once a week would be better.  An outdated ldefs-boot creates 
unnecessary warnings during the build.

>
> (It wouldn't make any difference to the issue that prompted this 
> report.)
>

That depends on the chosen solution.  You suggested copying ldefs-boot.el 
onto loaddefs.el when it is more recent.  I'm not entirely sure, but it 
seems to me that forcing a regeneration of the loaddefs files during make 
whenever ldefs-boot.el is more recent than loaddefs.el would be a better 
solution.

Message #74 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sat, 01 May 2021 11:25:40 +0300

> Date: Sat, 01 May 2021 08:09:42 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: Eli Zaretskii <eliz <at> gnu.org>, 48117 <at> debbugs.gnu.org
> 
> You suggested copying ldefs-boot.el onto loaddefs.el when it is more
> recent.  I'm not entirely sure, but it seems to me that forcing a
> regeneration of the loaddefs files during make whenever
> ldefs-boot.el is more recent than loaddefs.el would be a better
> solution.

Your proposal would produce marginally better results for a
significantly longer build time, so I don't think it's a net win.

Message #77 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sat, 01 May 2021 09:20:51 +0000

>> You suggested copying ldefs-boot.el onto loaddefs.el when it is more 
>> recent.  I'm not entirely sure, but it seems to me that forcing a 
>> regeneration of the loaddefs files during make whenever ldefs-boot.el 
>> is more recent than loaddefs.el would be a better solution.
>
> Your proposal would produce marginally better results for a 
> significantly longer build time, so I don't think it's a net win.
>

If ldefs-boot.el is updated, say, once a week, this would force the 
regeneration of the loaddefs files at most once a week.  Wouldn't that be 
a reasonable compromise?  On my computer, regenerating the loaddefs files 
takes about 10 seconds, or ~3% of the time of a make bootstrap.

Message #80 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sat, 01 May 2021 12:34:46 +0300

> Date: Sat, 01 May 2021 09:20:51 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
> 
> > Your proposal would produce marginally better results for a 
> > significantly longer build time, so I don't think it's a net win.
> 
> If ldefs-boot.el is updated, say, once a week, this would force the 
> regeneration of the loaddefs files at most once a week.  Wouldn't that be 
> a reasonable compromise?

Compromise between which alternatives?

The "marginally better" results in your proposal are that if someone
updates from Git when he/she is in the middle of some development,
then loaddefs.el are made up-to-date immediately, as opposed to
_maybe_ waiting for the next update of ldefs-boot.el.  (I say "maybe"
because in general loaddefs.el _are_ updated as part of routine
builds, just not 100% reliably so.)

> On my computer, regenerating the loaddefs files takes about 10
> seconds, or ~3% of the time of a make bootstrap.

Keep in mind that some people use less powerful machines.  And the
bootstrap time is not relevant, because loaddefs.el is completely
regenerated during bootstrap anyway.  The time that is relevant is the
time of just "make", and that is usually quite short, even on slow
machines.

Message #83 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sat, 01 May 2021 12:29:12 +0000

>>> Your proposal would produce marginally better results for a 
>>> significantly longer build time, so I don't think it's a net win.
>>
>> If ldefs-boot.el is updated, say, once a week, this would force the 
>> regeneration of the loaddefs files at most once a week.  Wouldn't that 
>> be a reasonable compromise?
>
> Compromise between which alternatives?
>

I see at least the following possible alternatives:

- the current situation, which you describe in your original post, in 
which the loaddefs files need to be regenerated manually (inconvenient)

- regenerating the loaddefs files for each make invocation (inefficient)

- copying the ldefs-boot.el onto loaddefs.el when it is more recent (which 
IIUC could lose local additions to loaddefs.el)

- automatically regenerating the loaddefs files when ldefs-boot.el is more 
recent than loaddefs.el (which I understand could be a bit slow)

- issue only a warning when make is invoked and ldefs-boot.el is more 
recent than loaddefs.el (and perhaps add a autoloads or loaddefs target to 
the main Makefile)

Message #86 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sat, 01 May 2021 16:00:53 +0300

> Date: Sat, 01 May 2021 12:29:12 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
> 
> > Compromise between which alternatives?
> >
> 
> I see at least the following possible alternatives:

My point is that the gain from your proposal wrt the next best
alternative is small, but the price is too high.

Message #89 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Glenn Morris <rgm <at> gnu.org>, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sun, 02 May 2021 09:28:04 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> 1) autoload generation is slow.
>
> Based on my latest experience, I think this is somewhat exaggerated,
> especially given that our builds became slower lately.

Has it?  I haven't noticed any substantial slow-down...  (Unless you
mean with native-comp switched on.)

In any case, autoload generation takes a substantial chunk of time --
especially since it's single-threaded.  Glenn's suggestion to copy over
ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
also like if that file was regenerated more often than once a month...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #92 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: rgm <at> gnu.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sun, 02 May 2021 10:41:01 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: Glenn Morris <rgm <at> gnu.org>,  48117 <at> debbugs.gnu.org
> Date: Sun, 02 May 2021 09:28:04 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> 1) autoload generation is slow.
> >
> > Based on my latest experience, I think this is somewhat exaggerated,
> > especially given that our builds became slower lately.
> 
> Has it?  I haven't noticed any substantial slow-down...  (Unless you
> mean with native-comp switched on.)

Yes, native-comp, certainly.  But the build became slower a year or so
ago as well, I no longer remember why.

> In any case, autoload generation takes a substantial chunk of time --
> especially since it's single-threaded.  Glenn's suggestion to copy over
> ldefs-boot.el (if it's newer) sounds like a good idea to me.

Yes, I agree.

Message #95 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Sun, 02 May 2021 12:59:17 -0400

Glenn Morris wrote:

> Another idea, FWIW:
> In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> older than ldefs-boot.el, start by copying the latter to the former.
> This ought to limit how outdated loaddefs can get?

BTW, I think this might actually make things worse in terms of detecting
new secondary loaddefs files like texinfo-loaddefs (not that those
happen very often), by bumping the timestamp on the primary loaddefs.

Message #98 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Sun, 02 May 2021 21:54:42 +0300

> From: Glenn Morris <rgm <at> gnu.org>
> Cc: 48117 <at> debbugs.gnu.org
> Date: Sun, 02 May 2021 12:59:17 -0400
> 
> Glenn Morris wrote:
> 
> > Another idea, FWIW:
> > In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> > older than ldefs-boot.el, start by copying the latter to the former.
> > This ought to limit how outdated loaddefs can get?
> 
> BTW, I think this might actually make things worse in terms of detecting
> new secondary loaddefs files like texinfo-loaddefs (not that those
> happen very often), by bumping the timestamp on the primary loaddefs.

But there's no such thing as detecting new secondary loaddefs, right?
The only remedy for that is "make autoloads-force", right?

Message #101 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Mon, 03 May 2021 08:41:49 +0000

>> Another idea, FWIW: In the make rule for $(lisp)/loaddefs.el, if 
>> loaddefs.el exists and is older than ldefs-boot.el, start by copying 
>> the latter to the former. This ought to limit how outdated loaddefs can 
>> get?
>
> BTW, I think this might actually make things worse in terms of detecting 
> new secondary loaddefs files like texinfo-loaddefs (not that those 
> happen very often), by bumping the timestamp on the primary loaddefs.
>

Which is one of the reasons why I still believe that my proposed solution 
is better: whenever lisp/ldefs-boot.el is more recent than 
lisp/loaddefs.el, issue a warning when make is invoked to suggest the 
regeneration of _all_ autoload files, and add a command to do this in the 
main Makefile: make autoloads would do find -name '*loaddefs.el' -delete 
&& make -C lisp autoloads.  In theory this should have the same effect as 
make -C lisp autoloads-force, but that command is only marginally faster, 
and I much prefer the clarity / determinicity of a "delete all and 
regenerate" operation.

Message #104 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Glenn Morris <rgm <at> gnu.org>, Eli Zaretskii <eliz <at> gnu.org>,
 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Wed, 8 Dec 2021 01:36:25 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> In any case, autoload generation takes a substantial chunk of time --
> especially since it's single-threaded.  Glenn's suggestion to copy over
> ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
> also like if that file was regenerated more often than once a month...)

If Eli agrees, I'm happy to update ldefs-boot.el more frequently.

Just let me know how often you want it updated and I will make it so.

Message #107 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: Glenn Morris <rgm <at> gnu.org>, Eli Zaretskii <eliz <at> gnu.org>,
 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Wed, 08 Dec 2021 01:40:14 +0100

Stefan Kangas <stefan <at> marxist.se> writes:

> If Eli agrees, I'm happy to update ldefs-boot.el more frequently.
>
> Just let me know how often you want it updated and I will make it so.

I usually just update it now if I add an ;;;###autoload somewhere --
updating that file kinda goes hand in hand with that.  (I added a new
"make ldefs-boot.el" Makefile target to make it more convenient.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #110 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: rgm <at> gnu.org, larsi <at> gnus.org, 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50;
 Update of loaddefs.el during normal build is unreliable
Date: Wed, 08 Dec 2021 05:32:16 +0200

> From: Stefan Kangas <stefan <at> marxist.se>
> Date: Wed, 8 Dec 2021 01:36:25 +0100
> Cc: Eli Zaretskii <eliz <at> gnu.org>, Glenn Morris <rgm <at> gnu.org>, 48117 <at> debbugs.gnu.org
> 
> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
> 
> > In any case, autoload generation takes a substantial chunk of time --
> > especially since it's single-threaded.  Glenn's suggestion to copy over
> > ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
> > also like if that file was regenerated more often than once a month...)
> 
> If Eli agrees, I'm happy to update ldefs-boot.el more frequently.

I see no down-sides to updating it frequently, so it's fine with me.

Message #113 received at 48117 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 48117 <at> debbugs.gnu.org
Subject: Re: bug#48117: 28.0.50; Update of loaddefs.el during normal build
 is unreliable
Date: Sun, 05 Jun 2022 18:25:37 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Then compare the old loaddefs.el with the newly created one: you will
> see many changes, depending on when was the last time you bootstrapped
> or otherwise regenerated loaddefs.el from scratch.
>
> This happens because regeneration of the parts of loaddefs.el affected
> by Lisp changes is unreliable and misses some changes, in particular
> those where autoloads from some Lisp files are redirected to private
> *-loaddefs.el files instead of the common loaddefs.el.

This should now be fixed in Emacs 29 -- the private loaddefs files
updates should be handled as well as the main one.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.