GNU bug report logs - #59692
29.0.60; Reduce installation size by disabling byte-compilation for some lisp files that won't benefit

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 59692 in the body.
You can then email your comments to 59692 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: daanturo <daanturo <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.0.60; Reduce installation size by disabling byte-compilation for
 some lisp files that won't benefit
Date: Wed, 30 Nov 2022 03:07:47 +0700

[Message part 1 (text/plain, inline)]

Recently, after each rebuild, I noticed that Emacs' total installation
size has become bigger overtime. That's understandably unavoidable but
there's room for improvement. By analyzing disk usage I discovered
that:

"lisp/leim/"'s compiled elisp files are taking the most significant disk
storage.

By inspecting some of those, I don't think we are able to gain any
practical benefit by compiling them, they don't contain (~)any
function/macro definitions at all.

So I try to disable byte-compilation (by either dir-local vars or
file-local vars) in:

- lisp/leim/ : I disable for the whole directory

And some more .el(s) that I don't find so essential to compile:

- lisp/play/ : not so serious or critical to workflows

- lisp/cedet/{ede,semantic,srecode}/ : we now have LSP/eglot!
- lisp/cedet/{ede.el,semantic.el} : same as above, also a big file by
itself

(I wished pulse.el, mode-local.el, data-debug.el were moved out of this
lisp/cedet/, then putting a single .dir-local.el there would be the
simplest)

- lisp/obsolete/ : already obsolete anyway, but some packages may still
depend on them so no-native-compile only

Then build Emacs again.

The final size of my Emacs installation, compared to before applying
those no-compile local variables:
298.79 MiB -> 253.09 MiB
That's 45.7 MiB disk saved. About 15.3% of total shaved!

Please consider at least "lisp/leim/"'s gigantic files such as
"ZIRANMA.el", "ARRAY30.el", "ja-dic.el" ,etc. as I find those the most
disk-expensive when compiled, yet questionable benefit and waste of time
spending to build.


In GNU Emacs 29.0.60 (build 2, x86_64-pc-linux-gnu, GTK+ Version
3.24.35, cairo version 1.17.6) of 2022-11-30 built on dan-laptop
Repository revision: ec7908204e3d1d415a17813a1fac1a8de334d3b1
Repository branch: makepkg
System Description: Arch Linux

Configured using:
'configure --sysconfdir=/etc --prefix=/usr --libexecdir=/usr/lib
--localstatedir=/var
'--program-transform-name='\''s/\(ctags\)/\1.emacs/'\''' --with-json
--with-libsystemd --with-mailutils --with-modules
--with-native-compilation --with-pgtk --without-xaw3d --with-sound=alsa
--with-tree-sitter --with-xinput2 --with-xwidgets
--with-native-compilation=aot --without-compress-install
'CFLAGS=-march=native -O2 -pipe -fno-plt -fexceptions
-Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
-fstack-clash-protection -fcf-protection'
LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM XWIDGETS GTK3 ZLIB

-- 
Daanturo.

[0001-No-compile-for-some-lisp-directories-and-files.patch (text/x-patch, attachment)]

Message #8 received at 59692 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: daanturo <daanturo <at> gmail.com>
Cc: 59692 <at> debbugs.gnu.org
Subject: Re: bug#59692: 29.0.60;
 Reduce installation size by disabling byte-compilation for some lisp
 files that won't benefit
Date: Tue, 29 Nov 2022 22:18:01 +0200

severity 59692 wishlist
thanks

> Date: Wed, 30 Nov 2022 03:07:47 +0700
> From: daanturo <daanturo <at> gmail.com>
> 
> Recently, after each rebuild, I noticed that Emacs' total installation
> size has become bigger overtime. That's understandably unavoidable but
> there's room for improvement. By analyzing disk usage I discovered
> that:
> 
> "lisp/leim/"'s compiled elisp files are taking the most significant disk
> storage.
> 
> By inspecting some of those, I don't think we are able to gain any
> practical benefit by compiling them, they don't contain (~)any
> function/macro definitions at all.
> 
> So I try to disable byte-compilation (by either dir-local vars or
> file-local vars) in:
> 
> - lisp/leim/ : I disable for the whole directory
> 
> And some more .el(s) that I don't find so essential to compile:
> 
> - lisp/play/ : not so serious or critical to workflows
> 
> - lisp/cedet/{ede,semantic,srecode}/ : we now have LSP/eglot!
> - lisp/cedet/{ede.el,semantic.el} : same as above, also a big file by
> itself
> 
> (I wished pulse.el, mode-local.el, data-debug.el were moved out of this
> lisp/cedet/, then putting a single .dir-local.el there would be the
> simplest)
> 
> - lisp/obsolete/ : already obsolete anyway, but some packages may still
> depend on them so no-native-compile only
> 
> Then build Emacs again.
> 
> The final size of my Emacs installation, compared to before applying
> those no-compile local variables:
> 298.79 MiB -> 253.09 MiB
> That's 45.7 MiB disk saved. About 15.3% of total shaved!
> 
> Please consider at least "lisp/leim/"'s gigantic files such as
> "ZIRANMA.el", "ARRAY30.el", "ja-dic.el" ,etc. as I find those the most
> disk-expensive when compiled, yet questionable benefit and waste of time
> spending to build.

Thanks.

FWIW, I consider this to be a waste of our resources and a long-term
maintenance burden that I cannot justify, even if I accept your conclusions
(and I'm not at all convinced that they are completely correct, especially
since you didn't show any measurements of code speed with and without
byte-compilation).  50 MiB of disk space is too small to justify keeping
track of all the files, making decisions which will and will not be
compiled, etc.  Disk space is cheap these days.

So I don't think we should do this.

Message #15 received at 59692 <at> debbugs.gnu.org (full text, mbox):

From: daanturo <daanturo <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 59692 <at> debbugs.gnu.org
Subject: Re: bug#59692: 29.0.60; Reduce installation size by disabling
 byte-compilation for some lisp files that won't benefit
Date: Thu, 1 Dec 2022 01:58:45 +0700

[Message part 1 (text/plain, inline)]

> measurements of code speed with and without byte-compilation

I tried to evaluate performance different:

### Load time

Helper macro:
```emacs-lisp
(defmacro my-with-deferred-gc (&rest body)
  `(dlet ((gc-cons-threshold most-positive-fixnum) (gc-cons-percentage 0.75))
     ,@body))
```

(Each of those measures was executed in a new Emacs instance, as the
second `load` is (for me) cheaper than the first `load`).

#### lisp/leim/ja-dic

(my-with-deferred-gc (benchmark-progn (load
"/usr/share/emacs/29.0.60/lisp/leim/ja-dic/ja-dic")))

> .el:
Loading /usr/share/emacs/29.0.60/lisp/leim/ja-dic/ja-dic.el (source)...done
Elapsed time: 0.707891s
> .elc:
Loading /usr/share/emacs/29.0.60/lisp/leim/ja-dic/ja-dic...done
Elapsed time: 0.104832s
> .eln:
Loading /usr/share/emacs/29.0.60/lisp/leim/ja-dic/ja-dic (native compiled elisp)...done
Elapsed time: 0.104676s


#### Other auto-generated lisp files in lisp/leim/quail, all of them are
Chinese at the moment, the largests being ZIRANMA.el, ARRAY30.el:



(my-with-deferred-gc (benchmark-progn (load
"/usr/share/emacs/29.0.60/lisp/leim/quail/ZIRANMA")))

> .el:
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ZIRANMA.el (source)...done
Elapsed time: 0.037230s
> .elc:
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ZIRANMA...done
Elapsed time: 0.058042s
> .eln:
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ZIRANMA (native compiled elisp)...
Elapsed time: 0.062904s

(my-with-deferred-gc (benchmark-progn (load
"/usr/share/emacs/29.0.60/lisp/leim/quail/ARRAY30")))

> .el
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ARRAY30.el (source)...done
Elapsed time: 0.031518s
> .elc
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ARRAY30...done
Elapsed time: 0.037412s
> .eln
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ARRAY30 (native compiled elisp)...done
Elapsed time: 0.041110s


(my-with-deferred-gc (benchmark-progn (load
"/usr/share/emacs/29.0.60/lisp/leim/quail/QJ")))
> .el
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/QJ.el (source)...done
Elapsed time: 0.001649s
> .elc
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/QJ...done
Elapsed time: 0.004481s
> .eln
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/QJ (native compiled elisp)...done
Elapsed time: 0.005001s



For "ja-dic.el", byte-compilation clearly helped speedup loading. But
native-compilation doesn't offer any more significantly. As I inspect
"ja-dic.elc", byte-compilation did by expanding macros to
`(defconst var literal-list)` forms, I think that native compilation
won't offer more about run time performance.


For auto-generated files under lisp/leim/quail (ZIRANMA, ARRAY30, QJ
tested above), sometimes compilation even slows down the loading process
compared to just interpretation.


About the run time performance, since I don't know Chinese, I will
approximately benchmark lisp/leim/quail/vnvni.el (Vietnamese) instead as
this file is quite similar to ZIRANMA.el: both has only 2 forms --
(quail-define-package ...) and (quail-define-rules ...).

ydotool is used (https://github.com/ReimuNotMoe/ydotool) to perform
auto-typing as in bench-quail.el attached. The function to be
benchmarked is #'quail-input-method.

emacs -q -l bench-quail.el --eval "(progn (my-to-new-buffer) (my-benchmark-quail nil))"
emacs -q -l bench-quail.el --eval "(progn (my-to-new-buffer) (my-benchmark-quail 'compiled))"


Benchmarking results: (note that :num means the total key pressed by ydotool).

Loading /usr/share/emacs/29.0.60/lisp/leim/quail/vnvni.el (source)...done
Elapsed time: 0.016844s (0.008795s in 1 GCs)
(:compiled nil (:num 145 :total-time 2.118776153 :average 0.014612249331034484))


Loading /usr/share/emacs/29.0.60/lisp/leim/quail/vnvni (native compiled elisp)...done
Elapsed time: 0.034253s (0.009630s in 1 GCs)
(:compiled compiled (:num 145 :total-time 2.117245182999999 :average 0.014601690917241374))



So I conclude that those quail lisp package files' compilation statuses,
whether compiled or not, doesn't affect run-time performance at all.

### lisp/international/titdic-cnv.el's pinyin-convert

It produce a single form: (defconst var literal-list) so the
corresponding produced ".elc" file is just the same with new lines
joined. Also the times taken to load lisp/language/pinyin.elc and
/lisp/language/pinyin.el aren't different anyway.

### About the patch

In the patch below, I disabled native-compile for "ja-dic" while keeping
byte-compile (by modifying `generate-lisp-file-trailer` keyword args),
and disable compilation altogether for files produced by titdic-convert,
miscdic-convert, pinyin-convert. There's no new .dir-locals files
introduced like the previous patch.

> Disk space is cheap these days.

But it's always better to be light if we don't suffer any penalties. A
little number of megabytes saved will always be appreciated by people
with metered connection and/or limited bandwidth, or people who have
multiple versions of Emacs installed. Disk space is indeed becoming
cheaper, but with varying pace among countries. I am currently using a
laptop which only has 238GiB & un-expandable, that explains my joy when
pacman shows net upgrade size being negative during system updates.

Thanks to the patch, I saved 20.20 MiB by a couple of flag flips without
making any new files. Please consider again if you think it makes more
sense than my previous justifications, thank you.

-- 
Daanturo.

[bench-quail.el (text/x-emacs-lisp, attachment)]

[0001-Inhibit-compilation-for-some-quail-packages-generators.patch (text/x-patch, attachment)]

Message #20 received at 59692-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 59692-done <at> debbugs.gnu.org, daanturo <daanturo <at> gmail.com>
Subject: Re: bug#59692: 29.0.60; Reduce installation size by disabling
 byte-compilation for some lisp files that won't benefit
Date: Mon, 12 Dec 2022 17:04:20 -0800

Eli Zaretskii <eliz <at> gnu.org> writes:

> FWIW, I consider this to be a waste of our resources and a long-term
> maintenance burden that I cannot justify, even if I accept your conclusions
> (and I'm not at all convinced that they are completely correct, especially
> since you didn't show any measurements of code speed with and without
> byte-compilation).  50 MiB of disk space is too small to justify keeping
> track of all the files, making decisions which will and will not be
> compiled, etc.  Disk space is cheap these days.
>
> So I don't think we should do this.

I agree, so I'm closing this as wontfix.

Message #23 received at 59692-done <at> debbugs.gnu.org (full text, mbox):

From: daanturo <daanturo <at> gmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>
Cc: 59692-done <at> debbugs.gnu.org
Subject: Re: bug#59692: 29.0.60; Reduce installation size by disabling
 byte-compilation for some lisp files that won't benefit
Date: Tue, 13 Dec 2022 13:18:53 +0700

Please take a glance at my second attempt:
https://lists.gnu.org/archive/html/bug-gnu-emacs/2022-11/msg02818.html

I did proper benchmark, also the patch only changed a couple of lines without
creating any new files.

"""
lisp/international/ja-dic-cnv.el | 3 ++-
lisp/international/titdic-cnv.el | 6 +++---
2 files changed, 5 insertions(+), 4 deletions(-)
"""


On 13/12/2022 08:04, Stefan Kangas wrote:
> Eli Zaretskii <eliz <at> gnu.org> writes:
>
>> ...you didn't show any measurements of code speed with and without
>> byte-compilation)...
>> ...keeping
>> track of all the files, making decisions which will and will not be
>> compiled, etc. ...
> I agree, so I'm closing this as wontfix.

-- 
Daanturo.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.