Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues

From: Pip Cet <[email-masked]>
Date: Sun, 19 Jan 2025 17:28:41 +0000

"Alexis Purslane via \"Bug reports for GNU Emacs, the Swiss army knife of text editors\"" <[email-masked]> writes:

> I've been using scratch/igc for the past few days (see details below for
> the exact version and situation) and have been having some /interesting/
> experiences:

Thanks for sharing them!  Please try to catch a freezing Emacs and do
not kill it; hopefully, we can then use gdb to extract enough
information to let us fix the bug.  Instructions below.

>     1. It seems slower than Emacs 29.4 with GCMH (gcmh-idle-timer set to

I don't know how gcmh works.

>     'auto, so it GCs usually after less than a second of idle time, and
>     gcmh-max-cons-threshold set to 200MB so that if there's a very
>     intense operation going on that's allocating a lot of memory, it
>     will actually GC before it gets out of hand and the idle GC pause
>     would freeze Emacs, but most of the time it'll almost never GC
>     unless I'm idle) when opening a lot of files (for instance, when
>     running org-agenda). About twice as slow. Startup is faster, though.

Does "twice as slow" mean that long tasks take Emacs twice as long, or
that latency appears to have doubled?

feature/igc isn't optimized particularly well right now.  IMHO, it's
already more usable than "master", but I care about latency, not CPU
usage.

I don't know why startup is faster.

I'm confident that we'll be able to outperform the traditional GC for
some users, and come close for all others.  A forced full GC will
probably take longer than an alloc.c GC run (it might not: string
compaction isn't efficient, for example).

>     2. It uses monotonically more and more memory throughout a session,

That's bad.  I thought it was my fault, but I've seen the same thing
here.

>     even when doing things that shouldn't cause new memory to be
>     allocated from the OS, eventually a really, really large amount
>     (I've seen 2GB) even though the amount of memory it claims it's

Is this memory actually used, or is it virtual memory which was never
paged in?  One good way to do that is to create a coredump file from gdb
attached to Emacs, which should not kill Emacs.

>     using when I run memory-report isn't that large. E.g., right now
>     it's using 981 MB (just went up from 700 in the last few minutes
>     despite only writing in this buffer the entire time), and
>     memory-report says:

I'll check whether memory-report does anything useful for the IGC build,
right now; if it doesn't, we should fix it.

However, M-x igc-stats and M-x igc-roots-stats may provide further
insight (use the "s" key to get a snapshot.  Sometimes I have to hit "a"
first, but that's possibly local breakage).

>     Estimated Emacs Memory Usage
>
>     73 MiB  Total Buffer Memory Usage
>     18 MiB  Memory Used By Global Variables
>    9.8 MiB  Memory Used By Symbol Plists
>    1.1 MiB  Total Image Cache Size
>        0 B  Reserved (But Unused) Object Memory
>        0 B  Overall Object Memory Usage

That looks negligible.

> Object Storage
>        0 B  Strings
>        0 B  Vectors
>        0 B  Floats
>        0 B  Conses
>        0 B  Symbols
>        0 B  Intervals
>        0 B  Buffer-Objects

So we're not scanning those at all :-)

> Largest Variables
>
>      2 MiB  load-history
>    1.5 MiB  ucs-normalize-hangul-translation-alist
>      1 MiB  nerd-icons/mdicon-alist
>    746 KiB  easy-menu-converted-items-table
>    659 KiB  face--new-frame-defaults
>    613 KiB  sly-common-lisp-system-indentation
>    565 KiB  undo-equiv-table
>    498 KiB  gnus-newsrc-hashtb
>    495 KiB  gnus-newsrc-alist
>    494 KiB  nnimap-current-infos
>    413 KiB  uni-confusable-table
>    305 KiB  definition-prefixes
>    285 KiB  nerd-icons/faicon-alist
>    234 KiB  minor-mode-map-alist
>    201 KiB  doom-themes-base-faces
>    189 KiB  org-entities
>    180 KiB  company-keywords-alist
>    157 KiB  common-lisp-hyperspec--symbols
>    149 KiB  global-map
>    149 KiB  help-quick-use-map

Negligible, too.

>     3. Eventually, Emacs will freeze irrecoverably (i.e., C-g doesn't
>     work), probably trying to GC, causing total lossage of my Emacs
>     session and necessitating a restart.

Ouch.  It'd be very important if you could catch Emacs in this state.

Are you using anything that creates many child processes? If that is the
problem, it might be possible to recover from it.

If you haven't started Emacs in GDB, the next time it freezes, please
try running

ps aux | grep emacs ==> determine pid of emacs

gdb -p <pid of emacs>

(you might need sudo because some people think it's more secure to run
gdb as root than it is to allow users to trace their own processes)

At the gdb prompt (note that "gcore" will produce a large file; please
save it, along with the freezing emacs binary (called "emacs" and its
pdump file "emacs.pdmp")):

source /path/to/emacs/src/.gdbinit
bt full
gcore

> I may have configured it wrong, or it may be an issue particular to my
> system, hence the debug info below.

I haven't seen uninterruptible freezes here; I have had to use the magic
"C-g C-g C-g" triple-quit once in a while, because magit sometimes seems
to take very long.  Just to be sure, triple C-g doesn't fix the problem
for you?

> In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
>  3.24.43, cairo version 1.18.2) of 2025-01-16 built on fedora
> Repository revision: 6185a57afed3ef02b2608e6bc476d117b02c8e81
> Repository branch: scratch/igc
> System Description: Fedora Linux 41.20241229.0 (Silverblue)
>
> Configured using:
>  'configure
>  CPPFLAGS=-I/home/alexispurslane/Development/scratch/emacs/mps/mps-artifacts
>  LDFLAGS=-L/home/alexispurslane/Development/scratch/emacs/mps/mps-artifacts
>  --with-cairo --with-dbus --with-gif --with-gpm=no --with-harfbuzz
>  --with-jpeg --with-modules --with-native-compilation=aot --with-png
>  --with-rsvg --with-sqlite3 --with-tiff --with-tree-sitter --with-webp
>  --with-x-toolkit=gtk3 --with-xinput2 --with-xpm --with-mps=yes
>  --with-pgtk --prefix=/var/home/alexispurslane/.local'
>
> Configured features:
> CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
> LIBSELINUX LIBXML2 MODULES MPS NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
> PNG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM
> GTK3 ZLIB
  ^^^^

We're working on fixing some GTK memory leaks in bug#75636; it's
possible those are partially responsible for growing memory usage.

> Features:
> (gnus-draft gnus-async shrface embark-org ox-odt rng-loc rng-uri
> ...
> magit-section cursor-sensor dash compat plz warnings color dns
  ^^^^^^^^^^^^^

I see some magit-related features here, but not the plain "magit"
feature.  Is that correct?

> Memory information:
> ((conses 24 0 0) (symbols 56 0 0) (strings 40 0 0)
>  (string-bytes 1 0) (vectors 24 0) (vector-slots 8 0 0)
>  (floats 24 0 0) (intervals 64 0 0) (buffers 1000 0))

Also something we have to fix.

Pip