GNU bug report logs - #75672
31.0.50; scratch/igc memory usage/collection issues

Previous Next

Package: emacs;

Reported by: Alexis Purslane <alexispurslane <at> pm.me>

Date: Sun, 19 Jan 2025 17:01:02 UTC

Severity: normal

Found in version 31.0.50

Done: Pip Cet <pipcet <at> protonmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75672 in the body.
You can then email your comments to 75672 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 17:01:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Alexis Purslane <alexispurslane <at> pm.me>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs <at> gnu.org. (Sun, 19 Jan 2025 17:01:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alexis Purslane <alexispurslane <at> pm.me>
To: bug-gnu-emacs <at> gnu.org
Subject: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 16:59:46 +0000
[Message part 1 (text/plain, inline)]
I've been using scratch/igc for the past few days (see details below for
the exact version and situation) and have been having some /interesting/
experiences:

    1. It seems slower than Emacs 29.4 with GCMH (gcmh-idle-timer set to
    'auto, so it GCs usually after less than a second of idle time, and
    gcmh-max-cons-threshold set to 200MB so that if there's a very
    intense operation going on that's allocating a lot of memory, it
    will actually GC before it gets out of hand and the idle GC pause
    would freeze Emacs, but most of the time it'll almost never GC
    unless I'm idle) when opening a lot of files (for instance, when
    running org-agenda). About twice as slow. Startup is faster, though.

    2. It uses monotonically more and more memory throughout a session,
    even when doing things that shouldn't cause new memory to be
    allocated from the OS, eventually a really, really large amount
    (I've seen 2GB) even though the amount of memory it claims it's
    using when I run memory-report isn't that large. E.g., right now
    it's using 981 MB (just went up from 700 in the last few minutes
    despite only writing in this buffer the entire time), and
    memory-report says:

    Estimated Emacs Memory Usage

    73 MiB  Total Buffer Memory Usage
    18 MiB  Memory Used By Global Variables
   9.8 MiB  Memory Used By Symbol Plists
   1.1 MiB  Total Image Cache Size
       0 B  Reserved (But Unused) Object Memory
       0 B  Overall Object Memory Usage

Object Storage

       0 B  Strings
       0 B  Vectors
       0 B  Floats
       0 B  Conses
       0 B  Symbols
       0 B  Intervals
       0 B  Buffer-Objects

Largest Buffers

    67 MiB  *eshell*
     1 MiB  *sly-events for sbcl*
   966 KiB  init.el
   640 KiB  *sly-compilation*
   478 KiB   *nnimap 127.0.0.1 1143  *nntpd**
   364 KiB  *sent mail to bug-gnu-emacs <at> gnu.org*
   336 KiB  *sly-mrepl for sbcl*
   293 KiB  *unsent mail to bug-gnu-emacs <at> gnu.org*
   285 KiB  *Summary Sent*
   246 KiB  main.lisp
   183 KiB  tools.lisp
   161 KiB  video.lisp
   130 KiB   *sly-2*
   130 KiB   *sly-3*
    66 KiB  *Messages*
    66 KiB   *code-conversion-work*
    47 KiB  *Group*
    37 KiB   *which-key*
    36 KiB  *Async-native-compile-log*
    29 KiB  *sly-description*

Largest Variables

     2 MiB  load-history
   1.5 MiB  ucs-normalize-hangul-translation-alist
     1 MiB  nerd-icons/mdicon-alist
   746 KiB  easy-menu-converted-items-table
   659 KiB  face--new-frame-defaults
   613 KiB  sly-common-lisp-system-indentation
   565 KiB  undo-equiv-table
   498 KiB  gnus-newsrc-hashtb
   495 KiB  gnus-newsrc-alist
   494 KiB  nnimap-current-infos
   413 KiB  uni-confusable-table
   305 KiB  definition-prefixes
   285 KiB  nerd-icons/faicon-alist
   234 KiB  minor-mode-map-alist
   201 KiB  doom-themes-base-faces
   189 KiB  org-entities
   180 KiB  company-keywords-alist
   157 KiB  common-lisp-hyperspec--symbols
   149 KiB  global-map
   149 KiB  help-quick-use-map

    3. Eventually, Emacs will freeze irrecoverably (i.e., C-g doesn't
    work), probably trying to GC, causing total lossage of my Emacs
    session and necessitating a restart.

I may have configured it wrong, or it may be an issue particular to my
system, hence the debug info below.


In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.43, cairo version 1.18.2) of 2025-01-16 built on fedora
Repository revision: 6185a57afed3ef02b2608e6bc476d117b02c8e81
Repository branch: scratch/igc
System Description: Fedora Linux 41.20241229.0 (Silverblue)

Configured using:
 'configure
 CPPFLAGS=-I/home/alexispurslane/Development/scratch/emacs/mps/mps-artifacts
 LDFLAGS=-L/home/alexispurslane/Development/scratch/emacs/mps/mps-artifacts
 --with-cairo --with-dbus --with-gif --with-gpm=no --with-harfbuzz
 --with-jpeg --with-modules --with-native-compilation=aot --with-png
 --with-rsvg --with-sqlite3 --with-tiff --with-tree-sitter --with-webp
 --with-x-toolkit=gtk3 --with-xinput2 --with-xpm --with-mps=yes
 --with-pgtk --prefix=/var/home/alexispurslane/.local'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
LIBSELINUX LIBXML2 MODULES MPS NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
PNG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM
GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  eshell-syntax-highlighting-global-mode: t
  global-fish-completion-mode: t
  fish-completion-mode: t
  eat-eshell-mode: t
  gnus-desktop-notify-mode: t
  gnus-undo-mode: t
  sly-symbol-completion-mode: t
  editorconfig-mode: t
  corfu-popupinfo-mode: t
  recentf-mode: t
  nerd-icons-completion-mode: t
  marginalia-mode: t
  icomplete-vertical-mode: t
  icomplete-mode: t
  which-key-mode: t
  spacious-padding-mode: t
  global-visual-fill-column-mode: t
  global-treesit-auto-mode: t
  electric-pair-mode: t
  repeat-mode: t
  delete-selection-mode: t
  motion-selection-mode: t
  god-local-mode: t
  windmove-mode: t
  winner-mode: t
  savehist-mode: t
  pixel-scroll-precision-mode: t
  minibuffer-depth-indicate-mode: t
  global-auto-revert-mode: t
  override-global-mode: t
  display-time-mode: t
  display-battery-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  global-visual-line-mode: t
  visual-line-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/var/home/alexispurslane/.emacs.d/elpa/which-key-20240620.2145/which-key hides /var/home/alexispurslane/.local/share/emacs/31.0.50/lisp/which-key
/var/home/alexispurslane/.emacs.d/elpa/transient-20250108.1351/transient hides /var/home/alexispurslane/.local/share/emacs/31.0.50/lisp/transient
/var/home/alexispurslane/.emacs.d/elpa/editorconfig-20241027.1815/editorconfig-tools hides /var/home/alexispurslane/.local/share/emacs/31.0.50/lisp/editorconfig-tools
/var/home/alexispurslane/.emacs.d/elpa/editorconfig-20241027.1815/editorconfig-fnmatch hides /var/home/alexispurslane/.local/share/emacs/31.0.50/lisp/editorconfig-fnmatch
/var/home/alexispurslane/.emacs.d/elpa/editorconfig-20241027.1815/editorconfig hides /var/home/alexispurslane/.local/share/emacs/31.0.50/lisp/editorconfig
/var/home/alexispurslane/.emacs.d/elpa/editorconfig-20241027.1815/editorconfig-core-handle hides /var/home/alexispurslane/.local/share/emacs/31.0.50/lisp/editorconfig-core-handle
/var/home/alexispurslane/.emacs.d/elpa/editorconfig-20241027.1815/editorconfig-core hides /var/home/alexispurslane/.local/share/emacs/31.0.50/lisp/editorconfig-core
/var/home/alexispurslane/.emacs.d/elpa/editorconfig-20241027.1815/editorconfig-conf-mode hides /var/home/alexispurslane/.local/share/emacs/31.0.50/lisp/editorconfig-conf-mode

Features:
(gnus-draft gnus-async shrface embark-org ox-odt rng-loc rng-uri
rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util ox-latex ox-icalendar org-agenda ox-html table
ox-ascii ox-publish ox org-attach org-element org-persist org-id
org-refile org-element-ast inline avl-tree orgtbl-ascii-plot org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-src sh-script smie
executable ob-comint org-pcomplete org-list org-footnote org-faces
org-entities org-version ob-emacs-lisp ob-core ob-eval org-cycle
org-table ol org-fold org-fold-core org-keys oc org-loaddefs cal-menu
calendar cal-loaddefs org-compat org-macs qp gnus-ml nnfolder nndraft
nnmh nnselect utf-7 epa-file gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-cache textsec uni-scripts idna-mapping
ucs-normalize uni-confusable textsec-check ispell checkdoc lisp-mnt
elisp-def ert debug backtrace find-func f s highlight-defined advice
shadow sort smtpmail-async ecomplete mail-extr emacsbug char-fold
pcmpl-x pcmpl-unix company-oddmuse company-keywords company-etags
company-gtags company-dabbrev-code company-dabbrev company-files
company-clang company-capf company-cmake company-semantic
company-template company-bbdb esh-autosuggest company em-term em-script
em-pred em-hist em-glob em-extpipe em-basic em-banner
eshell-syntax-highlighting em-prompt em-alias esh-help man em-unix
fish-completion em-cmpl eshell-prompt-extras em-dirs em-ls eshell
esh-mode esh-var eat term ehelp esh-cmd esh-ext esh-proc esh-opt esh-io
esh-arg esh-module esh-module-loaddefs esh-util gnus-demon nntp
gnus-desktop-notify smtpmail gnus-registry registry eieio-base gnus-art
mm-uu mml2015 mm-view mml-smime smime gnutls dig gnus-sum gnus-group
gnus-undo gnus-start gnus-dbus gnus-cloud nnimap nnmail mail-source nnoo
gnus-spec gnus-win gnus-int gnus-range imap rfc2104 utf7 gnus nnheader
range embark-consult embark ffap apheleia apheleia-rcs apheleia-dp
apheleia-formatters apheleia-utils apheleia-log
apheleia-formatter-context whitespace misearch multi-isearch puni
sly-asdf grep sly-fancy sly-tramp tramp trampver tramp-integration
tramp-message tramp-compat xdg shell pcomplete parse-time iso8601
tramp-loaddefs sly-indentation sly-cl-indent sly-stickers pulse hi-lock
sly-trace-dialog sly-fontifying-fu sly-package-fu sly-scratch
sly-fancy-trace sly-fancy-inspector sly-mrepl sly-autodoc sly-parse
network-stream nsm help-fns radix-tree mule-util vc-git files-x
sly-macrostep macrostep sly-overlay sly sly-completion sly-buttons
sly-messages sly-common apropos etags fileloop generator xref arc-mode
archive-mode hyperspec lisp-extra-font-lock highlight-numbers
parent-mode noutline outline flymake project compile comint ansi-osc
ansi-color display-line-numbers diff-hl log-view log-edit message
sendmail yank-media dired-subtree dired-hacks-utils dired-aux dired
dired-loaddefs rfc822 mml mml-sec epa derived epg rfc6068 epg-config
gnus-util mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047
rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils
mailheader add-log pcvs-util vc-dir vc vc-dispatcher diff-mode
track-changes editorconfig editorconfig-core editorconfig-core-handle
editorconfig-fnmatch yasnippet-capf thingatpt common-lisp-snippets
yasnippet eldoc-box hl-todo ligature corfu-popupinfo nerd-icons-corfu
corfu comp comp-cstr comp-run comp-common consult recentf tree-widget
orderless nerd-icons-completion marginalia icomplete which-key ement
ement-notifications ement-notify notifications ement-room transient
bookmark face-remap shr text-property-search pixel-fill kinsoku url-file
puny svg dom ewoc ement-lib ement-api ement-structs ement-macros
magit-section cursor-sensor dash compat plz warnings color dns
nerd-icons nerd-icons-faces nerd-icons-data nerd-icons-data-mdicon
nerd-icons-data-flicon nerd-icons-data-codicon nerd-icons-data-devicon
nerd-icons-data-sucicon nerd-icons-data-wicon nerd-icons-data-faicon
nerd-icons-data-powerline nerd-icons-data-octicon
nerd-icons-data-pomicon nerd-icons-data-ipsicon spacious-padding
almost-mono-black-theme doom-themes doom-themes-base visual-fill-column
treesit-auto treesit hl-line elec-pair repeat delsel
motion-selection-mode god-mode-isearch god-mode time-date async
disp-table windmove winner savehist pixel-scroll cua-base ring mb-depth
help-at-pt autorevert filenotify cus-edit pp cus-load wid-edit pcase
finder-inf almost-mono-themes edmacro kmacro use-package-bind-key
bind-key easy-mmode time format-spec battery dbus xml gcmh cl-extra
help-mode use-package-ensure use-package-core package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs icons password-cache json
subr-x map byte-opt gv bytecomp byte-compile url-vars rx profiler
cl-loaddefs cl-lib info CLEDE-autoloads almost-mono-themes-autoloads
apheleia-autoloads async-autoloads auto-highlight-symbol-autoloads
breadcrumb-autoloads calibredb-autoloads centered-window-autoloads
clojure-ts-mode-autoloads common-lisp-snippets-autoloads
consult-gnome-search-autoloads consult-notes-autoloads corfu-autoloads
dape-autoloads darkroom-autoloads devdocs-autoloads diff-hl-autoloads
dired-sidebar-autoloads dired-subtree-autoloads
dired-hacks-utils-autoloads docker-autoloads aio-autoloads
doom-themes-autoloads eat-autoloads editorconfig-autoloads
eldoc-box-autoloads elisp-def-autoloads elisp-demos-autoloads
embark-consult-autoloads consult-autoloads embark-autoloads
ement-autoloads enlight-autoloads esh-autosuggest-autoloads
company-autoloads esh-help-autoloads eshell-prompt-extras-autoloads
eshell-syntax-highlighting-autoloads evil-cleverparens-autoloads
evil-collection-autoloads annalist-autoloads evil-god-state-autoloads
evil-org-autoloads evil-textobj-tree-sitter-autoloads expreg-autoloads
exwm-autoloads fish-completion-autoloads flymake-proselint-autoloads
flymake-vale-autoloads forge-autoloads closql-autoloads
emacsql-autoloads gcmh-autoloads ghub-autoloads glsl-mode-autoloads
gnuplot-autoloads gnuplot-mode-autoloads gnus-desktop-notify-autoloads
god-mode-autoloads helpful-autoloads elisp-refs-autoloads f-autoloads
highlight-blocks-autoloads highlight-defined-autoloads
highlight-function-calls-autoloads highlight-numbers-autoloads
highlight-parentheses-autoloads highlight-stages-autoloads
highlight-thing-autoloads highlight-unique-symbol-autoloads
deferred-autoloads hl-todo-autoloads htmlize-autoloads
hungry-delete-autoloads keycast-autoloads latex-preview-pane-autoloads
ligature-autoloads lisp-extra-font-lock-autoloads magit-autoloads
marginalia-autoloads markdown-mode-autoloads mathjax-autoloads
mood-line-autoloads motion-selection-mode-autoloads
nerd-icons-completion-autoloads nerd-icons-corfu-autoloads
nerd-icons-dired-autoloads nerd-icons-autoloads nov-autoloads
esxml-autoloads kv-autoloads orderless-autoloads org-mime-autoloads
orgtbl-ascii-plot-autoloads ox-rss-autoloads
package-lint-flymake-autoloads package-lint-autoloads
pandoc-mode-autoloads paredit-autoloads parent-mode-autoloads
persist-autoloads plz-autoloads poet-theme-autoloads pos-tip-autoloads
puni-autoloads rainbow-identifiers-autoloads request-autoloads
shrface-autoloads language-detection-autoloads sly-asdf-autoloads
popup-autoloads sly-macrostep-autoloads macrostep-autoloads
sly-overlay-autoloads sly-autoloads smartparens-autoloads
spacious-padding-autoloads svg-lib-autoloads symbol-overlay-autoloads
syncthing-autoloads tablist-autoloads taxy-magit-section-autoloads
taxy-autoloads magit-section-autoloads theme-anchor-autoloads
toc-org-autoloads transient-autoloads treemacs-evil-autoloads
treemacs-autoloads cfrs-autoloads posframe-autoloads ht-autoloads
hydra-autoloads lv-autoloads pfuture-autoloads ace-window-autoloads
avy-autoloads s-autoloads dash-autoloads evil-autoloads
goto-chg-autoloads treepy-autoloads treesit-auto-autoloads
vimgolf-autoloads visual-fill-column-autoloads wgrep-autoloads
which-key-autoloads whole-line-or-region-autoloads with-editor-autoloads
xelb-autoloads yaml-autoloads yasnippet-capf-autoloads
yasnippet-autoloads rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/pgtk-win pgtk-win term/common-win touch-screen pgtk-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
multi-tty move-toolbar make-network-process native-compile mps emacs)

Memory information:
((conses 24 0 0) (symbols 56 0 0) (strings 40 0 0)
 (string-bytes 1 0) (vectors 24 0) (vector-slots 8 0 0)
 (floats 24 0 0) (intervals 64 0 0) (buffers 1000 0))
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 17:29:02 GMT) Full text and rfc822 format available.

Message #8 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Alexis Purslane <alexispurslane <at> pm.me>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 17:28:41 +0000
"Alexis Purslane via \"Bug reports for GNU Emacs, the Swiss army knife of text editors\"" <bug-gnu-emacs <at> gnu.org> writes:

> I've been using scratch/igc for the past few days (see details below for
> the exact version and situation) and have been having some /interesting/
> experiences:

Thanks for sharing them!  Please try to catch a freezing Emacs and do
not kill it; hopefully, we can then use gdb to extract enough
information to let us fix the bug.  Instructions below.

>     1. It seems slower than Emacs 29.4 with GCMH (gcmh-idle-timer set to

I don't know how gcmh works.

>     'auto, so it GCs usually after less than a second of idle time, and
>     gcmh-max-cons-threshold set to 200MB so that if there's a very
>     intense operation going on that's allocating a lot of memory, it
>     will actually GC before it gets out of hand and the idle GC pause
>     would freeze Emacs, but most of the time it'll almost never GC
>     unless I'm idle) when opening a lot of files (for instance, when
>     running org-agenda). About twice as slow. Startup is faster, though.

Does "twice as slow" mean that long tasks take Emacs twice as long, or
that latency appears to have doubled?

feature/igc isn't optimized particularly well right now.  IMHO, it's
already more usable than "master", but I care about latency, not CPU
usage.

I don't know why startup is faster.

I'm confident that we'll be able to outperform the traditional GC for
some users, and come close for all others.  A forced full GC will
probably take longer than an alloc.c GC run (it might not: string
compaction isn't efficient, for example).

>     2. It uses monotonically more and more memory throughout a session,

That's bad.  I thought it was my fault, but I've seen the same thing
here.

>     even when doing things that shouldn't cause new memory to be
>     allocated from the OS, eventually a really, really large amount
>     (I've seen 2GB) even though the amount of memory it claims it's

Is this memory actually used, or is it virtual memory which was never
paged in?  One good way to do that is to create a coredump file from gdb
attached to Emacs, which should not kill Emacs.

>     using when I run memory-report isn't that large. E.g., right now
>     it's using 981 MB (just went up from 700 in the last few minutes
>     despite only writing in this buffer the entire time), and
>     memory-report says:

I'll check whether memory-report does anything useful for the IGC build,
right now; if it doesn't, we should fix it.

However, M-x igc-stats and M-x igc-roots-stats may provide further
insight (use the "s" key to get a snapshot.  Sometimes I have to hit "a"
first, but that's possibly local breakage).

>     Estimated Emacs Memory Usage
>
>     73 MiB  Total Buffer Memory Usage
>     18 MiB  Memory Used By Global Variables
>    9.8 MiB  Memory Used By Symbol Plists
>    1.1 MiB  Total Image Cache Size
>        0 B  Reserved (But Unused) Object Memory
>        0 B  Overall Object Memory Usage

That looks negligible.

> Object Storage
>
>        0 B  Strings
>        0 B  Vectors
>        0 B  Floats
>        0 B  Conses
>        0 B  Symbols
>        0 B  Intervals
>        0 B  Buffer-Objects

So we're not scanning those at all :-)

> Largest Variables
>
>      2 MiB  load-history
>    1.5 MiB  ucs-normalize-hangul-translation-alist
>      1 MiB  nerd-icons/mdicon-alist
>    746 KiB  easy-menu-converted-items-table
>    659 KiB  face--new-frame-defaults
>    613 KiB  sly-common-lisp-system-indentation
>    565 KiB  undo-equiv-table
>    498 KiB  gnus-newsrc-hashtb
>    495 KiB  gnus-newsrc-alist
>    494 KiB  nnimap-current-infos
>    413 KiB  uni-confusable-table
>    305 KiB  definition-prefixes
>    285 KiB  nerd-icons/faicon-alist
>    234 KiB  minor-mode-map-alist
>    201 KiB  doom-themes-base-faces
>    189 KiB  org-entities
>    180 KiB  company-keywords-alist
>    157 KiB  common-lisp-hyperspec--symbols
>    149 KiB  global-map
>    149 KiB  help-quick-use-map

Negligible, too.

>     3. Eventually, Emacs will freeze irrecoverably (i.e., C-g doesn't
>     work), probably trying to GC, causing total lossage of my Emacs
>     session and necessitating a restart.

Ouch.  It'd be very important if you could catch Emacs in this state.

Are you using anything that creates many child processes? If that is the
problem, it might be possible to recover from it.

If you haven't started Emacs in GDB, the next time it freezes, please
try running

ps aux | grep emacs ==> determine pid of emacs

gdb -p <pid of emacs>

(you might need sudo because some people think it's more secure to run
gdb as root than it is to allow users to trace their own processes)

At the gdb prompt (note that "gcore" will produce a large file; please
save it, along with the freezing emacs binary (called "emacs" and its
pdump file "emacs.pdmp")):

source /path/to/emacs/src/.gdbinit
bt full
gcore

> I may have configured it wrong, or it may be an issue particular to my
> system, hence the debug info below.

I haven't seen uninterruptible freezes here; I have had to use the magic
"C-g C-g C-g" triple-quit once in a while, because magit sometimes seems
to take very long.  Just to be sure, triple C-g doesn't fix the problem
for you?

> In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
>  3.24.43, cairo version 1.18.2) of 2025-01-16 built on fedora
> Repository revision: 6185a57afed3ef02b2608e6bc476d117b02c8e81
> Repository branch: scratch/igc
> System Description: Fedora Linux 41.20241229.0 (Silverblue)
>
> Configured using:
>  'configure
>  CPPFLAGS=-I/home/alexispurslane/Development/scratch/emacs/mps/mps-artifacts
>  LDFLAGS=-L/home/alexispurslane/Development/scratch/emacs/mps/mps-artifacts
>  --with-cairo --with-dbus --with-gif --with-gpm=no --with-harfbuzz
>  --with-jpeg --with-modules --with-native-compilation=aot --with-png
>  --with-rsvg --with-sqlite3 --with-tiff --with-tree-sitter --with-webp
>  --with-x-toolkit=gtk3 --with-xinput2 --with-xpm --with-mps=yes
>  --with-pgtk --prefix=/var/home/alexispurslane/.local'
>
> Configured features:
> CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
> LIBSELINUX LIBXML2 MODULES MPS NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
> PNG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM
> GTK3 ZLIB
  ^^^^

We're working on fixing some GTK memory leaks in bug#75636; it's
possible those are partially responsible for growing memory usage.

> Features:
> (gnus-draft gnus-async shrface embark-org ox-odt rng-loc rng-uri
> ...
> magit-section cursor-sensor dash compat plz warnings color dns
  ^^^^^^^^^^^^^

I see some magit-related features here, but not the plain "magit"
feature.  Is that correct?

> Memory information:
> ((conses 24 0 0) (symbols 56 0 0) (strings 40 0 0)
>  (string-bytes 1 0) (vectors 24 0) (vector-slots 8 0 0)
>  (floats 24 0 0) (intervals 64 0 0) (buffers 1000 0))

Also something we have to fix.

Pip





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 17:38:02 GMT) Full text and rfc822 format available.

Message #11 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Alexis Purslane <alexispurslane <at> pm.me>, 75672 <at> debbugs.gnu.org
Cc: gerd <at> gnu.org, acorallo <at> gnu.org, Pip Cet <pipcet <at> protonmail.com>
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 11:36:58 -0600
Alexis Purslane via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs <at> gnu.org> writes:

> I've been using scratch/igc for the past few days (see details below for
> the exact version and situation) and have been having some /interesting/
> experiences:
>
>     1. It seems slower than Emacs 29.4 with GCMH (gcmh-idle-timer set to
>     'auto, so it GCs usually after less than a second of idle time, and
>     gcmh-max-cons-threshold set to 200MB so that if there's a very
>     intense operation going on that's allocating a lot of memory, it
>     will actually GC before it gets out of hand and the idle GC pause
>     would freeze Emacs, but most of the time it'll almost never GC
>     unless I'm idle) when opening a lot of files (for instance, when
>     running org-agenda). About twice as slow. Startup is faster, though.

Thanks for the report.  I'm repyling only to the GCMH part here.

For context, GCMH is a package by Andrea that tries to avoid GC by
setting a high `gc-cons-threshold` when Emacs is in use, and then
triggers GC when Emacs is idle.  Details here:

    https://elpa.gnu.org/packages/gcmh.html

Do we expect that running GCMH will be useful with IGC?

First, I'm not sure that `gc-cons-threshold` will even affect MPS.

Second, we already run 'igc_on_idle' periodically.

Third, GCMH will run `garbage-collect`, which doesn't trigger a full GC
with MPS, AFAIU by design.  GCMH would need to run `igc--collect`, but
according to Gerd we should treat that function as a debugging tool.

Maybe GCMH should simply do nothing if (featurep 'mps)?




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 17:42:02 GMT) Full text and rfc822 format available.

Message #14 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: alexis purslane <alexispurslane <at> pm.me>
To: "stefankangas <at> gmail.com" <stefankangas <at> gmail.com>
Cc: "pipcet <at> protonmail.com" <pipcet <at> protonmail.com>,
 "75672 <at> debbugs.gnu.org" <75672 <at> debbugs.gnu.org>,
 "acorallo <at> gnu.org" <acorallo <at> gnu.org>, "gerd <at> gnu.org" <gerd <at> gnu.org>
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 17:41:19 +0000
[Message part 1 (text/plain, inline)]

On 1/19/25 12:36 PM, Stefan Kangas <stefankangas <at> gmail.com> wrote:

>  Alexis Purslane via "Bug reports for GNU Emacs, the Swiss army knife of
>  text editors" <bug-gnu-emacs <at> gnu.org> writes:
>  
>  > I've been using scratch/igc for the past few days (see details below for
>  > the exact version and situation) and have been having some /interesting/
>  > experiences:
>  >
>  >     1. It seems slower than Emacs 29.4 with GCMH (gcmh-idle-timer set to
>  >     'auto, so it GCs usually after less than a second of idle time, and
>  >     gcmh-max-cons-threshold set to 200MB so that if there's a very
>  >     intense operation going on that's allocating a lot of memory, it
>  >     will actually GC before it gets out of hand and the idle GC pause
>  >     would freeze Emacs, but most of the time it'll almost never GC
>  >     unless I'm idle) when opening a lot of files (for instance, when
>  >     running org-agenda). About twice as slow. Startup is faster, though.
>  
>  Thanks for the report.  I'm repyling only to the GCMH part here.
>  
>  For context, GCMH is a package by Andrea that tries to avoid GC by
>  setting a high `gc-cons-threshold` when Emacs is in use, and then
>  triggers GC when Emacs is idle.  Details here:
>  
>      https://elpa.gnu.org/packages/gcmh.html
>  
>  Do we expect that running GCMH will be useful with IGC?
>  
>  First, I'm not sure that `gc-cons-threshold` will even affect MPS.
>  
>  Second, we already run 'igc_on_idle' periodically.
>  
>  Third, GCMH will run `garbage-collect`, which doesn't trigger a full GC
>  with MPS, AFAIU by design.  GCMH would need to run `igc--collect`, but
>  according to Gerd we should treat that function as a debugging tool.
>  
>  Maybe GCMH should simply do nothing if (featurep 'mps)?
>  

I use GCMH with emacs 29.4, but I have it turned off for emacs 31.0.50, Because I figured it probably wouldn't do anything, and even if it did, it would probably muddy the waters for debugging purposes.
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 17:43:01 GMT) Full text and rfc822 format available.

Message #17 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: alexis purslane <alexispurslane <at> pm.me>
To: "pipcet <at> protonmail.com" <pipcet <at> protonmail.com>
Cc: "75672 <at> debbugs.gnu.org" <75672 <at> debbugs.gnu.org>
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 17:42:15 +0000
[Message part 1 (text/plain, inline)]
-------- Original Message --------
On 1/19/25 12:28 PM, Pip Cet <pipcet <at> protonmail.com> wrote:

>  "Alexis Purslane via \"Bug reports for GNU Emacs, the Swiss army knife of text editors\"" <bug-gnu-emacs <at> gnu.org> writes:
>  
>  > I've been using scratch/igc for the past few days (see details below for
>  > the exact version and situation) and have been having some /interesting/
>  > experiences:
>  
>  Thanks for sharing them!  Please try to catch a freezing Emacs and do
>  not kill it; hopefully, we can then use gdb to extract enough
>  information to let us fix the bug.  Instructions below.
>  
>  >     1. It seems slower than Emacs 29.4 with GCMH (gcmh-idle-timer set to
>  
>  I don't know how gcmh works.
>  
>  >     'auto, so it GCs usually after less than a second of idle time, and
>  >     gcmh-max-cons-threshold set to 200MB so that if there's a very
>  >     intense operation going on that's allocating a lot of memory, it
>  >     will actually GC before it gets out of hand and the idle GC pause
>  >     would freeze Emacs, but most of the time it'll almost never GC
>  >     unless I'm idle) when opening a lot of files (for instance, when
>  >     running org-agenda). About twice as slow. Startup is faster, though.
>  
>  Does "twice as slow" mean that long tasks take Emacs twice as long, or
>  that latency appears to have doubled?
>  
>  feature/igc isn't optimized particularly well right now.  IMHO, it's
>  already more usable than "master", but I care about latency, not CPU
>  usage.
>  
>  I don't know why startup is faster.
>  
>  I'm confident that we'll be able to outperform the traditional GC for
>  some users, and come close for all others.  A forced full GC will
>  probably take longer than an alloc.c GC run (it might not: string
>  compaction isn't efficient, for example).
>  
>  >     2. It uses monotonically more and more memory throughout a session,
>  
>  That's bad.  I thought it was my fault, but I've seen the same thing
>  here.
>  
>  >     even when doing things that shouldn't cause new memory to be
>  >     allocated from the OS, eventually a really, really large amount
>  >     (I've seen 2GB) even though the amount of memory it claims it's
>  
>  Is this memory actually used, or is it virtual memory which was never
>  paged in?  One good way to do that is to create a coredump file from gdb
>  attached to Emacs, which should not kill Emacs.
>  
>  >     using when I run memory-report isn't that large. E.g., right now
>  >     it's using 981 MB (just went up from 700 in the last few minutes
>  >     despite only writing in this buffer the entire time), and
>  >     memory-report says:
>  
>  I'll check whether memory-report does anything useful for the IGC build,
>  right now; if it doesn't, we should fix it.
>  
>  However, M-x igc-stats and M-x igc-roots-stats may provide further
>  insight (use the "s" key to get a snapshot.  Sometimes I have to hit "a"
>  first, but that's possibly local breakage).
>  
>  >     Estimated Emacs Memory Usage
>  >
>  >     73 MiB  Total Buffer Memory Usage
>  >     18 MiB  Memory Used By Global Variables
>  >    9.8 MiB  Memory Used By Symbol Plists
>  >    1.1 MiB  Total Image Cache Size
>  >        0 B  Reserved (But Unused) Object Memory
>  >        0 B  Overall Object Memory Usage
>  
>  That looks negligible.
>  
>  > Object Storage
>  >
>  >        0 B  Strings
>  >        0 B  Vectors
>  >        0 B  Floats
>  >        0 B  Conses
>  >        0 B  Symbols
>  >        0 B  Intervals
>  >        0 B  Buffer-Objects
>  
>  So we're not scanning those at all :-)
>  
>  > Largest Variables
>  >
>  >      2 MiB  load-history
>  >    1.5 MiB  ucs-normalize-hangul-translation-alist
>  >      1 MiB  nerd-icons/mdicon-alist
>  >    746 KiB  easy-menu-converted-items-table
>  >    659 KiB  face--new-frame-defaults
>  >    613 KiB  sly-common-lisp-system-indentation
>  >    565 KiB  undo-equiv-table
>  >    498 KiB  gnus-newsrc-hashtb
>  >    495 KiB  gnus-newsrc-alist
>  >    494 KiB  nnimap-current-infos
>  >    413 KiB  uni-confusable-table
>  >    305 KiB  definition-prefixes
>  >    285 KiB  nerd-icons/faicon-alist
>  >    234 KiB  minor-mode-map-alist
>  >    201 KiB  doom-themes-base-faces
>  >    189 KiB  org-entities
>  >    180 KiB  company-keywords-alist
>  >    157 KiB  common-lisp-hyperspec--symbols
>  >    149 KiB  global-map
>  >    149 KiB  help-quick-use-map
>  
>  Negligible, too.
>  
>  >     3. Eventually, Emacs will freeze irrecoverably (i.e., C-g doesn't
>  >     work), probably trying to GC, causing total lossage of my Emacs
>  >     session and necessitating a restart.
>  
>  Ouch.  It'd be very important if you could catch Emacs in this state.
>  
>  Are you using anything that creates many child processes? If that is the
>  problem, it might be possible to recover from it.
>  
>  If you haven't started Emacs in GDB, the next time it freezes, please
>  try running
>  
>  ps aux | grep emacs ==> determine pid of emacs
>  
>  gdb -p <pid of emacs>
>  
>  (you might need sudo because some people think it's more secure to run
>  gdb as root than it is to allow users to trace their own processes)
>  
>  At the gdb prompt (note that "gcore" will produce a large file; please
>  save it, along with the freezing emacs binary (called "emacs" and its
>  pdump file "emacs.pdmp")):
>  
>  source /path/to/emacs/src/.gdbinit
>  bt full
>  gcore
>  
>  > I may have configured it wrong, or it may be an issue particular to my
>  > system, hence the debug info below.
>  
>  I haven't seen uninterruptible freezes here; I have had to use the magic
>  "C-g C-g C-g" triple-quit once in a while, because magit sometimes seems
>  to take very long.  Just to be sure, triple C-g doesn't fix the problem
>  for you?
>  
>  > In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
>  >  3.24.43, cairo version 1.18.2) of 2025-01-16 built on fedora
>  > Repository revision: 6185a57afed3ef02b2608e6bc476d117b02c8e81
>  > Repository branch: scratch/igc
>  > System Description: Fedora Linux 41.20241229.0 (Silverblue)
>  >
>  > Configured using:
>  >  'configure
>  >  CPPFLAGS=-I/home/alexispurslane/Development/scratch/emacs/mps/mps-artifacts
>  >  LDFLAGS=-L/home/alexispurslane/Development/scratch/emacs/mps/mps-artifacts
>  >  --with-cairo --with-dbus --with-gif --with-gpm=no --with-harfbuzz
>  >  --with-jpeg --with-modules --with-native-compilation=aot --with-png
>  >  --with-rsvg --with-sqlite3 --with-tiff --with-tree-sitter --with-webp
>  >  --with-x-toolkit=gtk3 --with-xinput2 --with-xpm --with-mps=yes
>  >  --with-pgtk --prefix=/var/home/alexispurslane/.local'
>  >
>  > Configured features:
>  > CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
>  > LIBSELINUX LIBXML2 MODULES MPS NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
>  > PNG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM
>  > GTK3 ZLIB
>    ^^^^
>  
>  We're working on fixing some GTK memory leaks in bug#75636; it's
>  possible those are partially responsible for growing memory usage.
>  
>  > Features:
>  > (gnus-draft gnus-async shrface embark-org ox-odt rng-loc rng-uri
>  > ...
>  > magit-section cursor-sensor dash compat plz warnings color dns
>    ^^^^^^^^^^^^^
>  
>  I see some magit-related features here, but not the plain "magit"
>  feature.  Is that correct?
>  
>  > Memory information:
>  > ((conses 24 0 0) (symbols 56 0 0) (strings 40 0 0)
>  >  (string-bytes 1 0) (vectors 24 0) (vector-slots 8 0 0)
>  >  (floats 24 0 0) (intervals 64 0 0) (buffers 1000 0))
>  
>  Also something we have to fix.
>  
>  Pip
>  

I didn't realize I could attach GDB to a running emacs process. That's a huge help. I'll do that as soon as I can.
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 17:45:03 GMT) Full text and rfc822 format available.

Message #20 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: Alexis Purslane <alexispurslane <at> pm.me>, 75672 <at> debbugs.gnu.org,
 acorallo <at> gnu.org, gerd <at> gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 17:43:51 +0000
"Stefan Kangas" <stefankangas <at> gmail.com> writes:

> Alexis Purslane via "Bug reports for GNU Emacs, the Swiss army knife of
> text editors" <bug-gnu-emacs <at> gnu.org> writes:
>
>> I've been using scratch/igc for the past few days (see details below for
>> the exact version and situation) and have been having some /interesting/
>> experiences:
>>
>>     1. It seems slower than Emacs 29.4 with GCMH (gcmh-idle-timer set to
>>     'auto, so it GCs usually after less than a second of idle time, and
>>     gcmh-max-cons-threshold set to 200MB so that if there's a very
>>     intense operation going on that's allocating a lot of memory, it
>>     will actually GC before it gets out of hand and the idle GC pause
>>     would freeze Emacs, but most of the time it'll almost never GC
>>     unless I'm idle) when opening a lot of files (for instance, when
>>     running org-agenda). About twice as slow. Startup is faster, though.
>
> Thanks for the report.  I'm repyling only to the GCMH part here.
>
> For context, GCMH is a package by Andrea that tries to avoid GC by
> setting a high `gc-cons-threshold` when Emacs is in use, and then
> triggers GC when Emacs is idle.  Details here:
>
>     https://elpa.gnu.org/packages/gcmh.html
>
> Do we expect that running GCMH will be useful with IGC?

No.  We want to optimize IGC's usage of MPS.

> First, I'm not sure that `gc-cons-threshold` will even affect MPS.

It shouldn't.  The decision what to do with an incremental garbage
collector is so different that we cannot do anything useful with that
variable.

> Second, we already run 'igc_on_idle' periodically.

We'll have a background thread soon, to see how that works :-)

> Third, GCMH will run `garbage-collect`, which doesn't trigger a full GC
> with MPS, AFAIU by design.  GCMH would need to run `igc--collect`, but
> according to Gerd we should treat that function as a debugging tool.
>
> Maybe GCMH should simply do nothing if (featurep 'mps)?

SGTM.  Andrea?

Pip





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 17:46:01 GMT) Full text and rfc822 format available.

Message #23 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: Alexis Purslane <alexispurslane <at> pm.me>, 75672 <at> debbugs.gnu.org,
 acorallo <at> gnu.org, gerd <at> gnu.org, Pip Cet <pipcet <at> protonmail.com>
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 18:45:17 +0100
Stefan Kangas <stefankangas <at> gmail.com> writes:

> Do we expect that running GCMH will be useful with IGC?

Absolutely no.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 17:55:02 GMT) Full text and rfc822 format available.

Message #26 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: Alexis Purslane <alexispurslane <at> pm.me>, 75672 <at> debbugs.gnu.org,
 acorallo <at> gnu.org, gerd <at> gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 09:54:12 -0800
Pip Cet <pipcet <at> protonmail.com> writes:

>> First, I'm not sure that `gc-cons-threshold` will even affect MPS.
>
> It shouldn't.  The decision what to do with an incremental garbage
> collector is so different that we cannot do anything useful with that
> variable.

That's what I was thinking, indeed.  So I documented that in commit
4fd6a61605b.  Please take a look, and feel free to tweak it.

We will also eventually need to rewrite (info "(elisp) Garbage
Collection") on feature/igc, but I left that alone for now.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 19 Jan 2025 18:04:02 GMT) Full text and rfc822 format available.

Message #29 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: alexis purslane <alexispurslane <at> pm.me>
Cc: "pipcet <at> protonmail.com" <pipcet <at> protonmail.com>,
 "75672 <at> debbugs.gnu.org" <75672 <at> debbugs.gnu.org>
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 19:02:56 +0100
alexis purslane <alexispurslane <at> pm.me> writes:

> I didn't realize I could attach GDB to a running emacs process. That's
> a huge help. I'll do that as soon as I can.

BTW, since Pip mentioned igc-stats:

There is also igc-start-collection-stats and igc-stop-collecting-stats.
This takes igc-stats every N seconds, either as CSV als in an SQLite db.

What's missing is a good interface to plot the results. In DBBrowser one
can display simple plots for SQL queries, but that's kind of
inconvenient.

Anyway, just wanted to mention that.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 02:34:02 GMT) Full text and rfc822 format available.

Message #32 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Alexis Purslane <alexispurslane <at> pm.me>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 02:33:15 +0000
[Message part 1 (text/plain, inline)]
"Pip Cet" <pipcet <at> protonmail.com> writes:

> However, M-x igc-stats and M-x igc-roots-stats may provide further
> insight (use the "s" key to get a snapshot.  Sometimes I have to hit "a"
> first, but that's possibly local breakage).

igc-stats:
    
IGC_OBJ_BLV                                944           45312         48            48
IGC_OBJ_BUILTIN_SUBR                         0               0          0             0
IGC_OBJ_BUILTIN_SYMBOL                       0               0          0             0
IGC_OBJ_BUILTIN_THREAD                       0               0          0             0
IGC_OBJ_BYTES                                0               0          0             0
IGC_OBJ_CONS                           2603329        62479896         24            24
IGC_OBJ_DUMPED_BIGNUM_DATA                   0               0          0             0
IGC_OBJ_DUMPED_BUFFER_TEXT                   0               0          0             0
IGC_OBJ_DUMPED_BYTES                         5           19936       3987         19528
IGC_OBJ_DUMPED_CHARSET_TABLE                 1           60480      60480         60480
IGC_OBJ_DUMPED_CODE_SPACE_MASKS              1           10248      10248         10248
IGC_OBJ_FACE                              1341          418392        312           312
IGC_OBJ_FACE_CACHE                           3             144         48            48
IGC_OBJ_FLOAT                             8582          205968         24            24
IGC_OBJ_FWD                                  0               0          0             0
IGC_OBJ_HANDLER                             27            8424        312           312
IGC_OBJ_IMAGE                               12            2976        248           248
IGC_OBJ_IMAGE_CACHE                          1              56         56            56
IGC_OBJ_INTERVAL                         82228         5262592         64            64
IGC_OBJ_INVALID                              0               0          0             0
IGC_OBJ_ITREE_NODE                         219           19272         88            88
IGC_OBJ_ITREE_TREE                          31             992         32            32
IGC_OBJ_MARKER_VECTOR                      113          148016       1309         32784
IGC_OBJ_PAD                              16002        31187680       1948       4967448
IGC_OBJ_STRING                          331439        13257560         40            40
IGC_OBJ_STRING_DATA                     343491        27504664         80        262152
IGC_OBJ_SYMBOL                           50996         2855776         56            56
IGC_OBJ_VECTOR                          268293        25462616         94        524304
IGC_OBJ_WEAK_HASH_TABLE_STRONG_PART         28          216800       7742         63328
IGC_OBJ_WEAK_HASH_TABLE_WEAK_PART           28           51424       1836         15632
PVEC_BIGNUM                              15118          483776         32            32
PVEC_BOOL_VECTOR                           438           17440         39            40
PVEC_BUFFER                                202          202000       1000          1000
PVEC_CHAR_TABLE                           1486          916112        616           624
PVEC_CLOSURE                             27259         1399256         51            64
PVEC_CONDVAR                                 0               0          0             0
PVEC_FINALIZER                               8             320         40            40
PVEC_FONT                                26460         3185160        120           312
PVEC_FRAME                                   3            2112        704           704
PVEC_FREE                                61916         2012024         32           120
PVEC_HASH_TABLE                           5281          464728         88            88
PVEC_MARKER                              22598         1265488         56            56
PVEC_MISC_PTR                                0               0          0             0
PVEC_MODULE_FUNCTION                         0               0          0             0
PVEC_MODULE_GLOBAL_REFERENCE                 0               0          0             0
PVEC_MUTEX                                   0               0          0             0
PVEC_NATIVE_COMP_UNIT                      427           75152        176           176
PVEC_NORMAL_VECTOR                       71693         9205512        128        524304
PVEC_OBARRAY                                88            2816         32            32
PVEC_OTHER                                   0               0          0             0
PVEC_OVERLAY                               219            8760         40            40
PVEC_PROCESS                                49           18424        376           376
PVEC_RECORD                               3425          208504         60           424
PVEC_SQLITE                                  0               0          0             0
PVEC_SUBR                                27492         2639232         96            96
PVEC_SUB_CHAR_TABLE                       3961         3324248        839          1048
PVEC_SYMBOL_WITH_POS                         0               0          0             0
PVEC_TERMINAL                                1             552        552           552
PVEC_THREAD                                  0               0          0             0
PVEC_TS_COMPILED_QUERY                       0               0          0             0
PVEC_TS_NODE                                 0               0          0             0
PVEC_TS_PARSER                               0               0          0             0
PVEC_USER_PTR                                0               0          0             0
PVEC_WEAK_HASH_TABLE                        28            1120         40            40
PVEC_WINDOW                                 30           16560        552           552
PVEC_WINDOW_CONFIGURATION                  111           13320        120           120
PVEC_XWIDGET                                 0               0          0             0
PVEC_XWIDGET_VIEW                            0               0          0             0
commit-limit                                 1              -1          1             0
committed                                    1       400138240  400138240             0
pause-time                                 nil             0.1        nil           nil
reserved                                     1       536875008  536875008             0
spare                                      nil            0.75        nil           nil
spare-committed                              1       217436160  217436160             0

igc-roots-stats:

bc-stack                       ambig               1         4194304
buffer                         ambig               2            1216
charset-table                  ambig               1           60480
control stack                  ambig               1             nil
dump-pins                      ambig               1              24
exact                          exact               1            8192
exact-n                        exact            1286         1112856
exact-ptr                      exact               7              56
kdb-buffer                     ambig               1          262144
lispsym                        exact               1           99120
main-thread                    exact               1             536
pure                           ambig               1         5166672
rdstack                        exact               1             528
specpdl                        exact               1           12256
staticvec                      exact               1           16384
terminal-list                  ambig               1               8
tty-list                       exact               1               8
xpalloc-ambig                  ambig               1            8040
xpalloc-exact                  exact               2             856
xzalloc-ambig                  ambig              11         1286552

I hope that helps -- I certainly have no idea how to read it.

Oh, and by the way - does anyone know how to do a /wide/ reply in a
GNUS Article buffer to the selected part of that message, instead of just a
regular reply via R? So I don't forget to Cc debbugs by accident.
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 02:35:02 GMT) Full text and rfc822 format available.

Message #35 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Alexis Purslane <alexispurslane <at> pm.me>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 02:34:11 +0000
[Message part 1 (text/plain, inline)]
> We're working on fixing some GTK memory leaks in bug#75636; it's
> possible those are partially responsible for growing memory usage.

I rebuilt Emacs with lucid instead of GTK just for the hell of it, and
it's still exhibiting exorbitant monotonicly increasing memory usage, so
I don't think it's GTK. I've had one Emacs open for about two hours,
just with GNUS open in it, doing nothing, on another workspace, and it's
using 533MB. The one I'm using for messing around with SLY and Common
Lisp is using 1.5GB currently, and I've seen it reach similar hights
without an inferior lisp running so I don't think it's that. Still
waiting for a freeze to happen again to attach GDB to it. 
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 03:16:01 GMT) Full text and rfc822 format available.

Message #38 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Alexis Purslane <alexispurslane <at> pm.me>, Pip Cet <pipcet <at> protonmail.com>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 19 Jan 2025 21:15:18 -0600
Alexis Purslane via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs <at> gnu.org> writes:

> Oh, and by the way - does anyone know how to do a /wide/ reply in a
> GNUS Article buffer to the selected part of that message, instead of just a
> regular reply via R? So I don't forget to Cc debbugs by accident.

S v runs the command gnus-summary-very-wide-reply




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 13:00:02 GMT) Full text and rfc822 format available.

Message #41 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexis Purslane <alexispurslane <at> pm.me>
Cc: pipcet <at> protonmail.com, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 14:59:40 +0200
> Cc: 75672 <at> debbugs.gnu.org
> Date: Mon, 20 Jan 2025 02:34:11 +0000
> From:  Alexis Purslane via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> I rebuilt Emacs with lucid instead of GTK just for the hell of it, and
> it's still exhibiting exorbitant monotonicly increasing memory usage, so
> I don't think it's GTK. I've had one Emacs open for about two hours,
> just with GNUS open in it, doing nothing, on another workspace, and it's
> using 533MB. The one I'm using for messing around with SLY and Common
> Lisp is using 1.5GB currently, and I've seen it reach similar hights
> without an inferior lisp running so I don't think it's that. Still
> waiting for a freeze to happen again to attach GDB to it. 

Is this memory really used, or is that just the free memory glibc
keeps to itself and doesn't return to the system?  Can you show the
output of "M-x malloc-info" from the session with 1.5GB footprint?
Does "M-x malloc-trim" make the memory footprint smaller?




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 13:48:01 GMT) Full text and rfc822 format available.

Message #44 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexis Purslane <alexispurslane <at> pm.me>
Cc: pipcet <at> protonmail.com, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 15:47:45 +0200
> Cc: 75672 <at> debbugs.gnu.org
> Date: Mon, 20 Jan 2025 02:33:15 +0000
> From:  Alexis Purslane via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> committed                                    1       400138240  400138240             0
> pause-time                                 nil             0.1        nil           nil
> reserved                                     1       536875008  536875008             0
> spare                                      nil            0.75        nil           nil
> spare-committed                              1       217436160  217436160             0

AFAIU, these 3 values mean that MPS uses about 1.15GB of memory
(unless "reserved" includes "committed").  If "reserved" means memory
MPS reserved but does not use, the question is why does it need so
much reserved memory?




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 14:20:01 GMT) Full text and rfc822 format available.

Message #47 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Alexis Purslane <alexispurslane <at> pm.me>, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 14:19:29 +0000
"Eli Zaretskii" <eliz <at> gnu.org> writes:

>> Cc: 75672 <at> debbugs.gnu.org
>> Date: Mon, 20 Jan 2025 02:33:15 +0000
>> From:  Alexis Purslane via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>>
>> committed                                    1       400138240  400138240             0
>> pause-time                                 nil             0.1        nil           nil
>> reserved                                     1       536875008  536875008             0
>> spare                                      nil            0.75        nil           nil
>> spare-committed                              1       217436160  217436160             0
>
> AFAIU, these 3 values mean that MPS uses about 1.15GB of memory
> (unless "reserved" includes "committed").  If "reserved" means memory

MPS "reserved" means "mmapped, not necessarily faulted in or used".
MPS "committed" means "faulted in and possibly in use by MPS or the
client program".
MPS "spare-committed" means "in use by MPS, not the client program".

So committed - spare-commited is how much usable memory MPS thinks has
been allocated, about 183 MB.

IIUC, the Emacs process was much larger than that, so something else is
wrong: either we got those numbers wrong somehow, or it's malloc()'d or
mmap()'d memory.

> MPS reserved but does not use, the question is why does it need so
> much reserved memory?

It's virtual memory: MPS assumes virtual memory is very cheap.  That's
not really true for processes that fork() a lot, so this assumption may
have to be fine-tuned.

Pip





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 14:52:01 GMT) Full text and rfc822 format available.

Message #50 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Alexis Purslane <alexispurslane <at> pm.me>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 14:51:25 +0000
"Alexis Purslane" <alexispurslane <at> pm.me> writes:

>> We're working on fixing some GTK memory leaks in bug#75636; it's
>> possible those are partially responsible for growing memory usage.
>
> I rebuilt Emacs with lucid instead of GTK just for the hell of it, and
> it's still exhibiting exorbitant monotonicly increasing memory usage, so
> I don't think it's GTK. I've had one Emacs open for about two hours,

Thanks!  That narrows it down.

What worries me most is the continuing increase: MPS should be better at
handing back memory to the operating system than the old allocator, not
worse.

> just with GNUS open in it, doing nothing, on another workspace, and it's
> using 533MB. The one I'm using for messing around with SLY and Common
> Lisp is using 1.5GB currently, and I've seen it reach similar hights

Just to clarify, this is physical memory used, not virtual size, right?

> without an inferior lisp running so I don't think it's that. Still
> waiting for a freeze to happen again to attach GDB to it.

Thanks!

Pip





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 16:53:01 GMT) Full text and rfc822 format available.

Message #53 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: alexispurslane <at> pm.me, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 18:52:26 +0200
> Date: Mon, 20 Jan 2025 14:19:29 +0000
> From: Pip Cet <pipcet <at> protonmail.com>
> Cc: Alexis Purslane <alexispurslane <at> pm.me>, 75672 <at> debbugs.gnu.org
> 
> "Eli Zaretskii" <eliz <at> gnu.org> writes:
> 
> >> Cc: 75672 <at> debbugs.gnu.org
> >> Date: Mon, 20 Jan 2025 02:33:15 +0000
> >> From:  Alexis Purslane via "Bug reports for GNU Emacs,
> >>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> >>
> >> committed                                    1       400138240  400138240             0
> >> pause-time                                 nil             0.1        nil           nil
> >> reserved                                     1       536875008  536875008             0
> >> spare                                      nil            0.75        nil           nil
> >> spare-committed                              1       217436160  217436160             0
> >
> > AFAIU, these 3 values mean that MPS uses about 1.15GB of memory
> > (unless "reserved" includes "committed").  If "reserved" means memory
> 
> MPS "reserved" means "mmapped, not necessarily faulted in or used".
> MPS "committed" means "faulted in and possibly in use by MPS or the
> client program".
> MPS "spare-committed" means "in use by MPS, not the client program".
> 
> So committed - spare-commited is how much usable memory MPS thinks has
> been allocated, about 183 MB.

I agree, but did I say something different?

The question now becomes: how was the 1.5GB figure Alex reported
measured? did it include the reserved memory, or didn't it?

> It's virtual memory: MPS assumes virtual memory is very cheap.  That's
> not really true for processes that fork() a lot, so this assumption may
> have to be fine-tuned.

AFAIK, fork'ed process uses copy-on-write, so we should be okay.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 19:03:02 GMT) Full text and rfc822 format available.

Message #56 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Alexis Purslane <alexispurslane <at> pm.me>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 19:02:06 +0000
[Message part 1 (text/plain, inline)]
"Eli Zaretskii" <eliz <at> gnu.org> writes:

> The question now becomes: how was the 1.5GB figure Alex reported
> measured? did it include the reserved memory, or didn't it?

Finally checked. It was purely resident memory. Total virtual memory usage is about 3x higher.
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 19:06:02 GMT) Full text and rfc822 format available.

Message #59 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Alexis Purslane <alexispurslane <at> pm.me>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 19:05:45 +0000
[Message part 1 (text/plain, inline)]
"Eli Zaretskii" <eliz <at> gnu.org> writes:

> Does "M-x malloc-trim" make the memory footprint smaller?

I haven't had an Emacs session open long enough in a bit to get it up to
1.3GB again, but I did have a 550MB session open, so I ran M-x
malloc-trim on it. That brought it back down to 407MB of resident memory.
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 23:33:04 GMT) Full text and rfc822 format available.

Message #62 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Alexis Purslane <alexispurslane <at> pm.me>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: Pip Cet <pipcet <at> protonmail.com>, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 19:13:36 +0000
[Message part 1 (text/plain, inline)]
"Stefan Kangas" <stefankangas <at> gmail.com> writes:

> S v runs the command gnus-summary-very-wide-reply

That doesn't quite do what I want, since it only works in Summary
buffers, not Article buffers, and doesn't include the original, let
alone allow me to reply only to particular parts of the origina. But I
discovered S W works as wide reply with original in Summary /and/ in
Article buffers (less to remember) /and/ also allows me to reply to
selected parts of the original message in Article buffers, which is
exactly what I wanted.
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Mon, 20 Jan 2025 23:58:01 GMT) Full text and rfc822 format available.

Message #65 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: alexispurslane <at> pm.me, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 21:07:55 +0000
"Eli Zaretskii" <eliz <at> gnu.org> writes:

>> Date: Mon, 20 Jan 2025 14:19:29 +0000
>> From: Pip Cet <pipcet <at> protonmail.com>
>> Cc: Alexis Purslane <alexispurslane <at> pm.me>, 75672 <at> debbugs.gnu.org
>>
>> "Eli Zaretskii" <eliz <at> gnu.org> writes:
>>
>> >> Cc: 75672 <at> debbugs.gnu.org
>> >> Date: Mon, 20 Jan 2025 02:33:15 +0000
>> >> From:  Alexis Purslane via "Bug reports for GNU Emacs,
>> >>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>> >>
>> >> committed                                    1       400138240  400138240             0
>> >> pause-time                                 nil             0.1        nil           nil
>> >> reserved                                     1       536875008  536875008             0
>> >> spare                                      nil            0.75        nil           nil
>> >> spare-committed                              1       217436160  217436160             0
>> >
>> > AFAIU, these 3 values mean that MPS uses about 1.15GB of memory
>> > (unless "reserved" includes "committed").  If "reserved" means memory
>>
>> MPS "reserved" means "mmapped, not necessarily faulted in or used".
>> MPS "committed" means "faulted in and possibly in use by MPS or the
>> client program".
>> MPS "spare-committed" means "in use by MPS, not the client program".
>>
>> So committed - spare-commited is how much usable memory MPS thinks has
>> been allocated, about 183 MB.
>
> I agree, but did I say something different?

Not at all!  I was merely summarizing the (unusual) definitions MPS
uses, and explain why they amount to 183MB of MPS memory in use by
Emacs, not 1.15GB.

> The question now becomes: how was the 1.5GB figure Alex reported
> measured? did it include the reserved memory, or didn't it?

Even if it did, MPS accounts for at most 512MB: reserved is an upper
limit on how much memory is in use.  So we'll have to look for other
culprits, or for very strange MPS/igc bugs.

>> It's virtual memory: MPS assumes virtual memory is very cheap.  That's
>> not really true for processes that fork() a lot, so this assumption may
>> have to be fine-tuned.
>
> AFAIK, fork'ed process uses copy-on-write, so we should be okay.

It does use copy-on-write, but on my machine, at least, Linux appears to
copy the entire page table on every fork, which makes fork quite a bit
more expensive in terms of CPU time than I thought it would be.  This is
noticeable for large processes, as I've previously described.

Pip





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Tue, 21 Jan 2025 00:43:03 GMT) Full text and rfc822 format available.

Message #68 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexis Purslane <alexispurslane <at> pm.me>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Mon, 20 Jan 2025 21:35:15 +0200
> Date: Mon, 20 Jan 2025 19:05:45 +0000
> From: Alexis Purslane <alexispurslane <at> pm.me>
> Cc: 75672 <at> debbugs.gnu.org
> 
> "Eli Zaretskii" <eliz <at> gnu.org> writes:
> 
> > Does "M-x malloc-trim" make the memory footprint smaller?
> 
> I haven't had an Emacs session open long enough in a bit to get it up to
> 1.3GB again, but I did have a 550MB session open, so I ran M-x
> malloc-trim on it. That brought it back down to 407MB of resident memory.

Then this is almost certainly the usual modus operandi of glibc, and
malloc-trim was added to Emacs precisely for these cases.  For
absolute certainty, use "M-x malloc-info" and examine the resulting
report to determine which part of the malloc arena is in the free
lists.

I see no problems here, and nothing that MPS does which we didn't see
in Emacs before.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Tue, 21 Jan 2025 13:37:01 GMT) Full text and rfc822 format available.

Message #71 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: alexispurslane <at> pm.me, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Tue, 21 Jan 2025 15:35:51 +0200
> Date: Mon, 20 Jan 2025 21:07:55 +0000
> From: Pip Cet <pipcet <at> protonmail.com>
> Cc: alexispurslane <at> pm.me, 75672 <at> debbugs.gnu.org
> 
> "Eli Zaretskii" <eliz <at> gnu.org> writes:
> 
> >> >> committed                                    1       400138240  400138240             0
> >> >> pause-time                                 nil             0.1        nil           nil
> >> >> reserved                                     1       536875008  536875008             0
> >> >> spare                                      nil            0.75        nil           nil
> >> >> spare-committed                              1       217436160  217436160             0
> >> >
> >> > AFAIU, these 3 values mean that MPS uses about 1.15GB of memory
> >> > (unless "reserved" includes "committed").  If "reserved" means memory
> >>
> >> MPS "reserved" means "mmapped, not necessarily faulted in or used".
> >> MPS "committed" means "faulted in and possibly in use by MPS or the
> >> client program".
> >> MPS "spare-committed" means "in use by MPS, not the client program".
> >>
> >> So committed - spare-commited is how much usable memory MPS thinks has
> >> been allocated, about 183 MB.
> >
> > I agree, but did I say something different?
> 
> Not at all!  I was merely summarizing the (unusual) definitions MPS
> uses, and explain why they amount to 183MB of MPS memory in use by
> Emacs, not 1.15GB.

That depends on the definition of "in use".  The reserved memory
addresses cannot be used by any other process, so in some sense they
are "in use".

> > The question now becomes: how was the 1.5GB figure Alex reported
> > measured? did it include the reserved memory, or didn't it?
> 
> Even if it did, MPS accounts for at most 512MB: reserved is an upper
> limit on how much memory is in use.  So we'll have to look for other
> culprits, or for very strange MPS/igc bugs.

So you are saying "reserved" includes "committed"?

I'd still like to see the results of memory-info, since that will tell
us what glibc knowns about the memory.  AFAIU, MPS manages only part
of the memory of the process, right?

> >> It's virtual memory: MPS assumes virtual memory is very cheap.  That's
> >> not really true for processes that fork() a lot, so this assumption may
> >> have to be fine-tuned.
> >
> > AFAIK, fork'ed process uses copy-on-write, so we should be okay.
> 
> It does use copy-on-write, but on my machine, at least, Linux appears to
> copy the entire page table on every fork, which makes fork quite a bit
> more expensive in terms of CPU time than I thought it would be.  This is
> noticeable for large processes, as I've previously described.

What is the size of the page table in that case?




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Tue, 21 Jan 2025 13:55:01 GMT) Full text and rfc822 format available.

Message #74 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Alexis Purslane <alexispurslane <at> pm.me>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Pip Cet <pipcet <at> protonmail.com>, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Tue, 21 Jan 2025 13:54:01 +0000
[Message part 1 (text/plain, inline)]
"Eli Zaretskii" <eliz <at> gnu.org> writes:

> I'd still like to see the results of memory-info, since that will tell
> us what glibc knowns about the memory.  AFAIU, MPS manages only part
> of the memory of the process, right?

I've had to restart my Emacs session a few times for unrelated reasons,
so I'm still waiting for it to climb back up to >1GB to report that
back. I got close just now but (again) had to reboot to fix a different
problem I'm trying to solve. 
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Tue, 21 Jan 2025 14:22:02 GMT) Full text and rfc822 format available.

Message #77 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexis Purslane <alexispurslane <at> pm.me>
Cc: pipcet <at> protonmail.com, 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Tue, 21 Jan 2025 16:21:00 +0200
> Date: Tue, 21 Jan 2025 13:54:01 +0000
> From: Alexis Purslane <alexispurslane <at> pm.me>
> Cc: Pip Cet <pipcet <at> protonmail.com>, 75672 <at> debbugs.gnu.org
> 
> "Eli Zaretskii" <eliz <at> gnu.org> writes:
> 
> > I'd still like to see the results of memory-info, since that will tell
> > us what glibc knowns about the memory.  AFAIU, MPS manages only part
> > of the memory of the process, right?
> 
> I've had to restart my Emacs session a few times for unrelated reasons,
> so I'm still waiting for it to climb back up to >1GB to report that
> back. I got close just now but (again) had to reboot to fix a different
> problem I'm trying to solve. 

Thanks.  There's no rush.  It is best to produce this report when the
memory footprint of Emacs is indeed very large, so that the report is
easier to interpret.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Thu, 23 Jan 2025 01:48:02 GMT) Full text and rfc822 format available.

Message #80 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Alexis Purslane <alexispurslane <at> pm.me>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Thu, 23 Jan 2025 01:46:51 +0000
[Message part 1 (text/plain, inline)]
"Alexis Purslane" <alexispurslane <at> pm.me> writes:

An Emacs session got up to 1GB again (even after an M-x malloc-trim,
which shaved like 100MB off!) so I set about trying to figure out how to
get a GDB attached to it to dump its stderr so I could get the output of
M-x malloc-info, but like ten minutes into that (GDB was missing some
plugin so I couldn't run the code inside it necessary to open the file
and redirect the output) suddenly it just.. freed all that memory and
went down to 140MB of resident memory. But it /was/ resident memory, not
virtual memory, that it was using, since it's still using 2GB of virtual
memory.
[publickey - alexispurslane@pm.me - 0x41E61568.asc (application/pgp-keys, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Thu, 23 Jan 2025 07:27:01 GMT) Full text and rfc822 format available.

Message #83 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Alexis Purslane <alexispurslane <at> pm.me>
Cc: 75672 <at> debbugs.gnu.org
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Thu, 23 Jan 2025 09:26:23 +0200
> Date: Thu, 23 Jan 2025 01:46:51 +0000
> From: Alexis Purslane <alexispurslane <at> pm.me>
> Cc: 75672 <at> debbugs.gnu.org
> 
> An Emacs session got up to 1GB again (even after an M-x malloc-trim,
> which shaved like 100MB off!) so I set about trying to figure out how to
> get a GDB attached to it to dump its stderr so I could get the output of
> M-x malloc-info, but like ten minutes into that (GDB was missing some
> plugin so I couldn't run the code inside it necessary to open the file
> and redirect the output)

To avoid the need to use such complicated tricks, you could start
Emacs with stderr redirected to a file to begin with.

> suddenly it just.. freed all that memory and went down to 140MB of
> resident memory.

That's typical of glibc, and also for situations where some relatively
small chunk of used memory prevents releasing a large chunk of free
memory.  So I still don't see any signs of significant leaks here,
maybe just something related to how the MPS-managed memory is laid out
in the address space (do we even have any ways of controlling or
changing that?).

> But it /was/ resident memory, not virtual memory, that it was using,
> since it's still using 2GB of virtual memory.

You mean, Emacs now reserves 2GB, but has only 140MB committed?




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Fri, 21 Feb 2025 18:09:05 GMT) Full text and rfc822 format available.

Message #86 received at 75672 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: bug-gnu-emacs <at> gnu.org, 75672 <at> debbugs.gnu.org,
 Alexis Purslane <alexispurslane <at> pm.me>
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Fri, 21 Feb 2025 18:08:35 +0000
"Alexis Purslane via \"Bug reports for GNU Emacs, the Swiss army knife of text editors\"" <bug-gnu-emacs <at> gnu.org> writes:

> I've been using scratch/igc for the past few days (see details below for
> the exact version and situation) and have been having some /interesting/
> experiences:

Alexis, can you open a new bug for new observations?  That would help us
keep track of what's fixed and what isn't.

Thanks!

Pip





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Fri, 21 Feb 2025 18:09:08 GMT) Full text and rfc822 format available.

Reply sent to Pip Cet <pipcet <at> protonmail.com>:
You have taken responsibility. (Sun, 02 Mar 2025 18:39:01 GMT) Full text and rfc822 format available.

Notification sent to Alexis Purslane <alexispurslane <at> pm.me>:
bug acknowledged by developer. (Sun, 02 Mar 2025 18:39:02 GMT) Full text and rfc822 format available.

Message #94 received at 75672-done <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: bug-gnu-emacs <at> gnu.org, 75672-done <at> debbugs.gnu.org, alexispurslane <at> pm.me
Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues
Date: Sun, 02 Mar 2025 18:38:46 +0000
"Pip Cet via \"Bug reports for GNU Emacs, the Swiss army knife of text editors\"" <bug-gnu-emacs <at> gnu.org> writes:

> "Alexis Purslane via \"Bug reports for GNU Emacs, the Swiss army knife of text editors\"" <bug-gnu-emacs <at> gnu.org> writes:
>
>> I've been using scratch/igc for the past few days (see details below for
>> the exact version and situation) and have been having some /interesting/
>> experiences:
>
> Alexis, can you open a new bug for new observations?  That would help us
> keep track of what's fixed and what isn't.

I'm closing this bug, but thanks for the report and please feel free to
report further observations!

Pip





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#75672; Package emacs. (Sun, 02 Mar 2025 18:40:03 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 31 Mar 2025 11:24:12 GMT) Full text and rfc822 format available.

This bug report was last modified 81 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.