Subject: bug#27194: 25.2; gnus: nnmaildir: article numbers keep growing -> nnmaildir--new-number slow

From: [email-masked]
Date: Fri, 02 Jun 2017 08:38:32 +0200

In Gnus' nnmaildir backend, article numbers grow without bounds.
Getting new news and moving articles into and out of those of my inbox
groups (that see a lot of churn) took really long lately (up to several
minutes).  Profiling revealed that 99% of this time was spent inside
nnmaildir--new-number, and indeed, the affected .nnmaildir/num/
directories contained thousands of hardlinks to article numbers.

For each affected group, I got rid of these exessive article number
hardlinks as follows:

1. move all messages from group to group.new
2. edit the parameters of group.new to match those of group
3. delete group
4. rename group.new to group

(I'm not sure this procedure is guaranteed to be free of unintended side
effects, but I have not observed any.)

Since then, Gnus is snappy again.

This is apparently a poorly-designed data structure / algorithm.  If
nnmaildir requires gapless article number sequences, why doesn't it just
store the highest allocated number?  Why does it waste time on
sequential search (as I suspect)?  If it does not rely on gapless
article numbers, why doesn't it remove hardlinks corresponding to
articles that no longer exist in this group? (After moving all messages
out of an affected group, all 15000+ hardlinks were still present in
.nnmaildir/num/.)

In GNU Emacs 25.2.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.22.10)
 of 2017-04-22 built on juergen
Windowing system distributor 'The X.Org Foundation', version 11.0.11903000
System Description:	Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe
 -fstack-protector-strong' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES

Important settings:
  value of $LC_COLLATE: de_AT.UTF-8
  value of $LC_MONETARY: de_AT.UTF-8
  value of $LC_TIME: de_AT.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Org

Minor modes in effect:
  diff-auto-refine-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  org-indent-mode: t
  show-paren-mode: t
  shell-dirtrack-mode: t
  pdf-occur-global-minor-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Expiring articles...done
No more unseen articles
No more unread articles
Expiring articles...done
Saving /home/piater/.newsrc.eld...
Saving file /home/piater/.newsrc.eld...
Wrote /home/piater/.newsrc.eld
Saving /home/piater/.newsrc.eld...done
(No files need saving)
Scanning for dabbrevs...done

Load-path shadows:
None found.

Features:
(shadow emacsbug latexenc org-colview pcmpl-unix pcmpl-gnu gnus-eform
gnus-fun kmacro gnus-uu yenc profiler eimp ffap locate pdf-sync
pdf-annot pdf-outline pdf-links pdf-history tabify image-file
org-capture term ehelp ispell gnus-draft grep sh-script smie tramp-cache
view woman man vc-git magit-blame magit-stash magit-bisect magit-remote
magit-commit magit-sequence magit magit-apply magit-wip magit-log
magit-diff smerge-mode diff-mode magit-core magit-autorevert autorevert
filenotify magit-process magit-mode magit-popup magit-git crm
magit-section magit-utils git-commit log-edit pcvs-util add-log
with-editor async-bytecomp async tramp-sh dash cal-move diary-lib
diary-loaddefs cal-iso org-indent org-rmail org-mhe org-irc org-info
org-gnus org-docview doc-view org-bibtex bibtex org-bbdb org-w3m
org-agenda mm-extern mm-url nxml-uchnm rng-xsd xsd-regexp rng-cmpct
rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt
rng-util rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap nxml-util
nxml-glyph nxml-enc xmltok dabbrev mailalias smtpmail sendmail dired-aux
flow-fill gnus-dup misearch multi-isearch gnus-cite smiley shr-color
color url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util url-parse url-vars qp shr dom subr-x
mm-archive mail-extr gnus-async gnus-bcklg sort gnus-ml disp-table
gnus-topic utf-7 network-stream nsm starttls imap epa-file epa derived
nndraft nnmh nnmaildir gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig
mailcap nntp gnus-cache nnir gnus-sum gnus-group gnus-undo gnus-start
gnus-cloud nnimap nnmail mail-source tls gnutls utf7 netrc nnoo
parse-time gnus-spec gnus-int gnus-range message idna rfc822 mml mml-sec
epg mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mailabbrev gmm-utils mailheader gnus-win mmm-mode mmm-univ
mmm-class mmm-region mmm-auto mmm-vars mmm-utils mmm-compat paren gnus
gnus-ems nnheader mail-utils gnus-dired ox-latex ox-icalendar ox-html
ox-ascii ox-publish ox org-element org-id org org-macro org-footnote
org-pcomplete org-list org-faces org-entities org-version ob-emacs-lisp
ob ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint
ob-core ob-eval org-compat org-macs org-loaddefs cal-menu calendar
cal-loaddefs ido ess-toolbar ess-mouse mouseme thingatpt browse-url
ess-menu ess-swv ess-noweb ess-noweb-font-lock-mode ess-bugs-l essd-els
ess-sas-d ess-sas-l ess-sas-a ess-sta-d ess-sta-l cc-vars cc-defs
make-regexp ess-sp6-d ess-dde ess-sp3-d ess-julia julia-mode ess-r-d
ess-r-syntax ess-r-completion ess-roxy essddr noutline outline
easy-mmode hideshow ess-help ess-r-package ess-s-l ess ess-inf
ess-tracebug tramp tramp-compat auth-source cl-seq gnus-util mm-util
help-fns mail-prsvr password-cache tramp-loaddefs trampver ucs-normalize
shell pcomplete ess-mode ess-noweb-mode ess-utils ess-generics cl
ess-custom executable ess-compat ess-site pdf-occur ibuf-ext ibuffer
tablist tablist-filter semantic/wisent/comp semantic/wisent
semantic/wisent/wisent semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw eieio eieio-core cl-macs
mode-local find-func cedet dired pdf-isearch let-alist pdf-misc imenu
pdf-tools compile comint ansi-color ring cus-edit cus-start cus-load
wid-edit pdf-view bookmark pp jka-compr pdf-cache pdf-info tq pdf-util
advice format-spec image-mode mozmail server finder-inf info package
epg-config seq byte-opt gv bytecomp byte-compile cl-extra help-mode
easymenu cconv cl-loaddefs pcase cl-lib time-date mule-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 4492676 448493)
 (symbols 48 279807 1477)
 (miscs 40 5257 10111)
 (strings 32 755088 73077)
 (string-bytes 1 46054567)
 (vectors 16 304729)
 (vector-slots 8 4348424 117230)
 (floats 8 9603 2491)
 (intervals 56 295380 4749)
 (buffers 976 313))