Subject: 31.0.50; Dired misses or double-processes files when auto-revert-mode is enabled

From: Tassilo Horn <[email-masked]>
Date: Fri, 17 Jan 2025 08:42:42 +0100

I have a directory test/ with files foobar.N for N in 0 to 99 and do:

emacs -Q test
M-x auto-revert-mode RET
% m .* RET               ; mark all files
Z                        ; compress them all
Z                        ; uncompress them all again
Z                        ; again and again...
...

I always wait until the (un)compress operations are all done before
pressing Z again.  But even though, at some Z you will notice that your
directory doesn't contain only gz or only non-gz files but a mix of
both!

It seems the reason is that auto-revert-mode at some point reverts the
buffer at random points in time while dired is still (un)compressing and
that changes the order of files so that it either misses files or
processes some files twice.

It also seems it is more likely to catch the error when the files take
some time to (un)compress, so I filled them with

   head -c 1000000 /dev/urandom | strings

i.e., random but long enough content.

I've had that issue just half an hour ago with a directory containing
similar gzipped large logfiles.  There, the error hit me so hard that I
basically had an infloop where the same files seemed to be uncompressed
and then compressed over and over again, with just a single "mark all"
and dired-do-compress operation.

In GNU Emacs 31.0.50 (build 79, x86_64-pc-linux-gnu, GTK+ Version
 3.24.43, cairo version 1.18.2) of 2025-01-17 built on thinkpad-t440p
Repository revision: 37b5b3ea91a4ed005664540091e5150d2454d8d6
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --with-tree-sitter --with-pgtk --with-modules'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
LCMS2 LIBOTF LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM GTK3 ZLIB

Important settings:
  value of $LC_MONETARY: de_DE.utf8
  value of $LC_NUMERIC: de_DE.utf8
  value of $LC_TIME: de_DE.utf8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: mu4e:main

Minor modes in effect:
  breadcrumb-mode: t
  editorconfig-mode: t
  global-aggressive-indent-mode: t
  pdf-occur-global-minor-mode: t
  diredfl-global-mode: t
  mu4e-search-minor-mode: t
  mu4e-update-minor-mode: t
  mu4e-context-minor-mode: t
  mu4e-modeline-mode: t
  which-key-mode: t
  highlight-parentheses-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  server-mode: t
  corfu-popupinfo-mode: t
  corfu-history-mode: t
  global-corfu-mode: t
  corfu-mode: t
  vertico-mode: t
  marginalia-mode: t
  minibuffer-depth-indicate-mode: t
  global-eldoc-diffstat-mode: t
  switchy-window-minor-mode: t
  electric-pair-mode: t
  recentf-mode: t
  override-global-mode: t
  repeat-mode: t
  global-so-long-mode: t
  save-place-mode: t
  savehist-mode: t
  puni-global-mode: t
  puni-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  buffer-read-only: t
  column-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  overwrite-mode: overwrite-mode-binary

Load-path shadows:
~/Repos/el/mu/mu4e/mu4e hides ~/Repos/el/mu/build/mu4e/mu4e
~/Repos/el/mu/mu4e/mu4e-modeline hides ~/Repos/el/mu/build/mu4e/mu4e-modeline
~/Repos/el/mu/mu4e/mu4e-context hides ~/Repos/el/mu/build/mu4e/mu4e-context
~/Repos/el/mu/mu4e/mu4e-main hides ~/Repos/el/mu/build/mu4e/mu4e-main
~/Repos/el/mu/mu4e/mu4e-vars hides ~/Repos/el/mu/build/mu4e/mu4e-vars
~/Repos/el/mu/mu4e/mu4e-window hides ~/Repos/el/mu/build/mu4e/mu4e-window
~/Repos/el/mu/mu4e/mu4e-speedbar hides ~/Repos/el/mu/build/mu4e/mu4e-speedbar
~/Repos/el/mu/mu4e/mu4e-view hides ~/Repos/el/mu/build/mu4e/mu4e-view
~/Repos/el/mu/mu4e/mu4e-thread hides ~/Repos/el/mu/build/mu4e/mu4e-thread
~/Repos/el/mu/mu4e/mu4e-bookmarks hides ~/Repos/el/mu/build/mu4e/mu4e-bookmarks
~/Repos/el/mu/mu4e/mu4e-org hides ~/Repos/el/mu/build/mu4e/mu4e-org
~/Repos/el/mu/mu4e/mu4e-lists hides ~/Repos/el/mu/build/mu4e/mu4e-lists
~/Repos/el/mu/mu4e/mu4e-actions hides ~/Repos/el/mu/build/mu4e/mu4e-actions
~/Repos/el/mu/mu4e/mu4e-helpers hides ~/Repos/el/mu/build/mu4e/mu4e-helpers
~/Repos/el/mu/mu4e/mu4e-search hides ~/Repos/el/mu/build/mu4e/mu4e-search
~/Repos/el/mu/mu4e/mu4e-server hides ~/Repos/el/mu/build/mu4e/mu4e-server
~/Repos/el/mu/mu4e/mu4e-obsolete hides ~/Repos/el/mu/build/mu4e/mu4e-obsolete
~/Repos/el/mu/mu4e/mu4e-update hides ~/Repos/el/mu/build/mu4e/mu4e-update
~/Repos/el/mu/mu4e/mu4e-draft hides ~/Repos/el/mu/build/mu4e/mu4e-draft
~/Repos/el/mu/mu4e/mu4e-message hides ~/Repos/el/mu/build/mu4e/mu4e-message
~/Repos/el/mu/mu4e/mu4e-compose hides ~/Repos/el/mu/build/mu4e/mu4e-compose
~/Repos/el/mu/mu4e/mu4e-headers hides ~/Repos/el/mu/build/mu4e/mu4e-headers
~/Repos/el/mu/mu4e/mu4e-query-items hides ~/Repos/el/mu/build/mu4e/mu4e-query-items
~/Repos/el/mu/mu4e/mu4e-notification hides ~/Repos/el/mu/build/mu4e/mu4e-notification
~/Repos/el/mu/mu4e/mu4e-contacts hides ~/Repos/el/mu/build/mu4e/mu4e-contacts
~/Repos/el/mu/mu4e/mu4e-transient hides ~/Repos/el/mu/build/mu4e/mu4e-transient
~/Repos/el/mu/mu4e/mu4e-icalendar hides ~/Repos/el/mu/build/mu4e/mu4e-icalendar
~/Repos/el/mu/mu4e/mu4e-mark hides ~/Repos/el/mu/build/mu4e/mu4e-mark
~/Repos/el/mu/mu4e/mu4e-contrib hides ~/Repos/el/mu/build/mu4e/mu4e-contrib
~/Repos/el/mu/mu4e/mu4e-folders hides ~/Repos/el/mu/build/mu4e/mu4e-folders
~/Repos/el/mu/mu4e/mu4e-mime-parts hides ~/Repos/el/mu/build/mu4e/mu4e-mime-parts
/home/horn/.emacs.d/elpa/ef-themes-1.9.0/theme-loaddefs hides /home/horn/Repos/el/emacs/lisp/theme-loaddefs

Features:
(etags fileloop shortdoc dired-aux dabbrev cape-keyword cape shadow sort
expreg cap-words superword subword face-remap mail-extr emacsbug
misearch multi-isearch eglot external-completion jsonrpc flymake ert
debug backtrace cus-start view help-fns radix-tree tramp-cmds puni
display-fill-column-indicator display-line-numbers tsdh-light-theme
generic yaml-mode fish-mode cargo xref cargo-process rust-utils
rust-mode-treesitter rust-ts-mode rust-mode rust-playpen rust-cargo
rust-common rust-rustfmt rust-compile web-mode disp-table
auctex-autoloads tex-site breadcrumb pulse project editorconfig
editorconfig-core editorconfig-core-handle editorconfig-fnmatch
elfeed-show elfeed-search vc-hg vc-bzr vc-src vc-sccs vc-svn vc-cvs
vc-rcs log-view debbugs-browse elfeed-csv elfeed elfeed-curl elfeed-log
elfeed-db elfeed-lib avl-tree url-queue xml-query hl-todo
aggressive-indent rainbow-mode pdf-occur tablist tablist-filter
semantic/wisent/comp semantic/wisent semantic/wisent/wisent
semantic/util-modes semantic/util semantic semantic/tag semantic/lex
semantic/fw mode-local cedet pdf-isearch pdf-misc pdf-tools pdf-view
jka-compr pdf-cache pdf-info tq pdf-util pdf-macs image-mode exif vc-git
vc-dir ewoc epa-file trashed diredfl dired-x eshell esh-cmd generator
esh-ext esh-proc esh-opt esh-io esh-arg esh-module esh-module-loaddefs
esh-util mu4e-icalendar gnus-icalendar icalendar diary-lib
diary-loaddefs mu4e mu4e-org mu4e-notification notifications mu4e-main
smtpmail mu4e-view mu4e-mime-parts mu4e-headers mu4e-thread mu4e-actions
org-capture org-refile org ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-macro org-src sh-script smie executable ob-comint org-pcomplete
org-list org-footnote org-faces org-entities ob-emacs-lisp ob-core
ob-eval org-cycle org-table ol org-fold org-fold-core org-keys oc
org-loaddefs cal-menu calendar cal-loaddefs org-compat org-version
org-macs mu4e-compose mu4e-draft gnus-msg mu4e-search mu4e-lists
mu4e-bookmarks mu4e-mark mu4e-message flow-fill mule-util mu4e-contacts
mu4e-update mu4e-folders mu4e-context mu4e-query-items mu4e-server
mu4e-modeline mu4e-vars mu4e-helpers mu4e-config mu4e-window
magit-bookmark bookmark ido mu4e-obsolete hippie-exp auto-dictionary
flyspell ispell tramp-smb which-key highlight-parentheses restclient
advice forge-repos forge-tablist hl-line forge-topics forge-commands
forge-semi forge-bitbucket buck forge-gogs gogs forge-gitea gtea
forge-gitlab glab forge-github ghub-graphql treepy gsexp ghub let-alist
forge-forgejo forge-notify forge-revnote forge-pullreq forge-issue
forge-topic yaml eieio-custom forge-post markdown-mode noutline outline
forge-repo forge forge-core forge-db closql emacsql-sqlite emacsql
emacsql-compiler eieio-base magit-submodule magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func imenu magit-diff smerge-mode diff diff-mode track-changes
git-commit log-edit pcvs-util add-log magit-core magit-autorevert
autorevert filenotify magit-margin magit-transient magit-process
with-editor comp comp-cstr server magit-mode benchmark magit-git
magit-base magit-section cursor-sensor crm dash visual-filename-abbrev
rg vc vc-dispatcher rg-info-hack rg-menu transient rg-ibuffer rg-result
wgrep-rg wgrep rg-history rg-header ibuf-ext ibuffer ibuffer-loaddefs
grep debbugs soap-client url-http url-auth url-gw nsm warnings rng-xsd
rng-dt rng-util xsd-regexp debbugs-compat bug-reference thingatpt
kind-icon svg-lib color corfu-popupinfo corfu-history corfu vertico
marginalia icomplete mb-depth eldoc-diffstat use-package-diminish
switchy-window compat elec-pair recentf tree-widget edmacro kmacro
use-package-bind-key bind-key diminish repeat toml-ts-mode json-ts-mode
c++-ts-mode c-ts-mode java-ts-mode c-ts-common find-func treesit so-long
saveplace tramp-cache time-stamp tramp-sh tramp trampver
tramp-integration files-x tramp-message tramp-compat shell pcomplete
format-spec tramp-loaddefs savehist smiley gnus-art mm-uu mml2015
mm-view mml-smime smime gnutls dig gnus-sum shr pixel-fill kinsoku
url-file svg dom gnus-group gnus-undo gnus-start gnus-dbus dbus compile
comint ansi-osc ansi-color ring comp-run comp-common rx xml gnus-cloud
nnimap nnmail mail-source utf7 nnoo parse-time iso8601 gnus-spec
gnus-int gnus-range message sendmail yank-media puny dired
dired-loaddefs rfc822 mml mml-sec epa derived epg rfc6068 epg-config
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mailabbrev gmm-utils mailheader gnus-win gnus nnheader
gnus-util text-property-search time-date mm-util mail-prsvr mail-utils
range ef-themes cl-extra help-mode use-package-ensure use-package-core
finder-inf cus-edit pp cus-load wid-edit aggressive-indent-autoloads
auto-dictionary-autoloads breadcrumb-autoloads cape-autoloads
cargo-autoloads clojure-mode-autoloads corfu-autoloads
csv-mode-autoloads debbugs-autoloads diminish-autoloads
diredfl-autoloads eat-autoloads ef-themes-autoloads
eldoc-diffstat-autoloads elfeed-autoloads ement-autoloads
expreg-autoloads fish-mode-autoloads forge-autoloads closql-autoloads
emacsql-autoloads ghub-autoloads highlight-parentheses-autoloads
hl-todo-autoloads kind-icon-autoloads magit-autoloads pcase
marginalia-autoloads markdown-mode-autoloads mastodon-autoloads
pdf-tools-autoloads persist-autoloads plz-autoloads puni-autoloads
easy-mmode rainbow-mode-autoloads rcirc-color-autoloads
request-autoloads restclient-autoloads rg-autoloads rust-mode-autoloads
svg-lib-autoloads symbol-overlay-autoloads tablist-autoloads
taxy-magit-section-autoloads taxy-autoloads magit-section-autoloads
dash-autoloads tp-autoloads trashed-autoloads treepy-autoloads
vertico-autoloads visual-filename-abbrev-autoloads web-mode-autoloads
wgrep-autoloads info with-editor-autoloads yaml-autoloads
yaml-mode-autoloads package browse-url xdg url url-proxy url-privacy
url-expand url-methods url-history url-cookie generate-lisp-file
url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs icons password-cache json subr-x map byte-opt
gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc iso-transl
tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
touch-screen pgtk-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk pgtk lcms2 multi-tty move-toolbar
make-network-process tty-child-frames native-compile emacs)

Memory information:
((conses 16 1064655 204213) (symbols 48 58002 17) (strings 32 291793 11325)
 (string-bytes 1 8415463) (vectors 16 109237) (vector-slots 8 1246508 142280)
 (floats 8 918 970) (intervals 56 29636 1707) (buffers 992 35))