Subject: bug#46379: 28.0.50; regression in lm-commentary, take two

From: Matt Armstrong <[email-masked]>
Date: Sun, 07 Feb 2021 16:03:54 -0800

This is a re-send of Bug#46364, which ended up closed by mutual
agreement without a clear description of the problem.

The issue is a behavior change in `lm-commentary' that probably deserves
some consideration, because it seems suboptimal in this example. The
behavior change is most likely caused by commit
963a9ffd66cb29f0370e9a4b854dddda242c54a6.

Prior to that commit, and in Emacs 27.1, the function returns the file's
commentary as an verbatim substring of the elisp source, including the
elisp comment characters, commentary headline, etc. E.g. a string like
this (I'll quote the string for email sanity):

> ";;; Commentary:
> ;;
> ;; This package extends and configures `dired' with features found in
> ;; almost all 'file managers', and also some unique features:
> ;;
> ;;   * Resilient dedicated dual-pane frame.
> ;;     * similar look to 'midnight commander'.
> ;;     * intelligent recovery of manually altered frame configuration
> ;;     * exit diredc/dired cleanly and totally
[...]

After that commit, the function returns a "sanitized" version of the
same content, such as:

> "
> This package extends and configures `dired' with features found in
> almost all 'file managers', and also some unique features:
> 
> * Resilient dedicated dual-pane frame.
> * similar look to 'midnight commander'.
> * intelligent recovery of manually altered frame configuration
> * exit diredc/dired cleanly and totally
[...]

The original Bug#46364 merely said the function "does not correctly
indent sub-headings." From the above, we can see this. Perhaps a clearer
way to say this would be that `lm-commentary' now seems to strip all
leading whitespace from every line, as a "sanitization" step, and this
has the unsatisfying side effect of ruining any indentation formatting
in the original commentary.

We need not go farther than Emacs' own lisp/align.el to see a similar
problem.

In Emacs 27, (lm-commentary "align.el") returns:

> ";;; Commentary:
> 
> ;; This mode allows you to align regions in a context-sensitive fashion.
> ;; The classic use is to align assignments:
> ;;
> ;;    int a = 1;
> ;;    short foo = 2;
> ;;    double blah = 4;
> ;;
> ;; becomes
> ;;
> ;;    int    a    = 1;
> ;;    short  foo  = 2;
> ;;    double blah = 4;
> 
> "

And in Emacs 28 (mainline):

> "This mode allows you to align regions in a context-sensitive fashion.
> The classic use is to align assignments:
> 
> int a = 1;
> short foo = 2;
> double blah = 4;
> 
> becomes
> 
> int    a    = 1;
> short  foo  = 2;
> double blah = 4;"

Perhaps this should be re-thought?

Note that Bug#46364 also mentioned MELPA, which is relevant only because
in some cases MELPA code calls `lm-commentary' to form user-facing
description of a package.

In GNU Emacs 28.0.50 (build 1, x86_64-apple-darwin20.2.0, NS appkit-2022.20 Version 11.1 (Build 20C69))
 of 2021-02-05 built on matts-mbp-2016.lan
Repository revision: 0484879d3b0f81222bacbc3c9655d1cfcdb5d321
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2022
System Description:  macOS 11.1

Configured features:
ACL GMP GNUTLS JPEG LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER PNG THREADS
TIFF TOOLKIT_SCROLL_BARS ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  msb-mode: t
  winner-mode: t
  electric-pair-mode: t
  auto-insert-mode: t
  display-time-mode: t
  show-paren-mode: t
  keyfreq-autosave-mode: t
  keyfreq-mode: t
  icomplete-mode: t
  magit-auto-revert-mode: t
  global-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  global-flycheck-mode: t
  which-key-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr msb emacsbug sendmail org-element avl-tree ol-w3m
ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect gnus-search eieio-opt
speedbar ezimage dframe gnus-art mm-uu mml2015 mm-view mml-smime smime
dig gnus-sum gnus-group gnus-undo gnus-start gnus-dbus dbus gnus-cloud
nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range
gnus-win ol-eww eww xdg url-queue mm-url gnus nnheader ol-docview
doc-view jka-compr image-mode exif ol-bibtex bibtex ol-bbdb winner
cus-start cus-load elec-pair slime-fancy slime-indentation
slime-cl-indent cl-indent slime-trace-dialog slime-fontifying-fu
slime-package-fu slime-references slime-compiler-notes-tree
slime-scratch slime-presentations bridge slime-macrostep macrostep
slime-mdot-fu slime-enclosing-context slime-fuzzy slime-fancy-trace
slime-fancy-inspector slime-c-p-c slime-editing-commands slime-autodoc
slime-repl elp slime-parse slime gud apropos arc-mode archive-mode
hyperspec slime-autoloads warnings org-protocol elisp-format protbuf
org-drill persist org-id org-agenda org-refile org ob ob-tangle ob-ref
ob-lob ob-table ob-exp org-macro org-footnote org-src ob-comint
org-pcomplete org-list org-faces org-entities org-version ob-emacs-lisp
ob-core ob-eval org-table ol org-keys org-compat org-macs org-loaddefs
cal-menu calendar cal-loaddefs diminish ctypes skeleton autoinsert time
paren ert debug backtrace keyfreq generic icomplete highlight-symbol
hi-lock cl racket-mode racket-bug-report racket-collection tq
racket-repl-buffer-name racket-stepper racket-logger racket-profile
racket-smart-open racket-xp racket-xp-complete racket-show pos-tip
racket-imenu racket-edit hideshow racket-repl semantic/symref/grep
semantic/symref semantic/util-modes semantic/util semantic pp
semantic/tag semantic/lex semantic/fw mode-local cedet racket-eldoc
racket-describe shr kinsoku svg xml dom racket-visit racket-complete
racket-common racket-parens racket-indent racket-font-lock racket-ppss
racket-keywords-and-builtins racket-doc racket-cmd racket-util
racket-browse-url racket-custom sh-script smie executable company-lsp
company lsp-mode lsp-protocol tree-widget wid-edit pcase network-stream
nsm markdown-mode noutline outline lv inline ht f s ewoc dash-functional
bindat projectile grep ibuf-ext ibuffer ibuffer-loaddefs cider tramp-sh
cider-debug cider-browse-ns cider-mode cider-inspector cider-completion
cider-profile cider-eval cider-repl-history pulse cider-repl
cider-resolve cider-test cider-overlays cider-stacktrace cider-doc
cider-browse-spec cider-clojuredocs cider-eldoc cider-client
cider-common cider-connection cider-util color cider-popup
sesman-browser nrepl-client tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 ls-lisp queue
nrepl-dict cider-compat spinner parseedn parseclj-parser parseclj-lex a
sesman vc vc-dispatcher edmacro kmacro clojure-mode lisp-mnt align
magit-submodule magit-obsolete magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log which-func imenu
magit-diff smerge-mode diff diff-mode magit-core magit-autorevert
autorevert filenotify magit-margin magit-transient magit-process
magit-mode git-commit transient format-spec magit-git magit-section
magit-utils crm log-edit message rmc puny dired dired-loaddefs rfc822
mml mml-sec epa derived epg epg-config gnus-util rmail rmail-loaddefs
time-date mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047
rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils
mailheader pcvs-util add-log with-editor async-bytecomp async shell
pcomplete server ido flycheck find-func rx dash go-mode find-file ffap
thingatpt etags fileloop generator xref project compile
text-property-search comint ansi-color ring which-key advice
exec-path-from-shell cl-extra help-mode use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
easy-mmode use-package-core finder-inf info package easymenu browse-url
url url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
iso-transl tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/ns-win ns-win ucs-normalize mule-util
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads kqueue cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 724471 37341)
 (symbols 48 55018 42)
 (strings 32 239918 4858)
 (string-bytes 1 7129508)
 (vectors 16 84926)
 (vector-slots 8 932975 39107)
 (floats 8 726 157)
 (intervals 56 550 234)
 (buffers 984 12))