Subject: bug#77754: 31.0.50; completion-table-with-quoting and completion-lazy-hilit don't work together

From: Spencer Baugh <[email-masked]>
Date: Fri, 11 Apr 2025 17:47:09 -0400

1. touch 'foo bxx' 'foo byy'
2. emacs -Q
3. M-x shell
4. Type "echo foo" and hit TAB; it will complete to "echo foo\ b"
5. Hit ? to bring up *Completions*
6. Note that the highlighting is incorrect; the next character should
   be highlighted in pink as the next difference, but it's not.

If you modify minibuffer-completion-help to not bind
completion-lazy-hilit to t (which I recently added), the highlighting
will be correct.

This is because completion-lazy-hilit=t makes the highlighting function
run on the *quoted* string (with the \), whereas
completion-lazy-hilit=nil runs highlighting on the *unquoted* string,
which is then highlighted correctly.

IMO, we should fix this by changing the API exposed for
completion-table-with-quoting.  As mentioned in a comment in
completion--twq-all:

   ;; The better solution is to not quote the *Completions* display,
   ;; which nicely circumvents the problem.

If we just stopped quoting *Completions*, we would avoid this problem.

This would necessitate adding some new way to quote the completion
before inserting it.  We could add some new piece of completion metadata
which provides a function which gets called when a completion candidate
is about to be inserted, and returns a string that should be inserted
instead - it can perform the quoting.  (Or it could even be a new
completion table action?)

I'm happy to make that change if that sounds reasonable to others.

(While adding this, I'd also like to make this new function return a
cons whose cdr is the new location of point after inserting the
candidate - that would be useful for various other fixes, including
making the emacs22 style behave better)

In GNU Emacs 31.0.50 (build 9, x86_64-pc-linux-gnu, GTK+ Version
 3.22.30, cairo version 1.15.12) of 2025-04-08 built on igm-qws-u22796a
Repository revision: fb4ae239053b5e03ade2af28573fe6608458a8b7
Repository branch: HEAD
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Rocky Linux 8.10 (Green Obsidian)

Configured using:
 'configure -C --with-gif=no'

Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSELINUX
LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM
XINERAMA XINPUT2 XPM XRANDR GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Eshell

Minor modes in effect:
  eshell-prompt-mode: t
  eshell-pred-mode: t
  eshell-hist-mode: t
  eshell-cmpl-mode: t
  eshell-proc-mode: t
  eshell-arg-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug lisp-mnt message mailcap yank-media puny
dired dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg
rfc6068 epg-config gnus-util time-date mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils pcmpl-unix cl-extra
help-mode em-unix em-term term disp-table ehelp em-script em-prompt
em-pred em-ls em-hist em-glob em-extpipe em-cmpl em-dirs em-basic
em-banner em-alias esh-mode esh-var eshell esh-cmd generator esh-ext
esh-proc esh-opt esh-io esh-arg esh-module esh-module-loaddefs esh-util
pp cl-seq project byte-opt gv sh-script smie treesit executable compile
text-property-search files-x shell pcomplete comint subr-x ansi-osc
ansi-color ring comp-run bytecomp byte-compile comp-common rx warnings
icons cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd touch-screen tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk
x-toolkit xinput2 x multi-tty move-toolbar make-network-process
tty-child-frames native-compile emacs)

Memory information:
((conses 16 136537 11841) (symbols 48 9968 0) (strings 32 57881 2879)
 (string-bytes 1 2842600) (vectors 16 25782)
 (vector-slots 8 321965 10866) (floats 8 62 66) (intervals 56 493 7)
 (buffers 984 16))