GNU bug report logs - #79279
28.2; Combining characters cause formatting problems when showing CAP candidates

Previous Next

To reply to this bug, email your comments to 79279 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Safronov <saf.dmitry <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.2; Combining characters cause formatting problems when showing CAP
 candidates
Date: Wed, 20 Aug 2025 17:44:12 +0200

[Message part 1 (text/plain, inline)]

When triggering completion-at-point for a Unicode symbol in Julia
mode, description
strings for some completion candidates in the minibuffer appear
misaligned, when the first character in a description string is a
combining diacritical mark (see the
attached screenshot). This is apparently because the space character immediately
preseeding the combining diacritical mark gets "consumed", that is, combined
with the diacritical mark. This issue is not specific to Julia mode,
but appears to be a more general problem of presenting text containing
leading combining characters.

In GNU Emacs 28.2 (build 1, x86_64-apple-darwin18.7.0, NS
appkit-1671.60 Version 10.14.6 (Build 18G95))
of 2022-09-12 built on builder10-14.lan
Windowing system distributor 'Apple', version 10.3.1894
System Description:  Mac OS X 10.15.7

Configured using:
'configure --with-ns '--enable-locallisppath=/Library/Application
Support/Emacs/${version}/site-lisp:/Library/Application Support/Emacs/
site-lisp' --with-modules'

Configured features:
ACL GMP GNUTLS JSON LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER THREADS
TOOLKIT_SCROLL_BARS ZLIB

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LC_CTYPE: UTF-8
  value of $LANG: en_US
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  display-line-numbers-mode: t
  vertico-mode: t
  global-company-mode: t
  company-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  global-auto-revert-mode: t
  save-place-mode: t
  adaptive-wrap-prefix-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  window-divider-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  global-visual-line-mode: t
  visual-line-mode: t
  transient-mark-mode: t
  auto-save-visited-mode: t

Load-path shadows:
~/.emacs.d/lisp/markdown-mode hides
/Users/dmitry/.emacs.d/elpa/markdown-mode-20250403.1127/markdown-mode
/Users/dmitry/.emacs.d/elpa/jsonrpc-1.0.25/jsonrpc hides
/Applications/Emacs.app/Contents/Resources/lisp/jsonrpc
/Users/dmitry/.emacs.d/elpa/xref-1.7.0/xref hides
/Applications/Emacs.app/Contents/Resources/lisp/progmodes/xref
/Users/dmitry/.emacs.d/elpa/project-0.11.1/project hides
/Applications/Emacs.app/Contents/Resources/lisp/progmodes/project
/Users/dmitry/.emacs.d/elpa/eldoc-1.15.0/eldoc hides
/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/eldoc

Features:
(whitespace pp shadow sort mail-extr emacsbug message rmc puny dired
dired-loaddefs rfc822 mml mml-sec epa epg rfc6068 epg-config gnus-util
rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-
util mail-prsvr mail-utils imenu eieio-opt speedbar ezimage dframe find-
func shortdoc text-property-search help-fns radix-tree python tramp-sh
tramp tramp-loaddefs trampver tramp-integration files-x tramp-compat
shell pcomplete ls-lisp format-spec markdown-mode color thingatpt vc-git
diff-mode vc-dispatcher server display-line-numbers bm neotree vertico
compat compat-29 company-dabbrev-code company-dabbrev company-keywords
company-yasnippet company-capf company yasnippet pcase cl-extra help-
mode julia-mode derived julia-mode-latexsubs lua-mode advice rx comint
ansi-color ring deft edmacro kmacro taskpaper-mode cal-iso parse-time
iso8601 time-date noutline outline holidays-de holidays hol-loaddefs cal
-menu calendar cal-loaddefs autorevert filenotify hl-line saveplace
adaptive-wrap easy-mmode disp-table atom-one-dark-theme finder-inf info
package browse-url url url-proxy url-privacy url-expand url-methods url-
history url-cookie url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs password-
cache json subr-x map url-vars seq byte-opt gv bytecomp byte-compile
cconv cl-loaddefs cl-lib iso-transl tooltip eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/ns-
win ns-win ucs-normalize mule-util term/common-win tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode lisp
-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-
core term/tty-colors frame minibuffer cl-generic cham georgian utf-8-
lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese
eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese composite emoji-zwj charscript charprop case-
table epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded
nadvice button loaddefs faces cus-face macroexp files window text-
properties overlay sha1 md5 base64 format env code-pages mule custom
widget hashtable-print-readable backquote threads kqueue cocoa ns multi-
tty make-network-process emacs)

Memory information:
((conses 16 498967 48249)
(symbols 48 27693 3)
(strings 32 168335 5079)
(string-bytes 1 3797261)
(vectors 16 42646)
(vector-slots 8 1438070 195491)
(floats 8 254 307)
(intervals 56 942 365)
(buffers 992 17))

[Screen Shot 2025-08-20 at 17.08.48.png (image/png, attachment)]

Message #8 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dmitry Safronov <saf.dmitry <at> gmail.com>
Cc: 79279 <at> debbugs.gnu.org
Subject: Re: bug#79279: 28.2;
 Combining characters cause formatting problems when showing CAP
 candidates
Date: Thu, 21 Aug 2025 08:04:06 +0300

> From: Dmitry Safronov <saf.dmitry <at> gmail.com>
> Date: Wed, 20 Aug 2025 17:44:12 +0200
> 
> When triggering completion-at-point for a Unicode symbol in Julia
> mode, description
> strings for some completion candidates in the minibuffer appear
> misaligned, when the first character in a description string is a
> combining diacritical mark (see the
> attached screenshot). This is apparently because the space character immediately
> preseeding the combining diacritical mark gets "consumed", that is, combined
> with the diacritical mark. This issue is not specific to Julia mode,
> but appears to be a more general problem of presenting text containing
> leading combining characters.

Thanks.  However, I'm not sure I understand which part of Emacs or of
some add-on package produces this display, so it is hard to guess
where to look for code which needs to be fixed to avoid this problem.
And you haven't provided a recipe for reproducing the issue, which
makes it harder yet.

Presenting text which starts with a combining character indeed needs
some special code, but that code should be in the Lisp program which
creates the text.  And given what you tell, I don't yet have an idea
where that Lisp program is: in Emacs core, in Julia, somewhere else?

Could you show a recipe for reproducing the problem?  Preferably, such
a recipe should start from "emacs -Q" and turn on any optional
features needed to show the problem.  If possible, the recipe should
not include any third-party packages if you can demonstrate the
problematic completion buffer display without those packages.  E.g.,
if the problem is in completion-at-point, can you show which commands
to invoke to produce display such as in your screenshot, preferably
without involving Julia?

Message #11 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dmitry Safronov <saf.dmitry <at> gmail.com>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>, Juri Linkov <juri <at> linkov.net>
Cc: 79279 <at> debbugs.gnu.org
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Thu, 21 Aug 2025 17:00:52 +0300

[Please use Reply All to reply, to keep the bug tracker on the CC list.]

> From: Dmitry Safronov <saf.dmitry <at> gmail.com>
> Date: Thu, 21 Aug 2025 13:25:58 +0200
> 
> > Presenting text which starts with a combining character indeed needs
> some special code, but that code should be in the Lisp program which
> creates the text.  And given what you tell, I don't yet have an idea
> where that Lisp program is: in Emacs core, in Julia, somewhere else?
> 
> The text was indeed produced by the julia-mode package, namely by the
> `julia--latexsub-capf-list` function for creating a CAPF completion
> table for that mode.
> However, the list of completion candidates (based on that completion
> table) in the
> "*Completions*" buffer was produced by the Emacs core (`display-completion-list`
> function defined in `minibuffer.el`).
> 
> Therefore, I think, a proper handling of annotation text which starts with a
> combining character should nevertheless be addressed in Emacs core (for example,
> by prepending annotations starting with a combining character by a
> zero-width space
> U+200B).

But display-completion-list just shows the strings it gets as its
argument.  AFAIU, theses strings already include both the completion
candidate and its visualization (which causes the problem).  If that
is correct, then the code which generates those strings is the one we
need to fix.  Can you (or someone else of the people I CC) point to
the code which generates those strings?

Message #14 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79279 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 Dmitry Safronov <saf.dmitry <at> gmail.com>
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Thu, 21 Aug 2025 19:04:17 +0300

[Message part 1 (text/plain, inline)]

>> The text was indeed produced by the julia-mode package, namely by the
>> `julia--latexsub-capf-list` function for creating a CAPF completion
>> table for that mode.
>> However, the list of completion candidates (based on that completion
>> table) in the
>> "*Completions*" buffer was produced by the Emacs core (`display-completion-list`
>> function defined in `minibuffer.el`).
>
> But display-completion-list just shows the strings it gets as its
> argument.  AFAIU, theses strings already include both the completion
> candidate and its visualization (which causes the problem).  If that
> is correct, then the code which generates those strings is the one we
> need to fix.  Can you (or someone else of the people I CC) point to
> the code which generates those strings?

This was already fixed in julia-mode long ago.
The current version adds the preceding space character:

  :annotation-function (lambda (s)
                         (concat " " (gethash s julia-mode-latexsubs)))

So there is no such problem anymore:

[vec.png (image/png, attachment)]

[Message part 3 (text/plain, inline)]

PS: To make it even nicer, it's better to show the character
before completions like in 'C-x 8 RET TAB' with something like:

  :affixation-function (lambda (strings)
                         (mapcar (lambda (s)
                                   (list s (concat (gethash s julia-mode-latexsubs) "\t") ""))
                                 strings))

Message #17 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: 79279 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca, saf.dmitry <at> gmail.com
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Thu, 21 Aug 2025 21:19:12 +0300

> From: Juri Linkov <juri <at> linkov.net>
> Cc: Dmitry Safronov <saf.dmitry <at> gmail.com>,  Stefan Monnier
>  <monnier <at> iro.umontreal.ca>,  79279 <at> debbugs.gnu.org
> Date: Thu, 21 Aug 2025 19:04:17 +0300
> 
> >> The text was indeed produced by the julia-mode package, namely by the
> >> `julia--latexsub-capf-list` function for creating a CAPF completion
> >> table for that mode.
> >> However, the list of completion candidates (based on that completion
> >> table) in the
> >> "*Completions*" buffer was produced by the Emacs core (`display-completion-list`
> >> function defined in `minibuffer.el`).
> >
> > But display-completion-list just shows the strings it gets as its
> > argument.  AFAIU, theses strings already include both the completion
> > candidate and its visualization (which causes the problem).  If that
> > is correct, then the code which generates those strings is the one we
> > need to fix.  Can you (or someone else of the people I CC) point to
> > the code which generates those strings?
> 
> This was already fixed in julia-mode long ago.
> The current version adds the preceding space character:
> 
>   :annotation-function (lambda (s)
>                          (concat " " (gethash s julia-mode-latexsubs)))
> 
> So there is no such problem anymore:

So you think we don't need to solve this in minibuffer.el for all the
users of this?

Message #20 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79279 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca, saf.dmitry <at> gmail.com
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Thu, 21 Aug 2025 21:24:51 +0300

>> >> The text was indeed produced by the julia-mode package, namely by the
>> >> `julia--latexsub-capf-list` function for creating a CAPF completion
>> >> table for that mode.
>> >> However, the list of completion candidates (based on that completion
>> >> table) in the
>> >> "*Completions*" buffer was produced by the Emacs core (`display-completion-list`
>> >> function defined in `minibuffer.el`).
>> >
>> > But display-completion-list just shows the strings it gets as its
>> > argument.  AFAIU, theses strings already include both the completion
>> > candidate and its visualization (which causes the problem).  If that
>> > is correct, then the code which generates those strings is the one we
>> > need to fix.  Can you (or someone else of the people I CC) point to
>> > the code which generates those strings?
>> 
>> This was already fixed in julia-mode long ago.
>> The current version adds the preceding space character:
>> 
>>   :annotation-function (lambda (s)
>>                          (concat " " (gethash s julia-mode-latexsubs)))
>> 
>> So there is no such problem anymore:
>
> So you think we don't need to solve this in minibuffer.el for all the
> users of this?

I think it's responsibility of the caller to add a space
when it's known that the completion candidate might contain
a combining character.

Message #23 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Safronov <saf.dmitry <at> gmail.com>
To: Juri Linkov <juri <at> linkov.net>
Cc: 79279 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 monnier <at> iro.umontreal.ca
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Thu, 21 Aug 2025 22:43:26 +0200

I experimented a bit using different fonts and came to the conclusion that the
formatting issue appears only if a font does not support proper
rendering of combined
characters. Ironically, the JuliaMono font does support it and it
causes the formatting problem
described in the original post.

> I think it's responsibility of the caller to add a space
> when it's known that the completion candidate might contain
> a combining character.

So well, nevermind, I will open an issue at
<https://github.com/JuliaEditorSupport/julia-emacs>.

On Thu, 21 Aug 2025 at 20:25, Juri Linkov <juri <at> linkov.net> wrote:
>
> >> >> The text was indeed produced by the julia-mode package, namely by the
> >> >> `julia--latexsub-capf-list` function for creating a CAPF completion
> >> >> table for that mode.
> >> >> However, the list of completion candidates (based on that completion
> >> >> table) in the
> >> >> "*Completions*" buffer was produced by the Emacs core (`display-completion-list`
> >> >> function defined in `minibuffer.el`).
> >> >
> >> > But display-completion-list just shows the strings it gets as its
> >> > argument.  AFAIU, theses strings already include both the completion
> >> > candidate and its visualization (which causes the problem).  If that
> >> > is correct, then the code which generates those strings is the one we
> >> > need to fix.  Can you (or someone else of the people I CC) point to
> >> > the code which generates those strings?
> >>
> >> This was already fixed in julia-mode long ago.
> >> The current version adds the preceding space character:
> >>
> >>   :annotation-function (lambda (s)
> >>                          (concat " " (gethash s julia-mode-latexsubs)))
> >>
> >> So there is no such problem anymore:
> >
> > So you think we don't need to solve this in minibuffer.el for all the
> > users of this?
>
> I think it's responsibility of the caller to add a space
> when it's known that the completion candidate might contain
> a combining character.

Message #26 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: 79279 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca, saf.dmitry <at> gmail.com
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Fri, 22 Aug 2025 09:23:39 +0300

> From: Juri Linkov <juri <at> linkov.net>
> Cc: saf.dmitry <at> gmail.com,  monnier <at> iro.umontreal.ca,  79279 <at> debbugs.gnu.org
> Date: Thu, 21 Aug 2025 21:24:51 +0300
> 
> >> >> The text was indeed produced by the julia-mode package, namely by the
> >> >> `julia--latexsub-capf-list` function for creating a CAPF completion
> >> >> table for that mode.
> >> >> However, the list of completion candidates (based on that completion
> >> >> table) in the
> >> >> "*Completions*" buffer was produced by the Emacs core (`display-completion-list`
> >> >> function defined in `minibuffer.el`).
> >> >
> >> > But display-completion-list just shows the strings it gets as its
> >> > argument.  AFAIU, theses strings already include both the completion
> >> > candidate and its visualization (which causes the problem).  If that
> >> > is correct, then the code which generates those strings is the one we
> >> > need to fix.  Can you (or someone else of the people I CC) point to
> >> > the code which generates those strings?
> >> 
> >> This was already fixed in julia-mode long ago.
> >> The current version adds the preceding space character:
> >> 
> >>   :annotation-function (lambda (s)
> >>                          (concat " " (gethash s julia-mode-latexsubs)))
> >> 
> >> So there is no such problem anymore:
> >
> > So you think we don't need to solve this in minibuffer.el for all the
> > users of this?
> 
> I think it's responsibility of the caller to add a space
> when it's known that the completion candidate might contain
> a combining character.

The alternative is for us to always add a special character there,
like ZWNJ or maybe invisible TAB, like we do in other places where we
want to prevent character composition.  But if you think this is
unnecessary, I'm okay with leaving that to the calling Lisp program.

Message #29 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79279 <at> debbugs.gnu.org, saf.dmitry <at> gmail.com, Juri Linkov <juri <at> linkov.net>
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Fri, 22 Aug 2025 03:29:02 -0400

> The alternative is for us to always add a special character there,
> like ZWNJ or maybe invisible TAB, like we do in other places where we
> want to prevent character composition.

That would be The Right Thing to do, I think.


        Stefan

Message #32 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79279 <at> debbugs.gnu.org, saf.dmitry <at> gmail.com, Juri Linkov <juri <at> linkov.net>
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Fri, 22 Aug 2025 03:38:02 -0400

>> The alternative is for us to always add a special character there,
>> like ZWNJ or maybe invisible TAB, like we do in other places where we
>> want to prevent character composition.
> That would be The Right Thing to do, I think.

[ Sorry, sent it a bit precipitously. ]

In practice it's rare for completion candidates to start with combining
characters, so it's maybe not a bad tradeoff to dump that burden on the
few completion tables where we know it's more likely to occur.

Still, if it can be done cheaply enough in `minibuffer.el`, it's much
better, because the poor souls writing those completion tables are
unlikely to be familiar with the issue and how best to circumvent it.


        Stefan

Message #35 received at 79279 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 79279 <at> debbugs.gnu.org, saf.dmitry <at> gmail.com, juri <at> linkov.net
Subject: Re: bug#79279: 28.2; Combining characters cause formatting problems
 when showing CAP candidates
Date: Sat, 23 Aug 2025 15:59:30 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: Juri Linkov <juri <at> linkov.net>,  saf.dmitry <at> gmail.com,
>   79279 <at> debbugs.gnu.org
> Date: Fri, 22 Aug 2025 03:38:02 -0400
> 
> >> The alternative is for us to always add a special character there,
> >> like ZWNJ or maybe invisible TAB, like we do in other places where we
> >> want to prevent character composition.
> > That would be The Right Thing to do, I think.
> 
> [ Sorry, sent it a bit precipitously. ]
> 
> In practice it's rare for completion candidates to start with combining
> characters, so it's maybe not a bad tradeoff to dump that burden on the
> few completion tables where we know it's more likely to occur.
> 
> Still, if it can be done cheaply enough in `minibuffer.el`, it's much
> better, because the poor souls writing those completion tables are
> unlikely to be familiar with the issue and how best to circumvent it.

AFAIU, the problem happens when we insert two strings one after the
other in completion--insert:

    ;; If `str' is a list that has 2 elements,
    ;; then the second element is a suffix annotation.
    ;; If `str' has 3 elements, then the second element
    ;; is a prefix, and the third element is a suffix.
    (let* ((prefix (when (nth 2 str) (nth 1 str)))
           (suffix (or (nth 2 str) (nth 1 str))))
      (when prefix
        (let ((beg (point))
              (end (progn (insert prefix) (point))))
          (add-text-properties beg end `(mouse-face nil completion--string ,(car str)))))
      (completion--insert (car str) group-fun)
      (let ((beg (point))
            (end (progn (insert suffix) (point))))

We could handle the problem there, I think.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.