GNU bug report logs - #77872
[PATCH 1/2] ansi-term: ignore CSI commands with subparams

Previous Next

Full log

Message #17 received at 77872 <at> debbugs.gnu.org (full text, mbox):

From: Jared Finder <jared <at> finder.org>
To: Johannes Altmanninger <aclopte <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77872 <at> debbugs.gnu.org
Subject: Re: bug#77872: [PATCH 1/2] ansi-term: ignore CSI commands with
 subparams
Date: Fri, 18 Apr 2025 22:33:36 -0700

On 2025-04-18 02:00, Johannes Altmanninger wrote:
> On Fri, Apr 18, 2025 at 09:05:33AM +0300, Eli Zaretskii wrote:
>> > Cc: Johannes Altmanninger <aclopte <at> gmail.com>
>> > From: Johannes Altmanninger <aclopte <at> gmail.com>
>> > Date: Thu, 17 Apr 2025 20:46:32 +0200
>> >
>> > * lisp/term.el (term-handle-ansi-escape): Check if parameters contain
>> > subparameters (values separated by colon) and ignore those commands.
>> > This prevents incorrect interpretation of advanced terminal features
>> > that aren't yet properly supported, such as curly underlines (\e[4:3m).
>> >
>> > Consider ignoring whitespace changes when reviewing this patch.
>> > The essence of this change is the addition at the beginning of
>> > term-handle-ansi-escape, to skip all CSI commands that contain at
>> > least two subparameters (e.g. the ones that contain at least one ":").
>> >
>> > Some background / longer description:
>> >
>> > If I run
>> >
>> > 	emacs --eval '(ansi-term "/bin/bash")'
>> >
>> > and type
>> >
>> > 	printf '\e[4:3mHELLO\e[m\n'
>> >
>> > then I get text with a straight underline.
>> >
>> > This is not what I expect, because "\e[4:3m" is supposed to turn on
>> > curly/wavy underlines, see https://sw.kovidgoyal.net/kitty/underlines/.
>> >
>> > My program wants to use curly underlines to indicate errors. Rendering
>> > those as straight underlines is noisy and confusing because it suggests
>> > a different meaning (in my program).
>> 
>> Why cannot you solve this problem by a suitable configuration of faces
>> in Emacs?
> 
> I'm implementing the default behavior for a shell; it doesn't behoove
> the shell to touch the user's (Emacs) configuration.
> 
> Now of course I could theoretically achieve my goal with this logic:
> 
>     if the terminal supports underline styles (terminfo Smulx or Su):
>         use curly underlines
>     else:
>         use no underline at all
> 
> where the condition could be checked by:
> 1. reading the terminfo database based on $TERM
> 2. querying the terminal's embedded terminfo database via XTGETTCAP
> 
> Option 1 is not great because terminfo frequently has false negatives,
> and many of those cannot easily be fixed because they use a generic
> $TERM (that's the problem with user-agents).  For example, GNOME
> Terminal and KDE's Konsole both support styled underlines but their
> TERM=xterm-256color says otherwise.
> 
>     # False negative.
>     $ infocmp -x xterm-256color | grep -E 'Smulx|Su'
> 
>     # Another false negative (here the terminfo db is "fixable").
>     $ echo $TERM
>     foot
>     $ infocmp -x $TERM | grep -E 'Smulx|Su'
> 
>     # True positive.
>     $ infocmp -x kitty | grep -E 'Smulx|Su'
>     	Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, TS=\E]2;,
>     $ infocmp -x xterm-kitty | grep -E 'Smulx|Su'
>     	am, ccc, hs, km, mc5i, mir, msgr, npc, xenl, Su, Tc, XF, fullkbd,
>     	Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, Sync=\EP=%p1%ds\E\\,
> 
> Note that only few terminals are "fixable" (by virtue of using a
> non-impersonating $TERM), That comes with the added burden that one
> may need to copy terminfo files to remote servers etc., which is
> probably part of why few terminals do this.
> 
> Option 2 (XTGETTCAP) always gives the right answer because we are
> asking the terminal directly -- both the request and response are
> control sequences written to the pty.  However it's only implemented
> by xterm, kitty and foot so far, so I only use it if there is no
> other good option.
> 
> But I think there *is* a better (long-term) option here: if we can
> assume that a terminal ignores commands that it doesn't recognize,
> we don't need to query the capability at all, thus making it much
> simpler to use curly underlines in e.g. a bash script. In practice,
> this is already the case for a lot of other sequences.
> 
> Now the case of «printf "\e[4:3m"» may not be as obvious.
> One argument for ignoring the sequence it is that if a user really
> wants ansi-term's current behavior of "use curly underlines if
> supported, else use straight underlines", they can simply print both:
> "\e[4m\e[4:3m", or equivalently "\e[4;4:3m".
> 
> XTerm ignores "\e[4:3m" and most the dozens of terminals I tested
> follow suit.  However I give you that there are some (important)
> terminals render curly underline as straight underline:
> 
> - emacs ansi-term
> - emacs-vterm
> - GNU screen
> - terminology
> - tmux
> - Vim
> 
> I plan to propose patches for all of them (ansi-term happened to be
> the first)  We'll see what other terminal developers think. I can
> link the discussions here. I guess I'll try tmux next.
> 
> Notice that most of these are terminals that run inside another
> terminal.  So I guess the reason for this automatic fallback could
> theoretically be that "tmux/screen/'emacs -nw' etc. cannot accurately
> detect whether the underlying terminal supports curly underlines,
> so we make the decision for them and give them straight underlines
> when in doubt".  But that would be somewhat odd (because it "blocks"
> progress) and actually, I haven't found that to be the case for the
> terminals where I already looked at the git-blame (ansi-term and
> vterm -- for both of them, the missing XTerm compatibility seems like
> an oversight).  Note that the curly underline feature is mostly used
> by text editors, which may be why no one had encountered this problem
> yet with Emacs terminals.
> 
>> 
>> > I'd rather ansi-term ignore the curly underline command until it's
>> > implemented.
>> 
>> We could have this behavior as an option, conditioned on some user
>> option, perhaps.
> 
> I would not be interested in an off-by-default option because if a
> user says "this doesn't work in ansi-term" then I can already tell
> them "use a different terminal, such as M-x vterm", so it wouldn't
> really reduce effort.
> 
> I would be surprised if better XTerm compatibility would make things
> worse but let's see what the consensus is.
> 
> If it's not acceptable to ignore curly underline sequences, I can
> probably also implement them in term.el (i.e. actually recognize
> \e[4:3m).

I think this is a much preferred option. Emacs 30 in a terminal already 
supports displaying wavy underlines. Adding support for wavy underlines 
to ansi-term looks like it wouldn't be harder than the proposed patch.

> That would give us curly underlines in graphical emacs, but it
> would barely change the "emacs -nw" behavior in practice, because,
> as described above, asking terminfo for Smulx/Su has a lot of false
> negatives.  Patch 2/2 would change "emacs -nw" to not
> emit anything if the capability is not advertised.
> So my perspective is: if we want to implement \e[4:3m now, we should
> also take the second patch to actually fix my use case for wrong
> terminfo.

(Note for other readers, patch 2/2 is in another bug: 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=77871)

> In future, to get out of this mess, we should consider not asking
> for Smulx/Su and simply emit the styled sequences unconditionally.
> Unless Emacs still needs to run on hardware terminals that misinterpret
> those, that would be the sanest option going forward.  It will
> essentially force all terminal emulators to be compatible with this,
> which is fine I think?  From the terminals I have tested, emitting
> \e[4:3m would only cause regressions on these terminals: abduco, dvtm,
> JetBrains IDE terminals and urxvt.  It should be easy to fix them or
> add workarounds; in a few years it should no longer be an issue.

If I'm understanding this right, you're suggesting to have Emacs always 
emit the wavy underline escape sequence for :underline (:style wave), 
even on terminals it believes does not support wavy underlines. This 
seems like a significant behavior change. While libraries packaged 
within Emacs look to be good about using a face supports spec when 
defining their faces, the same can't be said for external Emacs 
libraries.

I also don't understand how this makes things better. Are there 
terminals that you encounter that Emacs does not accurately report 
(display-supports-face-attributes-p '(:underline (:style wave))) for? 
Local testing on a Mac with iTerm2, kitty, Alacritty, Apple 
Terminal.app, Rio, and WezTerm correctly reported t vs nil.

  -- MJF

>> > FWIW Emacs itself already supports various underline styles, even in
>> > a TTY -- see e.g. 9f589eb9240 (Add support for colored and styled
>> > underlines on tty frames, 2023-04-20). It would be fairly easy to
>> > add support for curly underlines to ansi-term.
>> >
>> > In general, I wonder what's story on Emacs terminals. ansi-term
>> > has some other compatibility issues, for example it cannot parse
>> > OSC (Operating System Command) sequences, which causes friction
>> > with the fish shell. We can fix this fairly easily; alternatively
>> > maybe default to another terminal implementation such as vterm
>> > (https://github.com/akermu/emacs-libvterm) that offers better
>> > xterm-compatibility.
>> 
>> You don't tell in which Emacs version you see these issues.  Please
>> tell, it might be important.
> 
> Ah it seems like "cannot parse OSC" was the wrong conclusion because
> a bash command like
> 
>     printf "\x1b]133;A;special_key=1\x07"
> 
> is already correctly ignored (though there is the problem that the
> prompt is not redrawn).
> 
> The actual issue I'm seeing is that running fish shell version 4.0.1,
> which prints the above command at startup, causes this text to show
> in the terminal:
> 
>     133;A;special_key=1
> 
> This is with both Emacs 30.1 and latest master (2808bef2522).  fish
> assumes basic VT100 compatibility.  Perhaps this is a timing issue.
> I'm in a good position to extract a minimal reproducer, I can follow
> up with that (probably in a separate thread?).
> 
>> 
>> Jared, any comments to the above and/or to the patch?
>> 
>> In any case, we are unable to accept such a large contribution without
>> you assigning the copyright for your changes to the FSF.  If you are
>> willing to do that, I will send you the form to fill and the
>> instructions to go with it, to start your legal paperwork rolling.
> 
> sure.
> 
>> 
>> Thanks.
>> 
>> > ---
>> >  lisp/term.el | 230 ++++++++++++++++++++++++++++-----------------------

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.