Subject: bug#15444: One character can be lost if colors are enabled

From: Egmont Koblinger <[email-masked]>
Date: Wed, 27 Mar 2024 09:15:00 +0100

Hello,

VTE co-developer here. (VTE, a.k.a. libvte is the terminal emulation
widget behind GNOME Terminal and several other terminal emulator
apps.)

Note that this very bug is also filed at https://savannah.gnu.org/bugs/?36831 .

---

The "background color erase" ("bce") feature of most terminals suffers
from a fundamental design flaw. You cannot "just print" some piece of
text containing non-default background color, and expect it to appear
on the screen correctly.

By "just printing" what I mean is that you don't go into the business
of querying the terminal size, tracking the cursor position, knowing
which characters will be double wide, computing where the line would
wrap to the next one, etc. I'm only considering simple apps that print
messages using printf() or similar, like grep does.

The problem is that if you're at the bottom of the screen and
autowrapping occurs, the entire new line is added with the currently
active background color, which might not be the terminal's default
background color. If that new line isn't filled with text entirely
then the rest remains with that color.

The workaround applied by "grep" by default is to switch back to the
default background color and emit the sequence that clears to the end
of the line. This, while fixes the background color issue (apart from
a possible temporary flicker if the terminal doesn't process grep's
output in a single step and updates its display in the middle),
introduces a different bug: possibly chopping off one letter of
output.

For the sake of explaining it, assume 80 columns numbered from 1 to
80. The way terminals behave is: After printing 78 chars the cursor is
in column 79. After printing one more char the cursor is in column 80.
After printing the 80th char in the 80th column, the cursor remains in
column 80 (over this newly printed character) (*), but a special flag
is set, in Xterm's source I believe it's called "do_wrap", it denotes
that the next character will wrap to the next line.

((*) VTE hides the cursor in this case, but it's irrelevant for this bug.)

Some operations, like handling a backspace, or clearing to the end of
the line unfortunately don't take this "do_wrap" flag into account.
That is, after printing 80 characters and being in the "do_wrap"
state, the "clear to end of line" escape sequence only checks that the
cursor is in column 80, therefore clears everything in column 80 and
its right (i.e. column 80 only). And it also clears the "do_wrap"
flag.

Perhaps about 8-10 years ago I talked about this conceptual problem
with the developer of xterm and ncurses, and he wasn't interested in
doing anything about the situation. I, however, still was.

---

In VTE I made two modifications, i.e. at two places we intentionally
deviate from the standard behavior to fix this nonsense and make grep
work as expected. One of the workarounds is for when grep is asked to
emit that clear-to-eol, one is for when it isn't.

In https://bugzilla.gnome.org/show_bug.cgi?id=740789 (2014) I changed
the behavior of the clear-to-eol escape sequence. If in that "do_wrap"
state then it doesn't clear the last character, nor this "do_wrap"
state, it becomes no-op.

In https://bugzilla.gnome.org/show_bug.cgi?id=754596 (2015-2017) I
modified the bce feature to fill the newly appearing line with the
current background only if scrolling occurs due to an explicit newline
or escape sequence, but not when it occurs to text autowrapping, in
which case we use the terminal's default background color.

Neither of these changes cause any problem in real life that we're
aware of. The first iteration of the bce change in 2015 had some
problem that I fixed in 2017. We haven't received any bugreport since
then about either of these intentional deviations from the standard.

The result is that grep's output renders correctly in VTE, no matter
if you use its "ne" feature or not.

However, the goal should be to fix it in all terminals out there.

---

So, what should "grep" do?

Its maintainers, or anyone interested in pushing this through (it's
not going to be me, sorry) could talk to Xterm's and other terminals'
authors to change the default, or find a solution. They could point
out that these two workarounds in VTE have turned out to be fine, and
suggest to change the default behavior to these. (However, I wouldn't
expect everyone to agree. I'd expect that many of these folks would
insist on strictly adhering to the standards, rather than
acknowledging that the standard is conceptually broken and being okay
with deviating from it.)

As for clear-to-eol, I think urxvt kept the official behavior, but
added the same sequence with numeric parameter 3, i.e. \e[3K, to
perform the slightly modified (fixed) behavior. Could be a reasonable
route to go down.

As for the immediate future, i.e. assuming that grep maintainers don't
want to or don't succeed in driving such a change across leading /
reference terminals (probably most notably Xterm) and then wait for
years for this to become widespread, the question is: Would you rather
have a wrong background color, or a letter missing? I'm puzzled how
upstream grep, for many-many years, has chosen to possibly chop off a
letter as its default behavior, I firmly believe that it's by far the
worse of the two.

Or start thinking about brand new workarounds. Like Vincent's idea
with a space + backspace. Or space + backspace + clear-to-eol. Or
space + clear-to-eol. These _could_ be robust against issues at the
right margin. Obviously these brainstorming ideas I'm randomly
throwing in now have to be tested carefully across many popular
terminals. Maybe with both possible values of Xterm's reverse
wraparound, maybe with only the default because the other value causes
ncurses corruptions in Xterm
(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067679#10) so you
can't really count on that anyway.

Also, I'm not sure if currently grep emits clear-to-eol wherever the
background color changes within a line, or at the end of the line
after reverting to the terminal's default. Workarounds involving space
/ backspace chars probably only work with the latter approach.

Also, a minor issue, but could be important / annoying to some users:
printing space / backspace can modify how trailing spaces get
copy-pasted in some terminals. It would be nice to copy-paste grep's
output exactly as it appeared in the file. But if that's the price to
pay to fix the other two problems, probably it's still the least bad.

e.