GNU bug report logs - #79298
patch: full color in windows terminal

Previous Next

To reply to this bug, email your comments to 79298 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: "bug-gnu-emacs <at> gnu.org" <bug-gnu-emacs <at> gnu.org>
Subject: Re: patch: full color in windows terminal
Date: Sun, 24 Aug 2025 02:27:35 +0000

[Message part 1 (text/plain, inline)]

Apologies, previous patch inherited an unrelated bug fix. Fixed patch attached, fixed commit logs below.

* configure.ac: Compiler flag for setting w32 console color space.
* lisp/term/w32console.el: Define and apply different color spaces.
* src/term.c: Set tty max colors based on compiler flag.
* src/w32console.c: Write face colors to console via VT sequences.

[0001-Windows-Terminal-full-color-support.patch (application/octet-stream, attachment)]

Message #8 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Sun, 24 Aug 2025 10:20:08 +0300

merge 79298 79297
thanks

> Date: Sun, 24 Aug 2025 02:27:35 +0000
> From:  Ewan via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> * configure.ac: Compiler flag for setting w32 console color space.
> * lisp/term/w32console.el: Define and apply different color spaces.
> * src/term.c: Set tty max colors based on compiler flag.
> * src/w32console.c: Write face colors to console via VT sequences.

Thanks.  Such a large contribution will need you to sign a
copyright-assignment agreement with the FSF.  If you agree to do it, I
will send you the form to fill and the instructions to go with it.

> --- a/configure.ac
> +++ b/configure.ac
> @@ -299,6 +299,21 @@ AC_ARG_WITH([all],
>    [with_features=$withval],
>    [with_features=yes])
>  
> +AC_ARG_WITH([w32-vt-color],
> +[AS_HELP_STRING([--with-w32-vt-color=ARG],
> +[use VT sequences for w32 console color (ARG one of: no, 16, 156, 24bit)])],
> +[    case "${withval}" in
> +     no )    val=no ;;
> +     16 )    val=16 ;;
> +     256 )   val=256 ;;
> +     24bit ) val=24bit ;;
> +     * )
> +     AC_MSG_ERROR(['--with-vt-color=$withval' is invalid;
> +     the value should be 'no', '16', '256', or '24bit']) ;;
> +     esac
> +     with_w32_vt_color=$val
> +])

Why do we need this configure-time switch? why not compile this
feature into Emacs unconditionally?

> +(defun w32con-define-256-colors ()
> +  "Defines 256 color space"

The first line of a doc string should be a single complete sentence,
and must end in a period (here and elsewhere in the patch).

> +  ;; Define the color space
> +  (tty-color-clear)
> +  (let ((ncolors (display-color-cells)))
> +    (cond ((= ncolors 16777216) (w32con-define-24bit-colors))
> +          ((= ncolors 265       (w32con-define-256-colors)))
> +          (t                    (w32con-define-base-colors))))
>    (clear-face-cache)
>    ;; Figure out what are the colors of the console window, and set up
>    ;; the background-mode correspondingly.
> diff --git a/src/term.c b/src/term.c
> index a1e3f6312c6..e762024bdd5 100644
> --- a/src/term.c
> +++ b/src/term.c
> @@ -4686,7 +4686,17 @@ use the Bourne shell command 'TERM=...; export TERM' (C-shell:\n\
>       don't think we're losing anything by turning it off.  */
>    tty->line_ins_del_ok = 0;
>  
> -  tty->TN_max_colors = 16;  /* Must be non-zero for tty-display-color-p.  */
> +  /* Support full range of colors in new windows terminal app */
> +  #if defined(USE_W32CONVTCOLOR_16)
> +    tty->TN_max_colors = 16;
> +  #elif defined(USE_W32CONVTCOLOR_256)
> +    tty->TN_max_colors = 256;
> +  #elif defined(USE_W32CONVTCOLOR_24BIT)
> +    tty->TN_max_colors = 16777216;
> +  #else
> +    /* Must be non-zero for tty-display-color-p.  */
> +    tty->TN_max_colors = 16;
> +  #endif

I think it's sub-optimal to have this conditioned on the build-time
conditions.  I think users should be able to change the number of
supported colors when starting Emacs (using the --color command-line
option) and/or from within a running Emacs session by changing the
value of a variable.  Could you please rework the patch to allow that?

> +#if defined(USE_W32CONVTCOLOR_16) || defined(USE_W32CONVTCOLOR_256) || defined(USE_W32CONVTCOLOR_24BIT)
> +	  if (!WriteConsole (cur_screen, conversion_buffer,
> +			     coding->produced, &r, NULL))
> +	    {
> +	      printf ("Failed writing console characters: %lu\n",
> +		      GetLastError ());
> +	      fflush (stdout);
> +	    }

We cannot use such printf's in a production session.  I guess this is
debugging code that should be removed?

> +#if defined(USE_W32CONVTCOLOR_16) || defined(USE_W32CONVTCOLOR_256) || defined(USE_W32CONVTCOLOR_24BIT)
> +  DWORD mode;
> +  GetConsoleMode (cur_screen, &mode);
> +  mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
> +  SetConsoleMode (cur_screen, mode);
> +#endif

AFAIK, the ENABLE_VIRTUAL_TERMINAL_PROCESSING is not supported on
Windows 9X (which we still strive to support).  So either we should
make sure it is ignored on those old systems, or we should disable
these enhanced color modes on those systems.

> +	      snprintf(p, 10, "[%lum", fg + 30);

Our style is to leave 1 space between the function's name and the
following opening parenthesis.

> +	      WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> +	    }
> +	  if (fg >= 8 && fg < 16)
> +	    {
> +	      char p[10];
> +	      snprintf(p, 10, "[%lum", fg - 8 + 90);
> +	      WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> +	    }
> +	  if (fg >= 16 && fg < 256)
> +	    {
> +	      char p[20];
> +	      snprintf(p, 20, "[38;5;%lum", fg);
> +	      WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> +	    }
> +	  if (bg >= 0 && bg < 8)
> +	    {
> +	      char p[10];
> +	      snprintf(p, 10, "[%lum", bg + 40);
> +	      WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> +	    }
> +	  if (bg >= 8 && bg < 16)
> +	    {
> +	      char p[10];
> +	      snprintf(p, 10, "[%lum", bg - 8 + 100);
> +	      WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> +	    }
> +	  if (bg>= 16 && bg < 256)
> +	    {
> +	      char p[20];
> +	      snprintf(p, 20, "[48;5;%lum", bg);
> +	      WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> +	    }
> +	}
> +      else if (tty->TN_max_colors == 16777216)
> +
> +
> +	{
> +	  char p[30];
> +	  snprintf(p,30, "[38;2;%lu;%lu;%lum", fg/65536, (fg/256)&255, fg&255);
> +	  WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> +
> +	  char q[30];
> +	  snprintf(q, 30, "[48;2;%lu;%lu;%lum", bg/65536, (bg/256)&255, bg&255);
> +	  WriteConsole (cur_screen, q, strlen(q), &r, NULL);
> +	}
> +    }

Can we make this code more compact, like by using fewer snprintf calls
(since they are all so much alike), and just a single WriteConsole
call for each of foreground and background colors?

Finally, these changes will need a NEWS entry and suitable
additions/changes to the user manual.

Thanks again for working on this.

Message #13 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>,
 "79298 <at> debbugs.gnu.org" <79298 <at> debbugs.gnu.org>
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Mon, 25 Aug 2025 06:09:37 +0000

> merge 79298 79297

> Such a large contribution will need you to sign a
> copyright-assignment agreement with the FSF.


I assumed as much, and have sent off my details in the form request to assign <at> gnu.org.
If it's easier for you to just send me the form, that works too!

> > --- a/configure.ac
> > +++ b/configure.ac
> > @@ -299,6 +299,21 @@ AC_ARG_WITH([all],
> > [with_features=$withval],
> > [with_features=yes])
> > 
> > +AC_ARG_WITH([w32-vt-color],
> > +[AS_HELP_STRING([--with-w32-vt-color=ARG],
> > +[use VT sequences for w32 console color (ARG one of: no, 16, 156, 24bit)])],
> > ...
> 
> Why do we need this configure-time switch? why not compile this
> feature into Emacs unconditionally?


This work was inherited from someone else, and I had wondered the same...
I had some worries about supporting Windows Console Host (still 16-color),
but on further review, they've implemented a mapping from 24bit RGB.
Happy to remove the compiler flags and default to 24bit!

> I think it's sub-optimal to have this conditioned on the build-time
> conditions. I think users should be able to change the number of
> supported colors when starting Emacs (using the --color command-line
> option) and/or from within a running Emacs session by changing the
> value of a variable. Could you please rework the patch to allow that?


Will do (just need to figure out where the --color arg gets processed).

> > +#if defined(USE_W32CONVTCOLOR_16) || defined(USE_W32CONVTCOLOR_256) || defined(USE_W32CONVTCOLOR_24BIT)
> > + if (!WriteConsole (cur_screen, conversion_buffer,
> > + coding->produced, &r, NULL))
> > + {
> > + printf ("Failed writing console characters: %lu\n",
> > + GetLastError ());
> > + fflush (stdout);
> > + }
> 
> We cannot use such printf's in a production session. I guess this is
> debugging code that should be removed?


I suspect so, and the existing ones got duplicated in the changes I inherited.
I'll remove them in the revised patch.

> > +#if defined(USE_W32CONVTCOLOR_16) || defined(USE_W32CONVTCOLOR_256) || defined(USE_W32CONVTCOLOR_24BIT)
> > + DWORD mode;
> > + GetConsoleMode (cur_screen, &mode);
> > + mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
> > + SetConsoleMode (cur_screen, mode);
> > +#endif
> 
> AFAIK, the ENABLE_VIRTUAL_TERMINAL_PROCESSING is not supported on
> Windows 9X (which we still strive to support).


Will it be sufficient to disable the enhanced color modes on start up,
but still allow W9 users to activate them (at their own peril)?

> > + WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> > + }
> > + if (fg >= 8 && fg < 16)
> > + {
> > + char p[10];
> > + snprintf(p, 10, "[%lum", fg - 8 + 90);
> > + WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> > + }
> > + ...
> 
> Can we make this code more compact, like by using fewer snprintf calls
> (since they are all so much alike), and just a single WriteConsole
> call for each of foreground and background colors?


More compact, definitely, and we should actually be able to get away with:
- one WriteConsole call for turning the face on
- one WriteConsole call for writing the text
- one WriteConsole call for turning the face off

> Finally, these changes will need a NEWS entry and suitable
> additions/changes to the user manual.


I'll add a NEWS entry.
I'm not sure what exactly editing the manual entails, but happy to do so.


I'll make these changes in the next few days and send off a new patch.

Best,

- Ewan

Message #16 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Mon, 25 Aug 2025 13:50:12 +0300

> Date: Mon, 25 Aug 2025 06:09:37 +0000
> From: Ewan <ewan <at> etown.dev>
> 
> 
> > merge 79298 79297
> 
> > Such a large contribution will need you to sign a
> > copyright-assignment agreement with the FSF.
> 
> 
> I assumed as much, and have sent off my details in the form request to assign <at> gnu.org.
> If it's easier for you to just send me the form, that works too!

Thanks, form sent off-list.

> > > --- a/configure.ac
> > > +++ b/configure.ac
> > > @@ -299,6 +299,21 @@ AC_ARG_WITH([all],
> > > [with_features=$withval],
> > > [with_features=yes])
> > > 
> > > +AC_ARG_WITH([w32-vt-color],
> > > +[AS_HELP_STRING([--with-w32-vt-color=ARG],
> > > +[use VT sequences for w32 console color (ARG one of: no, 16, 156, 24bit)])],
> > > ...
> > 
> > Why do we need this configure-time switch? why not compile this
> > feature into Emacs unconditionally?
> 
> 
> This work was inherited from someone else, and I had wondered the same...

Hmm... the fact that some of the code is "inherited from someone else"
worries me.  If the code you took from someone else's work is
substantial (more than, say, 15 lines), we will need that other person
to assign the copyright to the FSF, in addition to your assignment.

Can you tell who is that other person and how much of the code he/she
wrote?

> > I think it's sub-optimal to have this conditioned on the build-time
> > conditions. I think users should be able to change the number of
> > supported colors when starting Emacs (using the --color command-line
> > option) and/or from within a running Emacs session by changing the
> > value of a variable. Could you please rework the patch to allow that?
> 
> 
> Will do (just need to figure out where the --color arg gets processed).

It is processed in lisp/startup.el.

> > AFAIK, the ENABLE_VIRTUAL_TERMINAL_PROCESSING is not supported on
> > Windows 9X (which we still strive to support).
> 
> Will it be sufficient to disable the enhanced color modes on start up,
> but still allow W9 users to activate them (at their own peril)?

Yes.

> > > + WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> > > + }
> > > + if (fg >= 8 && fg < 16)
> > > + {
> > > + char p[10];
> > > + snprintf(p, 10, "[%lum", fg - 8 + 90);
> > > + WriteConsole (cur_screen, p, strlen(p), &r, NULL);
> > > + }
> > > + ...
> > 
> > Can we make this code more compact, like by using fewer snprintf calls
> > (since they are all so much alike), and just a single WriteConsole
> > call for each of foreground and background colors?
> 
> 
> More compact, definitely, and we should actually be able to get away with:
> - one WriteConsole call for turning the face on
> - one WriteConsole call for writing the text
> - one WriteConsole call for turning the face off
> 
> > Finally, these changes will need a NEWS entry and suitable
> > additions/changes to the user manual.
> 
> 
> I'll add a NEWS entry.

Thanks.

> I'm not sure what exactly editing the manual entails, but happy to do so.

At least the description of --color in the user manual should be
adjusted: it currently says that it has no effect on Windows.  I think
the way to turn on/off 24-bit colors in a running session should also
be mentioned in the MS-Windows-specific appendix to the user manual.

> I'll make these changes in the next few days and send off a new patch.

TIA

Message #19 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Mon, 25 Aug 2025 12:06:22 +0000

> Hmm... the fact that some of the code is "inherited from someone else"
> worries me. If the code you took from someone else's work is
> substantial (more than, say, 15 lines), we will need that other person
> to assign the copyright to the FSF, in addition to your assignment.
> Can you tell who is that other person and how much of the code he/she
> wrote?

The initial patch drew heavily (>15 lines) from this public fork of emacs:
https://github.com/tsuu32/emacs-w32con-vt

From breadcrumbs in the working tree and other GNU mailing list discussions,
I gather the owner of that repo has this address: tsuucat <at> icloud.com,
and has contributed patches in the past, but I don't know them.

The revised patch differs substantially, but I suspect it will keep some
small chunks of color-computing code, as the forumulas are what they are.


> > Will do (just need to figure out where the --color arg gets processed).
> It is processed in lisp/startup.el.

Thank you!


> At least the description of --color in the user manual should be
> adjusted: it currently says that it has no effect on Windows. I think
> the way to turn on/off 24-bit colors in a running session should also
> be mentioned in the MS-Windows-specific appendix to the user manual.

I'll take a look!

Message #22 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Mon, 25 Aug 2025 16:31:19 +0300

> Date: Mon, 25 Aug 2025 12:06:22 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: 79298 <at> debbugs.gnu.org
> 
> > Hmm... the fact that some of the code is "inherited from someone else"
> > worries me. If the code you took from someone else's work is
> > substantial (more than, say, 15 lines), we will need that other person
> > to assign the copyright to the FSF, in addition to your assignment.
> > Can you tell who is that other person and how much of the code he/she
> > wrote?
> 
> The initial patch drew heavily (>15 lines) from this public fork of emacs:
> https://github.com/tsuu32/emacs-w32con-vt
> 
> >From breadcrumbs in the working tree and other GNU mailing list discussions,
> I gather the owner of that repo has this address: tsuucat <at> icloud.com,
> and has contributed patches in the past, but I don't know them.
> 
> The revised patch differs substantially, but I suspect it will keep some
> small chunks of color-computing code, as the forumulas are what they are.

Can you identify those formulas in the patch you submitted?

Message #25 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Mon, 25 Aug 2025 20:23:52 +0000

> > The initial patch drew heavily (>15 lines) from this public fork of emacs:
> > https://github.com/tsuu32/emacs-w32con-vt
> > 
> > The revised patch differs substantially, but I suspect it will keep some
> > small chunks of color-computing code, as the forumulas are what they are.
> 
> 
> Can you identify those formulas in the patch you submitted?

In lisp/term/w32console.el, the definition and application of convert-to-16-bit 
in the body of w32con-define-256-colors, and the overall of shape of the
w32con-define-N-colors functions.

In src/w32console.c, within turn_on_face, the conversions of fg and bg which 
yield substituends in the calls to snprintf .

Message #28 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Tue, 26 Aug 2025 14:02:37 +0300

> Date: Mon, 25 Aug 2025 20:23:52 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: 79298 <at> debbugs.gnu.org
> 
> > > The initial patch drew heavily (>15 lines) from this public fork of emacs:
> > > https://github.com/tsuu32/emacs-w32con-vt
> > > 
> > > The revised patch differs substantially, but I suspect it will keep some
> > > small chunks of color-computing code, as the forumulas are what they are.
> > 
> > 
> > Can you identify those formulas in the patch you submitted?
> 
> In lisp/term/w32console.el, the definition and application of convert-to-16-bit 
> in the body of w32con-define-256-colors, and the overall of shape of the
> w32con-define-N-colors functions.
> 
> In src/w32console.c, within turn_on_face, the conversions of fg and bg which 
> yield substituends in the calls to snprintf .

Thanks.  Is there some Microsoft documentation that describes how to
convert fg and bg colors to virtual console escape sequences?  If so,
can you point to it?

Message #31 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>,
 "79298 <at> debbugs.gnu.org" <79298 <at> debbugs.gnu.org>
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 27 Aug 2025 04:29:00 +0000

On Tuesday, August 26th, 2025 at 4:03 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > Date: Mon, 25 Aug 2025 20:23:52 +0000
>
> > From: Ewan ewan <at> etown.dev
> > Cc: 79298 <at> debbugs.gnu.org
> >
> > > > The initial patch drew heavily (>15 lines) from this public fork of emacs:
> > > > https://github.com/tsuu32/emacs-w32con-vt
> > > >
> > > > The revised patch differs substantially, but I suspect it will keep some
> > > > small chunks of color-computing code, as the forumulas are what they are.
> > >
> > > Can you identify those formulas in the patch you submitted?
> >
> > In lisp/term/w32console.el, the definition and application of convert-to-16-bit
> > in the body of w32con-define-256-colors, and the overall of shape of the
> > w32con-define-N-colors functions.

Note, those functions appear to be pulled, with minor changes, from xterm.el.

> > In src/w32console.c, within turn_on_face, the conversions of fg and bg which
> > yield substituends in the calls to snprintf .
>
>
> Thanks. Is there some Microsoft documentation that describes how to
> convert fg and bg colors to virtual console escape sequences? If so,
> can you point to it?

The conversion is derived from the table in this spec:
https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#text-formatting

I can also include the link in the source code, but it may go dead some day.

Message #34 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 27 Aug 2025 15:11:15 +0300

> Date: Wed, 27 Aug 2025 04:29:00 +0000
> From: Ewan <ewan <at> etown.dev>
> 
> On Tuesday, August 26th, 2025 at 4:03 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > > > Can you identify those formulas in the patch you submitted?
> > >
> > > In lisp/term/w32console.el, the definition and application of convert-to-16-bit
> > > in the body of w32con-define-256-colors, and the overall of shape of the
> > > w32con-define-N-colors functions.
> 
> Note, those functions appear to be pulled, with minor changes, from xterm.el.
> 
> > > In src/w32console.c, within turn_on_face, the conversions of fg and bg which
> > > yield substituends in the calls to snprintf .
> >
> >
> > Thanks. Is there some Microsoft documentation that describes how to
> > convert fg and bg colors to virtual console escape sequences? If so,
> > can you point to it?
> 
> The conversion is derived from the table in this spec:
> https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#text-formatting

Thanks.  I think with this information, and with the rewriting to make
the code more concise (as we already discussed), the actual code taken
from the other person is mall and/or trivial enough for us not to
bother about the legal issues.

> I can also include the link in the source code, but it may go dead some day.

Please do include the link.  We will update it as needed.

Message #37 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 03 Sep 2025 09:35:37 +0000

[Message part 1 (text/plain, inline)]

I'm still waiting on a disclaimer, but in the meantime, I have attached 
the revised patch (with a slightly broadened scope).

Basically, it's much smoother to do a wholesale migration from the Windows
Console API to ASCII control sequences, controlled by an auto-set feature flag.

However,

(1) ASCII control sequences require moving the cursor during redisplay,
(2) the Windows system cursor picks up these movements, and
(3) redisplay is slow enough that cursor jumps to the echo area are perceptible.

So, I've also implemented some changes to cursor handling in Windows TTY.

If the existing variable 'w32-use-visible-system-caret is nil (the default),
then we always hide the system cursor, and Emacs handles cursor display.
This is the smoothest UX, currently limited to a non-blinking solid cursor.
Otherwise, the system cursor is used, but we hide it momentarily when
redisplaying the echo area, resulting in a slight cursor flicker in-place
(the alternative being a noticeable "flashing" of the cursor in the echo area).

More details in the included news entries and changes to the manual.

Logs:
* doc/emacs/cmdargs.texi: Added 24-bit option and note re: --color on Windows.
* doc/emacs/msdos.texi: Documented VT sequence migration and feature flag(s).
* etc/NEWS.30: Entries on VT sequences (color) and cursor display.
* lisp/term/tty-colors.el: Added '24bit cell in tty-color-mode-alist.
* lisp/term/w32console.el: Changes/additions enabling VT sequence migration.
* src/dispnew.c: WINDOWSNT-specific segments for w32- and Emacs-drawn cursor(s).
* src/term.c: Extended existing TTY color infrastructure to include WINDOWSNT.
* src/termchar.h: Added extern decl. (tty_setup_colors) for use in w32console.c.
* src/w32console.c: Migration to ASCII control sequences (feature-flagged).
* src/xdisp.c: Modified #if condition to call set_tty_color_mode for WINDOWSNT.

[0001-Windows-TTY-Migration-to-Virtual-Terminal-Sequences.patch (application/octet-stream, attachment)]

Message #40 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 03 Sep 2025 17:44:48 +0300

> Date: Wed, 03 Sep 2025 09:35:37 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: 79298 <at> debbugs.gnu.org
> 
> I'm still waiting on a disclaimer, but in the meantime, I have attached 
> the revised patch (with a slightly broadened scope).

Thanks.

> Basically, it's much smoother to do a wholesale migration from the Windows
> Console API to ASCII control sequences, controlled by an auto-set feature flag.
> 
> However,
> 
> (1) ASCII control sequences require moving the cursor during redisplay,
> (2) the Windows system cursor picks up these movements, and
> (3) redisplay is slow enough that cursor jumps to the echo area are perceptible.
> 
> So, I've also implemented some changes to cursor handling in Windows TTY.
> 
> If the existing variable 'w32-use-visible-system-caret is nil (the default),
> then we always hide the system cursor, and Emacs handles cursor display.
> This is the smoothest UX, currently limited to a non-blinking solid cursor.
> Otherwise, the system cursor is used, but we hide it momentarily when
> redisplaying the echo area, resulting in a slight cursor flicker in-place
> (the alternative being a noticeable "flashing" of the cursor in the echo area).

I wish you brought this up for discussion before actually coding (and
documenting) these extensive changes.  I'd rather we stayed with the
original use of the Windows Console APIs for everything except the
true color support, if possible.  AFAIK, there are no fundamental
incompatibilities between these two features (if I'm mistaken, please
describe the problems), and we need to keep the old Console APIs
anyway for as long as we support the older Windows systems.  Plus,
AFAIK users can choose to use the legacy ConHost even on newer Windows
versions, and I wouldn't want us to have to tell the users to tweak
their Emacs should it turn out ConHost causes problems with the
Virtual Terminal control sequences.  And you have already bumped into
problems with this, in the form of cursor flashing momentarily in the
wrong places and slower redisplay.

In addition, the changes you propose introduce Windows-specific code
into platform-independent source files, like dispnew.c, something we
generally prefer to avoid.

So let's take a step back and discuss the need for this part of the
changes: what problems will we encounter if we leave cursor handling
and the rest of console management to the old Console APIs, and use
the new ASCII sequences only for true color?  If the problems are not
too hard to solve, I'd prefer to leave that code alone, with minimal
necessary changes.  If nothing else, it leaves use with code which was
tested by many users during a long time.

Thanks again for working on this.

Message #43 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 03 Sep 2025 15:54:11 +0000

> > (1) ASCII control sequences require moving the cursor during redisplay,
> > (2) the Windows system cursor picks up these movements, and
> > (3) redisplay is slow enough that cursor jumps to the echo area are perceptible.
> > 
> > So, I've also implemented some changes to cursor handling in Windows TTY.
> 
> 
> I wish you brought this up for discussion before actually coding (and
> documenting) these extensive changes. I'd rather we stayed with the
> original use of the Windows Console APIs for everything except the
> true color support, if possible. AFAIK, there are no fundamental
> incompatibilities between these two features (if I'm mistaken, please
> describe the problems), and we need to keep the old Console APIs
> anyway for as long as we support the older Windows systems. Plus,
> AFAIK users can choose to use the legacy ConHost even on newer Windows
> versions, and I wouldn't want us to have to tell the users to tweak
> their Emacs should it turn out ConHost causes problems with the
> Virtual Terminal control sequences. And you have already bumped into
> problems with this, in the form of cursor flashing momentarily in the
> wrong places and slower redisplay.

Not a problem, the implementation will be of use to me regardless of 
whether it goes in to master, the documentation is just to explain it 
to others. However, there is, I think, a fundamental incompatibility 
between the mechanism supporting true color, and the way we were doing 
things. There's a short bit about this in the manual entry. We could 
stick with use of the API to move the  cursor, clear lines, etc. - 
that's all just a matter of choice (before, we jumped through hoops with 
the API to achieve things there are now specific control sequences for). 

But the previous implementation achieved color display by direct writes
to regions of the screen (using WriteConsoleOutput etc.), not requiring 
cursor movement. The cursor was only moved once, at the very end of an 
update cycle. That was what enabled the stable cursor appearance (and
there are comments about this "trick" dating back to MSDOS). But for 
true color support, that is not an option. We have to use WriteConsole 
(or printf, etc.), which move the cursor during the write, and that is
what (re-)introduces the "unstable cursor" problem. AFAIK, it is not 
that redisplay is now inherently slower, per se... It is that it now 
requires cursor movement, and if the MSDOS commentary still holds up, 
this is "expensive" at a low level - enough so that it is perceptible,
mostly because we make such frequent use of the echo area.

> In addition, the changes you propose introduce Windows-specific code
> into platform-independent source files, like dispnew.c, something we
> generally prefer to avoid.

Agreed, this is not ideal, I just couldn't think of another way to solve
the unstable cursor problem. I've been careful to ensure the functional
changes are only present when compiled for Windows, but if there is an 
alternative, I'm all for it.

> So let's take a step back and discuss the need for this part of the
> changes: what problems will we encounter if we leave cursor handling
> and the rest of console management to the old Console APIs, and use
> the new ASCII sequences only for true color? If the problems are not
> too hard to solve, I'd prefer to leave that code alone, with minimal
> necessary changes. If nothing else, it leaves use with code which was
> tested by many users during a long time.

In short, ASCII sequences for true color destabilize the system cursor 
appearance, because they undermine the "trick" we were turning.
As far as I can tell, there is no way around that.

> AFAIK users can choose to use the legacy ConHost even on newer Windows
> versions, and I wouldn't want us to have to tell the users to tweak
> their Emacs should it turn out ConHost causes problems with the
> Virtual Terminal control sequences.

Three things give me comfort on this front. One, Microsoft has added 
"support" for ASCII sequences (at least for color) to the Console.

"Some virtual terminal emulators support a palette of colors greater 
than the 16 colors provided by the Windows Console. For these extended 
colors, the Windows Console will choose the nearest appropriate color 
from the existing 16 color table for display."

Two, when initializing a TTY session on Windows, we request the 
ENABLE_VIRTUAL_TERMINAL_PROCESSING output mode. If not available,
e.g. on legacy systems, we disable the feature flag and revert to 
the older color display "trick".

Finally, the Emacs-drawn cursor replacing the system cursor when
w32-use-visible-system-caret is nil, is independent of ASCII sequence 
support. It depends only on the 'cursor' face being defined (I believe
it always will be), and just adds an additional pass, for at most two 
rows, through w32con_write_glyphs, which branches between old and new
based on the feature flag, and so should work with the legacy system.


I share the concern about this all being relatively untested. I'll put
it through the motions in my (limited) use of Windows, but there 
will probably be some bugs I don't catch. In any case, it's made my 
use of terminal Emacs on Windows tolerable (if not pleasant!).

Message #46 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 03 Sep 2025 16:09:50 +0000

> Finally, the Emacs-drawn cursor replacing the system cursor when
> w32-use-visible-system-caret is nil, is independent of ASCII sequence
> support. It depends only on the 'cursor' face being defined (I believe
> it always will be), and just adds an additional pass, for at most two
> rows, through w32con_write_glyphs, which branches between old and new
> based on the feature flag, and so should work with the legacy system.

Correction, I managed to avoid the need for additional passes. Rather,
it just forces full re-write (disabling optimizations), for at most 
two rows (the one where the cursor was, and the one where it will be).

Message #49 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 03 Sep 2025 19:38:06 +0300

> Date: Wed, 03 Sep 2025 15:54:11 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: 79298 <at> debbugs.gnu.org
> 
> Not a problem, the implementation will be of use to me regardless of 
> whether it goes in to master, the documentation is just to explain it 
> to others. However, there is, I think, a fundamental incompatibility 
> between the mechanism supporting true color, and the way we were doing 
> things. There's a short bit about this in the manual entry. We could 
> stick with use of the API to move the  cursor, clear lines, etc. - 
> that's all just a matter of choice (before, we jumped through hoops with 
> the API to achieve things there are now specific control sequences for). 
> 
> But the previous implementation achieved color display by direct writes
> to regions of the screen (using WriteConsoleOutput etc.), not requiring 
> cursor movement. The cursor was only moved once, at the very end of an 
> update cycle. That was what enabled the stable cursor appearance (and
> there are comments about this "trick" dating back to MSDOS). But for 
> true color support, that is not an option. We have to use WriteConsole 
> (or printf, etc.), which move the cursor during the write, and that is
> what (re-)introduces the "unstable cursor" problem. AFAIK, it is not 
> that redisplay is now inherently slower, per se... It is that it now 
> requires cursor movement, and if the MSDOS commentary still holds up, 
> this is "expensive" at a low level - enough so that it is perceptible,
> mostly because we make such frequent use of the echo area.

Why can't we turn off the cursor before calling WriteConsoleOutput,
then turn it on again, after the call returns and we move the cursor
to its original place?

> I share the concern about this all being relatively untested. I'll put
> it through the motions in my (limited) use of Windows, but there 
> will probably be some bugs I don't catch. In any case, it's made my 
> use of terminal Emacs on Windows tolerable (if not pleasant!).

For this reason, I'd prefer to make as few changes as possible in the
console-related management code.  Where we must do that, fine, but
let's try leaving the rest intact, as it was tested by time and many
users (and we had in the past some tricky issues, which are now
resolved).

Message #52 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 03 Sep 2025 17:05:56 +0000

> > But the previous implementation achieved color display by direct writes
> > to regions of the screen (using WriteConsoleOutput etc.), not requiring
> > cursor movement. The cursor was only moved once, at the very end of an
> > update cycle. That was what enabled the stable cursor appearance (and
> > there are comments about this "trick" dating back to MSDOS). But for
> > true color support, that is not an option. We have to use WriteConsole
> > (or printf, etc.), which move the cursor during the write, and that is
> > what (re-)introduces the "unstable cursor" problem. AFAIK, it is not
> > that redisplay is now inherently slower, per se... It is that it now
> > requires cursor movement, and if the MSDOS commentary still holds up,
> > this is "expensive" at a low level - enough so that it is perceptible,
> > mostly because we make such frequent use of the echo area.
> 
> 
> Why can't we turn off the cursor before calling WriteConsoleOutput,
> then turn it on again, after the call returns and we move the cursor
> to its original place?

WriteConsoleOutput (used by the old code) doesn't move the cursor, but
can't be used for ASCII control sequences, and is being phased out by 
Microsoft. WriteConsole is required for ASCII sequences, and moves the
cursor. Hiding the system cursor before every move/call to WriteConsole 
and showing it after, makes for a very unstable cursor (flickers wildly).

Previously, we maintained the "cursor" throughout the update as an internal 
coordinate variable, without "realizing" the position in the system cursor,
through a call to SetConsoleCursorPosition, until the after-update hook. 
That is no longer possible -- the system cursor position must be realized 
in real-time. Hiding the system cursor the whole time, and replacing it
with a (nominal) "cursor" drawn in with the rest of the glyphs, achieves 
the same effect that we had before, in light of the new constraint. The
(actual, system) cursor jumps around, invisibly, during the write, while 
the (nominal, visible) cursor stays stable, then makes one discrete move 
(when its new row is written).

Message #55 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Wed, 03 Sep 2025 18:00:49 +0000

> > AFAIK users can choose to use the legacy ConHost even on newer Windows
> > versions, and I wouldn't want us to have to tell the users to tweak
> > their Emacs should it turn out ConHost causes problems with the
> > Virtual Terminal control sequences.
> 
> 
> Three things give me comfort on this front. One, Microsoft has added
> "support" for ASCII sequences (at least for color) to the Console.

I have confirmed that the conhost.exe which ships with Win11 does
support the full range of ASCII control sequences. Emacs with the 
above patch has no apparent differences from 30.2 release when run
with -q in ConHost. I don't have any older systems to test this on.

Message #58 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Thu, 04 Sep 2025 07:42:02 +0300

> Date: Wed, 03 Sep 2025 17:05:56 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: 79298 <at> debbugs.gnu.org
> 
> WriteConsoleOutput (used by the old code) doesn't move the cursor, but
> can't be used for ASCII control sequences, and is being phased out by 
> Microsoft. WriteConsole is required for ASCII sequences, and moves the
> cursor. Hiding the system cursor before every move/call to WriteConsole 
> and showing it after, makes for a very unstable cursor (flickers wildly).

I meant hiding it at the beginning of an update (in
w32con_update_begin) and showing it at the end of the update, which we
already do.  Then we can move it at will without it appearing on the
screen, right?

> Previously, we maintained the "cursor" throughout the update as an internal 
> coordinate variable, without "realizing" the position in the system cursor,
> through a call to SetConsoleCursorPosition, until the after-update hook. 
> That is no longer possible -- the system cursor position must be realized 
> in real-time. Hiding the system cursor the whole time, and replacing it
> with a (nominal) "cursor" drawn in with the rest of the glyphs, achieves 
> the same effect that we had before, in light of the new constraint. The
> (actual, system) cursor jumps around, invisibly, during the write, while 
> the (nominal, visible) cursor stays stable, then makes one discrete move 
> (when its new row is written).

It sounds like you are describing what I suggested above, so we agree
on how to handle this when WriteConsole is in use?  But what do you
mean by "replacing it with a (nominal) "cursor" drawn in with the rest
of the glyphs" in this context?  I thought if we hide the cursor, it
is simply not shown, although the system maintains its position, and
so should we.  Then why "replacing", and what is the "nominal cursor"?

Message #61 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Thu, 04 Sep 2025 07:56:05 +0300

> Date: Wed, 03 Sep 2025 18:00:49 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: 79298 <at> debbugs.gnu.org
> 
> > > AFAIK users can choose to use the legacy ConHost even on newer Windows
> > > versions, and I wouldn't want us to have to tell the users to tweak
> > > their Emacs should it turn out ConHost causes problems with the
> > > Virtual Terminal control sequences.
> > 
> > 
> > Three things give me comfort on this front. One, Microsoft has added
> > "support" for ASCII sequences (at least for color) to the Console.
> 
> I have confirmed that the conhost.exe which ships with Win11 does
> support the full range of ASCII control sequences. Emacs with the 
> above patch has no apparent differences from 30.2 release when run
> with -q in ConHost. I don't have any older systems to test this on.

I know that Windows 11 has ConHost which supports the control
sequences.  It's the potential issues with older systems (AFAIR,
ConHost premiered in XP) that I'm bothered with.  Although I don't
expect any such issues, it would be nice not to risk them at all, by
keeping as much of the current code as possible.  If someone does
report such issues, testing a fix for them will be quite tricky, given
the almost non-existent access to such old systems, and the fact that
Emacs compiled with MinGW64 cannot even run on them.

Message #64 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Thu, 04 Sep 2025 06:01:41 +0000

> I meant hiding it at the beginning of an update (in
> w32con_update_begin) and showing it at the end of the update, which we
> already do. Then we can move it at will without it appearing on the
> screen, right?

Correct, and I tried this initially, but while it fixes the "jumping" 
problem, it causes the cursor to flash in-place irregularly, since it 
is hidden during updates, and updates have a variable duration (meaning 
you can't mitigate it with a tailored system cursor blink rate). The 
strategy of hiding it only when updating the echo area was a response to
this issue. It minimizes the "off" time, so the flashing in-place is 
reduced to a slight flicker.

> > Previously, we maintained the "cursor" throughout the update as an internal
> > coordinate variable, without "realizing" the position in the system cursor,
> > through a call to SetConsoleCursorPosition, until the after-update hook.
> > That is no longer possible -- the system cursor position must be realized
> > in real-time. Hiding the system cursor the whole time, and replacing it
> > with a (nominal) "cursor" drawn in with the rest of the glyphs, achieves
> > the same effect that we had before, in light of the new constraint. The
> > (actual, system) cursor jumps around, invisibly, during the write, while
> > the (nominal, visible) cursor stays stable, then makes one discrete move
> > (when its new row is written).
> 
> 
> It sounds like you are describing what I suggested above, so we agree
> on how to handle this when WriteConsole is in use? 

Not quite. By "the whole time", I mean always, whether updating or not.
We effectively replace the internal coordinate variable with the actual
system cursor by making it invisible at all times, and moving it around.

> But what do you mean by "replacing it with a (nominal) "cursor" drawn 
> in with the rest of the glyphs" in this context? 
> ... why "replacing", and what is the "nominal cursor"?

You "replace" the (previously, visible) system cursor with a glyph that
has its regular face_id temporarily replaced by the CURSOR_FACE_ID. The 
"nominal cursor" is then just a temporary assignment of the cursor face
to a whichever glyph will coincide with the (invisible) system cursor at 
the end of the update cycle. You compute this position just before the 
glyph matrix is written to the screen buffer. Visually, it is like the 
solid GUI cursor, as its appearance is determined by the cursor face.

Here's a recent build, in case you want to try for yourself:
https://github.com/ewantown/wemacs-build/releases/tag/v31.35.20250904.0a313d5

And here's a video demo, in case you don't want to run strangers' code:
https://sendvid.com/o4rr6p2b

> I know that Windows 11 has ConHost which supports the control
> sequences.  It's the potential issues with older systems (AFAIR,
> ConHost premiered in XP) that I'm bothered with.  Although I don't
> expect any such issues, it would be nice not to risk them at all, by
> keeping as much of the current code as possible. If someone does
> report such issues, testing a fix for them will be quite tricky, given
> the almost non-existent access to such old systems, and the fact that
> Emacs compiled with MinGW64 cannot even run on them.

I see, fair concern. FWIW I have not /removed/ any of the old code. 
The old path is still intact. At runtime you can ensure that it is used 
by setting w32-use-virtual-terminal-sequences to nil (done automatically
if VT processing is not available), and w32-use-visible-system-cursor to 
non-nil (manual - existing default is nil, unless using a screen-reader).
Not quite as comforting as a static guarantee of backward compatibility,
but I've tried to thread the new code through the old, branching on the
feature flags, so the old path remains functionally untouched.

Message #67 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Thu, 04 Sep 2025 09:37:28 +0300

> Date: Thu, 04 Sep 2025 06:01:41 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: 79298 <at> debbugs.gnu.org
> 
> > I meant hiding it at the beginning of an update (in
> > w32con_update_begin) and showing it at the end of the update, which we
> > already do. Then we can move it at will without it appearing on the
> > screen, right?
> 
> Correct, and I tried this initially, but while it fixes the "jumping" 
> problem, it causes the cursor to flash in-place irregularly, since it 
> is hidden during updates, and updates have a variable duration (meaning 
> you can't mitigate it with a tailored system cursor blink rate). The 
> strategy of hiding it only when updating the echo area was a response to
> this issue. It minimizes the "off" time, so the flashing in-place is 
> reduced to a slight flicker.
> 
> > > Previously, we maintained the "cursor" throughout the update as an internal
> > > coordinate variable, without "realizing" the position in the system cursor,
> > > through a call to SetConsoleCursorPosition, until the after-update hook.
> > > That is no longer possible -- the system cursor position must be realized
> > > in real-time. Hiding the system cursor the whole time, and replacing it
> > > with a (nominal) "cursor" drawn in with the rest of the glyphs, achieves
> > > the same effect that we had before, in light of the new constraint. The
> > > (actual, system) cursor jumps around, invisibly, during the write, while
> > > the (nominal, visible) cursor stays stable, then makes one discrete move
> > > (when its new row is written).
> > 
> > 
> > It sounds like you are describing what I suggested above, so we agree
> > on how to handle this when WriteConsole is in use? 
> 
> Not quite. By "the whole time", I mean always, whether updating or not.
> We effectively replace the internal coordinate variable with the actual
> system cursor by making it invisible at all times, and moving it around.
> 
> > But what do you mean by "replacing it with a (nominal) "cursor" drawn 
> > in with the rest of the glyphs" in this context? 
> > ... why "replacing", and what is the "nominal cursor"?
> 
> You "replace" the (previously, visible) system cursor with a glyph that
> has its regular face_id temporarily replaced by the CURSOR_FACE_ID. The 
> "nominal cursor" is then just a temporary assignment of the cursor face
> to a whichever glyph will coincide with the (invisible) system cursor at 
> the end of the update cycle. You compute this position just before the 
> glyph matrix is written to the screen buffer. Visually, it is like the 
> solid GUI cursor, as its appearance is determined by the cursor face.
> 
> Here's a recent build, in case you want to try for yourself:
> https://github.com/ewantown/wemacs-build/releases/tag/v31.35.20250904.0a313d5
> 
> And here's a video demo, in case you don't want to run strangers' code:
> https://sendvid.com/o4rr6p2b

I'm not yet sure I understand the issues and the proposed solutions.

Let's back up a notch: Emacs in -nw mode on Unix systems already hides
the cursor before writing glyphs, see tty_write_glyphs_1 and
tty_write_glyphs_with_face.  Why cannot we do the same in
w32con_write_glyphs and w32con_write_glyphs_with_face, when
WriteConsole is in use?  Or what are the problems with doing that?

In general, drawing our own cursor on TTY frames sounds like a
complication I would like to avoid.  We don't do anything like that on
Unix, so I'd prefer Windows to work similarly, using the system
cursor.

> > I know that Windows 11 has ConHost which supports the control
> > sequences.  It's the potential issues with older systems (AFAIR,
> > ConHost premiered in XP) that I'm bothered with.  Although I don't
> > expect any such issues, it would be nice not to risk them at all, by
> > keeping as much of the current code as possible. If someone does
> > report such issues, testing a fix for them will be quite tricky, given
> > the almost non-existent access to such old systems, and the fact that
> > Emacs compiled with MinGW64 cannot even run on them.
> 
> I see, fair concern. FWIW I have not /removed/ any of the old code. 
> The old path is still intact. At runtime you can ensure that it is used 
> by setting w32-use-virtual-terminal-sequences to nil (done automatically
> if VT processing is not available), and w32-use-visible-system-cursor to 
> non-nil (manual - existing default is nil, unless using a screen-reader).
> Not quite as comforting as a static guarantee of backward compatibility,
> but I've tried to thread the new code through the old, branching on the
> feature flags, so the old path remains functionally untouched.

Yes, I understand.  But I would prefer to leave the old code used
always, not just under the compatibility option (which should be
needed only in rare cases anyway).

Message #70 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Thu, 04 Sep 2025 09:40:18 +0000

> I'm not yet sure I understand the issues and the proposed solutions.
> 
> Let's back up a notch: Emacs in -nw mode on Unix systems already hides
> the cursor before writing glyphs, see tty_write_glyphs_1 and
> tty_write_glyphs_with_face. Why cannot we do the same in
> w32con_write_glyphs and w32con_write_glyphs_with_face, when
> WriteConsole is in use? Or what are the problems with doing that?

On Unix systems, this rapid off/on cycle appears to be fast enough
to be imperceptible. Not so on Windows. My understanding is that
cursor changes are just less efficient on Windows.

Perhaps some visuals will help. Just FYI, all of these effects are 
more pronounced with a colored background, because of the effect of
colored spaces on write_row (in dispnew.c), and even more so when the 
screen is full or changing significantly.

Here is a baseline - no hiding of the cursor, so it jumps:
https://sendvid.com/s25e79ou

This one is from a build where the cursor is hidden before and shown
after writing glyphs, as on Unix systems:
https://sendvid.com/41falj8d

And this one is where I ended up, hiding the cursor just for the 
call to write_row where the row to be written is the echo area.
The "flicker" is almost imperceptible, but still noticeable when 
attending to the cursor, and esp. on my (quite large) monitor,
when the 'q' in 'setq' is entered:
https://sendvid.com/p43vhzav

Frankly, the last of these is probably sufficient, and would 
go unnoticed by the vast majority of users. But it required 
changes to dispnew.c anyway, and being a perfectionist, I opted 
for the perfectly stable option (drawing our own cursor):
https://sendvid.com/6rxlxsfh


> Yes, I understand. But I would prefer to leave the old code used
> always, not just under the compatibility option (which should be
> needed only in rare cases anyway).

Totally understandable. It's fairly easy to just pull out the new
bits added to handle cursor movement, clearing lines, etc., with
ASCII sequences, and we just lose a tiny bit of efficiency.

Message #73 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Thu, 04 Sep 2025 13:11:54 +0300

> Date: Thu, 04 Sep 2025 09:40:18 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: 79298 <at> debbugs.gnu.org
> 
> 
> > I'm not yet sure I understand the issues and the proposed solutions.
> > 
> > Let's back up a notch: Emacs in -nw mode on Unix systems already hides
> > the cursor before writing glyphs, see tty_write_glyphs_1 and
> > tty_write_glyphs_with_face. Why cannot we do the same in
> > w32con_write_glyphs and w32con_write_glyphs_with_face, when
> > WriteConsole is in use? Or what are the problems with doing that?
> 
> On Unix systems, this rapid off/on cycle appears to be fast enough
> to be imperceptible. Not so on Windows. My understanding is that
> cursor changes are just less efficient on Windows.

Why "rapid"?  AFAICT, we hide the cursor before the first write, and
unhide it only in update_end_hook, i.e. when we finished updating the
entire frame.

> Perhaps some visuals will help.

Thanks, but they don't.  Movies that show screens don't work well IME
because the basic timing on the original system is basically lost.

Tell you what: why don't you post a patch that includes all we were
talking about, i.e. leaving most of the code not related to colors
unmodified and using the system cursor, and including the call to
hide/show the cursor modeled after what the Unix systems do, and we
then take it from there?  Maybe I'm missing something important here,
but then I should be able to see if after applying the patch and
building Emacs.

Thanks.

Message #76 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Fri, 05 Sep 2025 20:47:56 +0000

[Message part 1 (text/plain, inline)]

> Tell you what: why don't you post a patch that includes all we were
> talking about, i.e. leaving most of the code not related to colors
> unmodified and using the system cursor, and including the call to
> hide/show the cursor modeled after what the Unix systems do, and we
> then take it from there? Maybe I'm missing something important here,
> but then I should be able to see if after applying the patch and
> building Emacs.

Sure -- attached. Just ignore the documentation, comments, and cruft 
for now, I will clean it all up once we get this figured out.

[0001-w32-Virtual-Terminal-Sequences-min.patch (application/octet-stream, attachment)]

Message #79 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Ewan <ewan <at> etown.dev>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Sat, 06 Sep 2025 09:27:10 +0200

Ewan via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs <at> gnu.org> writes:

Hi,

just a comment from the sideline:

> diff --git a/etc/NEWS.30 b/etc/NEWS.30
> index dfe6c5cafe6..814ca843be5 100644
> --- a/etc/NEWS.30
> +++ b/etc/NEWS.30
> @@ -15,6 +15,35 @@ in older Emacs versions.
>  You can narrow news to a specific version by calling 'view-emacs-news'
>  with a prefix argument or by typing 'C-u C-h C-n'.
>  
> +
> +* Changes in Emacs 30.3

I might be wrong, but I believe the change is for master (Emacs
31.1). So pls modify etc/NEWS instead of etc/NEWS.30.

Best cregards, Michael.

Message #82 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Ewan <ewan <at> etown.dev>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Sat, 06 Sep 2025 09:19:40 +0000

[Message part 1 (text/plain, inline)]

> I might be wrong, but I believe the change is for master (Emacs
> 31.1). So pls modify etc/NEWS instead of etc/NEWS.30.

Ahh, thank you - updated patches attached.

...-minimal: system cursor, VT sequences just for color.
...-full:    system or Emacs-drawn cursor, complete migration to VT sequences.

[0001-w32-VT-Sequences-minimal.patch (application/octet-stream, attachment)]

[0001-w32-VT-Sequences-full.patch (application/octet-stream, attachment)]

Message #85 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Ewan <ewan <at> etown.dev>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Sat, 06 Sep 2025 11:47:26 +0200

Ewan <ewan <at> etown.dev> writes:

Hi,

Another nitpick:

> -draws its own cursor to indicate the position of point.  A
> +draws its own cursor to indicate the position of point. A

Please use always two spaces after a period, which finishes a
sentence. Everywhere in doc and comments.

(You don't need to show new versions of the patches just because of this.)

Best regards, Michael.

Message #88 received at 79298 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ewan <ewan <at> etown.dev>
Cc: 79298 <at> debbugs.gnu.org
Subject: Re: bug#79298: patch: full color in windows terminal
Date: Sun, 14 Sep 2025 15:03:52 +0300

> Date: Sat, 06 Sep 2025 09:19:40 +0000
> From: Ewan <ewan <at> etown.dev>
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 79298 <at> debbugs.gnu.org
> 
> > I might be wrong, but I believe the change is for master (Emacs
> > 31.1). So pls modify etc/NEWS instead of etc/NEWS.30.
> 
> Ahh, thank you - updated patches attached.
> 
> ...-minimal: system cursor, VT sequences just for color.
> ...-full:    system or Emacs-drawn cursor, complete migration to VT sequences.

Thanks.  For now, let's focus on the "minimal" patch only.  I'm also
ignoring the documentation for the time being.  My plan is to arrive
at the agreed-upon initial patch for the code, then try it locally and
fix any problems, then finalize it (including the docs).

So the comments below are intended to (a) slightly cleanup the design
and implementation; and (b) avoid unnecessary changes.

> +;; W32 uses different color indexes than standard
>  (defvar w32-tty-standard-colors
>    '(("black"          0     0     0     0)
>      ("blue"           1     0     0 52480) ; MediumBlue
> @@ -43,56 +42,133 @@
>      ("lightmagenta"  13 65535     0 65535) ; Magenta
>      ("yellow"        14 65535 65535     0) ; Yellow
>      ("white"         15 65535 65535 65535))
> -"A list of VGA console colors, their indices and 16-bit RGB values.")
> +  "A list of VGA console colors, their indices and 16-bit RGB values.")
> +
> +;; When using VT sequences for color, use xterm-like indices
> +(defvar w32-tty-virtual-terminal-base-colors
> +  '(("black"          0     0     0     0)
> +    ("red"            1 45568  8704  8704) ; FireBrick

What is the rationale for using different RGB components in the VT
mode?  The RGB components of the 16 colors were determined by
comparing X and PC colors on the same screen.  I presume that the VT
mode uses the same colors, so I don't understand why we need different
definitions.  Can you explain?

>  (defun terminal-init-w32console ()
>    "Terminal initialization function for w32 console."
>    ;; Share function key initialization with w32 gui frames
>    (x-setup-function-keys (selected-frame))
>    ;; Set terminal and keyboard encodings to the current OEM codepage.
>    (let ((oem-code-page-coding
> -	 (intern (format "cp%d" (w32-get-console-codepage))))
> -	(oem-code-page-output-coding
> -	 (intern (format "cp%d" (w32-get-console-output-codepage))))
> -	oem-cs-p oem-o-cs-p)
> -	(setq oem-cs-p (coding-system-p oem-code-page-coding))
> -	(setq oem-o-cs-p (coding-system-p oem-code-page-output-coding))
> -	(when oem-cs-p
> -	  (set-keyboard-coding-system oem-code-page-coding)
> -	  (set-terminal-coding-system
> -	   (if oem-o-cs-p oem-code-page-output-coding oem-code-page-coding))
> -          ;; Since we changed the terminal encoding, we need to repeat
> -          ;; the test for Unicode quotes being displayable.
> -          (startup--setup-quote-display)))
> -  (let* ((colors w32-tty-standard-colors)
> -         (color (car colors)))
> -    (tty-color-clear)
> -    (while colors
> -      (tty-color-define (car color) (cadr color) (cddr color))
> -      (setq colors (cdr colors)
> -            color (car colors))))
> -  (clear-face-cache)
> -  ;; Figure out what are the colors of the console window, and set up
> -  ;; the background-mode correspondingly.
> -  (let* ((screen-color (get-screen-color))
> -	 (bg (cadr screen-color))
> -	 (descr (tty-color-by-index bg))
> -	 r g b bg-mode)
> -    (setq r (nth 2 descr)
> -	  g (nth 3 descr)
> -	  b (nth 4 descr))
> -    (if (< (+ r g b) (* .6 (+ 65535 65535 65535)))
> -	(setq bg-mode 'dark)
> -      (setq bg-mode 'light))
> -    (set-terminal-parameter nil 'background-mode bg-mode))
> -  (tty-set-up-initial-frame-faces)
> +         (intern (format "cp%d" (w32-get-console-codepage))))
> +        (oem-code-page-output-coding
> +         (intern (format "cp%d" (w32-get-console-output-codepage))))

In the interests of making the patch smaller and easier to review,
could you either avoid changing whitespace, or produce the patch using
the --ignore-space-changes command-line switch?  AFAIU, most of the
above are whitespace changes (TABs to SPCes, right?), but the code is
basically left intact, by an large.

> --- a/src/term.c
> +++ b/src/term.c
> @@ -73,7 +73,6 @@ static void clear_tty_hooks (struct terminal *terminal);
>  static void set_tty_hooks (struct terminal *terminal);
>  static void dissociate_if_controlling_tty (int fd);
>  static void delete_tty (struct terminal *);
> -
>  #endif /* !HAVE_ANDROID */

There are quite a few changes that add or remove empty lines, please
avoid that, to make the patch smaller and more to-the-point.

> @@ -346,10 +345,10 @@ tty_hide_cursor (struct tty_display_info *tty)
>  {
>    if (tty->cursor_hidden == 0)
>      {
> -      tty->cursor_hidden = 1;
>  #ifdef WINDOWSNT
>        w32con_hide_cursor ();
>  #else
> +      tty->cursor_hidden = 1;
>        OUTPUT_IF (tty, tty->TS_cursor_invisible);

Is this change of order important?  If so, why?  If it isn't
important, please leave the original order intact, again to make the
patch smaller and easier to grasp.

Same in other places where AFAICT you've modified the code for
stylistic reasons, without actually changing it.

> @@ -2264,18 +2263,42 @@ tty_setup_colors (struct tty_display_info *tty, int mode)
>        default:
>  	tty_default_color_capabilities (tty, 0);
>  	break;
> -      case 8:	/* 8 standard ANSI colors */
> -	tty->TS_orig_pair = "\033[0m";
> +      case 8: /* 8 standard ANSI colors */
> +	tty->TS_orig_pair = "\x1b[0m";
> +	tty->TS_set_foreground = "\x1b[3%dm";
> +	tty->TS_set_background = "\x1b[4%dm";
>  #ifdef TERMINFO
> -	tty->TS_set_foreground = "\033[3%p1%dm";
> -	tty->TS_set_background = "\033[4%p1%dm";
> -#else
> -	tty->TS_set_foreground = "\033[3%dm";
> -	tty->TS_set_background = "\033[4%dm";
> +	tty->TS_set_foreground = "\x1b[3%p1%dm";
> +	tty->TS_set_background = "\x1b[4%p1%dm";
> +#endif
> +#ifdef WINDOWSNT
> +	tty->TS_orig_pair = "\x1b[39m\x1b[49m";
> +	tty->TS_set_foreground = "\x1b[%lum";
> +	tty->TS_set_background = "\x1b[%lum";

I see you've consistently replaced \033 with \x1b.  Any reason for
that?  If it's possible to use \033 as in original code, it will again
make the patch smaller and easier to read.

Also, WINDOWSNT and TERMINFO are mutually-exclusive, so please use
#elif to show that.

>  	  {
> -	    tty->TS_set_foreground = "\033[%?%p1%{8}%<%t3%p1%d%e38;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%d%;m";
> -	    tty->TS_set_background = "\033[%?%p1%{8}%<%t4%p1%d%e48;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%d%;m";
> +	    tty->TS_set_foreground = "\x1b[%?%p1%{8}%<%t3%p1%d%e38;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%d%;m";
> +	    tty->TS_set_background = "\x1b[%?%p1%{8}%<%t4%p1%d%e48;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%d%;m";
>  	    tty->TN_max_colors = 16777216;
>  	  }

This AFAIU also merely replaces \033 with \x1b.

> +#define SEQMAX 256 /* Arbitrary upper limit on VT sequence size */

GNU coding conventions frown on arbitrary limits.  At the very least,
let's make sure the SSPRINTF macro bails out if it needs to produce
more than SEQMAX bytes, and let's have eassert there in case it bails
out early because the caller wanted more than that -- so that any such
problems could be caught early.

> +/* For debugging */
> +static void
> +vt_seq_error (char *seq)

Please explain in a comment how this supposed to be used for
debugging.

> +	      turn_on_face (f, face_id);
> +	      WriteConsole (cur_screen, conversion_buffer,
> +			    coding->produced, &r, NULL);

Why do you use WriteConsole here, but WriteConsoleA elsewhere?  Is
there a difference?  If not, let's be consistent.

>  static void
>  w32con_update_begin (struct frame * f)
>  {
> +  current_tty = FRAME_TTY (f);
> +
> +  if (!w32_use_virtual_terminal_sequences
> +      && current_tty->TN_max_colors > 16)
> +    {
> +      tty_setup_colors (current_tty, 16);
> +      safe_calln (Qw32con_set_up_initial_frame_faces);
> +    }

Why is this needed?  This will call to Lisp each redisplay cycle,
which is expensive and will slow down redisplay.  Why do we need to
setup colors and faces each redisplay cycle?  We never needed this
before.

> +/* returns the pixel value for the given index into VT base color map */
> +static unsigned long pixel_cache[16];
> +static unsigned long
> +get_pixel (unsigned long index)
> +{
> +  unsigned int i = (unsigned int) index;
> +  if (i > 15) return 0;
> +  if (i == 0 || pixel_cache[i] > 0)
> +    return pixel_cache[i];
> +
> +  Lisp_Object pix = safe_calln (Qw32con_get_pixel, make_ufixnum (i));
> +  pixel_cache[i] = (unsigned long) XUFIXNUM (pix);
> +  return pixel_cache[i];
> +}

This should be implemented in C to avoid a costly call into Lisp (each
time we need to use a different color!).  Reusing existing code is
important, but speed of redisplay trumps that by a large margin.

> +  DEFVAR_BOOL ("w32-use-virtual-terminal-sequences",
> +		w32_use_virtual_terminal_sequences,
> +		doc: /* If non-nil w32 console uses terminal sequences for some output processing.
> +This variable is set automatically based on the capabilities of the terminal.
> +It determines the number and indices of colors used for faces in the terminal.
> +If the terminal cannot handle VT sequences, the update hook triggers recomputation of faces.
> +See `w32con-set-up-initial-frame-faces', which should be called after setting this variable 
> +manually in a running session. */);
> +  w32_use_virtual_terminal_sequences = 0;

Do we want to support setting this from Lisp or by the user (in
addition to setting it automatically at startup)?  If yes, does
setting it require some initialization?

Thanks again for working on this.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.