GNU bug report logs - #14403
24.3.50; [regression] Typing non-ascii characters on a non-GUI MS-Windows session

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 14403 in the body.
You can then email your comments to 14403 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: dmoncayo <at> gmail.com
To: bug-gnu-emacs <at> gnu.org
Subject: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI MS-Windows
	session
Date: Tue, 14 May 2013 21:11:22 +0200

I see this problem in [2] but not in [1].

Recipe from "emacs -Q -nw":
  Type these characters: áéíóúñç

Note: With my spanish keyboard, I input the accented vowels by typing
the dead key [´] and then the vowel key (e.g [a]).  As for the "ñ" and
the "ç" characters, they have their own keys.

Well, in [1], I see the expected characters in the buffer (áéíóúñç), but
in [2] I see these characters instead: ßÚݾ·±þ


--- Footnotes ---

[1] Build from the emacs-24 branch:

  In GNU Emacs 24.3.1 (i386-mingw-nt6.1.7601)
   of 2013-04-08 on LEG570
  Bzr revision: 111342 rgm <at> gnu.org-20130406230553-iw6czqc1chq2peld
  Windowing system distributor `Microsoft Corp.', version 6.1.7601
  Configured using:
   `configure --with-gcc (4.7) --no-opt --enable-checking --cflags
   -IC:/emacs/libs/libXpm-3.5.10/include -IC:/emacs/libs/libXpm-3.5.10/src
   -IC:/emacs/libs/libpng-dev_1.4.3-1_win32/include
   -IC:/emacs/libs/zlib-dev_1.2.5-2_win32/include
   -IC:/emacs/libs/giflib-4.1.4-1-lib/include
   -IC:/emacs/libs/jpeg-6b-4-lib/include
   -IC:/emacs/libs/tiff-3.8.2-1-lib/include
   -IC:/emacs/libs/libxml2-2.7.8-w32-bin/include/libxml2
   -IC:/emacs/libs/gnutls-3.1.10-w32/include
   -IC:/emacs/libs/libiconv-1.14-2-mingw32-dev/include'

  Important settings:
    value of $LANG: ENU
    locale-coding-system: cp1252
    default enable-multibyte-characters: t


[2] Build from the trunk:

  In GNU Emacs 24.3.50.1 (i386-mingw-nt6.1.7601)
   of 2013-04-21 on LEG570
  Bzr revision: 112347 xfq.free <at> gmail.com-20130421120104-6nn62drdem8o2cud
  Windowing system distributor `Microsoft Corp.', version 6.1.7601
  Configured using:
   `configure --with-gcc (4.7) --no-opt --enable-checking --cflags
   -IC:/emacs/libs/libXpm-3.5.10/include -IC:/emacs/libs/libXpm-3.5.10/src
   -IC:/emacs/libs/libpng-dev_1.4.3-1_win32/include
   -IC:/emacs/libs/zlib-dev_1.2.5-2_win32/include
   -IC:/emacs/libs/giflib-4.1.4-1-lib/include
   -IC:/emacs/libs/jpeg-6b-4-lib/include
   -IC:/emacs/libs/tiff-3.8.2-1-lib/include
   -IC:/emacs/libs/libxml2-2.7.8-w32-bin/include/libxml2
   -IC:/emacs/libs/gnutls-3.1.10-w32/include
   -IC:/emacs/libs/libiconv-1.14-2-mingw32-dev/include'
  
  Important settings:
    value of $LANG: ENU
    locale-coding-system: cp1252
    default enable-multibyte-characters: t

Message #8 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: dmoncayo <at> gmail.com
Cc: 14403 <at> debbugs.gnu.org
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI
	MS-Windows	session
Date: Tue, 14 May 2013 22:48:45 +0300

> From: dmoncayo <at> gmail.com
> Date: Tue, 14 May 2013 21:11:22 +0200
> 
> 
> I see this problem in [2] but not in [1].
> 
> Recipe from "emacs -Q -nw":
>   Type these characters: áéíóúñç
> 
> Note: With my spanish keyboard, I input the accented vowels by typing
> the dead key [´] and then the vowel key (e.g [a]).  As for the "ñ" and
> the "ç" characters, they have their own keys.
> 
> Well, in [1], I see the expected characters in the buffer (áéíóúñç), but
> in [2] I see these characters instead: ßÚݾ·±þ

This is an exact duplicate of 14368, discussed just yesterday.

Message #13 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 14403 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI
	MS-Windows	session
Date: Tue, 14 May 2013 18:28:35 -0400

>> I see this problem in [2] but not in [1].
>> Recipe from "emacs -Q -nw":
>> Type these characters: áéíóúñç
>> Note: With my spanish keyboard, I input the accented vowels by typing
>> the dead key [´] and then the vowel key (e.g [a]).  As for the "ñ" and
>> the "ç" characters, they have their own keys.
>> Well, in [1], I see the expected characters in the buffer (áéíóúñç), but
>> in [2] I see these characters instead: ßÚݾ·±þ

> This is an exact duplicate of 14368, discussed just yesterday.

Hmm, this looks different, since it doesn't seem to use quail, but
instead relies on the "native input method".  Is it the case that the
non-GUI code in Windows receives decoded chars in `read_char', contrary
to the posix code which receives encoded chars there?


        Stefan

Message #16 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 14403 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI
	MS-Windows	session
Date: Wed, 15 May 2013 11:13:03 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: dmoncayo <at> gmail.com,  14403 <at> debbugs.gnu.org
> Date: Tue, 14 May 2013 18:28:35 -0400
> 
> > This is an exact duplicate of 14368, discussed just yesterday.
> 
> Hmm, this looks different, since it doesn't seem to use quail

Different, yes.  But caused by the same change in keyboard.c.

> Is it the case that the non-GUI code in Windows receives decoded
> chars in `read_char', contrary to the posix code which receives
> encoded chars there?

Both GUI and non-GUI keyboard input on Windows produce Unicode
codepoints of the characters the user types.  The produced input
events have the 'kind' of either ASCII_KEYSTROKE_EVENT or
MULTIBYTE_CHAR_KEYSTROKE_EVENT.  They are returned via this call
sequence:

  read_char -> kbd_buffer_get_event -> make_lispy_event

I guess the solution is to tell read_decoded_char more about the event
that produced the character?

Or maybe we should not set the keyboard encoding to the console
codepage on Windows (although I have no idea what kind of breakage
this could cause)?  What setting of keyboard-coding-system tells the
condition below that no decoding is needed?

Of course, we could always augment this condition:

      if (!((FRAME_TERMCAP_P (frame) || FRAME_MSDOS_P (frame))
	    && (TERMINAL_KEYBOARD_CODING (terminal)->common_flags
		& CODING_REQUIRE_DECODING_MASK)))
	return nextevt;		/* No decoding needed.  */

to do something special for MS-Windows, but that sounds kludgey.

Message #19 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 14403 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50; [regression] Typing non-ascii characters on a
	non-GUI MS-Windows	session
Date: Wed, 22 May 2013 12:06:47 -0400

> Of course, we could always augment this condition:

>       if (!((FRAME_TERMCAP_P (frame) || FRAME_MSDOS_P (frame))
> 	    && (TERMINAL_KEYBOARD_CODING (terminal)->common_flags
> 		& CODING_REQUIRE_DECODING_MASK)))
> 	return nextevt;		/* No decoding needed.  */

Before my change, the test was just

          (TERMINAL_KEYBOARD_CODING (terminal)->common_flags
 	   & CODING_REQUIRE_DECODING_MASK)

but was done inside tty_read_avail_input.  AFAICT, tty_read_avail_input
is only used under POSIX ttys and MSDOS, hence this bug: FRAME_TERMCAP_P
is true for w32 non-GUI frames even though they don't use
tty_read_avail_input (they use w32_console_read_socket instead).

So a crude fix would be to check
"terminal->read_socket_hook == &tty_read_avail_input".

What would be a good test to cleanly distinguish posix ttys from w32
"ttys"?


        Stefan

Message #22 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Juanma Barranquero <lekktu <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 14403 <at> debbugs.gnu.org, Dani Moncayo Melgar <dmoncayo <at> gmail.com>
Subject: Re: bug#14403: 24.3.50; [regression] Typing non-ascii characters on a
	non-GUI MS-Windows session
Date: Wed, 22 May 2013 22:22:47 +0200

On Tue, May 14, 2013 at 9:48 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:

> This is an exact duplicate of 14368, discussed just yesterday.

Isn't this in fact a regression to bug#12055?  I can reproduce the bug
with console codepage 850, it works OK with 1252.

    J

Message #25 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juanma Barranquero <lekktu <at> gmail.com>
Cc: 14403 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI MS-Windows
	session
Date: Wed, 22 May 2013 23:34:38 +0300

> From: Juanma Barranquero <lekktu <at> gmail.com>
> Date: Wed, 22 May 2013 22:22:47 +0200
> Cc: Dani Moncayo Melgar <dmoncayo <at> gmail.com>, 14403 <at> debbugs.gnu.org
> 
> On Tue, May 14, 2013 at 9:48 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> > This is an exact duplicate of 14368, discussed just yesterday.
> 
> Isn't this in fact a regression to bug#12055?

No.

> I can reproduce the bug with console codepage 850, it works OK with
> 1252.

Revert the changes in keyboard.c I pointed to and try again.  Does the
bug still happen?

Message #28 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 14403 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI
	MS-Windows	session
Date: Wed, 22 May 2013 23:39:02 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: dmoncayo <at> gmail.com, 14403 <at> debbugs.gnu.org
> Date: Wed, 22 May 2013 12:06:47 -0400
> 
> So a crude fix would be to check
> "terminal->read_socket_hook == &tty_read_avail_input".

Yes, crude.

> What would be a good test to cleanly distinguish posix ttys from w32
> "ttys"?

How are they different?

And what about the Leim use case, which trips there as well?

Message #31 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 14403 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI
	MS-Windows	session
Date: Wed, 22 May 2013 17:22:15 -0400

>> So a crude fix would be to check
>> "terminal->read_socket_hook == &tty_read_avail_input".
> Yes, crude.
>> What would be a good test to cleanly distinguish posix ttys from w32
>> "ttys"?
> How are they different?

posix ttys only emit bytes (aka encoded characters) in the event queue,
whereas according to this bug, w32 ttys emit characters (aka decoded
characters).

> And what about the Leim use case, which trips there as well?

While the Leim use case comes from the same commit, its source is
slightly different, so the two fixes are unrelated.  I'm working on
a patch for that problem, but it doesn't fix this problem.


        Stefan

Message #34 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: 14403 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI
	MS-Windows	session
Date: Thu, 23 May 2013 05:46:15 +0300

> From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
> Cc: dmoncayo <at> gmail.com, 14403 <at> debbugs.gnu.org
> Date: Wed, 22 May 2013 17:22:15 -0400
> 
> >> So a crude fix would be to check
> >> "terminal->read_socket_hook == &tty_read_avail_input".
> > Yes, crude.
> >> What would be a good test to cleanly distinguish posix ttys from w32
> >> "ttys"?
> > How are they different?
> 
> posix ttys only emit bytes (aka encoded characters) in the event queue,
> whereas according to this bug, w32 ttys emit characters (aka decoded
> characters).

Agree, but then shouldn't we distinguish between them based on this
difference, rather than look for some external attribute?

Message #39 received at 14403-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 14403-done <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI
	MS-Windows	session
Date: Thu, 23 May 2013 10:02:14 -0400

I installed the patch below which I hope fixes this problem.


        Stefan


=== modified file 'src/ChangeLog'
--- src/ChangeLog	2013-05-22 21:35:00 +0000
+++ src/ChangeLog	2013-05-23 13:22:33 +0000
@@ -1,3 +1,7 @@
+2013-05-23  Stefan Monnier  <monnier <at> iro.umontreal.ca>
+
+	* keyboard.c (read_decoded_char): Don't decode under w32 (bug#14403).
+
 2013-05-22  Barry OReilly  <gundaetiapo <at> gmail.com>  (tiny change)
 
 	* casetab.c (init_casetab_once): Fix last change (bug#14424).

=== modified file 'src/keyboard.c'
--- src/keyboard.c	2013-04-14 20:33:57 +0000
+++ src/keyboard.c	2013-05-23 13:21:53 +0000
@@ -6827,6 +6827,8 @@
   /* XXX I think the following code should be moved to separate hook
      functions in system-dependent files.  */
 #ifdef WINDOWSNT
+  /* FIXME: AFAIK, tty_read_avail_input is not used under w32 since the non-GUI
+     code sets read_socket_hook to w32_console_read_socket instead!  */
   return 0;
 #else /* not WINDOWSNT */
   if (! tty->term_initted)      /* In case we get called during bootstrap.  */
@@ -8700,6 +8702,10 @@
     {
       Lisp_Object nextevt
 	= read_char (commandflag, map, prev_event, used_mouse_menu, NULL);
+#ifdef WINDOWSNT
+      /* w32_console already returns decoded events.  */
+      return nextevt;
+#else
       struct frame *frame = XFRAME (selected_frame);
       struct terminal *terminal = frame->terminal;
       if (!((FRAME_TERMCAP_P (frame) || FRAME_MSDOS_P (frame))
@@ -8750,6 +8756,7 @@
 	      = Fcons (events[--n], Vunread_command_events);
 	  return events[0];
 	}
+#endif
     }
 }
 

Message #53 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 14403 <at> debbugs.gnu.org, dmoncayo <at> gmail.com
Subject: Re: bug#14403: 24.3.50;
	[regression] Typing non-ascii characters on a non-GUI
	MS-Windows	session
Date: Thu, 23 May 2013 19:23:55 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: dmoncayo <at> gmail.com,  14403-done <at> debbugs.gnu.org
> Date: Thu, 23 May 2013 10:02:14 -0400
> 
> I installed the patch below which I hope fixes this problem.

Thanks.  It seems to work for me, but let's wait for the Latin-1 guys
to see if it does for them.

Message #56 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Dani Moncayo <dmoncayo <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, 14403 <at> debbugs.gnu.org
Subject: Re: bug#14403: 24.3.50; [regression] Typing non-ascii characters on a
	non-GUI MS-Windows session
Date: Thu, 23 May 2013 19:22:49 +0200

>> I installed the patch below which I hope fixes this problem.
>
> Thanks.  It seems to work for me, but let's wait for the Latin-1 guys
> to see if it does for them.

It works fine here too.

Thank you.

--
Dani Moncayo

Message #59 received at 14403 <at> debbugs.gnu.org (full text, mbox):

From: Juanma Barranquero <lekktu <at> gmail.com>
To: Dani Moncayo <dmoncayo <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 14403 <at> debbugs.gnu.org
Subject: Re: bug#14403: 24.3.50; [regression] Typing non-ascii characters on a
	non-GUI MS-Windows session
Date: Thu, 23 May 2013 19:38:05 +0200

On Thu, May 23, 2013 at 7:22 PM, Dani Moncayo <dmoncayo <at> gmail.com> wrote:

> It works fine here too.

Same here.

   J

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.