Subject: Re: bug#396: 23.0.60; Quail (swedish postfix) broken in console

From: Stefan Monnier <[email-masked]>
Date: Sun, 15 Jun 2008 00:46:36 -0400

> Since the merge of the unicode branch on Feb 1, the quail package
> swedish-postfix has been broken in console.  When using X11 it works.
> Also, there is no problem to use the Swedish keyboard, so I believe that
> the problem is in quail.

> C-x b f o o RET C-x RET C-\ s w e d i s h - p o s t
> f i x RET a a SPC SPC

> Instead of the string "å  " I end up with 堠 (#x5820).  

Yuck!
Here's what seems to be happening: quail takes a sequence of bytes from
your terminal, interprets them as ascii chars, and turns this sequence
into another sequence, which is composed of latin-1 chars, but which
encoded-kbd then takes for bytes coming straight from your terminal, so
it decodes this sequence as a utf-8 encoded char: "a a" turns into #xe5
and somehow encoded-kbd confuses "#xe5 SPC SPC" for "#xE5 #xA0 #xA0".

I've installed the patch below which should fix your problem most of
the time.  But to fix it right, I think we'd need to hook encoded-kbd
earlier in the chain so it gets processed before input-methods are run.

        Stefan

--- encoded-kb.el.~1.45.~	2008-05-06 23:35:23.000000000 -0400
+++ encoded-kb.el	2008-06-15 00:39:58.000000000 -0400
@@ -219,8 +219,9 @@
 
 (defun encoded-kbd-self-insert-utf-8 (arg)
   (interactive "p")
-  (let ((char (encoded-kbd-last-key))
-	len)
+  (let* ((lead (encoded-kbd-last-key))
+         (char lead)
+         len event)
     (cond ((< char #xE0)
 	   (setq len 1 char (logand char #x1F)))
 	  ((< char #xF0)
@@ -230,8 +231,22 @@
 	  (t
 	   (setq len 4 char 0)))
     (while (> len 0)
-      (setq char (logior (lsh char 6) (logand (read-char-exclusive) #x3F))
-	    len (1- len)))
+      (setq event (read-char-exclusive))
+      (if (and (>= event #x80) (< event #xc0))
+          ;; Valid utf-8 sequence.
+          (setq char (logior (lsh char 6) (- event #x80))
+                len (1- len))
+        ;; Invalid utf-8 sequence.  Might be because Quail got involved
+        ;; in-between and the bytes we thought we were reading were actually
+        ;; latin-1 chars.  Let's presume that `event' is the second "byte",
+        ;; i.e. there weren't any "apprently correct" between `lead' and
+        ;; `event': it's easy to recover in this case, and the more general
+        ;; case seems pretty unlikely.
+        ;; FIXME: We should really do encoded-kbd decoding before processing
+        ;; input-methods.
+        (push event unread-command-events)
+        (setq char lead)
+        (setq len 0)))
     (vector char)))
 
 (defun encoded-kbd-setup-keymap (keymap coding)