Subject: term.el sometimes prints undecoded multibyte UTF-8 chars
From: Stephane Zermatten <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Date: Mon, 31 Mar 2025 17:18:35 +0300
List-Id: <debbugs-submit.debbugs.gnu.org>

Tags: patch

If I run a shell in a terminal with M-x term, with a very unicode-heavy
prompt (fish 3.6 + tide), sometimes the Unicode characters are printed
undecoded.

One possible cause of this might be unfortunate chunking in the middle
of a character, which the attached patch fixes.

Without the patch, if I type this in M-x term /usr/bin/bash

for j in $(seq 0 3); do
  for i in $(seq 0 30); do
    printf '\xf0\x9f'; sleep 0.1; printf '\x98\x80';
  done;
  echo;
done

I get
 \360\237\203\022\360\...

Instead of:
 😀😀😀😀😀😀😀...

With the patch included, I get the correct output.

The issue comes from an incorrect check (> count partial 0), which
should really be (and (>= count partial) (> partial 0)), but I
simplified that to (> partial 0) in the patch, because the while loop
guarantees (>= count partial).

I rewrote the existing test to cover this case, and try out multiple
different combination of chunks.

I'm still looking into other causes of the issue, but this, at least,
seems like an easy fix.

In GNU Emacs 30.1 (build 2, x86_64-apple-darwin23.6.0, NS appkit-2487.70
 Version 14.7.4 (Build 23H420)) of 2025-03-24 built on boomer.zia
Windowing system distributor 'Apple', version 10.3.2487
System Description:  macOS 14.7.4

Configured using:
 'configure --disable-dependency-tracking --disable-silent-rules
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs-plus@30/30.1/share/info/emacs
 --prefix=/usr/local/Cellar/emacs-plus@30/30.1
 --with-native-compilation=aot --with-xml2 --with-gnutls
 --without-compress-install --without-dbus --without-imagemagick
 --with-modules --with-rsvg --with-webp --with-ns
 --disable-ns-self-contained 'CFLAGS=-O2 -DFD_SETSIZE=10000
 -DDARWIN_UNLIMITED_SELECT -I/usr/local/opt/sqlite/include
 -I/usr/local/opt/gcc/include -I/usr/local/opt/libgccjit/include'
 'LDFLAGS=-L/usr/local/opt/sqlite/lib -L/usr/local/lib/gcc/14
 -I/usr/local/opt/gcc/include -I/usr/local/opt/libgccjit/include''

From 2bb6cec8f4f72009bcde1edab367f90ab82e5e2a Mon Sep 17 00:00:00 2001
From: Stephane Zermatten <[email-masked]>
Date: Mon, 31 Mar 2025 16:41:08 +0300
Subject: [PATCH] Fix issue with very short multibyte character chunk.

Before this change, a chunk containing only a part
of a multibyte character would be discarded and
displayed undecoded on the terminal.

* lisp/term.el
---
 lisp/term.el            |  2 +-
 test/lisp/term-tests.el | 15 ++++++++-------
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/lisp/term.el b/lisp/term.el
index 862103d88e6..a971300c055 100644
--- a/lisp/term.el
+++ b/lisp/term.el
@@ -3116,7 +3116,7 @@ term-emulate-terminal
                                                           (- count 1 partial)))
                                       'eight-bit))
                         (incf partial))
-                      (when (> count partial 0)
+                      (when (> partial 0)
                         (setq term-terminal-undecoded-bytes
                               (substring decoded-substring (- partial)))
                         (setq decoded-substring
diff --git a/test/lisp/term-tests.el b/test/lisp/term-tests.el
index 5ef8c1174df..aad84e171b2 100644
--- a/test/lisp/term-tests.el
+++ b/test/lisp/term-tests.el
@@ -402,13 +402,14 @@ term-to-margin
 (ert-deftest term-decode-partial () ;; Bug#25288.
   "Test multibyte characters sent into multiple chunks."
   ;; Set `locale-coding-system' so test will be deterministic.
-  (let* ((locale-coding-system 'utf-8-unix)
-         (string (make-string 7 ?ш))
-         (bytes (encode-coding-string string locale-coding-system)))
-    (should (equal string
-                   (term-test-screen-from-input
-                    40 1 `(,(substring bytes 0 (/ (length bytes) 2))
-                           ,(substring bytes (/ (length bytes) 2))))))))
+  (let ((locale-coding-system 'utf-8-unix))
+    (should (equal "шшш" (term-test-screen-from-input
+                          40 1 '("\321" "\210\321\210\321\210"))))
+    (should (equal "шшш" (term-test-screen-from-input
+                          40 1 '("\321\210\321" "\210\321\210"))))
+    (should (equal "шшш" (term-test-screen-from-input
+                          40 1 '("\321\210\321\210\321" "\210"))))))
+
 (ert-deftest term-undecodable-input () ;; Bug#29918.
   "Undecodable bytes should be passed through without error."
   (let* ((locale-coding-system 'utf-8-unix) ; As above.
-- 
2.47.0