GNU bug report logs - #37580
26.3; setting buffer as unibyte temporarily may change buffer contents

Previous Next

Full log

Message #11 received at 37580 <at> debbugs.gnu.org (full text, mbox):

From: ynyaaa <at> gmail.com
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 37580 <at> debbugs.gnu.org
Subject: Re: bug#37580: 26.3;
 setting buffer as unibyte temporarily may change buffer contents
Date: Sun, 06 Oct 2019 02:18:08 +0900

Eli Zaretskii <eliz <at> gnu.org> writes:
> I don't think this is a bug.  Changing the multibyte-ness of a buffer
> really does change the contents.  You should only do that where it
> makes sense.

Sometimes I find broken utf-8 texts on the Internet.
Some characters are split into surrogate pairs, and each surrogate
character is encoded as if it is a normal BMP character.

utf-8 coding system does not decode such sequences.
Changing multibyte-ness converts them to surrogate characters.
And encode-decode process with utf-16be outputs the intended characeters.

Suppose the character is #x10000,
the correspoding pair is (#xD800 #xDC00).
The miss-encoded sequence is:
  (encode-coding-string "\xD800\xDC00" 'utf-8)
  => "\355\240\200\355\260\200"

It is not decoded with utf-8.
  (decode-coding-string (encode-coding-string "\xD800\xDC00" 'utf-8)
                        'utf-8)
  => "\355\240\200\355\260\200"

Changing multibyte-ness, the sequence is converted into surrogate
characters.
  (with-temp-buffer
    (insert (encode-coding-string "\xD800\xDC00" 'utf-8))
    (set-buffer-multibyte nil)
    (set-buffer-multibyte t)
    (buffer-string))
  => "\xD800\xDC00"

The surrogate pair can be converted into the original character.
  (decode-coding-string (encode-coding-string "\xD800\xDC00" 'utf-16be)
                        'utf-16be)
  => "\x10000"

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.