Subject: Re: bug#55777: [PATCH] Improve documentation of `string-to-multibyte', `string-to-unibyte'

From: Richard Hansen <[email-masked]>
Date: Fri, 3 Jun 2022 23:28:51 -0400

On 6/3/22 03:02, Eli Zaretskii wrote:
> Thanks, but please explain the motivation for these changes.

The motivation is in the commit message, which I revised in the
attached patch to hopefully make it more clear.

> In particular, why would we need to describe in a doc string such 
> intimate details of our current implementation?
There is a fair amount of implementation detail right now; the patch
doesn't significantly change that. But I revised the patch to remove
some of the detail.

> If there was some situation where you needed these details for some 
> Lisp program, please describe that situation.
I'm trying to understand some inconsistent behavior I'm observing
while writing code to process binary data, and I found the existing
documentation lacking.

     ;; Unibyte vs. multibyte characters:
     (eq ?\xff ?\x3fffff)                           ; t (ok)
     (eq (aref "\x3fffff" 0) (aref "\xff" 0))       ; t (ok)
     (eq (aref "\x3fffff 😀" 0) (aref "\xff 😀" 0)) ; t (ok)
     (eq (aref "\xff" 0) (aref "\xff 😀" 0))        ; nil (expected t)

     ;; Unibyte vs. multibyte strings:
     (multibyte-string-p "\xff")                    ; nil (ok)
     (multibyte-string-p "\x3fffff")                ; nil (ok???)
     (string= "\xff" (string-to-multibyte "\xff"))  ; nil (expected t)

     ;; Char code vs. Unicode codepoint:
     (string= "😀\xff" "😀\x3fffff")                ; t (ok)
     (string= "😀\N{U+ff}" "😀\xff")                ; nil (ok)
     (string= "😀\N{U+ff}" "😀\x3fffff")            ; nil (ok)
     (string= "😀ÿ" "😀\N{U+ff}")                   ; t (ok)
     (string= "😀ÿ" "😀\xff")                       ; nil (ok)
     (string= "😀ÿ" "😀\x3fffff")                   ; nil (ok)
     (eq ?\N{U+ff} ?\xff)                           ; t (expected nil)
     (eq ?\N{U+ff} ?\x3fffff)                       ; t (expected nil)
     (eq ?ÿ ?\xff)                                  ; t (expected nil)
     (eq ?ÿ ?\x3fffff)                              ; t (expected nil)
From 6813b0a43250c9633d84d72418904025e973f1c8 Mon Sep 17 00:00:00 2001
From: Richard Hansen <[email-masked]>
Date: Fri, 3 Jun 2022 01:04:41 -0400
Subject: [PATCH] Improve documentation of `string-to-multibyte',
 `string-to-unibyte'

* doc/lispref/nonascii.texi (Converting Representations): Remove
confusing sentence from `string-to-multibyte' documentation (by
definition unibyte strings can only contain ASCII and eight-bit
characters, so there's no need to assume that unibyte strings only
contain those characters).  Fix description of the characters that
will cause `string-to-unibyte' to signal an error (`eight-bit'
characters are OK).  Remove some implementation details that are
discussed in the xrefed section.  Word the documentation for the two
functions similarly so that it is clear they are inverses of each
other.
---
 doc/lispref/nonascii.texi | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/doc/lispref/nonascii.texi b/doc/lispref/nonascii.texi
index d7d25dc36a..e8f02d8f2f 100644
--- a/doc/lispref/nonascii.texi
+++ b/doc/lispref/nonascii.texi
@@ -271,20 +271,19 @@ Converting Representations
 @defun string-to-multibyte string
 This function returns a multibyte string containing the same sequence
 of characters as @var{string}.  If @var{string} is a multibyte string,
-it is returned unchanged.  The function assumes that @var{string}
-includes only @acronym{ASCII} characters and raw 8-bit bytes; the
-latter are converted to their multibyte representation corresponding
-to the codepoints @code{#x3FFF80} through @code{#x3FFFFF}, inclusive
-(@pxref{Text Representations, codepoints}).
+it is returned unchanged.  Otherwise, byte values are transformed to
+their corresponding multibyte codepoints (@acronym{ASCII} characters
+and characters in the @code{eight-bit} charset).  @xref{Text
+Representations, codepoints}.
 @end defun
 
 @defun string-to-unibyte string
 This function returns a unibyte string containing the same sequence of
-characters as @var{string}.  It signals an error if @var{string}
-contains a non-@acronym{ASCII} character.  If @var{string} is a
-unibyte string, it is returned unchanged.  Use this function for
-@var{string} arguments that contain only @acronym{ASCII} and eight-bit
-characters.
+characters as @var{string}.  If @var{string} is a unibyte string, it
+is returned unchanged.  Otherwise, @acronym{ASCII} characters and
+characters in the @code{eight-bit} charset are converted to their
+corresponding byte values.  It signals an error if any other character
+is encountered.  @xref{Text Representations, codepoints}.
 @end defun
 
 @defun byte-to-string byte
-- 
2.36.1