GNU bug report logs - #43632
Raw bytes printed as latin-1 in echo area and *Messages*

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 43632 in the body.
You can then email your comments to 43632 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: bug-gnu-emacs <at> gnu.org
Subject: Raw bytes printed as latin-1 in echo area and *Messages*
Date: Sat, 26 Sep 2020 14:51:22 +0200

 M-: "\377"
 => "ÿ"

in both the echo area and in *Messages*. The expected message is "\377".

The same thing happens with

 (prin1 "\377" t)

This anomaly was first observed by Lars Ingebrigtsen.

Message #8 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 16:05:14 +0200

On Sep 26 2020, Mattias Engdegård wrote:

>  M-: "\377"
>  => "ÿ"
>
> in both the echo area and in *Messages*. The expected message is "\377".

That's because "\377" is a unibyte string.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #11 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 17:14:20 +0300

> From: Mattias Engdegård <mattiase <at> acm.org>
> Date: Sat, 26 Sep 2020 14:51:22 +0200
> 
>  M-: "\377"
>  => "ÿ"
> 
> in both the echo area and in *Messages*. The expected message is "\377".
> 
> The same thing happens with
> 
>  (prin1 "\377" t)
> 
> This anomaly was first observed by Lars Ingebrigtsen.

It is not an anomaly.  If you want to see escapes, set
print-escape-nonascii non-nil.

Also note that what you see is the result of 'eval' printing the
result, the real result (as returned by prin1) is a unibyte string:

  (multibyte-string-p (prin1 "\377")) => nil

(Yes, this is very confusing.)

Message #14 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 16:22:10 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> It is not an anomaly.  If you want to see escapes, set
> print-escape-nonascii non-nil.
>
> Also note that what you see is the result of 'eval' printing the
> result, the real result (as returned by prin1) is a unibyte string:
>
>   (multibyte-string-p (prin1 "\377")) => nil
>
> (Yes, this is very confusing.)

Could we do something about this confusion?

This came up because I just couldn't make sense of what I was seeing
when trying to work with raw bytes -- it seemed to be that Emacs was
auto-promoting unibyte strings (with bytes >127) to multibyte strings...
until I started calling multibyte-string-p on everything instead.

I'm guessing I'm not the only person confused by this stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #17 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: mattiase <at> acm.org, 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 17:26:04 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: Mattias Engdegård <mattiase <at> acm.org>,
>   43632 <at> debbugs.gnu.org
> Date: Sat, 26 Sep 2020 16:22:10 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > It is not an anomaly.  If you want to see escapes, set
> > print-escape-nonascii non-nil.
> >
> > Also note that what you see is the result of 'eval' printing the
> > result, the real result (as returned by prin1) is a unibyte string:
> >
> >   (multibyte-string-p (prin1 "\377")) => nil
> >
> > (Yes, this is very confusing.)
> 
> Could we do something about this confusion?

I don't want to say no, but I'm sure it is at least very hard.

> This came up because I just couldn't make sense of what I was seeing
> when trying to work with raw bytes -- it seemed to be that Emacs was
> auto-promoting unibyte strings (with bytes >127) to multibyte strings...
> until I started calling multibyte-string-p on everything instead.

We indeed convert to multibyte when we insert text into a multibyte
buffer, and that's a feature.

> I'm guessing I'm not the only person confused by this stuff.

You are not.  I learned to use multibyte-string-p when in doubt.

Message #20 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 18:00:55 +0200

26 sep. 2020 kl. 16.14 skrev Eli Zaretskii <eliz <at> gnu.org>:

First, thank you for your quick answer.

> If you want to see escapes, set
> print-escape-nonascii non-nil.

Certainly, but that isn't needed when printing to a buffer, as in (prin1 "\377").

The print-escape-nonascii docs say

  When the output goes in a multibyte buffer, this feature is
  enabled regardless of the value of the variable.

Why, then, is the echo area not treated as a multibyte buffer in this regard? Is there a practical reason or is it an artefact of history that cannot be changed?

(Note that the echo area buffers, the minibuffer, and *Messages* are all multibyte.)

If the behaviour of (prin1 x t) cannot be changed, then what about eval-expression (M-|)? Being an interactive command, surely compatibility isn't an obstacle to making it more useful?

I'm assuming that it would be more useful to see raw bytes shown octal-escaped or otherwise visually distinct from their interpretation as Latin-1. If nothing else, it would make sense to have the same behaviour as when evaluating something in *scratch*.

It would also be interesting to know why, when print-escape-nonascii is nil, unibyte strings are decoded specifically as Latin-1 (and not, say, UTF-8). I presume it is an artefact of history.

Message #23 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 19:10:10 +0300

> From: Mattias Engdegård <mattiase <at> acm.org>
> Date: Sat, 26 Sep 2020 18:00:55 +0200
> Cc: 43632 <at> debbugs.gnu.org
> 
> Why, then, is the echo area not treated as a multibyte buffer in
> this regard?

It is.  That's not the reason.

> If the behaviour of (prin1 x t) cannot be changed, then what about eval-expression (M-|)? Being an interactive command, surely compatibility isn't an obstacle to making it more useful?

We were talking about M-:.  And that does call prin1.  And prin1 does
produce a unibyte string, the reason for the display that confused you
is elsewhere.

> It would also be interesting to know why, when print-escape-nonascii is nil, unibyte strings are decoded specifically as Latin-1 (and not, say, UTF-8). I presume it is an artefact of history.

They are not decoded as Latin-1, they are simply inserted into a
buffer as a single byte.  And we do it regardless of
print-escape-nonascii, because \377 is the Unicode (and Latin-1)
codepoint of ÿ.

So it isn't a historical artifact.  I think it's simply a consequence
of how print functions work.

Message #26 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 18:53:39 +0200

26 sep. 2020 kl. 18.10 skrev Eli Zaretskii <eliz <at> gnu.org>:

> We were talking about M-:.

Sorry, typo of mine. Thanks for the correction.

>  And that does call prin1.  And prin1 does
> produce a unibyte string, the reason for the display that confused you
> is elsewhere.

We probably disagree about the details here, but let's leave prin1 aside; it doesn't need to be changed in order to improve the M-: user experience.
What about this little tweak?

--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -1797,6 +1797,7 @@ eval-expression
   (let ((print-length (unless no-truncate eval-expression-print-length))
         (print-level  (unless no-truncate eval-expression-print-level))
         (eval-expression-print-maximum-character char-print-limit)
+        (print-escape-nonascii t)
         (deactivate-mark))
     (let ((out (if insert-value (current-buffer) t)))
       (prog1

That way, it works more like the other interactive Lisp evaluation commands in this particular respect.
Do you think a NEWS entry is called for?

Message #29 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 19:57:21 +0300

> From: Mattias Engdegård <mattiase <at> acm.org>
> Date: Sat, 26 Sep 2020 18:53:39 +0200
> Cc: 43632 <at> debbugs.gnu.org
> 
> --- a/lisp/simple.el
> +++ b/lisp/simple.el
> @@ -1797,6 +1797,7 @@ eval-expression
>    (let ((print-length (unless no-truncate eval-expression-print-length))
>          (print-level  (unless no-truncate eval-expression-print-level))
>          (eval-expression-print-maximum-character char-print-limit)
> +        (print-escape-nonascii t)
>          (deactivate-mark))
>      (let ((out (if insert-value (current-buffer) t)))
>        (prog1

We are going to disregard user preferences regarding escapes?  For an
obscure use case?  That doesn't make too much sense to me.

Message #32 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: mattiase <at> acm.org
Cc: 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 20:27:23 +0300

> Date: Sat, 26 Sep 2020 19:57:21 +0300
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: 43632 <at> debbugs.gnu.org
> 
> > From: Mattias Engdegård <mattiase <at> acm.org>
> > Date: Sat, 26 Sep 2020 18:53:39 +0200
> > Cc: 43632 <at> debbugs.gnu.org
> > 
> > --- a/lisp/simple.el
> > +++ b/lisp/simple.el
> > @@ -1797,6 +1797,7 @@ eval-expression
> >    (let ((print-length (unless no-truncate eval-expression-print-length))
> >          (print-level  (unless no-truncate eval-expression-print-level))
> >          (eval-expression-print-maximum-character char-print-limit)
> > +        (print-escape-nonascii t)
> >          (deactivate-mark))
> >      (let ((out (if insert-value (current-buffer) t)))
> >        (prog1
> 
> We are going to disregard user preferences regarding escapes?  For an
> obscure use case?  That doesn't make too much sense to me.

I'd rather risk something like the below instead:

diff --git a/src/print.c b/src/print.c
index 0ecc98f..5d878c9 100644
--- a/src/print.c
+++ b/src/print.c
@@ -1993,6 +1993,10 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
                     }
                   else if (print_escape_control_characters && c_iscntrl (c))
 		    octalout (c, SDATA (obj), i_byte, size_byte, printcharfun);
+		  else if (!multibyte
+			   && SINGLE_BYTE_CHAR_P (c)
+			   && !ASCII_CHAR_P (c))
+		    printchar (BYTE8_TO_CHAR (c), printcharfun);
                   else
                     printchar (c, printcharfun);
 		  need_nonhex = false;

Message #35 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sat, 26 Sep 2020 21:53:01 +0200

26 sep. 2020 kl. 18.57 skrev Eli Zaretskii <eliz <at> gnu.org>:

> We are going to disregard user preferences regarding escapes?

If you mean print-escape-nonascii then it has no effect on values printed from interactive Lisp evaluation in the *scratch* buffer (C-j) either. Since that behaviour is less deceptive and appears more useful in general, it would both bring some consistency to the mess and make users happier. It not only fixes M-: but also C-x C-e and C-M-x.

> I'd rather risk something like the below instead:

By all means -- almost as good and better then what we have today.

Message #40 received at 43632-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: 43632-done <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sun, 27 Sep 2020 08:56:23 +0300

> From: Mattias Engdegård <mattiase <at> acm.org>
> Date: Sat, 26 Sep 2020 21:53:01 +0200
> Cc: 43632 <at> debbugs.gnu.org
> 
> > I'd rather risk something like the below instead:
> 
> By all means -- almost as good and better then what we have today.

Thanks, installed on the master branch.  Let's see what, if anything,
this will break.

Message #43 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mattiase <at> acm.org, 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sun, 27 Sep 2020 12:58:50 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> I'd rather risk something like the below instead:

That's definitely an improvement, thanks.  It's still slightly confusing
(but not as confusing as before).

In the *scratch* buffer, C-x C-e:

"\377"
=> "\377"

But the echo area says "\xff".

because I have (setq display-raw-bytes-as-hex t).

(string-to-multibyte "\377")
=> "\377"

And the echo area says "\377", as expected.

Hm.  Or perhaps that's not confusing?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #46 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mattiase <at> acm.org, 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sun, 27 Sep 2020 13:13:31 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> In the *scratch* buffer, C-x C-e:
>> 
>> "\377"
>> => "\377"
>> 
>> But the echo area says "\xff".
>> 
>> because I have (setq display-raw-bytes-as-hex t).
>
> So you've explicitly asked for that, no?

I have, so I guess the only confusing thing is that it's still "\377" in
the scratch buffer?

>> (string-to-multibyte "\377")
>> => "\377"
>> 
>> And the echo area says "\377", as expected.
>
> That's unrelated: Emacs behaved the same before my changes.

Yup.  But don't we call these "raw bytes", too?  I was just idly wondering
whether the display of these should also be affected by
`display-raw-bytes-as-hex'...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #49 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: mattiase <at> acm.org, 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sun, 27 Sep 2020 14:25:49 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: mattiase <at> acm.org,  43632 <at> debbugs.gnu.org
> Date: Sun, 27 Sep 2020 13:13:31 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> In the *scratch* buffer, C-x C-e:
> >> 
> >> "\377"
> >> => "\377"
> >> 
> >> But the echo area says "\xff".
> >> 
> >> because I have (setq display-raw-bytes-as-hex t).
> >
> > So you've explicitly asked for that, no?
> 
> I have, so I guess the only confusing thing is that it's still "\377" in
> the scratch buffer?

In the scratch buffer you have a string of 4 ASCII characters, not of
1 raw byte, right?

> >> (string-to-multibyte "\377")
> >> => "\377"
> >> 
> >> And the echo area says "\377", as expected.
> >
> > That's unrelated: Emacs behaved the same before my changes.
> 
> Yup.  But don't we call these "raw bytes", too?  I was just idly wondering
> whether the display of these should also be affected by
> `display-raw-bytes-as-hex'...

No, because what you have in *scratch* is again a string of 4 ASCII
characters.

Message #52 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: mattiase <at> acm.org, 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sun, 27 Sep 2020 14:05:25 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: mattiase <at> acm.org,  43632 <at> debbugs.gnu.org
> Date: Sun, 27 Sep 2020 12:58:50 +0200
> 
> In the *scratch* buffer, C-x C-e:
> 
> "\377"
> => "\377"
> 
> But the echo area says "\xff".
> 
> because I have (setq display-raw-bytes-as-hex t).

So you've explicitly asked for that, no?

> (string-to-multibyte "\377")
> => "\377"
> 
> And the echo area says "\377", as expected.

That's unrelated: Emacs behaved the same before my changes.

Message #55 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mattiase <at> acm.org, 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sun, 27 Sep 2020 13:36:41 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> I have, so I guess the only confusing thing is that it's still "\377" in
>> the scratch buffer?
>
> In the scratch buffer you have a string of 4 ASCII characters, not of
> 1 raw byte, right?

Ah, yes.

>> >> (string-to-multibyte "\377")
>> >> => "\377"
>> >> 
>> >> And the echo area says "\377", as expected.
>> >
>> > That's unrelated: Emacs behaved the same before my changes.
>> 
>> Yup.  But don't we call these "raw bytes", too?  I was just idly wondering
>> whether the display of these should also be affected by
>> `display-raw-bytes-as-hex'...
>
> No, because what you have in *scratch* is again a string of 4 ASCII
> characters.

Here I was talking about what's displayed in the echo area.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #58 received at 43632 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: mattiase <at> acm.org, 43632 <at> debbugs.gnu.org
Subject: Re: bug#43632: Raw bytes printed as latin-1 in echo area and
 *Messages*
Date: Sun, 27 Sep 2020 14:43:09 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: mattiase <at> acm.org,  43632 <at> debbugs.gnu.org
> Date: Sun, 27 Sep 2020 13:36:41 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> >> (string-to-multibyte "\377")
> >> >> => "\377"
> >> >> 
> >> >> And the echo area says "\377", as expected.
> >> >
> >> > That's unrelated: Emacs behaved the same before my changes.
> >> 
> >> Yup.  But don't we call these "raw bytes", too?  I was just idly wondering
> >> whether the display of these should also be affected by
> >> `display-raw-bytes-as-hex'...
> >
> > No, because what you have in *scratch* is again a string of 4 ASCII
> > characters.
> 
> Here I was talking about what's displayed in the echo area.  :-)

That, too, is a string of 4 ASCII characters.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.