GNU bug report logs - #24741
25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 24741 in the body.
You can then email your comments to 24741 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: John Wiegley <jwiegley <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Cc: Simon Peyton Jones <simonpj <at> microsoft.com>
Subject: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode
Date: Wed, 19 Oct 2016 10:32:36 -0700

[Message part 1 (text/plain, inline)]

I have a friend who is using Emacs 24.5 on Windows, and who is experiencing a
strange behavior: Unicode quotes display properly, but after *invoking* a
'make' command -- even with its output fully redirected to a file -- buffer
decoding begins to fail, causing later quotes to be display as escape codes.

I'm attaching a screenshot from his machine, to show the behavior:

[Untitled.png (image/png, inline)]

[Message part 3 (text/plain, inline)]

Note that in a fresh shell, cat'ing the saved file (test-output) does NOT mess
up the quotes. So it's a mysterious side-effect of calling 'make'. Also,
saying 'make xxxx' where xxxx is not a valid target, produces the appropriate
error message and does not mess up the quotes.

Any ideas?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[signature.asc (application/pgp-signature, inline)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Peyton Jones <simonpj <at> microsoft.com>
To: John Wiegley <jwiegley <at> gmail.com>, "bug-gnu-emacs <at> gnu.org"
 <bug-gnu-emacs <at> gnu.org>
Subject: RE: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode
Date: Wed, 19 Oct 2016 19:07:44 +0000

Thanks John.

It's emacs 24.5.1 (686-pc-mingw32)

It's not 'make' per se; for example 'make --version' or 'make boot' (which does nothing) is fine.  The invocation of make that causes the funny quotes to appear invokes Python, if that helps.

Deeply mysterious.

If I start a new shell, I get quotes looking nice again.  Could something in the shell environment be affected by invoking python?

Simon

| -----Original Message-----
| From: John Wiegley [mailto:jwiegley <at> gmail.com]
| Sent: 19 October 2016 18:33
| To: bug-gnu-emacs <at> gnu.org
| Cc: Simon Peyton Jones <simonpj <at> microsoft.com>
| Subject: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode
| 
| I have a friend who is using Emacs 24.5 on Windows, and who is
| experiencing a strange behavior: Unicode quotes display properly, but
| after *invoking* a 'make' command -- even with its output fully
| redirected to a file -- buffer decoding begins to fail, causing later
| quotes to be display as escape codes.
| 
| I'm attaching a screenshot from his machine, to show the behavior:

Message #11 received at 24741 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Simon Peyton Jones <simonpj <at> microsoft.com>
Cc: jwiegley <at> gmail.com, 24741 <at> debbugs.gnu.org
Subject: Re: bug#24741: 25.1.50;
 Buffer encoding corrupted up by 'make' in shell-mode
Date: Thu, 20 Oct 2016 10:14:55 +0300

> From: Simon Peyton Jones <simonpj <at> microsoft.com>
> Date: Wed, 19 Oct 2016 19:07:44 +0000
> 
> It's not 'make' per se; for example 'make --version' or 'make boot' (which does nothing) is fine.  The invocation of make that causes the funny quotes to appear invokes Python, if that helps.
> 
> Deeply mysterious.
> 
> If I start a new shell, I get quotes looking nice again.  Could something in the shell environment be affected by invoking python?

There aren't enough data in the report, only hints.  The image posted
by John indicates a Unixy shell is also involved, so it could be some
issue with the shell itself or its environment (I'm guessing the shell
is either a Cygwin or an MSYS2 Bash, which might mean the Emacs setup
of running them involves setting some coding-systems to UTF-8, for
example).

Then there are 'make' and Python, which are somehow related, but it's
unclear how, due to lack of a complete description.

In general, this looks like UTF-8 output by a subprocess gets decoded
using the Windows codepage.  But it's impossible to tell where exactly
is the problem without knowing the details.

IOW, a reproducible recipe, starting from "emacs -Q", and including
only those customizations that are needed for the reproduction, is
necessary to analyze this problem.  It is most likely a problem either
with user customization of the coding-systems, or with some subtle
incompatibility between Cygwin/MSYS programs and the native Windows
build of Emacs.

Message #14 received at 24741 <at> debbugs.gnu.org (full text, mbox):

From: John Wiegley <jwiegley <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 24741 <at> debbugs.gnu.org, Simon Peyton Jones <simonpj <at> microsoft.com>
Subject: Re: bug#24741: 25.1.50;
 Buffer encoding corrupted up by 'make' in shell-mode
Date: Thu, 20 Oct 2016 12:19:30 -0700

Eli makes an excellent point: We should try starting Emacs with the -Q flag
next, and attempt a reproduction following the same steps.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Message #17 received at 24741 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: John Wiegley <jwiegley <at> gmail.com>
Cc: 24741 <at> debbugs.gnu.org, simonpj <at> microsoft.com
Subject: Re: bug#24741: 25.1.50;
 Buffer encoding corrupted up by 'make' in shell-mode
Date: Fri, 21 Oct 2016 09:42:49 +0300

> From: John Wiegley <jwiegley <at> gmail.com>
> Cc: Simon Peyton Jones <simonpj <at> microsoft.com>,  24741 <at> debbugs.gnu.org
> Date: Thu, 20 Oct 2016 12:19:30 -0700
> 
> Eli makes an excellent point: We should try starting Emacs with the -Q flag
> next, and attempt a reproduction following the same steps.

Actually, posting here the relevant parts of Simon's customizations
(the ones that are related to coding-systems and to running shells)
might also help identify the probable causes.

Message #20 received at 24741 <at> debbugs.gnu.org (full text, mbox):

From: Simon Peyton Jones <simonpj <at> microsoft.com>
To: Eli Zaretskii <eliz <at> gnu.org>, John Wiegley <jwiegley <at> gmail.com>
Cc: "24741 <at> debbugs.gnu.org" <24741 <at> debbugs.gnu.org>,
 Simon Peyton Jones <simonpj <at> microsoft.com>
Subject: RE: bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in
 shell-mode
Date: Fri, 21 Oct 2016 21:20:34 +0000

Thanks everyone.  It's a bizarre bug!  But it's one that does make my life a lot worse, because my compiler's error messages are littered with quotes, each of which turns into 12 characters of encoding, as you can see.

If you want me to try emacs -Q, just let me know how to do that and what info to collect.  I've never done that before.

Customisations. In my .emacs I have the following, which John Wiegley gave to me in the first place

(setq-default locale-coding-system 'utf-8-unix)
(setq-default buffer-file-coding-system 'utf-8-unix)
(set-default-coding-systems 'utf-8-unix)
(set-terminal-coding-system 'utf-8-unix)
(set-keyboard-coding-system 'utf-8-unix)
(set-selection-coding-system 'utf-8-unix)
(prefer-coding-system 'utf-8-unix)

(defun utf8-shell ()
  (interactive)
  (set-default-coding-systems 'utf-8-unix)
  (shell))

(setq w32-get-true-file-attributes nil)

Simon


|  -----Original Message-----
|  From: Eli Zaretskii [mailto:eliz <at> gnu.org]
|  Sent: 21 October 2016 07:43
|  To: John Wiegley <jwiegley <at> gmail.com>
|  Cc: Simon Peyton Jones <simonpj <at> microsoft.com>; 24741 <at> debbugs.gnu.org
|  Subject: Re: bug#24741: 25.1.50; Buffer encoding corrupted up by
|  'make' in shell-mode
|  
|  > From: John Wiegley <jwiegley <at> gmail.com>
|  > Cc: Simon Peyton Jones <simonpj <at> microsoft.com>,
|  24741 <at> debbugs.gnu.org
|  > Date: Thu, 20 Oct 2016 12:19:30 -0700
|  >
|  > Eli makes an excellent point: We should try starting Emacs with the
|  -Q
|  > flag next, and attempt a reproduction following the same steps.
|  
|  Actually, posting here the relevant parts of Simon's customizations
|  (the ones that are related to coding-systems and to running shells)
|  might also help identify the probable causes.

Message #23 received at 24741 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Simon Peyton Jones <simonpj <at> microsoft.com>
Cc: jwiegley <at> gmail.com, 24741 <at> debbugs.gnu.org
Subject: Re: bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in
 shell-mode
Date: Sat, 22 Oct 2016 10:17:48 +0300

> From: Simon Peyton Jones <simonpj <at> microsoft.com>
> CC: "24741 <at> debbugs.gnu.org" <24741 <at> debbugs.gnu.org>, Simon Peyton Jones
> 	<simonpj <at> microsoft.com>
> Date: Fri, 21 Oct 2016 21:20:34 +0000
> 
> If you want me to try emacs -Q, just let me know how to do that and what info to collect.  I've never done that before.

Start Emacs with "emacs -Q", then try reproducing the problem by doing
whatever you do when the problem normally happens.

If "emacs -Q" lacks some optional features and custom settings of
certain variables, load those features and customize those variables
as part of the recipe (i.e. record everything you need to do starting
from "emacs -Q" until you are able to run the commands that produce
the problematic output).

Then post here the full recipe for reproducing the problem, including
everything you have recorded on the way.

> Customisations. In my .emacs I have the following, which John Wiegley gave to me in the first place
> 
> (setq-default locale-coding-system 'utf-8-unix)
> (setq-default buffer-file-coding-system 'utf-8-unix)
> (set-default-coding-systems 'utf-8-unix)
> (set-terminal-coding-system 'utf-8-unix)
> (set-keyboard-coding-system 'utf-8-unix)
> (set-selection-coding-system 'utf-8-unix)
> (prefer-coding-system 'utf-8-unix)
> 
> (defun utf8-shell ()
>   (interactive)
>   (set-default-coding-systems 'utf-8-unix)
>   (shell))

This is already dead wrong on Windows, especially setting the default
encodings to UTF-8.  It cannot possibly work well on Windows.  And you
shouldn't need this.

What happens if you remove these and try again -- does the problem
still appear?

Where did you get your Bash, your make.exe, and your Python?  If they
are from the MSYS2 project, the only customization of coding-systems
you may need is of process-coding-system-alist.  If you do that, make
sure the encoding is still your system codepage, and only the decoding
part is UTF-8.  That's because the encoding is used to encode the
command-line arguments, and you certainly don't want them to be
encoded in UTF-8 on Windows, because this is unsupported.

Message #28 received at 24741-done <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: 24741-done <at> debbugs.gnu.org
Subject: Re: bug#24741: 25.1.50;
 Buffer encoding corrupted up by 'make' in shell-mode
Date: Wed, 07 Dec 2016 15:29:42 -0500

Eli Zaretskii wrote:

> Start Emacs with "emacs -Q", then try reproducing the problem by doing
> whatever you do when the problem normally happens.
>
> If "emacs -Q" lacks some optional features and custom settings of
> certain variables, load those features and customize those variables
> as part of the recipe (i.e. record everything you need to do starting
> from "emacs -Q" until you are able to run the commands that produce
> the problematic output).
>
> Then post here the full recipe for reproducing the problem, including
> everything you have recorded on the way.
[...]
> This is already dead wrong on Windows, especially setting the default
> encodings to UTF-8.  It cannot possibly work well on Windows.  And you
> shouldn't need this.
>
> What happens if you remove these and try again -- does the problem
> still appear?
>
> Where did you get your Bash, your make.exe, and your Python?  If they
> are from the MSYS2 project, the only customization of coding-systems
> you may need is of process-coding-system-alist.  If you do that, make
> sure the encoding is still your system codepage, and only the decoding
> part is UTF-8.  That's because the encoding is used to encode the
> command-line arguments, and you certainly don't want them to be
> encoded in UTF-8 on Windows, because this is unsupported.

No further reponse, closing.
Possibly a bit soon, so feel free to reopen.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.