GNU bug report logs - #5794
23.1; UnicodeEncodeError in python-shell

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 5794 in the body.
You can then email your comments to 5794 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Max Arnold <lwarxx <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 23.1; UnicodeEncodeError in python-shell
Date: Mon, 29 Mar 2010 11:38:09 +0700

When python shell invoked via C-c C-c, its stdout encoding is set to None and unicode
characters can not be printed. Simple test case (0xA9 should produce unicode symbol (c) ):

1. Invoke python shell manually:

M-x python-shell
>>> print u'\xA9'
©
>>>

2. And this is how to trigger the error: close python shell opened at step (1), create
new buffer containing the same print command, switch it to python-mode and press
C-c C-c (python-send-buffer). Spawned python shell will show a traceback:

>>> Traceback (most recent call last):
  File "/tmp/py5308nvC", line 1, in <module>
    print u'\xA9'
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128)
>>>

Emacs was running with -nw mode in rxvt-unicode terminal, system-wide LANG is set
to ru_RU.UTF-8. On step (1) sys.stdout.encoding is UTF-8, on step (2) it is None.

The same issue was found using python-mode.el (https://bugs.launchpad.net/python-mode/+bug/550661).
For more details see this thread: http://lists.gnu.org/archive/html/help-gnu-emacs/2010-03/msg00233.html


In GNU Emacs 23.1.1 (i686-pc-linux-gnu, GTK+ Version 2.14.7)
 of 2009-12-15 on bbone
configured using `configure  '--prefix=/usr' '--build=i686-pc-linux-gnu'
'--host=i686-pc-linux-gnu' '--mandir=/usr/share/man' '--infodir=/usr/share/info'
'--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib'
'--program-suffix=-emacs-23' '--infodir=/usr/share/info/emacs-23'
'--with-sound' '--with-x' '--with-toolkit-scroll-bars' '--without-gif'
'--without-jpeg' '--without-png' '--without-rsvg' '--without-tiff'
'--without-xpm' '--with-xft' '--without-libotf' '--without-m17n-flt'
'--with-x-toolkit=gtk' '--without-hesiod' '--without-kerberos'
'--without-kerberos5' '--with-gpm' '--with-dbus' 'build_alias=i686-pc-linux-gnu'
'host_alias=i686-pc-linux-gnu' 'CFLAGS=-march=i686 -O2 -pipe' 'LDFLAGS=-Wl,-O1''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ru_RU.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  show-paren-mode: t
  iswitchb-mode: t
  global-hl-line-mode: t
  ergoemacs-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
ESC a r e p o TAB r t - e TAB RET

Recent messages:
("emacs")
Loading ~/.emacs.d/ergoemacs-mode/5.1/ergoemacs-mode (compiled; note, source file is newer)...
Loading functions...done
Loading ergoemacs-unbind...done
Loading ergoemacs-layout-us...done
Loading ~/.emacs.d/ergoemacs-mode/5.1/ergoemacs-mode (compiled; note, source file is newer)...done
Ido mode enabled [3 times]
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

Message #8 received at 5794 <at> debbugs.gnu.org (full text, mbox):

From: Chong Yidong <cyd <at> stupidchicken.com>
To: Dave Love <fx <at> gnu.org>
Cc: Max Arnold <lwarxx <at> gmail.com>, 5794 <at> debbugs.gnu.org
Subject: Re: 23.1; UnicodeEncodeError in python-shell
Date: Sat, 10 Apr 2010 14:21:40 -0400

> When python shell invoked via C-c C-c, its stdout encoding is set to
> None and unicode characters can not be printed. Simple test case (0xA9
> should produce unicode symbol (c) ):
>
> 1. Invoke python shell manually:
>
> M-x python-shell
> >>> print u'\xA9'
> ©
> >>>
>
> 2. And this is how to trigger the error: close python shell opened at
> step (1), create new buffer containing the same print command, switch
> it to python-mode and press C-c C-c (python-send-buffer). Spawned
> python shell will show a traceback:

> >>> Traceback (most recent call last):
>   File "/tmp/py5308nvC", line 1, in <module>
>     print u'\xA9'
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in
> position 0: ordinal not in range(128)

This is due to the following line in `run-python', which I merged from
the upstream python.el on 2008-02-21:

       ;; Suppress use of pager for help output:
       (process-connection-type nil))

Dave, do you remember what the rationale for this is?  (I don't
understand the comment.)

I'm not sure why using a pipe rather than a pty makes a difference,
though.

Message #11 received at 5794 <at> debbugs.gnu.org (full text, mbox):

From: Max Arnold <lwarxx <at> gmail.com>
To: Chong Yidong <cyd <at> stupidchicken.com>
Cc: 5794 <at> debbugs.gnu.org, Dave Love <fx <at> gnu.org>
Subject: Re: 23.1; UnicodeEncodeError in python-shell
Date: Sun, 11 Apr 2010 09:47:52 +0700

On Sat, Apr 10, 2010 at 02:21:40PM -0400, Chong Yidong wrote:
> > >>> Traceback (most recent call last):
> >   File "/tmp/py5308nvC", line 1, in <module>
> >     print u'\xA9'
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in
> > position 0: ordinal not in range(128)
> 
> This is due to the following line in `run-python', which I merged from
> the upstream python.el on 2008-02-21:
> 
>        ;; Suppress use of pager for help output:
>        (process-connection-type nil))
> 
> Dave, do you remember what the rationale for this is?  (I don't
> understand the comment.)
> 
> I'm not sure why using a pipe rather than a pty makes a difference,
> though.

This is probably for python builtin help() function, which can use pager
to scroll documentation page by page. When help() detects that terminal is
a pipe, it disables interactive scrolling and just prints full output as is.
Try this for example:

import os
help(os)

Fortunately, python understands $PAGER environment variable and will use it
for scrolling, so PAGER="cat" will have the same effect as using pipe.

Hope this helps.

Message #14 received at 5794 <at> debbugs.gnu.org (full text, mbox):

From: Dave Love <fx <at> gnu.org>
To: Chong Yidong <cyd <at> stupidchicken.com>
Cc: Max Arnold <lwarxx <at> gmail.com>,
	"5794 <at> debbugs.gnu.org" <5794 <at> debbugs.gnu.org>
Subject: Re: 23.1; UnicodeEncodeError in python-shell
Date: Tue, 13 Apr 2010 00:04:58 +0100

Chong Yidong <cyd <at> stupidchicken.com> writes:

> This is due to the following line in `run-python', which I merged from
> the upstream python.el on 2008-02-21:
>
>        ;; Suppress use of pager for help output:
>        (process-connection-type nil))
>
> Dave, do you remember what the rationale for this is?  (I don't
> understand the comment.)

It's the most likely way to suppress interactive i/o generally, which is
why it should normally be used for inferior interpreters and things like
VC, apart from the reasons in the manual to avoid PTYs.

> I'm not sure why using a pipe rather than a pty makes a difference,
> though.

It's certainly odd behaviour, but as you're dropping python.el for
python-mode.el, you needn't worry about it.  Amongst other things,
python-mode.el doesn't have the features that require results from the
sub-process.

Message #17 received at 5794 <at> debbugs.gnu.org (full text, mbox):

From: Dave Love <fx <at> gnu.org>
To: Max Arnold <lwarxx <at> gmail.com>
Cc: "5794 <at> debbugs.gnu.org" <5794 <at> debbugs.gnu.org>,
	Chong Yidong <cyd <at> stupidchicken.com>
Subject: Re: 23.1; UnicodeEncodeError in python-shell
Date: Tue, 13 Apr 2010 00:08:16 +0100

Max Arnold <lwarxx <at> gmail.com> writes:

> Fortunately, python understands $PAGER environment variable and will use it
> for scrolling, so PAGER="cat" will have the same effect as using pipe.

I assume it will lose on MS Windows, which is why I rejected it, not
that I have any interest in MS Windows.

For a working version see http://www.loveshack.ukfsn.org/emacs/.  As far
as I know it doesn't have any unfixed reported bugs, or any of the bugs
people keep reporting to me against the fork in Emacs.  You might want
it anyway when things like symbol help and completion are dropped from
Emacs, or for things like Python 3 support now.

Message #20 received at 5794 <at> debbugs.gnu.org (full text, mbox):

From: Chong Yidong <cyd <at> stupidchicken.com>
To: Max Arnold <lwarxx <at> gmail.com>
Cc: 5794 <at> debbugs.gnu.org
Subject: Re: 23.1; UnicodeEncodeError in python-shell
Date: Tue, 24 Aug 2010 16:22:41 -0400

>>> When python shell invoked via C-c C-c, its stdout encoding is set to
>>> None and unicode characters can not be printed.
>
>> This is due to the following line in `run-python'
>>   (process-connection-type nil))
>> I'm not sure why using a pipe rather than a pty makes a difference,
>> though.
>
> This is probably for python builtin help() function, which can use
> pager to scroll documentation page by page. When help() detects that
> terminal is a pipe, it disables interactive scrolling and just prints
> full output as is.

I've changed python.el in the trunk to use a pty; this should remove the
problem.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.