GNU bug report logs - #23095
24.5; python-shell-unbuffered causes problems on Windows

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 23095 in the body.
You can then email your comments to 23095 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Will Wykeham <will <at> wykeham.net>
To: bug-gnu-emacs <at> gnu.org
Subject: 24.5; python-shell-unbuffered causes problems on Windows
Date: Tue, 22 Mar 2016 15:01:27 +0000

The relevant context is mostly below, but the important bits are:
Emacs 24.5
Microsoft Windows 8
Python 2.7.11

From a user facing point of view, the issue I'm having is that 'out of
the box', the function python-shell-prompt-detect produces the warning
"Warning (emacs): Python shell prompts cannot be detected".

To reduce the test case down, this demonstrates the core problem:

(let ((process-environment (python-shell-calculate-process-environment))
       (code (concat "import sys\n"
                    "print(sys.ps1)")))
  (with-temp-buffer
    (let ((code-file (python-shell--save-temp-file code)))
      (call-process "python" code-file '(t nil) nil "-i")
      (delete-file code-file))
    (buffer-string)))

In an 'out of the box' emacs, with 'python-shell-unbuffered' set to t as
default, this evaluates to an empty string.

If python-shell-unbuffered is then set to nil, then the above evaluates to
the expected ">>>\n".

I haven't got to the bottom of why unbuffered is causing this
problem. My suspicion is that it relates to the fact that
PYTHONUNBUFFERED on Windows also causes python to process stdin/stdout
as binary (which also affects line ending processing).

Happy to provide any further information, testing etc. to help narrow
down the issue.

Regards,
Will

********************************************

In GNU Emacs 24.5.1 (i686-pc-mingw32)
 of 2015-04-11 on LEG570
Windowing system distributor `Microsoft Corp.', version 6.3.9600
Configured using:
 `configure --prefix=/c/usr --host=i686-pc-mingw32'

Important settings:
  value of $LANG: ENG
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  which-function-mode: t
  diff-auto-refine-mode: t
  global-highlight-parentheses-mode: t
  highlight-parentheses-mode: t
  show-paren-mode: t
  dtrt-indent-mode: t
  iswitchb-mode: t
  shell-dirtrack-mode: t
  winner-mode: t
  global-ede-mode: t
  global-semantic-mru-bookmark-mode: t
  global-semanticdb-minor-mode: t
  global-semantic-decoration-mode: t
  global-semantic-idle-completions-mode: t
  global-semantic-idle-scheduler-mode: t
  global-semantic-idle-summary-mode: t
  global-semantic-highlight-func-mode: t
  global-semantic-stickyfunc-mode: t
  semantic-mode: t
  delete-selection-mode: t
  cua-mode: t
  global-auto-revert-mode: t
  desktop-save-mode: t
  savehist-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Type "q" in help window to restore its previous buffer.

Quit [2 times]
Mark set [2 times]
""
Mark set
Quit
Mark saved where search started
Type "q" in help window to restore its previous buffer.

""

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug conf-mode nxml-uchnm rng-xsd xsd-regexp
rng-cmpct nxml-mode-expansions html-mode-expansions sgml-mode rng-nxml
rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util
rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-glyph
nxml-enc xmltok browse-url url-util cua-rect rect expand-region
text-mode-expansions cc-mode-expansions python-el-fgallina-expansions
er-basic-expansions expand-region-core expand-region-custom
semantic/tag-write semantic/edit debug misearch multi-isearch
semantic/db-typecache semantic/imenu semantic/sb ede/dired vc-hg xhg-dvc
xhg xhg-annotate xhg-mq xhg-log vc-svn eldoc which-func imenu
semantic/tag-file semantic/db-file data-debug cedet-files tla hl-line
smerge-mode dvc-state dvc-config dvc-diff dvc-fileinfo diff tla-gnus
dvc-gnus gnus-art mm-uu mml2015 mm-view mml-smime smime dig mailcap
gnus-sum nnoo gnus-group gnus-undo nnmail mail-source gnus-start
gnus-spec gnus-int gnus-range message rfc822 mml mml-sec mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
gnus-win gnus gnus-ems nnheader dvc-cmenu dvc-about dvc-version
dvc-revlist semantic/wisent/python semantic/decorate/include warnings
semantic/dep semantic/wisent/python-wy semantic/wisent
semantic/wisent/wisent rx superword subword python json ede/cpp-root
ede/emacs bzr-core tla-core pp tla-autoconf tla-defs dvc-log vc
vc-dispatcher add-log diff-mode ediff-merg ediff-wind ediff-diff
ediff-mult ediff-help ediff-init ediff-util ediff xdarcs-core xgit-core
xhg-core xmtn-minimal kalasm-mode qmake-mode bb-mode dts-mode php-mode
etags cc-langs generic generic-x cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs dvc-autoloads
dvc-unified dvc-tips dired-x dvc-core edebug dvc-lisp dvc-buffers
derived dvc-ui dvc-register dvc-utils dvc-emacs ewoc dvc-defs dvc-site
highlight-parentheses paren dtrt-indent iswitchb tramp tramp-compat
tramp-loaddefs trampver shell pcomplete format-spec icomplete+ icomplete
icicles icicles-mode easy-mmode dired icicles-cmd cus-edit cus-start
cus-load icicles-mcmd icicles-mac icicles-fn icicles-face icicles-var
icicles-opt ffap url-parse auth-source gnus-util password-cache url-vars
color-theme sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils reporter thingatpt hexrgb two-column cl-macs cl winner
windmove framemove findstr grep compile comint ansi-color ede/speedbar
ede/files ede ede/base ede/auto ede/source eieio-speedbar eieio-custom
wid-edit semantic/mru-bookmark semantic/db-mode semantic/decorate/mode
semantic/complete semantic/idle eieio-opt help-mode speedbar sb-image
dframe find-func semantic/ia semantic/analyze/refs semantic/db-find
semantic/db-ref semantic/senator semantic/decorate ring pulse
semantic/analyze semantic/sort semantic/scope semantic/analyze/fcn
semantic/db gv eieio-base semantic/ctxt semantic/format ezimage
semantic/tag-ls semantic/find semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw eieio byte-opt bytecomp
byte-compile cl-extra cconv eieio-core mode-local cedet edmacro kmacro
advice help-fns delsel cua-base saveplace autorevert filenotify desktop
frameset cl-loaddefs cl-lib savehist server info easymenu package
epg-config time-date tooltip electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 ls-lisp w32-common-fns disp-table w32-win
w32-vars tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer nadvice loaddefs button faces cus-face macroexp
files text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
w32notify w32 multi-tty emacs)

Memory information:
((conses 8 649133 135398)
 (symbols 32 52813 0)
 (miscs 32 11322 2303)
 (strings 16 127824 18777)
 (string-bytes 1 3792793)
 (vectors 8 54846)
 (vector-slots 4 1586068 18114)
 (floats 8 526 1355)
 (intervals 28 5955 102)
 (buffers 508 40))

Message #8 received at 23095 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Will Wykeham <will <at> wykeham.net>
Cc: 23095 <at> debbugs.gnu.org
Subject: Re: bug#23095: 24.5;
 python-shell-unbuffered causes problems on Windows
Date: Wed, 23 Mar 2016 17:46:28 +0200

> Date: Tue, 22 Mar 2016 15:01:27 +0000
> From: Will Wykeham <will <at> wykeham.net>
> 
> (let ((process-environment (python-shell-calculate-process-environment))
>        (code (concat "import sys\n"
>                     "print(sys.ps1)")))
>   (with-temp-buffer
>     (let ((code-file (python-shell--save-temp-file code)))
>       (call-process "python" code-file '(t nil) nil "-i")
>       (delete-file code-file))
>     (buffer-string)))
> 
> In an 'out of the box' emacs, with 'python-shell-unbuffered' set to t as
> default, this evaluates to an empty string.
> 
> If python-shell-unbuffered is then set to nil, then the above evaluates to
> the expected ">>>\n".
> 
> I haven't got to the bottom of why unbuffered is causing this
> problem. My suspicion is that it relates to the fact that
> PYTHONUNBUFFERED on Windows also causes python to process stdin/stdout
> as binary (which also affects line ending processing).

On Windows, Emacs communicates with subprocesses via pipes, which are
buffered.

Message #11 received at 23095 <at> debbugs.gnu.org (full text, mbox):

From: Will Wykeham <will <at> wykeham.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 23095 <at> debbugs.gnu.org
Subject: Re: bug#23095: 24.5;
 python-shell-unbuffered causes problems on Windows
Date: Wed, 23 Mar 2016 16:05:38 +0000

On 23 March 2016 at 15:46, Eli Zaretskii <eliz <at> gnu.org> wrote:
>> Date: Tue, 22 Mar 2016 15:01:27 +0000
>> From: Will Wykeham <will <at> wykeham.net>
>>
>> (let ((process-environment (python-shell-calculate-process-environment))
>>        (code (concat "import sys\n"
>>                     "print(sys.ps1)")))
>>   (with-temp-buffer
>>     (let ((code-file (python-shell--save-temp-file code)))
>>       (call-process "python" code-file '(t nil) nil "-i")
>>       (delete-file code-file))
>>     (buffer-string)))
>>
>> In an 'out of the box' emacs, with 'python-shell-unbuffered' set to t as
>> default, this evaluates to an empty string.
>>
>> If python-shell-unbuffered is then set to nil, then the above evaluates to
>> the expected ">>>\n".
>>
>> I haven't got to the bottom of why unbuffered is causing this
>> problem. My suspicion is that it relates to the fact that
>> PYTHONUNBUFFERED on Windows also causes python to process stdin/stdout
>> as binary (which also affects line ending processing).
>
> On Windows, Emacs communicates with subprocesses via pipes, which are
> buffered.

Ah. In which case, it sounds like the appropriate thing would be to
set the default value of 'python-shell-unbuffered' to nil on Windows.

Will

Message #14 received at 23095 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> users.sourceforge.net>
To: Will Wykeham <will <at> wykeham.net>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 23095 <at> debbugs.gnu.org
Subject: Re: bug#23095: 24.5;
 python-shell-unbuffered causes problems on Windows
Date: Sun, 03 Dec 2017 00:43:38 -0500

close 23095 26.1
unarchive 21376
merge 23095 21376
quit

Will Wykeham <will <at> wykeham.net> writes:

>>> I haven't got to the bottom of why unbuffered is causing this
>>> problem. My suspicion is that it relates to the fact that
>>> PYTHONUNBUFFERED on Windows also causes python to process stdin/stdout
>>> as binary (which also affects line ending processing).
>>
>> On Windows, Emacs communicates with subprocesses via pipes, which are
>> buffered.
>
> Ah. In which case, it sounds like the appropriate thing would be to
> set the default value of 'python-shell-unbuffered' to nil on Windows.

So it turns out this is actually some weird bug in w32 Python 2.x, we
work around it in Emacs 26 now.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21376#11

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.