GNU bug report logs - #23387
25.0.93; Unicode quote inserted in Python mode outside of strings

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 23387 in the body.
You can then email your comments to 23387 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Philipp Stephani <p.stephani2 <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 25.0.93; Unicode quote inserted in Python mode outside of strings
Date: Wed, 27 Apr 2016 16:14:57 +0200

emacs -Q
M-x electric-quote-mode
M-x customize-variable RET electric-quote-string, set it to t, save
M-x python-mode
Hit '

Expected: ' (ASCII apostrophe) is inserted because point is outside a string
Actual: ’ (Unicode quote) is inserted

This seems to happen only in Python mode, not in e.g. emacs-lisp-mode.



In GNU Emacs 25.0.93.5 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
 of 2016-04-24 built on localhost
Repository revision: 0cd2e923dba8d8c7128b0c084ce6af22069e8db5
Windowing system distributor 'The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04 LTS

Configured using:
 'configure --with-modules
 LDFLAGS=-Wl,-fuse-ld=gold,--export-dynamic-symbol=__google_auxv'

Configured features:
XPM JPEG TIFF GIF PNG SOUND GSETTINGS NOTIFY FREETYPE XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-quote-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Electric-Quote mode enabled
Quit
Creating customization items...
Creating customization items ...done
Resetting customization items...done
Creating customization setup...done
To install your edits, invoke [State] and choose the Set operation
Source file ‘/usr/local/google/home/phst/ThirdParty/Emacs/lisp/progmodes/python.el’ newer than byte-compiled file
Can’t guess python-indent-offset, using defaults: 4
Source file ‘/usr/local/google/home/phst/ThirdParty/Emacs/lisp/dired.el’ newer than byte-compiled file

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired rfc822 mml mml-sec epg
epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils
python tramp-sh tramp tramp-compat auth-source cl-seq eieio eieio-core
cl-macs gnus-util mm-util mail-prsvr password-cache tramp-loaddefs
trampver ucs-normalize shell pcomplete format-spec advice json map seq
byte-opt gv bytecomp byte-compile cl-extra cconv comint ring ansi-color
cus-edit wid-edit thingatpt help-fns help-mode easymenu cl-loaddefs
pcase cl-lib cus-start cus-load time-date mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 239874 9745)
 (symbols 48 24466 0)
 (miscs 40 400 230)
 (strings 32 29068 7264)
 (string-bytes 1 922223)
 (vectors 16 39237)
 (vector-slots 8 730263 4252)
 (floats 8 240 60)
 (intervals 56 261 0)
 (buffers 976 14)
 (heap 1024 32672 1237))

-- 
Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

Diese E-Mail ist vertraulich.  Wenn Sie nicht der richtige Adressat sind,
leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen
Sie die E-Mail und alle Anhänge.  Vielen Dank.

This e-mail is confidential.  If you are not the right addressee please do not
forward it, please inform the sender, and please erase this e-mail including
any attachments.  Thanks.

Message #8 received at 23387 <at> debbugs.gnu.org (full text, mbox):

From: Philipp Stephani <p.stephani2 <at> gmail.com>
To: 23387 <at> debbugs.gnu.org
Subject: Re: bug#23387: 25.0.93; Unicode quote inserted in Python mode outside
 of strings
Date: Thu, 28 Apr 2016 17:11:53 +0000

[Message part 1 (text/plain, inline)]

Philipp Stephani <p.stephani2 <at> gmail.com> schrieb am Mi., 27. Apr. 2016 um
16:16 Uhr:

>
> emacs -Q
> M-x electric-quote-mode
> M-x customize-variable RET electric-quote-string, set it to t, save
> M-x python-mode
> Hit '
>
> Expected: ' (ASCII apostrophe) is inserted because point is outside a
> string
> Actual: ’ (Unicode quote) is inserted
>
> This seems to happen only in Python mode, not in e.g. emacs-lisp-mode.
>
> Also happens in C++ mode, and probably others.

[Message part 2 (text/html, inline)]

Message #11 received at 23387 <at> debbugs.gnu.org (full text, mbox):

From: Philipp Stephani <p.stephani2 <at> gmail.com>
To: 23387 <at> debbugs.gnu.org
Subject: Re: bug#23387: 25.0.93; Unicode quote inserted in Python mode outside
 of strings
Date: Thu, 28 Apr 2016 17:22:06 +0000

[Message part 1 (text/plain, inline)]

Philipp Stephani <p.stephani2 <at> gmail.com> schrieb am Do., 28. Apr. 2016 um
19:11 Uhr:

> Philipp Stephani <p.stephani2 <at> gmail.com> schrieb am Mi., 27. Apr. 2016 um
> 16:16 Uhr:
>
>>
>> emacs -Q
>> M-x electric-quote-mode
>> M-x customize-variable RET electric-quote-string, set it to t, save
>> M-x python-mode
>> Hit '
>>
>> Expected: ' (ASCII apostrophe) is inserted because point is outside a
>> string
>> Actual: ’ (Unicode quote) is inserted
>>
>> This seems to happen only in Python mode, not in e.g. emacs-lisp-mode.
>>
>> Also happens in C++ mode, and probably others.
>

The root cause is that electric-quote-post-self-insert-function uses
syntax-ppss to check whether point is in a string or comment. Before it
makes any replacement, the ASCII apostrophe is still in the buffer, so
syntax-ppss considers point to be inside a string. It looks like calling
syntax-ppss with (forward-point -1) as argument instead of (point) should
fix it, but I'm not sure whether that's the right way.

[Message part 2 (text/html, inline)]

Message #14 received at 23387 <at> debbugs.gnu.org (full text, mbox):

From: Philipp Stephani <p.stephani2 <at> gmail.com>
To: 23387 <at> debbugs.gnu.org
Subject: Re: bug#23387: 25.0.93; Unicode quote inserted in Python mode outside
 of strings
Date: Sat, 30 Apr 2016 11:59:16 +0000

[Message part 1 (text/plain, inline)]

Philipp Stephani <p.stephani2 <at> gmail.com> schrieb am Do., 28. Apr. 2016 um
19:22 Uhr:

> Philipp Stephani <p.stephani2 <at> gmail.com> schrieb am Do., 28. Apr. 2016 um
> 19:11 Uhr:
>
>> Philipp Stephani <p.stephani2 <at> gmail.com> schrieb am Mi., 27. Apr. 2016
>> um 16:16 Uhr:
>>
>>>
>>> emacs -Q
>>> M-x electric-quote-mode
>>> M-x customize-variable RET electric-quote-string, set it to t, save
>>> M-x python-mode
>>> Hit '
>>>
>>> Expected: ' (ASCII apostrophe) is inserted because point is outside a
>>> string
>>> Actual: ’ (Unicode quote) is inserted
>>>
>>> This seems to happen only in Python mode, not in e.g. emacs-lisp-mode.
>>>
>>> Also happens in C++ mode, and probably others.
>>
>
> The root cause is that electric-quote-post-self-insert-function uses
> syntax-ppss to check whether point is in a string or comment. Before it
> makes any replacement, the ASCII apostrophe is still in the buffer, so
> syntax-ppss considers point to be inside a string. It looks like calling
> syntax-ppss with (forward-point -1) as argument instead of (point) should
> fix it, but I'm not sure whether that's the right way.
>

I've attached a patch.

[Message part 2 (text/html, inline)]

[0001-Fix-insertion-of-Unicode-quotes-in-strings.patch (application/octet-stream, attachment)]

Message #19 received at 23387-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Philipp Stephani <p.stephani2 <at> gmail.com>
Cc: 23387-done <at> debbugs.gnu.org
Subject: Re: 25.0.93; Unicode quote inserted in Python mode outside of strings
Date: Mon, 2 May 2016 09:04:18 -0700

[Message part 1 (text/plain, inline)]

Thanks for the bug report and patch, but I ran into a problem with the 
patch. Suppose I do this:

emacs -Q
M-x electric-quote-mode
M-x customize-variable RET electric-quote-string, set it to t, save
M-x python-mode
'abc'

That is, I type apostrophe-a-b-c-apostrophe in Python mode. With the 
patch, the buffer will contain:

'abc’

The opening apostrophe is OK, but the closing apostrophe is considered 
to be inside a string so it is transmogrified into a right single 
quotation mark. A Python programmer would want both to be apostrophes.

I installed the attached patches into emacs-25 instead, as they should 
address this problem too. Please give it a try.

[0001-Don-t-electrically-quote-in-Python.patch (application/x-patch, attachment)]

[0002-Add-electric-quote-string-unit-test.patch (application/x-patch, attachment)]

Message #22 received at 23387 <at> debbugs.gnu.org (full text, mbox):

From: Philipp Stephani <p.stephani2 <at> gmail.com>
To: 23387 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Cc: 23387-done <at> debbugs.gnu.org
Subject: Re: bug#23387: 25.0.93; Unicode quote inserted in Python mode outside
 of strings
Date: Tue, 03 May 2016 20:44:27 +0000

[Message part 1 (text/plain, inline)]

Paul Eggert <eggert <at> cs.ucla.edu> schrieb am Mo., 2. Mai 2016 um 18:05 Uhr:

> Thanks for the bug report and patch, but I ran into a problem with the
> patch. Suppose I do this:
>
> emacs -Q
> M-x electric-quote-mode
> M-x customize-variable RET electric-quote-string, set it to t, save
> M-x python-mode
> 'abc'
>
> That is, I type apostrophe-a-b-c-apostrophe in Python mode. With the
> patch, the buffer will contain:
>
> 'abc’
>
> The opening apostrophe is OK, but the closing apostrophe is considered
> to be inside a string so it is transmogrified into a right single
> quotation mark. A Python programmer would want both to be apostrophes.
>

I'm not sure, I think if a user has electric-quote-string enabled, that's
the behavior they want, because it's inside a string. However, I can see
that both approaches are valid. There is probably no good way that would
work in all circumstances.

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.