Subject: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

From: Da Zhang <[email-masked]>
Date: Fri, 05 Jun 2015 12:33:59 -0400

From: Da Zhang <[email-masked]>
To: [email-masked]
Subject: 24.5; shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly
--text follows this line--

Dear Emacs developers/maintainers,

I am writing this email to report an issue that has been bothering me
for a while.

I use python mode (python.el) for some python programming. When I invoke
the command python-shell-send-buffer, my Emacs always reports "Searching
for program: no such file or directory,
c\:/Anaconda3/Scripts/ipython.exe".

I traced down the cause of this problem, and found the following lines
in python.el:
(defun python-shell-parse-command ()    ;FIXME: why name it "parse"?
"Calculate the string used to execute the inferior Python process."
;; FIXME: process-environment doesn't seem to be used anywhere within
;; this let.
(let ((process-environment (python-shell-calculate-process-environment))
(exec-path (python-shell-calculate-exec-path)))
(format "%s %s"
;; FIXME: Why executable-find?
(shell-quote-argument (executable-find python-shell-interpreter))
python-shell-interpreter-args)))

The function call
(shell-quote-argument (executable-find python-shell-interpreter))
changed the path of my python interpreter from
"c:/Anaconda3/Scripts/ipython.exe"
to
"c\\:/Anaconda3/Scripts/ipython.exe".

I worked around this problem by replacing the function call
(shell-quote-argument (executable-find python-shell-interpreter))
with
(executable-find python-shell-interpreter) ;; edited by DZ on
2015/06/05: (shell-quote-argument) caused the problem of
"c\\:/Anaconda3/Scripts/ipython.exe"

To further analyze the problem, I found the function
shell-quote-argument defined in subr.el does not deal with the case when
bash from cygwin is used.

(defun shell-quote-argument (argument)
"Quote ARGUMENT for passing as argument to an inferior shell."
(cond
((and (eq system-type 'windows-nt) (w32-shell-dos-semantics))
;; First, quote argument so that CommandLineToArgvW will
;; understand it.  See
;; http://msdn.microsoft.com/en-us/library/17w5ykft%28v=vs.85%29.aspx
;; After we perform that level of quoting, escape shell
;; metacharacters so that cmd won't mangle our argument.  If the
;; argument contains no double quote characters, we can just
;; surround it with double quotes.  Otherwise, we need to prefix
;; each shell metacharacter with a caret.

(setq argument
;; escape backslashes at end of string
(replace-regexp-in-string
"\\(\\\\*\\)$"
"\\1\\1"
;; escape backslashes and quotes in string body
(replace-regexp-in-string
"\\(\\\\*\\)\""
"\\1\\1\\\\\""
argument)))

(if (string-match "[%!\"]" argument)
(concat
"^\""
(replace-regexp-in-string
"\\([%!()\"<>&|^]\\)"
"^\\1"
argument)
"^\"")
(concat "\"" argument "\"")))
(t
(if (equal argument "")
"''"
;; Quote everything except POSIX filename characters.
;; This should be safe enough even for really weird shells.
(replace-regexp-in-string
"\n" "'\n'"
(replace-regexp-in-string "[^-0-9a-zA-Z_./\n]" "\\\\\\&" argument))))
))

I added the following lines in the cond statement, to solve the problem.
((and (eq system-type 'windows-nt) (if (string-match ".*cygwin.*" (w32-shell-name)) t nil)) ;; DZ's edit on 6/5/2015
(if (equal argument "")
"''"
;; Quote everything except POSIX filename characters.
;; This should be safe enough even for really weird shells.
(replace-regexp-in-string
"\n" "'\n'"
(replace-regexp-in-string "[^-0-9a-zA-Z_./\n:]" "\\\\\\&" argument))))
((eq system-type 'ms-dos)
;; Quote using double quotes, but escape any existing quotes in
;; the argument with backslashes.
(let ((result "")
(start 0)
end)
(if (or (null (string-match "[^\"]" argument))
(< (match-end 0) (length argument)))
(while (string-match "[\"]" argument start)
(setq end (match-beginning 0)
result (concat result (substring argument start end)
"\\" (substring argument end (1+ end)))
start (1+ end))))
(concat "\"" result (substring argument start) "\"")))

However, the edits does not change the behavior of subr.el.

I worked with Eli Zaretskii on similar problem before, and here I report a more thorough analysis in the hope some experts more capable than me could help to solve this problem.

Thanks.

Da Zhang

In GNU Emacs 24.5.1 (x86_64-w64-mingw32)
of 2015-05-16 on KAEL
Windowing system distributor `Microsoft Corp.', version 6.3.9600
Configured using:
`configure --prefix=/z/emacs --host=x86_64-w64-mingw32
--target=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --with-wide-int
--with-jpeg --with-xpm --with-png --with-tiff --with-rsvg --with-xml2
--with-gnutls --with-sound=yes --with-file-notification=yes
--without-dbus --without-imagemagick 'CFLAGS=-O3 -fomit-frame-pointer
-g0 -pipe' 'LDFLAGS=-static-libgcc -static-libstdc++ -static -s
-Wl,-s''

Important settings:
value of $LANG: ENU
locale-coding-system: utf-8

Major mode: Emacs-Lisp

Minor modes in effect:
guide-key-mode: t
recentf-mode: t
winner-mode: t
cua-mode: t
global-company-mode: t
company-mode: t
rainbow-delimiters-mode: t
sml-modeline-mode: t
global-undo-tree-mode: t
undo-tree-mode: t
anything-dired-mode: Enable anything completion in Dired functions.
Bindings affected are C, R, S, H.
This is deprecated for Emacs24+ users, use `ac-mode' instead.
ido-vertical-mode: t
ido-everywhere: t
tabbar-mwheel-mode: t
tabbar-mode: t
shell-dirtrack-mode: t
desktop-save-mode: t
TeX-PDF-mode: t
global-auto-revert-mode: t
display-time-mode: t
auto-image-file-mode: t
show-paren-mode: t
which-function-mode: t
delete-selection-mode: t
global-hl-line-mode: t
diff-auto-refine-mode: t
tooltip-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
column-number-mode: t
line-number-mode: t
transient-mark-mode: t
abbrev-mode: t

Recent messages:
Starting new Ispell process aspell with default dictionary...
Fontifying README.W32... (regexps...........................................)
Starting new Ispell process aspell with default dictionary...
Wrote c:/home/zhangda/.emacs.d/.emacs.desktop.lock
Desktop: 1 frame, 5 buffers restored.
(Shell command succeeded with no output)
Desktop: 1 frame, 5 buffers restored.
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark saved where search started [2 times]
Mark activated

Load-path shadows:
c:/home/zhangda/.emacs.d/elpa/anything-20130606.946/anything-complete hides c:/home/zhangda/.emacs.d/elpa/anything-complete-1.86/anything-complete
c:/home/zhangda/.emacs.d/elpa/anything-20130606.946/anything-match-plugin hides c:/home/zhangda/.emacs.d/elpa/anything-match-plugin-1.27/anything-match-plugin

Features:
(shadow sort mail-extr warnings emacsbug sendmail misearch multi-isearch
swiper ivy darkroom eim eim-extra cdlatex texmathp flyspell ispell
org-rmail org-mhe org-irc org-info org-gnus org-docview doc-view
jka-compr image-mode org-bibtex bibtex org-bbdb org-w3m indent-guide
python-cell outline-magic auto-complete popup highlight-indentation
sphinx-doc guide-key popwin company-files company-oddmuse
company-keywords company-etags company-gtags company-dabbrev-code
company-dabbrev company-capf company-cmake company-xcode company-clang
company-semantic company-eclim company-template company-css company-nxml
company-bbdb vc-hg recentf tree-widget helm-files helm-buffers
helm-elscreen helm-tags helm-bookmark helm-adaptive helm-info bookmark
pp helm-locate helm-help helm-match-plugin helm-grep helm-regexp
helm-plugin helm-external helm-net helm-utils helm helm-source f s
ucs-normalize ace-jump-zap ace-jump-mode winner smartparens-config
smartparens-python smartparens-latex smartparens dash cua-base
visual-regexp eldoc company elpy pyvenv elpy-refactor files-x etags
flymake-cursor flymake-python-pyflakes flymake-easy
python-el-fgallina-expansions python json iedit iedit-lib smex
rainbow-delimiters re-builder matlab-load rainbow-mode google-c-style
fbib sml-modeline framemove windmove weblogger xml-rpc timezone url-http
tls url-auth url-gw expand-region text-mode-expansions
cc-mode-expansions the-org-mode-expansions latex-mode-expansions
er-basic-expansions expand-region-core expand-region-custom undo-tree
diff anything-config browse-url rx anything-match-plugin xml url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap tramp tramp-compat tramp-loaddefs trampver
ffap url-parse auth-source eieio eieio-core gnus-util password-cache
url-vars anything bm find-dired help-mode dired+ image-dired dired-x
dired-aux header2 color-theme-dawn-night color-theme ido-vertical-mode
ido browse-kill-ring tabbar ibuf-macs ibuffer server org-bullets
ox-beamer ox-latex ox-icalendar ox-html ox-ascii ox-publish ox
org-element ob-makefile ob-sh shell ob-R ob-python ob-perl ob-org
ob-matlab ob-octave ob-latex ob-gnuplot ob-plantuml ob-ditaa ob-css
ob-calc calc-store calc-trail calc-ext calc calc-loaddefs calc-macs ob-C
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs org org-macro org-footnote org-pcomplete pcomplete
org-list org-faces org-entities noutline outline org-version
ob-emacs-lisp ob ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys
ob-comint ob-core ob-eval org-compat org-macs org-loaddefs find-func
cal-menu calendar cal-loaddefs muse-wiki muse-colors muse-docbook
muse-texinfo texnfo-upd texinfo muse-latex muse-html muse-xml-common
cus-edit cus-start cus-load muse-publish muse-project muse-protocols
muse-regexps wid-edit derived muse muse-nested-tags muse-mode flymake
tex-mik preview prv-emacs reporter desktop frameset latex tex-style
tex-buf tex crm cl-macs autorevert filenotify time image-file avoid
paren which-func imenu delsel col-highlight vline highlight-symbol
thingatpt highlight-parentheses hl-line byte-opt bytecomp byte-compile
cl-extra cconv hlinum linum edmacro kmacro w32-browser htmlize-view
htmlize cl gv cl-loaddefs cl-lib ahg vc-annotate vc vc-dispatcher dired
grep compile ewoc log-edit message format-spec rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log diff-mode easy-mmode cygwin-mount ange-ftp comint
ansi-color ring anything-autoloads anything-complete-autoloads
anything-match-plugin-autoloads color-file-completion-autoloads
color-theme-autoloads tex-site csv-mode-autoloads etags-table-autoloads
flymake-shell-autoloads gnuplot-mode-autoloads hide-lines-autoloads
htmlize-autoloads http-post-simple-autoloads less-autoloads
muse-autoloads py-import-check-autoloads pydoc-info advice help-fns
info-look rainbow-mode-autoloads info easymenu sml-mode-autoloads
vline-autoloads weblogger-autoloads package epg-config time-date tooltip
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32
ls-lisp w32-common-fns disp-table w32-win w32-vars tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment lisp-mode prog-mode
register page menu-bar rfn-eshadow timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
w32notify w32 multi-tty emacs)

Memory information:
((conses 16 670478 66958)
(symbols 56 63650 0)
(miscs 48 489 281)
(strings 32 157872 41458)
(string-bytes 1 4522676)
(vectors 16 75479)
(vector-slots 8 1116100 22330)
(floats 8 400 929)
(intervals 56 3470 0)
(buffers 960 18))