GNU bug report logs - #20744
python.el tries to undo shell quoting with split-string-and-unquote

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 20744 in the body.
You can then email your comments to 20744 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Da Zhang <zhangda82 <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot
 figure out path correctly
Date: Fri, 05 Jun 2015 12:33:59 -0400

[Message part 1 (text/html, inline)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Da Zhang <zhangda82 <at> gmail.com>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot
 figure out path correctly
Date: Fri, 05 Jun 2015 23:13:18 +0300

> Date: Fri, 05 Jun 2015 12:33:59 -0400
> From: Da Zhang <zhangda82 <at> gmail.com>
> 
> I use python mode (python.el) for some python programming. When I invoke
> the command python-shell-send-buffer, my Emacs always reports "Searching
> for program: no such file or directory,
> c\:/Anaconda3/Scripts/ipython.exe".
> 
> I traced down the cause of this problem, and found the following lines
> in python.el:
> (defun python-shell-parse-command () ;FIXME: why name it "parse"?
> "Calculate the string used to execute the inferior Python process."
> ;; FIXME: process-environment doesn't seem to be used anywhere within
> ;; this let.
> (let ((process-environment (python-shell-calculate-process-environment))
> (exec-path (python-shell-calculate-exec-path)))
> (format "%s %s"
> ;; FIXME: Why executable-find?
> (shell-quote-argument (executable-find python-shell-interpreter))
> python-shell-interpreter-args)))
> 
> The function call 
> (shell-quote-argument (executable-find python-shell-interpreter))
> changed the path of my python interpreter from
> "c:/Anaconda3/Scripts/ipython.exe"
> to
> "c\\:/Anaconda3/Scripts/ipython.exe".
> 
> I worked around this problem by replacing the function call
> (shell-quote-argument (executable-find python-shell-interpreter))
> with
> (executable-find python-shell-interpreter) ;; edited by DZ on
> 2015/06/05: (shell-quote-argument) caused the problem of
> "c\\:/Anaconda3/Scripts/ipython.exe"
> 
> To further analyze the problem, I found the function
> shell-quote-argument defined in subr.el does not deal with the case when
> bash from cygwin is used.

For the record, this is bug #20237.

> I added the following lines in the cond statement, to solve the problem.
> ((and (eq system-type 'windows-nt) (if (string-match ".*cygwin.*"
> (w32-shell-name)) t nil)) ;; DZ's edit on 6/5/2015
> (if (equal argument "")
> "''"
> ;; Quote everything except POSIX filename characters.
> ;; This should be safe enough even for really weird shells.
> (replace-regexp-in-string
> "\n" "'\n'"
> (replace-regexp-in-string "[^-0-9a-zA-Z_./\n:]" "\\\\\\&" argument))))

I'm sorry, but I don't see how we can accept such a change: it means
that when the Cygwin Bash is used as the shell, the ':' character will
not be quoted, which I think could cause trouble in some cases.

Maybe some Bash expert could chime in and tell whether problems are
possible with Bash in this case.

Is there any way you could avoid using the Cygwin Bash as the shell
when running the native Windows build of Emacs?  That would make the
problem go away without any need for changes.

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Da Zhang <zhangda82 <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot
 figure out path correctly
Date: Fri, 05 Jun 2015 16:50:46 -0400

[Message part 1 (text/html, inline)]

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Da Zhang <zhangda82 <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot
 figure out path correctly
Date: Fri, 05 Jun 2015 17:05:37 -0400

[Message part 1 (text/html, inline)]

[gaefedad.png (image/png, inline)]

Message #17 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 20744 <at> debbugs.gnu.org, Da Zhang <zhangda82 <at> gmail.com>
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Sun, 07 Jun 2015 14:01:31 -0400

Eli Zaretskii wrote:

> I'm sorry, but I don't see how we can accept such a change: it means
> that when the Cygwin Bash is used as the shell, the ':' character will
> not be quoted, which I think could cause trouble in some cases.
>
> Maybe some Bash expert could chime in and tell whether problems are
> possible with Bash in this case.

Not an expert (and didn't read the rest of the report), but I can't
think of a reason why ':' would need quoting in Bash (unlike eg ';' or '&').
What kind of thing are you thinking of?

Message #20 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Sun, 07 Jun 2015 22:11:49 +0300

> From: Glenn Morris <rgm <at> gnu.org>
> Cc: Da Zhang <zhangda82 <at> gmail.com>,  20744 <at> debbugs.gnu.org
> Date: Sun, 07 Jun 2015 14:01:31 -0400
> 
> Eli Zaretskii wrote:
> 
> > I'm sorry, but I don't see how we can accept such a change: it means
> > that when the Cygwin Bash is used as the shell, the ':' character will
> > not be quoted, which I think could cause trouble in some cases.
> >
> > Maybe some Bash expert could chime in and tell whether problems are
> > possible with Bash in this case.
> 
> Not an expert (and didn't read the rest of the report), but I can't
> think of a reason why ':' would need quoting in Bash (unlike eg ';' or '&').
> What kind of thing are you thinking of?

PATH-style lists, maybe.

Anyway, if we think ':' doesn't need to be quoted, how about removing
it from the Posix branch of shell-quote-argument?  Then we won't need
ugly hacks that try to detect Cygwin Bash used from a native Windows
Emacs.

Message #23 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Sun, 07 Jun 2015 20:07:10 -0400

Eli Zaretskii wrote:

> Anyway, if we think ':' doesn't need to be quoted, how about removing
> it from the Posix branch of shell-quote-argument?

The comment there says "This should be safe enough even for really weird
shells." That's a much broader question than just Bash! :)

But I thought of an example that affects Bash:

scp localfile remotehost:

If "localfile" contains ":", things will get confused.
So it's probably best to leave it as-is.

Message #26 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Sun, 07 Jun 2015 20:24:40 -0400

Glenn Morris wrote:

> scp localfile remotehost:

Well, that's not a shell issue, though. It's an scp issue.
So it depends exactly what shell-quote-argument is for...

Message #29 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Sun, 07 Jun 2015 20:47:09 -0400

I think the actual answer for Bash is the "metacharacters" list in the
Bash manual, plus the "!" character (for history), as per:

http://www.gnu.org/software/bash/manual/html_node/Quoting.html

That doesn't include ":", but then there's those "really weird shells",
whatever they may be...

Message #32 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Mon, 08 Jun 2015 05:43:25 +0300

> From: Glenn Morris <rgm <at> gnu.org>
> Cc: 20744 <at> debbugs.gnu.org,  zhangda82 <at> gmail.com
> Date: Sun, 07 Jun 2015 20:47:09 -0400
> 
> 
> I think the actual answer for Bash is the "metacharacters" list in the
> Bash manual, plus the "!" character (for history), as per:
> 
> http://www.gnu.org/software/bash/manual/html_node/Quoting.html
> 
> That doesn't include ":", but then there's those "really weird shells",
> whatever they may be...

Perhaps we could have a branch for Bash only.

Message #35 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Glenn Morris <rgm <at> gnu.org>, 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Mon, 08 Jun 2015 09:43:32 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> That doesn't include ":", but then there's those "really weird shells",
>> whatever they may be...
>
> Perhaps we could have a branch for Bash only.

Tramp would happily join that branch, for "really weird shells on a
remote host".

Message #38 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Glenn Morris <rgm <at> gnu.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Mon, 08 Jun 2015 10:03:44 -0400

>> Anyway, if we think ':' doesn't need to be quoted, how about removing
>> it from the Posix branch of shell-quote-argument?
> The comment there says "This should be safe enough even for really weird
> shells." That's a much broader question than just Bash! :)

Let's stop escaping the : and if/when someone comes up with a very weird
shell where this is a problem, we'll see what can be done.


        Stefan

Message #41 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: rgm <at> gnu.org, 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Mon, 08 Jun 2015 17:31:27 +0300

> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: Glenn Morris <rgm <at> gnu.org>,  20744 <at> debbugs.gnu.org,  zhangda82 <at> gmail.com
> Date: Mon, 08 Jun 2015 09:43:32 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> That doesn't include ":", but then there's those "really weird shells",
> >> whatever they may be...
> >
> > Perhaps we could have a branch for Bash only.
> 
> Tramp would happily join that branch, for "really weird shells on a
> remote host".

Meaning you are for such a change, or against it?  (Sorry, it wasn't
clear from what you said.)

Message #44 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rgm <at> gnu.org, 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Mon, 08 Jun 2015 17:14:33 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> >> That doesn't include ":", but then there's those "really weird shells",
>> >> whatever they may be...
>> >
>> > Perhaps we could have a branch for Bash only.
>> 
>> Tramp would happily join that branch, for "really weird shells on a
>> remote host".
>
> Meaning you are for such a change, or against it?  (Sorry, it wasn't
> clear from what you said.)

Bash is always a pain to handle, there are exceptions in Tramp's code
for that. But having a branch for this is not realistic for Tramp,
because Tramp doesn't distinguish remote shells, it only expects that
they speak a bourne-shell dialect.

My message was rather a sigh than a request.

Best regards, Michael.

Message #47 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: rgm <at> gnu.org, 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Mon, 08 Jun 2015 19:22:31 +0300

> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: rgm <at> gnu.org,  20744 <at> debbugs.gnu.org,  zhangda82 <at> gmail.com
> Date: Mon, 08 Jun 2015 17:14:33 +0200
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> >> That doesn't include ":", but then there's those "really weird shells",
> >> >> whatever they may be...
> >> >
> >> > Perhaps we could have a branch for Bash only.
> >> 
> >> Tramp would happily join that branch, for "really weird shells on a
> >> remote host".
> >
> > Meaning you are for such a change, or against it?  (Sorry, it wasn't
> > clear from what you said.)
> 
> Bash is always a pain to handle, there are exceptions in Tramp's code
> for that. But having a branch for this is not realistic for Tramp,
> because Tramp doesn't distinguish remote shells, it only expects that
> they speak a bourne-shell dialect.
> 
> My message was rather a sigh than a request.

How about Stefan's suggestion to stop quoting the colon and see if
anybody hollers?  Are you okay with that, or will you holler?

Message #50 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rgm <at> gnu.org, 20744 <at> debbugs.gnu.org, zhangda82 <at> gmail.com
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Mon, 08 Jun 2015 18:34:03 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> How about Stefan's suggestion to stop quoting the colon and see if
> anybody hollers?  Are you okay with that, or will you holler?

That would be OK. We'll hear it, when Tramp users start to holler.

Best regards, Michael.

Message #53 received at 20744 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: Da Zhang <zhangda82 <at> gmail.com>
Cc: 20744 <at> debbugs.gnu.org, eliz <at> gnu.org
Subject: Re: bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on
 Windows cannot figure out path correctly
Date: Tue, 09 Aug 2016 20:51:24 -0400

retitle 20744 python.el tries to undo shell quoting with split-string-and-unquote
tags 20744 confirmed
found 20744 25.1
quit

Da Zhang <zhangda82 <at> gmail.com> writes:

> From: Da Zhang <zhangda82 <at> gmail.com>
> To: bug-gnu-emacs <at> gnu.org
> Subject: 24.5; shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly
> --text follows this line--
>
> Dear Emacs developers/maintainers,
>
> I am writing this email to report an issue that has been bothering me
> for a while.
>
> I use python mode (python.el) for some python programming. When I invoke
> the command python-shell-send-buffer, my Emacs always reports "Searching
> for program: no such file or directory,
> c\:/Anaconda3/Scripts/ipython.exe".
>
[...]
>
> The function call 
> (shell-quote-argument (executable-find python-shell-interpreter))
> changed the path of my python interpreter from
> "c:/Anaconda3/Scripts/ipython.exe"
> to
> "c\\:/Anaconda3/Scripts/ipython.exe".
>
> I worked around this problem by replacing the function call
> (shell-quote-argument (executable-find python-shell-interpreter))
> with
> (executable-find python-shell-interpreter) ;; edited by DZ on
> 2015/06/05: (shell-quote-argument) caused the problem of
> "c\\:/Anaconda3/Scripts/ipython.exe"
>
> To further analyze the problem, I found the function
> shell-quote-argument defined in subr.el does not deal with the case when
> bash from cygwin is used.
>
[...]

Actually, bash, regardless of being on cygwin or not, should have no
problem with an escaped ":".  The actual problem here is that python.el
is using shell-quote-argument but then not using a shell.  The result of
python-shell-calculate-command (formerly known as
python-shell-parse-command) gets passed to python-shell-make-comint as
the CMD argument:

(defun python-shell-make-comint (cmd proc-name &optional show internal)
  [...]
          (let* ((cmdlist (split-string-and-unquote cmd))
                 (buffer (apply #'make-comint-in-buffer proc-name proc-buffer-name
                                interpreter nil args))

make-comint-in-buffer uses start-file-process, i.e., no shell is called.
So python.el is trying to undo shell-quote-argument with
split-string-and-unquote, this is the bug.

To see this on any system, compare

    C-u M-x run-python RET 'python' -i

this fails, while

    C-u M-x run-python RET "python" -i

this succeeds (assuming you have python in exec-path).

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.