GNU bug report logs -
#68559
[PATCH] Improve Python shell completion
Previous Next
Reported by: Liu Hui <liuhui1610 <at> gmail.com>
Date: Thu, 18 Jan 2024 04:50:01 UTC
Severity: wishlist
Tags: patch
Done: Eli Zaretskii <eliz <at> gnu.org>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 68559 in the body.
You can then email your comments to 68559 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 18 Jan 2024 04:50:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Liu Hui <liuhui1610 <at> gmail.com>:
New bug report received and forwarded. Copy sent to
bug-gnu-emacs <at> gnu.org.
(Thu, 18 Jan 2024 04:50:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi,
Currently python-shell-completion-at-point doesn't respect the
delimiter setting of readline completer and always split the text,
resulting in poor completions when completers that don't need word
splitting, e.g. jedi[1], is used. This patch fixes the problem.
Meanwhile, this patch adds an option 'python-shell-readline-completer'
to make Python shell provide better completion experience (e.g.
completing function parameters, dictionary keys) by default if users
are using vanilla Python interpreter with jedi or using IPython, where
the completion support for IPython has been enhanced (e.g. showing
type annotations and function signatures).
BTW, I think it may make sense to move Python snippets to a separate
file to make them easier to be maintained.
Thanks.
Best,
[1] https://jedi.readthedocs.io/en/latest/docs/usage.html#repl-completion
[0001-Improve-Python-shell-completion.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 18 Jan 2024 06:40:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> From: Liu Hui <liuhui1610 <at> gmail.com>
> Date: Thu, 18 Jan 2024 12:48:37 +0800
>
> Currently python-shell-completion-at-point doesn't respect the
> delimiter setting of readline completer and always split the text,
> resulting in poor completions when completers that don't need word
> splitting, e.g. jedi[1], is used. This patch fixes the problem.
>
> Meanwhile, this patch adds an option 'python-shell-readline-completer'
> to make Python shell provide better completion experience (e.g.
> completing function parameters, dictionary keys) by default if users
> are using vanilla Python interpreter with jedi or using IPython, where
> the completion support for IPython has been enhanced (e.g. showing
> type annotations and function signatures).
>
> BTW, I think it may make sense to move Python snippets to a separate
> file to make them easier to be maintained.
Thanks. Let's see what others think about this (I don't use Python
enough to have an opinion that matters).
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Sun, 21 Jan 2024 09:36:01 GMT)
Full text and
rfc822 format available.
Message #11 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii wrote:
>
> > From: Liu Hui <liuhui1610 <at> gmail.com>
> > Date: Thu, 18 Jan 2024 12:48:37 +0800
> >
> > Currently python-shell-completion-at-point doesn't respect the
> > delimiter setting of readline completer and always split the text,
> > resulting in poor completions when completers that don't need word
> > splitting, e.g. jedi[1], is used. This patch fixes the problem.
> >
> > Meanwhile, this patch adds an option 'python-shell-readline-completer'
> > to make Python shell provide better completion experience (e.g.
> > completing function parameters, dictionary keys) by default if users
> > are using vanilla Python interpreter with jedi or using IPython, where
> > the completion support for IPython has been enhanced (e.g. showing
> > type annotations and function signatures).
> >
> > BTW, I think it may make sense to move Python snippets to a separate
> > file to make them easier to be maintained.
>
> Thanks. Let's see what others think about this (I don't use Python
> enough to have an opinion that matters).
Thanks for the patch. I have tried it and found it to be a very
powerful enhancement.
Is it possible to allow completion of keyword arguments in a
multi-line function call like the following?
#+begin_src python
re.split(
'b',
'abc',
maxs
#+end_src
I am not sure if Python snippets should be separated. Do other
language supports keep them separate?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Tue, 23 Jan 2024 11:33:01 GMT)
Full text and
rfc822 format available.
Message #14 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
kobarity <kobarity <at> gmail.com> 于2024年1月21日周日 17:34写道：
> Is it possible to allow completion of keyword arguments in a
> multi-line function call like the following?
>
> #+begin_src python
> re.split(
> 'b',
> 'abc',
> maxs
> #+end_src
I have added experimental support in the attached patch, can you test
it please?
> I am not sure if Python snippets should be separated. Do other
> language supports keep them separate?
I think it allows to reduce code duplication (e.g. the IPython support
code existing in both python-shell-completion-setup-code and
python-shell-readline-ipython-setup-code) and make code more simple.
It seems other languages do not contain as many code snippets as
python.el.
[0001-Improve-Python-shell-completion.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Tue, 23 Jan 2024 14:16:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui wrote:
> kobarity <kobarity <at> gmail.com> 于2024年1月21日周日 17:34写道：
>
> > Is it possible to allow completion of keyword arguments in a
> > multi-line function call like the following?
> >
> > #+begin_src python
> > re.split(
> > 'b',
> > 'abc',
> > maxs
> > #+end_src
>
> I have added experimental support in the attached patch, can you test
> it please?
Thank you. I tried the new patch and confirmed that the Python Shell
buffer allows keyword argument completion on multi-line function
calls. However, my expectation is that it can be done in Python
buffer as well (after calling `python-shell-send-buffer'). Is this
also possible?
> > I am not sure if Python snippets should be separated. Do other
> > language supports keep them separate?
>
> I think it allows to reduce code duplication (e.g. the IPython support
> code existing in both python-shell-completion-setup-code and
> python-shell-readline-ipython-setup-code) and make code more simple.
> It seems other languages do not contain as many code snippets as
> python.el.
Sorry, I misunderstood that snippets meant skeletons. I agree that
python.el has long Python codes, which is not easy to maintain. Are
you proposing to separate the snippets into .py file? If it's
acceptable from the Emacs' source code management perspective, I think
that is a good approach.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 24 Jan 2024 10:08:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
kobarity <kobarity <at> gmail.com> 于2024年1月23日周二 22:15写道：
> Thank you. I tried the new patch and confirmed that the Python Shell
> buffer allows keyword argument completion on multi-line function
> calls. However, my expectation is that it can be done in Python
> buffer as well (after calling `python-shell-send-buffer'). Is this
> also possible?
Thank you for the feedback. I have updated the patch to add support in
Python buffer.
> > > I am not sure if Python snippets should be separated. Do other
> > > language supports keep them separate?
> >
> > I think it allows to reduce code duplication (e.g. the IPython support
> > code existing in both python-shell-completion-setup-code and
> > python-shell-readline-ipython-setup-code) and make code more simple.
> > It seems other languages do not contain as many code snippets as
> > python.el.
>
> Sorry, I misunderstood that snippets meant skeletons. I agree that
> python.el has long Python codes, which is not easy to maintain. Are
> you proposing to separate the snippets into .py file? If it's
> acceptable from the Emacs' source code management perspective, I think
> that is a good approach.
Yes, I'd like to at least put some flags and functions shared by
python-shell-completion-native-setup and python-*-setup-code in a
separate .py file. On the other hand, they are not changed frequently,
so it is not in urgent need.
[0001-Improve-Python-shell-completion-bug-68559.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 25 Jan 2024 15:40:01 GMT)
Full text and
rfc822 format available.
Message #23 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui wrote:
> kobarity <kobarity <at> gmail.com> 于2024年1月23日周二 22:15写道：
> > Thank you. I tried the new patch and confirmed that the Python Shell
> > buffer allows keyword argument completion on multi-line function
> > calls. However, my expectation is that it can be done in Python
> > buffer as well (after calling `python-shell-send-buffer'). Is this
> > also possible?
> Thank you for the feedback. I have updated the patch to add support in
> Python buffer.
Thank you. It's working very nice.
I apologize for reporting in the piecemeal fashion, but I have noticed
the following differences from the current completion.
1. Module names cannot be completed in Python buffer.
2. Completion is not working in comments.
I'm not sure if the completion should work in comments, but some
people might want it because the current python-mode allows it in both
Python buffer and Python Shell buffer.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 26 Jan 2024 10:13:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
kobarity <kobarity <at> gmail.com> 于2024年1月25日周四 23:38写道：
>
> Liu Hui wrote:
> > kobarity <kobarity <at> gmail.com> 于2024年1月23日周二 22:15写道：
> > > Thank you. I tried the new patch and confirmed that the Python Shell
> > > buffer allows keyword argument completion on multi-line function
> > > calls. However, my expectation is that it can be done in Python
> > > buffer as well (after calling `python-shell-send-buffer'). Is this
> > > also possible?
> > Thank you for the feedback. I have updated the patch to add support in
> > Python buffer.
>
> Thank you. It's working very nice.
>
> I apologize for reporting in the piecemeal fashion, but I have noticed
> the following differences from the current completion.
No worries. I appreciate your valuable feedback for improving the
quality of the patch.
> 1. Module names cannot be completed in Python buffer.
Can you elaborate? The completion of module names never worked for me
except using IPython with non-native completion mechanism.
Because module names cannot be completed using vanilla python
interpreter in the terminal, they are also not completed even with the
patch, unless jedi is available.
> 2. Completion is not working in comments.
>
> I'm not sure if the completion should work in comments, but some
> people might want it because the current python-mode allows it in both
> Python buffer and Python Shell buffer.
Thanks for pointing it out. The attached patch should now restore
previous basic completion behavior in comments.
[0001-Improve-Python-shell-completion-bug-68559.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Sun, 28 Jan 2024 13:23:02 GMT)
Full text and
rfc822 format available.
Message #29 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui wrote:
> kobarity <kobarity <at> gmail.com> 于2024年1月25日周四 23:38写道：
> > 1. Module names cannot be completed in Python buffer.
> Can you elaborate? The completion of module names never worked for me
> except using IPython with non-native completion mechanism.
>
> Because module names cannot be completed using vanilla python
> interpreter in the terminal, they are also not completed even with the
> patch, unless jedi is available.
Hmm, I must have done something wrong. I cannot reproduce the problem
of module name completion even with the previous patch.
As for the current python-mode, I was able to complete the module
names because I enabled Jedi in PYTHONSTARTUP. Sorry for the confusion.
I'm experiencing strange behavior regarding completion of import
statement in a block in Python buffer. If I try to type the following
lines and then try to complete it, it will fail.
#+begin_src python
try:
im
#+end_src
However, when I try to complete at the beginning of the second line:
#+begin_src python
try:
#+end_src
"import" keyword also appears as a candidate. If I cancel the
candidates and type "im" and try to complete it, it will succeed.
This behavior does not occur in Python Shell buffer.
Another thing I noticed is the multi-line import statement. If the
import statement is one-line, each items (IGNORECASE and MULTILINE in
the example below) can be completed.
#+begin_src python
from re import IGNORECASE, MULTILINE
#+end_src
However, they cannot be completed if the import statement spans
multi-line.
#+begin_src python
from re import (
IGN
#+end_src
This happens in both Python buffer and Python Shell buffer. Perhaps
this is a limitation of Jedi completer?
> > 2. Completion is not working in comments.
> >
> > I'm not sure if the completion should work in comments, but some
> > people might want it because the current python-mode allows it in both
> > Python buffer and Python Shell buffer.
>
> Thanks for pointing it out. The attached patch should now restore
> previous basic completion behavior in comments.
Thanks, I confirmed that basic completion (without keyword argument
completion) can be performed within comments.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Mon, 29 Jan 2024 13:16:02 GMT)
Full text and
rfc822 format available.
Message #32 received at 68559 <at> debbugs.gnu.org (full text, mbox):
kobarity wrote:
> Liu Hui wrote:
> > Thanks for pointing it out. The attached patch should now restore
> > previous basic completion behavior in comments.
>
> Thanks, I confirmed that basic completion (without keyword argument
> completion) can be performed within comments.
One additional comment regarding ERT. pyenv installs an ipython shim
to the shims directory, if there is at least one virtualenv providing
ipython. It can be found by `executable-find'. However, it is not
usable unless the virtualenv is activated. As a result, the
`python-shell-completion-at-point-ipython' ERT would fail unless a
virtualenv which provides ipython is activated.
This may be something pyenv users should take care of, but it is more
convenient for pyenv users like me to make the following changes.
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index ae23ff8ebe2..9452f136cfb 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4854,7 +4854,10 @@ python-shell-completion-at-point-ipython
"Check if Python shell completion works for IPython."
(let ((python-shell-interpreter "ipython")
(python-shell-interpreter-args "-i --simple-prompt"))
- (skip-unless (executable-find python-shell-interpreter))
+ (skip-unless
+ (and
+ (executable-find python-shell-interpreter)
+ (eql (call-process python-shell-interpreter nil nil nil "--version") 0)))
(python-tests-with-temp-buffer-with-shell
""
(python-shell-with-shell-buffer
Severity set to 'wishlist' from 'normal'
Request was from
Stefan Kangas <stefankangas <at> gmail.com>
to
control <at> debbugs.gnu.org.
(Tue, 30 Jan 2024 00:45:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 01 Feb 2024 09:53:02 GMT)
Full text and
rfc822 format available.
Message #37 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> Date: Mon, 29 Jan 2024 22:15:20 +0900
> From: kobarity <kobarity <at> gmail.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,
> 68559 <at> debbugs.gnu.org
>
> kobarity wrote:
> > Liu Hui wrote:
> > > Thanks for pointing it out. The attached patch should now restore
> > > previous basic completion behavior in comments.
> >
> > Thanks, I confirmed that basic completion (without keyword argument
> > completion) can be performed within comments.
>
> One additional comment regarding ERT. pyenv installs an ipython shim
> to the shims directory, if there is at least one virtualenv providing
> ipython. It can be found by `executable-find'. However, it is not
> usable unless the virtualenv is activated. As a result, the
> `python-shell-completion-at-point-ipython' ERT would fail unless a
> virtualenv which provides ipython is activated.
>
> This may be something pyenv users should take care of, but it is more
> convenient for pyenv users like me to make the following changes.
Do we have a consensus about this issue? Should I install the
original patch, or are you still discussing it, and an updated patch
will be posted soon?
Thanks.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 01 Feb 2024 14:41:02 GMT)
Full text and
rfc822 format available.
Message #40 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii wrote:
> > Date: Mon, 29 Jan 2024 22:15:20 +0900
> > From: kobarity <kobarity <at> gmail.com>
> > Cc: Eli Zaretskii <eliz <at> gnu.org>,
> > 68559 <at> debbugs.gnu.org
> >
> > kobarity wrote:
> > > Liu Hui wrote:
> > > > Thanks for pointing it out. The attached patch should now restore
> > > > previous basic completion behavior in comments.
> > >
> > > Thanks, I confirmed that basic completion (without keyword argument
> > > completion) can be performed within comments.
> >
> > One additional comment regarding ERT. pyenv installs an ipython shim
> > to the shims directory, if there is at least one virtualenv providing
> > ipython. It can be found by `executable-find'. However, it is not
> > usable unless the virtualenv is activated. As a result, the
> > `python-shell-completion-at-point-ipython' ERT would fail unless a
> > virtualenv which provides ipython is activated.
> >
> > This may be something pyenv users should take care of, but it is more
> > convenient for pyenv users like me to make the following changes.
>
> Do we have a consensus about this issue? Should I install the
> original patch, or are you still discussing it, and an updated patch
> will be posted soon?
>
> Thanks.
Although I think there are a few things left unanswered, the latest
patch sent on Jan 26th is very worthwhile and stable enough to be
installed. So, unless Liu is preparing a new patch, I agree that the
latest patch should be installed. That way more people can try it and
find the problem, if any.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 01 Feb 2024 15:04:01 GMT)
Full text and
rfc822 format available.
Message #43 received at 68559 <at> debbugs.gnu.org (full text, mbox):
kobarity <kobarity <at> gmail.com> 于2024年2月1日周四 22:40写道：
>
> Eli Zaretskii wrote:
> > Do we have a consensus about this issue? Should I install the
> > original patch, or are you still discussing it, and an updated patch
> > will be posted soon?
> >
> > Thanks.
>
> Although I think there are a few things left unanswered, the latest
> patch sent on Jan 26th is very worthwhile and stable enough to be
> installed. So, unless Liu is preparing a new patch, I agree that the
> latest patch should be installed. That way more people can try it and
> find the problem, if any.
Sorry for the late reply. Recent events have occupied all my time. I
plan to update the patch in the next day or two to address remaining
concerns.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Sun, 04 Feb 2024 12:11:02 GMT)
Full text and
rfc822 format available.
Message #46 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
kobarity <kobarity <at> gmail.com> writes:
> I'm experiencing strange behavior regarding completion of import
> statement in a block in Python buffer. If I try to type the following
> lines and then try to complete it, it will fail.
>
> #+begin_src python
> try:
> im
> #+end_src
The problem can be reproduced by running python with jedi completer in
a terminal. The reason is that readline completer can only see the
text in the current line, i.e. " im"; in this case, jedi does not
complete the text to " import".
Similarly, the jedi completer cannot complete "except" after ex:
try:
pass
ex|
> However, when I try to complete at the beginning of the second line:
>
> #+begin_src python
> try:
>
> #+end_src
>
>
> "import" keyword also appears as a candidate. If I cancel the
> candidates and type "im" and try to complete it, it will succeed.
It is because jedi produces completions including " import" for
blank string " ". Due to the same prefix between " " and
" im", completion cache is reused for the latter. Then " import"
can be completed.
It is more a limitation of readline completer than a problem with
jedi, as we cannot provide proper completion context for jedi. We may
define a custom completer to combine jedi and rlcompleter, e.g.
(setq python-shell-readline-completer "
def __PYTHON_EL_setup_readline_completer():
import readline, rlcompleter
import re, sys, os, __main__
from jedi import Interpreter
class MyJediRL:
def __init__(self):
self.rlcompleter = rlcompleter.Completer()
self.rldelim = readline.get_completer_delims()
def complete(self, text, state):
if state == 0:
sys.path.insert(0, os.getcwd())
try:
interpreter = Interpreter(text, [__main__.__dict__])
completions = interpreter.complete(fuzzy=False)
self.matches = [
text[:len(text) - c._like_name_length] + c.name_with_symbols
for c in completions
]
# try rlcompleter
sub = re.split('[' + re.escape(self.rldelim) + ']', text)[-1]
i = 0
while True:
completion = self.rlcompleter.complete(sub, i)
if not completion:
break
i += 1
completion = text[:len(text)-len(sub)] + completion.rstrip(' ()')
if completion not in self.matches:
self.matches.append(completion)
except:
raise
finally:
sys.path.pop(0)
try:
return self.matches[state]
except IndexError:
return None
readline.set_completer(MyJediRL().complete)
readline.set_completer_delims('')")
> Another thing I noticed is the multi-line import statement. If the
> import statement is one-line, each items (IGNORECASE and MULTILINE in
> the example below) can be completed.
>
> #+begin_src python
> from re import IGNORECASE, MULTILINE
> #+end_src
>
>
> However, they cannot be completed if the import statement spans
> multi-line.
>
> #+begin_src python
> from re import (
> IGN
> #+end_src
>
> This happens in both Python buffer and Python Shell buffer. Perhaps
> this is a limitation of Jedi completer?
Yes. Because readline completer cannot see cross-line context, I added
the function "python-shell-completion-extra-context" in previous patch
to address the case of multi-line function call. I have updated the
attached patch to handle multi-line import statement.
The change to python-tests.el has been incorporated in the patch.
Thanks!
[0001-Improve-Python-shell-completion-bug-68559.patch (text/x-diff, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Sun, 04 Feb 2024 14:36:01 GMT)
Full text and
rfc822 format available.
Message #49 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui wrote:
> kobarity <kobarity <at> gmail.com> writes:
> > I'm experiencing strange behavior regarding completion of import
> > statement in a block in Python buffer. If I try to type the following
> > lines and then try to complete it, it will fail.
> >
> > #+begin_src python
> > try:
> > im
> > #+end_src
>
> The problem can be reproduced by running python with jedi completer in
> a terminal. The reason is that readline completer can only see the
> text in the current line, i.e. " im"; in this case, jedi does not
> complete the text to " import".
>
> Similarly, the jedi completer cannot complete "except" after ex:
>
> try:
> pass
> ex|
>
> > However, when I try to complete at the beginning of the second line:
> >
> > #+begin_src python
> > try:
> >
> > #+end_src
> >
> >
> > "import" keyword also appears as a candidate. If I cancel the
> > candidates and type "im" and try to complete it, it will succeed.
>
> It is because jedi produces completions including " import" for
> blank string " ". Due to the same prefix between " " and
> " im", completion cache is reused for the latter. Then " import"
> can be completed.
>
> It is more a limitation of readline completer than a problem with
> jedi, as we cannot provide proper completion context for jedi. We may
> define a custom completer to combine jedi and rlcompleter, e.g.
>
> (setq python-shell-readline-completer "
> def __PYTHON_EL_setup_readline_completer():
> import readline, rlcompleter
> import re, sys, os, __main__
> from jedi import Interpreter
>
> class MyJediRL:
> def __init__(self):
> self.rlcompleter = rlcompleter.Completer()
> self.rldelim = readline.get_completer_delims()
>
> def complete(self, text, state):
> if state == 0:
> sys.path.insert(0, os.getcwd())
> try:
> interpreter = Interpreter(text, [__main__.__dict__])
> completions = interpreter.complete(fuzzy=False)
> self.matches = [
> text[:len(text) - c._like_name_length] + c.name_with_symbols
> for c in completions
> ]
>
> # try rlcompleter
> sub = re.split('[' + re.escape(self.rldelim) + ']', text)[-1]
> i = 0
> while True:
> completion = self.rlcompleter.complete(sub, i)
> if not completion:
> break
> i += 1
> completion = text[:len(text)-len(sub)] + completion.rstrip(' ()')
> if completion not in self.matches:
> self.matches.append(completion)
> except:
> raise
> finally:
> sys.path.pop(0)
> try:
> return self.matches[state]
> except IndexError:
> return None
>
> readline.set_completer(MyJediRL().complete)
> readline.set_completer_delims('')")
Thank you for the detailed explanation and the workaround. I
confirmed that the problem is solved by the above workaround. Just to
confirm, are you of the opinion that this workaround should not be the
default?
> > Another thing I noticed is the multi-line import statement. If the
> > import statement is one-line, each items (IGNORECASE and MULTILINE in
> > the example below) can be completed.
> >
> > #+begin_src python
> > from re import IGNORECASE, MULTILINE
> > #+end_src
> >
> >
> > However, they cannot be completed if the import statement spans
> > multi-line.
> >
> > #+begin_src python
> > from re import (
> > IGN
> > #+end_src
> >
> > This happens in both Python buffer and Python Shell buffer. Perhaps
> > this is a limitation of Jedi completer?
>
> Yes. Because readline completer cannot see cross-line context, I added
> the function "python-shell-completion-extra-context" in previous patch
> to address the case of multi-line function call. I have updated the
> attached patch to handle multi-line import statement.
Thank you very much. I confirmed that the new patch allows completion
of multi-line import statements.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Mon, 05 Feb 2024 15:04:02 GMT)
Full text and
rfc822 format available.
Message #52 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Sun, Feb 4, 2024 at 10:35 PM kobarity <kobarity <at> gmail.com> wrote:
> > It is more a limitation of readline completer than a problem with
> > jedi, as we cannot provide proper completion context for jedi. We may
> > define a custom completer to combine jedi and rlcompleter, e.g.
> >
> > (setq python-shell-readline-completer "
> > def __PYTHON_EL_setup_readline_completer():
> > import readline, rlcompleter
> > import re, sys, os, __main__
> > from jedi import Interpreter
> >
> > class MyJediRL:
> > def __init__(self):
> > self.rlcompleter = rlcompleter.Completer()
> > self.rldelim = readline.get_completer_delims()
> >
> > def complete(self, text, state):
> > if state == 0:
> > sys.path.insert(0, os.getcwd())
> > try:
> > interpreter = Interpreter(text, [__main__.__dict__])
> > completions = interpreter.complete(fuzzy=False)
> > self.matches = [
> > text[:len(text) - c._like_name_length] + c.name_with_symbols
> > for c in completions
> > ]
> >
> > # try rlcompleter
> > sub = re.split('[' + re.escape(self.rldelim) + ']', text)[-1]
> > i = 0
> > while True:
> > completion = self.rlcompleter.complete(sub, i)
> > if not completion:
> > break
> > i += 1
> > completion = text[:len(text)-len(sub)] + completion.rstrip(' ()')
> > if completion not in self.matches:
> > self.matches.append(completion)
> > except:
> > raise
> > finally:
> > sys.path.pop(0)
> > try:
> > return self.matches[state]
> > except IndexError:
> > return None
> >
> > readline.set_completer(MyJediRL().complete)
> > readline.set_completer_delims('')")
>
> Thank you for the detailed explanation and the workaround. I
> confirmed that the problem is solved by the above workaround. Just to
> confirm, are you of the opinion that this workaround should not be the
> default?
I'm not sure if we should add more Python code in the form of strings
to python.el, which increases maintenance burden IMO. Maybe they could
be distributed separately at ELPA/Git forges.
Actually, I'm considering simplifying this patch to mainly fix the bug
that python shell completion doesn't respect the delimiter of readline
completer. The new patch has been attached. It should support
the completer defined in the PYTHONSTARTUP file, e.g., jedi or a
custom completer like the above one. WDYT?
[0001-Respect-the-delimiter-of-completer-in-Python-shell-c.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Tue, 06 Feb 2024 01:27:02 GMT)
Full text and
rfc822 format available.
Message #55 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Mon, Feb 5, 2024 at 11:03 PM Liu Hui <liuhui1610 <at> gmail.com> wrote:
> > Thank you for the detailed explanation and the workaround. I
> > confirmed that the problem is solved by the above workaround. Just to
> > confirm, are you of the opinion that this workaround should not be the
> > default?
>
> I'm not sure if we should add more Python code in the form of strings
> to python.el, which increases maintenance burden IMO. Maybe they could
> be distributed separately at ELPA/Git forges.
>
> Actually, I'm considering simplifying this patch to mainly fix the bug
> that python shell completion doesn't respect the delimiter of readline
> completer. The new patch has been attached. It should support
> the completer defined in the PYTHONSTARTUP file, e.g., jedi or a
> custom completer like the above one. WDYT?
I have updated documentation about readline completer and fixed an
error in commit log in the attach patch.
[0001-Respect-the-delimiter-of-completer-in-Python-shell-c.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Tue, 06 Feb 2024 15:13:01 GMT)
Full text and
rfc822 format available.
Message #58 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui wrote:
> On Mon, Feb 5, 2024 at 11:03 PM Liu Hui <liuhui1610 <at> gmail.com> wrote:
>
> > > Thank you for the detailed explanation and the workaround. I
> > > confirmed that the problem is solved by the above workaround. Just to
> > > confirm, are you of the opinion that this workaround should not be the
> > > default?
> >
> > I'm not sure if we should add more Python code in the form of strings
> > to python.el, which increases maintenance burden IMO. Maybe they could
> > be distributed separately at ELPA/Git forges.
> >
> > Actually, I'm considering simplifying this patch to mainly fix the bug
> > that python shell completion doesn't respect the delimiter of readline
> > completer. The new patch has been attached. It should support
> > the completer defined in the PYTHONSTARTUP file, e.g., jedi or a
> > custom completer like the above one. WDYT?
>
> I have updated documentation about readline completer and fixed an
> error in commit log in the attach patch.
Thanks, I understood your concern. I tried the new patch and
confirmed that your MyJediRL can be used as a custom completer by
storing it in a file specified in the PYTHONSTARTUP environment
variable. This may be the better way to go.
Sorry, I noticed one more thing. As written in the comments below, it
would be better to use `line-beginning-position' and
`line-end-position' instead of `pos-bol' and `pos-eol'.
;; This is a GNU ELPA :core package. Avoid functionality that is not
;; compatible with the version of Emacs recorded above.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 07 Feb 2024 13:23:02 GMT)
Full text and
rfc822 format available.
Message #61 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Tue, Feb 6, 2024 at 11:12 PM kobarity <kobarity <at> gmail.com> wrote:
> Sorry, I noticed one more thing. As written in the comments below, it
> would be better to use `line-beginning-position' and
> `line-end-position' instead of `pos-bol' and `pos-eol'.
>
> ;; This is a GNU ELPA :core package. Avoid functionality that is not
> ;; compatible with the version of Emacs recorded above.
Thanks for pointing it out! `pos-bol' was used because it can ignore
the prompt in `python-shell--get-multiline-input'. Fortunately, I find
`pos-bol' and `pos-eol' are available in the compat package since
29.1.1.0, so it should be enough to upgrade the compat version in
Package-Requires. I have updated the attached patch.
[0001-Respect-the-delimiter-of-completer-in-Python-shell-c.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 07 Feb 2024 15:21:02 GMT)
Full text and
rfc822 format available.
Message #64 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui wrote:
> On Tue, Feb 6, 2024 at 11:12 PM kobarity <kobarity <at> gmail.com> wrote:
>
> > Sorry, I noticed one more thing. As written in the comments below, it
> > would be better to use `line-beginning-position' and
> > `line-end-position' instead of `pos-bol' and `pos-eol'.
> >
> > ;; This is a GNU ELPA :core package. Avoid functionality that is not
> > ;; compatible with the version of Emacs recorded above.
>
> Thanks for pointing it out! `pos-bol' was used because it can ignore
> the prompt in `python-shell--get-multiline-input'. Fortunately, I find
> `pos-bol' and `pos-eol' are available in the compat package since
> 29.1.1.0, so it should be enough to upgrade the compat version in
> Package-Requires. I have updated the attached patch.
I see. The patch looks good to me. I appreciate your efforts!
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 08 Feb 2024 12:14:02 GMT)
Full text and
rfc822 format available.
Message #67 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> Date: Thu, 08 Feb 2024 00:19:55 +0900
> From: kobarity <kobarity <at> gmail.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,
> 68559 <at> debbugs.gnu.org
>
> Liu Hui wrote:
> > On Tue, Feb 6, 2024 at 11:12 PM kobarity <kobarity <at> gmail.com> wrote:
> >
> > > Sorry, I noticed one more thing. As written in the comments below, it
> > > would be better to use `line-beginning-position' and
> > > `line-end-position' instead of `pos-bol' and `pos-eol'.
> > >
> > > ;; This is a GNU ELPA :core package. Avoid functionality that is not
> > > ;; compatible with the version of Emacs recorded above.
> >
> > Thanks for pointing it out! `pos-bol' was used because it can ignore
> > the prompt in `python-shell--get-multiline-input'. Fortunately, I find
> > `pos-bol' and `pos-eol' are available in the compat package since
> > 29.1.1.0, so it should be enough to upgrade the compat version in
> > Package-Requires. I have updated the attached patch.
>
> I see. The patch looks good to me. I appreciate your efforts!
Thanks, I installed it on the master branch.
What about the other patch, one you posted in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
Does that one need to be installed as well, or is it included in the
one I just installed?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 08 Feb 2024 13:34:02 GMT)
Full text and
rfc822 format available.
Message #70 received at 68559 <at> debbugs.gnu.org (full text, mbox):
On Thu, Feb 8, 2024 at 8:13 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> What about the other patch, one you posted in
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
>
> Does that one need to be installed as well, or is it included in the
> one I just installed?
The installed one is complete. Thanks.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 08 Feb 2024 13:47:02 GMT)
Full text and
rfc822 format available.
Message #73 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> From: Liu Hui <liuhui1610 <at> gmail.com>
> Date: Thu, 8 Feb 2024 21:33:10 +0800
> Cc: kobarity <kobarity <at> gmail.com>, 68559 <at> debbugs.gnu.org
>
> On Thu, Feb 8, 2024 at 8:13 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > What about the other patch, one you posted in
> >
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
> >
> > Does that one need to be installed as well, or is it included in the
> > one I just installed?
>
> The installed one is complete. Thanks.
Great, thanks. Then this bug can be closed now, right?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 08 Feb 2024 14:17:01 GMT)
Full text and
rfc822 format available.
Message #76 received at 68559 <at> debbugs.gnu.org (full text, mbox):
On Thu, Feb 8, 2024 at 9:46 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Liu Hui <liuhui1610 <at> gmail.com>
> > Date: Thu, 8 Feb 2024 21:33:10 +0800
> > Cc: kobarity <kobarity <at> gmail.com>, 68559 <at> debbugs.gnu.org
> >
> > On Thu, Feb 8, 2024 at 8:13 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > > What about the other patch, one you posted in
> > >
> > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
> > >
> > > Does that one need to be installed as well, or is it included in the
> > > one I just installed?
> >
> > The installed one is complete. Thanks.
>
> Great, thanks. Then this bug can be closed now, right?
Yes, please.
Reply sent
to
Eli Zaretskii <eliz <at> gnu.org>:
You have taken responsibility.
(Thu, 08 Feb 2024 16:44:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
Liu Hui <liuhui1610 <at> gmail.com>:
bug acknowledged by developer.
(Thu, 08 Feb 2024 16:44:01 GMT)
Full text and
rfc822 format available.
Message #81 received at 68559-done <at> debbugs.gnu.org (full text, mbox):
> From: Liu Hui <liuhui1610 <at> gmail.com>
> Date: Thu, 8 Feb 2024 22:16:23 +0800
> Cc: kobarity <at> gmail.com, 68559 <at> debbugs.gnu.org
>
> On Thu, Feb 8, 2024 at 9:46 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > > From: Liu Hui <liuhui1610 <at> gmail.com>
> > > Date: Thu, 8 Feb 2024 21:33:10 +0800
> > > Cc: kobarity <kobarity <at> gmail.com>, 68559 <at> debbugs.gnu.org
> > >
> > > On Thu, Feb 8, 2024 at 8:13 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > >
> > > > What about the other patch, one you posted in
> > > >
> > > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
> > > >
> > > > Does that one need to be installed as well, or is it included in the
> > > > one I just installed?
> > >
> > > The installed one is complete. Thanks.
> >
> > Great, thanks. Then this bug can be closed now, right?
>
> Yes, please.
Thanks, closing.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 15 Feb 2024 14:46:01 GMT)
Full text and
rfc822 format available.
Message #84 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
[python-tests.log (application/octet-stream, attachment)]
Did not alter fixed versions and reopened.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Thu, 15 Feb 2024 16:27:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 15 Feb 2024 16:38:02 GMT)
Full text and
rfc822 format available.
Message #89 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> Date: Thu, 15 Feb 2024 15:43:32 +0100
> Cc: Eli Zaretskii <eliz <at> gnu.org>,
> kobarity <at> gmail.com,
> 68559 <at> debbugs.gnu.org
> The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
>
> It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
The python-*-completion-at-point-* tests never worked for me, and I
always assumed it was something specific to MS-Windows. But maybe
not. There was no change in the tests that fail for me before and
after the recent changes related to Python.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 15 Feb 2024 16:50:02 GMT)
Full text and
rfc822 format available.
Message #92 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> Cc: liuhui1610 <at> gmail.com, kobarity <at> gmail.com, 68559 <at> debbugs.gnu.org
> Date: Thu, 15 Feb 2024 18:37:13 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
>
> > It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
>
> The python-*-completion-at-point-* tests never worked for me, and I
> always assumed it was something specific to MS-Windows. But maybe
> not. There was no change in the tests that fail for me before and
> after the recent changes related to Python.
And, btw, on this Gnu/Linux system:
$ uname -a
Linux (REDACTED) 5.15.0-94-generic #104+11.0trisquel25 SMP Sat Feb 10 06:24:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
all the Python tests that failed for you still pass, with the current
master. So I wonder what is it that causes those failures on your
system.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 15 Feb 2024 17:23:02 GMT)
Full text and
rfc822 format available.
Message #95 received at 68559 <at> debbugs.gnu.org (full text, mbox):
15 feb. 2024 kl. 17.48 skrev Eli Zaretskii <eliz <at> gnu.org>:
> The python-*-completion-at-point-* tests never worked for me, and I
> always assumed it was something specific to MS-Windows. But maybe
> not. There was no change in the tests that fail for me before and
> after the recent changes related to Python.
Right, I don't think the Windows port is obscure enough to deserve test failures either; it's an unnecessary impediment. If it cannot easily be fixed then it's better to mark them to be skipped on Windows so that you don't see it every time you run 'make check'.
Completion in the Python shell worked on macOS before the change but now pressing TAB causes the process to hang, so something was clearly broken by the patch.
> all the Python tests that failed for you still pass, with the current
> master. So I wonder what is it that causes those failures on your
> system.
The standard Python installation on macOS uses a libedit-based readline module and the patch author probably didn't test the changes with that configuration.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 03:26:01 GMT)
Full text and
rfc822 format available.
Message #98 received at 68559 <at> debbugs.gnu.org (full text, mbox):
On Thu, Feb 15, 2024 at 10:43 PM Mattias Engdegård
<mattias.engdegard <at> gmail.com> wrote:
>
> The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
>
> It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
Hi,
According to the log, the problem may be related to the json library,
which is introduced by the patch to convert the completion results
from Python to elisp. What is the version of Python on macOS? AFAIK
the json library is added to the Standard library since version 2.6.
If the library is not available, could you setup a Python virtual
environment containing the library and run the python tests in it?
Thanks!
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 04:09:02 GMT)
Full text and
rfc822 format available.
Message #101 received at 68559 <at> debbugs.gnu.org (full text, mbox):
On Fri, Feb 16, 2024 at 12:37 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> > Date: Thu, 15 Feb 2024 15:43:32 +0100
> > Cc: Eli Zaretskii <eliz <at> gnu.org>,
> > kobarity <at> gmail.com,
> > 68559 <at> debbugs.gnu.org
> > The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
> >
> > It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
>
> The python-*-completion-at-point-* tests never worked for me, and I
> always assumed it was something specific to MS-Windows. But maybe
> not. There was no change in the tests that fail for me before and
> after the recent changes related to Python.
The Python shell completion relies on the readline module, which is
not available for Python on MS-Windows. According to the instruction
in python.el:
;; readline based shell (it's known to work with PyPy). If your
;; Python installation lacks readline (like CPython for Windows),
;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
;; should suffice. To troubleshoot why you are not getting any
;; completions, you can try the following in your Python shell:
;; >>> import readline, rlcompleter
;; If you see an error, then you need to either install pyreadline or
;; setup custom code that avoids that dependency.
It may be necessary to install pyreadline (for Python 2.7) or
pyreadline3 (for Python 3).
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 07:43:01 GMT)
Full text and
rfc822 format available.
Message #104 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> From: Liu Hui <liuhui1610 <at> gmail.com>
> Date: Fri, 16 Feb 2024 12:06:38 +0800
> Cc: Mattias Engdegård <mattias.engdegard <at> gmail.com>,
> kobarity <at> gmail.com, 68559 <at> debbugs.gnu.org
>
> On Fri, Feb 16, 2024 at 12:37 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > The python-*-completion-at-point-* tests never worked for me, and I
> > always assumed it was something specific to MS-Windows. But maybe
> > not. There was no change in the tests that fail for me before and
> > after the recent changes related to Python.
>
> The Python shell completion relies on the readline module, which is
> not available for Python on MS-Windows. According to the instruction
> in python.el:
>
> ;; readline based shell (it's known to work with PyPy). If your
> ;; Python installation lacks readline (like CPython for Windows),
> ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
> ;; should suffice. To troubleshoot why you are not getting any
> ;; completions, you can try the following in your Python shell:
>
> ;; >>> import readline, rlcompleter
>
> ;; If you see an error, then you need to either install pyreadline or
> ;; setup custom code that avoids that dependency.
I don't know if I have CPython, but the above does show an error
message.
> It may be necessary to install pyreadline (for Python 2.7) or
> pyreadline3 (for Python 3).
I will see if I can do that, thanks.
Regardless, patches to the test suite to skip the tests which rely on
those modules, if they aren't installed, will be welcome.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 09:36:01 GMT)
Full text and
rfc822 format available.
Message #107 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui wrote:
> On Thu, Feb 15, 2024 at 10:43 PM Mattias Engdegård
> <mattias.engdegard <at> gmail.com> wrote:
> >
> > The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
> >
> > It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
>
> Hi,
>
> According to the log, the problem may be related to the json library,
> which is introduced by the patch to convert the completion results
> from Python to elisp. What is the version of Python on macOS? AFAIK
> the json library is added to the Standard library since version 2.6.
> If the library is not available, could you setup a Python virtual
> environment containing the library and run the python tests in it?
> Thanks!
This is caused by the fact that the input is echoed back on MacOS
Python. This is described in etc/PROBLEMS:
*** In Inferior Python mode, input is echoed and native completion doesn't work.
The following log shows that it tries to parse the echoed back string
as a JSON string.
Test python-completion-at-point-1 backtrace:
json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
I think the echoed back string were ignored before the patch is
applied. However, after applying the patch, the returned string is
now parsed as JSON, which I believe is the reason for the error.
So one workaround would be to remove the echoed back string before
parsing as JSON.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 11:47:02 GMT)
Full text and
rfc822 format available.
Message #110 received at 68559 <at> debbugs.gnu.org (full text, mbox):
16 feb. 2024 kl. 10.34 skrev kobarity <kobarity <at> gmail.com>:
> This is caused by the fact that the input is echoed back on MacOS
> Python.
Right, that bug needs to be fixed as well, but the echo problem existed prior to the change that broke the tests.
An alternative might be to disable the tty echo altogether. If I do it right after process creation then it has no effect; presumably Python or its readline module turns on echo just a bit later on. Anyway, running
import tty
tty.setraw(0)
in the python shell seems to put it right, with working completion and the annoying echo gone.
> So one workaround would be to remove the echoed back string before
> parsing as JSON.
Yes, either that or turning off echo in the tty.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 12:53:02 GMT)
Full text and
rfc822 format available.
Message #113 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> Cc: mattias.engdegard <at> gmail.com, kobarity <at> gmail.com, 68559 <at> debbugs.gnu.org
> Date: Fri, 16 Feb 2024 09:41:37 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
>
> > It may be necessary to install pyreadline (for Python 2.7) or
> > pyreadline3 (for Python 3).
>
> I will see if I can do that, thanks.
I've now installed pyreadline3, and I can confirm that the completion
tests which were failing before now pass.
Thanks.
> Regardless, patches to the test suite to skip the tests which rely on
> those modules, if they aren't installed, will be welcome.
This is still true.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 15:27:02 GMT)
Full text and
rfc822 format available.
Message #116 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Mattias Engdegård wrote:
>
> 16 feb. 2024 kl. 10.34 skrev kobarity <kobarity <at> gmail.com>:
>
> > This is caused by the fact that the input is echoed back on MacOS
> > Python.
>
> Right, that bug needs to be fixed as well, but the echo problem existed prior to the change that broke the tests.
>
> An alternative might be to disable the tty echo altogether. If I do it right after process creation then it has no effect; presumably Python or its readline module turns on echo just a bit later on. Anyway, running
>
> import tty
> tty.setraw(0)
>
> in the python shell seems to put it right, with working completion and the annoying echo gone.
>
> > So one workaround would be to remove the echoed back string before
> > parsing as JSON.
>
> Yes, either that or turning off echo in the tty.
I made prototype patches for each method. I don't use Mac so it would
be helpful if you could try these.
0001-Remove-echoed-back-string-in-python-shell-completion.patch
extracts only the last line to exclude echoed back strings.
0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
the Inferior Python tty to raw mode. python-ffap-module-path-1 will
no longer need to be skipped on Mac. If it is safe to set tty to raw
mode on all UNIX based systems, I prefer this method.
By the way, is it necessary to send
`python-shell-completion-setup-code' for every completion in
`python-shell-completion-get-completions'? To me it seems sufficient
to send it once at initialization.
[0001-Remove-echoed-back-string-in-python-shell-completion.patch (application/octet-stream, attachment)]
[0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch (application/octet-stream, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 15:55:02 GMT)
Full text and
rfc822 format available.
Message #119 received at 68559 <at> debbugs.gnu.org (full text, mbox):
> Date: Sat, 17 Feb 2024 00:24:29 +0900
> From: kobarity <kobarity <at> gmail.com>
> Cc: Liu Hui <liuhui1610 <at> gmail.com>,
> Eli Zaretskii <eliz <at> gnu.org>,
> 68559 <at> debbugs.gnu.org
> > Right, that bug needs to be fixed as well, but the echo problem existed prior to the change that broke the tests.
> >
> > An alternative might be to disable the tty echo altogether. If I do it right after process creation then it has no effect; presumably Python or its readline module turns on echo just a bit later on. Anyway, running
> >
> > import tty
> > tty.setraw(0)
> >
> > in the python shell seems to put it right, with working completion and the annoying echo gone.
> >
> > > So one workaround would be to remove the echoed back string before
> > > parsing as JSON.
> >
> > Yes, either that or turning off echo in the tty.
>
> I made prototype patches for each method. I don't use Mac so it would
> be helpful if you could try these.
>
> 0001-Remove-echoed-back-string-in-python-shell-completion.patch
> extracts only the last line to exclude echoed back strings.
>
> 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> the Inferior Python tty to raw mode. python-ffap-module-path-1 will
> no longer need to be skipped on Mac. If it is safe to set tty to raw
> mode on all UNIX based systems, I prefer this method.
Will this work on MS-Windows as well? If you are unsure, would you
please tell me how to test whether this works on Windows, so I could
collect the information for you?
Thanks.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 16 Feb 2024 20:12:01 GMT)
Full text and
rfc822 format available.
Message #122 received at 68559 <at> debbugs.gnu.org (full text, mbox):
16 feb. 2024 kl. 16.24 skrev kobarity <kobarity <at> gmail.com>:
> I made prototype patches for each method. I don't use Mac so it would
> be helpful if you could try these.
Nice, thank you! I can confirm that they both appear to work, at least in the sense that the python-tests pass (except for the ones skipped intentionally), and the python shell behaves reasonable.
Only the set-tty-mode patch eliminates echo in the interactive python shell; in that sense it's preferable.
Both produce the very annoying warning
Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally. Consider installing the python package "readline".
which is not even correct since the standard Python does have a working readline module, even if it uses libedit.
> 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> the Inferior Python tty to raw mode. python-ffap-module-path-1 will
> no longer need to be skipped on Mac.
Right, that test runs and passes.
> If it is safe to set tty to raw
> mode on all UNIX based systems, I prefer this method.
Same here. I see no reason why it wouldn't be safe, either.
> By the way, is it necessary to send
> `python-shell-completion-setup-code' for every completion in
> `python-shell-completion-get-completions'? To me it seems sufficient
> to send it once at initialization.
Indeed, it does seem a bit extravagant.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Sat, 17 Feb 2024 04:38:02 GMT)
Full text and
rfc822 format available.
Message #125 received at 68559 <at> debbugs.gnu.org (full text, mbox):
kobarity <kobarity <at> gmail.com> writes:
> 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> the Inferior Python tty to raw mode. python-ffap-module-path-1 will
> no longer need to be skipped on Mac. If it is safe to set tty to raw
> mode on all UNIX based systems, I prefer this method.
How about the following change, which only affects libedit-based
readline? It may enable native completion on mac, but I cannot test
it.
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b7e43f3fc68..f59bc19367b 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4286,6 +4286,9 @@ (defcustom python-shell-completion-setup-code
except:
pass
else:
+ if readline.__doc__ and 'libedit' in readline.__doc__:
+ import tty
+ tty.setraw(0)
# Try to reuse current completer.
completer = readline.get_completer()
if not completer:
@@ -4471,8 +4474,8 @@ (defun python-shell-completion-native-setup ()
instance.rlcomplete = new_completer
if readline.__doc__ and 'libedit' in readline.__doc__:
- raise Exception('''libedit based readline is known not to work,
- see etc/PROBLEMS under \"In Inferior Python mode, input is echoed\".''')
+ import tty
+ tty.setraw(0)
readline.parse_and_bind('bind ^I rl_complete')
else:
readline.parse_and_bind('tab: complete')
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Sat, 17 Feb 2024 13:23:01 GMT)
Full text and
rfc822 format available.
Message #128 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
17 feb. 2024 kl. 05.36 skrev Liu Hui <liuhui1610 <at> gmail.com>:
> How about the following change, which only affects libedit-based
> readline? It may enable native completion on mac, but I cannot test
> it.
Thank you, but that change alone doesn't seem to suffice for python-tests to pass. Log attached.
[python-tests.log (application/octet-stream, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Sat, 17 Feb 2024 13:36:02 GMT)
Full text and
rfc822 format available.
Message #131 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii wrote:
> > 0001-Remove-echoed-back-string-in-python-shell-completion.patch
> > extracts only the last line to exclude echoed back strings.
> >
> > 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> > the Inferior Python tty to raw mode. python-ffap-module-path-1 will
> > no longer need to be skipped on Mac. If it is safe to set tty to raw
> > mode on all UNIX based systems, I prefer this method.
>
> Will this work on MS-Windows as well? If you are unsure, would you
> please tell me how to test whether this works on Windows, so I could
> collect the information for you?
It does not affect MS-Windows because it sets only when the tty
library can be imported. Although the tty library exists in
MS-Windows, importing the tty library will result in an error because
there is no underlying termios library.
Mattias Engdegård wrote:
>
> 16 feb. 2024 kl. 16.24 skrev kobarity <kobarity <at> gmail.com>:
>
> > I made prototype patches for each method. I don't use Mac so it would
> > be helpful if you could try these.
>
> Nice, thank you! I can confirm that they both appear to work, at least in the sense that the python-tests pass (except for the ones skipped intentionally), and the python shell behaves reasonable.
>
> Only the set-tty-mode patch eliminates echo in the interactive python shell; in that sense it's preferable.
Thank you for testing my patches.
> Both produce the very annoying warning
>
> Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally. Consider installing the python package "readline".
>
> which is not even correct since the standard Python does have a working readline module, even if it uses libedit.
You are right. Maybe the package name should be fixed to "gnureadline"
as described in etc/PROBLEMS.
On Mac, it might be better to set the default value of
`python-shell-completion-native-enable' to nil.
> > 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> > the Inferior Python tty to raw mode. python-ffap-module-path-1 will
> > no longer need to be skipped on Mac.
>
> Right, that test runs and passes.
>
> > If it is safe to set tty to raw
> > mode on all UNIX based systems, I prefer this method.
>
> Same here. I see no reason why it wouldn't be safe, either.
Thanks.
> > By the way, is it necessary to send
> > `python-shell-completion-setup-code' for every completion in
> > `python-shell-completion-get-completions'? To me it seems sufficient
> > to send it once at initialization.
>
> Indeed, it does seem a bit extravagant.
This would be one of the items for future improvement.
Liu Hui wrote:
>
> kobarity <kobarity <at> gmail.com> writes:
>
> > 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> > the Inferior Python tty to raw mode. python-ffap-module-path-1 will
> > no longer need to be skipped on Mac. If it is safe to set tty to raw
> > mode on all UNIX based systems, I prefer this method.
>
> How about the following change, which only affects libedit-based
> readline? It may enable native completion on mac, but I cannot test
> it.
>
> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index b7e43f3fc68..f59bc19367b 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -4286,6 +4286,9 @@ (defcustom python-shell-completion-setup-code
> except:
> pass
> else:
> + if readline.__doc__ and 'libedit' in readline.__doc__:
> + import tty
> + tty.setraw(0)
> # Try to reuse current completer.
> completer = readline.get_completer()
> if not completer:
> @@ -4471,8 +4474,8 @@ (defun python-shell-completion-native-setup ()
> instance.rlcomplete = new_completer
>
> if readline.__doc__ and 'libedit' in readline.__doc__:
> - raise Exception('''libedit based readline is known not to work,
> - see etc/PROBLEMS under \"In Inferior Python mode, input is echoed\".''')
> + import tty
> + tty.setraw(0)
> readline.parse_and_bind('bind ^I rl_complete')
> else:
> readline.parse_and_bind('tab: complete')
Disabling echo back may not be sufficient to enable native completions
on Mac. I have not tried raw mode, but have tried
readline.parse_and_bind('setty -echo') and
readline.parse_and_bind('edit on'). Native completions could be
enabled, but it was unstable.
I have no objection if Mac users check the above patch and if it is
OK.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Mon, 19 Feb 2024 15:07:02 GMT)
Full text and
rfc822 format available.
Message #134 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Eli Zaretskii [2024-02-15 18:48 +0200] wrote:
>> Cc: liuhui1610 <at> gmail.com, kobarity <at> gmail.com, 68559 <at> debbugs.gnu.org
>> Date: Thu, 15 Feb 2024 18:37:13 +0200
>> From: Eli Zaretskii <eliz <at> gnu.org>
>>
>> > It seems to have something to do with completion. Maybe the new code is
>> sensitive to details of Python's command line editor? The standard Python
>> interpreter is unlikely to use GNU readline, for example.
>>
>> The python-*-completion-at-point-* tests never worked for me, and I
>> always assumed it was something specific to MS-Windows. But maybe
>> not. There was no change in the tests that fail for me before and
>> after the recent changes related to Python.
>
> And, btw, on this Gnu/Linux system:
>
> $ uname -a
> Linux (REDACTED) 5.15.0-94-generic #104+11.0trisquel25 SMP Sat Feb 10 06:24:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
>
> all the Python tests that failed for you still pass, with the current
> master. So I wonder what is it that causes those failures on your
> system.
BTW, I have been seeing an IPython test failure on GNU/Linux:
[ipython.txt.gz (application/gzip, attachment)]
[Message part 3 (text/plain, inline)]
Any pointers?
Thanks,
--
Basil
$ ipython --version
8.21.0
$ python --version
Python 3.12.2
In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
version 1.18.0, Xaw3d scroll bars) of 2024-02-19 built on tia
Repository revision: 8f260bb93f534b24d9a93d3315804ffe0c1fec4f
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101011
System Description: Debian GNU/Linux trixie/sid
Configured using:
'configure 'CFLAGS=-Og -ggdb3' -C --prefix=/home/blc/.local
--enable-checking=structs --without-native-compilation
--with-file-notification=yes --with-x-toolkit=lucid --with-x'
Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XAW3D XDBE XIM XINPUT2 XPM
LUCID ZLIB
Important settings:
value of $LANG: en_IE.UTF-8
value of $XMODIFIERS: @im=ibus
locale-coding-system: utf-8-unix
Major mode: Lisp Interaction
Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
minibuffer-regexp-mode: t
line-number-mode: t
indent-tabs-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
Load-path shadows:
None found.
Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd touch-screen tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
xinput2 x multi-tty move-toolbar make-network-process emacs)
Memory information:
((conses 16 39117 8464) (symbols 48 5217 0) (strings 32 13299 1612)
(string-bytes 1 311978) (vectors 16 9245)
(vector-slots 8 110970 8413) (floats 8 23 25) (intervals 56 266 0)
(buffers 984 10))
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Tue, 20 Feb 2024 04:49:02 GMT)
Full text and
rfc822 format available.
Message #137 received at 68559 <at> debbugs.gnu.org (full text, mbox):
On Mon, Feb 19, 2024 at 9:18 PM Basil L. Contovounesios
<basil <at> contovou.net> wrote:
>
> BTW, I have been seeing an IPython test failure on GNU/Linux:
>
>
> Any pointers?
>
> Thanks,
Hi,
I cannot reproduce the problem with Python 3.12 and IPython 8.21. Can
you check the results with the following steps? Thanks.
1. start Python shell with IPython interpreter, i.e.
(setq python-shell-interpreter "ipython")
(setq python-shell-interpreter-args "-i --simple-prompt")
M-x run-python
2. M-x python-shell-completion-native-turn-off
3. eval code in the Python shell:
import re
from IPython.core.completer import provisionalcompleter
with provisionalcompleter():
print(list(get_ipython().Completer.completions('re.split("a",
"abc", maxs', 25)))
expected output: [<Completion start=21 end=25 text='maxsplit='
type='param', signature='?',>]
__PYTHON_EL_get_completions('re.split("a", "abc", maxs')
expected output: '[["maxsplit=", 21, 25, "param", ""]]'
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Tue, 20 Feb 2024 10:19:01 GMT)
Full text and
rfc822 format available.
Message #140 received at 68559 <at> debbugs.gnu.org (full text, mbox):
17 feb. 2024 kl. 14.33 skrev kobarity <kobarity <at> gmail.com>:
> On Mac, it might be better to set the default value of
> `python-shell-completion-native-enable' to nil.
Not sure why the inferior Python process doesn't act on a TAB being sent to it. Is the tty somehow in a state that disables readline/libedit? Something that Emacs does when setting up the pty?
Of course from a software engineering point of view, it's silly to send what essentially are edit keystrokes to Python and then screen-scrape the output. A proper interaction protocol would be the way to go, and would work equally well on any platform including Windows.
> Disabling echo back may not be sufficient to enable native completions
> on Mac. I have not tried raw mode, but have tried
> readline.parse_and_bind('setty -echo') and
> readline.parse_and_bind('edit on'). Native completions could be
> enabled, but it was unstable.
>
> I have no objection if Mac users check the above patch and if it is
> OK.
Afraid it wasn't.
Thanks for your patches. I suggest we apply your set-tty-raw patch on master now since it cures the test failures without breaking anything else (on Mac; I'm assuming no regression elsewhere).
Would you like me to do that for you?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Tue, 20 Feb 2024 13:17:02 GMT)
Full text and
rfc822 format available.
Message #143 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Liu Hui [2024-02-20 12:46 +0800] wrote:
> On Mon, Feb 19, 2024 at 9:18 PM Basil L. Contovounesios
> <basil <at> contovou.net> wrote:
>>
>> BTW, I have been seeing an IPython test failure on GNU/Linux:
>> Any pointers?
>
> I cannot reproduce the problem with Python 3.12 and IPython 8.21. Can
> you check the results with the following steps? Thanks.
Thanks. These all work as expected in 'emacs -Q', which made me realise
what may be the problem:
- the Emacs test suite runs under HOME=/nonexistent
- but PATH is unchanged
- I have the latest IPython installed locally using pipx:
$ ls -l $(which ipython)
lrwxrwxrwx 1 blc blc 53 Feb 20 11:46
/home/blc/.local/bin/ipython ->
/home/blc/.local/share/pipx/venvs/ipython/bin/ipython
This is confirmed by the following experiment:
[prynt.diff (text/x-diff, inline)]
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index af6c199b5bd..3393f93542b 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4883,6 +4883,7 @@ python-shell-completion-at-point-ipython
(python-tests-with-temp-buffer-with-shell
""
(python-shell-with-shell-buffer
+ (make-temp-file "my-py-out-" nil ".txt" (buffer-string))
(python-shell-completion-native-turn-off)
(python-tests--completion-module)
(python-tests--completion-parameters)
[Message part 3 (text/plain, inline)]
Running 'make TEST_LOAD_EL=no test/python-tests' now results in the
following output file:
[my-py-out-kFh7He.txt (text/plain, inline)]
/home/blc/.local/share/pipx/venvs/ipython/lib/python3.12/site-packages/IPython/paths.py:69: UserWarning: IPython parent '/nonexistent' is not a writable location, using a temp directory.
warn("IPython parent '{0}' is not a writable location,"
Python 3.12.2 (main, Feb 18 2024, 16:59:56) [GCC 13.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.21.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]:
[Message part 5 (text/plain, inline)]
In this case list(get_ipython().Completer.completions(...)) is empty.
I'm not sure what the best way to detect or work around this is.
Any ideas?
[ It's a problem shared, to an extent, with Eglot tests, since LSP servers are
often installed in one's HOME. ]
Thanks,
--
Basil
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 21 Feb 2024 10:02:02 GMT)
Full text and
rfc822 format available.
Message #146 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Tue, Feb 20, 2024 at 9:15 PM Basil L. Contovounesios
<basil <at> contovou.net> wrote:
>
> Liu Hui [2024-02-20 12:46 +0800] wrote:
>
> > On Mon, Feb 19, 2024 at 9:18 PM Basil L. Contovounesios
> > <basil <at> contovou.net> wrote:
> >>
> >> BTW, I have been seeing an IPython test failure on GNU/Linux:
> >> Any pointers?
> >
> > I cannot reproduce the problem with Python 3.12 and IPython 8.21. Can
> > you check the results with the following steps? Thanks.
>
> Thanks. These all work as expected in 'emacs -Q', which made me realise
> what may be the problem:
>
> - the Emacs test suite runs under HOME=/nonexistent
> - but PATH is unchanged
> - I have the latest IPython installed locally using pipx:
> $ ls -l $(which ipython)
> lrwxrwxrwx 1 blc blc 53 Feb 20 11:46
> /home/blc/.local/bin/ipython ->
> /home/blc/.local/share/pipx/venvs/ipython/bin/ipython
>
> This is confirmed by the following experiment:
>
>
> Running 'make TEST_LOAD_EL=no test/python-tests' now results in the
> following output file:
>
>
> In this case list(get_ipython().Completer.completions(...)) is empty.
Thank you for the investigation! I didn't realize the problem is
related to HOME=/nonexistent in 'make test'. Now I can also reproduce
it. In fact, I found that the test failure was caused by Jedi, as it
attempts to write cache to a non-existent directory.
When Jedi is directly used as the completion backend, more test
failures will be triggered:
PYTHONSTARTUP="$(python -m jedi repl)" make TEST_LOAD_EL=no test/python-tests
If the cache directory is writable, all tests will pass:
PYTHONSTARTUP="$(python -m jedi repl)" XDG_CACHE_HOME=~/.cache make
TEST_LOAD_EL=no test/python-tests
The attached patch should fix the problem.
[0001-Fix-Python-shell-completion-test-failures.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 21 Feb 2024 13:17:01 GMT)
Full text and
rfc822 format available.
Message #149 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Mattias Engdegård wrote:
> 17 feb. 2024 kl. 14.33 skrev kobarity <kobarity <at> gmail.com>:
> > On Mac, it might be better to set the default value of
> > `python-shell-completion-native-enable' to nil.
> Not sure why the inferior Python process doesn't act on a TAB being sent to it. Is the tty somehow in a state that disables readline/libedit? Something that Emacs does when setting up the pty?
On Mac, there are two independent problems. One problem, which
existed before Liu's patch was applied, is that native completions
cannot be enabled. The other problem is that the test now fails after
applying Liu's patch. Suppressing echo back solves the latter, but
not the former. Because of the former, leaving
`python-shell-completion-native-enable' at the default setting of t
will result in the warning.
There are two completions methods, native and non-native. If native
completions cannot be used, it falls back to non-native completions.
Native completions is triggered by sending TAB character, while
non-native completions is triggered by sending and evaluating
__PYTHON_EL_get_completions().
As far as I have tried, native completions cannot be enabled for
libedit-based readline package, even on Linux.
So, there are two ways for Mac users to avoid the warning. One is to
use gnureadline instead of libedit, and the other is to give up native
completions and set `python-shell-completion-native-enable' to nil.
This remains correct even with echo back disabled.
> Of course from a software engineering point of view, it's silly to send what essentially are edit keystrokes to Python and then screen-scrape the output. A proper interaction protocol would be the way to go, and would work equally well on any platform including Windows.
I think the protocol between python.el and inferior Python process is
already platform independent. Protocol violations are echo back.
> Thanks for your patches. I suggest we apply your set-tty-raw patch on master now since it cures the test failures without breaking anything else (on Mac; I'm assuming no regression elsewhere).
>
> Would you like me to do that for you?
Yes, please. Thank you. I have no commit right.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 21 Feb 2024 14:57:01 GMT)
Full text and
rfc822 format available.
Message #152 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui [2024-02-21 18:00 +0800] wrote:
> The attached patch should fix the problem.
Thanks! The patch fixes the error, but that's because
python-shell-completion-at-point-ipython is now skipped:
in particular, python-shell-readline-completer-delims evaluates to
"\s\t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?" rather than the empty string.
Any idea why that happens?
> - (progn
> - (run-python nil t)
> - (insert ,contents)
> - (goto-char (point-min))
> - (python-tests-shell-wait-for-prompt)
> - ,@body)
> + ;; Prevent test failures when Jedi is used as a completion
> + ;; backend, either directly or indirectly (e.g., via
> + ;; IPython). Jedi needs to store cache, but the
> + ;; "/nonexistent" HOME directory is not writable.
> + (ert-with-temp-directory cache-dir
^^^^^^^^^
Should this be an uninterned symbol instead?
> + (with-environment-variables (("XDG_CACHE_HOME" cache-dir))
> + (run-python nil t)
> + (insert ,contents)
> + (goto-char (point-min))
> + (python-tests-shell-wait-for-prompt)
> + ,@body))
> (when (python-shell-get-buffer)
> (python-shell-with-shell-buffer
> (let (kill-buffer-hook kill-buffer-query-functions)
Thanks,
--
Basil
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 21 Feb 2024 18:22:01 GMT)
Full text and
rfc822 format available.
Message #155 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
21 feb. 2024 kl. 14.13 skrev kobarity <kobarity <at> gmail.com>:
> As far as I have tried, native completions cannot be enabled for
> libedit-based readline package, even on Linux.
Did you find out why? Python responds nicely to TAB if run from a terminal, so what is it that we do in Emacs that prevents it from doing so? A TTY setting, or an environment variable (the TERM=dumb)?
> So, there are two ways for Mac users to avoid the warning. One is to
> use gnureadline instead of libedit, and the other is to give up native
> completions and set `python-shell-completion-native-enable' to nil.
Preferably neither. Emacs should adapt to the system environment, and in particular not warn about the default environment. A warning is an indication of a possible risk or other problem, and there is none here.
At most, python-mode could show a calm notice on the echo line but even that is a stretch. What do you think about this rough patch?
[nowarn.diff (application/octet-stream, attachment)]
[Message part 3 (text/plain, inline)]
> I think the protocol between python.el and inferior Python process is
> already platform independent. Protocol violations are echo back.
No, I meant a protocol that allows Emacs to act as a first-class Python front-end, not simulate a terminal, send keystrokes, use heuristics to determine what in the output stream is a prompt, REPL value, error, completion etc.
For example, it's a bit silly to input multi-line code in the REPL as a sequence of individual single-line commands, when we actually are inside a text editor that can edit multi-line Python code without a problem.
(I'm not suggesting that you or anybody in particular should do this; just that it's feasible. It would clearly be quite some work!)
>> Thanks for your patches. I suggest we apply your set-tty-raw patch on master now since it cures the test failures without breaking anything else (on Mac; I'm assuming no regression elsewhere).
>>
>> Would you like me to do that for you?
>
> Yes, please.
Done!
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 22 Feb 2024 10:42:01 GMT)
Full text and
rfc822 format available.
Message #158 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Wed, Feb 21, 2024 at 10:55 PM Basil L. Contovounesios
<basil <at> contovou.net> wrote:
>
> Liu Hui [2024-02-21 18:00 +0800] wrote:
>
> > The attached patch should fix the problem.
>
> Thanks! The patch fixes the error, but that's because
> python-shell-completion-at-point-ipython is now skipped:
Only the native completion part is skipped for the reason below.
> in particular, python-shell-readline-completer-delims evaluates to
> "\s\t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?" rather than the empty string.
"\s\t\n`..." is the delimiter used by rlcompleter, which is the
default completer used by the readline. rlcompleter cannot complete
module names or parameters, so in this case the native completion part
is skipped.
The test is intended to be used with Jedi as the completion backend,
e.g. setting PYTHONSTARTUP="$(python -m jedi repl)", or with a custom
IPython completer defined in the PYTHONSTARTUP file. I have updated the
patch to make the test use Jedi when possible.
> > - (progn
> > - (run-python nil t)
> > - (insert ,contents)
> > - (goto-char (point-min))
> > - (python-tests-shell-wait-for-prompt)
> > - ,@body)
> > + ;; Prevent test failures when Jedi is used as a completion
> > + ;; backend, either directly or indirectly (e.g., via
> > + ;; IPython). Jedi needs to store cache, but the
> > + ;; "/nonexistent" HOME directory is not writable.
> > + (ert-with-temp-directory cache-dir
> ^^^^^^^^^
> Should this be an uninterned symbol instead?
Fixed.
[0001-Fix-Python-shell-completion-test-failures.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 22 Feb 2024 14:06:02 GMT)
Full text and
rfc822 format available.
Message #161 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui [2024-02-22 18:31 +0800] wrote:
> On Wed, Feb 21, 2024 at 10:55 PM Basil L. Contovounesios
> <basil <at> contovou.net> wrote:
>>
>> Liu Hui [2024-02-21 18:00 +0800] wrote:
>>
>> > The attached patch should fix the problem.
>>
>> Thanks! The patch fixes the error, but that's because
>> python-shell-completion-at-point-ipython is now skipped:
>
> Only the native completion part is skipped for the reason below.
>
>> in particular, python-shell-readline-completer-delims evaluates to
>> "\s\t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?" rather than the empty string.
>
> "\s\t\n`..." is the delimiter used by rlcompleter, which is the
> default completer used by the readline. rlcompleter cannot complete
> module names or parameters, so in this case the native completion part
> is skipped.
>
> The test is intended to be used with Jedi as the completion backend,
> e.g. setting PYTHONSTARTUP="$(python -m jedi repl)", or with a custom
> IPython completer defined in the PYTHONSTARTUP file. I have updated the
> patch to make the test use Jedi when possible.
Thanks, looks fine to me and runs without issue.
The python-shell-completion-at-point-ipython test is still skipped, with
or without specifying PYTHONSTARTUP="$(python -m jedi repl)", but like
you suggest that's not necessarily a problem.
I'm guessing you don't have write access to emacs.git, but have signed
the CA? If so, and if there are no other comments/objections in a few
days, I'll apply the patch in your name.
Thanks,
--
Basil
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 22 Feb 2024 16:18:02 GMT)
Full text and
rfc822 format available.
Message #164 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Mattias Engdegård wrote:
> 21 feb. 2024 kl. 14.13 skrev kobarity <kobarity <at> gmail.com>:
> > As far as I have tried, native completions cannot be enabled for
> > libedit-based readline package, even on Linux.
> Did you find out why? Python responds nicely to TAB if run from a terminal, so what is it that we do in Emacs that prevents it from doing so? A TTY setting, or an environment variable (the TERM=dumb)?
No, not exactly, but I don't think it is related to terminal settings.
python.el does not parse the completion candidates shown on the
terminal. Instead, it expects the candidates in a particular format,
which I call the protocol between python.el and inferior Python
process.
__PYTHON_EL_native_completion_setup() defined in
`python-shell-completion-native-setup' sets up the completer to do so.
We can test it by removing empty lines and pasting into Python REPL
outside Emacs. An example of typing "ra" and TAB using readline would
look like this.
>>> raraise
range
0__dummy_completion__ 1__dummy_completion__
>>> ra
However, when I use libedit-based readline and remove the raise line
in __PYTHON_EL_native_completion_setup(), it behaves like this.
>>> raraise
range
Please note that there is no dummy completions, and the new prompt is
not shown. When I hit TAB again, it would be like this.
>>> raraise
range
raise
range
0__dummy_completion__ 0__dummy_completion__ 0__dummy_completion__ 1__dummy_completion__
>>> ra
I think this difference in behavior is the reason why Native
completions does not work with libedit.
> > So, there are two ways for Mac users to avoid the warning. One is to
> > use gnureadline instead of libedit, and the other is to give up native
> > completions and set `python-shell-completion-native-enable' to nil.
>
> Preferably neither. Emacs should adapt to the system environment, and in particular not warn about the default environment. A warning is an indication of a possible risk or other problem, and there is none here.
>
> At most, python-mode could show a calm notice on the echo line but even that is a stretch. What do you think about this rough patch?
Personally, I think this patch would be fine.
I think we can also improve Non-native completions. Current
implementation sends the definition of __PYTHON_EL_get_completions()
every time. However, sending it once during initialization should be
sufficient. Worse, if the number of characters sent exceeds
`comint-max-line-length', it will be sent via file. This is happening
in your environment. Here is the log you presented.
Test python-completion-at-point-1 backtrace:
json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
Thanks to the echo back, we can see __PYTHON_EL_eval_file() was used.
> > I think the protocol between python.el and inferior Python process is
> > already platform independent. Protocol violations are echo back.
>
> No, I meant a protocol that allows Emacs to act as a first-class Python front-end, not simulate a terminal, send keystrokes, use heuristics to determine what in the output stream is a prompt, REPL value, error, completion etc.
>
> For example, it's a bit silly to input multi-line code in the REPL as a sequence of individual single-line commands, when we actually are inside a text editor that can edit multi-line Python code without a problem.
>
> (I'm not suggesting that you or anybody in particular should do this; just that it's feasible. It would clearly be quite some work!)
I am not opposed to this approach, but as you wrote, it is very
different from the current inferior Python implementation.
Jupyter-emacs's approach may be similar to some extent. It uses zmq
as the communication channel.
https://github.com/emacs-jupyter/jupyter
> >> Thanks for your patches. I suggest we apply your set-tty-raw patch on master now since it cures the test failures without breaking anything else (on Mac; I'm assuming no regression elsewhere).
> >> Would you like me to do that for you?
> > Yes, please.
> Done!
Thank you!
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 23 Feb 2024 11:03:02 GMT)
Full text and
rfc822 format available.
Message #167 received at 68559 <at> debbugs.gnu.org (full text, mbox):
22 feb. 2024 kl. 17.15 skrev kobarity <kobarity <at> gmail.com>:
>> Did you find out why? Python responds nicely to TAB if run from a terminal, so what is it that we do in Emacs that prevents it from doing so? A TTY setting, or an environment variable (the TERM=dumb)?
>
> No, not exactly, but I don't think it is related to terminal settings.
Well it must be something. I'll see if I can make sense of it.
> I think this difference in behavior is the reason why Native
> completions does not work with libedit.
Thanks for explaining.
> Personally, I think this patch would be fine.
Thank you, now pushed to master.
> I think we can also improve Non-native completions. Current
> implementation sends the definition of __PYTHON_EL_get_completions()
> every time. However, sending it once during initialization should be
> sufficient. Worse, if the number of characters sent exceeds
> `comint-max-line-length', it will be sent via file. This is happening
> in your environment. Here is the log you presented.
>
> Test python-completion-at-point-1 backtrace:
> json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
>
> Thanks to the echo back, we can see __PYTHON_EL_eval_file() was used.
Right. (The need for a file is just an artefact of Comint limitations, isn't it?)
> I am not opposed to this approach, but as you wrote, it is very
> different from the current inferior Python implementation.
> Jupyter-emacs's approach may be similar to some extent. It uses zmq
> as the communication channel.
Thank you, yes, in a sense it's a (small) step towards a more notebook-like interaction model, but Comint already has some of that. This problem is not unique to Python: other REPLs have to solve the problem of multi-line input as well, and it would serve the user if they all worked in the same way (as far as the language differences allow).
In Python's case, it would liberate us from the constraints of the standard terminal REPL.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 23 Feb 2024 13:10:02 GMT)
Full text and
rfc822 format available.
Message #170 received at 68559 <at> debbugs.gnu.org (full text, mbox):
On Thu, Feb 22, 2024 at 9:56 PM Basil L. Contovounesios
<basil <at> contovou.net> wrote:
>
> Liu Hui [2024-02-22 18:31 +0800] wrote:
>
> > On Wed, Feb 21, 2024 at 10:55 PM Basil L. Contovounesios
> > <basil <at> contovou.net> wrote:
> >>
> >> Liu Hui [2024-02-21 18:00 +0800] wrote:
> >>
> >> > The attached patch should fix the problem.
> >>
> >> Thanks! The patch fixes the error, but that's because
> >> python-shell-completion-at-point-ipython is now skipped:
> >
> > Only the native completion part is skipped for the reason below.
> >
> >> in particular, python-shell-readline-completer-delims evaluates to
> >> "\s\t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?" rather than the empty string.
> >
> > "\s\t\n`..." is the delimiter used by rlcompleter, which is the
> > default completer used by the readline. rlcompleter cannot complete
> > module names or parameters, so in this case the native completion part
> > is skipped.
> >
> > The test is intended to be used with Jedi as the completion backend,
> > e.g. setting PYTHONSTARTUP="$(python -m jedi repl)", or with a custom
> > IPython completer defined in the PYTHONSTARTUP file. I have updated the
> > patch to make the test use Jedi when possible.
>
> Thanks, looks fine to me and runs without issue.
>
> The python-shell-completion-at-point-ipython test is still skipped, with
> or without specifying PYTHONSTARTUP="$(python -m jedi repl)", but like
> you suggest that's not necessarily a problem.
I have no idea why the test is skipped. You may still check if native
completion works for IPython with following steps:
1. PYTHONSTARTUP="$(python -m jedi repl)" emacs -Q
2. start Python shell with IPython interpreter, i.e.
(setq python-shell-interpreter "ipython")
(setq python-shell-interpreter-args "-i --simple-prompt")
M-x run-python
There should be text "REPL completion using Jedi xxx" before the
first prompt in the Python shell buffer, and a message "Shell
native completion is enabled.".
3. type "import ab"/"open(enc" and press TAB
expected result: "import abc"/"open(encoding="
> I'm guessing you don't have write access to emacs.git, but have signed
> the CA? If so, and if there are no other comments/objections in a few
> days, I'll apply the patch in your name.
Yes, I've signed the CA. Thanks.
> Is your GitHub username ilupin by any chance?
>
> I ask because there is one commit in emacs.git from someone with the
> same name as you, but with the email address
> ilupin <at> users.noreply.github.com (I'm guessing the commit was imported
> from eglot.git).
>
> If that was from you, do you have any objection to me mapping
> ilupin <at> users.noreply.github.com to liuhui1610 <at> gmail.com in the .mailmap
> file in emacs.git? Then all commits will show up under a single name
> and email address.
Yes. I have no objection, thanks.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 23 Feb 2024 14:41:02 GMT)
Full text and
rfc822 format available.
Message #173 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Hi Liu,
I noticed that with Non-native completions, Jedi completion is not
enabled even if PYTHONSTARTUP is set and "REPL completion using Jedi
0.18.2" is shown in inferior Python. Is this expected behavior? Jedi
completion is enabled regardless of PYTHONSTARTUP setting when I use
IPython with Non-native completions.
Mattias Engdegård wrote:
> > I think we can also improve Non-native completions. Current
> > implementation sends the definition of __PYTHON_EL_get_completions()
> > every time. However, sending it once during initialization should be
> > sufficient. Worse, if the number of characters sent exceeds
> > `comint-max-line-length', it will be sent via file. This is happening
> > in your environment. Here is the log you presented.
> >
> > Test python-completion-at-point-1 backtrace:
> > json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
> >
> > Thanks to the echo back, we can see __PYTHON_EL_eval_file() was used.
>
> Right. (The need for a file is just an artefact of Comint limitations, isn't it?)
Yes, I think it is the limitation of comint. However, if the
definition is omitted, it is unlikely to reach that limit, since all
that is required is to send the string like
'__PYTHON_EL_get_completions("subproc")'.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Mon, 26 Feb 2024 11:09:02 GMT)
Full text and
rfc822 format available.
Message #176 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Fri, Feb 23, 2024 at 10:39 PM kobarity <kobarity <at> gmail.com> wrote:
>
> Hi Liu,
>
> I noticed that with Non-native completions, Jedi completion is not
> enabled even if PYTHONSTARTUP is set and "REPL completion using Jedi
> 0.18.2" is shown in inferior Python. Is this expected behavior? Jedi
> completion is enabled regardless of PYTHONSTARTUP setting when I use
> IPython with Non-native completions.
Thanks for pointing out the problem! The attached patch should enable
Jedi completion for the non-native case.
On Fri, Feb 16, 2024 at 3:41 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > The Python shell completion relies on the readline module, which is
> > not available for Python on MS-Windows. According to the instruction
> > in python.el:
> >
> > ;; readline based shell (it's known to work with PyPy). If your
> > ;; Python installation lacks readline (like CPython for Windows),
> > ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
> > ;; should suffice. To troubleshoot why you are not getting any
> > ;; completions, you can try the following in your Python shell:
> >
> > ;; >>> import readline, rlcompleter
> >
> > ;; If you see an error, then you need to either install pyreadline or
> > ;; setup custom code that avoids that dependency.
>
> I don't know if I have CPython, but the above does show an error
> message.
>
> > It may be necessary to install pyreadline (for Python 2.7) or
> > pyreadline3 (for Python 3).
>
> I will see if I can do that, thanks.
>
> Regardless, patches to the test suite to skip the tests which rely on
> those modules, if they aren't installed, will be welcome.
Such tests should be skipped now with this patch.
[0001-Detect-the-readline-support-for-Python-shell-complet.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Mon, 26 Feb 2024 12:47:02 GMT)
Full text and
rfc822 format available.
Message #179 received at 68559 <at> debbugs.gnu.org (full text, mbox):
26 feb. 2024 kl. 12.06 skrev Liu Hui <liuhui1610 <at> gmail.com>:
> Thanks for pointing out the problem! The attached patch should enable
> Jedi completion for the non-native case.
This patch does not break anything further on macOS using the standard system Python.
I just thought you would be happy to know.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Mon, 26 Feb 2024 15:11:02 GMT)
Full text and
rfc822 format available.
Message #182 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Mattias Engdegård wrote:
>
> 26 feb. 2024 kl. 12.06 skrev Liu Hui <liuhui1610 <at> gmail.com>:
>
> > Thanks for pointing out the problem! The attached patch should enable
> > Jedi completion for the non-native case.
>
> This patch does not break anything further on macOS using the standard system Python.
> I just thought you would be happy to know.
Thanks, I confirmed that the patch enables Jedi completion for the
non-native mode.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 28 Feb 2024 14:49:02 GMT)
Full text and
rfc822 format available.
Message #185 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui [2024-02-23 21:07 +0800] wrote:
> On Thu, Feb 22, 2024 at 9:56 PM Basil L. Contovounesios
> <basil <at> contovou.net> wrote:
>>
>> The python-shell-completion-at-point-ipython test is still skipped, with
>> or without specifying PYTHONSTARTUP="$(python -m jedi repl)", but like
>> you suggest that's not necessarily a problem.
>
> I have no idea why the test is skipped. You may still check if native
> completion works for IPython with following steps:
>
> 1. PYTHONSTARTUP="$(python -m jedi repl)" emacs -Q
Oops:
$ PYTHONSTARTUP="$(python -m jedi repl)" ./src/emacs -Q
/home/blc/.pyenv/versions/3.12.2/bin/python: No module named jedi
Once I 'pip install jedi' the rest works as expected, and
python-shell-completion-at-point-ipython is no longer skipped.
>> I'm guessing you don't have write access to emacs.git, but have signed
>> the CA? If so, and if there are no other comments/objections in a few
>> days, I'll apply the patch in your name.
>
> Yes, I've signed the CA. Thanks.
Done:
Fix Python shell completion test failures
8a2d013be37 2024-02-28 15:25:56 +0100
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=8a2d013be37
>> If that was from you, do you have any objection to me mapping
>> ilupin <at> users.noreply.github.com to liuhui1610 <at> gmail.com in the .mailmap
>> file in emacs.git? Then all commits will show up under a single name
>> and email address.
>
> Yes. I have no objection, thanks.
Done:
; * .mailmap: Fix GitHub address (bug#68559#170).
1ddd9c8e29f 2024-02-28 15:30:41 +0100
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=1ddd9c8e29f
Thanks,
--
Basil
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 28 Feb 2024 14:51:02 GMT)
Full text and
rfc822 format available.
Message #188 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui [2024-02-26 19:06 +0800] wrote:
> +(add-hook 'python-shell-first-prompt-hook
> + 'python-shell-readline-detect -90)
This depth currently prepends the function to the hook, but in Emacs 24
it instead appends to the hook. Is that a problem?
> + (skip-when (null python-shell-readline-completer-delims))
skip-unless here and below?
Thanks,
--
Basil
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 06 Mar 2024 10:16:01 GMT)
Full text and
rfc822 format available.
Message #191 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Wed, Feb 28, 2024 at 10:50 PM Basil L. Contovounesios
<basil <at> contovou.net> wrote:
>
> Liu Hui [2024-02-26 19:06 +0800] wrote:
>
> > +(add-hook 'python-shell-first-prompt-hook
> > + 'python-shell-readline-detect -90)
>
> This depth currently prepends the function to the hook, but in Emacs 24
> it instead appends to the hook. Is that a problem?
Thanks for pointing it out! It is a problem because
python-shell-readline-detect is expected to be executed before
python-shell-completion-native-turn-on-maybe-with-msg, i.e. another
function in the hook. I have updated the patch.
> > + (skip-when (null python-shell-readline-completer-delims))
>
> skip-unless here and below?
Done.
[0001-Detect-the-readline-support-for-Python-shell-complet.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Fri, 08 Mar 2024 15:45:01 GMT)
Full text and
rfc822 format available.
Message #194 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui [2024-03-06 18:14 +0800] wrote:
> I have updated the patch.
Thanks! The build and tests succeed here, without skipped tests.
> + (unless (string-match-p "No readline support" output)
Nit: why not plain 'string-search' instead of a regexp search?
> - (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
> + (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
> + (string= python-shell-readline-completer-delims ""))))))
Just curious: what does the empty string signify?
[ If it's not just a dumb question, perhaps the meaning could be added
to the variable's docstring/commentary. ]
--
Basil
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Mon, 11 Mar 2024 11:38:02 GMT)
Full text and
rfc822 format available.
Message #197 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Fri, Mar 8, 2024 at 11:44 PM Basil L. Contovounesios
<basil <at> contovou.net> wrote:
>
> Liu Hui [2024-03-06 18:14 +0800] wrote:
>
> > I have updated the patch.
>
> Thanks! The build and tests succeed here, without skipped tests.
>
> > + (unless (string-match-p "No readline support" output)
>
> Nit: why not plain 'string-search' instead of a regexp search?
Done.
> > - (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
> > + (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
> > + (string= python-shell-readline-completer-delims ""))))))
>
> Just curious: what does the empty string signify?
>
> [ If it's not just a dumb question, perhaps the meaning could be added
> to the variable's docstring/commentary. ]
The empty string means no characters are considered delimiters and the
readline completion could consider the entire line of input without
breaking it into parts based on typical delimiters like spaces or
punctuation. The docstring is updated in the attached patch.
[0001-Detect-the-readline-support-for-Python-shell-complet.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Mon, 11 Mar 2024 16:04:02 GMT)
Full text and
rfc822 format available.
Message #200 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui [2024-03-11 19:35 +0800] wrote:
> On Fri, Mar 8, 2024 at 11:44 PM Basil L. Contovounesios
> <basil <at> contovou.net> wrote:
>>
>> Liu Hui [2024-03-06 18:14 +0800] wrote:
>>
>> > - (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
>> > + (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
>> > + (string= python-shell-readline-completer-delims ""))))))
>>
>> Just curious: what does the empty string signify?
>>
>> [ If it's not just a dumb question, perhaps the meaning could be added
>> to the variable's docstring/commentary. ]
>
> The empty string means no characters are considered delimiters and the
> readline completion could consider the entire line of input without
> breaking it into parts based on typical delimiters like spaces or
> punctuation. The docstring is updated in the attached patch.
Thanks! But that makes me wonder: in the cases where we check
(string= python-shell-readline-completer-delims "")
is there a possibility that the variable's value will be nil?
(In which case we should compare with equal instead of string=.)
--
Basil
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Wed, 13 Mar 2024 10:25:02 GMT)
Full text and
rfc822 format available.
Message #203 received at 68559 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Tue, Mar 12, 2024 at 12:02 AM Basil L. Contovounesios
<basil <at> contovou.net> wrote:
>
> Liu Hui [2024-03-11 19:35 +0800] wrote:
>
> > On Fri, Mar 8, 2024 at 11:44 PM Basil L. Contovounesios
> > <basil <at> contovou.net> wrote:
> >>
> >> Liu Hui [2024-03-06 18:14 +0800] wrote:
> >>
> >> > - (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
> >> > + (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
> >> > + (string= python-shell-readline-completer-delims ""))))))
> >>
> >> Just curious: what does the empty string signify?
> >>
> >> [ If it's not just a dumb question, perhaps the meaning could be added
> >> to the variable's docstring/commentary. ]
> >
> > The empty string means no characters are considered delimiters and the
> > readline completion could consider the entire line of input without
> > breaking it into parts based on typical delimiters like spaces or
> > punctuation. The docstring is updated in the attached patch.
>
> Thanks! But that makes me wonder: in the cases where we check
>
> (string= python-shell-readline-completer-delims "")
>
> is there a possibility that the variable's value will be nil?
> (In which case we should compare with equal instead of string=.)
Yes, it is a string with native completion and may be nil with
non-native completion. I have updated the patch. Thanks.
[0001-Detect-the-readline-support-for-Python-shell-complet.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Thu, 14 Mar 2024 14:25:01 GMT)
Full text and
rfc822 format available.
Message #206 received at 68559 <at> debbugs.gnu.org (full text, mbox):
Liu Hui [2024-03-13 18:21 +0800] wrote:
> On Tue, Mar 12, 2024 at 12:02 AM Basil L. Contovounesios
> <basil <at> contovou.net> wrote:
>>
>> Thanks! But that makes me wonder: in the cases where we check
>>
>> (string= python-shell-readline-completer-delims "")
>>
>> is there a possibility that the variable's value will be nil?
>> (In which case we should compare with equal instead of string=.)
>
> Yes, it is a string with native completion and may be nil with
> non-native completion.
Sorry, I completely forgot that string= works on symbols too. I was
worried that (string= nil "") would signal an error, but it actually
does what we want.
> I have updated the patch.
Thanks, pushed:
Detect the readline support for Python shell completion
a7057745f5e 2024-03-14 15:09:56 +0100
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=a7057745f5e
Is there more left to do as part of this bug report?
--
Basil
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#68559; Package
emacs.
(Sat, 16 Mar 2024 06:52:01 GMT)
Full text and
rfc822 format available.
Message #209 received at 68559 <at> debbugs.gnu.org (full text, mbox):
On Thu, Mar 14, 2024 at 10:24 PM Basil L. Contovounesios
<basil <at> contovou.net> wrote:
>
> Liu Hui [2024-03-13 18:21 +0800] wrote:
>
> > On Tue, Mar 12, 2024 at 12:02 AM Basil L. Contovounesios
> > <basil <at> contovou.net> wrote:
> >>
> >> Thanks! But that makes me wonder: in the cases where we check
> >>
> >> (string= python-shell-readline-completer-delims "")
> >>
> >> is there a possibility that the variable's value will be nil?
> >> (In which case we should compare with equal instead of string=.)
> >
> > Yes, it is a string with native completion and may be nil with
> > non-native completion.
>
> Sorry, I completely forgot that string= works on symbols too. I was
> worried that (string= nil "") would signal an error, but it actually
> does what we want.
>
> > I have updated the patch.
>
> Thanks, pushed:
>
> Detect the readline support for Python shell completion
> a7057745f5e 2024-03-14 15:09:56 +0100
> https://git.sv.gnu.org/cgit/emacs.git/commit/?id=a7057745f5e
>
> Is there more left to do as part of this bug report?
I think there is none.
Reply sent
to
Eli Zaretskii <eliz <at> gnu.org>:
You have taken responsibility.
(Sat, 16 Mar 2024 08:28:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Liu Hui <liuhui1610 <at> gmail.com>:
bug acknowledged by developer.
(Sat, 16 Mar 2024 08:28:02 GMT)
Full text and
rfc822 format available.
Message #214 received at 68559-done <at> debbugs.gnu.org (full text, mbox):
> From: Liu Hui <liuhui1610 <at> gmail.com>
> Date: Sat, 16 Mar 2024 14:49:20 +0800
> Cc: kobarity <kobarity <at> gmail.com>, Mattias Engdegård <mattias.engdegard <at> gmail.com>,
> Eli Zaretskii <eliz <at> gnu.org>, 68559 <at> debbugs.gnu.org
>
> On Thu, Mar 14, 2024 at 10:24 PM Basil L. Contovounesios
> <basil <at> contovou.net> wrote:
> >
> > Thanks, pushed:
> >
> > Detect the readline support for Python shell completion
> > a7057745f5e 2024-03-14 15:09:56 +0100
> > https://git.sv.gnu.org/cgit/emacs.git/commit/?id=a7057745f5e
> >
> > Is there more left to do as part of this bug report?
>
> I think there is none.
Thanks, closing.
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Sat, 13 Apr 2024 11:24:12 GMT)
Full text and
rfc822 format available.
This bug report was last modified 1 year and 65 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.