Subject: python.el: native completion: more problems (and solutions)

From: Carlos Pita <[email-masked]>
Date: Tue, 3 Feb 2015 09:40:21 -0300

X-Debbugs-CC: [email-masked]
Tags: patch

Hi Fabian,

besides the ones l have already reported and patched in [1] there are
some remaining problems with the native completion approach:

1) The waiting timeout is too low for import completions, as imports
take a bit more time than simple dirs. But making the timeout larger
is bad for cases when there really is no completion at all for the
current prefix. I want to avoid this trade off.

2) When all the completions share a common prefix, readline completes
inline up to this common prefix, so the delete-line-command sequence
length is wrong. For example, say the current prefix is "x" and that
its completions are "xxy", "xxz". Readline will complete "xx" inline,
but delete-line-command will only delete one "x".

3) A pager could be enabled to show long lists of completions.

My solutions for 1 and 2 simply extend the dummy completion hack I
introduced in [1]:

1') Ensure there is always one completion at least (a fortiori, I'm
ensuring there are at least two completions, to force listing instead
of inline completion). So we can set a larger timeout *just for the
first* accept-process-output without the risk of waiting too much when
there is really no completion available.

2') Ensure there is a completion with a different prefix, by providing
a dummy randomly prefixed completion.

For 3 I just disabled paged completions for readline, but I don't know
how to do that for libedit.

Attached is a patch with all the described changes, applied on top of
my patch for [1]. Anyway, it should be easy to selectively apply the
changes manually.

All in all I find the solution pretty simple and it leverages my
previous trick. The completer is a bit more complex now as it has to
keep track of a couple of states in order to generate the correct
sequence of completions (dummies included):

    def completer(text, state,
                  real_completer=readline.get_completer(),
                  last_state=[None]):
        completion = None
        if state == 0:
            last_state[0] = None
        if last_state[0] is None:
            completion = real_completer(text, state)
            if not completion:
                last_state[0] = state
        if (state == last_state[0] or
            state - 1 == last_state[0] == 0):
            import random
            completion = '%s%d__dummy_completion__' % (
                text, random.randint(1000, 10000))
        return completion
    readline.set_completer(completer)

Cheers
--
Carlos

[1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19736
diff --git a/.emacs.d/lisp/python.el b/.emacs.d/lisp/python.el
index ca69b68..15041a3 100644
--- a/.emacs.d/lisp/python.el
+++ b/.emacs.d/lisp/python.el
@@ -3071,28 +3071,38 @@ When a match is found, native completion is disabled."
   "Try to setup native completion, return non-nil on success."
   (let ((process (python-shell-get-process)))
     (python-shell-send-string
-     (funcall
-      'mapconcat
-      #'identity
-      (list
-       "try:"
-       "    import readline"
-       "    def completer(text, state, c=readline.get_completer()):"
-       "        completion = c(text, state)"
-       "        if not completion and state == 1:"
-       "            return text + '__dummy_completion__'"
-       "        else:"
-       "            return completion"
-       "    readline.set_completer(completer)"
-       "    if readline.__doc__ and 'libedit' in readline.__doc__:"
-       "        readline.parse_and_bind('bind ^I rl_complete')"
-       "    else:"
-       "        readline.parse_and_bind('tab: complete')"
-       "    print ('python.el: readline is available')"
-       "    del completer, readline  # Some cleanup"
-       "except:"
-       "    print ('python.el: readline not available')")
-      "\n")
+     ;; Ensure at least two completions:
+     ;;  - Random prefix avoids common prefix inline completion
+     ;;  - Two completions avoid single inline completion
+     ;;  - One sure completion avoids waiting output timeout
+     "try:
+    import readline
+    def completer(text, state,
+                  real_completer=readline.get_completer(),
+                  last_state=[None]):
+        completion = None
+        if state == 0:
+            last_state[0] = None
+        if last_state[0] is None:
+            completion = real_completer(text, state)
+            if not completion:
+                last_state[0] = state
+        if (state == last_state[0] or
+            state - 1 == last_state[0] == 0):
+            import random
+            completion = '%s%d__dummy_completion__' % (
+                text, random.randint(1000, 10000))
+        return completion
+    readline.set_completer(completer)
+    if readline.__doc__ and 'libedit' in readline.__doc__:
+        readline.parse_and_bind('bind ^I rl_complete')
+    else:
+        readline.parse_and_bind('tab: complete')
+        readline.parse_and_bind('set page-completions off')
+    print('python.el: readline is available')
+    del completer, readline  # Env cleanup
+except:
+    print('python.el: readline not available')"
      process)
     (python-shell-accept-process-output process)
     (when (save-excursion
@@ -3206,9 +3216,7 @@ completion."
                             #'comint-redirect-filter original-filter-fn))
                 (set-process-filter process #'comint-redirect-filter))
               (process-send-string process input-to-send)
-              (accept-process-output
-               process
-               python-shell-completion-native-output-timeout)
+              (accept-process-output process 0.3)
               ;; XXX: can't use `python-shell-accept-process-output'
               ;; here because there are no guarantees on how output
               ;; ends.  The workaround here is to call
@@ -3217,13 +3225,12 @@ completion."
               (while (accept-process-output
                       process
                       python-shell-completion-native-output-timeout))
-              (cl-remove
-               (concat input "__dummy_completion__")
+              (cl-remove-if
+               (lambda (c) (string-match "__dummy_completion__" c))
                (split-string
                 (buffer-substring-no-properties
                  (point-min) (point-max))
-                separators t)
-               :test #'string=)))
+                separators t))))
         (set-process-filter process original-filter-fn)))))
 
 (defun python-shell-completion-get-completions (process import input)