GNU bug report logs - #15295
which-func-mode slow in long Python tuple

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 15295 in the body.
You can then email your comments to 15295 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dale <dale <at> codefu.org>
To: bug-gnu-emacs <at> gnu.org
Subject: which-func-mode slow in long Python tuple
Date: Fri, 06 Sep 2013 19:47:19 -0500

	I happen to have a Python source file that has a relatively long tuple 
at the module top level, i.e. a Python source file containing:

----------
foo = (
    "item 1",
    "item 2",
    # ...and so on for ~500 lines
)
----------

I also use which-function-mode.  If I go to the end of that tuple and 
move the cursor in to it, Emacs becomes unusably slow.  It will appear 
to lock up and eat 100% CPU for 10-20 seconds each time I move the 
cursor within the end of that tuple.  Emacs remains responsive at the 
top of the tuple.

	I think this is happening because python-info-current-defun is slow 
when dealing with long tuples.  (Maybe lists, dicts, and other things 
too; I only tested tuples.)  Here's some elisp to produce a test case 
and benchmark python-info-current-defun:

----------
(progn
  (set-buffer (generate-new-buffer "*test*"))
  (python-mode)
  (insert "foo = (\n")
  (dotimes (_ 500) (insert "    \"bar\",\n"))
  (insert ")\n")
  (forward-line -2)
  (message "%S" (benchmark-run (python-info-current-defun))))
----------

This makes a python-mode buffer named "*test*" containing only a 
500-item Python tuple, as in my above example.  On my hardware, the 
above benchmark-run yields a result such as "(7.364507 131 
0.9572049999999979)", i.e. 7.3 seconds to run.

	Once that *test* buffer is created, feel free to turn on 
which-function-mode in there and see that Emacs locks up every time you 
move the cursor around in the end of that tuple.  (which-function-mode 
seems to be taking about twice the time reported by benchmark-run. 
Perhaps it's calling python-info-current-defun twice?)

	I have reproduced this behavior with "emacs -Q" using an Emacs I just 
built from trunk, looks like revision 114162.  (I get Emacs from Git, 
where the master branch is 0f1532f2fe2.)  I have also reproduced this 
with python.el from the emacs-24 branch, looks like revision 111403.

	Thanks to everyone who develops Emacs, an indispensable tool for me!

Dale

Message #8 received at 15295 <at> debbugs.gnu.org (full text, mbox):

From: Alex V. Koval <alex <at> ua2web.com>
To: 15295 <at> debbugs.gnu.org
Subject: python mode slow to unusable
Date: Sat, 26 Oct 2013 12:16:56 +0300

For me this is happen as well. Emacs, starting from version 24.3 became
so slow in Python mode that I had to tell all developers at our company
to use version 24.2 until I sorted this out.

Sit today and started trying various emacs versions, and calling
different functions. The suggested test case from original author above,
runs with this benchmark:

(7.3956507 53 1.8788885930000063)

In fact, when I enable which-function-mode and just try to open
one of our project files, it reads it 62 seconds. *Same* file opened
with emacs 24.2 reads < 1second. 

Same thing happens when I try to call 'help-imenu' - 46 seconds. In
emacs 24.2 - less then 1 second.

I have this bug in version 24.3 and 'bzr' current:

 * Emacs branch: trunk
 * Revision: 114814
 * Emacs version number: 24.3.50

Please tell me what additional information should I provide.
Not very big expert in Lisp but may try to debug it more
to detail.

WBR, Alex

Message #11 received at 15295 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Alex V. Koval <alex <at> ua2web.com>
Cc: 15295 <at> debbugs.gnu.org
Subject: Re: bug#15295: python mode slow to unusable
Date: Sat, 26 Oct 2013 13:39:21 +0200

Alex V. Koval <alex <at> ua2web.com> writes:

> For me this is happen as well. Emacs, starting from version 24.3 became
> so slow in Python mode that I had to tell all developers at our company
> to use version 24.2 until I sorted this out.
>
> Sit today and started trying various emacs versions, and calling
> different functions. The suggested test case from original author above,
> runs with this benchmark:
>
> (7.3956507 53 1.8788885930000063)

I profiled a bit, and, at least in this example, these two functions
seem to be extremely inefficient in combination:

(defun python-nav-beginning-of-statement ()
  "Move to start of current statement."
  (interactive "^")
  (while (and (or (back-to-indentation) t)
              (not (bobp))
              (when (or
                     (save-excursion
                       (forward-line -1)
                       (python-info-line-ends-backslash-p))
                     (python-syntax-context 'string)
                     (python-syntax-context 'paren))
                (forward-line -1))))
  (point-marker))

(defun python-info-line-ends-backslash-p (&optional line-number)
  "Return non-nil if current line ends with backslash.
With optional argument LINE-NUMBER, check that line instead."
  (save-excursion
    (save-restriction
      (widen)
      (when line-number
        (python-util-goto-line line-number))
      (while (and (not (eobp))
                  (goto-char (line-end-position))
                  (python-syntax-context 'paren)
                  (not (equal (char-before (point)) ?\\)))
        (forward-line 1))
      (when (equal (char-before) ?\\)
        (point-marker)))))

They consume most of the time used.  While the first function goes
backward, the second goes forward to the end in every loop cycle.  This
makes the thing O(n^2), with n being the number of lines of the
expression.

I don't know Python, so I can't make any suggestions.  Who can?  At
least, changing the order of `or' expressions in
`python-nav-beginning-of-statement' seems to help in the example case:

(defun python-nav-beginning-of-statement ()
  "Move to start of current statement."
  (interactive "^")
  (while (and (or (back-to-indentation) t)
              (not (bobp))
              (when (or
                     (python-syntax-context 'string)
                     (python-syntax-context 'paren)
                     (save-excursion
                       (forward-line -1)
                       (python-info-line-ends-backslash-p)))
                (forward-line -1))))
  (point-marker))

It's also not efficient how often `syntax-ppss' is called all the time.


Regards,

Michael.

Message #14 received at 15295 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Fabián Ezequiel Gallina <fgallina <at> gnu.org>
Cc: 15295 <at> debbugs.gnu.org
Subject: Re: bug#15295: python mode slow to unusable
Date: Sun, 27 Oct 2013 00:23:51 -0400

Could you take a look at this, as well?


        Stefan

>>>>> "Michael" == Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> Alex V. Koval <alex <at> ua2web.com> writes:
>> For me this is happen as well. Emacs, starting from version 24.3 became
>> so slow in Python mode that I had to tell all developers at our company
>> to use version 24.2 until I sorted this out.
>> 
>> Sit today and started trying various emacs versions, and calling
>> different functions. The suggested test case from original author above,
>> runs with this benchmark:
>> 
>> (7.3956507 53 1.8788885930000063)

> I profiled a bit, and, at least in this example, these two functions
> seem to be extremely inefficient in combination:

> (defun python-nav-beginning-of-statement ()
>   "Move to start of current statement."
>   (interactive "^")
>   (while (and (or (back-to-indentation) t)
>               (not (bobp))
>               (when (or
>                      (save-excursion
>                        (forward-line -1)
>                        (python-info-line-ends-backslash-p))
>                      (python-syntax-context 'string)
>                      (python-syntax-context 'paren))
>                 (forward-line -1))))
>   (point-marker))

> (defun python-info-line-ends-backslash-p (&optional line-number)
>   "Return non-nil if current line ends with backslash.
> With optional argument LINE-NUMBER, check that line instead."
>   (save-excursion
>     (save-restriction
>       (widen)
>       (when line-number
>         (python-util-goto-line line-number))
>       (while (and (not (eobp))
>                   (goto-char (line-end-position))
>                   (python-syntax-context 'paren)
>                   (not (equal (char-before (point)) ?\\)))
>         (forward-line 1))
>       (when (equal (char-before) ?\\)
>         (point-marker)))))

> They consume most of the time used.  While the first function goes
> backward, the second goes forward to the end in every loop cycle.  This
> makes the thing O(n^2), with n being the number of lines of the
> expression.

> I don't know Python, so I can't make any suggestions.  Who can?  At
> least, changing the order of `or' expressions in
> `python-nav-beginning-of-statement' seems to help in the example case:

> (defun python-nav-beginning-of-statement ()
>   "Move to start of current statement."
>   (interactive "^")
>   (while (and (or (back-to-indentation) t)
>               (not (bobp))
>               (when (or
>                      (python-syntax-context 'string)
>                      (python-syntax-context 'paren)
>                      (save-excursion
>                        (forward-line -1)
>                        (python-info-line-ends-backslash-p)))
>                 (forward-line -1))))
>   (point-marker))

> It's also not efficient how often `syntax-ppss' is called all the time.


> Regards,

> Michael.

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Röhler <andreas.roehler <at> easy-emacs.de>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#15295: python mode slow to unusable
Date: Sun, 27 Oct 2013 09:01:13 +0100

Am 27.10.2013 05:23, schrieb Stefan Monnier:
> Could you take a look at this, as well?
>
>
>          Stefan
>
>>>>>> "Michael" == Michael Heerdegen <michael_heerdegen <at> web.de> writes:
>
>> Alex V. Koval <alex <at> ua2web.com> writes:
>>> For me this is happen as well. Emacs, starting from version 24.3 became
>>> so slow in Python mode that I had to tell all developers at our company
>>> to use version 24.2 until I sorted this out.
>>>
>>> Sit today and started trying various emacs versions, and calling
>>> different functions. The suggested test case from original author above,
>>> runs with this benchmark:
>>>
>>> (7.3956507 53 1.8788885930000063)
>
>> I profiled a bit, and, at least in this example, these two functions
>> seem to be extremely inefficient in combination:
>
>> (defun python-nav-beginning-of-statement ()
>>    "Move to start of current statement."
>>    (interactive "^")
>>    (while (and (or (back-to-indentation) t)
>>                (not (bobp))
>>                (when (or
>>                       (save-excursion
>>                         (forward-line -1)
>>                         (python-info-line-ends-backslash-p))
>>                       (python-syntax-context 'string)
>>                       (python-syntax-context 'paren))
>>                  (forward-line -1))))
>>    (point-marker))
>
>> (defun python-info-line-ends-backslash-p (&optional line-number)
>>    "Return non-nil if current line ends with backslash.
>> With optional argument LINE-NUMBER, check that line instead."
>>    (save-excursion
>>      (save-restriction
>>        (widen)
>>        (when line-number
>>          (python-util-goto-line line-number))
>>        (while (and (not (eobp))
>>                    (goto-char (line-end-position))
>>                    (python-syntax-context 'paren)
>>                    (not (equal (char-before (point)) ?\\)))
>>          (forward-line 1))
>>        (when (equal (char-before) ?\\)
>>          (point-marker)))))
>
>> They consume most of the time used.  While the first function goes
>> backward, the second goes forward to the end in every loop cycle.  This
>> makes the thing O(n^2), with n being the number of lines of the
>> expression.
>
>> I don't know Python, so I can't make any suggestions.  Who can?  At
>> least, changing the order of `or' expressions in
>> `python-nav-beginning-of-statement' seems to help in the example case:
>
>> (defun python-nav-beginning-of-statement ()
>>    "Move to start of current statement."
>>    (interactive "^")
>>    (while (and (or (back-to-indentation) t)
>>                (not (bobp))
>>                (when (or
>>                       (python-syntax-context 'string)
>>                       (python-syntax-context 'paren)
>>                       (save-excursion
>>                         (forward-line -1)
>>                         (python-info-line-ends-backslash-p)))
>>                  (forward-line -1))))
>>    (point-marker))
>
>> It's also not efficient how often `syntax-ppss' is called all the time.
>
>
>> Regards,
>
>> Michael.
>
>
>
>
>
>

IMO it's a matter of coding style.

IIUC Emacs hackers should be warned somewhere in Elisp manual to code like

python-syntax-context

does. Python.el is not the only place where it's done like this.

It looks nice, but seems to port some dangers WRT speed.

Message #22 received at 15295-done <at> debbugs.gnu.org (full text, mbox):

From: fgallina <at> gnu.org (Fabián Ezequiel Gallina)
To: 15295-done <at> debbugs.gnu.org
Date: Tue, 24 Dec 2013 17:08:09 -0300

Fixed in revno 115736.

Thanks Dale for such detailed recipe.

This patch banishes initial thoughts of `python-syntax-context' being a
bad idea.  `python-syntax-context' is nothing than a thin semantic
wrapper over `syntax-ppss'. It makes code easier to grasp for newcomers
to Elisp and has almost no impact on itself, it's optional argument is a
`syntax-ppss' list which can be used instead to lower the amount of
calls to it (as it is happening in this new patch I've just committed).

The problem here was that `python-nav-beginning-of-statement' was coded
awfully (looking for the statement beginning line by line). Now it
should be extremely fast compared to that.

Using OP's suggested recipe, here are the elp results for when
which-func is triggered inside the big tuple:

    python-info-current-defun                      2           0.003719249   0.0018596245
    python-nav-beginning-of-defun                  2           0.0036946010  0.0018473005
    python-nav--beginning-of-defun                 2           0.003685751   0.0018428755
    python-nav-backward-block                      2           0.001836524   0.000918262
    python-nav-forward-block                       2           0.0018315750  0.0009157875
    python-info-looking-at-beginning-of-defun      6           0.000889166   0.0001481943
    python-nav-beginning-of-statement              4           0.000437251   0.0001093127
    python-syntax-context-type                     6           5.009e-06     8.348...e-07

And this is the benchmark-run result: (0.020715153 0 0.0)


Regards,
Fabián.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.