Subject: bug#9610: closed (Re: bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files)

From: [email-masked] (GNU bug Tracking System)
Date: Tue, 27 Sep 2011 17:25:02 +0000

Your bug report

#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

which was filed against the emacs,org-mode package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
9610: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9610
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files
From: Eli Zaretskii <[email-masked]>
To: Steve Revilak <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Tue, 27 Sep 2011 20:23:02 +0300
List-Id: <debbugs-submit.debbugs.gnu.org>

> Date: Tue, 27 Sep 2011 09:47:45 -0400
> From: Steve Revilak <[email-masked]>
> Cc: [email-masked], Bastien Guerry <[email-masked]>, [email-masked]
> 
> My bug report contained two scenarios to produce sluggish response and
> high CPU utilization.  These scenarios were
> 
>   (1) move point to the end of buffer, and start typing
> 
>   (2) Put point in line 1, column zero; press and hold the down (then up)
>   arrows to move point down (then up) the org-mode buffer.

Sorry, I missed the second one.

> Setting bidi-paragraph-direction: left-to-right eliminates the problem
> with scenario (1).  However, it has no effect on scenario (2).

Yes, there was another reason for the slowdown.  I found an
optimization of the new display that solves the problem for me with
your sample Org file.  The fix is committed as revision 105941 on the
trunk.  If you cannot or don't want to build the trunk, the patch is
reproduced below; please try it.  On my machine, the cursor motion is
now as fast as in Emacs 23 with the same file.

I'm closing the bug.  Feel free to reopen if there are any leftovers.

Thanks.

=== modified file 'src/ChangeLog'
--- src/ChangeLog	2011-09-27 08:37:07 +0000
+++ src/ChangeLog	2011-09-27 17:18:31 +0000
@@ -1,3 +1,11 @@
+2011-09-27  Eli Zaretskii  <[email-masked]>
+
+	* xdisp.c (handle_invisible_prop): If invisible text ends on a
+	newline, reseat the iterator instead of bidi-iterating there one
+	character at a time.  (Bug#9610)
+	(BUFFER_POS_REACHED_P, move_it_in_display_line_to): Bail when past
+	TO_CHARPOS if the bidi iterator is at base embedding level.
+
 2011-09-27  Andreas Schwab  <[email-masked]>
 
 	* lread.c (readevalloop): Use correct code for NBSP.

=== modified file 'src/xdisp.c'
--- src/xdisp.c	2011-09-24 16:28:25 +0000
+++ src/xdisp.c	2011-09-27 17:18:31 +0000
@@ -4056,40 +4056,67 @@ handle_invisible_prop (struct it *it)
 	  /* The position newpos is now either ZV or on visible text.  */
 	  if (it->bidi_p && newpos < ZV)
 	    {
-	      /* With bidi iteration, the region of invisible text
-		 could start and/or end in the middle of a non-base
-		 embedding level.  Therefore, we need to skip
-		 invisible text using the bidi iterator, starting at
-		 IT's current position, until we find ourselves
-		 outside the invisible text.  Skipping invisible text
-		 _after_ bidi iteration avoids affecting the visual
-		 order of the displayed text when invisible properties
-		 are added or removed.  */
-	      if (it->bidi_it.first_elt && it->bidi_it.charpos < ZV)
-		{
-		  /* If we were `reseat'ed to a new paragraph,
-		     determine the paragraph base direction.  We need
-		     to do it now because next_element_from_buffer may
-		     not have a chance to do it, if we are going to
-		     skip any text at the beginning, which resets the
-		     FIRST_ELT flag.  */
-		  bidi_paragraph_init (it->paragraph_embedding,
-				       &it->bidi_it, 1);
-		}
-	      do
+	      EMACS_INT bpos = CHAR_TO_BYTE (newpos);
+
+	      if (FETCH_BYTE (bpos) == '\n'
+		  || (newpos > BEGV && FETCH_BYTE (bpos - 1) == '\n'))
 		{
-		  bidi_move_to_visually_next (&it->bidi_it);
+		  /* If the invisible text ends on a newline or the
+		     character after a newline, we can avoid the
+		     costly, character by character, bidi iteration to
+		     newpos, and instead simply reseat the iterator
+		     there.  That's because all bidi reordering
+		     information is tossed at the newline.  This is a
+		     big win for modes that hide complete lines, like
+		     Outline, Org, etc.  (Implementation note: the
+		     call to reseat_1 is necessary, because it signals
+		     to the bidi iterator that it needs to reinit its
+		     internal information when the next element for
+		     display is requested.  */
+		  struct text_pos tpos;
+
+		  SET_TEXT_POS (tpos, newpos, bpos);
+		  reseat_1 (it, tpos, 0);
+		}
+	      else	/* Must use the slow method.  */
+		{
+		  /* With bidi iteration, the region of invisible text
+		     could start and/or end in the middle of a
+		     non-base embedding level.  Therefore, we need to
+		     skip invisible text using the bidi iterator,
+		     starting at IT's current position, until we find
+		     ourselves outside the invisible text.  Skipping
+		     invisible text _after_ bidi iteration avoids
+		     affecting the visual order of the displayed text
+		     when invisible properties are added or
+		     removed.  */
+		  if (it->bidi_it.first_elt && it->bidi_it.charpos < ZV)
+		    {
+		      /* If we were `reseat'ed to a new paragraph,
+			 determine the paragraph base direction.  We
+			 need to do it now because
+			 next_element_from_buffer may not have a
+			 chance to do it, if we are going to skip any
+			 text at the beginning, which resets the
+			 FIRST_ELT flag.  */
+		      bidi_paragraph_init (it->paragraph_embedding,
+					   &it->bidi_it, 1);
+		    }
+		  do
+		    {
+		      bidi_move_to_visually_next (&it->bidi_it);
+		    }
+		  while (it->stop_charpos <= it->bidi_it.charpos
+			 && it->bidi_it.charpos < newpos);
+		  IT_CHARPOS (*it) = it->bidi_it.charpos;
+		  IT_BYTEPOS (*it) = it->bidi_it.bytepos;
+		  /* If we overstepped NEWPOS, record its position in
+		     the iterator, so that we skip invisible text if
+		     later the bidi iteration lands us in the
+		     invisible region again. */
+		  if (IT_CHARPOS (*it) >= newpos)
+		    it->prev_stop = newpos;
 		}
-	      while (it->stop_charpos <= it->bidi_it.charpos
-		     && it->bidi_it.charpos < newpos);
-	      IT_CHARPOS (*it) = it->bidi_it.charpos;
-	      IT_BYTEPOS (*it) = it->bidi_it.bytepos;
-	      /* If we overstepped NEWPOS, record its position in the
-		 iterator, so that we skip invisible text if later the
-		 bidi iteration lands us in the invisible region
-		 again. */
-	      if (IT_CHARPOS (*it) >= newpos)
-		it->prev_stop = newpos;
 	    }
 	  else
 	    {
@@ -7880,7 +7907,9 @@ move_it_in_display_line_to (struct it *i
   ((op & MOVE_TO_POS) != 0					\
    && BUFFERP (it->object)					\
    && (IT_CHARPOS (*it) == to_charpos				\
-       || (!it->bidi_p && IT_CHARPOS (*it) > to_charpos)	\
+       || ((!it->bidi_p						\
+	    || BIDI_AT_BASE_LEVEL (it->bidi_it))		\
+	   && IT_CHARPOS (*it) > to_charpos)			\
        || (it->what == IT_COMPOSITION				\
 	   && ((IT_CHARPOS (*it) > to_charpos			\
 		&& to_charpos >= it->cmp_it.charpos)		\
@@ -7912,7 +7941,13 @@ move_it_in_display_line_to (struct it *i
       if ((op & MOVE_TO_POS) != 0
 	  && BUFFERP (it->object)
 	  && it->method == GET_FROM_BUFFER
-	  && ((!it->bidi_p && IT_CHARPOS (*it) > to_charpos)
+	  && (((!it->bidi_p
+		/* When the iterator is at base embedding level, we
+		   are guaranteed that characters are delivered for
+		   display in strictly increasing order of their
+		   buffer positions.  */
+		|| BIDI_AT_BASE_LEVEL (it->bidi_it))
+	       && IT_CHARPOS (*it) > to_charpos)
 	      || (it->bidi_p
 		  && (prev_method == GET_FROM_IMAGE
 		      || prev_method == GET_FROM_STRETCH

Subject: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files
From: Steve Revilak <[email-masked]>
To: [email-masked]
Date: Mon, 26 Sep 2011 22:50:25 -0400
List-Id: <debbugs-submit.debbugs.gnu.org>

This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgement at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':

I'd like to report an org-mode regression issue.  When working with
large .org files, Emacs 24.0.90 becomes sluggish, and consumes large
amounts of CPU.

I've attached an .org file (sample.org.gz) that causes emacs to exhibit
this behavior.  I apologize for the large attachment, but I have not
been able to reproduce this issue with small .org files.

Steps to reproduce:

(1) gunzip sample.org.gz

(2) run "top" in a terminal window.  We'll use top to monitor emacs CPU
utilization.

(3) Start emacs as "emacs -Q sample.org".  Emacs should open the file
with org-mode as the major mode.

(4) Press M-> to position point at the end of the buffer.

(5) Press RET and start typing.  You don't need to type anything
meaningful: nonsense like "fdfsds fdsfds fdsfdsf fdsfdfsds" will do.
Continue typing for 10-20 seconds.

(6) While typing notice the delay between pressing keys on the keyboard,
and the characters being displayed in the buffer.  To me, this feels
similar to typing characters into an ssh session, over a slow
communications link.

(7) While typing, observe emacs CPU utilization in top.  While
performing step (5), I observed emacs using 100% CPU.  (My machine has a
quad-core 2.66 GHz Intel Xeon processor; emacs was fully utilizing one
of the CPU cores).

(8) Repeat steps (3)--(7) with emacs 23.3.  With emacs 23.3, the delays
in (6) are not present.  As for step (7), I could not get emacs 23.3 to
use more than 2--4% CPU (as reported via top).

Uname for my system is
Linux sunny 2.6.37.6-0.7-desktop #1 SMP PREEMPT 2011-07-21 02:17:24 +0200 x86_64 x86_64 x86_64 GNU/Linux

Another thing you can try: put point in line 1, column zero.  Press and
hold the keyboard down arrow until point reaches the end of buffer;
then, press and hold the keyboard up arrow until point reaches the first
line of the buffer.  Repeat this a few times.  With emacs 24, the point
movement is `jerky', and emacs uses ~ 47% CPU (as reported by top).
With emacs 23.3, the point movement is `smooth', and emacs 23.3 uses
2--4% CPU (as reported by top).

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
     `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/local/emacs-24.0.90.1/share/emacs/24.0.90/etc/DEBUG.

In GNU Emacs 24.0.90.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.22.1)
  of 2011-09-26 on sunny
Windowing system distributor `The X.Org Foundation', version 11.0.10903000
configured using `configure  '--prefix=/usr/local/emacs-24.0.90.1/''

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: C
   value of $LC_CTYPE: nil
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: en_US.UTF-8
   value of $XMODIFIERS: @im=local
   locale-coding-system: utf-8-unix
   default enable-multibyte-characters: t

Major mode: Perl

Minor modes in effect:
   diff-auto-refine-mode: t
   shell-dirtrack-mode: t
   display-time-mode: t
   tooltip-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   column-number-mode: t
   line-number-mode: t

Recent input:
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v <down-mouse-1> <mouse-1> 
C-x k <return> <down-mouse-1> <mouse-1> M-x r e p o 
r t - e m a <tab> <return>

Recent messages:
Saving file /home/srevilak/sample.org...
Wrote /home/srevilak/sample.org
Mark set
Saving file /home/srevilak/sample.org...
Wrote /home/srevilak/sample.org
FOLDED
CHILDREN
SUBTREE
byte-code: Beginning of buffer [3 times]
scroll-up-command: End of buffer [5 times]
scroll-up-command: End of buffer

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr message sendmail rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mailabbrev mail-utils gmm-utils mailheader
emacsbug perl-mode tabify pcmpl-unix vc-bzr vc-sccs vc-svn vc-dir
org-clock dired-aux log-view vc-rcs two-column term disp-table ehelp
electric browse-url mule-util cal-move org-table dired pcvs pcvs-parse
pcvs-info pcvs-defs ewoc smerge-mode executable pcmpl-cvs ansi-color
ispell diff-mode help-mode view thingatpt reftex-parse newcomment
multi-isearch parse-time vc-cvs log-edit pcvs-util add-log vc ediff-merg
ediff-diff ediff-wind ediff-help ediff-util ediff-mult ediff-init ediff
vc-dispatcher skeleton reftex-vcr reftex-dcr reftex reftex-vars tex-mode
compile shell latexenc diary-lib diary-loaddefs cal-iso vc-git org-wl
org-w3m org-vm org-rmail org-mhe org-mew org-irc org-jsinfo org-infojs
org-html format-spec org-exp ob-exp org-exp-blocks org-info org-gnus
org-docview org-bibtex bibtex org-bbdb org-agenda org byte-opt warnings
bytecomp byte-compile cconv macroexp advice help-fns advice-preload
ob-emacs-lisp ob-tangle ob-ref ob-lob ob-table org-footnote org-src
ob-comint comint ring ob-keys ob ob-eval org-pcomplete pcomplete
org-list org-faces org-compat org-entities org-macs noutline outline
easy-mmode regexp-opt cal-menu easymenu calendar cal-loaddefs edmacro
kmacro paren time time-date tooltip ediff-hook vc-hooks lisp-float-type
mwheel x-win x-dnd tool-bar dnd fontset image fringe lisp-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer loaddefs
button faces cus-face files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)