Subject: Re: bug#10183: small bidi problems in the org mode and in the mini buffer

From: Eli Zaretskii <[email-masked]>
Date: Sat, 03 Dec 2011 10:12:26 +0200

> Date: Thu, 1 Dec 2011 14:13:52 -0800 (PST)
> From: nabil-82 <[email-masked]>
> 
> Let's take o1.org, in the ("SHOW ALL")- form  
> --- o1.org ---
> * א
> א
> 
>               e *
>                  e
> --- o1.org ---
> 
> (the logical order is:
>  * - space - Alef - newline - 
>  Alef - newline - 
>  newline - 
>  * - space - e - newline - 
>  e - newline )
> 
> So, everything is fine. But if both items are folded:
> 
> a) I see this by starting with "emacs":
> --- o1.org ---
> א *...
> * e...
> --- o1.org ---
> So, we have problems:
> The first heading is not on the right and the three points are not on the
> left. By typing TAB in the first heading it jumps to correct side (the right
> side and the points disappear).

This bug should be fixed in revision 106587 on the trunk.  The patch
appears near the end of this message.

This completes the solution of the problems described in the original
report, so I'm closing the bug.  In the future, please file separate
bug reports about unrelated problems.

> I expect this - and I get it by 24.0.90:
> 
> --- o1.org ---
>            ...א *
> * e...
> --- o1.org ---

I introduced a display optimization between 24.0.90 and 24.0.91, which
made Org Mode much faster, but it also introduced this bug, now fixed.

> - I've just tested it with emacs -Q:
> org doesn't know there at all that the line in arabic or hebrew starts from
> right. It writes the words correctly but doesn't realize that it is an
> rtl-paragraph.

By default, Org Mode forces left-to-right paragraph direction.  (I
guess you have bidi-paragraph-direction reset to nil somewhere.)  This
is done on purpose: Org buffers can look and behave in confusing ways
when the paragraph direction of the heading lines is sometimes LTR and
sometimes RTL.  Some of the more advanced Org features may not live
happily with this.  So I recommend sticking to the default LTR
paragraph direction in Org buffers.

=== modified file 'src/ChangeLog'
--- src/ChangeLog	2011-12-02 10:19:49 +0000
+++ src/ChangeLog	2011-12-03 07:59:23 +0000
@@ -1,3 +1,9 @@
+2011-12-03  Eli Zaretskii  <[email-masked]>
+
+	* xdisp.c (handle_invisible_prop): If the invisible text ends just
+	before a newline, prepare the bidi iterator for consuming the
+	newline, and keep the current paragraph direction.  (Bug#10183)
+
 2011-12-02  Juri Linkov  <[email-masked]>
 
 	* search.c (Fword_search_regexp): New Lisp function created from

=== modified file 'src/xdisp.c'
--- src/xdisp.c	2011-11-27 04:43:11 +0000
+++ src/xdisp.c	2011-12-03 07:59:23 +0000
@@ -4093,26 +4093,37 @@ handle_invisible_prop (struct it *it)
 	  if (it->bidi_p && newpos < ZV)
 	    {
 	      EMACS_INT bpos = CHAR_TO_BYTE (newpos);
-
-	      if (FETCH_BYTE (bpos) == '\n'
-		  || (newpos > BEGV && FETCH_BYTE (bpos - 1) == '\n'))
+	      int on_newline = FETCH_BYTE (bpos) == '\n';
+	      int after_newline =
+		newpos <= BEGV || FETCH_BYTE (bpos - 1) == '\n';
+
+	      /* If the invisible text ends on a newline or on a
+		 character after a newline, we can avoid the costly,
+		 character by character, bidi iteration to NEWPOS, and
+		 instead simply reseat the iterator there.  That's
+		 because all bidi reordering information is tossed at
+		 the newline.  This is a big win for modes that hide
+		 complete lines, like Outline, Org, etc.  */
+	      if (on_newline || after_newline)
 		{
-		  /* If the invisible text ends on a newline or the
-		     character after a newline, we can avoid the
-		     costly, character by character, bidi iteration to
-		     newpos, and instead simply reseat the iterator
-		     there.  That's because all bidi reordering
-		     information is tossed at the newline.  This is a
-		     big win for modes that hide complete lines, like
-		     Outline, Org, etc.  (Implementation note: the
-		     call to reseat_1 is necessary, because it signals
-		     to the bidi iterator that it needs to reinit its
-		     internal information when the next element for
-		     display is requested.  */
 		  struct text_pos tpos;
+		  bidi_dir_t pdir = it->bidi_it.paragraph_dir;
 
 		  SET_TEXT_POS (tpos, newpos, bpos);
 		  reseat_1 (it, tpos, 0);
+		  /* If we reseat on a newline, we need to prep the
+		     bidi iterator for advancing to the next character
+		     after the newline, keeping the current paragraph
+		     direction (so that PRODUCE_GLYPHS does TRT wrt
+		     prepending/appending glyphs to a glyph row).  */
+		  if (on_newline)
+		    {
+		      it->bidi_it.first_elt = 0;
+		      it->bidi_it.paragraph_dir = pdir;
+		      it->bidi_it.ch = '\n';
+		      it->bidi_it.nchars = 1;
+		      it->bidi_it.ch_len = 1;
+		    }
 		}
 	      else	/* Must use the slow method.  */
 		{
@@ -4121,11 +4132,11 @@ handle_invisible_prop (struct it *it)
 		     non-base embedding level.  Therefore, we need to
 		     skip invisible text using the bidi iterator,
 		     starting at IT's current position, until we find
-		     ourselves outside the invisible text.  Skipping
-		     invisible text _after_ bidi iteration avoids
-		     affecting the visual order of the displayed text
-		     when invisible properties are added or
-		     removed.  */
+		     ourselves outside of the invisible text.
+		     Skipping invisible text _after_ bidi iteration
+		     avoids affecting the visual order of the
+		     displayed text when invisible properties are
+		     added or removed.  */
 		  if (it->bidi_it.first_elt && it->bidi_it.charpos < ZV)
 		    {
 		      /* If we were `reseat'ed to a new paragraph,