GNU bug report logs - #68446
29.1.90; Bidi right-to-left paragraphs missing text in Org mode

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 68446 in the body.
You can then email your comments to 68446 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Thamer Mahmoud <thamer.mahmoud <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode
Date: Sun, 14 Jan 2024 13:21:57 +0300

Hi,

1. In emacs -Q, create an empty buffer with Org mode active and type:

a [[link]]

2. Evaluate: (setq bidi-paragraph-direction 'right-to-left)
3. Note the "a" and link are no longer visible.

In GNU Emacs 29.1.90 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.39, cairo version 1.18.0)
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12101009
System Description: Debian GNU/Linux trixie/sid

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
LIBSELINUX LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP
X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

-- 
Regards,
Thamer

Message #8 received at 68446 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Thamer Mahmoud <thamer.mahmoud <at> gmail.com>,
 Ihor Radchenko <yantar92 <at> posteo.net>
Cc: 68446 <at> debbugs.gnu.org
Subject: Re: bug#68446: 29.1.90;
 Bidi right-to-left paragraphs missing text in Org mode
Date: Sun, 14 Jan 2024 13:24:59 +0200

> From: Thamer Mahmoud <thamer.mahmoud <at> gmail.com>
> Date: Sun, 14 Jan 2024 13:21:57 +0300
> 
> 1. In emacs -Q, create an empty buffer with Org mode active and type:
> 
> a [[link]]
> 
> 2. Evaluate: (setq bidi-paragraph-direction 'right-to-left)
> 3. Note the "a" and link are no longer visible.

I think it's an Org bug: it should prevent bidi reordering inside the
"[[link]]" string.  For example, wrap the "[[link]]" thing in LRO..PDF
bidi controls.  Because without that, the brackets can be mirrored by
bidi reordering and the BPA algorithm, and the link is no longer in
the form that Org expects.  The result is that the entire text becomes
invisible.

A work-around is to do one of the following:

  . insert one or more L2R characters after the "[[link]]", or
  . set bidi-inhibit-bpa to a non-nil value

I don't see an Emacs bug here, surprising as it may sound.  Lisp
programs that depend on particular sequence of characters on display
should be aware that bidi reordering can affect that.

Adding Ihor to the discussion.

Message #11 received at 68446 <at> debbugs.gnu.org (full text, mbox):

From: Ihor Radchenko <yantar92 <at> posteo.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 68446 <at> debbugs.gnu.org, Thamer Mahmoud <thamer.mahmoud <at> gmail.com>
Subject: Re: bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text
 in Org mode
Date: Sun, 14 Jan 2024 12:11:03 +0000

Eli Zaretskii <eliz <at> gnu.org> writes:

>> 1. In emacs -Q, create an empty buffer with Org mode active and type:
>> 
>> a [[link]]
>> 
>> 2. Evaluate: (setq bidi-paragraph-direction 'right-to-left)
>> 3. Note the "a" and link are no longer visible.
>
> I think it's an Org bug: it should prevent bidi reordering inside the
> "[[link]]" string.  For example, wrap the "[[link]]" thing in LRO..PDF
> bidi controls.  Because without that, the brackets can be mirrored by
> bidi reordering and the BPA algorithm, and the link is no longer in
> the form that Org expects.  The result is that the entire text becomes
> invisible.

May you please elaborate? I do not see anything unexpected in the text
properties that Org mode applies to the link and the letter "a".

In fact, after following the reproducer, and enabling visible-mode, M-x
describe-text-properties on "a" yields

    Text content at position 101:
    
    
    There are text properties here:
      fontified            t

Yet, "a" is rendered invisible.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message #14 received at 68446 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ihor Radchenko <yantar92 <at> posteo.net>
Cc: 68446 <at> debbugs.gnu.org, thamer.mahmoud <at> gmail.com
Subject: Re: bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text
 in Org mode
Date: Sun, 14 Jan 2024 15:00:56 +0200

> From: Ihor Radchenko <yantar92 <at> posteo.net>
> Cc: Thamer Mahmoud <thamer.mahmoud <at> gmail.com>, 68446 <at> debbugs.gnu.org
> Date: Sun, 14 Jan 2024 12:11:03 +0000
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > I think it's an Org bug: it should prevent bidi reordering inside the
> > "[[link]]" string.  For example, wrap the "[[link]]" thing in LRO..PDF
> > bidi controls.  Because without that, the brackets can be mirrored by
> > bidi reordering and the BPA algorithm, and the link is no longer in
> > the form that Org expects.  The result is that the entire text becomes
> > invisible.
> 
> May you please elaborate? I do not see anything unexpected in the text
> properties that Org mode applies to the link and the letter "a".

(For some value of "expected".  Me, I don't understand more than half
of the properties there, wrt their expected effect on the text and its
display.  Are you surprised?)

Anyway, after performing the recipe, do this:

  . M-<
  . C-x =
  . C-f
  . C-x =
  . C-f
  . C-x =

etc.  IOW, move 1 buffer position forward at a time from BOB, and each
time show the character at that position.  Do you see something
strange?

Then do the same in a buffer under Fundamental mode with the same text
and the same value of bidi-paragraph-direction.  Can you explain what
Org does to cause the difference, which makes some characters be
"skipped" by C-f (due to point-adjustment feature), and probably also
makes all the characters invisible as result?

I can tell you what I see when stepping through the display code: it
skips all the characters as invisible.  I see that org-link is in the
buffer-invisibility-spec, but that's probably just the tip of the
iceberg, so I'm asking you below to describe what are all those faces
and properties for, and how they are related to the invisibility.

> In fact, after following the reproducer, and enabling visible-mode, M-x
> describe-text-properties on "a" yields
> 
>     Text content at position 101:
>     
>     
>     There are text properties here:
>       fontified            t
> 
> Yet, "a" is rendered invisible.

When you invoke visible-mode, the text is magically shown as expected.
Can you explain why is that?

If you invoke describe-text-properties on any character in the buffer,
there's no character with 'invisible' face, either.  And yet they are
invisible.  How come?

If you describe in enough details how the [[link]] thing is handled by
Org, maybe I could elaborate more than the general observations and
questions above.

To summarize, my point is simple: the Emacs bidirectional display is
responsible for reordering plain text with faces, and while doing so,
it more-or-less ignores invisible text.  Anything fancier: overlays,
images, text properties that affect display in some other ways,
etc. -- all these are not guaranteed to behave in reordered text as
they do in strict L2R text, and Lisp programs which create visual
effects using those fancier features should look out for problems, and
should use bidi formatting controls judiciously to prevent such
disasters.

(My guess is that the brackets in this case are reordered, and as
result cause everything to become invisible.  But that's just a guess,
and I don't expect to understand why without a lot more background and
help from Org experts.)

Message #17 received at 68446 <at> debbugs.gnu.org (full text, mbox):

From: Ihor Radchenko <yantar92 <at> posteo.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 68446 <at> debbugs.gnu.org, thamer.mahmoud <at> gmail.com
Subject: Re: bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text
 in Org mode
Date: Sun, 14 Jan 2024 14:41:21 +0000

Eli Zaretskii <eliz <at> gnu.org> writes:

>> May you please elaborate? I do not see anything unexpected in the text
>> properties that Org mode applies to the link and the letter "a".
>
> ...
> If you describe in enough details how the [[link]] thing is handled by
> Org, maybe I could elaborate more than the general observations and
> questions above.

Let's simplify the reproducer, getting rid of Org mode:

(let ((str (concat "a " (propertize "[" 'invisible t) "b" (propertize "]" 'invisible t))))
  (with-temp-buffer
    (display-buffer (current-buffer))
    (insert str)
    (read-char "'a b' is visible. Press any key.")
    (setq bidi-paragraph-direction 'right-to-left)
    (read-char "'a b' is invisible. Press any key.")))

Is this the above expected?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message #20 received at 68446 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ihor Radchenko <yantar92 <at> posteo.net>
Cc: 68446 <at> debbugs.gnu.org, thamer.mahmoud <at> gmail.com
Subject: Re: bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text
 in Org mode
Date: Sun, 14 Jan 2024 16:58:43 +0200

> From: Ihor Radchenko <yantar92 <at> posteo.net>
> Cc: thamer.mahmoud <at> gmail.com, 68446 <at> debbugs.gnu.org
> Date: Sun, 14 Jan 2024 14:41:21 +0000
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> May you please elaborate? I do not see anything unexpected in the text
> >> properties that Org mode applies to the link and the letter "a".
> >
> > ...
> > If you describe in enough details how the [[link]] thing is handled by
> > Org, maybe I could elaborate more than the general observations and
> > questions above.

Actually, I think I see the reason, and the problem is indeed in the
display code's logic in this case.

> Let's simplify the reproducer, getting rid of Org mode:
> 
> (let ((str (concat "a " (propertize "[" 'invisible t) "b" (propertize "]" 'invisible t))))
>   (with-temp-buffer
>     (display-buffer (current-buffer))
>     (insert str)
>     (read-char "'a b' is visible. Press any key.")
>     (setq bidi-paragraph-direction 'right-to-left)
>     (read-char "'a b' is invisible. Press any key.")))
> 
> Is this the above expected?

No.

Message #23 received at 68446 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: yantar92 <at> posteo.net, thamer.mahmoud <at> gmail.com
Cc: 68446 <at> debbugs.gnu.org
Subject: Re: bug#68446: 29.1.90;
 Bidi right-to-left paragraphs missing text in Org mode
Date: Sun, 04 Feb 2024 11:51:11 +0200

> Cc: 68446 <at> debbugs.gnu.org, thamer.mahmoud <at> gmail.com
> Date: Sun, 14 Jan 2024 16:58:43 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > From: Ihor Radchenko <yantar92 <at> posteo.net>
> > Cc: thamer.mahmoud <at> gmail.com, 68446 <at> debbugs.gnu.org
> > Date: Sun, 14 Jan 2024 14:41:21 +0000
> > 
> > Eli Zaretskii <eliz <at> gnu.org> writes:
> > 
> > >> May you please elaborate? I do not see anything unexpected in the text
> > >> properties that Org mode applies to the link and the letter "a".
> > >
> > > ...
> > > If you describe in enough details how the [[link]] thing is handled by
> > > Org, maybe I could elaborate more than the general observations and
> > > questions above.
> 
> Actually, I think I see the reason, and the problem is indeed in the
> display code's logic in this case.

There was an algorithmic flaw in handling invisible property when the
invisible text starts inside embedding level above the base paragraph
level: the code failed to account for the "backward" iteration which
is part of the bidi reordering, and instead always looked forward for
the first visible position.

> > Let's simplify the reproducer, getting rid of Org mode:
> > 
> > (let ((str (concat "a " (propertize "[" 'invisible t) "b" (propertize "]" 'invisible t))))
> >   (with-temp-buffer
> >     (display-buffer (current-buffer))
> >     (insert str)
> >     (read-char "'a b' is visible. Press any key.")
> >     (setq bidi-paragraph-direction 'right-to-left)
> >     (read-char "'a b' is invisible. Press any key.")))
> > 
> > Is this the above expected?
> 
> No.

Should be fixed now on the master branch.

Message #26 received at 68446 <at> debbugs.gnu.org (full text, mbox):

From: Ihor Radchenko <yantar92 <at> posteo.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 68446 <at> debbugs.gnu.org, thamer.mahmoud <at> gmail.com
Subject: Re: bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text
 in Org mode
Date: Sun, 04 Feb 2024 13:10:14 +0000

Eli Zaretskii <eliz <at> gnu.org> writes:

> Should be fixed now on the master branch.

I confirm that I can no longer reproduce the problem following the
original steps and my reproducer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message #31 received at 68446-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ihor Radchenko <yantar92 <at> posteo.net>
Cc: 68446-done <at> debbugs.gnu.org, thamer.mahmoud <at> gmail.com
Subject: Re: bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text
 in Org mode
Date: Sun, 04 Feb 2024 15:43:33 +0200

> From: Ihor Radchenko <yantar92 <at> posteo.net>
> Cc: thamer.mahmoud <at> gmail.com, 68446 <at> debbugs.gnu.org
> Date: Sun, 04 Feb 2024 13:10:14 +0000
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > Should be fixed now on the master branch.
> 
> I confirm that I can no longer reproduce the problem following the
> original steps and my reproducer.

Thanks, I'm therefore closing this bug.  We can reopen if someone
reports some issues or left-overs.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.