GNU bug report logs - #69385
30.0.50; Long lines with bidi text slow down Emacs

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 69385 in the body.
You can then email your comments to 69385 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: bug-gnu-emacs <at> gnu.org
Subject: 30.0.50; Long lines with bidi text slow down Emacs
Date: Sun, 25 Feb 2024 17:23:11 +0100

Displaying a buffer that contains a long line with bidirectional text
greatly slows down Emacs.  A simple reproduction is to copy the the
Arabic example from etc/HELLO (`C-h h'), yank it into a buffer
(fundamental-mode suffices), add " Hello ", and then create a single
line consisting of a large number of copies of these strings; on my
machine 500 copies clearly shows the slowdown, and with 800 copies it is
much worse.

There is no slowdown with a line of the same length consisting only of
RTL or only of LTR text, nor with the above test line when
bidi-display-reordering is set to nil in the buffer (but then, of
course, the Arabic is not displayed correctly).  It seems that the long
line optimizations added to Emacs 29 do not work with bidirectional
text.

(FTR, I encountered this issue with a program of mine that generates
Emacs Lisp files containing such long lines with bidirectional text.
These files are not intended for display but I was examining one and
experienced the slowdown.)


In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.38, cairo version 1.18.0) of 2024-02-20 built on strobelfs2
Repository revision: d9afa1f30fdf9d00b447fea0a8343397333e172f
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101009
System Description: Linux From Scratch r12.0-112

Configured using:
 'configure -C --with-xwidgets 'CFLAGS=-Og -g3'
 PKG_CONFIG_PATH=/opt/qt5/lib/pkgconfig'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER
PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS
TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM XWIDGETS GTK3 ZLIB

Message #8 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Sun, 25 Feb 2024 19:06:55 +0200

> Date: Sun, 25 Feb 2024 17:23:11 +0100
> From:  Stephen Berman via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> Displaying a buffer that contains a long line with bidirectional text
> greatly slows down Emacs.  A simple reproduction is to copy the the
> Arabic example from etc/HELLO (`C-h h'), yank it into a buffer
> (fundamental-mode suffices), add " Hello ", and then create a single
> line consisting of a large number of copies of these strings; on my
> machine 500 copies clearly shows the slowdown, and with 800 copies it is
> much worse.

It is not clear what exactly constitutes a slow-down in this recipe.
You describe how to create a test buffer, but not what you do
afterwards to demonstrate slow-down.  Please fill the gaps.

FTR, I tried C-f/C-b (no perceptible slow-down), C-v/M-v (likewise),
and C-n/C-p (no slow-down at the beginning of buffer, considerable
slow-down near the end: about 0.5 to 0.8 sec response time -- but this
is a non-optimized build; your optimized build should see about 0.1
sec to 0.2 sec).  Is this what you see?  If not, please tell what did
you do and what did you see, and please describe it in detail and with
timings if possible.

> There is no slowdown with a line of the same length consisting only of
> RTL or only of LTR text, nor with the above test line when
> bidi-display-reordering is set to nil in the buffer (but then, of
> course, the Arabic is not displayed correctly).

That's not what I see: setting bidi-display-reordering to nil doesn't
affect the slow-down on my system in any perceptible way.

> It seems that the long line optimizations added to Emacs 29 do not
> work with bidirectional text.

Long line optimizations don't kick in until the lines are longer than
the value of long-line-threshold, by default 50000 characters.  Since
800 copies of the Arabic greeting don't reach that threshold, the
optimizations are not in effect at all in this case.

> (FTR, I encountered this issue with a program of mine that generates
> Emacs Lisp files containing such long lines with bidirectional text.
> These files are not intended for display but I was examining one and
> experienced the slowdown.)

If your real-life cases are all with Arabic text, then the reason is
likely not bidirectional reordering of RTL text by itself (although it
does slow down redisplay to some degree), but the fact that all of the
Arabic text gets shaped by HarfBuzz via composition-function-table,
which involves calls from the C code of the display engine into Lisp,
which then turns and calls back into C.  This is relatively slow, but
we have no choice: Arabic shaping requires that all Arabic characters
be handed to the shaping engine for rendering, otherwise the display
will either be illegible for Arabic speakers or at least look ugly in
their eyes.

For now, I see no bugs here.  Maybe if you tell more we will arrive at
something that is a real problem, but what I've read and saw until now
is what I would expect in these cases.

Message #11 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Sun, 25 Feb 2024 19:03:24 +0100

On Sun, 25 Feb 2024 19:06:55 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> Date: Sun, 25 Feb 2024 17:23:11 +0100
>> From:  Stephen Berman via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>>
>> Displaying a buffer that contains a long line with bidirectional text
>> greatly slows down Emacs.  A simple reproduction is to copy the the
>> Arabic example from etc/HELLO (`C-h h'), yank it into a buffer
>> (fundamental-mode suffices), add " Hello ", and then create a single
>> line consisting of a large number of copies of these strings; on my
>> machine 500 copies clearly shows the slowdown, and with 800 copies it is
>> much worse.
>
> It is not clear what exactly constitutes a slow-down in this recipe.
> You describe how to create a test buffer, but not what you do
> afterwards to demonstrate slow-down.  Please fill the gaps.
>
> FTR, I tried C-f/C-b (no perceptible slow-down), C-v/M-v (likewise),
> and C-n/C-p (no slow-down at the beginning of buffer, considerable
> slow-down near the end: about 0.5 to 0.8 sec response time -- but this
> is a non-optimized build; your optimized build should see about 0.1
> sec to 0.2 sec).  Is this what you see?  If not, please tell what did
> you do and what did you see, and please describe it in detail and with
> timings if possible.

The first thing I tried, with both 500 and 800 copies, is M-> from
point-min: with 500 it took ~4 seconds, with 800 ~10 seconds, and each
time Emacs used 100% of one core for the duration (my machine has an
i7-8700 processor with 6 cores/12 threads).  After that even just
invoking commands like `C-x 1' or `M-x' and entering something in the
minibuffer shows a noticeable delay (several seconds) and 100% CPU.
When I typed `C-n' at point-min in the 500 copy buffer and held down the
key till the cursor froze, it took about a minute until it moved again
(to point-max), during which the core used by Emacs stayed at 100%.

>> There is no slowdown with a line of the same length consisting only of
>> RTL or only of LTR text, nor with the above test line when
>> bidi-display-reordering is set to nil in the buffer (but then, of
>> course, the Arabic is not displayed correctly).
>
> That's not what I see: setting bidi-display-reordering to nil doesn't
> affect the slow-down on my system in any perceptible way.

Very strange: here, after setting bidi-display-reordering to nil, M->
and M-< are practically instantaneous and even holding down C-n or C-p
traverses the entire buffer in a couple of seconds, with just a bit of
stuttering.

>> It seems that the long line optimizations added to Emacs 29 do not
>> work with bidirectional text.
>
> Long line optimizations don't kick in until the lines are longer than
> the value of long-line-threshold, by default 50000 characters.  Since
> 800 copies of the Arabic greeting don't reach that threshold, the
> optimizations are not in effect at all in this case.

Ah, ok.  Then this is a different long-line issue.

>> (FTR, I encountered this issue with a program of mine that generates
>> Emacs Lisp files containing such long lines with bidirectional text.
>> These files are not intended for display but I was examining one and
>> experienced the slowdown.)
>
> If your real-life cases are all with Arabic text, then the reason is
> likely not bidirectional reordering of RTL text by itself (although it
> does slow down redisplay to some degree), but the fact that all of the
> Arabic text gets shaped by HarfBuzz via composition-function-table,
> which involves calls from the C code of the display engine into Lisp,
> which then turns and calls back into C.  This is relatively slow, but
> we have no choice: Arabic shaping requires that all Arabic characters
> be handed to the shaping engine for rendering, otherwise the display
> will either be illegible for Arabic speakers or at least look ugly in
> their eyes.

I created a test buffer with 800 copies of the Hebrew HELLO text
concatenated with " Hello " and found similar slowdowns with M-> and C-n
as with the Arabic text.  IIUC displaying the Hebrew script does not
involve text shaping, or does it?

> For now, I see no bugs here.  Maybe if you tell more we will arrive at
> something that is a real problem, but what I've read and saw until now
> is what I would expect in these cases.

I can try other tests if you can suggest any you think will help.

Steve Berman

Message #14 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Sun, 25 Feb 2024 21:31:19 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Sun, 25 Feb 2024 19:03:24 +0100
> 
> The first thing I tried, with both 500 and 800 copies, is M-> from
> point-min: with 500 it took ~4 seconds, with 800 ~10 seconds, and each
> time Emacs used 100% of one core for the duration (my machine has an
> i7-8700 processor with 6 cores/12 threads).  After that even just
> invoking commands like `C-x 1' or `M-x' and entering something in the
> minibuffer shows a noticeable delay (several seconds) and 100% CPU.
> When I typed `C-n' at point-min in the 500 copy buffer and held down the
> key till the cursor froze, it took about a minute until it moved again
> (to point-max), during which the core used by Emacs stayed at 100%.

To make sure we are testing the same: what is the value of point-max
in that buffer?  I see something around 30k characters after making
800 copies.

> >> There is no slowdown with a line of the same length consisting only of
> >> RTL or only of LTR text, nor with the above test line when
> >> bidi-display-reordering is set to nil in the buffer (but then, of
> >> course, the Arabic is not displayed correctly).
> >
> > That's not what I see: setting bidi-display-reordering to nil doesn't
> > affect the slow-down on my system in any perceptible way.
> 
> Very strange: here, after setting bidi-display-reordering to nil, M->
> and M-< are practically instantaneous and even holding down C-n or C-p
> traverses the entire buffer in a couple of seconds, with just a bit of
> stuttering.

Then I don't know what is going on.  Maybe you have a very weak CPU or
something?  Or maybe your version of HarfBuzz is somehow much slower?

Just to be sure: this is in "emacs -Q", and you see the same with many
different fonts used for Arabic, not just with a few of them?

> > If your real-life cases are all with Arabic text, then the reason is
> > likely not bidirectional reordering of RTL text by itself (although it
> > does slow down redisplay to some degree), but the fact that all of the
> > Arabic text gets shaped by HarfBuzz via composition-function-table,
> > which involves calls from the C code of the display engine into Lisp,
> > which then turns and calls back into C.  This is relatively slow, but
> > we have no choice: Arabic shaping requires that all Arabic characters
> > be handed to the shaping engine for rendering, otherwise the display
> > will either be illegible for Arabic speakers or at least look ugly in
> > their eyes.
> 
> I created a test buffer with 800 copies of the Hebrew HELLO text
> concatenated with " Hello " and found similar slowdowns with M-> and C-n
> as with the Arabic text.  IIUC displaying the Hebrew script does not
> involve text shaping, or does it?

For the text in HELLO yes, it does: there are diacriticals (so-called
"niqqud") there.  But not all of the Hebrew text goes through the
shaper, only the combinations with diacriticals, so it should be
somewhat faster.  And if you remove the diacriticals, Emacs will not
use HarfBuzz at all for Hebrew.

> > For now, I see no bugs here.  Maybe if you tell more we will arrive at
> > something that is a real problem, but what I've read and saw until now
> > is what I would expect in these cases.
> 
> I can try other tests if you can suggest any you think will help.

I don't know what to suggest, because I see something very different,
and have never seen what you describe.  The only idea I have is to run
this under perf, but interpreting the results is not easy.

Can someone else reproduce Stephen's results?  I'm quite surprised by
what he describes, TBH.  That's not the performance of the Emacs
bidirectional display I'm used to see.

Message #17 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Sun, 25 Feb 2024 21:36:46 +0100

On Sun, 25 Feb 2024 21:31:19 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Sun, 25 Feb 2024 19:03:24 +0100
>> 
>> The first thing I tried, with both 500 and 800 copies, is M-> from
>> point-min: with 500 it took ~4 seconds, with 800 ~10 seconds, and each
>> time Emacs used 100% of one core for the duration (my machine has an
>> i7-8700 processor with 6 cores/12 threads).  After that even just
>> invoking commands like `C-x 1' or `M-x' and entering something in the
>> minibuffer shows a noticeable delay (several seconds) and 100% CPU.
>> When I typed `C-n' at point-min in the 500 copy buffer and held down the
>> key till the cursor froze, it took about a minute until it moved again
>> (to point-max), during which the core used by Emacs stayed at 100%.
>
> To make sure we are testing the same: what is the value of point-max
> in that buffer?  I see something around 30k characters after making
> 800 copies.

With 800 repetitions of the Arabic text from etc/HELLO ("السّلام عليكم")
concatenated with " Hello " with no newlines, point-max is 16001.  So
the length of the whole line is 16000 characters, and for the initial
string " Hello السّلام عليكم", M-= returns 20, so for 800 copies
point-max is what I would expect.  Do the Arabic characters in your font
occupy two columns each?  If the string I've tested is indeed that much
shorter than what you've test, it's even more mysterious that I see such
a slowdown and you don't...

>> >> There is no slowdown with a line of the same length consisting only of
>> >> RTL or only of LTR text, nor with the above test line when
>> >> bidi-display-reordering is set to nil in the buffer (but then, of
>> >> course, the Arabic is not displayed correctly).
>> >
>> > That's not what I see: setting bidi-display-reordering to nil doesn't
>> > affect the slow-down on my system in any perceptible way.
>> 
>> Very strange: here, after setting bidi-display-reordering to nil, M->
>> and M-< are practically instantaneous and even holding down C-n or C-p
>> traverses the entire buffer in a couple of seconds, with just a bit of
>> stuttering.
>
> Then I don't know what is going on.  Maybe you have a very weak CPU or
> something?  Or maybe your version of HarfBuzz is somehow much slower?

My CPU is Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, certainly not
superfast but not superslow either, e.g. bootstrapping Emacs (without
native compilation) takes just a few minutes (with native compilation
it's a couple of minutes longer).  My HarfBuzz is version 8.3.0.

> Just to be sure: this is in "emacs -Q", and you see the same with many
> different fonts used for Arabic, not just with a few of them?

Yes, this is with -Q.  The only fonts I've tried are Noto Naskh Arabic,
DejaVu Sans, and Amiri.  I'm not sure whether any other fonts I have
installed support Arabic (I only have 294 fonts installed).

>> > If your real-life cases are all with Arabic text, then the reason is
>> > likely not bidirectional reordering of RTL text by itself (although it
>> > does slow down redisplay to some degree), but the fact that all of the
>> > Arabic text gets shaped by HarfBuzz via composition-function-table,
>> > which involves calls from the C code of the display engine into Lisp,
>> > which then turns and calls back into C.  This is relatively slow, but
>> > we have no choice: Arabic shaping requires that all Arabic characters
>> > be handed to the shaping engine for rendering, otherwise the display
>> > will either be illegible for Arabic speakers or at least look ugly in
>> > their eyes.
>> 
>> I created a test buffer with 800 copies of the Hebrew HELLO text
>> concatenated with " Hello " and found similar slowdowns with M-> and C-n
>> as with the Arabic text.  IIUC displaying the Hebrew script does not
>> involve text shaping, or does it?
>
> For the text in HELLO yes, it does: there are diacriticals (so-called
> "niqqud") there.  But not all of the Hebrew text goes through the
> shaper, only the combinations with diacriticals, so it should be
> somewhat faster.  And if you remove the diacriticals, Emacs will not
> use HarfBuzz at all for Hebrew.

Ok, I repeated the test with Hebrew without niqqud ("שלום") and indeed,
M-> and M-< are much quicker and even holding down C-n and C-p is
quicker (it does freeze, but unfreezes more quickly).  So it seems the
bottleneck really is the shaping.

>> > For now, I see no bugs here.  Maybe if you tell more we will arrive at
>> > something that is a real problem, but what I've read and saw until now
>> > is what I would expect in these cases.
>> 
>> I can try other tests if you can suggest any you think will help.
>
> I don't know what to suggest, because I see something very different,
> and have never seen what you describe.  The only idea I have is to run
> this under perf, but interpreting the results is not easy.

I've never used perf, and don't even have it installed (I build the
kernel myself but I haven't built perf, but I could do that -- though
probably not in the very short term -- if it might be helpful).

> Can someone else reproduce Stephen's results?  I'm quite surprised by
> what he describes, TBH.  That's not the performance of the Emacs
> bidirectional display I'm used to see.

I also hope others chime in.

Steve Berman

Message #20 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 14:33:18 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Sun, 25 Feb 2024 21:36:46 +0100
> 
> > To make sure we are testing the same: what is the value of point-max
> > in that buffer?  I see something around 30k characters after making
> > 800 copies.
> 
> With 800 repetitions of the Arabic text from etc/HELLO ("السّلام عليكم")
> concatenated with " Hello " with no newlines, point-max is 16001.  So
> the length of the whole line is 16000 characters, and for the initial
> string " Hello السّلام عليكم", M-= returns 20, so for 800 copies
> point-max is what I would expect.  Do the Arabic characters in your font
> occupy two columns each?

No (and it doesn't matter, since I measured characters, not columns).
It's just that I copied more from the Arabic line than you did:
everything starting from the beginning of line.  Never mind, that.

> If the string I've tested is indeed that much shorter than what
> you've test, it's even more mysterious that I see such a slowdown
> and you don't...

Yes.

> > Then I don't know what is going on.  Maybe you have a very weak CPU or
> > something?  Or maybe your version of HarfBuzz is somehow much slower?
> 
> My CPU is Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, certainly not
> superfast but not superslow either, e.g. bootstrapping Emacs (without
> native compilation) takes just a few minutes (with native compilation
> it's a couple of minutes longer).  My HarfBuzz is version 8.3.0.

The CPU is fast.  My HarfBuzz version is much older, but I doubt that
HarfBuzz became so much slower for Arabic.  So it's very strange.  But
maybe it's worth to try downgrading to an earlier version, just to be
sure this is not the culprit?

> > Just to be sure: this is in "emacs -Q", and you see the same with many
> > different fonts used for Arabic, not just with a few of them?
> 
> Yes, this is with -Q.  The only fonts I've tried are Noto Naskh Arabic,
> DejaVu Sans, and Amiri.

Should be enough to draw conclusions, I think.

> Ok, I repeated the test with Hebrew without niqqud ("שלום") and indeed,
> M-> and M-< are much quicker and even holding down C-n and C-p is
> quicker (it does freeze, but unfreezes more quickly).  So it seems the
> bottleneck really is the shaping.

If, after starting Emacs, you set all the entries of Arabic characters
in composition-function-table to nil (e.g., using
set-char-table-range), do you see significant speedup?

Message #23 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 16:19:15 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Mon, 26 Feb 2024 15:13:38 +0100
> 
> On Mon, 26 Feb 2024 14:33:18 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> > The CPU is fast.  My HarfBuzz version is much older, but I doubt that
> > HarfBuzz became so much slower for Arabic.  So it's very strange.  But
> > maybe it's worth to try downgrading to an earlier version, just to be
> > sure this is not the culprit?
> 
> That's not so straightforward on this system, where I build system
> packages from source, so downgrading HarfBuzz may require also
> rebuilding packages that depend on it.  I have a somewhat older system
> on this machine with HarfBuzz 7.1.0, so I can easily try that.  What
> version of HarfBuzz are you using?

I have 2.4.0 here.

> > If, after starting Emacs, you set all the entries of Arabic characters
> > in composition-function-table to nil (e.g., using
> > set-char-table-range), do you see significant speedup?
> 
> Yes, there is a noticeable speedup when M-> and C-v, M-v, and even
> holding down C-n and C-p is faster, though they still freeze after a few
> seconds, but the freeze doesn't last as long as when it happens with the
> shaped Arabic script.  Also, probably unsurprisingly, the speed after
> disabling shaping is still much slower than keeping shaping enabled but
> setting bidi-display-reordering to nil.

Is this with text from HELLO, or with some other Arabic text?  The
prominent effect of bidi-display-reordering is actually surprising,
and not what I see here, with or without shaping.  Very strange.

Message #26 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: stephen.berman <at> gmx.net
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 16:22:59 +0200

> Cc: 69385 <at> debbugs.gnu.org
> Date: Mon, 26 Feb 2024 16:19:15 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > From: Stephen Berman <stephen.berman <at> gmx.net>
> > Cc: 69385 <at> debbugs.gnu.org
> > Date: Mon, 26 Feb 2024 15:13:38 +0100
> > 
> > On Mon, 26 Feb 2024 14:33:18 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> > 
> > > The CPU is fast.  My HarfBuzz version is much older, but I doubt that
> > > HarfBuzz became so much slower for Arabic.  So it's very strange.  But
> > > maybe it's worth to try downgrading to an earlier version, just to be
> > > sure this is not the culprit?
> > 
> > That's not so straightforward on this system, where I build system
> > packages from source, so downgrading HarfBuzz may require also
> > rebuilding packages that depend on it.  I have a somewhat older system
> > on this machine with HarfBuzz 7.1.0, so I can easily try that.  What
> > version of HarfBuzz are you using?
> 
> I have 2.4.0 here.

Btw, another possibility could be that this is specific to the Cairo
build.  So maybe try a build without Cairo.

Message #29 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 15:13:38 +0100

On Mon, 26 Feb 2024 14:33:18 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Sun, 25 Feb 2024 21:36:46 +0100
[...] 
>> My CPU is Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, certainly not
>> superfast but not superslow either, e.g. bootstrapping Emacs (without
>> native compilation) takes just a few minutes (with native compilation
>> it's a couple of minutes longer).  My HarfBuzz is version 8.3.0.
>
> The CPU is fast.  My HarfBuzz version is much older, but I doubt that
> HarfBuzz became so much slower for Arabic.  So it's very strange.  But
> maybe it's worth to try downgrading to an earlier version, just to be
> sure this is not the culprit?

That's not so straightforward on this system, where I build system
packages from source, so downgrading HarfBuzz may require also
rebuilding packages that depend on it.  I have a somewhat older system
on this machine with HarfBuzz 7.1.0, so I can easily try that.  What
version of HarfBuzz are you using?

[...]
>> Ok, I repeated the test with Hebrew without niqqud ("שלום") and indeed,
>> M-> and M-< are much quicker and even holding down C-n and C-p is
>> quicker (it does freeze, but unfreezes more quickly).  So it seems the
>> bottleneck really is the shaping.
>
> If, after starting Emacs, you set all the entries of Arabic characters
> in composition-function-table to nil (e.g., using
> set-char-table-range), do you see significant speedup?

Yes, there is a noticeable speedup when M-> and C-v, M-v, and even
holding down C-n and C-p is faster, though they still freeze after a few
seconds, but the freeze doesn't last as long as when it happens with the
shaped Arabic script.  Also, probably unsurprisingly, the speed after
disabling shaping is still much slower than keeping shaping enabled but
setting bidi-display-reordering to nil.

Steve Berman

Message #32 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 16:17:53 +0100

On Mon, 26 Feb 2024 16:22:59 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Mon, 26 Feb 2024 16:19:15 +0200
>> From: Eli Zaretskii <eliz <at> gnu.org>
>>
>> > From: Stephen Berman <stephen.berman <at> gmx.net>
>> > Cc: 69385 <at> debbugs.gnu.org
>> > Date: Mon, 26 Feb 2024 15:13:38 +0100
>> >
>> > On Mon, 26 Feb 2024 14:33:18 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
>> >
>> > > The CPU is fast.  My HarfBuzz version is much older, but I doubt that
>> > > HarfBuzz became so much slower for Arabic.  So it's very strange.  But
>> > > maybe it's worth to try downgrading to an earlier version, just to be
>> > > sure this is not the culprit?
>> >
>> > That's not so straightforward on this system, where I build system
>> > packages from source, so downgrading HarfBuzz may require also
>> > rebuilding packages that depend on it.  I have a somewhat older system
>> > on this machine with HarfBuzz 7.1.0, so I can easily try that.  What
>> > version of HarfBuzz are you using?
>>
>> I have 2.4.0 here.
>
> Btw, another possibility could be that this is specific to the Cairo
> build.  So maybe try a build without Cairo.

I did, and the slowdown seems just as bad as in the build with Cairo.

Steve Berman

Message #35 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 16:17:38 +0100

On Mon, 26 Feb 2024 16:19:15 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Mon, 26 Feb 2024 15:13:38 +0100
>>
>> On Mon, 26 Feb 2024 14:33:18 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
[...]
>> > If, after starting Emacs, you set all the entries of Arabic characters
>> > in composition-function-table to nil (e.g., using
>> > set-char-table-range), do you see significant speedup?
>>
>> Yes, there is a noticeable speedup when M-> and C-v, M-v, and even
>> holding down C-n and C-p is faster, though they still freeze after a few
>> seconds, but the freeze doesn't last as long as when it happens with the
>> shaped Arabic script.  Also, probably unsurprisingly, the speed after
>> disabling shaping is still much slower than keeping shaping enabled but
>> setting bidi-display-reordering to nil.
>
> Is this with text from HELLO, or with some other Arabic text?  The
> prominent effect of bidi-display-reordering is actually surprising,
> and not what I see here, with or without shaping.  Very strange.

All my tests in this thread have been with the Arabic example text in
HELLO, though the file with which I first noticed the slowdown contains
much more and different Arabic text, as well as ASCII and non-ASCII
characters without shaping.  Here, the speed difference, when executing
movement and other commands in a buffer with a long line of
bidirectional text containing shaped Arabic, between
bidi-display-reordering t and nil is like night and day.  If it's not
that way for you I can only assume it's due to another factor
distinguishing our systems that hasn't yet been identified.

Steve Berman

Message #38 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 16:59:46 +0100

On Mon, 26 Feb 2024 17:52:50 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Mon, 26 Feb 2024 16:17:53 +0100
>>
>> On Mon, 26 Feb 2024 16:22:59 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
>>
>> >> Cc: 69385 <at> debbugs.gnu.org
>> >> Date: Mon, 26 Feb 2024 16:19:15 +0200
>> >> From: Eli Zaretskii <eliz <at> gnu.org>
>> >>
>> >> > From: Stephen Berman <stephen.berman <at> gmx.net>
>> >> > Cc: 69385 <at> debbugs.gnu.org
>> >> > Date: Mon, 26 Feb 2024 15:13:38 +0100
>> >> >
>> >> > On Mon, 26 Feb 2024 14:33:18 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
>> >> >
>> >> > > The CPU is fast.  My HarfBuzz version is much older, but I doubt that
>> >> > > HarfBuzz became so much slower for Arabic.  So it's very strange.  But
>> >> > > maybe it's worth to try downgrading to an earlier version, just to be
>> >> > > sure this is not the culprit?
>> >> >
>> >> > That's not so straightforward on this system, where I build system
>> >> > packages from source, so downgrading HarfBuzz may require also
>> >> > rebuilding packages that depend on it.  I have a somewhat older system
>> >> > on this machine with HarfBuzz 7.1.0, so I can easily try that.  What
>> >> > version of HarfBuzz are you using?
>> >>
>> >> I have 2.4.0 here.
>> >
>> > Btw, another possibility could be that this is specific to the Cairo
>> > build.  So maybe try a build without Cairo.
>>
>> I did, and the slowdown seems just as bad as in the build with Cairo.
>
> Can you try the emacs-29 branch?

I already did and see the same slowdown with it as with master.  (Or did
you mean try emacs-29 without Cairo?  That I haven't done, but given I
saw no noticeable difference on master, I wouldn't expect to see one on
emacs-29 either.)

Steve Berman

Message #41 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 17:52:50 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Mon, 26 Feb 2024 16:17:53 +0100
> 
> On Mon, 26 Feb 2024 16:22:59 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> >> Cc: 69385 <at> debbugs.gnu.org
> >> Date: Mon, 26 Feb 2024 16:19:15 +0200
> >> From: Eli Zaretskii <eliz <at> gnu.org>
> >>
> >> > From: Stephen Berman <stephen.berman <at> gmx.net>
> >> > Cc: 69385 <at> debbugs.gnu.org
> >> > Date: Mon, 26 Feb 2024 15:13:38 +0100
> >> >
> >> > On Mon, 26 Feb 2024 14:33:18 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> >> >
> >> > > The CPU is fast.  My HarfBuzz version is much older, but I doubt that
> >> > > HarfBuzz became so much slower for Arabic.  So it's very strange.  But
> >> > > maybe it's worth to try downgrading to an earlier version, just to be
> >> > > sure this is not the culprit?
> >> >
> >> > That's not so straightforward on this system, where I build system
> >> > packages from source, so downgrading HarfBuzz may require also
> >> > rebuilding packages that depend on it.  I have a somewhat older system
> >> > on this machine with HarfBuzz 7.1.0, so I can easily try that.  What
> >> > version of HarfBuzz are you using?
> >>
> >> I have 2.4.0 here.
> >
> > Btw, another possibility could be that this is specific to the Cairo
> > build.  So maybe try a build without Cairo.
> 
> I did, and the slowdown seems just as bad as in the build with Cairo.

Can you try the emacs-29 branch?

Message #44 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 18:36:42 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Mon, 26 Feb 2024 16:59:46 +0100
> 
> On Mon, 26 Feb 2024 17:52:50 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> >> From: Stephen Berman <stephen.berman <at> gmx.net>
> >> Cc: 69385 <at> debbugs.gnu.org
> >> Date: Mon, 26 Feb 2024 16:17:53 +0100
> >>
> >> On Mon, 26 Feb 2024 16:22:59 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> >>
> >> >> Cc: 69385 <at> debbugs.gnu.org
> >> >> Date: Mon, 26 Feb 2024 16:19:15 +0200
> >> >> From: Eli Zaretskii <eliz <at> gnu.org>
> >> >>
> >> >> > From: Stephen Berman <stephen.berman <at> gmx.net>
> >> >> > Cc: 69385 <at> debbugs.gnu.org
> >> >> > Date: Mon, 26 Feb 2024 15:13:38 +0100
> >> >> >
> >> >> > On Mon, 26 Feb 2024 14:33:18 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> >> >> >
> >> >> > > The CPU is fast.  My HarfBuzz version is much older, but I doubt that
> >> >> > > HarfBuzz became so much slower for Arabic.  So it's very strange.  But
> >> >> > > maybe it's worth to try downgrading to an earlier version, just to be
> >> >> > > sure this is not the culprit?
> >> >> >
> >> >> > That's not so straightforward on this system, where I build system
> >> >> > packages from source, so downgrading HarfBuzz may require also
> >> >> > rebuilding packages that depend on it.  I have a somewhat older system
> >> >> > on this machine with HarfBuzz 7.1.0, so I can easily try that.  What
> >> >> > version of HarfBuzz are you using?
> >> >>
> >> >> I have 2.4.0 here.
> >> >
> >> > Btw, another possibility could be that this is specific to the Cairo
> >> > build.  So maybe try a build without Cairo.
> >>
> >> I did, and the slowdown seems just as bad as in the build with Cairo.
> >
> > Can you try the emacs-29 branch?
> 
> I already did and see the same slowdown with it as with master.  (Or did
> you mean try emacs-29 without Cairo?  That I haven't done, but given I
> saw no noticeable difference on master, I wouldn't expect to see one on
> emacs-29 either.)

I meant the emacs-29 branch in general, regardless of the exact
configuration.

I'm out of ideas, sorry.  The prominent effect of
bidi-display-reordering is especially puzzling.

Can you measure the time it takes to redraw the window after M->, with
and without bidi-display-reordering?  I'd like to have numbers here,
not just impressions.  For example, measure it with benchmark-run or
something similar.

Message #47 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 18:12:04 +0100

On Mon, 26 Feb 2024 18:36:42 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

[...]
> Can you measure the time it takes to redraw the window after M->, with
> and without bidi-display-reordering?  I'd like to have numbers here,
> not just impressions.  For example, measure it with benchmark-run or
> something similar.

On master started with -Q in a buffer containing a single line
consisting of 800 repetitions of "السّلام عليكم Hello " (in the buffer
the string " Hello " was to the left of the Arabic string because I
yanked the latter first), with bidi-display-reordering at its default
value t and with point at point-min, running (benchmark-run nil
(end-of-buffer)) returned:

(5.249231941 1 0.014300497000000023)

After moving point back to point-min and setting bidi-display-reordering
to nil, running (benchmark-run nil (end-of-buffer)) again now returned:

(0.034058467 0 0.0)

Steve Berman 

Message #50 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 21:18:25 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Mon, 26 Feb 2024 18:12:04 +0100
> 
> On Mon, 26 Feb 2024 18:36:42 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> [...]
> > Can you measure the time it takes to redraw the window after M->, with
> > and without bidi-display-reordering?  I'd like to have numbers here,
> > not just impressions.  For example, measure it with benchmark-run or
> > something similar.
> 
> On master started with -Q in a buffer containing a single line
> consisting of 800 repetitions of "السّلام عليكم Hello "

Oh, I think I see what's going on.  The buffer you create this way is
rendered as a paragraph with right-to-left base direction, right?
That is, it starts at the right side of the window, and C-f generally
moves to the left, yes?  By contrast, I thought you were doing this in
a buffer whose bidi-paragraph-direction is set to left-to-right, or
the text begins with Latin characters (as in "Arabic", the string the
begins the Arabic line in HELLO).

So now try your recipe, but set bidi-paragraph-direction to
left-to-right, or prepend a single ASCII character to the long line
(which will have the same effect on the base paragraph direction), and
then repeat your experiment, including the timings.  I think you will
see a big difference.

The reason is that, when a paragraph's direction is right-to-left,
inserting a new glyph into glyph matrices pushes all the previous
glyphs, thus reversing them on the fly.  Whereas in LTR paragraphs a
glyph is inserted by adding it to the end of the previous glyphs.  And
pushing is more expensive.  So now I understand why resetting
bidi-display-reordering had such a dramatic effect in your case: it
makes the paragraph render LTR as its side effect, which avoids the
costly pushing of glyphs.  In a very long line, this cost is very
high.

I will see if we can do better in this matter.

Message #53 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 26 Feb 2024 22:47:21 +0100

On Mon, 26 Feb 2024 21:18:25 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Mon, 26 Feb 2024 18:12:04 +0100
>> 
>> On Mon, 26 Feb 2024 18:36:42 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
>> 
>> [...]
>> > Can you measure the time it takes to redraw the window after M->, with
>> > and without bidi-display-reordering?  I'd like to have numbers here,
>> > not just impressions.  For example, measure it with benchmark-run or
>> > something similar.
>> 
>> On master started with -Q in a buffer containing a single line
>> consisting of 800 repetitions of "السّلام عليكم Hello "
>
> Oh, I think I see what's going on.  The buffer you create this way is
> rendered as a paragraph with right-to-left base direction, right?
> That is, it starts at the right side of the window, and C-f generally
> moves to the left, yes?

Correct.

>                          By contrast, I thought you were doing this in
> a buffer whose bidi-paragraph-direction is set to left-to-right, or
> the text begins with Latin characters (as in "Arabic", the string the
> begins the Arabic line in HELLO).
>
> So now try your recipe, but set bidi-paragraph-direction to
> left-to-right, or prepend a single ASCII character to the long line
> (which will have the same effect on the base paragraph direction), and
> then repeat your experiment, including the timings.  I think you will
> see a big difference.

I created the test buffer now by first typing "Hello" and the inserting
" السّلام عليكم " on the right, so the base paragraph direction was LTR.
Then I ran the benchmarks, and there is indeed a big difference, but one
that I think will surprise you.  First, the timing for the LTR run with
bidi-display-reordering set to nil was:

(0.035104007 0 0.0)

This is, unsurprisingly, almost the same as the corresponding test run
with RTL base paragraph direction, which was (0.034058467 0 0.0).

In contrast, recall that with bidi-display-reordering at the default
value t, the RTL test run was (5.249231941 1 0.014300497000000023), but
now the LTR test run gave this:

(10.613699099 1 0.012965359999999981)

Again, the only difference between the buffers is the order of the
strings and, consequently, the base paragraph direction.  Both buffers
consist of 800 copies of the strings and in both point-max is 16001.

The -- in view of your below explanation -- unexpectedly large timing
for the LTR buffer reminded me of the file in which I first encountered
the slowdown, which consists of a number of Emacs Lisp sexps.  The first
group are all ASCII characters, so the base paragraph direction is LTR.
Then there is a vector of 827 lists in a single line (remember, it's a
generated file), each list containing three strings, one in Arabic
script, one in English, one a transliteration of the Arabic which in
many cases uses non-ASCII characters.  The benchmark run for M-> in the
buffer visiting this file was:

(9.308704995000001 4 0.054923504999999984)

The buffer size here is considerably larger than the buffers for my test
runs; I couldn't remember it exactly, so I typed `M-: (point)' in the
buffer, with the result 43901.  It took a number of seconds for the
result to be displayed, during which Emacs was unresponsive, so I then
ran (benchmark-run nil (point)) in that buffer, with this result:

(9.030000000000001e-07 0 0.0)

I don't know what this result means, but the reported execution time
bears no relation to actual elapsed time until the value of point was
displayed.  So I reran the benchmark and also manually timed it with a
stopwatch.  This is the result of the second benchmark:

(3.1120000000000004e-06 0 0.0)

However, the stopwatch showed fully 15 seconds.  Maybe benchmark-run
doesn't work for point for some reason.  Given my doubts, I reran the
benchmark on the test LTR buffer, but the reported execution time was
only slightly different from before:

(10.317616279000001 0 0.0)

What do you make of these results?

> The reason is that, when a paragraph's direction is right-to-left,
> inserting a new glyph into glyph matrices pushes all the previous
> glyphs, thus reversing them on the fly.  Whereas in LTR paragraphs a
> glyph is inserted by adding it to the end of the previous glyphs.  And
> pushing is more expensive.  So now I understand why resetting
> bidi-display-reordering had such a dramatic effect in your case: it
> makes the paragraph render LTR as its side effect, which avoids the
> costly pushing of glyphs.  In a very long line, this cost is very
> high.

This sounds plausible, but my results seem to indicate there something
else (or more) going on.

> I will see if we can do better in this matter.

I appreciate whatever you can do.

Steve Berman

Message #56 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Tue, 27 Feb 2024 10:52:23 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Mon, 26 Feb 2024 22:47:21 +0100
> 
> I created the test buffer now by first typing "Hello" and the inserting
> " السّلام عليكم " on the right, so the base paragraph direction was LTR.
> Then I ran the benchmarks, and there is indeed a big difference, but one
> that I think will surprise you.

It doesn't, see below.

> First, the timing for the LTR run with bidi-display-reordering set
> to nil was:
> 
> (0.035104007 0 0.0)
> 
> This is, unsurprisingly, almost the same as the corresponding test run
> with RTL base paragraph direction, which was (0.034058467 0 0.0).
> 
> In contrast, recall that with bidi-display-reordering at the default
> value t, the RTL test run was (5.249231941 1 0.014300497000000023), but
> now the LTR test run gave this:
> 
> (10.613699099 1 0.012965359999999981)

Yes, that's the result of the fix I installed yesterday: it makes the
display correct, but it costs more CPU, sometimes much more.

To see why this is so costly, disable auto-composition-mode in the
buffer, and compare the times with bidi-display-reordering nil and
non-nil.  With auto-composition-mode disabled, you see the effect of
the reordering alone (because characters are still reordered for
display, just not shaped correctly, as C-f will show you), and it
should be quite small; at least, that's what I see here.  The
(mandatory) shaping of Arabic text is what takes most of the redisplay
time in these cases.  In particular, M-> does a recenter to show EOB
nicely, and that is very expensive in this case; disable that
recentering by using "(goto-char (point-max))" instead to see the
difference.

> The buffer size here is considerably larger than the buffers for my test
> runs; I couldn't remember it exactly, so I typed `M-: (point)' in the
> buffer, with the result 43901.  It took a number of seconds for the
> result to be displayed, during which Emacs was unresponsive, so I then
> ran (benchmark-run nil (point)) in that buffer, with this result:
> 
> (9.030000000000001e-07 0 0.0)
> 
> I don't know what this result means, but the reported execution time
> bears no relation to actual elapsed time until the value of point was
> displayed.  So I reran the benchmark and also manually timed it with a
> stopwatch.  This is the result of the second benchmark:
> 
> (3.1120000000000004e-06 0 0.0)
> 
> However, the stopwatch showed fully 15 seconds.  Maybe benchmark-run
> doesn't work for point for some reason.

A Lisp program that just calls '(point)' doesn't include the redisplay
caused by "M-:", which enters the minibuffer, and therefore triggers a
fairly thorough redisplay of the window, and by display of the result.
That's Emacs's MVC design in action for you.

> This sounds plausible, but my results seem to indicate there something
> else (or more) going on.

Yes: that something is character composition, which in the case of
Arabic text is quite expensive, because every chunk of Arabic text
between SPACEs goes through the shaping engine.

(This explains well why Emacs tries so hard to call the shaper only
when absolutely necessary, contrary to HarfBuzz folks' opinions that
we should pass all the text through the shaping engine, to get correct
display with ligatures, kerning, and other niceties.  They are right,
of course, but with the way character composition was designed in
Emacs, doing so would be prohibitive from the performance POV.)

> > I will see if we can do better in this matter.
> 
> I appreciate whatever you can do.

From what I've seen so far, it won't be easy, because it requires
changes to some key logic in the display code.  Hmm...

Message #59 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Tue, 27 Feb 2024 11:13:05 +0100

On Tue, 27 Feb 2024 10:52:23 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Mon, 26 Feb 2024 22:47:21 +0100
>> 
>> I created the test buffer now by first typing "Hello" and the inserting
>> " السّلام عليكم " on the right, so the base paragraph direction was LTR.
>> Then I ran the benchmarks, and there is indeed a big difference, but one
>> that I think will surprise you.
>
> It doesn't, see below.
>
>> First, the timing for the LTR run with bidi-display-reordering set
>> to nil was:
>> 
>> (0.035104007 0 0.0)
>> 
>> This is, unsurprisingly, almost the same as the corresponding test run
>> with RTL base paragraph direction, which was (0.034058467 0 0.0).
>> 
>> In contrast, recall that with bidi-display-reordering at the default
>> value t, the RTL test run was (5.249231941 1 0.014300497000000023), but
>> now the LTR test run gave this:
>> 
>> (10.613699099 1 0.012965359999999981)
>
> Yes, that's the result of the fix I installed yesterday: it makes the
> display correct, but it costs more CPU, sometimes much more.

My recollection is that I ran the benchmarks before updating master with
your fix, but perhaps I misremember.  So I just now ran the benchmarks
for the default value of bidi-display-reordering again on emacs-29,
which does not have your fix.  Here is the result for the buffer with
RTL base paragraph direction:

(5.334990188 0 0.0)

And for the buffer with LTR base paragraph direction:

(10.537410334999999 1 0.011070665000000035)

And I ran them again on master now, to be sure I'm testing with your
fix; here are the respective results:

(5.350407252 1 0.013722977000000025)
(10.55348192 1 0.014691314999999983)

So practically the same difference without and with your fix.

> To see why this is so costly, disable auto-composition-mode in the
> buffer, and compare the times with bidi-display-reordering nil and
> non-nil.  With auto-composition-mode disabled, you see the effect of
> the reordering alone (because characters are still reordered for
> display, just not shaped correctly, as C-f will show you), and it
> should be quite small; at least, that's what I see here.  The

This I can confirm (on master with your fix); in the RTL buffer with
bidi-display-reordering first t and then nil:

(0.049796204999999996 0 0.0)
(0.033475448 0 0.0)

and likewise in the LRT buffer:

(0.049654598 0 0.0)
(0.033401513 0 0.0)

> (mandatory) shaping of Arabic text is what takes most of the redisplay
> time in these cases.  In particular, M-> does a recenter to show EOB
> nicely, and that is very expensive in this case; disable that
> recentering by using "(goto-char (point-max))" instead to see the
> difference.

I guess by "does a recenter to show EOB nicely" you mean the call of
(recenter -3) at the end of the code in end-of-buffer, because after
calling (goto-char (point-max)) the last line is vertically centered in
the window?

>> The buffer size here is considerably larger than the buffers for my test
>> runs; I couldn't remember it exactly, so I typed `M-: (point)' in the
>> buffer, with the result 43901.  It took a number of seconds for the
>> result to be displayed, during which Emacs was unresponsive, so I then
>> ran (benchmark-run nil (point)) in that buffer, with this result:
>> 
>> (9.030000000000001e-07 0 0.0)
>> 
>> I don't know what this result means, but the reported execution time
>> bears no relation to actual elapsed time until the value of point was
>> displayed.  So I reran the benchmark and also manually timed it with a
>> stopwatch.  This is the result of the second benchmark:
>> 
>> (3.1120000000000004e-06 0 0.0)
>> 
>> However, the stopwatch showed fully 15 seconds.  Maybe benchmark-run
>> doesn't work for point for some reason.
>
> A Lisp program that just calls '(point)' doesn't include the redisplay
> caused by "M-:", which enters the minibuffer, and therefore triggers a
> fairly thorough redisplay of the window, and by display of the result.
> That's Emacs's MVC design in action for you.

Ah, ok; thanks for the explanation.

>> This sounds plausible, but my results seem to indicate there something
>> else (or more) going on.
>
> Yes: that something is character composition, which in the case of
> Arabic text is quite expensive, because every chunk of Arabic text
> between SPACEs goes through the shaping engine.
>
> (This explains well why Emacs tries so hard to call the shaper only
> when absolutely necessary, contrary to HarfBuzz folks' opinions that
> we should pass all the text through the shaping engine, to get correct
> display with ligatures, kerning, and other niceties.  They are right,
> of course, but with the way character composition was designed in
> Emacs, doing so would be prohibitive from the performance POV.)
>
>> > I will see if we can do better in this matter.
>> 
>> I appreciate whatever you can do.
>
>>From what I've seen so far, it won't be easy, because it requires
> changes to some key logic in the display code.  Hmm...

I'm afraid I can't help with the display code logic, but I'm happy to
test any ideas you have.

Steve Berman

Message #62 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: stephen.berman <at> gmx.net
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Sun, 03 Mar 2024 17:18:31 +0200

> Cc: 69385 <at> debbugs.gnu.org
> Date: Mon, 26 Feb 2024 21:18:25 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> The reason is that, when a paragraph's direction is right-to-left,
> inserting a new glyph into glyph matrices pushes all the previous
> glyphs, thus reversing them on the fly.  Whereas in LTR paragraphs a
> glyph is inserted by adding it to the end of the previous glyphs.  And
> pushing is more expensive.  So now I understand why resetting
> bidi-display-reordering had such a dramatic effect in your case: it
> makes the paragraph render LTR as its side effect, which avoids the
> costly pushing of glyphs.  In a very long line, this cost is very
> high.
> 
> I will see if we can do better in this matter.

Actually, the fact that glyphs are pushed is not the culprit here,
since we only push as many glyphs as fit in a screen line, and that
cannot be too many.

The real culprit is the way we look for the next (actually, previous)
composition that includes the given buffer position, when we scan
characters backward (due to reordering).  The current code causes us
to scan all the way to BOB, which is especially painful with long
lines without newlines.

I've coded an optimization for that, see the patch below.  Could you
please try running with this patch for a week or two, and see if you
can spot any problems?  I hope you are editing those files with
embedded Arabic frequently enough for these changes to be exercised.

If you see no problems after a week or two, I will install this.

Thanks.

diff --git a/src/bidi.c b/src/bidi.c
index 36d1a04..306b44a 100644
--- a/src/bidi.c
+++ b/src/bidi.c
@@ -754,6 +754,16 @@ bidi_cache_find_level_change (int level, int dir, bool before)
   return -1;
 }
 
+ptrdiff_t
+bidi_level_start (int level)
+{
+  ptrdiff_t slot = bidi_cache_find_level_change (level, -1, true);
+
+  if (slot >= 0)
+    return bidi_cache[slot].charpos;
+  return -1;
+}
+
 static void
 bidi_cache_ensure_space (ptrdiff_t idx)
 {
diff --git a/src/dispextern.h b/src/dispextern.h
index 5387cb4..1c3232f 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -3438,6 +3438,7 @@ #define TTY_CAP_STRIKE_THROUGH	0x20
 extern void *bidi_shelve_cache (void);
 extern void bidi_unshelve_cache (void *, bool);
 extern ptrdiff_t bidi_find_first_overridden (struct bidi_it *);
+extern ptrdiff_t bidi_level_start (int);
 
 /* Defined in xdisp.c */
 
diff --git a/src/xdisp.c b/src/xdisp.c
index d03769e..140d711 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4353,7 +4353,7 @@ compute_stop_pos (struct it *it)
          an automatic composition, limit the search of composable
          characters to that position.  */
       if (it->bidi_p && it->bidi_it.scan_dir < 0)
-	stoppos = -1;
+	stoppos = bidi_level_start (it->bidi_it.resolved_level) - 1;
       else if (!STRINGP (it->string)
 	       && it->cmp_it.stop_pos <= IT_CHARPOS (*it)
 	       && cmp_limit_pos > 0)
@@ -8712,9 +8712,8 @@ set_iterator_to_next (struct it *it, bool reseat_p)
 	      ptrdiff_t stop = it->end_charpos;
 
 	      if (it->bidi_it.scan_dir < 0)
-		/* Now we are scanning backward and don't know
-		   where to stop.  */
-		stop = -1;
+		/* Now we are scanning backward; figure out where to stop.  */
+		stop = bidi_level_start (it->bidi_it.resolved_level) - 1;
 	      composition_compute_stop_pos (&it->cmp_it, IT_CHARPOS (*it),
 					    IT_BYTEPOS (*it), stop, Qnil, true);
 	    }
@@ -8745,7 +8744,7 @@ set_iterator_to_next (struct it *it, bool reseat_p)
 		     re-compute the stop position for composition.  */
 		  ptrdiff_t stop = it->end_charpos;
 		  if (it->bidi_it.scan_dir < 0)
-		    stop = -1;
+		    stop = bidi_level_start (it->bidi_it.resolved_level) - 1;
 		  composition_compute_stop_pos (&it->cmp_it, IT_CHARPOS (*it),
 						IT_BYTEPOS (*it), stop, Qnil,
 						true);
@@ -9190,7 +9189,9 @@ get_visually_first_element (struct it *it)
 	  bytepos = IT_BYTEPOS (*it);
 	}
       if (it->bidi_it.scan_dir < 0)
-	stop = -1;
+	stop = STRINGP (it->string)
+	       ? -1
+	       : bidi_level_start (it->bidi_it.resolved_level) - 1;
       composition_compute_stop_pos (&it->cmp_it, charpos, bytepos, stop,
 				    it->string, true);
     }
@@ -9694,9 +9695,10 @@ next_element_from_buffer (struct it *it)
 		    && PT < it->end_charpos) ? PT : it->end_charpos;
 	}
       else
-	stop = it->bidi_it.scan_dir < 0 ? -1 : it->end_charpos;
-      if (CHAR_COMPOSED_P (it, IT_CHARPOS (*it), IT_BYTEPOS (*it),
-			   stop)
+	stop = it->bidi_it.scan_dir < 0
+	       ? bidi_level_start (it->bidi_it.resolved_level) - 1
+	       : it->end_charpos;
+      if (CHAR_COMPOSED_P (it, IT_CHARPOS (*it), IT_BYTEPOS (*it), stop)
 	  && next_element_from_composition (it))
 	{
 	  return true;

Message #65 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 04 Mar 2024 14:28:50 +0100

On Sun, 03 Mar 2024 17:18:31 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Mon, 26 Feb 2024 21:18:25 +0200
>> From: Eli Zaretskii <eliz <at> gnu.org>
>>
>> The reason is that, when a paragraph's direction is right-to-left,
>> inserting a new glyph into glyph matrices pushes all the previous
>> glyphs, thus reversing them on the fly.  Whereas in LTR paragraphs a
>> glyph is inserted by adding it to the end of the previous glyphs.  And
>> pushing is more expensive.  So now I understand why resetting
>> bidi-display-reordering had such a dramatic effect in your case: it
>> makes the paragraph render LTR as its side effect, which avoids the
>> costly pushing of glyphs.  In a very long line, this cost is very
>> high.
>>
>> I will see if we can do better in this matter.
>
> Actually, the fact that glyphs are pushed is not the culprit here,
> since we only push as many glyphs as fit in a screen line, and that
> cannot be too many.
>
> The real culprit is the way we look for the next (actually, previous)
> composition that includes the given buffer position, when we scan
> characters backward (due to reordering).  The current code causes us
> to scan all the way to BOB, which is especially painful with long
> lines without newlines.
>
> I've coded an optimization for that, see the patch below.  Could you
> please try running with this patch for a week or two, and see if you
> can spot any problems?

I applied the patch, rebuilt, started with -Q, made two test buffers
using 800 concatenated copies of a bidirectional Arabic + English
string, one with base paragraph direction RTL, one with LTR, and ran the
end-of-buffer benchmark; here are the results:

RTL with patch: (0.099171108 0 0.0)
LTR with patch: (0.143149541 1 0.01330051900000001)

Here for comparison my previous results without your patch:

RTL without patch: (5.249231941 1 0.014300497000000023)
LTR without patch: (10.613699099 1 0.012965359999999981)

That's more than 50 times faster for the RTL test and more than 70 times
faster for the LTR test -- impressive!

>                         I hope you are editing those files with
> embedded Arabic frequently enough for these changes to be exercised.

As I mentioned previously, my real files are programmatically generated
elisp files (so base paragraph direction LTR) not meant to be manually
edited or even just viewed by users of the program, and I haven't edited
them manually, and normally wouldn't.  But I just now ran the
end-of-buffer benchmark on one of them (the one I described previous,
containing a vector of 827 lists of bidirectional strings in a single
line), with this result:

(0.849369497 4 0.05337466599999996)

This was the timing without your patch:

(9.308704995000001 4 0.054923504999999984)

So for this file your patch yields "only" an almost 11 times faster
benchmark.  For navigation besides M-> and M-<, I find C-v, M-v, C-n,
C-p in the buffer visiting this file still very slow (noticeably more
than in the test buffers) and holding them down still freezes Emacs
(with C-n and C-p for many seconds) and uses 100% of a CPU core; though,
while I haven't tried timing these yet, my impression is that the
freezes are not as long as the ones I observed without your patch.
Also, there is still a marked delay when entering the minibuffer with
M-x or M-: or when switching to another buffer with C-x b, though
impressionistically no worse than the delays without your patch.  I'll
try to do more testing.

> If you see no problems after a week or two, I will install this.

Thanks.

Steve Berman

Message #68 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Mon, 04 Mar 2024 16:43:26 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Mon, 04 Mar 2024 14:28:50 +0100
> 
> On Sun, 03 Mar 2024 17:18:31 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> I applied the patch, rebuilt, started with -Q, made two test buffers
> using 800 concatenated copies of a bidirectional Arabic + English
> string, one with base paragraph direction RTL, one with LTR, and ran the
> end-of-buffer benchmark; here are the results:
> 
> RTL with patch: (0.099171108 0 0.0)
> LTR with patch: (0.143149541 1 0.01330051900000001)
> 
> Here for comparison my previous results without your patch:
> 
> RTL without patch: (5.249231941 1 0.014300497000000023)
> LTR without patch: (10.613699099 1 0.012965359999999981)
> 
> That's more than 50 times faster for the RTL test and more than 70 times
> faster for the LTR test -- impressive!
> 
> >                         I hope you are editing those files with
> > embedded Arabic frequently enough for these changes to be exercised.
> 
> As I mentioned previously, my real files are programmatically generated
> elisp files (so base paragraph direction LTR) not meant to be manually
> edited or even just viewed by users of the program, and I haven't edited
> them manually, and normally wouldn't.  But I just now ran the
> end-of-buffer benchmark on one of them (the one I described previous,
> containing a vector of 827 lists of bidirectional strings in a single
> line), with this result:
> 
> (0.849369497 4 0.05337466599999996)
> 
> This was the timing without your patch:
> 
> (9.308704995000001 4 0.054923504999999984)
> 
> So for this file your patch yields "only" an almost 11 times faster
> benchmark.  For navigation besides M-> and M-<, I find C-v, M-v, C-n,
> C-p in the buffer visiting this file still very slow (noticeably more
> than in the test buffers) and holding them down still freezes Emacs
> (with C-n and C-p for many seconds) and uses 100% of a CPU core; though,
> while I haven't tried timing these yet, my impression is that the
> freezes are not as long as the ones I observed without your patch.
> Also, there is still a marked delay when entering the minibuffer with
> M-x or M-: or when switching to another buffer with C-x b, though
> impressionistically no worse than the delays without your patch.  I'll
> try to do more testing.

Thanks for testing.  The above matches what I see on my system.  C-n
and C-p is known to be problematic in long lines, but these changes
speed them up as well, although perhaps not as well as the other
commands.

> > If you see no problems after a week or two, I will install this.
> 
> Thanks.

So I will wait for you to report any problems, and if no problems are
seen, will install in a week or so.

Message #71 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Thu, 07 Mar 2024 12:12:51 +0100

[Message part 1 (text/plain, inline)]

On Mon, 04 Mar 2024 16:43:26 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Mon, 04 Mar 2024 14:28:50 +0100
>>
>> On Sun, 03 Mar 2024 17:18:31 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
[...]
>> >                         I hope you are editing those files with
>> > embedded Arabic frequently enough for these changes to be exercised.
>>
>> As I mentioned previously, my real files are programmatically generated
>> elisp files (so base paragraph direction LTR) not meant to be manually
>> edited or even just viewed by users of the program, and I haven't edited
>> them manually, and normally wouldn't.  But I just now ran the
>> end-of-buffer benchmark on one of them (the one I described previous,
>> containing a vector of 827 lists of bidirectional strings in a single
>> line), with this result:
>>
>> (0.849369497 4 0.05337466599999996)
>>
>> This was the timing without your patch:
>>
>> (9.308704995000001 4 0.054923504999999984)
>>
>> So for this file your patch yields "only" an almost 11 times faster
>> benchmark.  For navigation besides M-> and M-<, I find C-v, M-v, C-n,
>> C-p in the buffer visiting this file still very slow (noticeably more
>> than in the test buffers) and holding them down still freezes Emacs
>> (with C-n and C-p for many seconds) and uses 100% of a CPU core; though,
>> while I haven't tried timing these yet, my impression is that the
>> freezes are not as long as the ones I observed without your patch.
>> Also, there is still a marked delay when entering the minibuffer with
>> M-x or M-: or when switching to another buffer with C-x b, though
>> impressionistically no worse than the delays without your patch.  I'll
>> try to do more testing.
>
> Thanks for testing.  The above matches what I see on my system.  C-n
> and C-p is known to be problematic in long lines, but these changes
> speed them up as well, although perhaps not as well as the other
> commands.
>
>> > If you see no problems after a week or two, I will install this.
>>
>> Thanks.
>
> So I will wait for you to report any problems, and if no problems are
> seen, will install in a week or so.

I haven't yet run into any issues concerning your patch, but I have
encountered a problem with another one of my generated files, which,
though independent of your patch (the problem also happens in emacs-29),
is an issue for bidirectional text in Emacs, so might be worth trying to
handle better.  If you want, I can open a separate bug to pursue this
issue, but for now I'll summarize what I've observed so far.

Most of the Arabic words in the problematic file are enclosed in the
bidirectional control characters POP DIRECTIONAL FORMATTING (#x202c) and
RIGHT-TO-LEFT EMBEDDING (#x202b).  I did not add these characters, but I
had copy-&-pasted most of the Arabic from a PDF file I did not create.
I don't know if PDFs of Arabic text normally contain these control
characters, but the consequences for Emacs were dramatic.  When I simply
visited this file in Emacs (started with -Q) there was an immediate
slowdown, and in top I could see Emacs using 100% of a CPU thread.  When
I ran the end-of-buffer benchmark on this file, the result (with your
patch) was:

(27.962602113 2 0.0226042269999999977)

However, the display of that result only appeared in the echo area after
more than a minute (I timed it with a stopwatch).  At that point the
mode line showed the buffer at 4% from the top, and the display remained
frozen afterwards.  After several minutes during which Emacs consumed
100% CPU, and I had switched the focus away from the Emacs frame, the
CPU consumption stopped, but as soon as I switch focus back to that
frame, it went back to 100%.  The display never changed from showing the
buffer at 4%, apparently being in some kind of infinite loop.  After
about 15 minutes I started gdb, attached the Emacs process and produced
a backtrace, which I've attached, in the hope it helps to diagnose the
problem.

The problem seems to be certainly related the the bidirectional control
characters, because I made a copy of the file and removed all
occurrences of these control characters from it, and then ran the
end-of-buffer benchmark, getting this result (with your patch):

(0.716104165 4 0.04223660400000001)

And the display updated normally and CPU consumption was normal.

Nevertheless, there seems to be something else besides the control
characters involved in this issue, because as a futher test, I created a
buffer consisting of more than 1000 copies of the test string
concatenating the Arabic example in etc/HELLO and "Hello", and manually
enclosed each Arabic word in the above control characters, but the
benchmark result in this buffer was not significantly different from the
result without the control characters (and similar to the above result
for the copy of the problematic file without the control characters),
and the display did not freeze.

Steve Berman

[Message part 2 (text/plain, attachment)]

Message #74 received at 69385 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385 <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Thu, 07 Mar 2024 13:25:03 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Thu, 07 Mar 2024 12:12:51 +0100
> 
> Most of the Arabic words in the problematic file are enclosed in the
> bidirectional control characters POP DIRECTIONAL FORMATTING (#x202c) and
> RIGHT-TO-LEFT EMBEDDING (#x202b).  I did not add these characters, but I
> had copy-&-pasted most of the Arabic from a PDF file I did not create.
> I don't know if PDFs of Arabic text normally contain these control
> characters, but the consequences for Emacs were dramatic.  When I simply
> visited this file in Emacs (started with -Q) there was an immediate
> slowdown, and in top I could see Emacs using 100% of a CPU thread.  When
> I ran the end-of-buffer benchmark on this file, the result (with your
> patch) was:
> 
> (27.962602113 2 0.0226042269999999977)
> 
> However, the display of that result only appeared in the echo area after
> more than a minute (I timed it with a stopwatch).  At that point the
> mode line showed the buffer at 4% from the top, and the display remained
> frozen afterwards.  After several minutes during which Emacs consumed
> 100% CPU, and I had switched the focus away from the Emacs frame, the
> CPU consumption stopped, but as soon as I switch focus back to that
> frame, it went back to 100%.  The display never changed from showing the
> buffer at 4%, apparently being in some kind of infinite loop.  After
> about 15 minutes I started gdb, attached the Emacs process and produced
> a backtrace, which I've attached, in the hope it helps to diagnose the
> problem.
> 
> The problem seems to be certainly related the the bidirectional control
> characters, because I made a copy of the file and removed all
> occurrences of these control characters from it, and then ran the
> end-of-buffer benchmark, getting this result (with your patch):
> 
> (0.716104165 4 0.04223660400000001)
> 
> And the display updated normally and CPU consumption was normal.
> 
> Nevertheless, there seems to be something else besides the control
> characters involved in this issue, because as a futher test, I created a
> buffer consisting of more than 1000 copies of the test string
> concatenating the Arabic example in etc/HELLO and "Hello", and manually
> enclosed each Arabic word in the above control characters, but the
> benchmark result in this buffer was not significantly different from the
> result without the control characters (and similar to the above result
> for the copy of the problematic file without the control characters),
> and the display did not freeze.

Please submit a separate bug report, and please post an example of a
problematic file with the report.

Thanks.

Message #79 received at 69385-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 69385-done <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Thu, 21 Mar 2024 10:29:12 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 69385 <at> debbugs.gnu.org
> Date: Thu, 07 Mar 2024 12:12:51 +0100
> 
> > So I will wait for you to report any problems, and if no problems are
> > seen, will install in a week or so.
> 
> I haven't yet run into any issues concerning your patch

No further comments, so I assume the patch is indeed safe, and I have
now installed it on the master branch.

I'm therefore closing this bug.  Thanks for testing the patch.

Message #82 received at 69385-done <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 69385-done <at> debbugs.gnu.org
Subject: Re: bug#69385: 30.0.50; Long lines with bidi text slow down Emacs
Date: Thu, 21 Mar 2024 16:08:11 +0100

On Thu, 21 Mar 2024 10:29:12 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 69385 <at> debbugs.gnu.org
>> Date: Thu, 07 Mar 2024 12:12:51 +0100
>>
>> > So I will wait for you to report any problems, and if no problems are
>> > seen, will install in a week or so.
>>
>> I haven't yet run into any issues concerning your patch
>
> No further comments, so I assume the patch is indeed safe, and I have
> now installed it on the master branch.
>
> I'm therefore closing this bug.  Thanks for testing the patch.

Thanks for the optimization!

Steve Berman

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.