GNU bug report logs - #67604
Motion problems with inline images

Previous Next

To reply to this bug, email your comments to 67604 AT debbugs.gnu.org.
There is no need to reopen the bug first.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Motion problems with inline images
Date: Sun, 3 Dec 2023 11:54:27 -0500

[Message part 1 (text/plain, inline)]

Following on from the recently fixed bug#67533, which related to incorrect pixel size measurements with inline images, one remaining issue, with that fix in place, relates to motion in buffers in visual line mode with inline images.  A video of the effect <https://gist.github.com/jdtsmith/ad765047a6afe20f353de573d8c07da9?permalink_comment_id=4780726#gistcomment-4780726> can be seen on the associated gist for bug#67533.

Here the issue is that next-line/forward-line/(vertical-motion 1) all skip from the Green inline SVG straight to the “JUMPS HERE” line, for certain frame widths (for me: 79 characters).  See reproduction code, below.  Depending on your font char size (mine is 7x14 pixels) this width may vary for you.  Note that multiple frame width produce the exact same wrapped appearance, but the motion bug shows up only for one of them.

There are other related phenomena, such as prev-line from an image at column 0 jumping to (near the) end of the previous screen line, but hopefully this is enough to identify the motion issue(s) at hand.

++++ 
(require 'svg)
(let ((buf "svg-file-motion-demo")
      (ims '(("red" 146 29)
	     ("green" 106 29)
	     ("blue" 151 29))))
  (with-current-buffer (get-buffer-create buf)
    (erase-buffer)
    (visual-line-mode 1)
    (insert "tellus.  $\\gamma(t) = \\log\\left(\\sqrt{\\tan(t)}\\right)$  Donec hendrerit tempor tellus.  $\\chi(y) = \\sqrt{\\frac{1}{\\log(y)}}$  Phasellus lacus.  $\\tau(t) = \\exp\\left(\\sqrt{\\exp(t)}\\right)$  Curabitur lacinia pulvinar nibh.
JUMPS HERE")
    (goto-char (point-min))
    (while  (re-search-forward (rx ?$ (* (not ?$)) ?$) nil t)
      (let* ((ov (make-overlay (match-beginning 0) (match-end 0)))
	     (im (pop ims))
	     (svg (svg-create (nth 1 im) (nth 2 im))))
	(svg-rectangle svg 0 0 (nth 1 im) (nth 2 im) :fill-color (car im))
	(overlay-put ov 'display (svg-image svg :ascent 'center)))))
  (pop-to-buffer buf))
++++

[Message part 2 (text/html, inline)]

Message #8 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Sun, 03 Dec 2023 19:11:50 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Sun, 3 Dec 2023 11:54:27 -0500
> 
> Following on from the recently fixed bug#67533, which related to incorrect pixel size measurements
> with inline images, one remaining issue, with that fix in place, relates to motion in buffers in visual line
> mode with inline images.  A video of the effect can be seen on the associated gist for bug#67533.
> 
> Here the issue is that next-line/forward-line/(vertical-motion 1) all skip from the Green inline SVG
> straight to the “JUMPS HERE” line, for certain frame widths (for me: 79 characters).  See
> reproduction code, below.  Depending on your font char size (mine is 7x14 pixels) this width may
> vary for you.  Note that multiple frame width produce the exact same wrapped appearance, but the
> motion bug shows up only for one of them.

I tried different frame widths, but couldn't reproduce the problem.
Not with the latest master branch, anyway, where I recently installed
the changes from bug#67533.

If someone can tell what frame width and default font height trigger
the problem, I will try to reproduce with those sizes.

Message #11 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Sun, 3 Dec 2023 15:46:38 -0500

[Message part 1 (text/plain, inline)]

> On Dec 3, 2023, at 12:11 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Sun, 3 Dec 2023 11:54:27 -0500
>> 
>> Following on from the recently fixed bug#67533, which related to incorrect pixel size measurements
>> with inline images, one remaining issue, with that fix in place, relates to motion in buffers in visual line
>> mode with inline images.  A video of the effect can be seen on the associated gist for bug#67533.
>> 
>> Here the issue is that next-line/forward-line/(vertical-motion 1) all skip from the Green inline SVG
>> straight to the “JUMPS HERE” line, for certain frame widths (for me: 79 characters).  See
>> reproduction code, below.  Depending on your font char size (mine is 7x14 pixels) this width may
>> vary for you.  Note that multiple frame width produce the exact same wrapped appearance, but the
>> motion bug shows up only for one of them.
> 
> I tried different frame widths, but couldn't reproduce the problem.
> Not with the latest master branch, anyway, where I recently installed
> the changes from bug#67533.
> 
> If someone can tell what frame width and default font height trigger
> the problem, I will try to reproduce with those sizes.

I just compiled a fresh NS build from master with your bug#67533 fix.  It triggers the skip-a-line bug for me, from emacs -q at frame-width=79.  But I can also confirm, it is font size dependent, likely occurring with only a precise combination of pixel sizes.  

Here’s an easy way to find it with your font, using the attached command:

Run the let-form mentioned in my initial message.
Resize the frame width down until the green image first moves to the beginning of a visual line (80 chars, for me).
Resize down one more column *without causing further wrap* (79 chars for me).
In the svg-file-motion-demo buffer, execute M-x my/find-skip-bug.
It should report the pixel width needed for the red SVG image on the first line to trigger the bug, and leave the demo buffer in that state.
Hitting down twice from (point-min) should land on JUMPS HERE, bypassing the stub line “pulvinar nibh".

++++

(defun my/find-skip-bug ()
  (interactive)
  (goto-char (point-min))
  (let* ((ov (car (overlays-at 10)))
	 (w 146)
	 (h 29)
	 (wc (frame-char-width))
	 (res
	  (cl-loop
	   for off from (- (- wc 2)) to (1- wc)
	   for sw = (+ w off)
	   for svg = (svg-create sw h) do
	   (message "Checking with red image width %d" sw)
	   (svg-rectangle svg 0 0 sw h :fill-color "red")
	   (overlay-put ov 'display (svg-image svg :ascent 'center))
	   if (save-excursion
		(next-line) (next-line)
		(beginning-of-visual-line)
		(looking-at "JUMPS HERE"))
	   return off
	   finally return nil)))
    (if res (message "Found Bug at offset %d = %d pixels" res (+ w res))
      (message "Did not find Bug"))))
  

[Message part 2 (text/html, inline)]

Message #14 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 04 Dec 2023 15:16:47 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Sun, 3 Dec 2023 15:46:38 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
> * Run the let-form mentioned in my initial message.
> * Resize the frame width down until the green image first moves to the beginning of a visual line (80
>  chars, for me).
> * Resize down one more column *without causing further wrap* (79 chars for me).
> * In the svg-file-motion-demo buffer, execute M-x my/find-skip-bug.
> * It should report the pixel width needed for the red SVG image on the first line to trigger the bug,
>  and leave the demo buffer in that state.

Instead of reporting the size, it says "Did not find Bug".

Message #17 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 4 Dec 2023 09:10:01 -0500

> On Dec 4, 2023, at 8:16 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Sun, 3 Dec 2023 15:46:38 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>> * Run the let-form mentioned in my initial message.
>> * Resize the frame width down until the green image first moves to the beginning of a visual line (80
>> chars, for me).
>> * Resize down one more column *without causing further wrap* (79 chars for me).
>> * In the svg-file-motion-demo buffer, execute M-x my/find-skip-bug.
>> * It should report the pixel width needed for the red SVG image on the first line to trigger the bug,
>> and leave the demo buffer in that state.
> 
> Instead of reporting the size, it says "Did not find Bug”.

Try increasing/decreasing width by one or two columns and repeating.  Note that if you go too far, you’ll get a false positive report when the green SVG wraps back to line 1. 

Message #20 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 04 Dec 2023 16:17:52 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Mon, 4 Dec 2023 09:10:01 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
> 
> 
> > On Dec 4, 2023, at 8:16 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> > 
> >> From: JD Smith <jdtsmith <at> gmail.com>
> >> Date: Sun, 3 Dec 2023 15:46:38 -0500
> >> Cc: 67604 <at> debbugs.gnu.org
> >> 
> >> * Run the let-form mentioned in my initial message.
> >> * Resize the frame width down until the green image first moves to the beginning of a visual line (80
> >> chars, for me).
> >> * Resize down one more column *without causing further wrap* (79 chars for me).
> >> * In the svg-file-motion-demo buffer, execute M-x my/find-skip-bug.
> >> * It should report the pixel width needed for the red SVG image on the first line to trigger the bug,
> >> and leave the demo buffer in that state.
> > 
> > Instead of reporting the size, it says "Did not find Bug”.
> 
> Try increasing/decreasing width by one or two columns and repeating.  Note that if you go too far, you’ll get a false positive report when the green SVG wraps back to line 1. 

Sorry, still no cigar.

Message #23 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 4 Dec 2023 09:25:16 -0500

[Message part 1 (text/plain, inline)]

Strange.  Can I ask your (frame-char-width) and (frame-char-height)?  I haven’t yet found a font or size where the bug isn’t discoverable.  Usually at multiple frame widths.

Just confirming your buffer looks like:

[PastedGraphic-1.png (image/png, inline)]

[Message part 3 (text/plain, inline)]

And that *Messages* includes something like:

Checking with red image width 139
Checking with red image width 140
Checking with red image width 141
Checking with red image width 142
Checking with red image width 143
Checking with red image width 144
Checking with red image width 145
Checking with red image width 146
Checking with red image width 147
Checking with red image width 148
Checking with red image width 149
Checking with red image width 150
Checking with red image width 151
Checking with red image width 152
Checking with red image width 153
Found Bug at offset 7 = 153 pixels

> On Dec 4, 2023, at 9:17 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Mon, 4 Dec 2023 09:10:01 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>> 
>> 
>>> On Dec 4, 2023, at 8:16 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
>>> 
>>>> From: JD Smith <jdtsmith <at> gmail.com>
>>>> Date: Sun, 3 Dec 2023 15:46:38 -0500
>>>> Cc: 67604 <at> debbugs.gnu.org
>>>> 
>>>> * Run the let-form mentioned in my initial message.
>>>> * Resize the frame width down until the green image first moves to the beginning of a visual line (80
>>>> chars, for me).
>>>> * Resize down one more column *without causing further wrap* (79 chars for me).
>>>> * In the svg-file-motion-demo buffer, execute M-x my/find-skip-bug.
>>>> * It should report the pixel width needed for the red SVG image on the first line to trigger the bug,
>>>> and leave the demo buffer in that state.
>>> 
>>> Instead of reporting the size, it says "Did not find Bug”.
>> 
>> Try increasing/decreasing width by one or two columns and repeating.  Note that if you go too far, you’ll get a false positive report when the green SVG wraps back to line 1.
> 
> Sorry, still no cigar.

Message #26 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 04 Dec 2023 16:37:22 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Mon, 4 Dec 2023 09:25:16 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
> Strange.  Can I ask your (frame-char-width) and (frame-char-height)?  I haven’t yet found a font or size where the bug isn’t discoverable.  Usually at multiple frame widths.

width = 8, height = 16

I also tried 7 and 15, also without success.

> Just confirming your buffer looks like:

Yes.

Message #29 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 4 Dec 2023 12:44:45 -0500

> On Dec 4, 2023, at 9:37 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Mon, 4 Dec 2023 09:25:16 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>> Strange.  Can I ask your (frame-char-width) and (frame-char-height)?  I haven’t yet found a font or size where the bug isn’t discoverable.  Usually at multiple frame widths.
> 
> width = 8, height = 16
> 
> I also tried 7 and 15, also without success.


Wonders never cease.  I just picked an 8x16 font and it worked fine, finding the bug at offset=2 (i.e. red svg=148 pixels wide). This was using the build from this morning with your recent xdisp.c fix.  To be sure we’re on the same page, I’ve included again the test code below.

My version info: GNU Emacs 30.0.50 (build 5, aarch64-apple-darwin23.1.0, NS appkit-2487.20 Version 14.1.2 (Build 23B92)) of 2023-12-04.   

I suppose this motion bug could be version-dependent?  If anyone else can confirm this on NS or other builds that would be most helpful.

One other scraping-the-barrel's-bottom idea: I’ve found before that (frame-char-width) when called “too early” yields incorrect/missing results; are you running the code interactively, or via a batch style setup?  Do you see a nice range of “Check with red image width …” messages?

++++
(require 'svg)
(let ((buf "svg-file-motion-demo")
      (ims '(("red" 146 29) ; 146 start
	     ("green" 108 29) ; 108 start
	     ("blue" 151 29))))
  (with-current-buffer (get-buffer-create buf)
    (erase-buffer)
    (visual-line-mode 1)
    (insert "tellus.  $\\gamma(t) = \\log\\left(\\sqrt{\\tan(t)}\\right)$  Donec hendrerit tempor tellus.  $\\chi(y) = \\sqrt{\\frac{1}{\\log(y)}}$  Phasellus lacus.  $\\tau(t) = \\exp\\left(\\sqrt{\\exp(t)}\\right)$  Curabitur lacinia pulvinar nibh.
JUMPS HERE")
    (goto-char (point-min))
    (while  (re-search-forward (rx ?$ (* (not ?$)) ?$) nil t)
      (let* ((ov (make-overlay (match-beginning 0) (match-end 0)))
	     (im (pop ims))
	     (svg (svg-create (nth 1 im) (nth 2 im))))
	(svg-rectangle svg 0 0 (nth 1 im) (nth 2 im) :fill-color (car im))
	(overlay-put ov 'display (svg-image svg :ascent 'center)))))
  (pop-to-buffer buf))

(defun my/find-skip-bug ()
  (interactive)
  (goto-char (point-min))
  (let* ((ov (car (overlays-at 10)))
	 (w 146)
	 (h 29)
	 (wc (frame-char-width))
	 (res
	  (cl-loop
	   for off from (- (- wc 2)) to (1- wc)
	   for sw = (+ w off)
	   for svg = (svg-create sw h) do
	   (message "Checking with red image width %d" sw)
	   (svg-rectangle svg 0 0 sw h :fill-color "red")
	   (overlay-put ov 'display (svg-image svg :ascent 'center))
	   if (save-excursion
		(next-line) (next-line)
		(beginning-of-visual-line)
		(looking-at "JUMPS HERE"))
	   return off
	   finally return nil)))
    (if res (message "Found Bug at offset %d = %d pixels" res (+ w res))
      (message "Did not find Bug"))))

Message #32 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 04 Dec 2023 20:20:40 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Mon, 4 Dec 2023 12:44:45 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
> Wonders never cease.  I just picked an 8x16 font and it worked fine, finding the bug at offset=2 (i.e. red svg=148 pixels wide). This was using the build from this morning with your recent xdisp.c fix.  To be sure we’re on the same page, I’ve included again the test code below.

Now I run it and I get

   Found Bug at offset -6 = 140 pixels

but the problem with cursor jumping to "JUMPS HERE" still doesn't
happen.

> One other scraping-the-barrel's-bottom idea: I’ve found before that (frame-char-width) when called “too early” yields incorrect/missing results; are you running the code interactively, or via a batch style setup?  Do you see a nice range of “Check with red image width …” messages?

I'm running the code interactively, and I do see those "Check..."
messages (mainly in *Messages*, as they are very quickly replaced by
the finishing message).

Message #35 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 4 Dec 2023 14:16:10 -0500

> On Dec 4, 2023, at 1:20 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Mon, 4 Dec 2023 12:44:45 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>> Wonders never cease.  I just picked an 8x16 font and it worked fine, finding the bug at offset=2 (i.e. red svg=148 pixels wide). This was using the build from this morning with your recent xdisp.c fix.  To be sure we’re on the same page, I’ve included again the test code below.
> 
> Now I run it and I get
> 
>   Found Bug at offset -6 = 140 pixels
> 
> but the problem with cursor jumping to "JUMPS HERE" still doesn't
> happen.


OK great.  Is it possible at offset = -6 = 140 pixels the green image has wrapped back onto line 1?  That’s a “false positive” bug.  The layout after my/find-skip-bug has found the magic width must look as in my prior screenshot (with green image at the beginning of screen line 2) or it’s not a real bug.

If it does look correct, from (point-min), any pair of next-line or (vertical-motion 1), should bypass scene line 3 = "pulvinar nibh" and land on screen line 4 = "JUMPS HERE", as that’s the literal test find-skip-bug performs!

Message #38 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 04 Dec 2023 21:38:29 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Mon, 4 Dec 2023 14:16:10 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
> > Now I run it and I get
> > 
> >   Found Bug at offset -6 = 140 pixels
> > 
> > but the problem with cursor jumping to "JUMPS HERE" still doesn't
> > happen.
> 
> 
> OK great.  Is it possible at offset = -6 = 140 pixels the green image has wrapped back onto line 1?  That’s a “false positive” bug.  The layout after my/find-skip-bug has found the magic width must look as in my prior screenshot (with green image at the beginning of screen line 2) or it’s not a real bug.

No, it looks with the green image at the visual end of a screen line.

> If it does look correct, from (point-min), any pair of next-line or
> (vertical-motion 1), should bypass scene line 3 = "pulvinar nibh"
> and land on screen line 4 = "JUMPS HERE",

It doesn't.  It works as expected.

Message #41 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 4 Dec 2023 16:05:56 -0500

> On Dec 4, 2023, at 2:38 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Mon, 4 Dec 2023 14:16:10 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>>> Now I run it and I get
>>> 
>>>  Found Bug at offset -6 = 140 pixels
>>> 
>>> but the problem with cursor jumping to "JUMPS HERE" still doesn't
>>> happen.
>> 
>> 
>> OK great.  Is it possible at offset = -6 = 140 pixels the green image has wrapped back onto line 1?  That’s a “false positive” bug.  The layout after my/find-skip-bug has found the magic width must look as in my prior screenshot (with green image at the beginning of screen line 2) or it’s not a real bug.
> 
> No, it looks with the green image at the visual end of a screen line.

Before or after the test?  If before the test, you haven’t tried this, please resize the frame until the green image wraps to visual line 2, then reduce width by one column more.

Message #44 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Tue, 05 Dec 2023 05:24:09 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Mon, 4 Dec 2023 16:05:56 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
> >> OK great.  Is it possible at offset = -6 = 140 pixels the green image has wrapped back onto line 1?  That’s a “false positive” bug.  The layout after my/find-skip-bug has found the magic width must look as in my prior screenshot (with green image at the beginning of screen line 2) or it’s not a real bug.
> > 
> > No, it looks with the green image at the visual end of a screen line.
> 
> Before or after the test?

After the test.

Message #47 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Tue, 5 Dec 2023 18:06:41 -0500

> On Dec 4, 2023, at 10:24 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Mon, 4 Dec 2023 16:05:56 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>>>> OK great.  Is it possible at offset = -6 = 140 pixels the green image has wrapped back onto line 1?  That’s a “false positive” bug.  The layout after my/find-skip-bug has found the magic width must look as in my prior screenshot (with green image at the beginning of screen line 2) or it’s not a real bug.
>>> 
>>> No, it looks with the green image at the visual end of a screen line.
>> 
>> Before or after the test?
> 
> After the test.

OK, so this was likely a false positive then.  I’m out of ideas.  Maybe Windows is magically immune.

For me both NS and Mac builds with your most recent xdisp.c fixes exhibit the same motion issue.  Do you have anyone you can call in to try the simple test on another build?

Thanks.

Message #50 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Wed, 06 Dec 2023 05:31:34 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Tue, 5 Dec 2023 18:06:41 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
> >> Before or after the test?
> > 
> > After the test.
> 
> OK, so this was likely a false positive then.  I’m out of ideas.  Maybe Windows is magically immune.

Unlikely.  More likely is that we need some specific metrics of the
displayed stuff to see the problem (which is therefore very rare).

> For me both NS and Mac builds with your most recent xdisp.c fixes exhibit the same motion issue.  Do you have anyone you can call in to try the simple test on another build?

Not really, no.  But reproducing the problem is just a step towards
debugging it, so the alternative is for you to step through next-line
and its subroutines (or in C through vertical-motion) and tell what
happens there, and preferably also why.  It isn't like I'm the only
one who should be able to read the code and understand where it fails.

Message #53 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Tue, 5 Dec 2023 23:33:42 -0500

[Message part 1 (text/plain, inline)]

> On Dec 5, 2023, at 10:31 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Tue, 5 Dec 2023 18:06:41 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>>>> Before or after the test?
>>> 
>>> After the test.
>> 
>> OK, so this was likely a false positive then.  I’m out of ideas.  Maybe Windows is magically immune.
> 
> Unlikely.  More likely is that we need some specific metrics of the
> displayed stuff to see the problem (which is therefore very rare).
> 
>> For me both NS and Mac builds with your most recent xdisp.c fixes exhibit the same motion issue.  Do you have anyone you can call in to try the simple test on another build?
> 
> Not really, no.  But reproducing the problem is just a step towards
> debugging it, so the alternative is for you to step through next-line
> and its subroutines

I stepped through next-line and subroutines via edebug and landed via line-move-visual on:

(vertical-motion  (cons (or goal-column
				     (if (consp temporary-goal-column)
					 (car temporary-goal-column)
				       temporary-goal-column))
				 arg))

So vertical-motion is where all fingers point.

> It isn't like I'm the only
> one who should be able to read the code and understand where it fails.

I agree with that, but unfortunately am not setup for it here and have next to no familiarity with Emacs’ C code.  I’m sorry I can’t be of more help.  But similar to how you were unable to work with dvisvgm and other packages, I don’t have access to gdb, as it is not supported on my architecture.

[Message part 2 (text/html, inline)]

Message #56 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Wed, 06 Dec 2023 14:25:01 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Tue, 5 Dec 2023 23:33:42 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
>  Not really, no.  But reproducing the problem is just a step towards
>  debugging it, so the alternative is for you to step through next-line
>  and its subroutines
> 
> I stepped through next-line and subroutines via edebug and landed via line-move-visual on:
> 
>  (vertical-motion  (cons (or goal-column
>      (if (consp temporary-goal-column)
>  (car temporary-goal-column)
>        temporary-goal-column))
>  arg))
> 
> So vertical-motion is where all fingers point.

Did you verify that goal-column and temporary-goal-column have correct
values in the case where the problem happens?

>  It isn't like I'm the only
>  one who should be able to read the code and understand where it fails.
> 
> I agree with that, but unfortunately am not setup for it here and have next to no familiarity with Emacs’
> C code.  I’m sorry I can’t be of more help.  But similar to how you were unable to work with dvisvgm
> and other packages, I don’t have access to gdb, as it is not supported on my architecture.

If someone can reproduce and debug the problem on a system other than
macOS, that would be some progress.  (If the problem is specific to
macOS, it is much less interesting, at least to me, since the display
code on macOS behaves differently in significant ways, and because
macOS is in general an idiosyncratic platform.)

Message #59 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Wed, 6 Dec 2023 13:29:54 -0500

[Message part 1 (text/plain, inline)]

> On Dec 6, 2023, at 7:25 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Tue, 5 Dec 2023 23:33:42 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>> Not really, no.  But reproducing the problem is just a step towards
>> debugging it, so the alternative is for you to step through next-line
>> and its subroutines
>> 
>> I stepped through next-line and subroutines via edebug and landed via line-move-visual on:
>> 
>> (vertical-motion  (cons (or goal-column
>>     (if (consp temporary-goal-column)
>> (car temporary-goal-column)
>>       temporary-goal-column))
>> arg))
>> 
>> So vertical-motion is where all fingers point.
> 
> Did you verify that goal-column and temporary-goal-column have correct
> values in the case where the problem happens?

Yes.  Sometimes the temporary-goal-column is 0 (if I C-a on a text line) or sometimes a fractional number like 66.5 or similar if I start my downward movement by first C-a’ing on the green image line.  In both cases the short screen line #3 line is skipped.

>> It isn't like I'm the only
>> one who should be able to read the code and understand where it fails.
>> 
>> I agree with that, but unfortunately am not setup for it here and have next to no familiarity with Emacs’
>> C code.  I’m sorry I can’t be of more help.  But similar to how you were unable to work with dvisvgm
>> and other packages, I don’t have access to gdb, as it is not supported on my architecture.
> 
> If someone can reproduce and debug the problem on a system other than
> macOS, that would be some progress.  (If the problem is specific to
> macOS, it is much less interesting, at least to me, since the display
> code on macOS behaves differently in significant ways, and because
> macOS is in general an idiosyncratic platform.)

I have made progress.  I built master on a remote Linux server and ran it via X forwarding.  With an 8x17 font, the bug is found at offset 2 = 148 pixel width of the red SVG, for the specific frame-width = 74.   I verified that (vertical-motion 1) causes the same line-skip error, when starting from the green image.  

This makes me think that your difficulty reproducing the bug on your system may result from a miscommunication.  To be sure, I’ve recorded a session of me (re-)discovering the bug at this comment <https://gist.github.com/jdtsmith/914c9f44ed5f5394e4ec188b00b09b47?permalink_comment_id=4784608#gistcomment-4784608>.  Hopefully that’s useful.  

If not, I did the build on my Linux server with debug options, and can run over gdb. If you can give me a simple recipe for where I should break and what I should look at, I’m happy to try.  But I suspect it will be 25x more efficient for you to have a look if you can reproduce.

Re MacOS, I’m surprised you wouldn’t say that Windows is similarly an idiosyncratic system for display and other purposes.

[Message part 2 (text/html, inline)]

Message #62 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 11 Dec 2023 14:06:38 -0500

> On Dec 5, 2023, at 10:31 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
>> OK, so this was likely a false positive then.  I’m out of ideas.  Maybe Windows is magically immune.
> 
> Unlikely.  More likely is that we need some specific metrics of the
> displayed stuff to see the problem (which is therefore very rare).

I agree with this sentiment: it does seem to require special tuning to reveal itself.  I wanted to check to see if you had received my news that I reproduced the line-skip motion bug on a GNU/Linux system.  That would seem to confirm it’s a general issue with the display/motion code dealing with inline images.  Let me know if I can produce anything else to help with this one.  Thanks.

Message #65 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 11 Dec 2023 21:44:02 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Mon, 11 Dec 2023 14:06:38 -0500
> Cc: 67604 <at> debbugs.gnu.org
> 
> 
> 
> > On Dec 5, 2023, at 10:31 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> >> OK, so this was likely a false positive then.  I’m out of ideas.  Maybe Windows is magically immune.
> > 
> > Unlikely.  More likely is that we need some specific metrics of the
> > displayed stuff to see the problem (which is therefore very rare).
> 
> I agree with this sentiment: it does seem to require special tuning to reveal itself.  I wanted to check to see if you had received my news that I reproduced the line-skip motion bug on a GNU/Linux system.  That would seem to confirm it’s a general issue with the display/motion code dealing with inline images.  Let me know if I can produce anything else to help with this one.  Thanks.

I've eventually succeeded in reproducing it.  I will get to it when I
have time; however, with the current tempest on emacs-devel and other
urgent issues, I don't know when will that be.

Message #68 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Mon, 11 Dec 2023 17:06:24 -0500

> On Dec 11, 2023, at 2:44 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Mon, 11 Dec 2023 14:06:38 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>> 
>> 
>>> On Dec 5, 2023, at 10:31 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
>>>> OK, so this was likely a false positive then.  I’m out of ideas.  Maybe Windows is magically immune.
>>> 
>>> Unlikely.  More likely is that we need some specific metrics of the
>>> displayed stuff to see the problem (which is therefore very rare).
>> 
>> I agree with this sentiment: it does seem to require special tuning to reveal itself.  I wanted to check to see if you had received my news that I reproduced the line-skip motion bug on a GNU/Linux system.  That would seem to confirm it’s a general issue with the display/motion code dealing with inline images.  Let me know if I can produce anything else to help with this one.  Thanks.
> 
> I've eventually succeeded in reproducing it.  I will get to it when I
> have time; however, with the current tempest on emacs-devel and other
> urgent issues, I don't know when will that be.

Thanks for the update, glad you could reproduce it.  It will be waiting when you have a moment; hope the dust settles soon.

Message #71 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Tue, 3 Sep 2024 14:00:39 -0400

> On Dec 11, 2023, at 2:44 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> I've eventually succeeded in reproducing it.  I will get to it when I
> have time; however, with the current tempest on emacs-devel and other
> urgent issues, I don't know when will that be.

Following up on this vertical-motion with inline images bug.  My earlier recipe still seems to find special image sizes where (vertical-motion 1) skips the 3rd line.  As a reminder this is occurring only for certain window widths, near and below those where the green bar has just wrapped to line 2.

Message #74 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Tue, 6 May 2025 23:30:02 -0400

> On Dec 11, 2023, at 2:44 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Mon, 11 Dec 2023 14:06:38 -0500
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>> 
>> 
>>> On Dec 5, 2023, at 10:31 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
>>>> OK, so this was likely a false positive then.  I’m out of ideas.  Maybe Windows is magically immune.
>>> 
>>> Unlikely.  More likely is that we need some specific metrics of the
>>> displayed stuff to see the problem (which is therefore very rare).
>> 
>> I agree with this sentiment: it does seem to require special tuning to reveal itself.  I wanted to check to see if you had received my news that I reproduced the line-skip motion bug on a GNU/Linux system.  That would seem to confirm it’s a general issue with the display/motion code dealing with inline images.  Let me know if I can produce anything else to help with this one.  Thanks.
> 
> I've eventually succeeded in reproducing it.  I will get to it when I
> have time; however, with the current tempest on emacs-devel and other
> urgent issues, I don't know when will that be.

This bug remains present in Emacs v30.  I have managed some progress on it and feel I am quite close to narrowing to the culprit.

I've managed to get lldb working with a debug build of Emacs.  I can rapidly reproduce the bug, and am able to set breakpoints, step through the code, and print variable values, etc.

I've debugged through Fvertical_motion into the various move_it functions in xdisp.c.  They are very complicated functions, but here is what I've determined, referencing xdisp.c:

1. In `move_it_to`, when moving one vertical position down (op=MOVE_TO_VPOS), this is attempted:

	  /* If no TO_CHARPOS and no TO_X specified, stop at the
	     start of the line TO_VPOS.  */
            ...
		skip = move_it_in_display_line_to (it, -1, -1, 0);

I gather `move_it_in_display_line_to' here is supposed to be moving to the end of the display line, with op=0.

2. In that function, there is a bit of logic that looks like:

	  int single_glyph_width = it->pixel_width / it->nglyphs;
	  int new_x;
	  for (i = 0; i < it->nglyphs; ++i, x = new_x)
	    {
	      new_x = x + single_glyph_width;

In this example, we are moving glyph-by-glyph, accumulating each glpyh's pixel widths into to the new_x position.  This is continued, until a glyph is found which no longer fits on the visible line:

	      if (/* Lines are continued.  */
		  it->line_wrap != TRUNCATE
		  && (/* And glyph doesn't fit on the line.  */
		      new_x > it->last_visible_x

Here is the problem: if the first glyph on the line is wide (e.g. the green SVG rectangle in the example we were playing with), its width is incorrectly encoded in it->pixel_width as 7 pixels (the width of my normal font).  

As a result, for this very first glyph, `move_it_in_display_line_to' increments the x pixel position by too small a value (7 instead of 108 pixels).  This causes IT to eventually move past the end of the line, and on to the next wrapped line.  And if that next wrapped line is short enough, it even moves past that on to the following line, thus causing the full line "skip" we observed.

It's important to note that even when the line skip bug isn't present, if the first glyph is wide, vertical motion does unexpected things because of this mis-reporting of the first glyph's width.  If a wide glyph wraps to column 0, this motion bug is present.  It does not require an image, only a wide glyph. For example, I removed the green rectangle and inserted in its place:

  (insert (propertize "x" 'display '(space :width (108)) 'face '(:background "gray")))

and precisely the same issues occur.

I further verified this by placing a single character on the visible line just before the green rectangle.  With this change, on the 2nd time it->pixel_width is consulted above, it has the correct value of 108.  Only the *first* wrapped column position if affected.  Nor does this happen at the "real" start of a line, prior to its first wrap.  Only *wrapped* wide glyphs at visual column 0 have misreported pixel widths.

What I'm looking for is a reason why a wide glyph (only) at the very start of a wrapped visual display line would have its pixel width incorrectly stored, as if it were just a regular width character.  

I tried to get a sense of where this IT pixel_width is getting set, but it's referenced all over xdisp.c, and I have only a rudimentary understanding of the iterator structure (and its glyph rows, etc.).

If you have any hunches where this mistaken it->pixel_width might be sneaking in for wrapped wide glyphs, I'd be very happy to investigate further using my debug setup here.

Thanks,

JD

Message #77 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Wed, 07 May 2025 15:00:35 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Tue, 6 May 2025 23:30:02 -0400
> Cc: 67604 <at> debbugs.gnu.org
> 
> > I've eventually succeeded in reproducing it.  I will get to it when I
> > have time; however, with the current tempest on emacs-devel and other
> > urgent issues, I don't know when will that be.
> 
> This bug remains present in Emacs v30.

Sorry, I couldn't find the time to look at this.

> What I'm looking for is a reason why a wide glyph (only) at the very start of a wrapped visual display line would have its pixel width incorrectly stored, as if it were just a regular width character.  
> 
> I tried to get a sense of where this IT pixel_width is getting set, but it's referenced all over xdisp.c, and I have only a rudimentary understanding of the iterator structure (and its glyph rows, etc.).
> 
> If you have any hunches where this mistaken it->pixel_width might be sneaking in for wrapped wide glyphs, I'd be very happy to investigate further using my debug setup here.

it->pixel_width comes from the call to PRODUCE_GLYPHS (which is a
macro that calls gui_produce_glyphs).  For characters,
gui_produce_glyphs does the job itself, under this condition:

  if (it->what == IT_CHARACTER)

For images, it calls produce_image_glyph.

I hope this is the information you were looking for.  If not, please
ask more specific questions.

Thanks.

Message #80 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Thu, 8 May 2025 16:58:36 -0400

[Message part 1 (text/plain, inline)]

> On May 7, 2025, at 8:00 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> 
>> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Tue, 6 May 2025 23:30:02 -0400
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>>> I've eventually succeeded in reproducing it.  I will get to it when I
>>> have time; however, with the current tempest on emacs-devel and other
>>> urgent issues, I don't know when will that be.
>> 
>> This bug remains present in Emacs v30.
> 
> Sorry, I couldn't find the time to look at this.

No worries at all. You juggle many things and it was a good impetus to get a src debugging setup working (I wish realgud worked better with lldb, but command line is ok for now).  It’s also a fairly subtle and specific bug, even though I think it underlies many(/all?) small movement oddities I’ve noticed over the years with inline wrapped wide glyphs.

>> What I'm looking for is a reason why a wide glyph (only) at the very start of a wrapped visual display line would have its pixel width incorrectly stored, as if it were just a regular width character.  
>> 
>> I tried to get a sense of where this IT pixel_width is getting set, but it's referenced all over xdisp.c, and I have only a rudimentary understanding of the iterator structure (and its glyph rows, etc.).
>> 
>> If you have any hunches where this mistaken it->pixel_width might be sneaking in for wrapped wide glyphs, I'd be very happy to investigate further using my debug setup here.
> 
> it->pixel_width comes from the call to PRODUCE_GLYPHS (which is a
> macro that calls gui_produce_glyphs).  For characters,
> gui_produce_glyphs does the job itself, under this condition:
> 
>  if (it->what == IT_CHARACTER)
> 
> For images, it calls produce_image_glyph.
> 
> I hope this is the information you were looking for.  If not, please
> ask more specific questions.

I appreciate these tips.

I took a long look, and my initial assessment was incorrect — the glyph pixel sizes are correct, but the iterator is confused about where the line start is.  IT thinks it is at the beginning of the line, but has in fact over-shot, just to the right of the wrapped wide glyph.

Here's a snapshot of the problem.  There are no images in the buffer any longer — the gray boxes are both simple specified space applied on single SPC chars.  The behavior is the same as before.  Point jumps from the beginning of the 2nd line on the light gray stretch right to JUMPS HERE.  Other forms of incorrect motion are evident, but this is the most dramatic, so I stick with it.

[PastedGraphic-6.png (image/png, inline)]

[Message part 3 (text/plain, inline)]

Here is the stack I'm investigating:

  * frame #0: 0x00000001000357c0 Emacs`move_it_in_display_line_to(it=0x0000000170793220, to_charpos=-1, to_x=-1, op=0x0) at xdisp.c:10118:11
    frame #1: 0x00000001000314b0 Emacs`move_it_to(it=0x0000000170793220, to_charpos=-1, to_x=-1, to_y=-1, to_vpos=1, op=4) at xdisp.c:10667:10
    frame #2: 0x000000010002cadc Emacs`move_it_by_lines(it=0x0000000170793220, dvpos=1) at xdisp.c:11240:7
    frame #3: 0x00000001001d787c Emacs`Fvertical_motion(lines=(EMACS_INT) $4 = 1, window=(struct Lisp_Symbol *) $22 = 0x0000000100a73070, cur_col=(struct Lisp_Symbol *) $43 = 0x0000000100a73070) at indent.c:2406:4
    frame #4: 0x0000000100240adc Emacs`funcall_subr_1(subr=0x000000010045aae0, numargs=1, args=(struct Lisp_Symbol *) $72 = 0x0000000218bbb270) at eval.c:3176:15

At this location, I have:

(lldb) br en
All breakpoints enabled. (1 breakpoints)
Process 35347 stopped
* thread #2, name = 'org.gnu.Emacs.lisp-main', stop reason = breakpoint 1.1
    frame #0: 0x00000001000357c0 Emacs`move_it_in_display_line_to(it=0x0000000170793220, to_charpos=-1, to_x=-1, op=0x0) at xdisp.c:10118:11
   10115    10116 Note that both for tabs and padding glyphs, all glyphs have
   10117 the same width.  */
-> 10118      if (it->nglyphs)
   10119 {
   10120  /* More than one glyph or glyph doesn't fit on line.  All
   10121     glyphs have the same width.  */
Target 0: (Emacs) stopped.
(lldb) p it->current
(display_pos) {
  pos = (charpos = 44, bytepos = 44)
  overlay_string_index = -1
  string_pos = (charpos = -1, bytepos = -1)
  dpvec_index = -1
}
(lldb) p it->hpos
(int) 0
(lldb) p it->current_x
(int) 7
(lldb) p it->last_visible_x
(int) 546
(lldb) p it->pixel_width 
(int) 7
(lldb) 

Note that the light gray specified space is at pos=43.  I conclude IT thinks it is at column 0 (assuming HPOS is meaningful), and has a current_x of 7 (one normal char width).  When in reality, due to the wide glyph, it got pushed ahead to the pink position above, at pos=44.  From there it iterates until new_x >= last_visible_x, and in so doing, "falls off the line".  If it falls to the final newline of the subsequent wrapped line, this results in a "line jump".

GLYPH SKIP:
===========

Narrowing down further, here is the location where point first gets pushed ahead to the pink box in the screenshot (in indent.c:Fvertical_motion, line 2315).  Just prior to this, IT is at buffer pos=1 (having been moved there by `reseat_at_previous_visible_line_start').  After this, it's at pos=44, on the pink box.

/* When the position we started from is covered by a display
  string, move_it_to will overshoot it, while vertical-motion
  wants to put the cursor _before_ the display string.  So in
  that case, we move to buffer position before the display
  string, and avoid overshooting.  But if the position before
  the display string is a newline, we don't do this, because
  otherwise we will end up in a screen line that is one too
  far back.  */
move_it_to (&it,
   (!disp_string_at_start_p
    || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
   ? PT
   : PT - 1,
   -1, -1, -1, MOVE_TO_POS);

This comment is very telling, and set off some bells.  That's seems to be EXACTLY what is happening here: `move_it_to(&it, PT, MOVE_TO_POS)' overshoots the display element (here, specified space), and lands at pos=44 (pink box).   

So why is this commented-upon "fix" not working?  Because  disp_string_at_start_p=false, since a display *image* or *specified space* is not a display *string*.

This flag is set above:

else if (it.method == GET_FROM_STRING)
{
 const char *s = SSDATA (it.string);
 const char *e = s + SBYTES (it.string);

 disp_string_at_start_p =
 /* If it.area is anything but TEXT_AREA, we need not bother
    about the display string, as it doesn't affect cursor
    positioning.  */
   it.area == TEXT_AREA
   && it.string_from_display_prop_p
   /* A display string on anything but buffer text (e.g., on
      an overlay string) doesn't affect cursor positioning.  */
   && (it.sp > 0 && it.stack[it.sp - 1].method == GET_FROM_BUFFER);

SOLUTION(S):
============

It would seem the simple solution is to do the same thing for images and stretches as is done for display strings: move to the position right *before* PT. 

That could be as simple as enabling disp_string_at_start_p when starting on a (real) IMAGE or STRETCH.  That does indeed fix the bug, but unfortunately it results in a new bug: now moving vertically *upwards* across wrapped wide glyphs jumps an extra line up.  Sigh.

I've tried various approaches, but have struggled to come up with a full fix.  

A few questions that might get me closer:

1. Why do IMAGES/STRETCHES get a different overshoot count?

      else
it_overshoot_count =
 /* If image_id is negative, it's a fringe bitmap, which by
    definition doesn't affect display in the text area.  */
 !((it.method == GET_FROM_IMAGE && it.image_id >= 0)
   || it.method == GET_FROM_STRETCH);

A (real) IMAGE or STRETCH at the starting position of the vertical movement changes the overshoot to 0 (it seems to be 1 by default).  Yet no allowance is made for reseat -> move_it_to skipping past the IMAGE/STRETCH, as is done for display *strings*.  I don't see why the mere presence of an IMAGE/STRETCH should tell you anything concrete about the overshoot.

2. How can you tell if the position prior to the IMAGE/STRETCH is on a prior visual line?

Later in `Fvertical_motion' there is this test, which looks very promising:

      else if (IT_CHARPOS (it) == PT - 1
      && FETCH_BYTE (PT_BYTE - 1) == '\n'
      && nlines <= 0)
{
 /* The position we started from was covered by a display
    property, so we moved to position before the string, and
    backed up one line, because the character at PT - 1 is
    a newline.  So we need one less line to go up (or exactly
    one line to go down if nlines == 0).  */
 nlines++;
 /* But we still need to record that one line, in order to
    return the correct value to the caller.  */
 vpos_init = -1;

 overshoot_handled = 1;
}

But note, it looks for an *explicit newline* before the display property, which is obviously not there for wrapped lines.  I've tried to use it.vpos to find out if you got moved by a screen line, but that seems to be more of a "delta vpos", apparently set only AFTER moving back to the starting point with move_it_to.

3. It seems what's relevant for wide glyphs is not only "Was there a newline behind you?", but "Was there a newline behind you, or was the position there at a lower vpos?".  Is there a way to use vpos to make a test like this?

The only idea I could think of is (assuming no newline behind):

a. First move to PT.  Store vpos.
b. If you started on a display property (not on a newline), now move back to PT-1.  
c. Store and compare the vpos values.
c. If new_vpos<stored_vpos, your display entity is at the beginning of a visual line, and you'd better update your nlines/overshoot.

But I haven't gotten it to work. I'm also not sure if that will mess up the x position, etc. 

I think this proposed fix should apply equally to display strings, images, and stretches which have been wrapped (it.line_wrap != TRUNCATE).  

Thoughts/ideas?

Message #83 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Fri, 09 May 2025 19:17:05 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Thu, 8 May 2025 16:58:36 -0400
> Cc: 67604 <at> debbugs.gnu.org
> 
> At this location, I have:
> 
> (lldb) br en
> All breakpoints enabled. (1 breakpoints)
> Process 35347 stopped
> * thread #2, name = 'org.gnu.Emacs.lisp-main', stop reason = breakpoint 1.1
>     frame #0: 0x00000001000357c0 Emacs`move_it_in_display_line_to(it=0x0000000170793220, to_charpos=-1, to_x=-1, op=0x0) at xdisp.c:10118:11
>    10115    10116 Note that both for tabs and padding glyphs, all glyphs have
>    10117 the same width.  */
> -> 10118      if (it->nglyphs)
>    10119 {
>    10120  /* More than one glyph or glyph doesn't fit on line.  All
>    10121     glyphs have the same width.  */
> Target 0: (Emacs) stopped.
> (lldb) p it->current
> (display_pos) {
>   pos = (charpos = 44, bytepos = 44)
>   overlay_string_index = -1
>   string_pos = (charpos = -1, bytepos = -1)
>   dpvec_index = -1
> }
> (lldb) p it->hpos
> (int) 0
> (lldb) p it->current_x
> (int) 7
> (lldb) p it->last_visible_x
> (int) 546
> (lldb) p it->pixel_width 
> (int) 7
> (lldb) 
> 
> Note that the light gray specified space is at pos=43.  I conclude IT thinks it is at column 0 (assuming HPOS is meaningful), and has a current_x of 7 (one normal char width).  When in reality, due to the wide glyph, it got pushed ahead to the pink position above, at pos=44.

Thanks, I think you made good progress.

But you are looking at the horizontal coordinates too early: they have
not yet been updated at that point.  They are updated inside the for
loop which starts a few lines down.

> From there it iterates until new_x >= last_visible_x, and in so doing, "falls off the line".  If it falls to the final newline of the subsequent wrapped line, this results in a "line jump".

You lost me here.  I don't see "new_x >= it->last_visible_x" in the
source, so which line did you have in mind?

> GLYPH SKIP:
> ===========
> 
> Narrowing down further, here is the location where point first gets pushed ahead to the pink box in the screenshot (in indent.c:Fvertical_motion, line 2315).  Just prior to this, IT is at buffer pos=1 (having been moved there by `reseat_at_previous_visible_line_start').  After this, it's at pos=44, on the pink box.
> 
> /* When the position we started from is covered by a display
>   string, move_it_to will overshoot it, while vertical-motion
>   wants to put the cursor _before_ the display string.  So in
>   that case, we move to buffer position before the display
>   string, and avoid overshooting.  But if the position before
>   the display string is a newline, we don't do this, because
>   otherwise we will end up in a screen line that is one too
>   far back.  */
> move_it_to (&it,
>    (!disp_string_at_start_p
>     || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
>    ? PT
>    : PT - 1,
>    -1, -1, -1, MOVE_TO_POS);
> 
> This comment is very telling, and set off some bells.  That's seems to be EXACTLY what is happening here: `move_it_to(&it, PT, MOVE_TO_POS)' overshoots the display element (here, specified space), and lands at pos=44 (pink box).   
> 
> So why is this commented-upon "fix" not working?  Because  disp_string_at_start_p=false, since a display *image* or *specified space* is not a display *string*.
> 
> This flag is set above:
> 
> else if (it.method == GET_FROM_STRING)
> {
>  const char *s = SSDATA (it.string);
>  const char *e = s + SBYTES (it.string);
> 
>  disp_string_at_start_p =
>  /* If it.area is anything but TEXT_AREA, we need not bother
>     about the display string, as it doesn't affect cursor
>     positioning.  */
>    it.area == TEXT_AREA
>    && it.string_from_display_prop_p
>    /* A display string on anything but buffer text (e.g., on
>       an overlay string) doesn't affect cursor positioning.  */
>    && (it.sp > 0 && it.stack[it.sp - 1].method == GET_FROM_BUFFER);
> 
> SOLUTION(S):
> ============
> 
> It would seem the simple solution is to do the same thing for images and stretches as is done for display strings: move to the position right *before* PT. 
> 
> That could be as simple as enabling disp_string_at_start_p when starting on a (real) IMAGE or STRETCH.  That does indeed fix the bug, but unfortunately it results in a new bug: now moving vertically *upwards* across wrapped wide glyphs jumps an extra line up.  Sigh.

I'm not sure your conclusions are correct.  If they were, it would
mean that vertical-motion doesn't work at all when the first glyph on
a line is an image or a stretch glyph (what you call "specified
space").  But that is not true, is it?

I think this problem is specific to the situation when the following
happen all at once:

 . lines are wrapped (it->line_wrap = WORD_WRAP)
 . the image or the stretch glyph don't fit on their line and are
   wrapped to the next line
 . vertical-motion is invoked when point is on or after that image or
   stretch

Am I right?

If I'm right, then we should try understanding why in this particular
situation the code doesn't work, whereas it does work when, say, the
image or stretch are in a non-continuation line or are not the first
glyph.

> 1. Why do IMAGES/STRETCHES get a different overshoot count?

I don't remember, sorry.  If you really need to know (and I'm not
sure), you will have to use "git -L" to see the history of this code,
and then look up the bug reports which led to this code.

> 2. How can you tell if the position prior to the IMAGE/STRETCH is on a prior visual line?

By looking at the Y coordinate (it.current_t or it.vpos), I'd say.

> Later in `Fvertical_motion' there is this test, which looks very promising:
> 
>       else if (IT_CHARPOS (it) == PT - 1
>       && FETCH_BYTE (PT_BYTE - 1) == '\n'
>       && nlines <= 0)

This only works if the previous line ended in a newline, but will not
work with continuation lines.

> But note, it looks for an *explicit newline* before the display property, which is obviously not there for wrapped lines.  I've tried to use it.vpos to find out if you got moved by a screen line, but that seems to be more of a "delta vpos", apparently set only AFTER moving back to the starting point with move_it_to.

Yes, vpos counts from zero, where the iterator was initialized.

> 3. It seems what's relevant for wide glyphs is not only "Was there a newline behind you?", but "Was there a newline behind you, or was the position there at a lower vpos?".  Is there a way to use vpos to make a test like this?

I don't know yet, the answer should reveal itself if you step through
the code and see what happens there in this particular situation (but
not in other similar ones).

> The only idea I could think of is (assuming no newline behind):
> 
> a. First move to PT.  Store vpos.
> b. If you started on a display property (not on a newline), now move back to PT-1.  
> c. Store and compare the vpos values.
> c. If new_vpos<stored_vpos, your display entity is at the beginning of a visual line, and you'd better update your nlines/overshoot.
> 
> But I haven't gotten it to work. I'm also not sure if that will mess up the x position, etc. 

It's too early to devise solutions, because we don't understand the
root cause of the problem yet.  At least I don't.

> I think this proposed fix should apply equally to display strings, images, and stretches which have been wrapped (it.line_wrap != TRUNCATE).  

I'm not yet convinced, since vertical-motion does work in general when
images and stretch glyphs are around.

Message #86 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: jdtsmith <at> gmail.com
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Fri, 09 May 2025 20:49:54 +0300

> Cc: 67604 <at> debbugs.gnu.org
> Date: Fri, 09 May 2025 19:17:05 +0300
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> I'm not yet convinced, since vertical-motion does work in general when
> images and stretch glyphs are around.

Btw, you seem to be working on the emacs-30 branch, or close to that.
Did you try this with master, and if so, did the same issue happen
there?

Message #89 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Fri, 9 May 2025 20:51:40 -0400

> On May 9, 2025, at 12:17 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> Thanks, I think you made good progress.
> 
> But you are looking at the horizontal coordinates too early: they have
> not yet been updated at that point.  They are updated inside the for
> loop which starts a few lines down.

Yeah it's a challenge to know when various members of IT are yet valid. 

>> From there it iterates until new_x >= last_visible_x, and in so doing, "falls off the line".  If it falls to the final newline of the subsequent wrapped line, this results in a "line jump".
> 
> You lost me here.  I don't see "new_x >= it->last_visible_x" in the
> source, so which line did you have in mind?

This was referring (obliquely) back to my prior message.  It happens a couple levels deeper, via move_it_by_lines (&it, min (PTRDIFF_MAX, nlines)), to move_it_to:

	  /* If no TO_CHARPOS and no TO_X specified, stop at the
	     start of the line TO_VPOS.  */
	  if ((op & (MOVE_TO_X | MOVE_TO_POS)) == 0)
	    {
	      if (it->vpos == to_vpos)
		{
		  reached = 1;
		  break;
		}
	      else
		skip = move_it_in_display_line_to (it, -1, -1, 0);

And in move_it_in_display_line_to(it, -1, -1, 0):

	      if (/* Lines are continued.  */
		  it->line_wrap != TRUNCATE
		  && (/* And glyph doesn't fit on the line.  */
		      new_x > it->last_visible_x

That new_x is getting incremented by it->pixel_width until it falls off the screen line, but I believe it starts at the wrong value, since it's pushed to the right of the stretch/image without knowing it.   That's the (as yet unsubstantiated) theory at least.

>> SOLUTION(S):
>> ============
>> 
>> It would seem the simple solution is to do the same thing for images and stretches as is done for display strings: move to the position right *before* PT. 
>> 
>> That could be as simple as enabling disp_string_at_start_p when starting on a (real) IMAGE or STRETCH.  That does indeed fix the bug, but unfortunately it results in a new bug: now moving vertically *upwards* across wrapped wide glyphs jumps an extra line up.  Sigh.
> 
> I'm not sure your conclusions are correct.  If they were, it would
> mean that vertical-motion doesn't work at all when the first glyph on
> a line is an image or a stretch glyph (what you call "specified
> space").  But that is not true, is it?

I admit I don't have a complete picture, but yes, a wrapped stretch/image at the beginning of a screen line does appear to cause other small vertical motion issues: cursor moves a bit off the expected column, after line move, etc.  I don't understand exactly why it needs this "fine tuning" to manifest as a full line skip, so I could easily be coming to an incorrect understanding (or more likely an understanding of a non-essential part of the problem).

> I think this problem is specific to the situation when the following
> happen all at once:
> 
> . lines are wrapped (it->line_wrap = WORD_WRAP)
> . the image or the stretch glyph don't fit on their line and are
>   wrapped to the next line
> . vertical-motion is invoked when point is on or after that image or
>   stretch

> Am I right?


Yes, this is all correct, except you only see the issue when point is on or directly after (next character) the image or stretch. 

And it's a good point.  The fact that it doesn't appear on normal continuation lines, only WORD_WRAP continuations must be significant.  And it's suspicious that it only appears when an image/stretch is finely balanced between not yet wrapped and just wrapped.  You might recall that I can only reproduce the bug when the window width shrinks just enough to wrap the green image, then one char narrower.  And then I still have to adjust the red image width on line 1 to "fine tune" the balance further (by "eating up" some of that one char wrap margin).

> If I'm right, then we should try understanding why in this particular
> situation the code doesn't work, whereas it does work when, say, the
> image or stretch are in a non-continuation line or are not the first
> glyph

Or indeed on a non WORD_WRAP continuation line, as far as I can tell.  And yet, I do certainly see that Fvertical_motion incorrectly returns IT to the pink box in the image I provided.  For example, if you introduce a normal character adjacent before the wrapped image, it does not get moved ahead.  And that all sounds very similar to the issue with display strings that is already treated in Fvertical_motion.  This happens via this combo:

       reseat_at_previous_visible_line_start (&it);
	....
       move_it_to (&it, PT, -11, -1, -1, MOVE_TO_POS);

which I gather is primarily for getting the correct x position. But your point raises the very good question of why this doesn't happen on normal continuation lines.  What's special about word wrap here?

>> 2. How can you tell if the position prior to the IMAGE/STRETCH is on a prior visual line?
> 
> By looking at the Y coordinate (it.current_t or it.vpos), I'd say.

But as you mentioned for horizontal positioning, it doesn't get updated with a valid value until you've performed the move.  So you'd have to do two moves, one before, one "on" the image/glpyh.  There's probably a better way.

>> Later in `Fvertical_motion' there is this test, which looks very promising:
>> 
>>      else if (IT_CHARPOS (it) == PT - 1
>>      && FETCH_BYTE (PT_BYTE - 1) == '\n'
>>      && nlines <= 0)
> 
> This only works if the previous line ended in a newline, but will not
> work with continuation lines.

Of course.  My point was that the commentary is highly suggestive of the very thing that went wrong:

	  /* The position we started from was covered by a display
	     property, so we moved to position before the string, and
	     backed up one line, because the character at PT - 1 is
	     a newline.  So we need one less line to go up (or exactly
	     one line to go down if nlines == 0).  */

So we'd just need another test for "PT-1 fell back to the prior screen line" to go along with checking for a newline to fix the "counter-bug" induced by my original fix.  But I agree we need a better understanding of the issue before devising a fix.

>> 3. It seems what's relevant for wide glyphs is not only "Was there a newline behind you?", but "Was there a newline behind you, or was the position there at a lower vpos?".  Is there a way to use vpos to make a test like this?
> 
> I don't know yet, the answer should reveal itself if you step through
> the code and see what happens there in this particular situation (but
> not in other similar ones).

I'll try stepping with the same configuration but without word wrapping and see what differs.

> It's too early to devise solutions, because we don't understand the
> root cause of the problem yet.  At least I don't.

Agreed.  I have hints of it, but no fully understanding.  Thanks for the good thoughts.  I'll keep experimenting.  If you have other thoughts of things to try, please do share.

Message #92 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Sat, 10 May 2025 10:35:04 -0400

> On May 9, 2025, at 1:49 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> Btw, you seem to be working on the emacs-30 branch, or close to that.
> Did you try this with master, and if so, did the same issue happen
> there?

I'm on emacs-30 branch.  I tested in a recent build of master and the bug remains there too.

Message #95 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Sat, 17 May 2025 23:13:45 -0400

[Message part 1 (text/plain, inline)]

I have some progress to report.  To keep things simple, I compared two cases, both moving vertically down by one line, starting with point on the just-wrapped green image:

1. The "fine-tuned" case, which skips a line.
2. A "nearby" case with the same window width that doesn't exhibit the bug (no skip).

Recall that this bug/no-bug tuning proceeds by making small changes to the width of the red image on the first screen line, effectively "pushing" the green image further along, pixel by pixel.  You can move between cases 1 and 2 by changing the red image width by as little as one pixel.

By comparing these two cases, I've tracked things back to where I started: `move_it_in_display_line_to'.  I investigated various backtraces, all of which land via vertical-motion in this function:

* thread #5, name = 'org.gnu.Emacs.lisp-main', stop reason = step over
  * frame #0: 0x0000000100038374 Emacs`move_it_in_display_line_to(it=0x0000000170793950, to_charpos=89, to_x=-1, op=MOVE_TO_POS) at xdisp.c:10594:10
    frame #1: 0x0000000100031bd0 Emacs`move_it_to(it=0x0000000170793950, to_charpos=89, to_x=-1, to_y=-1, to_vpos=-1, op=8) at xdisp.c:10841:9
    frame #2: 0x000000010002d600 Emacs`start_display(it=0x0000000170793950, w=0x000000012a808cd0, pos=(charpos = 89, bytepos = 89)) at xdisp.c:3763:4
    frame #3: 0x00000001001d72d4 Emacs`Fvertical_motion(lines=(EMACS_INT) $259 = 1, window=(struct Lisp_Symbol *) $277 = 0x0000000100a73070, cur_col=(struct Lisp_Symbol *) $298 = 0x0000000100a73070) at indent.c:2260:7

There are many paths from vertical-motion which lead here, but I suspect this is the critical one.  It occurs on `start_display', which does:

	  reseat_at_previous_visible_line_start (it);
	  move_it_to (it, CHARPOS (pos), -1, -1, -1, MOVE_TO_POS);

That move_it_to(MOVE_TO_POS) is what initially sets up the bug.  I believe the idea here is to correct initialize the position.

By setting breakpoints when the iterator is on the green image (charpos=89), I discovered the cause of the pixel-precise "fine-tuning" behavior.  It results from this code in xdisp.c:move_it_in_display_line_to:

	      if (/* Lines are continued.  */
		  it->line_wrap != TRUNCATE
		...
		  if (/* IT->hpos == 0 means the very first glyph
			 doesn't fit on the line, e.g. a wide
			 image.  */
		      it->hpos == 0  
		      || (new_x == it->last_visible_x  //**!!! FINE-TUNING
			  && FRAME_WINDOW_P (it->f)))
		    {
		      ++it->hpos;  //!!! The fine-tuned case lands here: new_x==last_visible_x
		      it->current_x = new_x;
		      /* The character's last glyph just barely fits
			 in this row.  */
			....

			  prev_method = it->method;
			  if (it->method == GET_FROM_BUFFER)
			    prev_pos = IT_CHARPOS (*it);
			  set_iterator_to_next (it, true);
			...
		  else if (wrap_it.sp >= 0)  ///**!!! THIS IS THE WRAP RESTORATION WE MISS
		    {
		      RESTORE_IT (it, &wrap_it, wrap_data);
		      atpos_it.sp = -1;
		      atx_it.sp = -1;
		    }

I.e. if the position of the green image is tuned so that it "barely" fits on a line, we take another code path.  For this special fine-tuned case, this prevents wrap_it from being restored, which leaves the it->current_x position after start_display at 0 instead of the correct value of 108 (the width of the green image).  The rest of the bug proceeds along the lines of my previous conjecture about point being pushed ahead past the wrapped image.

In terms of the special case path for "last glyph just fits":  this is appropriate for (non-word) window wrapping, but not, I believe word-wrapping.  Why? I noticed that wrapped words carry _all_ their trailing whitespace with them when they wrap (presumably to keep WS from wrapping to the start of a display line).  In other words, why care if the green image itself can "barely fit" on a line, when in fact, it will _already have wrapped_ at a larger window width, due to the trailing whitespace (of which there must be at least one).

BTW, the "barely fits" test I zeroed in on is similar in xdisp.c on master, only adding a check for line-number pixel width:

		  if (/* IT->hpos == 0 (modulo line-number width) means
			 the very first glyph doesn't fit on the line,
			 e.g., a wide image.  */
		      it->hpos == 0 + (it->lnum_width ? it->lnum_width + 2 : 0)
		      || (new_x == it->last_visible_x
			  && FRAME_WINDOW_P (it->f)))

Here is a proposed patch, which skips the fine-tuned branch when word-wrapping, if the next position can't wrap.  `char_can_wrap_before' is actually called before this, so it's possible the result could be saved and reused.   

Definitely poke around and be sure this makes sense.  That was a _lot_ of hunting!

[inline-image-line-skip.patch (application/octet-stream, attachment)]

Message #98 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Tue, 27 May 2025 16:23:30 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Sat, 17 May 2025 23:13:45 -0400
> Cc: 67604 <at> debbugs.gnu.org
> 
> I have some progress to report.  To keep things simple, I compared two cases, both moving vertically down by one line, starting with point on the just-wrapped green image:
> 
> 1. The "fine-tuned" case, which skips a line.
> 2. A "nearby" case with the same window width that doesn't exhibit the bug (no skip).

Thanks, and sorry for my long delays.  These issues need some time and
a quiet environment, and I don't easily get both together...

> In terms of the special case path for "last glyph just fits":  this is appropriate for (non-word) window wrapping, but not, I believe word-wrapping.  Why? I noticed that wrapped words carry _all_ their trailing whitespace with them when they wrap (presumably to keep WS from wrapping to the start of a display line).  In other words, why care if the green image itself can "barely fit" on a line, when in fact, it will _already have wrapped_ at a larger window width, due to the trailing whitespace (of which there must be at least one).

That condition, i.e.

		  if (/* IT->hpos == 0 means the very first glyph
			 doesn't fit on the line, e.g. a wide
			 image.  */
		      it->hpos == 0  
		      || (new_x == it->last_visible_x  //**!!! FINE-TUNING
			  && FRAME_WINDOW_P (it->f)))

is because GUI windows can show the cursor on the fringe.  So even if
a glyph "barely fits", we can show the cursor after it, and don't need
an extra column of the window's text-area for that purpose.  See
overflow-newline-into-fringe.

Given what you say, it might be TRT to not go there under word-wrap
when the next character can wrap, but please verify that:

 . it->current was already moved to the character after the image,
   because AFAIR char_can_wrap_after looks at the character at
   it->current
 . when this problem happens with the image right next to a newline
   that ends a line, we can still show the cursor on the fringe when
   you move point to the buffer position after the image, both with
   and without word-wrap

If the patch passes these two tests, feel free to install on the
master branch.

> That was a _lot_ of hunting!

It usually is, with tricky display engine issues such as this one.

Thanks!

Message #101 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Wed, 28 May 2025 19:42:50 -0400

[Message part 1 (text/plain, inline)]

> On May 27, 2025, at 9:23 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Sat, 17 May 2025 23:13:45 -0400
>> Cc: 67604 <at> debbugs.gnu.org
>> 
>> I have some progress to report.  To keep things simple, I compared two cases, both moving vertically down by one line, starting with point on the just-wrapped green image:
>> 
>> 1. The "fine-tuned" case, which skips a line.
>> 2. A "nearby" case with the same window width that doesn't exhibit the bug (no skip).
> 
> Thanks, and sorry for my long delays.  These issues need some time and
> a quiet environment, and I don't easily get both together...

No problem.  Indeed this stretch of code requires some serious concentration.  In fact here I concentrated so much in responding to you that another yet-deeper possibility occurred to me at the very end.  Apologies for the (possible) late plot twist.

>> In terms of the special case path for "last glyph just fits":  this is appropriate for (non-word) window wrapping, but not, I believe word-wrapping.  Why? I noticed that wrapped words carry _all_ their trailing whitespace with them when they wrap (presumably to keep WS from wrapping to the start of a display line).  In other words, why care if the green image itself can "barely fit" on a line, when in fact, it will _already have wrapped_ at a larger window width, due to the trailing whitespace (of which there must be at least one).
> 
> That condition, i.e.
> 
>  if (/* IT->hpos == 0 means the very first glyph
> doesn't fit on the line, e.g. a wide
> image.  */
>      it->hpos == 0  
>      || (new_x == it->last_visible_x  //**!!! FINE-TUNING
>  && FRAME_WINDOW_P (it->f)))
> 
> is because GUI windows can show the cursor on the fringe.  So even if
> a glyph "barely fits", we can show the cursor after it, and don't need
> an extra column of the window's text-area for that purpose.  See
> overflow-newline-into-fringe.

Got it, thanks.  I did notice that the overflow feature seems to work even when the branch isn't taken; see below.

> Given what you say, it might be TRT to not go there under word-wrap
> when the next character can wrap, but please verify that:

The idea is not to go there under word-wrap _unless_ you can wrap directly after (i.e. IT is on a WS).  Because otherwise, the just-fitting glyph at it->current (e.g., the green box) will certainly wrap — it only "just fits" if it's followed by a newline.  Technically it's true that a trailing WS that seems to "just fit" could in fact also wrap.  I.e. imagine something like the first trailing space just filling the window width in this:

 "XXXXX    "

But apparently avoiding the branch in this case doesn't matter in practice (likely because BUFFER_POS_REACHED_P works fine for such chars; see below).

> . it->current was already moved to the character after the image,
>   because AFAIR char_can_wrap_after looks at the character at
>   it->current

When the branch above would have been taken — an outcome which is now disabled by my patch — it->current is at pos=89, on the green box.  It seems `char_can_wrap_after' just checks if the current char is whitespace (false, for the green box).  So at pos=89, the previous char (88, a WS) could wrap after, and the current char (89, green box) can wrap before, so wrap_it is stored as a suitable wrap position.  After this test, at pos=89, may_wrap=false, since you can't wrap directly after a non-WS char.

I think this all checks out.  In fact on review I noticed we can just use the saved variable may_wrap instead of using char_can_wrap_after(it); they are equivalent.

> . when this problem happens with the image right next to a newline
>   that ends a line, we can still show the cursor on the fringe when
>   you move point to the buffer position after the image, both with
>   and without word-wrap

I added a newline after the green box.  By adjusting the red box width until the green box "just fits" on the line (144pix), I can confirm with this patch that the cursor correctly overflows from the green box into the fringe, with and without word wrap.  It would appear however that the above branch is now never taken with WORD_WRAP except when you are on a whitespace which "just fits".   But in all cases the fringe-newline still appears to work.

> If the patch passes these two tests, feel free to install on the
> master branch.

It did not pass test 1, but I don't think it should have, since I believe TRT is to ignore just-fitting non-WS glyphs that cannot wrap directly after (since they themselves will wrap!).

I modified the patch to give priority to the hpos==0 (overly wide image) test, and used the saved variable may_wrap; see attached.  

PLOT TWIST:
===========

This all works fine, but there was one tiny distinction that bothered me as I tested fringe newline overflow.   When moving past (with forward-char) the "just fits" green box when it is followed by a newline, rather than returning MOVE_POS_MATCH_OR_ZV, move_it_in_display_line_to now returns MOVE_LINE_CONTINUED.  Again, this results in no operational difference I could find, and maybe it matters not one bit, but it gave me pause.

This thought then brought me deeper, to this macro:

#define BUFFER_POS_REACHED_P()					\
  ((op & MOVE_TO_POS) != 0					\
   && BUFFERP (it->object)					\
   && (IT_CHARPOS (*it) == to_charpos				\
       || ((!it->bidi_p						\
	    || BIDI_AT_BASE_LEVEL (it->bidi_it))		\
	   && IT_CHARPOS (*it) > to_charpos)			\
       || (it->what == IT_COMPOSITION				\
	   && ((IT_CHARPOS (*it) > to_charpos			\
		&& to_charpos >= it->cmp_it.charpos)		\
	       || (IT_CHARPOS (*it) < to_charpos		\
		   && to_charpos <= it->cmp_it.charpos))))	\
   && (it->method == GET_FROM_BUFFER				\
       || (it->method == GET_FROM_DISPLAY_VECTOR		\
	   && it->dpvec + it->current.dpvec_index + 1 >= it->dpend)))

When the iterator arrives on the green box, it has reached its destination (to_charpos=89).  And yet, it does not stop.  Moving past the green box is what causes all the problems.  Why does it move on?  Everything here in this macro test is true, _except_ for it->method==GET_FROM_BUFFER and following.  As we're on an image, we have it->method=GET_FROM_IMAGE.  So, it fails.

Can you never actually reach a TO_CHARPOS position that is on an image (or stretch)?  Are you doomed to move past it?  That I don't understand.  If this macro returned True for pos=89, as I think it perhaps should, this whole issue would never have occurred.  The just-fits branch would notice that the destination position (namely, the green image) had been reached, save the position in atpos_it, and just prior to exiting, restore it.  

And yet, I think based on the various comments I've seen in the code about "moving past" images, BUFFER_POS_REACHED_P is perhaps intentionally False for images/stretches.  Is that your understanding? 

I noticed this ChangeLog entry from Richard in 2004:

	* xdisp.c (BUFFER_POS_REACHED_P): We haven't reached the specified
	position if we're reading from something other than the buffer.

Why?  Maybe that's from an earlier time and only meant for (e.g.) strings.  

If I revert the original patch, and instead just redefine the macro (see alt patch) to read:

...
   && (it->method == GET_FROM_BUFFER                            \
       || it->method == GET_FROM_IMAGE                          \
       || it->method == GET_FROM_STRETCH			\
       || (it->method == GET_FROM_DISPLAY_VECTOR		\
...

this also fixes the original line-skip bug, has all correct behavior including fringe newline overflow, etc., all without any change to the just-fits branch test or any of the return values.  Note that this macro is used 8 times in xdisp.c.  

JD

[inline-image-line-skip-2.patch (application/octet-stream, attachment)]

[inline-image-line-skip-alt.patch (application/octet-stream, attachment)]

[Message part 4 (text/plain, inline)]

Message #104 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Thu, 29 May 2025 10:17:58 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Wed, 28 May 2025 19:42:50 -0400
> Cc: 67604 <at> debbugs.gnu.org
> 
> #define BUFFER_POS_REACHED_P()					\
>   ((op & MOVE_TO_POS) != 0					\
>    && BUFFERP (it->object)					\
>    && (IT_CHARPOS (*it) == to_charpos				\
>        || ((!it->bidi_p						\
> 	    || BIDI_AT_BASE_LEVEL (it->bidi_it))		\
> 	   && IT_CHARPOS (*it) > to_charpos)			\
>        || (it->what == IT_COMPOSITION				\
> 	   && ((IT_CHARPOS (*it) > to_charpos			\
> 		&& to_charpos >= it->cmp_it.charpos)		\
> 	       || (IT_CHARPOS (*it) < to_charpos		\
> 		   && to_charpos <= it->cmp_it.charpos))))	\
>    && (it->method == GET_FROM_BUFFER				\
>        || (it->method == GET_FROM_DISPLAY_VECTOR		\
> 	   && it->dpvec + it->current.dpvec_index + 1 >= it->dpend)))
> 
> When the iterator arrives on the green box, it has reached its destination (to_charpos=89).  And yet, it does not stop.  Moving past the green box is what causes all the problems.  Why does it move on?  Everything here in this macro test is true, _except_ for it->method==GET_FROM_BUFFER and following.  As we're on an image, we have it->method=GET_FROM_IMAGE.  So, it fails.
> 
> Can you never actually reach a TO_CHARPOS position that is on an image (or stretch)?  Are you doomed to move past it?  That I don't understand.

We never return MOVE_POS_MATCH_OR_ZV unless we are iterating on buffer
text, yes.  That's because display properties and other similar
features can "cover" any number of buffer-text characters, and we
cannot know where it ends until we are done moving over it in its
entirety.  Maybe in your case the image only covered one buffer
position, but that is not always the case.  Or maybe you thought that
we return MOVE_POS_MATCH_OR_ZV when we are _at_ the position, whereas
in fact we return that when we got past the position (look at the
conditions in BUFFER_POS_REACHED_P that compare buffer positions).

So in this case, to return MOVE_POS_MATCH_OR_ZV, you need to move past
the image to the newline.  If that's what you did, and it->method was
still GET_FROM_IMAGE, please see why we don't update it->method,
because that is unexpected, I think.

> And yet, I think based on the various comments I've seen in the code about "moving past" images, BUFFER_POS_REACHED_P is perhaps intentionally False for images/stretches.  Is that your understanding? 

Yes, this is by design.  It's why Emacs can never place the cursor on
any character "covered" by the image, only either before or after the
image.

> I noticed this ChangeLog entry from Richard in 2004:
> 
> 	* xdisp.c (BUFFER_POS_REACHED_P): We haven't reached the specified
> 	position if we're reading from something other than the buffer.
> 
> Why?

Because unless we read from the buffer, we don't know what is the
correct buffer position.  I tried to explain that above.  So this
reason is still accurate.

Message #107 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Thu, 29 May 2025 09:56:55 -0400

[Message part 1 (text/plain, inline)]

> On May 29, 2025, at 3:17 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> When the iterator arrives on the green box, it has reached its destination (to_charpos=89).  And yet, it does not stop.  Moving past the green box is what causes all the problems.  Why does it move on?  Everything here in this macro test is true, _except_ for it->method==GET_FROM_BUFFER and following.  As we're on an image, we have it->method=GET_FROM_IMAGE.  So, it fails.
>> 
>> Can you never actually reach a TO_CHARPOS position that is on an image (or stretch)?  Are you doomed to move past it?  That I don't understand.
> 
> We never return MOVE_POS_MATCH_OR_ZV unless we are iterating on buffer
> text, yes.  That's because display properties and other similar
> features can "cover" any number of buffer-text characters, and we
> cannot know where it ends until we are done moving over it in its
> entirety.  

Makes sense (though see below).

> Maybe in your case the image only covered one buffer
> position, but that is not always the case.  

My image does cover 30 or so buffer positions.

> Or maybe you thought that
> we return MOVE_POS_MATCH_OR_ZV when we are _at_ the position, whereas
> in fact we return that when we got past the position (look at the
> conditions in BUFFER_POS_REACHED_P that compare buffer positions).

Ahh yes, I see that, hiding under the bidi_p branch of that test.  So we return when IT >= TO_CHARPOS.  When re-seating "to" the buffer position at the image where you started via start_display, it is too late to return _past_ the image; that's what causes this pixel overflow and line skip.

Question: if you are simply "moving back to the current position", as start_display does (so as to correctly calculate the x pixel position), why does it matter whether the image covers 1 or more buffer positions?  E.g., my green box covers pos=89-125 in my test setup.  If I'm moving "back" to pos=89 from BOL, should reaching the position at the precise start of the image span not constitute BUFFER_POS_REACHED_P?  

In other words, it seems we don't need to know where an image or stretch _ends_ to know that we've reached its _beginning_.  I.e. could that portion of the test be crafted as (see alt2 patch, attached):

   && (it->method == GET_FROM_BUFFER                            \
       || ((it->method == GET_FROM_IMAGE			\
	    || it->method == GET_FROM_STRETCH)			\
	   && IT_CHARPOS (*it) == to_charpos)                   \

??

> So in this case, to return MOVE_POS_MATCH_OR_ZV, you need to move past
> the image to the newline.  If that's what you did, and it->method was
> still GET_FROM_IMAGE, please see why we don't update it->method,
> because that is unexpected, I think.

In the case of a newline inserted right after a "just fitting on the line" green image (testing newline overflow into the fringe), this is indeed the current default scenario: we move past the image to the newline.  But, with my first patch, since it avoids the "just fits" branch, the one whose body starts:

		      ++it->hpos;
		      it->current_x = new_x;

		      /* The character's last glyph just barely fits
			 in this row.  */
		      if (i == it->nglyphs - 1)

we do not get a chance to notice ITERATOR_AT_END_OF_LINE_P and return early with MOVE_POS_MATCH_OR_ZV, instead returning MOVE_LINE_CONTINUED a bit further down, here:

		  move_trace ("move_it_in: continued at %td\n",
			      IT_CHARPOS (*it));
		  result = MOVE_LINE_CONTINUED;
		  break;

This would also be true for a normal (non-WS) just-fitting character followed by a newline.  Again, this could be a distinction without a difference, but I don't know what depends on the precise return value of move_it_in_display_line_to.

[inline-image-line-skip-alt2.patch (application/octet-stream, attachment)]

Message #110 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Thu, 29 May 2025 17:18:12 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Thu, 29 May 2025 09:56:55 -0400
> Cc: 67604 <at> debbugs.gnu.org
> 
> Question: if you are simply "moving back to the current position", as start_display does (so as to correctly calculate the x pixel position), why does it matter whether the image covers 1 or more buffer positions?  E.g., my green box covers pos=89-125 in my test setup.  If I'm moving "back" to pos=89 from BOL, should reaching the position at the precise start of the image span not constitute BUFFER_POS_REACHED_P?  

Because that position's character is not shown.  It's concealed by the
image.

> In other words, it seems we don't need to know where an image or stretch _ends_ to know that we've reached its _beginning_.  I.e. could that portion of the test be crafted as (see alt2 patch, attached):
> 
>    && (it->method == GET_FROM_BUFFER                            \
>        || ((it->method == GET_FROM_IMAGE			\
> 	    || it->method == GET_FROM_STRETCH)			\
> 	   && IT_CHARPOS (*it) == to_charpos)                   \
> 
> ??

No, that's against the contract of the move_it_* functions.  They
return after they reached (or passed) the specified position and
processed the display element at that position, i.e. computed its
metrics.  And you cannot do that if you stop before the image.

> > So in this case, to return MOVE_POS_MATCH_OR_ZV, you need to move past
> > the image to the newline.  If that's what you did, and it->method was
> > still GET_FROM_IMAGE, please see why we don't update it->method,
> > because that is unexpected, I think.
> 
> In the case of a newline inserted right after a "just fitting on the line" green image (testing newline overflow into the fringe), this is indeed the current default scenario: we move past the image to the newline.  But, with my first patch, since it avoids the "just fits" branch, the one whose body starts:
> 
> 		      ++it->hpos;
> 		      it->current_x = new_x;
> 
> 		      /* The character's last glyph just barely fits
> 			 in this row.  */
> 		      if (i == it->nglyphs - 1)
> 
> we do not get a chance to notice ITERATOR_AT_END_OF_LINE_P and return early with MOVE_POS_MATCH_OR_ZV, instead returning MOVE_LINE_CONTINUED a bit further down, here:
> 
> 		  move_trace ("move_it_in: continued at %td\n",
> 			      IT_CHARPOS (*it));
> 		  result = MOVE_LINE_CONTINUED;
> 		  break;
> 
> This would also be true for a normal (non-WS) just-fitting character followed by a newline.  Again, this could be a distinction without a difference, but I don't know what depends on the precise return value of move_it_in_display_line_to.

Sorry, you lost me here.  I asked a question: if the iterator moves to
the newline immediately after the image, does it->method become
GET_FROM_BUFFER?  If yes, why don't we return MOVE_POS_MATCH_OR_ZV?
If it->method is not GET_FROM_BUFFER, why not?

As for the return value of move_it_in_display_line_to, it _is_
important to some of its callers.  So it's important to get it right.

Message #113 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Fri, 30 May 2025 21:37:11 -0400

> On May 29, 2025, at 10:18 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:

> No, that's against the contract of the move_it_* functions.  They
> return after they reached (or passed) the specified position and
> processed the display element at that position, i.e. computed its
> metrics.  And you cannot do that if you stop before the image.

Aha, that's the crux of the matter.  Is this contract spelled out somewhere in a comment?  If not, would be very useful to have.

>>       ++it->hpos;
>>       it->current_x = new_x;
>> 
>>       /* The character's last glyph just barely fits
>>  in this row.  */
>>       if (i == it->nglyphs - 1)
>> 
>> we do not get a chance to notice ITERATOR_AT_END_OF_LINE_P and return early with MOVE_POS_MATCH_OR_ZV, instead returning MOVE_LINE_CONTINUED a bit further down, here:
>> 
>>   move_trace ("move_it_in: continued at %td\n",
>>       IT_CHARPOS (*it));
>>   result = MOVE_LINE_CONTINUED;
>>   break;
>> 
>> This would also be true for a normal (non-WS) just-fitting character followed by a newline.  Again, this could be a distinction without a difference, but I don't know what depends on the precise return value of move_it_in_display_line_to.
> 
> Sorry, you lost me here.  I asked a question: if the iterator moves to
> the newline immediately after the image, does it->method become
> GET_FROM_BUFFER?  

With my initial patch (`&& (it->line_wrap != WORD_WRAP || may_wrap)', the iterator does not move past the green image to the newline.  The reason is that the "just fits" branch is not taken.  It is during this (skipped) branch that we would have moved the iterator forward onto the newline:

 prev_method = it->method;
 if (it->method == GET_FROM_BUFFER)
   prev_pos = IT_CHARPOS (*it);
 set_iterator_to_next (it, true);

Instead, we return with MOVE_LINE_CONTINUED, from the location mentioned above, after the move_trace call (but note: point still does the right thing and the cursor overflows into the fringe).

It is this combination of moving forward past the image, AND not recognizing that the desired position has been reached (so that it cannot be restored via atpos_it) that leads to the line skip bug.

> If yes, why don't we return MOVE_POS_MATCH_OR_ZV?
> If it->method is not GET_FROM_BUFFER, why not?

Because we do not in fact move forward past the GET_FROM_IMAGE position.  Which sounds like a problem v.v. the "contract" you mention.  

> As for the return value of move_it_in_display_line_to, it _is_
> important to some of its callers.  So it's important to get it right.

I figured.  The conundrum seems to be:

1. move_it_in_display_line_to promises to move _past_ an image/stretch if the requested TO_CHARPOS falls on one.
2. Moving past a just-fitting image (which in actuality has always wrapped to the next display line with its trailing space, via WORD_WRAP) leaves us with the wrong idea about the starting x position, leading to vertical motion bugs.

I may need to set this aside for a bit and come back later.  In the meantime, I'd appreciate your thoughts on the following:

If the move_it_in_display_line_to is obligated to iterate past an image (even one at the end of the line that will be wrapped) so as to properly compute its metrics, must IT really be past the image when the function returns, or can the function restore a wrap position left of the image before returning?  Does that count as having satisfied the contract (i.e. do the "metrics" survive wrap_it or atpos_it restoration)?

Thanks

Message #116 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Sat, 31 May 2025 15:18:08 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Fri, 30 May 2025 21:37:11 -0400
> Cc: 67604 <at> debbugs.gnu.org
> 
> 
> 
> > On May 29, 2025, at 10:18 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> > No, that's against the contract of the move_it_* functions.  They
> > return after they reached (or passed) the specified position and
> > processed the display element at that position, i.e. computed its
> > metrics.  And you cannot do that if you stop before the image.
> 
> Aha, that's the crux of the matter.  Is this contract spelled out somewhere in a comment?  If not, would be very useful to have.

Not really spelled out.  It's my conclusion from what is expected.

> > Sorry, you lost me here.  I asked a question: if the iterator moves to
> > the newline immediately after the image, does it->method become
> > GET_FROM_BUFFER?  
> 
> With my initial patch (`&& (it->line_wrap != WORD_WRAP || may_wrap)', the iterator does not move past the green image to the newline.  The reason is that the "just fits" branch is not taken.  It is during this (skipped) branch that we would have moved the iterator forward onto the newline:
> 
>  prev_method = it->method;
>  if (it->method == GET_FROM_BUFFER)
>    prev_pos = IT_CHARPOS (*it);
>  set_iterator_to_next (it, true);
> 
> Instead, we return with MOVE_LINE_CONTINUED, from the location mentioned above, after the move_trace call (but note: point still does the right thing and the cursor overflows into the fringe).
> 
> It is this combination of moving forward past the image, AND not recognizing that the desired position has been reached (so that it cannot be restored via atpos_it) that leads to the line skip bug.
> 
> > If yes, why don't we return MOVE_POS_MATCH_OR_ZV?
> > If it->method is not GET_FROM_BUFFER, why not?
> 
> Because we do not in fact move forward past the GET_FROM_IMAGE position.  Which sounds like a problem v.v. the "contract" you mention.  

So you are saying that the patch you suggested is incorrect, and we
should do something else, and in particular make sure the call to
set_iterator_to_next _is_ made?

> > As for the return value of move_it_in_display_line_to, it _is_
> > important to some of its callers.  So it's important to get it right.
> 
> I figured.  The conundrum seems to be:
> 
> 1. move_it_in_display_line_to promises to move _past_ an image/stretch if the requested TO_CHARPOS falls on one.
> 2. Moving past a just-fitting image (which in actuality has always wrapped to the next display line with its trailing space, via WORD_WRAP) leaves us with the wrong idea about the starting x position, leading to vertical motion bugs.
> 
> I may need to set this aside for a bit and come back later.  In the meantime, I'd appreciate your thoughts on the following:
> 
> If the move_it_in_display_line_to is obligated to iterate past an image (even one at the end of the line that will be wrapped) so as to properly compute its metrics, must IT really be past the image when the function returns, or can the function restore a wrap position left of the image before returning?  Does that count as having satisfied the contract (i.e. do the "metrics" survive wrap_it or atpos_it restoration)?

This is a notoriously dark corner of move_it_* functions under
word-wrap.  I think you will find traces of it in several places if
you search for WORD_WRAP, for example at the beginning of
move_it_in_display_line.  So I think this tricky aspect is expected by
the rest of the code, should not get in the way in this case.

Message #119 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Wed, 4 Jun 2025 16:47:48 -0400

[Message part 1 (text/plain, inline)]

> On May 31, 2025, at 8:18 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> So you are saying that the patch you suggested is incorrect, and we
> should do something else, and in particular make sure the call to
> set_iterator_to_next _is_ made?

Yes, the former patch was too broad and thus somewhat incorrect,
but for different reasons (see below).

I've now very carefully traced through the entire flow for the line-skip
and non-line-skip (missing the bug by one pixel) cases.  

Here is a summary:

1.1 line-skip on wrapped inline images that appear to "just fit" on the prior line
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Bug #67604.  This bug results when an image or stretch /just fits/ as
  the last glyph within the window, and `overflow-newline-into-fringe'
  is enabled.  This causes an "early skip" of the image to the position
  just past it, which results in `move_it_to' incorrectly returning
  early at that position, with `current_x=0'.  This misleads
  `move_it_by_lines', which ends up moving past the "start of the line
  at `VPOS'", and in some situations can move right past the next line.
  This results in a line being "skipped" by `vertical-motion'.


1.1.1 Final assessment
----------------------

  It's pretty head-spinning the number of possible paths through this
  maze.  I'll try to summarize the situation as best I can, based on all
  my investigations so far, here comparing the line-skip and
  no-line-skip paths (which differ by one pixel in the red image).

  What is happening:

  1. `Fvertical_motion' starts by moving back to the beginning of the
     (real) line at `pos=1'.

  2. It then asks `move_it_to' to move back to `pos=89' (the start of
     the green box) using `op=MOVE_TO_POS'.

  3. `move_it_to' calls `move_it_in_display_line_to' with
     `TO_CHARPOS=89', which does one of two things, depending on whether
     the image "just fits" and the fine-tuned overflow branch is taken:

     A) If the image does not "just fit" (even by one pixel, no line
        skip bug):

        1) Instead of moving past the image on the first call,
           `move_it_in_display_line_to' moves `IT' to `pos=89', on the
           first buffer position covered by the image, leaving
           `current_x=451' (the `x' position at the /start/ of the
           image).

        2) `move_it_to' resets `current_x' to `line_start_x=0'
           ("Reset/increment for the next run"), and iterates.

        3) On the next iteration, `move_it_to' is still on the image
           with `method=GET_FROM_IMAGE'.

        4) `move_it_in_display_line_to' is therefore called with
           `pos=89' /again/.

        5) `IT' is /now/ moved to `pos=125', but with the correct
           `current_x=108'.

     B) If the image "just fits" (line skip bug):

        1) On the first call to `move_it_in_display_line_to', `IT' is
           moved past `pos=89' to `pos=125', just /after/ the image,
           leaving `current_x=it->last_visible_x=560'.

        2) `move_it_to' resets `current_x' to `line_start_x=0'
           ("Reset/increment for the next run").

        3) On the next iteration, `move_it_to' notices that `it' is
           already past the `TO_CHARPOS' and is on a `FROM_BUFFER'
           position (`pos=125'), and returns, via
           `skip=MOVE_POS_MATCH_OR_ZV'.

        4) `it' is now at `pos=125' (past the image), but claims
           `current_x=0' (because `move_it_to' explicitly reset
           `current_x' to that before returning).  *THIS IS INCORRECT*.

  4. Summary: depending on which path has been taken above,
     `move_it_to(OP=MOVE_TO_POS)' returns with `it' either holding the
     correct initial `x' position (`current_x=108'), or the incorrect
     position (`it->current_x = 0').

  5. Starting from this position, `Fvertical_motion' calls
     `move_it_by_lines' with `DVPOS=1' to move one line down, as
     requested.

  6. This calls `move_it_to' with `op=MOVE_TO_VPOS'.

  7. `move_it_to' calls `move_it_in_display_line_to(OP=0)' to "stop at
     the start of the line `TO_VPOS'".  Note that the latter mentions:

     ,----
     | /* Regardless of OP's value, stop upon reaching the end of the display line. */
     `----

  8. This tests whether you are still on the line by using the
     accumulated glyph widths.  When the starting `x' position is wrong,
     point moves TOO FAR, past the end of the line and even (in the
     right circumstances) onto the /next/ line.

  9. `move_it_to', and hence `vertical-motion', skips a line.

  Take-aways from this:

  - There is not apparently an absolute contract of all the `move_*'
    functions to move past images.  In the first call to
    `move_it_in_display_line_to' in the bug-free path (starting from
    `pos=1'), the call does /not/ move past the image.  Only on the
    subsequent call targeting the same position (and starting from that
    position) does it move past the image.
  - It's fairly clear what the core of the problem is: a small feature
    designed for `overflow-newline-into-fringe'[1] results in an
    incorrect `current_x' at the starting point of vertical motion when
    word-wrapping, for images or stretches at the end of the line which
    /just fit/ into the window, activating the overflow logic.
  - Less clear is /where to intervene/ to fix the bug, without
    disturbing anything else. Ideally the first call to
    `move_it_in_display_line_to' would not move past, even when the
    glyph just fits.


  [1] If you turn off `overflow-newline-into-fringe', this bug
  disappears.


1.1.2 Solution
--------------

  The solution is to localize as precisely as possible to prevent the
  "early return" of just-fitting glyphs from
  `move_it_in_display_line_to', when:

  1. The last method is `GET_FROM_IMAGE' or `GET_FROM_STRETCH', and
  2. `line_wrap' is `WORD_WRAP'.


> If the patch passes these two tests, feel free to install on the
> master branch.

I do not have commit privileges; whom should I contact?

Final patch attached.  All return values of move_it_in_display_line_to are now 
preserved, except the unwanted "early return" referenced above.  

This patch only affects the rare combination of:

  - moving in a display line to a position at the start of an image or
    stretch
  - which just fits on a line
  - while word-wrapping (so it will always be wrapped)
  - and `overflow-newline-into-fringe=t'

[inline-image-line-skip-innermost.patch (application/octet-stream, attachment)]

Message #122 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Sat, 07 Jun 2025 13:00:04 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Wed, 4 Jun 2025 16:47:48 -0400
> Cc: 67604 <at> debbugs.gnu.org
> 
> > If the patch passes these two tests, feel free to install on the
> > master branch.
> 
> I do not have commit privileges; whom should I contact?
> 
> Final patch attached.  All return values of move_it_in_display_line_to are now 
> preserved, except the unwanted "early return" referenced above.  
> 
> This patch only affects the rare combination of:
> 
>   - moving in a display line to a position at the start of an image or
>     stretch
>   - which just fits on a line
>   - while word-wrapping (so it will always be wrapped)
>   - and `overflow-newline-into-fringe=t'

Thanks, I've now installed this in your name on the master branch.

Is there anything else to do here, or should this bug now be closed?

Message #125 received at 67604 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67604 <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Sat, 7 Jun 2025 08:01:17 -0400

> Thanks, I've now installed this in your name on the master branch.
> 
> Is there anything else to do here, or should this bug now be closed?

Thanks. Nothing else, it can be closed. 

JD

Message #130 received at 67604-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67604-done <at> debbugs.gnu.org
Subject: Re: bug#67604: Motion problems with inline images
Date: Sat, 07 Jun 2025 15:40:42 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Sat, 7 Jun 2025 08:01:17 -0400
> Cc: 67604 <at> debbugs.gnu.org
> 
> 
> > Thanks, I've now installed this in your name on the master branch.
> > 
> > Is there anything else to do here, or should this bug now be closed?
> 
> Thanks. Nothing else, it can be closed. 

Done.  Thanks for all your hard work and for persevering.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.