Subject: [PATCH] Inconsistency in bitmap overlay drawing for macOS

From: Daniel Pettersson <[email-masked]>
Date: Tue, 25 Jun 2024 00:04:27 +0200

On macOS overlay-arrow is not draw on top of bitmaps already present
in the fringe, like its done on X Window System (src/xterm.c).  This is
to the detriment of gdb-mi.el where the overlay-arrow hides the
breakpoint when they exist on the same row.

It can be reproduced with the following:
(require 'gdb-mi)
(defvar overlay-arrow (make-marker))
(add-to-list 'overlay-arrow-variable-list 'overlay-arrow)
(setq fringe-indicator-alist '((overlay-arrow . hollow-right-triangle)))
(overlay-put (make-overlay (pos-bol) (1+ (pos-bol))) 'before-string
             (propertize " " 'display
                         `(left-fringe breakpoint warning)))
(move-marker overlay-arrow (pos-bol))

I expect that macOS is not using overlay_p param as intended as fixed by
attached patch.  But I was having an hard time finding any documentation
proving my point other then the implementation of it's X Window System
sibling in src/xterm.c.

>From 492a26a4296fc8a7558b488796840c22c977d75c Mon Sep 17 00:00:00 2001
From: Daniel Pettersson <[email-masked]>
Date: Mon, 24 Jun 2024 23:16:59 +0200
Subject: [PATCH] Fix inconsistency in bitmap overlay drawing for macOS

* src/nsterm.m (ns_draw_fringe_bitmap): Respect overlay_p by not
clearing fringe if set, as its done in xterm.
---
 src/nsterm.m | 32 ++++++++++++++++++--------------
 1 file changed, 18 insertions(+), 14 deletions(-)

diff --git a/src/nsterm.m b/src/nsterm.m
index 794630de1c1..fc9133071ed 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -2961,24 +2961,28 @@ Hide the window (X11 semantics)
   NSTRACE_MSG ("which:%d cursor:%d overlay:%d width:%d height:%d period:%d",
                p->which, p->cursor_p, p->overlay_p, p->wd, p->h, p->dh);
 
-  /* Work out the rectangle we will need to clear.  */
-  clearRect = NSMakeRect (p->x, p->y, p->wd, p->h);
+  /* Clear screen unless overlay.  */
+  if ( !p->overlay_p )
+    {
+      /* Work out the rectangle we will need to clear.  */
+      clearRect = NSMakeRect (p->x, p->y, p->wd, p->h);
 
-  if (p->bx >= 0 && !p->overlay_p)
-    clearRect = NSUnionRect (clearRect, NSMakeRect (p->bx, p->by, p->nx, p->ny));
+      if ( p->bx >= 0 )
+        clearRect = NSUnionRect (clearRect, NSMakeRect (p->bx, p->by, p->nx, p->ny));
 
-  /* Handle partially visible rows.  */
-  clearRect = NSIntersectionRect (clearRect, rowRect);
+      /* Handle partially visible rows.  */
+      clearRect = NSIntersectionRect (clearRect, rowRect);
 
-  /* The visible portion of imageRect will always be contained within
-     clearRect.  */
-  ns_focus (f, &clearRect, 1);
-  if (! NSIsEmptyRect (clearRect))
-    {
-      NSTRACE_RECT ("clearRect", clearRect);
+      /* The visible portion of imageRect will always be contained
+	 within clearRect.  */
+      ns_focus (f, &clearRect, 1);
+      if ( !NSIsEmptyRect (clearRect) )
+        {
+          NSTRACE_RECT ("clearRect", clearRect);
 
-      [[NSColor colorWithUnsignedLong:face->background] set];
-      NSRectFill (clearRect);
+          [[NSColor colorWithUnsignedLong:face->background] set];
+          NSRectFill (clearRect);
+        }
     }
 
   NSBezierPath *bmp = [fringe_bmp objectForKey:[NSNumber numberWithInt:p->which]];
-- 
2.39.3 (Apple Git-145)

In GNU Emacs 30.0.50 (build 1, aarch64-apple-darwin23.1.0, NS
 appkit-2487.20 Version 14.1.1 (Build 23B81)) of 2024-06-24 built on
 Daniels-Air
Windowing system distributor 'Apple', version 10.3.2487
System Description:  macOS 14.1.1

Configured using:
 'configure --with-xwidgets'

Configured features:
ACL DBUS GLIB GNUTLS LCMS2 LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER PNG
RSVG SQLITE3 THREADS TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM XWIDGETS
ZLIB