Subject: bug#22320: Overlays with an 'invisible property break stacking of overlay faces

From: Clément Pit--Claudel <[email-masked]>
Date: Thu, 7 Jan 2016 12:03:38 -0500

Hi Eli,

Thanks a lot for your response.

On 01/07/2016 10:54 AM, Eli Zaretskii wrote:
> It's a feature.  More accurately, it's the best solution found (more
> than 10 years ago!) for a difficult problem.  In a nutshell, in a
> general use case the invisible text can (and many times does) have
> non-trivial face attributes, and using those faces for displaying the
> ellipsis would result in each instance of the ellipsis to have some
> random face.  This "feature" makes that face predictable: when the
> invisible text has a face that is different from the last visible
> character before it, the ellipsis is displayed with the 'default'
> face.
>
> You can read the discussion which led to the current implementation
> here:
> 
>   http://lists.gnu.org/archive/html/emacs-pretest-bug/2005-04/msg00338.html

Interesting, thanks for the pointer. I read the discussion, but I see no rationale there for this choice. Was one ever given? Why is this better than, for example, keeping the face of the first invisible character and extending it to the whole ellipsis?

Am I mistaken to think that this choice is also inconsistent with the way faces on composed characters are handled? It seems that composing a sequence of characters into "..." will keep the face of the first compose character. That default is useful in many places (such as Arthur's nameless mode, or in flyspell or flycheck when an error covers a sequence of characters composed into a single one). Is there a good reason to treat ellipses standing for invisible spans differently?

The interactions of the current behaviour with transient mark mode are especially confusing: consider the two examples below:

(with-current-buffer (get-buffer-create "region covers "def", but ellipsis isn't highlighted")
  (erase-buffer)
  (fundamental-mode)
  (add-to-invisibility-spec '(outline . t))
  (insert "abcdefghi")
  (let ((ov (make-overlay 4 7)))
    (overlay-put ov 'invisible 'outline))
  (let ((ov (make-overlay 6 (point-max))))
    (overlay-put ov 'face 'region))
  (pop-to-buffer (current-buffer)))

(with-current-buffer (get-buffer-create "region now covers one more char, and ellispis is highlighed")
  (erase-buffer)
  (fundamental-mode)
  (add-to-invisibility-spec '(outline . t))
  (insert "abcdefghi")
  (let ((ov (make-overlay 4 7)))
    (overlay-put ov 'invisible 'outline))
  (let ((ov (make-overlay 5 (point-max))))
    (overlay-put ov 'face 'region))
  (pop-to-buffer (current-buffer)))

Using transient-mark-mode and selecting characters one by one from the end of the buffer, "i", "h", and "g" are progressively selected, then nothing happens despite "def" being in fact selected (so pressing C-w at that point kills more than highlighted), then "def" and "c" get highlighted at the same time. This means that the highlighted region doesn't actually correspond to the marked region. Is that also expected? (I realize that this can also happen if the point is in the middle of an invisible region, but that's a lot more of a corner case than using Shift+arrow keys to select a buffer section)

> Note that the display engine only looks at the face of the first
> invisible character; the rest are skipped without looking at their
> faces (or any other text properties or overlays).  So if you change
> your test case in a very minor way:
> 
>   (let ((ov (make-overlay 8 9)))
>     (overlay-put ov 'face '(:underline t)))
> 
> The "problem" doesn't happen.

That's true, but I have no control over this; the only thing that I see is that invisible sections of org Buffers do not have the right background (same for folded theorems in Coq buffers, due to the bug with font fallback and overlays).

>> Here is another (more "real-world") way to reproduce this issue (run the
>> snippet, then use C-x h in the newly created buffer to mark all text: the
>> invisible section isn't highlighted)
>>
>> (with-current-buffer (get-buffer-create "org-bug")
>>   (erase-buffer)
>>   (org-mode)
>>   (insert "* line1\n** line2\n* line3")
>>   (org-shifttab)
>>   (pop-to-buffer (current-buffer)))
> 
> You say that this use case is only broken in Emacs 25, but I see this
> in all the previous versions as well (as expected, given when this was
> coded).  So if you really see this working correctly in Emacs 24.4,
> then some other factors on your system were at work here.  Maybe you
> didn't test this exact test case there?

You're right; this example works fine in 24.3, but it is indeed broken in 24.5. I tested both with -Q:

   Works:  GNU Emacs 24.3.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8) of 2015-06-17 on clem-w50-mint
   Broken: GNU Emacs 24.5.2 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8) of 2015-06-20 on clem-w50-mint

Is this something that changed in org then?

>> Yet another way it to reproduce it is given below
>>
>> (with-current-buffer (get-buffer-create "flyspell-bug")
>>   (erase-buffer)
>>   (text-mode)
>>   (add-to-invisibility-spec '(outline . t))
>>   (insert "line1\nline2\nline3")
>>   (flyspell-check-region-doublons (point-min) (point-max))
>>   (let ((ov (make-overlay 7 12)))
>>     (overlay-put ov 'invisible 'outline))
>>   (pop-to-buffer (current-buffer)))
> 
> What exactly is the problem here?  I don't see anything that looks
> problematic.  In particular, there are no duplicate misspelling in
> this buffer, so flyspell-check-region-doublons should do nothing.
> What am I missing?

Indeed, I use aspell instead of ispell. It seems to ignore numbers, and thus flags the first two instances of "line" as duplicates.

>> Interestingly, the first example is broken in both 24.4 and emacs-25; but the 
>> second one is broken only in emacs-25 (the third one does not run in Emacs 
>> 24.4).
> 
> This behavior should be present in any Emacs released after Apr 2005.

I wonder what makes 24.3 behave differently in Org mode.

>> I wonder if this is connected to another issue that I've seen, where having a
>> composition (using prettify-symbols-mode) just before an invisible region would 
>> cause the background of the invisible region to be incorrect, but setting 
>> prettify-symbols-unprettify-at-point and moving the point to the edge of the 
>> symbol would cause the background to be fine. Unfortunately I don't have a good 
>> repro for that issue.
> 
> Hard to tell without a reproducible test case.

This is in fact due to font fallback. I opened a separate bug report, as it does not seem obviously related to the current issue.

Thanks,
Clément.