Subject: redisplay bugs

From: "Richard M. Stallman" <[email-masked]>
Date: Wed, 15 Oct 2008 12:10:01 -0400

I found this while reviewing old mail.

I think I may have fixed one or two of these bugs, but most of them
probably still exist.  There are four messages included below.

To: [email-masked]
From: Joe Wells <[email-masked]>
Date: Wed, 19 Sep 2007 14:18:07 +0100
Subject: anomalies of overlays and before-string and display properties

There are a number of strange anomalies (some clearly bugs) in the
behavior of overlays and before-string and display properties.

1. Faces on a display property on a before-string behave differently
   depending on where the display property is in the before-string.
   For a display property at the beginning of the before-string, its
   face is used and the before-string's face is ignored (I think this
   is correct).  For a display property not at the beginning of the
   before-string, its face is completely ignored, and the face of the
   before-string is used instead.  This is fairly clearly a bug.

2. When a substring of a before-string beginning at offset O has a
   display property which is a string S, the first O characters of S
   are not displayed.  If S is not at least O+1 characters long, then
   disastrous things start to happen.  This is clearly a bug.

3. The before-string somehow ÂÃ´Ã²Ã¼inheritsÂÃ´Ã²Ã½ face attributes from the face
   of the character that follows the start of the overlay.  (If the
   overlay is of length zero, this character is not even in the
   overlay.  See the recent discussion of how this problem is
   affecting linum.el.)  This means that in practice a before-string
   needs to use completely specified faces, which is obviously a real
   pain.

4. The display property of an overlay somehow ÂÃ´Ã²Ã¼inheritsÂÃ´Ã²Ã½ face
   attributes from the face of the character that follows the start of
   the overlay.  This has similar disadvantages to anomaly #3
   mentioned above.

5. The help-echo and mouse-face properties of an overlay with both a
   before-string and a display property only affect the material
   displayed by the overlay's display property.  One can work around
   the issue by copying these properties to the various display
   properties inside the before-string; however, this workaround is a
   pain and in the case of mouse-face it doesn't get the entire
   overlay highlighted at once but instead only the pieces are
   highlighted one at a time.  It would be nice to be able to set a
   single help-echo or a single mouse-face property that would affect
   the _entire_ overlay.

You can reproduce all of these anomalies with this function:

  (defun illustrate-anomalies ()
    (let ((s #1=#("\\abcd{VWXYZ}"
                  0 5 (face (:foreground "Purple" :background "yellow"))
                  5 12
                  (face
                   (:foreground
                    "DarkOliveGreen"
                    :weight bold :slant oblique :height 0.8))))
          (props
           '(help-echo
             #1#
             ;;face (:slant italic :foreground "red")
             display "]"
             before-string
             #("  "
               0 1
               (display
                #("|["
                  0 2 (;; help-echo #1#
                       face #2=(:strike-through t :background "white")))
                ;; help-echo #1#
                face #3=(:underline t))
               1 2
               (display
                #("||VWXYZ"
                  0 7 (;; help-echo #1#
                       ;; mouse-face (:background "darkseagreen2")
                       face #2#))
                ;; help-echo #1#
                face #3#))
             mouse-face (:background "darkseagreen2")))
          (buf (get-buffer-create "xyzzy")))
      (with-current-buffer buf
        (display-buffer buf)
        (erase-buffer)
        (dolist (o (overlays-in (point-min) (point-max)))
          (delete-overlay o))
        (insert s)
        (let ((o (make-overlay (point-min) (point-max))))
          (while props
            (overlay-put o (car props) (cadr props))
            (setq props (nthcdr 2 props)))))))

This example is derived from some work where I was trying to get
AUCTeX's folding mode to show the fontified and latex-previewed
contents of macro arguments.

I hope this problem report is useful.

Joe

======================================================================
In GNU Emacs 22.1.1 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
 of 2007-06-27 on artemis
Windowing system distributor `The X.Org Foundation', version 11.0.70000000
configured using `configure  '--prefix=/home/jbw/local2' '--enable-debug' '--disable-nls' '--with-x-toolkit=gtk' 'CFLAGS=-O0 -g3 -ggdb''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: jbw
  value of $LANG: nil
  locale-coding-system: utf-8
  default-enable-multibyte-characters: t

Minor modes in effect:
  outline-minor-mode: t
  desktop-save-mode: t
  url-handler-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  size-indication-mode: t
  line-number-mode: t
  transient-mark-mode: t

To: [email-masked]
From: Joe Wells <[email-masked]>
Date: Wed, 19 Sep 2007 14:27:22 +0100
Subject: Re: anomalies of overlays and before-string and display properties

By the way, with one exception, these problems also occur with
after-string properties.

The exception is that the overlay's mouse-face property does correctly
cover both the overlays display and after-string properties.  This is
more evidence that not covering the before-string property is a bug.

Strangely, the after-string also ÂÃ´Ã²Ã¼inheritsÂÃ´Ã²Ã½ face attributes from the
character after the beginning of the overlay, not the character just
before the _end_ of the overlay.  This is more evidence that this is a
bug and not a feature.

Joe

Joe Wells <[email-masked]> writes:

> There are a number of strange anomalies (some clearly bugs) in the
> behavior of overlays and before-string and display properties.
>
> 1. Faces on a display property on a before-string behave differently
>    depending on where the display property is in the before-string.
>    For a display property at the beginning of the before-string, its
>    face is used and the before-string's face is ignored (I think this
>    is correct).  For a display property not at the beginning of the
>    before-string, its face is completely ignored, and the face of the
>    before-string is used instead.  This is fairly clearly a bug.
>
> 2. When a substring of a before-string beginning at offset O has a
>    display property which is a string S, the first O characters of S
>    are not displayed.  If S is not at least O+1 characters long, then
>    disastrous things start to happen.  This is clearly a bug.
>
> 3. The before-string somehow ÂÃ´Ã²Ã¼inheritsÂÃ´Ã²Ã½ face attributes from the face
>    of the character that follows the start of the overlay.  (If the
>    overlay is of length zero, this character is not even in the
>    overlay.  See the recent discussion of how this problem is
>    affecting linum.el.)  This means that in practice a before-string
>    needs to use completely specified faces, which is obviously a real
>    pain.
>
> 4. The display property of an overlay somehow ÂÃ´Ã²Ã¼inheritsÂÃ´Ã²Ã½ face
>    attributes from the face of the character that follows the start of
>    the overlay.  This has similar disadvantages to anomaly #3
>    mentioned above.
>
> 5. The help-echo and mouse-face properties of an overlay with both a
>    before-string and a display property only affect the material
>    displayed by the overlay's display property.  One can work around
>    the issue by copying these properties to the various display
>    properties inside the before-string; however, this workaround is a
>    pain and in the case of mouse-face it doesn't get the entire
>    overlay highlighted at once but instead only the pieces are
>    highlighted one at a time.  It would be nice to be able to set a
>    single help-echo or a single mouse-face property that would affect
>    the _entire_ overlay.
>
> You can reproduce all of these anomalies with this function:
>
>   (defun illustrate-anomalies ()
>     (let ((s #1=#("\\abcd{VWXYZ}"
>                   0 5 (face (:foreground "Purple" :background "yellow"))
>                   5 12
>                   (face
>                    (:foreground
>                     "DarkOliveGreen"
>                     :weight bold :slant oblique :height 0.8))))
>           (props
>            '(help-echo
>              #1#
>              ;;face (:slant italic :foreground "red")
>              display "]"
>              before-string
>              #("  "
>                0 1
>                (display
>                 #("|["
>                   0 2 (;; help-echo #1#
>                        face #2=(:strike-through t :background "white")))
>                 ;; help-echo #1#
>                 face #3=(:underline t))
>                1 2
>                (display
>                 #("||VWXYZ"
>                   0 7 (;; help-echo #1#
>                        ;; mouse-face (:background "darkseagreen2")
>                        face #2#))
>                 ;; help-echo #1#
>                 face #3#))
>              mouse-face (:background "darkseagreen2")))
>           (buf (get-buffer-create "xyzzy")))
>       (with-current-buffer buf
>         (display-buffer buf)
>         (erase-buffer)
>         (dolist (o (overlays-in (point-min) (point-max)))
>           (delete-overlay o))
>         (insert s)
>         (let ((o (make-overlay (point-min) (point-max))))
>           (while props
>             (overlay-put o (car props) (cadr props))
>             (setq props (nthcdr 2 props)))))))
>
> This example is derived from some work where I was trying to get
> AUCTeX's folding mode to show the fontified and latex-previewed
> contents of macro arguments.
>
> I hope this problem report is useful.
>
> Joe
>
> ======================================================================
> In GNU Emacs 22.1.1 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
>  of 2007-06-27 on artemis
> Windowing system distributor `The X.Org Foundation', version 11.0.70000000
> configured using `configure  '--prefix=/home/jbw/local2' '--enable-debug' '--disable-nls' '--with-x-toolkit=gtk' 'CFLAGS=-O0 -g3 -ggdb''
>
> Important settings:
>   value of $LC_ALL: nil
>   value of $LC_COLLATE: nil
>   value of $LC_CTYPE: en_US.UTF-8
>   value of $LC_MESSAGES: nil
>   value of $LC_MONETARY: nil
>   value of $LC_NUMERIC: nil
>   value of $LC_TIME: jbw
>   value of $LANG: nil
>   locale-coding-system: utf-8
>   default-enable-multibyte-characters: t
>
> Minor modes in effect:
>   outline-minor-mode: t
>   desktop-save-mode: t
>   url-handler-mode: t
>   tooltip-mode: t
>   mouse-wheel-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   unify-8859-on-encoding-mode: t
>   utf-translate-cjk-mode: t
>   auto-compression-mode: t
>   temp-buffer-resize-mode: t
>   size-indication-mode: t
>   line-number-mode: t
>   transient-mark-mode: t

To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: Fwd: overlay face property not used for after-string 	property
From: Joe Wells <[email-masked]>
Date: Mon, 29 Oct 2007 09:57:59 +0000

Richard Stallman <[email-masked]> writes:

>     > Can someone write a patch so that the overlay's face property (and
>     > other display-related properties) won't apply to its before-string?
>     >
>     > Joe, what do you think about that as a solution?
>
>     I think that would be an improvement.  I think the best solution would
>     be for the face property of an overlay (and similar properties) to not
>     apply to the before-string, the after-string, _and_ the display
>     property of the overlay.
>
> Can someone implement that?  It should not be fundamentally hard;
> making the properties apply is what is hard.
>
>     Note that there are additional problems.  One example (which I have
>     already reported) is that the face of the character in the buffer
>     after the overlay start location (which is not even in the overlay if
>     the overlay is of length zero) is applied to the before-string and
>     display properties of the overlay.
>
> The fix we are discussing will get rid of this problem, right?

No, the problem with the before-string/display/after-string properties
being affected by a face from the buffer appears to be a completely
unrelated issue which will need to be debugged separately.

> 					Another example (which I think I
>     have not previously reported) is that the face of the character in the
>     buffer _after_ the overlay end location (which is _never_ in the
>     overlay) is applied to the after-string property.
>
> Let's consider that after this fix is done; perhaps this will
> solve that other problem as a byproduct.

I can't see any way that the change proposed here will help with that
problem.

By the way, changing topic to something else I mentioned in my earlier
message:ÃÂÃÂ ÃÂÃÂ Johan BockgÃÂÃÂ¥rd pointed out in another message that the
convenience functions font-lock-prepend-text-property and
font-lock-append-text-property already exist for the purpose of adding
faces to text (strings or buffer contents) that already have face
properties.  However, I have pointed out in a new bug report that
these functions don't work correctly for all possible face property
values (and indeed they sometimes raise errors).

-- 
Joe

To: [email-masked]
From: Joe Wells <[email-masked]>
Date: Tue, 18 Sep 2007 23:20:07 +0100
Subject: redisplay goes horribly wrong when a before-string contains
	multiple display properties

Redisplay goes horribly wrong when a before-string contains multiple
display properties.

Reproduce by evaluating this expression:

  (let ((buf (get-buffer-create "foo")))
    (with-current-buffer buf
      (display-buffer buf)
      (erase-buffer)
      (dolist (o (overlays-in (point-min) (point-max)))
        (delete-overlay o))
      (insert "ABC")
      (let ((o (make-overlay 2 3))
            (s (make-string 5 ?X)))
        (put-text-property 0 1 'display "1" s)
        (put-text-property 1 2 'display "2" s)
        (put-text-property 2 3 'display "3" s)
        (put-text-property 3 4 'display "4" s)
        (put-text-property 4 5 'display "5" s)
        (overlay-put o 'display "Y")
        (overlay-put o 'before-string s))))

The correct behavior would be to see the buffer ?foo? pop up
displaying the characters ?A12345YC?.  The horribly wrong behavior
that actually happens is that redisplay of the buffer stops completely
after ?A1? is shown.  Strangely, the ?XXXXX? (which should not be
displayed anywhere) will be displayed in the echo area.  Scary errors
start occurring if one tries to do work in the buffer after this.

By the way, it would be really nice if this worked, because it would
be an easy way to display a number of display specifications for a
single overlay, which would make it much easier to get the features of
AUCTeX's tex-fold.el to work together with the features of
latex-preview.  (What is needed is being able to display strings
interleaved with images.)

Joe

======================================================================
In GNU Emacs 22.1.1 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
 of 2007-06-27 on artemis
Windowing system distributor `The X.Org Foundation', version 11.0.70000000
configured using `configure  '--prefix=/home/jbw/local2' '--enable-debug' '--disable-nls' '--with-x-toolkit=gtk' 'CFLAGS=-O0 -g3 -ggdb''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: jbw
  value of $LANG: nil
  locale-coding-system: utf-8
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  TeX-source-specials-mode: t
  auto-fill-function: do-auto-fill
  shell-dirtrack-mode: t
  outline-minor-mode: t
  desktop-save-mode: t
  url-handler-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  size-indication-mode: t
  line-number-mode: t
  transient-mark-mode: t