GNU bug report logs - #44784
Combining accent is not combining

Previous Next

To reply to this bug, email your comments to 44784 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: bug-gnu-emacs <at> gnu.org
Subject: Combining accent is not combining
Date: Sat, 21 Nov 2020 21:17:09 +0200

[Message part 1 (text/plain, inline)]

Sorry that I noticed this regression only now,
but it's a pretty bad-looking regression:

0. emacs -Q
1. insert to the *scratch* buffer:

ä́  і́  Я́ 

In 26.3 everything is correct, and COMBINING ACUTE ACCENT
is combined above its previous character:

[comb26.png (image/png, inline)]

[Message part 3 (text/plain, inline)]

Composed with the following character(s) "́" using this font:
  xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1071 878 13 0 13 21 0 nil]
  [0 1 769 649 0 5 11 18 -14 [-15 -4 0]]
In 26.3 (x86_64-pc-linux-gnu, GTK+ Version 3.24.14)
Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LIBSYSTEMD LCMS2

But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:

[comb27.png (image/png, inline)]

[Message part 5 (text/plain, inline)]

Composed with the following character(s) "́" using this font:
  xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1071 878 13 0 13 16 0 nil]
  [0 1 769 649 13 4 11 17 -13 [0 0 0]]
In 27.1 (x86_64-pc-linux-gnu, GTK+ Version 3.24.20)
Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY INOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
some characters?  It fails with the scripts above, but still has no problems
with plain Latin characters.

Message #8 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Sat, 21 Nov 2020 22:05:52 +0200

> From: Juri Linkov <juri <at> linkov.net>
> Date: Sat, 21 Nov 2020 21:17:09 +0200
> 
> 0. emacs -Q
> 1. insert to the *scratch* buffer:
> 
> ä́  і́  Я́ 
> 
> In 26.3 everything is correct, and COMBINING ACUTE ACCENT
> is combined above its previous character:
> 
> 
> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:

On my system, this displays the same in both Emacs 26 and 27, the
latter using HarfBuzz.

> It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
> is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
> some characters?  It fails with the scripts above, but still has no problems
> with plain Latin characters.

That cannot be the reason.  Please try with other fonts, for
starters.  DejaVu Sans Mono is known to have problems with combining
accents, see this discussion:

  https://lists.freedesktop.org/archives/harfbuzz/2019-August/007422.html

Message #11 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Sat, 21 Nov 2020 21:08:20 +0100

On Nov 21 2020, Juri Linkov wrote:

> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:

Do they come from the same font?

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #14 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: juri <at> linkov.net
Cc: 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Sat, 21 Nov 2020 22:13:47 +0200

Btw, the report by Emacs 27's "C-u C-x =", viz.:

  Composed with the following character(s) "́" using this font:
    xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
  by these glyphs:
    [0 1 1071 878 13 0 13 16 0 nil]
    [0 1 769 649 13 4 11 17 -13 [0 0 0]]

explicitly says that character composition _did_ happen here, it's
just the display that is broken.  And I guess that moving point to
these characters will show a "fat" cursor that includes both the base
character and the accent, once again showing that Emacs did compose
the accent with the preceding character.

Message #17 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Sat, 21 Nov 2020 22:30:05 +0200

>> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:
>
> Do they come from the same font?

Yes, both use the same font:

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1

The component character(s) are displayed by these fonts (glyph codes):
 ́: ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x289)

Message #20 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Sat, 21 Nov 2020 22:35:11 +0200

> Btw, the report by Emacs 27's "C-u C-x =", viz.:
>
>   Composed with the following character(s) "́" using this font:
>     xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
>   by these glyphs:
>     [0 1 1071 878 13 0 13 16 0 nil]
>     [0 1 769 649 13 4 11 17 -13 [0 0 0]]
>
> explicitly says that character composition _did_ happen here, it's
> just the display that is broken.  And I guess that moving point to
> these characters will show a "fat" cursor that includes both the base
> character and the accent, once again showing that Emacs did compose
> the accent with the preceding character.

Actually, separate cursor is shown on a character and separate on its
combining character, so it seems they are not composed.

>> It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
>> is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
>> some characters?
>
> That cannot be the reason.  Please try with other fonts, for
> starters.  DejaVu Sans Mono is known to have problems with combining
> accents, see this discussion:

Yes, it displays correctly with other fonts, e.g. Noto Sans.
It would be a pity to switch to another font.  So this bug report
could be closed if nothing can be done in Emacs.

Message #23 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Sun, 22 Nov 2020 05:35:30 +0200

> From: Juri Linkov <juri <at> linkov.net>
> Cc: 44784 <at> debbugs.gnu.org
> Date: Sat, 21 Nov 2020 22:35:11 +0200
> 
> > Btw, the report by Emacs 27's "C-u C-x =", viz.:
> >
> >   Composed with the following character(s) "́" using this font:
> >     xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
> >   by these glyphs:
> >     [0 1 1071 878 13 0 13 16 0 nil]
> >     [0 1 769 649 13 4 11 17 -13 [0 0 0]]
> >
> > explicitly says that character composition _did_ happen here, it's
> > just the display that is broken.  And I guess that moving point to
> > these characters will show a "fat" cursor that includes both the base
> > character and the accent, once again showing that Emacs did compose
> > the accent with the preceding character.
> 
> Actually, separate cursor is shown on a character and separate on its
> combining character, so it seems they are not composed.

The 2 vectors you show above, let alone the "composed with" part,
explicitly say they _were_ composed.  So something else is going on
here.

Message #26 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 44784 <at> debbugs.gnu.org, Juri Linkov <juri <at> linkov.net>
Subject: Re: bug#44784: Combining accent is not combining
Date: Sun, 22 Nov 2020 11:12:03 +0100

On Sun, 22 Nov 2020 05:35:30 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Juri Linkov <juri <at> linkov.net>
>> Cc: 44784 <at> debbugs.gnu.org
>> Date: Sat, 21 Nov 2020 22:35:11 +0200
>> 
>> > Btw, the report by Emacs 27's "C-u C-x =", viz.:
>> >
>> >   Composed with the following character(s) "́" using this font:
>> >     xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
>> >   by these glyphs:
>> >     [0 1 1071 878 13 0 13 16 0 nil]
>> >     [0 1 769 649 13 4 11 17 -13 [0 0 0]]
>> >
>> > explicitly says that character composition _did_ happen here, it's
>> > just the display that is broken.  And I guess that moving point to
>> > these characters will show a "fat" cursor that includes both the base
>> > character and the accent, once again showing that Emacs did compose
>> > the accent with the preceding character.
>> 
>> Actually, separate cursor is shown on a character and separate on its
>> combining character, so it seems they are not composed.
>
> The 2 vectors you show above, let alone the "composed with" part,
> explicitly say they _were_ composed.  So something else is going on
> here.

I think it's a display problem with DejaVu Sans Mono, as you already
noted.  I also see what Juri sees, but when the cursor appears to be
over the separately displayed accent, `C-u C-x =' says the character at
point is SPC.

Steve Berman

Message #29 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 44784 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#44784: Combining accent is not combining
Date: Sun, 22 Nov 2020 17:28:08 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: Juri Linkov <juri <at> linkov.net>,  44784 <at> debbugs.gnu.org
> Date: Sun, 22 Nov 2020 11:12:03 +0100
> 
> > The 2 vectors you show above, let alone the "composed with" part,
> > explicitly say they _were_ composed.  So something else is going on
> > here.
> 
> I think it's a display problem with DejaVu Sans Mono, as you already
> noted.  I also see what Juri sees, but when the cursor appears to be
> over the separately displayed accent, `C-u C-x =' says the character at
> point is SPC.

Right.  I think the discussion on the HarfBuzz list to which I pointed
explains what happens: the problematic fonts don't provide the
information necessary to correctly locate the accent over some
character (specifically, precomposed characters that are already
accented).  The [0 0 0] part of the second glyph in the composition
info is a telltale sign of lack of that information: if you try the
same with a font where display is correct, you will see that the first
2 components are non-zero (notably, the first component should be
negative).

Message #32 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 44784 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#44784: Combining accent is not combining
Date: Sun, 22 Nov 2020 17:28:09 +0100

On Sun, 22 Nov 2020 17:28:08 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: Juri Linkov <juri <at> linkov.net>,  44784 <at> debbugs.gnu.org
>> Date: Sun, 22 Nov 2020 11:12:03 +0100
>> 
>> > The 2 vectors you show above, let alone the "composed with" part,
>> > explicitly say they _were_ composed.  So something else is going on
>> > here.
>> 
>> I think it's a display problem with DejaVu Sans Mono, as you already
>> noted.  I also see what Juri sees, but when the cursor appears to be
>> over the separately displayed accent, `C-u C-x =' says the character at
>> point is SPC.
>
> Right.  I think the discussion on the HarfBuzz list to which I pointed
> explains what happens: the problematic fonts don't provide the
> information necessary to correctly locate the accent over some
> character (specifically, precomposed characters that are already
> accented).  The [0 0 0] part of the second glyph in the composition
> info is a telltale sign of lack of that information: if you try the
> same with a font where display is correct, you will see that the first
> 2 components are non-zero (notably, the first component should be
> negative).

FTR, here's what `C-u C-x =' shows for the correct and incorrect
displays, respectively:

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 1110 243 4 1 3 7 0 nil]
  [0 1 769 690 0 -5 -1 11 -7 [1 0 0]]
with these character(s):
  ́ (#x301) COMBINING ACUTE ACCENT

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1110 917 8 1 8 10 0 nil]
  [0 1 769 649 8 3 8 11 -8 [0 0 0]]
with these character(s):
  ́ (#x301) COMBINING ACUTE ACCENT

Steve Berman

Message #35 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 44784 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#44784: Combining accent is not combining
Date: Sun, 22 Nov 2020 18:52:17 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: juri <at> linkov.net,  44784 <at> debbugs.gnu.org
> Date: Sun, 22 Nov 2020 17:28:09 +0100
> 
> FTR, here's what `C-u C-x =' shows for the correct and incorrect
> displays, respectively:
> 
> Composed with the following character(s) "́" using this font:
>   ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
> by these glyphs:
>   [0 1 1110 243 4 1 3 7 0 nil]
>   [0 1 769 690 0 -5 -1 11 -7 [1 0 0]]
> with these character(s):
>   ́ (#x301) COMBINING ACUTE ACCENT
> 
> Composed with the following character(s) "́" using this font:
>   ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 1110 917 8 1 8 10 0 nil]
>   [0 1 769 649 8 3 8 11 -8 [0 0 0]]
> with these character(s):
>   ́ (#x301) COMBINING ACUTE ACCENT

OK, zero WIDTH and negative LBEARING/RBEARING is another way to place
the accent correctly.

Message #40 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Wed, 09 Feb 2022 10:43:27 +0100

[Message part 1 (text/plain, inline)]

Juri Linkov <juri <at> linkov.net> writes:

> Yes, it displays correctly with other fonts, e.g. Noto Sans.
> It would be a pity to switch to another font.  So this bug report
> could be closed if nothing can be done in Emacs.

Skimming this bug report, it's not clear whether this was a problem with
the fonts or in Emacs.  If I do emacs -Q -fn "Noto Sans" with the
current trunk, I get:

[Message part 2 (image/png, inline)]

[Message part 3 (text/plain, inline)]

So the combining accent ̷ is combining with the o correctly here.

So is there anything more to be done here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #47 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Thu, 10 Feb 2022 10:40:55 +0200

[Message part 1 (text/plain, inline)]

>> Yes, it displays correctly with other fonts, e.g. Noto Sans.
>> It would be a pity to switch to another font.  So this bug report
>> could be closed if nothing can be done in Emacs.
>
> Skimming this bug report, it's not clear whether this was a problem with
> the fonts or in Emacs.  If I do emacs -Q -fn "Noto Sans" with the
> current trunk, I get:
>
> So the combining accent ̷ is combining with the o correctly here.
>
> So is there anything more to be done here?

I confirm that "DejaVu Sans Mono" is still broken:

[noncombining.png (image/png, inline)]

[Message part 3 (text/plain, inline)]

whereas other fonts can correctly handle combining characters:

[combining.png (image/png, inline)]

[Message part 5 (text/plain, inline)]

Please close this bug report if it's not possible
to implement some workaround for broken fonts.

Message #50 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: Juri Linkov <juri <at> linkov.net>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 44784 <at> debbugs.gnu.org
Subject: Re: bug#44784: Combining accent is not combining
Date: Thu, 10 Feb 2022 11:49:39 +0100

>>>>> On Thu, 10 Feb 2022 10:40:55 +0200, Juri Linkov <juri <at> linkov.net> said:

    Juri> Please close this bug report if it's not possible
    Juri> to implement some workaround for broken fonts.

Itʼs possible, since we do something similar for certain emoji
sequences. I question whether itʼs worth doing, though, it seems like
very much a rare condition, which is easily fixed by switching fonts.

Robert
-- 

Message #53 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Robert Pluim <rpluim <at> gmail.com>
Cc: 44784 <at> debbugs.gnu.org, Juri Linkov <juri <at> linkov.net>
Subject: Re: bug#44784: Combining accent is not combining
Date: Thu, 10 Feb 2022 12:19:11 +0100

Robert Pluim <rpluim <at> gmail.com> writes:

> Itʼs possible, since we do something similar for certain emoji
> sequences. I question whether itʼs worth doing, though, it seems like
> very much a rare condition, which is easily fixed by switching fonts.

I guess so.  I'm surprised that DejaVu Sans Mono is broken, though, so
perhaps there is something else going on.

This reminds me of a related thing I've wondered about: We don't combine
glyphs from different fonts, because this may look very ugly.  But has
anybody tried doing it anyway?  It may not look worse than what some
people are seeing today...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #56 received at 44784 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: rpluim <at> gmail.com, 44784 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#44784: Combining accent is not combining
Date: Thu, 10 Feb 2022 14:25:07 +0200

> Resent-From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
> Resent-CC: bug-gnu-emacs <at> gnu.org
> Resent-Sender: help-debbugs <at> gnu.org
> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Date: Thu, 10 Feb 2022 12:19:11 +0100
> Cc: 44784 <at> debbugs.gnu.org, Juri Linkov <juri <at> linkov.net>
> 
> Robert Pluim <rpluim <at> gmail.com> writes:
> 
> > Itʼs possible, since we do something similar for certain emoji
> > sequences. I question whether itʼs worth doing, though, it seems like
> > very much a rare condition, which is easily fixed by switching fonts.
> 
> I guess so.  I'm surprised that DejaVu Sans Mono is broken, though, so
> perhaps there is something else going on.
> 
> This reminds me of a related thing I've wondered about: We don't combine
> glyphs from different fonts, because this may look very ugly.  But has
> anybody tried doing it anyway?  It may not look worse than what some
> people are seeing today...

I think it depends on the two fonts and the characters to compose.  It
_could_ be much worse, there's no doubt about that.  Different fonts
use very different styles for the same glyphs.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.