Subject: bug#31315: wrong font encoding for fallback font

From: Werner LEMBERG <[email-masked]>
Date: Mon, 30 Apr 2018 09:21:06 +0200 (CEST)

The attached image shows that some CJK characters are displayed
incorrectly.  For the used outline font Emacs reports

  xft:-PfEd-AR PL UKai TW MBE-normal-normal-normal-*-18-*-*-*-*-0-iso10646-1

One character (the highlighted one) is missing in this font, and Emacs
uses a different font as a fallback:

  x:-misc-droid sans fallback-medium-r-normal--18-130-100-100-p-179-gb18030.2000-0

Note that the different font backend seems to produce the ugly
rendering; the font in question is the outline font
`DroidSansFallbackFull.ttf'.

The problem now is that the encoding of the fallback font is not
respected.  In the image, the highlighted character is U+83EF, but
Emacs incorrectly displays U+51BF instead.

The GB 18030 bytes to represent U+51BF are \x83\xEF; this clearly
shows that Emacs lacks an iconv call (or an equivalent to that);
instead, it seems to simply feed the Unicode value to the font
backend.

                        *   *   *

It's a completely different question why on my system Emacs uses a
font encoded in GB 18030 as a fallback font.  It's probably related to
the fact that I use `mew' as my e-mail program, manually extended to
cover GB 18030.  Unfortunately, I wasn't able yet to trigger the issue
with `emacs -Q' (which by default uses iso10646 for the fallback
font).  On the other hand, as soon as the problem happens, it happens
with any buffer containing CJK characters not displayable with the
current font, so it seems a genuine Emacs core bug.

In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.20.10)
 of 2018-02-12 built on linux
Repository revision: 3a718ffca097b35218c3e041a94adff937f3052f
Windowing system distributor 'The X.Org Foundation', version 11.0.11803000
System Description: openSUSE Leap 42.3

Recent messages:
Wrote /home/wl/Mail/draft/2
Draft is prepared
Sole completion
next-line: End of buffer [3 times]
Kill draft message? (y or n) y
Saving file /home/wl/Mail/draft/2...
Wrote /home/wl/Mail/draft/2
Draft was killed
Quit
Type C-x 1 to remove help window.  

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
GTK3 X11 THREADS LIBSYSTEMD JSON LCMS2

Important settings:
  value of $LANG: de_AT.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Summary

Minor modes in effect:
  shell-dirtrack-mode: t
  TeX-PDF-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/usr/local/share/emacs/site-lisp/thai-word hides /usr/local/share/emacs/27.0.50/lisp/language/thai-word

Features:
(shadow emacsbug message dired dired-loaddefs format-spec rfc822 mml
mml-sec epa derived epg gnus-util rmail rmail-loaddefs mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mail-utils preview prv-emacs
noutline outline tex-jp wid-edit descr-text apropos font-latex
plain-tex tex-buf latex easy-mmode tex-ispell tex-style tex-mode
compile shell pcomplete comint ansi-color ring latexenc misearch
multi-isearch shr-color color shr svg browse-url network-stream puny
nsm rmc starttls tls gnutls qp pp mew-varsx mew-unix elec-pair edmacro
kmacro rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse
rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode nxml-outln
nxml-rap nxml-util nxml-enc xmltok sgml-mode dom hideshow cal-menu
calendar cal-loaddefs mew-auth mew-config mew-imap2 mew-imap mew-nntp2
mew-nntp mew-pop mew-smtp mew-ssl mew-ssh mew-net mew-highlight
mew-sort mew-fib mew-ext mew-refile mew-demo mew-attach mew-draft
mew-message mew-thread mew-virtual mew-summary4 mew-summary3
mew-summary2 mew-summary mew-search mew-pick mew-passwd mew-scan
mew-syntax mew-bq mew-smime mew-pgp mew-header mew-exec mew-mark
mew-mime mew-edit mew-decode mew-encode mew-cache mew-minibuf
mew-complete mew-addrbook mew-local mew-vars3 mew-vars2 mew-vars
mew-env mew-mule3 mew-mule mew-gemacs mew-key mew-func mew-blvs
mew-const mew tex dbus xml crm advice auto-loads tex-site quail
help-mode cjktilde mm-util mail-prsvr disp-table finder-inf package
easymenu epg-config url-handlers url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs password-cache json map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib time-date
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 454688 29723)
 (symbols 48 49126 5)
 (miscs 40 4726 718)
 (strings 32 150458 3023)
 (string-bytes 1 3707077)
 (vectors 16 46771)
 (vector-slots 8 2048741 170478)
 (floats 8 303 739)
 (intervals 56 66880 624)
 (buffers 992 20)
 (heap 1024 70029 4352))