Subject: 26.3; Won't display Big5 characters using Xft
From: Brian Schack <[email-masked]>
To: [email-masked]
Date: Fri, 13 Mar 2020 15:47:57 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

My emacs (which is compiled with X11 and Xft, but not GTK or NextStep)
won't display Big5-encoded characters; the easiest way to see this is:

% emacs -Q
`C-h h'
`C-s big5('

On my system, the glyphs displayed are from the LastResort font.

After digging around, I think I know why it happens, and how to fix it.
I can give details on what's going on, but the fix is simple - change a
line in ftfont.c:

--- ../../ftfont.old.c	2020-03-11 14:57:35.000000000 -0700
+++ ../../ftfont.new.c	2020-03-11 14:57:44.000000000 -0700
@@ -108,7 +108,7 @@
     { "iso8859-15", { 0x00A0, 0x00A1, 0x00D0, 0x0152 }},
     { "iso8859-16", { 0x00A0, 0x0218}},
     { "gb2312.1980-0", { 0x4E13 }, "zh-cn"},
-    { "big5-0", { 0xF6B1 }, "zh-tw" },
+    { "big5-0", { 0x9C21 }, "zh-tw" },
     { "jisx0208.1983-0", { 0x4E55 }, "ja"},
     { "ksc5601.1985-0", { 0xAC00 }, "ko"},
     { "cns11643.1992-1", { 0xFE32 }, "zh-tw"},

Explanation: To check if a fontconfig font can support a character set,
Emacs uses a set of unique characters (all in Unicode), in
fcfont.c:fc_charset_table.  However, the character given for the big5-0
character set (0xF6B1) is *not* Unicode, it is big5.  Changing it to the
equivalent Unicode character (0x9C21) fixes the problem - Xft
successfully finds fonts other than LastResort to display the big5
characters.

Extra note: The file macfont.m has a similar mapping table, called
cf_charset_table.  It has this line for big5:

    { "big5-0", { /* 0xF6B1 in ftfont.c */ 0x4EDC }, CFSTR ("zh-Hant") },

It's clear that the maintainer of this file saw the problem and fixed it
in macfont.m, but didn't fix it in ftfont.c.  I have a version of emacs
compiled with NextStep stuff, and sure enough, it displays big5
characters correctly.

Hope this helps!  If you have any questions, let me know.

Brian

----------------------------------------------------------------------

In GNU Emacs 26.3 (build 1, x86_64-apple-darwin10.8.0, X toolkit, Xaw scroll bars)
 of 2020-02-23 built on Brian-Schacks-Computer.local
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
Recent messages:
nsmenu.m
nsselect.m
nsterm.m
Mark saved where search started
Making completion list...
Mark saved where search started
Mark set [2 times]
Mark saved where search started [2 times]
Finding changes in /Users/schack/Projects/Emacs/emacs-26.3/src/coding.c...done
Finding changes in /Users/schack/Projects/Emacs/emacs-26.3/src/ftfont.c...done

Configured using:
 'configure --with-x-toolkit=athena --without-ns --with-gnutls=no
 --with-xft'

Configured features:
XPM JPEG TIFF GIF PNG NOTIFY ACL LIBXML2 FREETYPE XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM THREADS

Important settings:
  value of $LC_CTYPE: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: C/*l

Minor modes in effect:
  diff-auto-refine-mode: t
  TeX-PDF-mode: t
  shell-dirtrack-mode: t
  desktop-save-mode: t
  show-paren-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow fortune emacsbug sendmail whitespace diff-mode vc face-remap
lisp-mnt url-http url-gw url-cache url-auth bbdb-gui sort gnus-cite
smiley mm-archive gnus-async gnus-bcklg qp gnus-ml disp-table pcmpl-unix
misearch multi-isearch w3m-cookie w3m-form w3m-filter w3m-bookmark
w3m-tabmenu w3m-session ffap w3m w3m-hist w3m-fb bookmark-w3m w3m-ems
w3m-ccl ccl w3m-favicon w3m-image w3m-proc w3m-util grep nndraft nnmh
nnrss xml mm-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util nnfolder utf-7 bbdb-gnus
bbdb-snarf mail-extr bbdb-com cl network-stream nsm starttls gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu
mml2015 mm-view mml-smime smime dig mailcap nntp gnus-cache multishell
multishell-list savehist cl-print debug conf-mode m4-mode cl-extra
help-mode vc-cvs perl-mode vc-dispatcher preview prv-emacs tex-buf
font-latex latex latex-flymake flymake-proc flymake warnings thingatpt
tex-ispell tex-style tex crm tex-mode compile shell sh-script smie
executable org-rmail org-mhe org-irc org-info org-gnus nnir gnus-sum
gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source tls
gnutls utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range message
rmc puny rfc822 mml mml-sec epa epg mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win gnus
nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
mail-utils mm-util mail-prsvr wid-edit org-docview doc-view image-mode
dired dired-loaddefs org-bibtex bibtex org-bbdb org-w3m elec-pair
jka-compr vc-filewise vc-rcs map cc-mode cc-fonts cc-guess cc-menus
cc-cmds finder-inf info tex-site gnuplot-autoloads package epg-config
url-handlers url-parse auth-source eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv org-element cl-seq avl-tree generator org advice
org-macro org-footnote org-pcomplete pcomplete org-list org-faces
org-entities noutline outline org-version ob-emacs-lisp ob ob-tangle
org-src ob-ref ob-lob ob-table ob-keys ob-exp ob-comint ob-core ob-eval
org-compat org-macs org-loaddefs format-spec find-func cal-menu easymenu
calendar cal-loaddefs desktop frameset w3m-load cc-styles cc-align
cc-engine cc-vars cc-defs offlineimap derived edmacro kmacro cl-loaddefs
cl-lib comint ansi-color ring bbdb-sc supercite easy-mmode regi
bbdb-autoloads bbdb timezone paren cus-start cus-load time-date
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads kqueue
dynamic-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 2080534 22721)
 (symbols 48 55526 2)
 (miscs 40 1884 1057)
 (strings 32 383670 8294)
 (string-bytes 1 13035276)
 (vectors 16 76063)
 (vector-slots 8 1577357 247498)
 (floats 8 446 356)
 (intervals 56 91697 663)
 (buffers 992 123))

-- 
NAPOLEON: What shall we do with this soldier, Guiseppe?  Everything he
says is wrong.

GUISEPPE: Make him a general, Excellency, and then everything he says
will be right.

  - G. B. Shaw, "The Man of Destiny"