Subject: bug#65996: closed (Re: bug#65996: 29.1; UCS normalization is wrong)

From: [email-masked] (GNU bug Tracking System)
Date: Sat, 16 Sep 2023 09:23:02 +0000

Your bug report

#65996: 29.1; UCS normalization is wrong

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
65996: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=65996
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#65996: 29.1; UCS normalization is wrong
From: Eli Zaretskii <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Date: Sat, 16 Sep 2023 12:21:42 +0300
List-Id: <debbugs-submit.debbugs.gnu.org>

> From: [email-masked]
> Date: Fri, 15 Sep 2023 21:49:38 +0900
> 
> 
> UCS normalization is wrong for some characters.
> 
> (1) NFD/NFKD decompostion is not done
>     U+1112E 𑄮 CHAKMA VOWEL SIGN O
>     U+1112F 𑄯 CHAKMA VOWEL SIGN AU
>     U+1134B 𑍋 GRANTHA VOWEL SIGN OO
>     U+1134C 𑍌 GRANTHA VOWEL SIGN AU
>     U+114BB 𑒻 TIRHUTA VOWEL SIGN AI
>     U+114BC 𑒼 TIRHUTA VOWEL SIGN O
>     U+114BE 𑒾 TIRHUTA VOWEL SIGN AU
>     U+115BA 𑖺 SIDDHAM VOWEL SIGN O
>     U+115BB 𑖻 SIDDHAM VOWEL SIGN AU
>     U+11938 𑤸 DIVES AKURU VOWEL SIGN O
> 
>     (let ((s "\U0001112E\U0001112F\U0001134B\U0001134C\
>     \U000114BB\U000114BC\U000114BE\U000115BA\U000115BB\U00011938"))
>       (require 'ucs-normalize)
>       (list (equal s (ucs-normalize-NFD-string s))
>             (equal s (ucs-normalize-NFKD-string s))))
>     =>(t t)
> 
> (2) NFKC/NFKD replacement is not done
>     U+1E030..U+1E06D Cyrillic MODIFIER LETTER or SUBSCRIPT
>     U+1EE00..U+1EEBB ARABIC MATHEMATICAL *
>     U+1FBF0..U+1FBF9 SEGMENTED DIGIT *
> 
>     (let* ((f (lambda (cell)
>                 (apply #'string (number-sequence (car cell) (cdr cell)))))
>            (s (mapconcat f '((#x1E030 . #x1E06D)
>                              (#x1EE00 . #x1EEBB)
>                              (#x1FBF0 . #x1FBF9)))))
>       (require 'ucs-normalize)
>       (list (equal s (ucs-normalize-NFKC-string s))
>             (equal s (ucs-normalize-NFKD-string s))))
>     =>(t t)

Thanks, fixed on the emacs-29 branch.

Once again, if (as I'm guessing) you found these problems by examining
the data in ucs-normalize.el, it would have greatly helped if you'd
pointed to the problematic data in your report.  Reverse-engineering
the sources of the problem from the behavior takes time, especially
when the relevant code is not trivial and was written by someone else.

Subject: 29.1; UCS normalization is wrong
From: [email-masked]
To: [email-masked]
Date: Fri, 15 Sep 2023 21:49:38 +0900
List-Id: <debbugs-submit.debbugs.gnu.org>

UCS normalization is wrong for some characters.

(1) NFD/NFKD decompostion is not done
    U+1112E 𑄮 CHAKMA VOWEL SIGN O
    U+1112F 𑄯 CHAKMA VOWEL SIGN AU
    U+1134B 𑍋 GRANTHA VOWEL SIGN OO
    U+1134C 𑍌 GRANTHA VOWEL SIGN AU
    U+114BB 𑒻 TIRHUTA VOWEL SIGN AI
    U+114BC 𑒼 TIRHUTA VOWEL SIGN O
    U+114BE 𑒾 TIRHUTA VOWEL SIGN AU
    U+115BA 𑖺 SIDDHAM VOWEL SIGN O
    U+115BB 𑖻 SIDDHAM VOWEL SIGN AU
    U+11938 𑤸 DIVES AKURU VOWEL SIGN O

    (let ((s "\U0001112E\U0001112F\U0001134B\U0001134C\
    \U000114BB\U000114BC\U000114BE\U000115BA\U000115BB\U00011938"))
      (require 'ucs-normalize)
      (list (equal s (ucs-normalize-NFD-string s))
            (equal s (ucs-normalize-NFKD-string s))))
    =>(t t)

(2) NFKC/NFKD replacement is not done
    U+1E030..U+1E06D Cyrillic MODIFIER LETTER or SUBSCRIPT
    U+1EE00..U+1EEBB ARABIC MATHEMATICAL *
    U+1FBF0..U+1FBF9 SEGMENTED DIGIT *

    (let* ((f (lambda (cell)
                (apply #'string (number-sequence (car cell) (cdr cell)))))
           (s (mapconcat f '((#x1E030 . #x1E06D)
                             (#x1EE00 . #x1EEBB)
                             (#x1FBF0 . #x1FBF9)))))
      (require 'ucs-normalize)
      (list (equal s (ucs-normalize-NFKC-string s))
            (equal s (ucs-normalize-NFKD-string s))))
    =>(t t)

In GNU Emacs 29.1 (build 2, x86_64-w64-mingw32) of 2023-08-02 built on
 AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Pro (v10.0.2009.19045.3448)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation=aot
 --without-compress-install --with-tree-sitter CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

Important settings:
  value of $LANG: JPN
  locale-coding-system: cp932

Major mode: Lisp Interaction

Minor modes in effect:
  highlight-changes-visible-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(misearch multi-isearch comp comp-cstr warnings icons rx emoji-labels
emoji multisession sqlite transient format-spec edmacro kmacro cl-extra
gnutls network-stream nsm mailalias smtpmail textsec uni-scripts url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util url-parse auth-source cl-seq
eieio eieio-core cl-macs json map url-vars idna-mapping ucs-normalize
uni-confusable textsec-check cl-print byte-opt gv bytecomp byte-compile
debug backtrace find-func hilit-chg wid-edit thingatpt help-fns
radix-tree help-mode pp shadow sort mail-extr emacsbug message mailcap
yank-media puny dired dired-loaddefs rfc822 mml mml-sec password-cache
epa derived epg rfc6068 epg-config gnus-util text-property-search
time-date subr-x mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils term/bobcat japan-util
rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 331760 49630)
 (symbols 48 14840 3)
 (strings 32 66748 8954)
 (string-bytes 1 1357518)
 (vectors 16 55924)
 (vector-slots 8 1637738 128446)
 (floats 8 68 385)
 (intervals 56 7100 2925)
 (buffers 984 18))