Subject: Possible bugs in filepos-to-bufferpos / bufferpos-to-filepos

From: Gregory Heytings <[email-masked]>
Date: Thu, 04 Mar 2021 21:21:24 +0000

(Disclaimer: I have no knowledge whatsoever about the ISO-2022-JP 
encoding, and although this looks like a bug, I'm not sure this is 
actually a bug; I report this at the suggesion of Eli in bug#46859.)

I downloaded the file [1], and converted it to the ISO-2022-JP encoding 
with iconv -t iso-2022-jp one.txt > iso-2022-jp.txt.  The resulting file 
is attached to this bug report.  It ends with two CRLFs, at byte offsets 
2993 and 2995.  However, after emacs -Q iso-2022-jp.txt, with M-: 
(goto-char (filepos-to-bufferpos POS 'exact)) we get:

POS = 2991, 2992: last but one visible character (HIRAGANA LETTER RU)
POS = 2993, 2994: last visible character (IDEOGRAPHIC FULL STOP)
POS = 2995, 2996: first CRLF
POS = 2997: second CRLF
POS = 2998: point-max
POS = 2999: first CRLF
POS = 3000, 3001: second CRLF
POS >= 3002: point-max

I would have expected:

POS = 2989, 2990: last but one visible character (HIRAGANA LETTER RU)
POS = 2991, 2992: last visible character (IDEOGRAPHIC FULL STOP)
POS = 2993, 2994: first CRLF
POS = 2995, 2996: second CRLF
POS >= 2997: point-max

The opposite operation M-: (bufferpos-to-filepos (- (point) POS) 'exact) 
apparently also has bugs; its return values are not coherent with the 
above ones:

POS = 0: 3003
POS = 1: 3001
POS = 2: 2999
POS = 3 (IDEOGRAPHIC FULL STOP): 2997
POS = 4 (HIRAGANA LETTER RU): 2995

I would have expected:

POS = 0: 2997
POS = 1: 2995
POS = 2: 2993
POS = 3 (IDEOGRAPHIC FULL STOP): 2991
POS = 4 (HIRAGANA LETTER RU): 2989

[1] https://darza.com/ecbackend/vendor/symfony/mime/Tests/Fixtures/samples/charsets/iso-2022-jp/one.txt

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.23, cairo version 1.16.0)
  of 2020-11-08, modified by Debian built on x86-ubc-01
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Debian GNU/Linux bullseye/sid

Configured using:
  'configure --build x86_64-linux-gnu --prefix=/usr
  --sharedstatedir=/var/lib --libexecdir=/usr/lib
  --localstatedir=/var/lib --infodir=/usr/share/info
  --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
  --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
  --with-sound=alsa --without-gconf --with-mailutils --build
  x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
  --libexecdir=/usr/lib --localstatedir=/var/lib
  --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
  --with-pop=yes
  --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
  --with-sound=alsa --without-gconf --with-mailutils --with-cairo
  --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
  'CFLAGS=-g -O2
  -fdebug-prefix-map=/build/emacs-6jKC2B/emacs-27.1+1=. -fstack-protector-strong
  -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
  -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD
JSON PDUMPER LCMS2 GMP

Important settings:
   value of $LANG: en_US.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)