GNU bug report logs - #11026
23.3; call-process-region deletes unicode chars

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 11026 in the body.
You can then email your comments to 11026 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Sean McLaughlin <seanmcl <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 23.3; call-process-region deletes unicode chars
Date: Fri, 16 Mar 2012 12:07:55 -0400

This bug report will be sent to the Free Software Foundation,
not to your local site managers!
Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your report will be posted to the bug-gnu-emacs <at> gnu.org mailing list
and the gnu.emacs.bug news group, and at http://debbugs.gnu.org.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug.  If you can, give
a recipe starting from `emacs -Q':

Create a buffer called 'bug' with multi-byte chars.  For example, with
the contents

Alpha = α

Run (call-process-region (point-min) (point-max) "cat" t t) in that
buffer.  It deletes the non-ascii unicode chars and leaves everything
else intact.

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/local/src/homebrew/Cellar/emacs/23.3/share/emacs/23.3/etc/DEBUG.


In GNU Emacs 23.3.1 (x86_64-apple-darwin10.7.0, NS apple-appkit-1038.35)
 of 2011-04-28 on seanmcl.local
Windowing system distributor `Apple', version 10.3.1038
configured using `configure
'--prefix=/usr/local/src/homebrew/Cellar/emacs/23.3' '--without-dbus'
'--enable-locallisppath=/usr/local/src/homebrew/share/emacs/site-lisp'
'--infodir=/usr/local/src/homebrew/Cellar/emacs/23.3/share/info/emacs'
'--with-ns' '--disable-ns-self-contained' 'CC=/usr/bin/cc' 'CFLAGS=-O3
-march=core2 -msse4.1 -w -pipe'
'LDFLAGS=-L/usr/local/src/homebrew/lib' 'CPPFLAGS=-isystem
/usr/local/src/homebrew/include''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: nil
  locale-coding-system: nil
  default enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  shell-dirtrack-mode: t
  show-paren-mode: t
  display-time-mode: t
  iswitchb-mode: t
  nxhtml-global-minor-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  abbrev-mode: t

Recent input:
C-SPC <down> C-w <down> <down> <down> <down> <down>
<down> <down> <down> <down> C-y <tab> C-k <up> <tab>
C-e <backspace> <tab> <down> <tab> C-a C-x C-s <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> C-SPC
<down> <down> C-w <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> C-u C-g C-y C-k <C-up> <C-up> C-SPC <C-down>
C-w <up> <up> <up> <up> <C-down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <up> <tab> <down>
<down> C-a C-o <tab> <tab> <up> <down> <tab> <up> C-e
<backspace> <backspace> <down> <tab> C-9 <up> <up>
<down> <down> <C-down> C-h f d e l e t e - f <tab>
i l e <return> <up> <down> ( d e l e t e - f i l e
SPC f i l e ) C-a C-x C-s <C-down> C-k C-k <left> C-x
C-e C-x C-s <C-tab> C-x b t e s t . <return> <tab>
<C-tab> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <C-left>
<C-left> <right> <right> <right> r C-x C-s <C-tab>
<C-down> <C-tab> <C-down> <left> C-x C-e <up> <up>
<up> <up> C-M-x <C-tab> <tab> <up> <tab> SPC SPC <tab>
C-x C-s <down> v a l SPC x SPC = SPC <return> l e t
SPC i n <return> 4 <return> e n d <tab> C-a C-x C-s
<up> <up> <up> <up> <right> <right> <right> \ a l p
h a <tab> <right> <right> <backspace> C-a C-x C-s <tab>
<tab> <C-down> <C-up> <C-down> <C-tab> <C-up> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <up>
<help-echo> <help-echo> <up> <down> M-x r e p o r SPC
e m SPC b u SPC <return>

Recent messages:
Wrote /Users/seanmcl/save/projects/sml-indent/sml-indent.el
SML\.indent-buffer
SML.indent-buffer
Error in post-command-hook: (error Marker does not point anywhere)
Saving file /Users/seanmcl/save/projects/sml-indent/test.sml...
Wrote /Users/seanmcl/save/projects/sml-indent/test.sml
Saving file /Users/seanmcl/save/projects/sml-indent/test.sml...
Wrote /Users/seanmcl/save/projects/sml-indent/test.sml
Saving file /Users/seanmcl/save/projects/sml-indent/test.sml...
Wrote /Users/seanmcl/save/projects/sml-indent/test.sml

Load-path shadows:
/Users/seanmcl/save/config/emacs/mac/init hides
/Users/seanmcl/save/config/emacs/init
/usr/local/share/emacs/site-lisp/nxhtml/util/htmlfontify hides
/usr/local/src/homebrew/Cellar/emacs/23.3/share/emacs/23.3/lisp/htmlfontify
/usr/local/share/emacs/site-lisp/nxhtml/util/chart hides
/usr/local/src/homebrew/Cellar/emacs/23.3/share/emacs/23.3/lisp/emacs-lisp/chart

Features:
(shadow sort mail-extr message ecomplete rfc822 mml mml-sec
password-cache mm-decode mm-bodies mm-encode mailabbrev nnheader
gnus-util netrc gmm-utils mailheader canlock sha1 hex-util hashcash
mail-utils emacsbug edmacro make-mode image-mode newcomment re-builder
tabify man ffap vc vc-dispatcher align kmacro sh-script rect ansi-color
shell mule-util bs find-dired tar-mode jka-compr executable dabbrev
cc-mode cc-fonts cc-menus cc-styles cc-align thingatpt debug
multi-isearch info cl-specs edebug find-func vc-git disp-table vc-hg
bookmark pp saveplace pc-select paren time cua-base ocaml-lib dired-x
dired-aux uniquify iswitchb whitespace nxhtml-autoload moz cc-cmds
cc-engine cc-vars cc-defs majmodpri nxhtml-menu udev-rinari udev-cedet
udev-ecb udev flymake-js flymake-php flymake css-color nxhtml-mode
html-quote tidy-xhtml ediff-merg ediff-diff ediff-wind ediff-help
ediff-util ediff-mult ediff-init ediff html-imenu imenu popcmp rngalt
rng-nxml loadhist appmenu-fold fold-dwim hideshow noutline outline
mumamo byte-opt warnings sgml-mode rng-valid rng-loc rng-uri rng-parse
nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode
nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc xmltok mlinks rx
appmenu flyspell ispell html-upl html-site url-http tls url url-proxy
url-privacy url-expand url-history mailcap url-auth mail-parse rfc2231
rfc2047 rfc2045 qp ietf-drums time-date url-cookie url-util url-gw
url-methods url-parse url-vars mm-util mail-prsvr ffip ourcomments-util
advice help-fns advice-preload recentf tree-widget ido grep apropos gimp
dired cus-edit cus-start cus-load wid-edit nxhtml-autostart sml-indent
tex-site twelf-font twelf caml tuareg speedbar sb-image ezimage dframe
assoc skeleton smie agda-input quail help-mode view esml-du-mlton bg-job
def-use-mode easy-mmode def-use-data def-use-sym def-use-util sml-mlton
SML sml-proc compile comint ring bytecomp byte-compile esml-mlb-mode
derived regexp-opt esml-util compat cl cl-19 tooltip ediff-hook vc-hooks
lisp-float-type mwheel ns-win easymenu tool-bar dnd fontset image fringe
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mldrag mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev loaddefs button minibuffer faces cus-face files text-properties
overlay md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process ns multi-tty
emacs)

Message #8 received at 11026 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Sean McLaughlin <seanmcl <at> gmail.com>
Cc: 11026 <at> debbugs.gnu.org
Subject: Re: bug#11026: 23.3; call-process-region deletes unicode chars
Date: Fri, 16 Mar 2012 20:57:22 +0200

> From: Sean McLaughlin <seanmcl <at> gmail.com>
> Date: Fri, 16 Mar 2012 12:07:55 -0400
> 
> Create a buffer called 'bug' with multi-byte chars.  For example, with
> the contents
> 
> Alpha = α
> 
> Run (call-process-region (point-min) (point-max) "cat" t t) in that
> buffer.  It deletes the non-ascii unicode chars and leaves everything
> else intact.

This is not a bug, but a cockpit error.

You need to bind coding-process-for-write to something sensible, like
utf-8.  Otherwise, `cat' will get unpredictable sequence of bytes
instead of the Unicode character, and will respond with another
unpredictable sequence.

I think this bug report should be closed.

Message #11 received at 11026 <at> debbugs.gnu.org (full text, mbox):

From: Sean McLaughlin <seanmcl <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11026 <at> debbugs.gnu.org
Subject: Re: bug#11026: 23.3; call-process-region deletes unicode chars
Date: Fri, 16 Mar 2012 15:03:30 -0400

I apologize.  I thought it was a bug because when I write the buffer
to a file and use call-process it works.

Thank you for your time.  I'll try to close the bug.

Sean

On Fri, Mar 16, 2012 at 2:57 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
>> From: Sean McLaughlin <seanmcl <at> gmail.com>
>> Date: Fri, 16 Mar 2012 12:07:55 -0400
>>
>> Create a buffer called 'bug' with multi-byte chars.  For example, with
>> the contents
>>
>> Alpha = α
>>
>> Run (call-process-region (point-min) (point-max) "cat" t t) in that
>> buffer.  It deletes the non-ascii unicode chars and leaves everything
>> else intact.
>
> This is not a bug, but a cockpit error.
>
> You need to bind coding-process-for-write to something sensible, like
> utf-8.  Otherwise, `cat' will get unpredictable sequence of bytes
> instead of the Unicode character, and will respond with another
> unpredictable sequence.
>
> I think this bug report should be closed.

Message #14 received at 11026 <at> debbugs.gnu.org (full text, mbox):

From: Sean McLaughlin <seanmcl <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11026 <at> debbugs.gnu.org
Subject: Re: bug#11026: 23.3; call-process-region deletes unicode chars
Date: Fri, 16 Mar 2012 15:06:14 -0400

For the record, the variable is called coding-system-for-write.

2012/3/16 Sean McLaughlin <seanmcl <at> gmail.com>:
> I apologize.  I thought it was a bug because when I write the buffer
> to a file and use call-process it works.
>
> Thank you for your time.  I'll try to close the bug.
>
> Sean
>
> On Fri, Mar 16, 2012 at 2:57 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
>>> From: Sean McLaughlin <seanmcl <at> gmail.com>
>>> Date: Fri, 16 Mar 2012 12:07:55 -0400
>>>
>>> Create a buffer called 'bug' with multi-byte chars.  For example, with
>>> the contents
>>>
>>> Alpha = α
>>>
>>> Run (call-process-region (point-min) (point-max) "cat" t t) in that
>>> buffer.  It deletes the non-ascii unicode chars and leaves everything
>>> else intact.
>>
>> This is not a bug, but a cockpit error.
>>
>> You need to bind coding-process-for-write to something sensible, like
>> utf-8.  Otherwise, `cat' will get unpredictable sequence of bytes
>> instead of the Unicode character, and will respond with another
>> unpredictable sequence.
>>
>> I think this bug report should be closed.

Message #17 received at 11026 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Sean McLaughlin <seanmcl <at> gmail.com>
Cc: 11026 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#11026: 23.3; call-process-region deletes unicode chars
Date: Sat, 17 Mar 2012 18:19:43 -0400

> For the record, the variable is called coding-system-for-write.

And there's also coding-system-for-read, since your code does a "encode
with coding-system-for-write, pass through cat, then decode with
coding-system-for-read".


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.