GNU bug report logs - #7626
23.2.91; Rmail shows incorrect message encoding in the mode line

unarchive 7626

From: Eli Zaretskii <eliz <at> gnu.org>
To: bug-gnu-emacs <at> gnu.org
Subject: 23.2.91; Rmail shows incorrect message encoding in the mode line
Date: Mon, 13 Dec 2010 00:21:12 +0200

The new rmail-mime feature causes Rmail to set
buffer-file-coding-system to an incorrect value.  Expected result:
buffer-file-coding-system is according to the charset= header, but in
fact buffer-file-coding-system is almost always set to undecided.

This is a regression from Emacs 23.2.90.

Also, once the headers were decoded by rfc2047-decode-region, it would
be good to set buffer-file-coding-system to last-coding-system-used,
if the buffer's encoding is still undecided.

In GNU Emacs 23.2.91.1 (i386-mingw-nt5.1.2600)
 of 2010-12-11 on HOME-C4E4A596F7
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  value of $XMODIFIERS: nil
  locale-coding-system: cp1255
  default enable-multibyte-characters: t

Major mode: RMAIL

Minor modes in effect:
  diff-auto-refine-mode: t
  desktop-save-mode: t
  show-paren-mode: t
  display-time-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  line-number-mode: t

Recent input:
C-x b I N B <tab> <return> <C-next> <C-next> <C-next> 
<C-next> <C-next> <C-next> <C-next> <C-next> <C-next> 
<M-end> <help-echo> M-x r e p o r t <tab> <return>

Recent messages:
setting up indent stuff
Indentation variables are now local.
Indentation setup for shell type sh
Setting up indent for shell type sh
setting up indent stuff
Indentation variables are now local.
Indentation setup for shell type sh
Note: file is write protected [3 times]
Wrote d:/usr/eli/.emacs.desktop.lock
Desktop: 234 buffers restored.

Load-path shadows:
None found.

Features:
(shadow mailalias sendmail emacsbug ld-script sh-script executable
dired-x dired-aux dired tcl comint ring make-mode nxml-uchnm rng-xsd
xsd-regexp rng-cmpct rng-nxml rng-valid rng-loc rng-uri rng-parse
nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode
nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc xmltok sgml-mode
arc-mode archive-mode parse-time conf-mode newcomment jka-compr
tar-mode add-log diff-mode texinfo generic vc-cvs cc-mode cc-fonts
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
regexp-opt flyspell ispell org-wl org-w3m org-vm org-rmail org-mhe
org-mew org-irc org-jsinfo org-infojs org-html org-exp org-exp-blocks
org-agenda org-info org-gnus org-bibtex org-bbdb org byte-opt warnings
bytecomp byte-compile advice help-fns advice-preload org-footnote
org-src org-list org-faces org-compat org-macs noutline outline
easy-mmode vc-bzr info rmailsum rmailmm message ecomplete rfc822 mml
easymenu mml-sec password-cache mm-decode mm-bodies mm-encode mailcap
mailabbrev nnheader gnus-util netrc gmm-utils wid-edit mailheader
canlock sha1 hex-util hashcash mail-parse rfc2231 rmail rfc2047
rfc2045 ietf-drums time-date qp mm-util mail-prsvr mail-utils desktop
server filecache saveplace generic-x paren battery time tooltip
ediff-hook vc-hooks lisp-float-type mwheel dos-w32 disp-table ls-lisp
w32-win w32-vars tool-bar dnd fontset image fringe lisp-mode register
page menu-bar rfn-eshadow timer select scroll-bar mldrag mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
loaddefs button minibuffer faces cus-face files text-properties
overlay md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process multi-tty
emacs)

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Mon, 13 Dec 2010 09:29:01 +0900

In article <83ei9miqef.fsf <at> gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

> The new rmail-mime feature causes Rmail to set
> buffer-file-coding-system to an incorrect value.  Expected result:
> buffer-file-coding-system is according to the charset= header, but in
> fact buffer-file-coding-system is almost always set to undecided.

> This is a regression from Emacs 23.2.90.

> Also, once the headers were decoded by rfc2047-decode-region, it would
> be good to set buffer-file-coding-system to last-coding-system-used,
> if the buffer's encoding is still undecided.

I'm now working on various bug-fixing and improvement of
rmail's mime handling, and fixing the above is included in
the work.  The problem is that which coding system to set if
the header and each text part of multipart have different
"charset".  At the moment, I'm going to commit a code that
set the FIRST coding system used for decoding some part
(header, parts of multipart, forwarded message, etc).

---
Kenichi Handa
handa <at> m17n.org

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Mon, 13 Dec 2010 05:47:07 +0200

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: 7626 <at> debbugs.gnu.org
> Date: Mon, 13 Dec 2010 09:29:01 +0900
> 
> I'm now working on various bug-fixing and improvement of
> rmail's mime handling, and fixing the above is included in
> the work.

Thank you.

> The problem is that which coding system to set if the header and
> each text part of multipart have different "charset".  At the
> moment, I'm going to commit a code that set the FIRST coding system
> used for decoding some part (header, parts of multipart, forwarded
> message, etc).

That might be a good logic.  The important thing here is to DTRT as
much as possible wrt replying to the message: rmail-reply sets the
buffer for composing the response to be encoded in the same encoding
as the original message.  So whatever you do in the Rmail buffer when
you display the message should yield an encoding most probably
suitable for the reply.

TIA

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Sat, 01 Jan 2011 13:20:38 +0200

The new Rmail/Rmail-MIME code (revision 100323 on the emacs-23 branch)
solved most of the problems originally reported here, but it still has
some issues.  Here are the problems I see with the new code, based on
less than an hour of testing:

 . If the message has an explicit specification of its charset, the
   EOL type of the RMAIL buffer is left unspecified (displayed as `:'
   in the mode line).  By contrast, when there's no charset= spec, the
   EOL type is correctly set to -unix.

 . Sometimes, the RMAIL buffer's encoding is set to windows-1252,
   which is not present anywhere in the message headers.  The only
   source of non-ASCII characters in the 2 messages where I saw this
   problem was in the RFC 2047 encoded addresses, but those used
   8859-1, as in "Jan =?iso-8859-1?Q?Dj=E4rv?=", not windows-1252.
   `describe-text-properties' indeed shows the charset of the decoded
   address as windows-1252, so the reason is probably somewhere in
   `rfc2047-decode-region'.

Thanks.

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7651: 23.2.91;
	Rmail doesn't allow displaying text attachments conveniently
Date: Wed, 12 Jan 2011 12:57:50 +0200

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: 7651 <at> debbugs.gnu.org
> Date: Wed, 12 Jan 2011 15:22:14 +0900
> 
> By the way, I can't reproduce the incorrect setting of
> buffer-file-coding-system.

You mean, the one with cp1252 I reported in bug#7626?  Or the one
with EOL being undecided?  I guess the former.

> Could you "recend" the
> problematic mail, and let me know the subject line in
> another mail so that I can identify that problematic mail.

I attach one such message below (indented 2 columns).  This is an even
worse offender: the message headers clearly say that it's in
ISO-8859-1, but I still get cp1252 in the Rmail buffer.  I don't know
where that value comes from.  FWIW, my default-buffer-file-coding-system
is iso-latin-1-dos and w32-system-coding-system is cp1255.

  From emacs-devel-bounces+eliz=gnu.org <at> gnu.org Mon Jan 03 16:34:24 2011
  Return-path: <emacs-devel-bounces+eliz=gnu.org <at> gnu.org>
  Envelope-to: eliz <at> gnu.org
  Delivery-date: Mon, 03 Jan 2011 16:34:24 -0500
  Received: from eggs.gnu.org ([140.186.70.92]:60304)
	  by fencepost.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32)
	  (Exim 4.69)
	  (envelope-from <emacs-devel-bounces+eliz=gnu.org <at> gnu.org>)
	  id 1PZs2q-000351-I2
	  for eliz <at> gnu.org; Mon, 03 Jan 2011 16:34:24 -0500
  Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	  (envelope-from <emacs-devel-bounces+eliz=gnu.org <at> gnu.org>)
	  id 1PZs2s-0002FG-RK
	  for eliz <at> gnu.org; Mon, 03 Jan 2011 16:34:27 -0500
  X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org
  X-Spam-Level: 
  X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE,
	  RFC_ABUSE_POST autolearn=unavailable version=3.3.1
  Received: from lists.gnu.org ([199.232.76.165]:51247)
	  by eggs.gnu.org with esmtp (Exim 4.71)
	  (envelope-from <emacs-devel-bounces+eliz=gnu.org <at> gnu.org>)
	  id 1PZs2s-0002FC-P5
	  for eliz <at> gnu.org; Mon, 03 Jan 2011 16:34:26 -0500
  Received: from localhost ([127.0.0.1]:58728 helo=lists.gnu.org)
	  by lists.gnu.org with esmtp (Exim 4.43)
	  id 1PZs2s-0007Ku-JW
	  for eliz <at> gnu.org; Mon, 03 Jan 2011 16:34:26 -0500
  Received: from [140.186.70.92] (port=32772 helo=eggs.gnu.org)
	  by lists.gnu.org with esmtp (Exim 4.43) id 1PZs2W-0007Kf-G9
	  for emacs-devel <at> gnu.org; Mon, 03 Jan 2011 16:34:05 -0500
  Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	  (envelope-from <jan.h.d <at> swipnet.se>) id 1PZs2V-0002An-Eb
	  for emacs-devel <at> gnu.org; Mon, 03 Jan 2011 16:34:04 -0500
  Received: from smtprelay-h22.telenor.se ([195.54.99.197]:44511)
	  by eggs.gnu.org with esmtp (Exim 4.71)
	  (envelope-from <jan.h.d <at> swipnet.se>)
	  id 1PZs2T-0002AG-TX; Mon, 03 Jan 2011 16:34:02 -0500
  Received: from ipb1.telenor.se (ipb1.telenor.se [195.54.127.164])
	  by smtprelay-h22.telenor.se (Postfix) with ESMTP id BDDEDEAD9B;
	  Mon,  3 Jan 2011 22:34:00 +0100 (CET)
  X-SENDER-IP: [85.225.45.100]
  X-LISTENER: [smtp.bredband.net]
  X-IronPort-Anti-Spam-Filtered: true
  X-IronPort-Anti-Spam-Result: AmIxAGjPIU1V4S1kPGdsb2JhbACIRJtwDAEBAQE1L7wMhUoEjiE
  X-IronPort-AV: E=Sophos;i="4.60,268,1291590000"; d="scan'208";a="163718935"
  Received: from c-642de155.25-1-64736c10.cust.bredbandsbolaget.se (HELO
	  coolsville.localdomain) ([85.225.45.100])
	  by ipb1.telenor.se with ESMTP; 03 Jan 2011 22:34:00 +0100
  Received: from [172.20.199.13] (zeplin [172.20.199.13])
	  by coolsville.localdomain (Postfix) with ESMTPSA id E262C7FA05A;
	  Mon,  3 Jan 2011 22:33:59 +0100 (CET)
  Message-ID: <4D2240C8.2060602 <at> swipnet.se>
  Date: Mon, 03 Jan 2011 22:34:00 +0100
  From: =?ISO-8859-1?Q?Jan_Dj=E4rv?= <jan.h.d <at> swipnet.se>
  User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; sv-SE;
	  rv:1.9.2.13) Gecko/20101129 Thunderbird/3.1.7
  MIME-Version: 1.0
  To: Chong Yidong <cyd <at> stupidchicken.com>
  References: <AANLkTin-OjWynp7GOEPnSTKxf2PXH1+t6LWbmQBwCCFD <at> mail.gmail.com>	<4D1B27AF.7010701 <at> swipnet.se>
	  <jwvpqskio2r.fsf-monnier+emacs <at> gnu.org>	<AANLkTi=Jh7pr9_w-9LfJhKBdfDwDuLYnRk5Ke6KyRRy4 <at> mail.gmail.com>	<4D1C6E7D.2040300 <at> swipnet.se>	<AANLkTim_habwF6AN=ncPjUey0umF22TZC_X_RJO2i845 <at> mail.gmail.com>	<4D1D0172.8080404 <at> swipnet.se>	<AANLkTimSLno7xk5xuM+zMM9dujyZoeofoC7vRKrpiQfp <at> mail.gmail.com>	<4D1DB555.5080002 <at> swipnet.se>
	  <4D1DBD4A.6010303 <at> swipnet.se>	<83d3oiaysj.fsf <at> gnu.org>
	  <4D1DD655.1040809 <at> swipnet.se>	<83aajmaxme.fsf <at> gnu.org>
	  <jwv62u9hohk.fsf-monnier+emacs <at> gnu.org>	<AANLkTi=JN3AjvyBcAC5uTi4pnyARnUkOO8U95vDiyWvY <at> mail.gmail.com>	<4D1E5987.2000502 <at> swipnet.se>
	  <jwvoc81fssl.fsf-monnier+emacs <at> gnu.org>	<83oc80q1k4.fsf <at> gnu.org>	<AANLkTinKjo8tDKMovGcpuXoHDKqJ_3ik1+fGjGua+bzN <at> mail.gmail.com>	<83bp40pr7m.fsf <at> gnu.org>
	  <4D1F73B1.3010701 <at> swipnet.se> <87r5cvz9ms.fsf <at> stupidchicken.com>
  In-Reply-To: <87r5cvz9ms.fsf <at> stupidchicken.com>
  Content-Type: text/plain; charset=ISO-8859-1; format=flowed
  Content-Transfer-Encoding: 7bit
  X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	  recognized.
  Cc: u.s.reddy <at> cs.bham.ac.uk, emacs user <user.emacs <at> gmail.com>,
	  emacs-devel <at> gnu.org, monnier <at> iro.umontreal.ca,
	  7517 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
  Subject: Re: bug#7517: 24.0.50; repeated crash under Mac OS X
  X-BeenThere: emacs-devel <at> gnu.org
  X-Mailman-Version: 2.1.5
  Precedence: list
  List-Id: "Emacs development discussions." <emacs-devel.gnu.org>
  List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/emacs-devel>,
	  <mailto:emacs-devel-request <at> gnu.org?subject=unsubscribe>
  List-Archive: <http://lists.gnu.org/archive/html/emacs-devel>
  List-Post: <mailto:emacs-devel <at> gnu.org>
  List-Help: <mailto:emacs-devel-request <at> gnu.org?subject=help>
  List-Subscribe: <http://lists.gnu.org/mailman/listinfo/emacs-devel>,
	  <mailto:emacs-devel-request <at> gnu.org?subject=subscribe>
  X-Mailman-Copy: yes
  Sender: emacs-devel-bounces+eliz=gnu.org <at> gnu.org
  Errors-To: emacs-devel-bounces+eliz=gnu.org <at> gnu.org
  X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
  X-RMAIL-ATTRIBUTES: -D------

  Done.

	  Jan D.

  Chong Yidong skrev 2011-01-02 15.02:
  > "Jan D."<jan.h.d <at> swipnet.se>  writes:
  >
  >> Eli Zaretskii skrev 2011-01-01 16:40:
  >>
  >>> right?
  >>>
  >>> I guess this bug report can be closed now.  Any objections?
  >>
  >> No.
  >
  > Please backport the fix to the emacs-23 branch, thanks.

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7651: 23.2.91;
	Rmail doesn't allow displaying text attachments conveniently
Date: Thu, 13 Jan 2011 10:21:33 +0900

In article <83ipxuie35.fsf <at> gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

> You mean, the one with cp1252 I reported in bug#7626?  Or the one
> with EOL being undecided?  I guess the former.

Yes.

> I attach one such message below (indented 2 columns).  This is an even
> worse offender: the message headers clearly say that it's in
> ISO-8859-1, but I still get cp1252 in the Rmail buffer.  I don't know
> where that value comes from.  FWIW, my default-buffer-file-coding-system
> is iso-latin-1-dos and w32-system-coding-system is cp1255.

I found that rfc2047-decode-region (called while preparing
the message header) sets last-coding-system-used to
windows-1252.  That is because mm-util.el defines
mm-charset-override-alist as this:

  '((gb2312 . gbk)
    (iso-8859-1 . windows-1252)
    (iso-8859-8 . windows-1255)
    (iso-8859-9 . windows-1254))

And explains it as this:

i.e. treat iso-8859-1 as windows-1252.  windows-1252 is a
superset of iso-8859-1."

At the moment, I don't know how (or where) to fix this
problem, but at least, it seems that setting
buffer-file-coding-system to windows-1252 for iso-8859-1
message won't cause an actual problem.

---
Kenichi Handa
handa <at> m17n.org

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7651: 23.2.91;
	Rmail doesn't allow displaying text attachments conveniently
Date: Thu, 13 Jan 2011 07:12:33 -0500

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: 7626 <at> debbugs.gnu.org
> Date: Thu, 13 Jan 2011 10:21:33 +0900
> 
> I found that rfc2047-decode-region (called while preparing
> the message header) sets last-coding-system-used to
> windows-1252.  That is because mm-util.el defines
> mm-charset-override-alist as this:
> 
>   '((gb2312 . gbk)
>     (iso-8859-1 . windows-1252)
>     (iso-8859-8 . windows-1255)
>     (iso-8859-9 . windows-1254))
> 
> And explains it as this:
> 
> i.e. treat iso-8859-1 as windows-1252.  windows-1252 is a
> superset of iso-8859-1."

I guess this is because some broken mailers state Latin-N while
actually using the corresponding Windows codepage, and when the
message includes characters not in Latin-N, they are not displayed
correctly.

But even if we keep this "feature" (see below for an alternative
suggestion), I don't see a need to force this on users.  We should
have a user option to do this.  Personally, I think it should be off
by default, but if someone feels strongly about that, I won't mind
having it on by default, in which case I will customize it on my
system.

> At the moment, I don't know how (or where) to fix this
> problem, but at least, it seems that setting
> buffer-file-coding-system to windows-1252 for iso-8859-1
> message won't cause an actual problem.

It's surprising and kludgey, IMO.  A better solution would be to
decode by windows-12XX, but set last-coding-system-used to Latin-N.
That way, only if I reuse the text that is not encodable by Latin-N, I
will need to do something.  Otherwise, I get to have the cake and eat
it, too.

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7651: 23.2.91;
	Rmail doesn't allow displaying text attachments conveniently
Date: Fri, 14 Jan 2011 13:23:11 +0900

In article <E1PdM2b-00011L-Cr <at> fencepost.gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

> > At the moment, I don't know how (or where) to fix this
> > problem, but at least, it seems that setting
> > buffer-file-coding-system to windows-1252 for iso-8859-1
> > message won't cause an actual problem.

> It's surprising and kludgey, IMO.  A better solution would be to
> decode by windows-12XX, but set last-coding-system-used to Latin-N.
> That way, only if I reuse the text that is not encodable by Latin-N, I
> will need to do something.  Otherwise, I get to have the cake and eat
> it, too.

I don't want to modify rfc2047.el nor mm-util.el now.  So,
I've just installed a little bit inefficient workaround
which does this:

After decoding each header, if rmail-mime-coding-system is
nil, set it to a cons (CODING-SYSTEM . nil).

After decoding each body, if rmail-mime-coding-system is nil
or a cons, set it to CODING-SYSTEM.

After decoding a whole message, if rmail-mime-coding-system
is a cons (i.e. only a header part is decoded), re-decode
the header while binding mm-charset-override-alist to nil,
and set rmail-mime-coding-system to last-coding-system-used.

Set buffer-file-coding-system to rmail-mime-coding-system.

By this, encoding specified for a body always takes
precedence which I think is the right thing.

---
Kenichi Handa
handa <at> m17n.org

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Fri, 14 Jan 2011 13:28:10 +0900

In article <83tyhsq395.fsf <at> gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

>  . If the message has an explicit specification of its charset, the
>    EOL type of the RMAIL buffer is left unspecified (displayed as `:'
>    in the mode line).  By contrast, when there's no charset= spec, the
>    EOL type is correctly set to -unix.

I think the buffer-file-coding-system of rmail buffer should
leave the EOL type unspecified in any cases.  So I added
this code:

	  (set-buffer-file-coding-system
	   (coding-system-base rmail-mime-coding-system) t t))

---
Kenichi Handa
handa <at> m17n.org

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Fri, 14 Jan 2011 12:22:30 -0500

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: 7626 <at> debbugs.gnu.org
> Date: Fri, 14 Jan 2011 13:28:10 +0900
> 
> In article <83tyhsq395.fsf <at> gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >  . If the message has an explicit specification of its charset, the
> >    EOL type of the RMAIL buffer is left unspecified (displayed as `:'
> >    in the mode line).  By contrast, when there's no charset= spec, the
> >    EOL type is correctly set to -unix.
> 
> I think the buffer-file-coding-system of rmail buffer should
> leave the EOL type unspecified in any cases.

Why would this be useful?  In any case, this is a change in behavior
from previous Rmail versions.

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Mon, 17 Jan 2011 10:22:35 +0900

In article <E1PdnM6-00077h-27 <at> fencepost.gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:
> > I think the buffer-file-coding-system of rmail buffer should
> > leave the EOL type unspecified in any cases.

> Why would this be useful?  In any case, this is a change in behavior
> from previous Rmail versions.

Rmail-mime does not perform file I/O, but decodes a text
that is already stored in a mbox buffer.  So, we don't have
any information about by what kind of EOL format the decoded
text should be encoded out to a file.  If we leave the EOL
type unspecified, it will be encoded by whatever EOL format
defined for the current system.

---
Kenichi Handa
handa <at> m17n.org

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Mon, 17 Jan 2011 06:06:20 -0500

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: 7626 <at> debbugs.gnu.org
> Date: Mon, 17 Jan 2011 10:22:35 +0900
> 
> Rmail-mime does not perform file I/O, but decodes a text
> that is already stored in a mbox buffer.

But the mbox format uses Unix EOLs.  In fact, Rmail will barf if you
say "C-u g FILE RET", and FILE has DOS EOLs.  So the EOL format is
known in advance in this case.

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Mon, 17 Jan 2011 20:35:38 +0900

In article <E1Pemui-00053A-Iv <at> fencepost.gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

> > Rmail-mime does not perform file I/O, but decodes a text
> > that is already stored in a mbox buffer.

> But the mbox format uses Unix EOLs.

On Windows too?  I didn't know that.

> In fact, Rmail will barf if you
> say "C-u g FILE RET", and FILE has DOS EOLs.  So the EOL format is
> known in advance in this case.

Are you arguing that typing 'o' (rmail-output) will write
DOS EOL file when EOL type of buffer-file-coding-system of
rmail buffer is undecided and the system_eol_type is DOS?
Then, isn't it a bug of rmail-output?

The reason I decided to leave EOL type undecided is for the
case of M-x write-region on rmail buffer.  In that case, I
think, following system_eol_type is the right thing.

---
Kenichi Handa
handa <at> m17n.org

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Mon, 17 Jan 2011 07:24:12 -0500

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: 7626 <at> debbugs.gnu.org
> Date: Mon, 17 Jan 2011 20:35:38 +0900
> 
> In article <E1Pemui-00053A-Iv <at> fencepost.gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > But the mbox format uses Unix EOLs.
> 
> On Windows too?  I didn't know that.

It must, because Rmail reads the mbox file or stream with
no-conversion.

> > In fact, Rmail will barf if you
> > say "C-u g FILE RET", and FILE has DOS EOLs.  So the EOL format is
> > known in advance in this case.
> 
> Are you arguing that typing 'o' (rmail-output) will write
> DOS EOL file when EOL type of buffer-file-coding-system of
> rmail buffer is undecided and the system_eol_type is DOS?
> Then, isn't it a bug of rmail-output?

That's a separate issue.  I just don't like seeing a buffer whose file
was read with no-conversion show ":" as the EOL mnemonic.  It's not a
catastrophe, I just don't see why we should change this aspect of
Rmail which is how it behaved for several Emacs releases.

> The reason I decided to leave EOL type undecided is for the
> case of M-x write-region on rmail buffer.  In that case, I
> think, following system_eol_type is the right thing.

But we don't behave like that with buffers that visit files, do we?
Why is this use-case different?

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: bug#7626: mm-charset-override-alist (was: bug#7626: bug#7651: 23.2.91;
	Rmail doesn't allow displaying text attachments conveniently)
Date: Tue, 18 Jan 2011 09:07:30 +0100

On Fri, Jan 14 2011, Kenichi Handa wrote:

> Eli Zaretskii <eliz <at> gnu.org> writes:
>> I guess this is because some broken mailers state Latin-N while
>> actually using the corresponding Windows codepage, and when the
>> message includes characters not in Latin-N, they are not displayed
>> correctly.

Exactly.

>> But even if we keep this "feature" (see below for an alternative
>> suggestion), I don't see a need to force this on users.  We should
>> have a user option to do this.  Personally, I think it should be off
>> by default, but if someone feels strongly about that, I won't mind
>> having it on by default, in which case I will customize it on my
>> system.

I think `mm-charset-override-alist' should be on by default.  Before I
added (back in 2005) we frequently had complains about "why doesn't
Gnus display this article correctly?".  I don't recall any real
problems with this.

>> > At the moment, I don't know how (or where) to fix this
>> > problem, but at least, it seems that setting
>> > buffer-file-coding-system to windows-1252 for iso-8859-1
>> > message won't cause an actual problem.
>
>> It's surprising and kludgey, IMO.  A better solution would be to
>> decode by windows-12XX, but set last-coding-system-used to Latin-N.
>> That way, only if I reuse the text that is not encodable by Latin-N, I
>> will need to do something.  Otherwise, I get to have the cake and eat
>> it, too.
>
> I don't want to modify rfc2047.el nor mm-util.el now.  

Thank you.

(I don't have an opinion about what Rmail should do.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Tue, 18 Jan 2011 20:44:21 +0900

In article <E1Peo84-0006Nu-O4 <at> fencepost.gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

> That's a separate issue.  I just don't like seeing a buffer whose file
> was read with no-conversion show ":" as the EOL mnemonic.

After the change of rmail to mbox-base, the buffer whose
file was read with no-conversion is " *message-viewer
RMAIL*", and the buffer "RMAIL" is what showing the decoding
result of a part of " *message-viewer RMAIL*".  Even if you
see "-1:" or "-1(unix)" in the modeline, it doesn't mean the
mbox file is read with iso-8859-1.

So, I don't understand why you so much care EOL part even if
you don't care that the text-encoding part is shown as "1"
(meaning iso-8859-1) instead of "=" (meaning no-conversion).

> It's not a
> catastrophe, I just don't see why we should change this aspect of
> Rmail which is how it behaved for several Emacs releases.

I haven't realized that the change results in the actual
difference on screen on Windows.  On GNU/Linux, both *-unix
and undecided are indicated by ":".

Anyway, if Windows users are already familiar with seeing
"(unix)" in the modeline of RMAIL buffer, don't want to
change it, and want to make M-x write-region to write out a
region of RMAIL buffer with Unix-style EOL as before, I'll
change the current behavior.

> > The reason I decided to leave EOL type undecided is for the
> > case of M-x write-region on rmail buffer.  In that case, I
> > think, following system_eol_type is the right thing.

> But we don't behave like that with buffers that visit files, do we?

Of course no, because in that case, it's the right thing to
follow the file's EOL format.  But, when you open a new
buffer, write a few lines of text, and save some region into
a file, the EOL type of the saved file is CRLF on Windows.

> Why is this use-case different?

As I wrote above, RMAIL buffer's buffer-file-coding-system
doesn't reflect how the corresponding mbox file is read.

---
Kenichi Handa
handa <at> m17n.org

From: Eli Zaretskii <eliz <at> gnu.org>
To: Reiner Steib <Reiner.Steib <at> gmx.de>
Cc: 7626 <at> debbugs.gnu.org, handa <at> m17n.org
Subject: Re: bug#7626: mm-charset-override-alist (was: bug#7626: bug#7651:
	23.2.91; Rmail doesn't allow displaying text attachments conveniently)
Date: Tue, 18 Jan 2011 10:10:31 -0500

> From: Reiner Steib <reinersteib+gmane <at> imap.cc>
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 7626 <at> debbugs.gnu.org
> Reply-To: Reiner Steib <Reiner.Steib <at> gmx.de>
> Date: Tue, 18 Jan 2011 09:07:30 +0100
> 
> I think `mm-charset-override-alist' should be on by default.  Before I
> added (back in 2005) we frequently had complains about "why doesn't
> Gnus display this article correctly?".  I don't recall any real
> problems with this.

2005 was 6 years ago.  Perhaps things changed since then.

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Tue, 18 Jan 2011 10:23:31 -0500

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: 7626 <at> debbugs.gnu.org
> Date: Tue, 18 Jan 2011 20:44:21 +0900
> 
> I haven't realized that the change results in the actual
> difference on screen on Windows.  On GNU/Linux, both *-unix
> and undecided are indicated by ":".
> 
> Anyway, if Windows users are already familiar with seeing
> "(unix)" in the modeline of RMAIL buffer, don't want to
> change it, and want to make M-x write-region to write out a
> region of RMAIL buffer with Unix-style EOL as before, I'll
> change the current behavior.

I don't know about others.  I'm familiar (and I customized the Unix
EOL mnemonic to just "/").

It's up to you.  You can close the bug if you want.

Thanks for working on this.

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org, Reiner Steib <Reiner.Steib <at> gmx.de>
Subject: Re: bug#7626: mm-charset-override-alist
Date: Tue, 18 Jan 2011 13:00:59 -0500

>> I think `mm-charset-override-alist' should be on by default.  Before I
>> added (back in 2005) we frequently had complains about "why doesn't
>> Gnus display this article correctly?".  I don't recall any real
>> problems with this.

> 2005 was 6 years ago.  Perhaps things changed since then.

AFAIK, none of the root causes for these problems have changed.

        Stefan

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 7626 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#7626: mm-charset-override-alist
Date: Wed, 19 Jan 2011 22:23:55 +0100

On Tue, Jan 18 2011, Stefan Monnier wrote:

>>> I think `mm-charset-override-alist' should be on by default.  Before I
>>> added (back in 2005) we frequently had complains about "why doesn't
>>> Gnus display this article correctly?".  I don't recall any real
>>> problems with this.
>
>> 2005 was 6 years ago.  Perhaps things changed since then.
>
> AFAIK, none of the root causes for these problems have changed.

I agree.  I found around 40 articles declared as iso-8859-* which
contain #x80 (the position of the EUR sign € in windows-1252) in my
local leafnode news spool which contains mostly technical usenet
groups (comp.*, de.comp.*, de.comm.*, ...) and mailing list from
gmane.org.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

From: Eli Zaretskii <eliz <at> gnu.org>
To: Reiner Steib <Reiner.Steib <at> gmx.de>
Cc: 7626 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca
Subject: Re: bug#7626: mm-charset-override-alist
Date: Wed, 19 Jan 2011 19:40:08 -0500

> From: Reiner Steib <reinersteib+gmane <at> imap.cc>
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 7626 <at> debbugs.gnu.org
> Reply-To: Reiner Steib <Reiner.Steib <at> gmx.de>
> Date: Wed, 19 Jan 2011 22:23:55 +0100
> 
> I found around 40 articles declared as iso-8859-* which
> contain #x80 (the position of the EUR sign € in windows-1252) in my
> local leafnode news spool which contains mostly technical usenet
> groups (comp.*, de.comp.*, de.comm.*, ...) and mailing list from
> gmane.org.

40 out of how many?

Anyway, Rmail is about reading mail, not about reading news groups.
rmail-reply uses the encoding of the message replied to for setting
the encoding of the reply.  This "feature" makes me reply with a
windows-12xx encoding to someone whose sole "fault" is that she has
Latin-X characters in her name, which is RFC-2047 encoded in the From:
header.  This is ugly and unnecessary.  Emacs should not do this
unless the problematic characters are really present in the message.

I guess I will simply customize away mm-charset-override-alist, if no
one shares these concerns.

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org, Reiner Steib <Reiner.Steib <at> gmx.de>
Subject: Re: bug#7626: mm-charset-override-alist
Date: Wed, 19 Jan 2011 20:57:05 -0500

> Anyway, Rmail is about reading mail, not about reading news groups.
> rmail-reply uses the encoding of the message replied to for setting
> the encoding of the reply.

Sounds like a problem in rmail-reply, then.  E.g. what happens if the
reply includes chars that aren't covered by that coding-system?

        Stefan

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org, Reiner.Steib <at> gmx.de
Subject: Re: bug#7626: mm-charset-override-alist
Date: Thu, 20 Jan 2011 11:05:42 +0900

In article <E1PfiZM-0005hs-Sq <at> fencepost.gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

> I guess I will simply customize away mm-charset-override-alist, if no
> one shares these concerns.

I think you don't need that with the latest code.

I wrote:

> I don't want to modify rfc2047.el nor mm-util.el now.  So,
> I've just installed a little bit inefficient workaround
> which does this:

> After decoding each header, if rmail-mime-coding-system is
> nil, set it to a cons (CODING-SYSTEM . nil).

> After decoding each body, if rmail-mime-coding-system is nil
> or a cons, set it to CODING-SYSTEM.

> After decoding a whole message, if rmail-mime-coding-system
> is a cons (i.e. only a header part is decoded), re-decode
> the header while binding mm-charset-override-alist to nil,
> and set rmail-mime-coding-system to last-coding-system-used.

> Set buffer-file-coding-system to rmail-mime-coding-system.

Have you tried it?

---
Kenichi Handa
handa <at> m17n.org

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 7626 <at> debbugs.gnu.org, Reiner.Steib <at> gmx.de
Subject: Re: bug#7626: mm-charset-override-alist
Date: Wed, 19 Jan 2011 21:08:49 -0500

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: Reiner Steib <Reiner.Steib <at> gmx.de>,  7626 <at> debbugs.gnu.org
> Date: Wed, 19 Jan 2011 20:57:05 -0500
> 
> > rmail-reply uses the encoding of the message replied to for setting
> > the encoding of the reply.
> 
> Sounds like a problem in rmail-reply, then.

If you have a better alternative for guessing the suitable encoding of
the reply, please suggest it.  For me, it works in 99% of cases.

> E.g. what happens if the reply includes chars that aren't covered by
> that coding-system?

As usual in Emacs: when you hit "C-c C-s" or "C-c C-c", Emacs asks for
a suitable encoding, showing the unencodable characters in a pop-up
buffer.  The message is sent using the encoding you selected once you
select it (and Emacs verified that it can encode every character).

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org, Reiner.Steib <at> gmx.de
Subject: Re: bug#7626: mm-charset-override-alist
Date: Wed, 19 Jan 2011 21:12:44 -0500

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: Reiner.Steib <at> gmx.de, 7626 <at> debbugs.gnu.org
> Date: Thu, 20 Jan 2011 11:05:42 +0900
> 
> > After decoding each header, if rmail-mime-coding-system is
> > nil, set it to a cons (CODING-SYSTEM . nil).
> 
> > After decoding each body, if rmail-mime-coding-system is nil
> > or a cons, set it to CODING-SYSTEM.

What is CODING-SYSTEM here?

> > After decoding a whole message, if rmail-mime-coding-system
> > is a cons (i.e. only a header part is decoded), re-decode
> > the header while binding mm-charset-override-alist to nil,
> > and set rmail-mime-coding-system to last-coding-system-used.
> 
> > Set buffer-file-coding-system to rmail-mime-coding-system.
> 
> Have you tried it?

Not yet.  I'm traveling, and I'm using the 23.2.91 pretest at the
moment, where the above doesn't exist.  I will try this as soon as I
can.  Thanks.

From: Eli Zaretskii <eliz <at> gnu.org>
To: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: mm-charset-override-alist
Date: Wed, 19 Jan 2011 21:27:18 -0500

> From: Eli Zaretskii <eliz <at> gnu.org>
> Date: Wed, 19 Jan 2011 19:40:08 -0500
> Cc: 7626 <at> debbugs.gnu.org
> Reply-To: Eli Zaretskii <eliz <at> gnu.org>
> 
> rmail-reply uses the encoding of the message replied to for setting
> the encoding of the reply.

Sorry, that was inaccurate and misleading.  It's not rmail-reply, it's
mail-yank-original who does that, and it only does it if the encoding
of the reply buffer is the default buffer-file-coding-system (i.e. was
not yet set in any non-trivial way).  Here's the relevant part of
mail-yank-original:

	    ;; If they yank the original text, the encoding of the
	    ;; original message is a better default than
	    ;; the default buffer-file-coding-system.
	    (and (coding-system-equal
		  (default-value 'buffer-file-coding-system)
		  buffer-file-coding-system)
		 (setq buffer-file-coding-system
		       (coding-system-change-text-conversion
			buffer-file-coding-system
			(coding-system-base
			 (with-current-buffer original
			   buffer-file-coding-system))))))

The logic behind this is that if your reply cites the original, it had
better start with the encoding of that original.

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org, Reiner.Steib <at> gmx.de
Subject: Re: bug#7626: mm-charset-override-alist
Date: Thu, 20 Jan 2011 12:51:59 +0900

In article <E1Pfk0y-0003Pf-Hi <at> fencepost.gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

> > > After decoding each header, if rmail-mime-coding-system is
> > > nil, set it to a cons (CODING-SYSTEM . nil).
> > 
> > > After decoding each body, if rmail-mime-coding-system is nil
> > > or a cons, set it to CODING-SYSTEM.

> What is CODING-SYSTEM here?

For a header part, last-coding-system-used set by
rfc2047-decode-region.  For a body part, a coding system
implied by MIME charset, or if it's not valid, whatever
Emacs detected.

---
Kenichi Handa
handa <at> m17n.org

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: 23.2.91;
	Rmail shows incorrect message encoding in the mode line
Date: Thu, 20 Jan 2011 12:53:20 +0900

In article <E1PfDP9-0003ub-E7 <at> fencepost.gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:
> > Anyway, if Windows users are already familiar with seeing
> > "(unix)" in the modeline of RMAIL buffer, don't want to
> > change it, and want to make M-x write-region to write out a
> > region of RMAIL buffer with Unix-style EOL as before, I'll
> > change the current behavior.

> I don't know about others.  I'm familiar (and I customized the Unix
> EOL mnemonic to just "/").

> It's up to you.  You can close the bug if you want.

Then please just advice me which is better for general
Windows users; write-region on RMAIL buffer writes out a
file of DOS-style EOL or Unix-style EOL.  If it's the
former, I'll keep the current behavior.  Otherwise I'll
change it back.

---
Kenichi Handa
handa <at> m17n.org

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org, Reiner.Steib <at> gmx.de
Subject: Re: bug#7626: mm-charset-override-alist
Date: Thu, 20 Jan 2011 10:32:37 -0500

>> > rmail-reply uses the encoding of the message replied to for setting
>> > the encoding of the reply.
>> Sounds like a problem in rmail-reply, then.
> If you have a better alternative for guessing the suitable encoding of
> the reply, please suggest it.  For me, it works in 99% of cases.

Using message.el ;-)

>> E.g. what happens if the reply includes chars that aren't covered by
>> that coding-system?
> As usual in Emacs: when you hit "C-c C-s" or "C-c C-c", Emacs asks for
> a suitable encoding, showing the unencodable characters in a pop-up
> buffer.  The message is sent using the encoding you selected once you
> select it (and Emacs verified that it can encode every character).

I guess that's workable.

        Stefan

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 7626 <at> debbugs.gnu.org, Reiner.Steib <at> gmx.de
Subject: Re: bug#7626: mm-charset-override-alist
Date: Thu, 20 Jan 2011 10:37:47 -0500

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: Reiner.Steib <at> gmx.de,  7626 <at> debbugs.gnu.org
> Date: Thu, 20 Jan 2011 10:32:37 -0500
> 
> > If you have a better alternative for guessing the suitable encoding of
> > the reply, please suggest it.  For me, it works in 99% of cases.
> 
> Using message.el ;-)

If someone could explain its logic in setting the encoding of a reply,
I might consider that.  I tried once to understand what it does, but
quickly got lost in a maze of twisty little passages all alike.  When
I insert the citation from the original message, the buffer's encoding
is invariably UTF-8, which is not TRT from my POV.  Maybe this gets
fixed before sending, but I got lost there.

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: mm-charset-override-alist
Date: Sat, 05 Feb 2011 15:36:49 +0200

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: Reiner.Steib <at> gmx.de, 7626 <at> debbugs.gnu.org
> Date: Thu, 20 Jan 2011 11:05:42 +0900
> 
> > I don't want to modify rfc2047.el nor mm-util.el now.  So,
> > I've just installed a little bit inefficient workaround
> > which does this:
> 
> > After decoding each header, if rmail-mime-coding-system is
> > nil, set it to a cons (CODING-SYSTEM . nil).
> 
> > After decoding each body, if rmail-mime-coding-system is nil
> > or a cons, set it to CODING-SYSTEM.
> 
> > After decoding a whole message, if rmail-mime-coding-system
> > is a cons (i.e. only a header part is decoded), re-decode
> > the header while binding mm-charset-override-alist to nil,
> > and set rmail-mime-coding-system to last-coding-system-used.
> 
> > Set buffer-file-coding-system to rmail-mime-coding-system.
> 
> Have you tried it?

I'm using it now, since it's part of the Emacs 23.2.93 pretest.

In general, it seems to work well, but I found one message where it
didn't DTRT.  This message:

  http://lists.gnu.org/archive/html/savannah-users/2011-02/msg00003.html

is shown with buffer-file-coding-system `us-ascii', whereas I would
expect it to be UTF-8, because the From: header is in UTF-8.

(I saved the raw message as it was found in my mailbox, in case you'd
need it for analysis.)

From: Kenichi Handa <handa <at> m17n.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: mm-charset-override-alist
Date: Mon, 07 Feb 2011 09:45:33 +0900

In article <8339o2a9hq.fsf <at> gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:

> In general, it seems to work well, but I found one message where it
> didn't DTRT.  This message:

>   http://lists.gnu.org/archive/html/savannah-users/2011-02/msg00003.html

> is shown with buffer-file-coding-system `us-ascii', whereas I would
> expect it to be UTF-8, because the From: header is in UTF-8.

> (I saved the raw message as it was found in my mailbox, in case you'd
> need it for analysis.)

Thank you.  Please send it it me.

---
Kenichi Handa
handa <at> m17n.org

From: Kenichi Handa <handa <at> m17n.org>
To: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: mm-charset-override-alist
Date: Mon, 07 Feb 2011 14:36:16 +0900

In article <8339o0ts6k.fsf <at> gnu.org>, Eli Zaretskii <eliz <at> gnu.org> writes:
> > >   http://lists.gnu.org/archive/html/savannah-users/2011-02/msg00003.html
> > 
> > > is shown with buffer-file-coding-system `us-ascii', whereas I would
> > > expect it to be UTF-8, because the From: header is in UTF-8.
> > 
> > > (I saved the raw message as it was found in my mailbox, in case you'd
> > > need it for analysis.)
> > 
> > Thank you.  Please send it it me.

> Attached.

Thank you.  In that mail, the first part of body explicitly
specifies this:

Content-Type: text/plain; charset=US-ASCII

So, according to this algorithm:

> > After decoding each header, if rmail-mime-coding-system is
> > nil, set it to a cons (CODING-SYSTEM . nil).
> 
> > After decoding each body, if rmail-mime-coding-system is nil
> > or a cons, set it to CODING-SYSTEM.
> 
> > After decoding a whole message, if rmail-mime-coding-system
> > is a cons (i.e. only a header part is decoded), re-decode
> > the header while binding mm-charset-override-alist to nil,
> > and set rmail-mime-coding-system to last-coding-system-used.
> 
> > Set buffer-file-coding-system to rmail-mime-coding-system.

the coding system is decided as us-ascii (the second step
above overrides a coding system from header).

---
Kenichi Handa
handa <at> m17n.org

From: Eli Zaretskii <eliz <at> gnu.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 7626 <at> debbugs.gnu.org
Subject: Re: bug#7626: mm-charset-override-alist
Date: Mon, 07 Feb 2011 01:07:34 -0500

> From: Kenichi Handa <handa <at> m17n.org>
> Cc: 7626 <at> debbugs.gnu.org
> Date: Mon, 07 Feb 2011 09:45:33 +0900
> 
> >   http://lists.gnu.org/archive/html/savannah-users/2011-02/msg00003.html
> 
> > is shown with buffer-file-coding-system `us-ascii', whereas I would
> > expect it to be UTF-8, because the From: header is in UTF-8.
> 
> > (I saved the raw message as it was found in my mailbox, in case you'd
> > need it for analysis.)
> 
> Thank you.  Please send it it me.

Sent via private email.

From: Glenn Morris <rgm <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 7626 <at> debbugs.gnu.org, Kenichi Handa <handa <at> m17n.org>
Subject: Re: bug#7626: mm-charset-override-alist
Date: Thu, 06 Oct 2011 20:32:11 -0400

IIUC there were a bunch of rmail changes since this. Is this report
still relevant?

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 7626-done <at> debbugs.gnu.org, handa <at> m17n.org
Subject: Re: bug#7626: mm-charset-override-alist
Date: Fri, 07 Oct 2011 14:14:26 +0200

> From: Glenn Morris <rgm <at> gnu.org>
> Cc: Kenichi Handa <handa <at> m17n.org>,  7626 <at> debbugs.gnu.org
> Date: Thu, 06 Oct 2011 20:32:11 -0400
> 
> 
> IIUC there were a bunch of rmail changes since this. Is this report
> still relevant?

Somewhere along the way, in this message:

  http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7626#56

I said that the bug could be closed.  I hope that expresses my answer
clearly enough ;-)

So I'm closing it now.

GNU bug report logs - #7626 23.2.91; Rmail shows incorrect message encoding in the mode line

GNU bug report logs - #7626
23.2.91; Rmail shows incorrect message encoding in the mode line