GNU bug report logs - #1174
23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 1174 in the body.
You can then email your comments to 1174 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: emacs-pretest-bug <at> gnu.org
Subject: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Wed, 15 Oct 2008 22:21:21 +0200

Gnus in Emacs 23 displays some German umlauts incorrectly while Gnus
in Emacs 22 displays the same message correctly. It only happens when
my boss uses thunderbird to reply to my UTF-8 mails from Gnus.

In Emacs 23 in the mail from my boss I get
Char: � (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4
and the character is displayed as \374
In an other UTF-8 mail written by me it is
Char: ü (252, #o374, #xfc, file ...) point=493 of 940 (52%) column=0


In Emacs 22 when I do C-x = on the same character in the mail from my
boss as above I get
Char: ü (2300, #o4374, #x8fc, file ...) point=212 of 461 (46%) column=4
and in the other mail
Char: ü (2300, #o4374, #x8fc, file ...) point=493 of 940 (52%) column=0

I could reproduce with CVS head from this evening, minimal .gnus just
specifying the select method and emacs -q.


In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.12.8)
 of 2008-10-15 on glasgow.frank-schmitt.net
Windowing system distributor `The X.Org Foundation', version 11.0.10300000
configured using `configure  '--prefix=/opt/emacs23''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: de_DE.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Group

Minor modes in effect:
  gnus-undo-mode: t
  show-paren-mode: t
  delete-selection-mode: t
  pc-selection-mode: t
  recentf-mode: t
  iswitchb-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> M-x g n u s <return> <down> C-u 1 0 <return> 
<up> <up> <return> t <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <return> 
<return> <return> <return> <return> <return> <down> 
<return> <down-mouse-1> <mouse-1> C-x = <help-echo> 
<help-echo> <help-echo> <down-mouse-2> <mouse-2> <help-echo> 
<down-mouse-1> <mouse-1> <right> <left> C-x = <help-echo> 
<down-mouse-1> <drag-mouse-1> <help-echo> <help-echo> 
<down-mouse-1> <mouse-1> <home> <S-end> <C-insert> 
<down> <home> <S-end> <C-insert> <help-echo> <down-mouse-1> 
<mouse-movement> <mouse-1> <help-echo> q <return> <return> 
SPC q L <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<down> <down> <down> <down> M-x e m a <tab> b u <tab> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> b u g <tab> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> r e p <tab> o <tab> 
r <tab> <return>

Recent messages:
IMAP mailbox `Uni' closed... OK
Retrieving newsgroup: nnimap+ltw:Uni...
nnimap: Updating info for nnimap+ltw:Uni...done
Fetching headers for nnimap+ltw:Uni...done
Scoring...done
Generating summary...done
nnimap: Setting marks in Uni...done
No more unread newsgroups
IMAP mailbox `Uni' closed... OK
Making completion list... [2 times]

Message #10 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Andreas Schwab <schwab <at> suse.de>
To: usenet <at> frank-schmitt.net
Cc: 1174 <at> debbugs.gnu.org, emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 16 Oct 2008 09:32:23 +0200

Frank Schmitt <ich <at> frank-schmitt.net> writes:

> In Emacs 23 in the mail from my boss I get
> Char: � (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4

How is that character encoded in the mail?

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab <at> suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #20 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: bug-gnu-emacs <at> gnu.org
Cc: emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 16 Oct 2008 10:36:36 +0200

Andreas Schwab <schwab <at> suse.de> writes:

> Frank Schmitt <ich <at> frank-schmitt.net> writes:
>
>> In Emacs 23 in the mail from my boss I get
>> Char: � (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4
>
> How is that character encoded in the mail?

You mean the headers of the mail?

Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

and in the working one 

Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #30 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Frank Schmitt <ich <at> frank-schmitt.net>, 1174 <at> debbugs.gnu.org
Cc: bug-gnu-emacs <at> gnu.org, emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 16 Oct 2008 19:39:23 +0200

> From: Frank Schmitt <ich <at> frank-schmitt.net>
> Date: Thu, 16 Oct 2008 10:36:36 +0200
> Cc: emacs-pretest-bug <at> gnu.org
> 
> Andreas Schwab <schwab <at> suse.de> writes:
> 
> > Frank Schmitt <ich <at> frank-schmitt.net> writes:
> >
> >> In Emacs 23 in the mail from my boss I get
> >> Char: � (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4
> >
> > How is that character encoded in the mail?
> 
> You mean the headers of the mail?

No, Andreas means what sequence of 8-bit bytes was used to encode the
character in the original message?  A utf-8 encoding uses several
bytes to encode a single character; can you please show those bytes
from the body of the original mail message?

Message #45 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 1174 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 16 Oct 2008 20:00:07 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Frank Schmitt <ich <at> frank-schmitt.net>
>> Date: Thu, 16 Oct 2008 10:36:36 +0200
>> Cc: emacs-pretest-bug <at> gnu.org
>> 
>> Andreas Schwab <schwab <at> suse.de> writes:
>> 
>> > Frank Schmitt <ich <at> frank-schmitt.net> writes:
>> >
>> >> In Emacs 23 in the mail from my boss I get
>> >> Char: ��� (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4
>> >
>> > How is that character encoded in the mail?
>> 
>> You mean the headers of the mail?
>
> No, Andreas means what sequence of 8-bit bytes was used to encode the
> character in the original message?  A utf-8 encoding uses several
> bytes to encode a single character; can you please show those bytes
> from the body of the original mail message?

But how should I do this? Just yanking? Here we go: �

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #60 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Frank Schmitt <ich <at> frank-schmitt.net>
Cc: 1174 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 16 Oct 2008 20:15:26 +0200

> From: Frank Schmitt <ich <at> frank-schmitt.net>
> Cc: 1174 <at> emacsbugs.donarmstrong.com,  bug-gnu-emacs <at> gnu.org,  emacs-pretest-bug <at> gnu.org
> Date: Thu, 16 Oct 2008 20:00:07 +0200
> 
> > No, Andreas means what sequence of 8-bit bytes was used to encode the
> > character in the original message?  A utf-8 encoding uses several
> > bytes to encode a single character; can you please show those bytes
> > from the body of the original mail message?
> 
> But how should I do this?

Do you still have the original mail text, or can access it?  If so,
something like "od -c" (outside Emacs) on that text or
"M-x find-file-literally" inside Emacs will show you the original
encoding of that character.

Message #75 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 1174 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 16 Oct 2008 22:05:32 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Frank Schmitt <ich <at> frank-schmitt.net>
>> Cc: 1174 <at> emacsbugs.donarmstrong.com,  bug-gnu-emacs <at> gnu.org,  emacs-pretest-bug <at> gnu.org
>> Date: Thu, 16 Oct 2008 20:00:07 +0200
>> 
>> > No, Andreas means what sequence of 8-bit bytes was used to encode the
>> > character in the original message?  A utf-8 encoding uses several
>> > bytes to encode a single character; can you please show those bytes
>> > from the body of the original mail message?
>> 
>> But how should I do this?
>
> Do you still have the original mail text, or can access it?  If so,
> something like "od -c" (outside Emacs) on that text or
> "M-x find-file-literally" inside Emacs will show you the original
> encoding of that character.

I opened the mail again in Gnus, saved it and saved with no-conversion
and said od -c on it. Here is the relevant part

0001100   i   l   i   e   n   .       I   c   h       w 374   s   s   t
0001120   e       a   u   c   h       n   i   c   h   t   ,       w   i

when I save the same buffer from Emacs 22 I get something different

0001100  \n   B   r   a   s   i   l   i   e   n   .       I   c   h    
0001120   w 303 274   s   s   t   e       a   u   c   h       n   i   c
0001140   h   t   ,       w   i   e       w   i   r       a   u   s    

so I went to my mail server and did it on the raw message there

0012240      \n   B   r   a   s   i   l   i   e   n   .       I   c   h
0012260       w 303 274   s   s   t   e       a   u   c   h       n   i
0012300   c   h   t   ,       w   i   e       w   i   r       a   u   s

so I guess the last two are correct.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #90 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Richard M. Stallman" <rms <at> gnu.org>
To: Frank Schmitt <ich <at> frank-schmitt.net>, 1174 <at> debbugs.gnu.org
Cc: emacs-pretest-bug <at> gnu.org, bug-submit-list <at> donarmstrong.com,
        bug-gnu-emacs <at> gnu.org, bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 01:15:05 -0400

It seems to me that coding system names should be interpreted in a
case-insensitive manner, always.

Message #105 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Andreas Schwab <schwab <at> suse.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 1174 <at> debbugs.gnu.org, Frank Schmitt <ich <at> frank-schmitt.net>,
        emacs-pretest-bug <at> gnu.org, bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 10:28:40 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Frank Schmitt <ich <at> frank-schmitt.net>
>> Date: Thu, 16 Oct 2008 10:36:36 +0200
>> Cc: emacs-pretest-bug <at> gnu.org
>> 
>> Andreas Schwab <schwab <at> suse.de> writes:
>> 
>> > Frank Schmitt <ich <at> frank-schmitt.net> writes:
>> >
>> >> In Emacs 23 in the mail from my boss I get
>> >> Char: � (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4
>> >
>> > How is that character encoded in the mail?
>> 
>> You mean the headers of the mail?
>
> No, Andreas means what sequence of 8-bit bytes was used to encode the
> character in the original message?

It was acatually the right hint.  ISTR there was a problem in Gnus with
8-bit encoded articles, but all I could find is
<http://permalink.gmane.org/gmane.emacs.gnus.general/67112>.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab <at> suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #120 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Andreas Schwab <schwab <at> suse.de>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 1174 <at> debbugs.gnu.org,
        emacs-pretest-bug <at> gnu.org, bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 10:52:15 +0200

Andreas Schwab <schwab <at> suse.de> writes:

> Eli Zaretskii <eliz <at> gnu.org> writes:
>
>>> From: Frank Schmitt <ich <at> frank-schmitt.net>
>>> Date: Thu, 16 Oct 2008 10:36:36 +0200
>>> Cc: emacs-pretest-bug <at> gnu.org
>>> 
>>> Andreas Schwab <schwab <at> suse.de> writes:
>>> 
>>> > Frank Schmitt <ich <at> frank-schmitt.net> writes:
>>> >
>>> >> In Emacs 23 in the mail from my boss I get
>>> >> Char: � (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4
>>> >
>>> > How is that character encoded in the mail?
>>> 
>>> You mean the headers of the mail?
>>
>> No, Andreas means what sequence of 8-bit bytes was used to encode the
>> character in the original message?
>
> It was acatually the right hint.  ISTR there was a problem in Gnus with
> 8-bit encoded articles, but all I could find is
> <http://permalink.gmane.org/gmane.emacs.gnus.general/67112>.

Yes, this seems to be the same problem I am seeing. Any idea on how I
could further debug this? It is a real show-stopper for me.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #135 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Andreas Schwab <schwab <at> suse.de>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 1174 <at> debbugs.gnu.org,
        emacs-pretest-bug <at> gnu.org, bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 11:12:00 +0200

Andreas Schwab <schwab <at> suse.de> writes:

> Eli Zaretskii <eliz <at> gnu.org> writes:
>
>>> From: Frank Schmitt <ich <at> frank-schmitt.net>
>>> Date: Thu, 16 Oct 2008 10:36:36 +0200
>>> Cc: emacs-pretest-bug <at> gnu.org
>>> 
>>> Andreas Schwab <schwab <at> suse.de> writes:
>>> 
>>> > Frank Schmitt <ich <at> frank-schmitt.net> writes:
>>> >
>>> >> In Emacs 23 in the mail from my boss I get
>>> >> Char: � (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4
>>> >
>>> > How is that character encoded in the mail?
>>> 
>>> You mean the headers of the mail?
>>
>> No, Andreas means what sequence of 8-bit bytes was used to encode the
>> character in the original message?
>
> It was acatually the right hint.  ISTR there was a problem in Gnus with
> 8-bit encoded articles, but all I could find is
> <http://permalink.gmane.org/gmane.emacs.gnus.general/67112>.

One more data point: It only happens in Mails fetched via nnimap. If I
reproduce with nntp (that means if I send a news articles with 8bit
encoding and UTF-8) everything is displayed correctly. 

As stated in the bug report cited by Andreas above nnimap does have a function

(defsubst imap-disable-multibyte ()
  "Enable multibyte in the current buffer."
  (when (fboundp 'set-buffer-multibyte)
    (set-buffer-multibyte nil)))

which is called for every buffer involved (see e.g. imap-open). Might
this be the problem?

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #150 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Frank Schmitt <ich <at> frank-schmitt.net>
Cc: schwab <at> suse.de, 1174 <at> debbugs.gnu.org, emacs-pretest-bug <at> gnu.org,
        bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 11:53:12 +0200

> From: Frank Schmitt <ich <at> frank-schmitt.net>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  1174 <at> emacsbugs.donarmstrong.com,  emacs-pretest-bug <at> gnu.org,  bug-gnu-emacs <at> gnu.org
> Date: Fri, 17 Oct 2008 10:52:15 +0200
> 
> Andreas Schwab <schwab <at> suse.de> writes:
> 
> > Eli Zaretskii <eliz <at> gnu.org> writes:
> >
> >>> From: Frank Schmitt <ich <at> frank-schmitt.net>
> >>> Date: Thu, 16 Oct 2008 10:36:36 +0200
> >>> Cc: emacs-pretest-bug <at> gnu.org
> >>> 
> >>> Andreas Schwab <schwab <at> suse.de> writes:
> >>> 
> >>> > Frank Schmitt <ich <at> frank-schmitt.net> writes:
> >>> >
> >>> >> In Emacs 23 in the mail from my boss I get
> >>> >> Char: � (4194300, #o17777774, #x3ffffc, raw-byte) point=212 of 461 (46%) column=4
> >>> >
> >>> > How is that character encoded in the mail?
> >>> 
> >>> You mean the headers of the mail?
> >>
> >> No, Andreas means what sequence of 8-bit bytes was used to encode the
> >> character in the original message?
> >
> > It was acatually the right hint.  ISTR there was a problem in Gnus with
> > 8-bit encoded articles, but all I could find is
> > <http://permalink.gmane.org/gmane.emacs.gnus.general/67112>.
> 
> Yes, this seems to be the same problem I am seeing. Any idea on how I
> could further debug this? It is a real show-stopper for me.

Is it true that this happens when the charset is stated as UTF-8
(upper-case), but works if it is utf-8 (lower-case)?  If so, then
Gnus, or one of the functions it calls, doesn't downcase the charset
name before it intern's it, to get the coding-system.

Or is the problem the "; format=flowed" thingy?

Message #165 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: schwab <at> suse.de, 1174 <at> debbugs.gnu.org, emacs-pretest-bug <at> gnu.org,
        bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 12:02:46 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Is it true that this happens when the charset is stated as UTF-8
> (upper-case), but works if it is utf-8 (lower-case)?  If so, then
> Gnus, or one of the functions it calls, doesn't downcase the charset
> name before it intern's it, to get the coding-system.
>
> Or is the problem the "; format=flowed" thingy?

I think neither of both. It must be a problem of raw 8bit encoded UTF-8
specific to nnimap.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #180 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: schwab <at> suse.de, 1174 <at> debbugs.gnu.org, emacs-pretest-bug <at> gnu.org,
        bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 12:08:15 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> > It was acatually the right hint.  ISTR there was a problem in Gnus with
>> > 8-bit encoded articles, but all I could find is
>> > <http://permalink.gmane.org/gmane.emacs.gnus.general/67112>.
>> 
>> Yes, this seems to be the same problem I am seeing. Any idea on how I
>> could further debug this? It is a real show-stopper for me.

Got it!

In lisp/gnus/nnimap.el there is 

(defun nnimap-demule (string)
  ;; BEWARE: we used to use string-as-multibyte here which is braindead
  ;; because it will turn accidental emacs-mule-valid byte sequences
  ;; into multibyte chars.  --Stef
  ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
  ;; that bad. --Simon
  (funcall (if (and (fboundp 'string-as-multibyte)
		    (subrp (symbol-function 'string-as-multibyte)))
	       'string-as-multibyte
	     'identity)
	   (or string "")))

which is called at several places. And this breaks it. If I change this
function so that string is not changed, my mails are displayed correctly.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #195 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Frank Schmitt <ich <at> frank-schmitt.net>
Cc: 1174 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
        schwab <at> suse.de, bug-gnu-emacs <at> gnu.org, emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 11:39:53 -0400

>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>   ;; because it will turn accidental emacs-mule-valid byte sequences
>   ;; into multibyte chars.  --Stef
>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>   ;; that bad. --Simon

Who's this Simon who reverted my change without even explaining why?


        Stefan

Message #210 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Simon Josefsson <jas <at> extundo.com>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org, Frank Schmitt <ich <at> frank-schmitt.net>
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 20:19:59 +0200

On Fri, Oct 17 2008, Stefan Monnier wrote:

>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>   ;; into multibyte chars.  --Stef
>>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>   ;; that bad. --Simon
>
> Who's this Simon who reverted my change without even explaining why?

The one who's listed as the author of nnimap.el (cc-ed).

>> which is called at several places. And this breaks it. If I change
>> this function so that string is not changed, my mails are displayed
>> correctly.

Does it work correctly when using Stefan's version?
( s/string-as-multibyte/string-to-multibyte/ ...)

(defun nnimap-demule (string)
  ;; BEWARE: we used to use string-as-multibyte here which is braindead
  ;; because it will turn accidental emacs-mule-valid byte sequences
  ;; into multibyte chars.  --Stef
  (funcall (if (and (fboundp 'string-to-multibyte)
		    (subrp (symbol-function 'string-to-multibyte)))
	       'string-to-multibyte
	     'identity)
	   (or string "")))

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #215 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Simon Josefsson <jas <at> extundo.com>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Fri, 17 Oct 2008 20:36:58 +0200

Reiner Steib <reinersteib+gmane <at> imap.cc> writes:

> On Fri, Oct 17 2008, Stefan Monnier wrote:
>
>>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>>   ;; into multibyte chars.  --Stef
>>>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>>   ;; that bad. --Simon
>>
>> Who's this Simon who reverted my change without even explaining why?
>
> The one who's listed as the author of nnimap.el (cc-ed).
>
>>> which is called at several places. And this breaks it. If I change
>>> this function so that string is not changed, my mails are displayed
>>> correctly.
>
> Does it work correctly when using Stefan's version?
> ( s/string-as-multibyte/string-to-multibyte/ ...)
>
> (defun nnimap-demule (string)
>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>   ;; because it will turn accidental emacs-mule-valid byte sequences
>   ;; into multibyte chars.  --Stef
>   (funcall (if (and (fboundp 'string-to-multibyte)
> 		    (subrp (symbol-function 'string-to-multibyte)))
> 	       'string-to-multibyte
> 	     'identity)
> 	   (or string "")))

Yes, it does.

Frank

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #220 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: bug-gnu-emacs <at> gnu.org
Cc: emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 23 Oct 2008 09:24:20 +0200

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>   ;; into multibyte chars.  --Stef
>>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>   ;; that bad. --Simon
>
> Who's this Simon who reverted my change without even explaining why?

So, what's the outcome? We found the place where the bug occurs, I can
confirm that with string-to-multibyte instead of string-as-multibyte
everything works as expected, what's next?

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #230 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Frank Schmitt <ich <at> frank-schmitt.net>
Cc: 1174 <at> debbugs.gnu.org, emacs-pretest-bug <at> gnu.org,
        bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 23 Oct 2008 09:58:55 -0400

>>> ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>> ;; because it will turn accidental emacs-mule-valid byte sequences
>>> ;; into multibyte chars.  --Stef
>>> ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>> ;; that bad. --Simon
>> 
>> Who's this Simon who reverted my change without even explaining why?

> So, what's the outcome? We found the place where the bug occurs, I can
> confirm that with string-to-multibyte instead of string-as-multibyte
> everything works as expected, what's next?

Next is to get this Simon to explain his change, so we can resolve the
conflict between mine and his.


        Stefan

Message #245 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: James Cloos <cloos <at> jhcloos.com>
To: 1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Tue, 28 Oct 2008 18:54:53 -0400

>>>>> "Frank" == Frank Schmitt <ich <at> frank-schmitt.net> writes:

[Catching up on mail backlog -JimC]

Frank> You mean the headers of the mail?

Frank> Content-Type: text/plain; charset=UTF-8; format=flowed
Frank> Content-Transfer-Encoding: 8bit

Frank> and in the working one 

Frank> Content-Type: text/plain; charset=utf-8
Frank> Content-Transfer-Encoding: quoted-printable

I reported this bug on the ding list some time back.

Shortly after the unicode-2 branch was merged into trunk changes were
made to gnus wrt unibyte vs multibyte buffers and/or strings at the
stage where it decodes article bodies.

The change broke much (all?) UTF-8 mail which has an 8bit CTE.

It had worked fine (in the unicode-2 branch) until then.

-JimC
-- 
James Cloos <cloos <at> jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Message #250 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: Simon Josefsson <jas <at> extundo.com>
Cc: Frank Schmitt <ich <at> frank-schmitt.net>,
        Stefan Monnier <monnier <at> iro.umontreal.ca>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Sat, 29 Nov 2008 13:08:20 +0100

On Fri, Oct 17 2008, Frank Schmitt wrote:

> Reiner Steib <reinersteib+gmane <at> imap.cc> writes:
>
>> On Fri, Oct 17 2008, Stefan Monnier wrote:
>>
>>>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>>>   ;; into multibyte chars.  --Stef
>>>>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>>>   ;; that bad. --Simon

Simon, could you please clarify why you reverted Stefan's change in
`nnimap-demule'?  It breaks reading UTF-8 articles via nnimap.el in
Emacs 23.

>>> Who's this Simon who reverted my change without even explaining why?
>>
>> The one who's listed as the author of nnimap.el (cc-ed).
>>
>>>> which is called at several places. And this breaks it. If I change
>>>> this function so that string is not changed, my mails are displayed
>>>> correctly.
>>
>> Does it work correctly when using Stefan's version?
>> ( s/string-as-multibyte/string-to-multibyte/ ...)
>>
>> (defun nnimap-demule (string)
>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>   ;; into multibyte chars.  --Stef
>>   (funcall (if (and (fboundp 'string-to-multibyte)
>> 		    (subrp (symbol-function 'string-to-multibyte)))
>> 	       'string-to-multibyte
>> 	     'identity)
>> 	   (or string "")))
>
> Yes, it does.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #255 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Frank Schmitt <ich <at> frank-schmitt.net>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Sat, 29 Nov 2008 13:18:21 +0100

Reiner Steib <reinersteib+gmane <at> imap.cc> writes:

> On Fri, Oct 17 2008, Frank Schmitt wrote:
>
>> Reiner Steib <reinersteib+gmane <at> imap.cc> writes:
>>
>>> On Fri, Oct 17 2008, Stefan Monnier wrote:
>>>
>>>>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>>>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>>>>   ;; into multibyte chars.  --Stef
>>>>>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>>>>   ;; that bad. --Simon
>
> Simon, could you please clarify why you reverted Stefan's change in
> `nnimap-demule'?  It breaks reading UTF-8 articles via nnimap.el in
> Emacs 23.

I don't recall, but people should notice relatively quickly if there are
problems in this area (wrong display of non-ascii) so feel free to
revert the patch or apply another patch instead.  It needs to be tested
under Emacs 22 too, though, if it is installed in the Gnus CVS.

/Simon

Message #260 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: Simon Josefsson <simon <at> josefsson.org>
Cc: Frank Schmitt <ich <at> frank-schmitt.net>, James Cloos <cloos <at> jhcloos.com>,
        Stefan Monnier <monnier <at> iro.umontreal.ca>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org,
        Clemens Schueller <cs.usenet.nospam <at> aon.at>
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Sat, 29 Nov 2008 16:30:18 +0100

On Sat, Nov 29 2008, Simon Josefsson wrote:

> Reiner Steib <reinersteib+gmane <at> imap.cc> writes:
>>>>>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>>>>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>>>>>   ;; into multibyte chars.  --Stef
>>>>>>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>>>>>   ;; that bad. --Simon
>>
>> Simon, could you please clarify why you reverted Stefan's change in
>> `nnimap-demule'?  It breaks reading UTF-8 articles via nnimap.el in
>> Emacs 23.
>
> I don't recall, but people should notice relatively quickly if there are
> problems in this area (wrong display of non-ascii) 

Hm, both changes happened 2004, but I don't recall any reports beside
the recent ones in 2008:

  http://thread.gmane.org/gmane.emacs.gnus.general/67112
  (bug#464, reported by James Cloos)
  http://thread.gmane.org/gmane.emacs.bugs/21524
  (bug#1174, reported by Frank Schmitt)
  
  ----------------------------
  revision 7.9
  date: 2004-09-13 13:52:48 +0200;  author: jas;  state: Exp;  lines: +5 -3
  (nnimap-demule): Revert 2004-08-30 change.
  ----------------------------

> so feel free to revert the patch or apply another patch instead.  It
> needs to be tested under Emacs 22 too, though,

I never saw this problem myself.  I cannot see any difference with a
few UTF-8 articles (C-T-E: 8bit, [1]), neither with Emacs 22 (with
current Gnus trunk) or Emacs trunk (Gnus 5.13 from there).  Stefan,
what are the "accidental emacs-mule-valid byte sequences" that trigger
this problem?  It would be good if someone could send me a problematic
article. [2]

I've just checked in (Gnus and Emacs) some code to debug this problem.
I'd like to ask those who saw the bug in Emacs 23 to test the articles
in question:

- With current Emacs 23 (Emacs CVS trunk)

- With Emacs 22 plus current Gnus CVS trunk (No Gnus)

- If you see wrong display: does it display correctly after evaluating
  the following:

  M-x gnus-backlog-shutdown RET

  (setq nnimap-demule-use-string-to-multibyte nil
        gnus-verbose 10)

  Check the *Messages* buffer for messages
  "nnimap-demule-use-string-to-multibyte: nil" to ensure that the
  article is decoded again with this setting.  You may need to
  re-enter the group.

> if it is installed in the Gnus CVS.

I don't want different code in Gnus and Emacs.  If all else fails, we
can make it conditional.

Bye, Reiner.

[1] Cc-s of the following articles (available on Gmane) from Aidan Kehoe:
    <18492.30425.377545.700503 <at> parhasard.net>
    <18518.43672.183610.662699 <at> parhasard.net>
    <18712.43474.265690.792714 <at> parhasard.net>

    Non-ascii characters in the attribution line ("scríobh"), the
    signature ("¿Dónde estará ahora mi sobrino Yoghurtu Nghé ..."),
    and "’", "İ", "ı"

[2] (push '(utf-8 . 8bit) mm-body-charset-encoding-alist)
    --> trying to produce a problematic article here:
    AE-Ä OE-Ö UE-Ü ae-ä oe-ö ue-ü ss-ß

    Should be sent with: 
      Content-Type: text/plain; charset=utf-8
      Content-Transfer-Encoding: 8bit
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #265 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Reiner Steib <Reiner.Steib <at> gmx.de>
Cc: Simon Josefsson <simon <at> josefsson.org>,
        Frank Schmitt <ich <at> frank-schmitt.net>, James Cloos <cloos <at> jhcloos.com>,
        ding <at> gnus.org, 1174 <at> debbugs.gnu.org,
        Clemens Schueller <cs.usenet.nospam <at> aon.at>
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Sat, 29 Nov 2008 16:30:49 -0500

> I never saw this problem myself.  I cannot see any difference with a
> few UTF-8 articles (C-T-E: 8bit, [1]), neither with Emacs 22 (with
> current Gnus trunk) or Emacs trunk (Gnus 5.13 from there).  Stefan,
> what are the "accidental emacs-mule-valid byte sequences" that trigger
> this problem?  It would be good if someone could send me a problematic
> article. [2]

In Emacs-22, the problem was more difficult to trigger: you had to
receive an email whose undecoded text contained emacs-mule escape
sequences, which is rather uncommon.  With Emacs-23, it's a lot more
common since the internal encoding has changed to a variant of utf-8:
an 8bit body using utf-8 will see its content unwillingly decoded during
nnimap-demule which leads to the bugs we've seen recently.

I'm pretty sure that string-as-multibyte is wrong here in general.
Maybe the problem is that nnimap-demule is used blindly in different
contexts where some need string-to-multibyte and some need
string-as-multibyte.  E.g. maybe Simon's problem was linked to imap
groups with non-ASCII chars in their names, rather than in the
message bodies.


        Stefan

Message #270 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: James Cloos <cloos <at> jhcloos.com>
To: Reiner Steib <Reiner.Steib <at> gmx.de>
Cc: Simon Josefsson <simon <at> josefsson.org>,
        Frank Schmitt <ich <at> frank-schmitt.net>,
        Stefan Monnier <monnier <at> iro.umontreal.ca>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org,
        Clemens Schueller <cs.usenet.nospam <at> aon.at>
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Sat, 29 Nov 2008 17:14:11 -0500

The easiest test case are utf-8 emails which are 8bit mime and which
fail to specify their charset, or specify utf-8 in some incorrect way.

The most common example may be commit messgages; the scripts usually do
not specify a charset since they commit messages and src could be in any
encoding and they send them out as is.

NB that this bug never showed up for me when using the unicode-2 branch
before that was pulled into mainline.  Some unibyte vs multibyte change
was made after that happened which started triggering the bug.

A possibly related issue is that (gnus-article-treat-dumbquotes) also
does not work for me anymore.  I do specify my own gnus-article-dumbquotes-map
in ~/.gnus, but the syntax is the same as what is still (defvar)ed in
gnus-art.el.  (I use utf-8 results rather than ascii fallbacks.)
The symptom is that the strings such as "\221" do not match the relevant
octets in the *Article* buffer, as they used to do.  This also stopped
working at the same time.

My goal would be for gnus to treat messages and mime blocks w/o a
charset as utf-8 rather than ascii by default.  That does work for qp
and base64, just not for 8bitmime.  Unless Stefan's patch is applied.
But it also must be easy to manually tell it to use some other charset
when one recognizes the need.  I can use (gnus-article-view-part-as-charset)
for a mime part (at least w/ Stefan's patch) but haven't managed to make
that work for a message w/o any attachments or inlines.

I'm running w/ Stefan's patch now and it works.  I'll try a new compile
tonight or tomorrow and test the committed code.

-JimC
-- 
James Cloos <cloos <at> jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Message #275 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Simon Josefsson <simon <at> josefsson.org>,
        Frank Schmitt <ich <at> frank-schmitt.net>, James Cloos <cloos <at> jhcloos.com>,
        ding <at> gnus.org, 1174 <at> debbugs.gnu.org,
        Clemens Schueller <cs.usenet.nospam <at> aon.at>
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Sun, 30 Nov 2008 14:12:02 +0100

On Sat, Nov 29 2008, Stefan Monnier wrote:

>> Stefan, what are the "accidental emacs-mule-valid byte sequences"
>> that trigger this problem?  It would be good if someone could send
>> me a problematic article. [2]
>
> In Emacs-22, the problem was more difficult to trigger: you had to
> receive an email whose undecoded text contained emacs-mule escape
> sequences, which is rather uncommon.  With Emacs-23, it's a lot more
> common since the internal encoding has changed to a variant of utf-8:
> an 8bit body using utf-8 will see its content unwillingly decoded during
> nnimap-demule which leads to the bugs we've seen recently.

Could you send me an article demonstrating the problem?

> I'm pretty sure that string-as-multibyte is wrong here in general.
> Maybe the problem is that nnimap-demule is used blindly in different
> contexts where some need string-to-multibyte and some need
> string-as-multibyte.  E.g. maybe Simon's problem was linked to imap
> groups with non-ASCII chars in their names, rather than in the
> message bodies.

I'm not familiar with the IMAP code, but AFAICS, `nnimap-demule' is
only used when getting headers or body:

| nnimap.el:611:	      headers (nnimap-demule

(defun nnimap-retrieve-headers-progress ()
  "Hook to insert NOV line for current article into `nntp-server-buffer'."

| nnimap.el:951:	 (nnimap-demule

(defun nnimap-callback (article gnus-callback buffer)
  (when (eq article (imap-current-message))
    (remove-hook 'imap-fetch-data-hook
		 (nnimap-make-callback article gnus-callback buffer))
    (with-current-buffer buffer
      (insert
       (with-current-buffer nnimap-server-buffer
	 (nnimap-demule
	  (if (imap-capability 'IMAP4rev1)
	      ;; xxx don't just use car? alist doesn't contain
	      ;; anything else now, but it might...
	      (nth 2 (car (imap-message-get article 'BODYDETAIL)))
	    (imap-message-get article 'RFC822)))))
      (nnheader-ms-strip-cr)
      (funcall gnus-callback t))))

| nnimap.el:977:		(insert (nnimap-demule (if detail

(defun nnimap-request-article-part (article part prop &optional
					    group server to-buffer detail)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #280 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Simon Josefsson <jas <at> extundo.com>
Cc: Frank Schmitt <ich <at> frank-schmitt.net>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Mon, 01 Dec 2008 16:04:01 -0500

> Simon, could you please clarify why you reverted Stefan's change in
> `nnimap-demule'?  It breaks reading UTF-8 articles via nnimap.el in
> Emacs 23.

Having looked at the code again, I'm more than ever confident that
string-to-unibyte is the right thing to use.  Maybe the code I installed
back then failed to fallback to string-as-unibyte when string-to-unibyte
was not available, which caused a bug for Simon?

In any case the newly committed code has a prenthesis typo that makes
it still use the old code and ignore the new config var
nnimap-demule-use-string-to-multibyte.

Also I recommend to just use the patch below instead.  The first hunk
removes an unnecessary use of nnimap-demule since the output will be
inserted into a unibyte buffer.


        Stefan


--- nnimap.el.~1.50.~	2008-12-01 15:38:55.000000000 -0500
+++ nnimap.el	2008-12-01 15:49:53.000000000 -0500
@@ -608,12 +608,11 @@
       (with-current-buffer nnimap-server-buffer
 	(setq uid imap-current-message
 	      mbx imap-current-mailbox
-	      headers (nnimap-demule
-		       (if (imap-capability 'IMAP4rev1)
+	      headers (if (imap-capability 'IMAP4rev1)
 			   ;; xxx don't just use car? alist doesn't contain
 			   ;; anything else now, but it might...
 			   (nth 2 (car (imap-message-get uid 'BODYDETAIL)))
-			 (imap-message-get uid 'RFC822.HEADER)))
+                        (imap-message-get uid 'RFC822.HEADER))
 	      lines (imap-body-lines (imap-message-body imap-current-message))
 	      chars (imap-message-get imap-current-message 'RFC822.SIZE)))
       (nnheader-insert-nov
@@ -901,40 +900,17 @@
   (when (nnimap-possibly-change-server server)
     (nnoo-status-message 'nnimap server)))
 
-(defvar nnimap-demule-use-string-to-multibyte (fboundp 'string-to-multibyte)
-  "Temporary internal debug variable.
-If you have problems (UTF-8 not decoded correctly on IMAP) with
-the default value, please report it as a bug!")
-;; FIXME: Clarify if we need to make this variable conditional on the Emacs
-;; version (Emacs 22 vs. Emacs 23;Emacs 21 doesn't have `string-to-multibyte'
-;; anyhow).  --rsteib
-;;
-;; http://thread.gmane.org/gmane.emacs.gnus.general/67112
-;; (bug#464, reported by James Cloos)
-;; http://thread.gmane.org/gmane.emacs.bugs/21524
-;; (bug#1174, reported by Frank Schmitt)
-
-(defun nnimap-demule (string)
-  ;; BEWARE: we used to use string-as-multibyte here which is braindead
-  ;; because it will turn accidental emacs-mule-valid byte sequences
-  ;; into multibyte chars.  --Stef
-  ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
-  ;; that bad. --Simon
-  (gnus-message 9 "nnimap-demule-use-string-to-multibyte: %s"
-		nnimap-demule-use-string-to-multibyte)
-  (if nnimap-demule-use-string-to-multibyte
-      ;; Stefan
-      (funcall (if (and (fboundp 'string-to-multibyte)
-			(subrp (symbol-function 'string-to-multibyte)))
-		   'string-to-multibyte
-		 'identity)
-	       (or string "")))
-  ;; Simon
-  (funcall (if (and (fboundp 'string-as-multibyte)
-		    (subrp (symbol-function 'string-as-multibyte)))
-	       'string-as-multibyte
-	     'identity)
-	   (or string "")))
+;; We used to use a string-as-multibyte here, but it is really incorrect.
+;; This function is used when we're about to insert a unibyte string
+;; into a potentially multibyte buffer.  The string is either an article
+;; header or body (or both?), undecoded.  When Emacs is asked to convert
+;; a unibyte string to multibyte, it may either use the equivalent of
+;; nothing (e.g. non-Mule XEmacs), string-make-unibyte (i.e. decode using
+;; locale), string-as-multibyte (decode using emacs-internal coding system)
+;; or string-to-multibyte (keep the data undecoded as a sequence of bytes).
+;; Only the last one preserves the data such that we can reliably later on
+;; decode the text using the mime info.
+(defalias 'nnimap-demule 'mm-string-to-multibyte)
 
 (defun nnimap-make-callback (article gnus-callback buffer)
   "Return a callback function."

Message #285 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Simon Josefsson <jas <at> extundo.com>, Frank Schmitt <ich <at> frank-schmitt.net>,
        ding <at> gnus.org, 1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Mon, 01 Dec 2008 23:48:32 +0100

On Mon, Dec 01 2008, Stefan Monnier wrote:

> Having looked at the code again, I'm more than ever confident that
> string-to-unibyte is the right thing to use.  Maybe the code I installed
> back then failed to fallback to string-as-unibyte when string-to-unibyte
> was not available, which caused a bug for Simon?

Yes, it didn't fall back to string-as-unibyte:

--- nnimap.el	17 Aug 2004 14:27:16 -0000	7.7
+++ nnimap.el	30 Aug 2004 18:13:58 -0000	7.8
[...]
@@ -845,9 +847,12 @@
     (nnoo-status-message 'nnimap server)))
 
 (defun nnimap-demule (string)
-  (funcall (if (and (fboundp 'string-as-multibyte)
-		    (subrp (symbol-function 'string-as-multibyte)))
-	       'string-as-multibyte
+  ;; BEWARE: we used to use string-as-multibyte here which is braindead
+  ;; because it will turn accidental emacs-mule-valid byte sequences
+  ;; into multibyte chars.  --Stef
+  (funcall (if (and (fboundp 'string-to-multibyte)
+		    (subrp (symbol-function 'string-to-multibyte)))
+	       'string-to-multibyte
 	     'identity)
 	   (or string "")))

> In any case the newly committed code has a prenthesis typo that makes
> it still use the old code and ignore the new config var
> nnimap-demule-use-string-to-multibyte.

Oops, stupid me.

> Also I recommend to just use the patch below instead.  The first hunk
> removes an unnecessary use of nnimap-demule since the output will be
> inserted into a unibyte buffer.

Thanks for your analysis. Please install the patch.  I'll pull it into
Gnus CVS ASAP (unless Miles syncs first).

> +;; We used to use a string-as-multibyte here, but it is really incorrect.
> +;; This function is used when we're about to insert a unibyte string
> +;; into a potentially multibyte buffer.  The string is either an article
> +;; header or body (or both?), undecoded.  When Emacs is asked to convert
> +;; a unibyte string to multibyte, it may either use the equivalent of
> +;; nothing (e.g. non-Mule XEmacs), string-make-unibyte (i.e. decode using
> +;; locale), string-as-multibyte (decode using emacs-internal coding system)
> +;; or string-to-multibyte (keep the data undecoded as a sequence of bytes).
> +;; Only the last one preserves the data such that we can reliably later on
> +;; decode the text using the mime info.
> +(defalias 'nnimap-demule 'mm-string-to-multibyte)

In Emacs 21 (which Gnus still aim to be compatible with), we have
string-as-multibyte, but not string-to-multibyte.  So your proposed
code (i.e. mm-string-to-multibyte) runs
  (string-as-multibyte (char-to-string string))
whereas we used to run
  (string-as-multibyte string)
Does char-to-string matter here?

(defalias 'mm-string-to-multibyte
  (cond
   ((featurep 'xemacs)
    'identity)
   ((fboundp 'string-to-multibyte)
    'string-to-multibyte)
   (t
    (lambda (string)
      "Return a multibyte string with the same individual chars as string."
      (mapconcat
       (lambda (ch) (mm-string-as-multibyte (char-to-string ch)))
       string "")))))

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #290 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Simon Josefsson <jas <at> extundo.com>
Cc: Frank Schmitt <ich <at> frank-schmitt.net>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Tue, 02 Dec 2008 02:36:31 -0500

> In Emacs 21 (which Gnus still aim to be compatible with), we have
> string-as-multibyte, but not string-to-multibyte.  So your proposed
> code (i.e. mm-string-to-multibyte) runs
>   (string-as-multibyte (char-to-string string))
> whereas we used to run
>   (string-as-multibyte string)
> Does char-to-string matter here?

> (defalias 'mm-string-to-multibyte
>   (cond
>    ((featurep 'xemacs)
>     'identity)
>    ((fboundp 'string-to-multibyte)
>     'string-to-multibyte)
>    (t
>     (lambda (string)
>       "Return a multibyte string with the same individual chars as string."
>       (mapconcat
>        (lambda (ch) (mm-string-as-multibyte (char-to-string ch)))
>        string "")))))

Oh, that's clever: yes, the mapconcat/char-to-string dance does make it
implement the string-to-multibyte behavior because doing the
string-as-multibyte conversion one byte at a time avoids the
problematic case.  To quote myself from mm-util.el:

     ;; string-as-multibyte often doesn't really do what you think it does.
     ;; Example:
     ;;    (aref (string-as-multibyte "\201") 0) -> 129 (aka ?\201)
     ;;    (aref (string-as-multibyte "\300") 0) -> 192 (aka ?\300)
     ;;    (aref (string-as-multibyte "\300\201") 0) -> 192 (aka ?\300)
     ;;    (aref (string-as-multibyte "\300\201") 1) -> 129 (aka ?\201)
     ;; but
     ;;    (aref (string-as-multibyte "\201\300") 0) -> 2240
     ;;    (aref (string-as-multibyte "\201\300") 1) -> <error>

Basically when the sring passed is made of a single byte,
string-as-multibyte is equal to string-to-multibyte, which is the
property ued by the code you quoted above to build a poor man's
string-to-multibyte.


        Stefan

Message #295 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Simon Josefsson <jas <at> extundo.com>, Frank Schmitt <ich <at> frank-schmitt.net>,
        ding <at> gnus.org, 1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 04 Dec 2008 20:43:41 +0100

On Tue, Dec 02 2008, Stefan Monnier wrote:

>> In Emacs 21 (which Gnus still aim to be compatible with), we have
>> string-as-multibyte, but not string-to-multibyte.  So your proposed
>> code (i.e. mm-string-to-multibyte) runs
>
>>   (string-as-multibyte (char-to-string string))
>> whereas we used to run
>>   (string-as-multibyte string)
>> Does char-to-string matter here?
[...]
>>     (lambda (string)
>>       "Return a multibyte string with the same individual chars as string."
>>       (mapconcat
>>        (lambda (ch) (mm-string-as-multibyte (char-to-string ch)))
>>        string "")))))
>
> Oh, that's clever: yes, the mapconcat/char-to-string dance does make it
> implement the string-to-multibyte behavior because doing the
> string-as-multibyte conversion one byte at a time avoids the
> problematic case.  

Good.  So I think you can close this bug.  Thanks.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #300 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Simon Josefsson <jas <at> extundo.com>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 04 Dec 2008 22:43:51 +0100

Reiner Steib <reinersteib+gmane <at> imap.cc> writes:

> On Tue, Dec 02 2008, Stefan Monnier wrote:
>
>>> In Emacs 21 (which Gnus still aim to be compatible with), we have
>>> string-as-multibyte, but not string-to-multibyte.  So your proposed
>>> code (i.e. mm-string-to-multibyte) runs
>>
>>>   (string-as-multibyte (char-to-string string))
>>> whereas we used to run
>>>   (string-as-multibyte string)
>>> Does char-to-string matter here?
> [...]
>>>     (lambda (string)
>>>       "Return a multibyte string with the same individual chars as string."
>>>       (mapconcat
>>>        (lambda (ch) (mm-string-as-multibyte (char-to-string ch)))
>>>        string "")))))
>>
>> Oh, that's clever: yes, the mapconcat/char-to-string dance does make it
>> implement the string-to-multibyte behavior because doing the
>> string-as-multibyte conversion one byte at a time avoids the
>> problematic case.  
>
> Good.  So I think you can close this bug.  Thanks.

Yes, I can confirm the bug is fixed in CVS head.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #305 received at 1174-done <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: 1174-done <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Thu, 04 Dec 2008 16:57:36 -0500

>> Oh, that's clever: yes, the mapconcat/char-to-string dance does make it
>> implement the string-to-multibyte behavior because doing the
>> string-as-multibyte conversion one byte at a time avoids the
>> problematic case.  
> Good.  So I think you can close this bug.  Thanks.

OK, done,


        Stefan

Message #310 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Frank Schmitt <ich <at> frank-schmitt.net>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Mon, 12 Jan 2009 11:54:01 +0100

I have recently upgraded my Gnus installation, and it included this fix.
Now every e-mail I send has non-ASCII characters pre-fixed with \201.
So the patch installed does not seem to be the right, or there is
something else wrong with my configuration.  Can anyone else reproduce
this?  I'm including 'åäö' in this e-mail for debugging.

I'll see if I can debug this further, and find the exact part of the
patch that cause the problem.

/Simon

Simon Josefsson <simon <at> josefsson.org> writes:

> Reiner Steib <reinersteib+gmane <at> imap.cc> writes:
>
>> On Fri, Oct 17 2008, Frank Schmitt wrote:
>>
>>> Reiner Steib <reinersteib+gmane <at> imap.cc> writes:
>>>
>>>> On Fri, Oct 17 2008, Stefan Monnier wrote:
>>>>
>>>>>>   ;; BEWARE: we used to use string-as-multibyte here which is braindead
>>>>>>   ;; because it will turn accidental emacs-mule-valid byte sequences
>>>>>>   ;; into multibyte chars.  --Stef
>>>>>>   ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be
>>>>>>   ;; that bad. --Simon
>>
>> Simon, could you please clarify why you reverted Stefan's change in
>> `nnimap-demule'?  It breaks reading UTF-8 articles via nnimap.el in
>> Emacs 23.
>
> I don't recall, but people should notice relatively quickly if there are
> problems in this area (wrong display of non-ascii) so feel free to
> revert the patch or apply another patch instead.  It needs to be tested
> under Emacs 22 too, though, if it is installed in the Gnus CVS.
>
> /Simon

Message #315 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Frank Schmitt <ich <at> frank-schmitt.net>
To: Simon Josefsson <simon <at> josefsson.org>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Mon, 12 Jan 2009 12:03:09 +0100

Simon Josefsson <simon <at> josefsson.org> writes:

> I have recently upgraded my Gnus installation, and it included this fix.
> Now every e-mail I send has non-ASCII characters pre-fixed with \201.
> So the patch installed does not seem to be the right, or there is
> something else wrong with my configuration.  Can anyone else reproduce
> this?  I'm including '\201å\201ä\201ö' in this e-mail for debugging.
>
> I'll see if I can debug this further, and find the exact part of the
> patch that cause the problem.

I see the \201 in your mail but for me everything is send and displayed
perfectly since the patch was applied. Maybe the difference is that you
are using Emacs 22? äöüß

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Message #320 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Simon Josefsson <simon <at> josefsson.org>
To: Frank Schmitt <ich <at> frank-schmitt.net>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Mon, 12 Jan 2009 12:10:56 +0100

Frank Schmitt <ich <at> frank-schmitt.net> writes:

> Simon Josefsson <simon <at> josefsson.org> writes:
>
>> I have recently upgraded my Gnus installation, and it included this fix.
>> Now every e-mail I send has non-ASCII characters pre-fixed with \201.
>> So the patch installed does not seem to be the right, or there is
>> something else wrong with my configuration.  Can anyone else reproduce
>> this?  I'm including '\201å\201ä\201ö' in this e-mail for debugging.
>>
>> I'll see if I can debug this further, and find the exact part of the
>> patch that cause the problem.
>
> I see the \201 in your mail but for me everything is send and displayed
> perfectly since the patch was applied. Maybe the difference is that you
> are using Emacs 22? äöüß

I'm testing Emacs 23 now and everything works fine.  So it seems the
problem only occurs in Emacs 22.

However, it seems the problem happens on _sending_ which confuses me.
Maybe the old patch just worked around the problem, and made \201 in
incoming e-mail disappear?  However, I think I would have noticed this
before anyway, because all my posts to mailing lists and news also has
\201 before any non-ascii character now (with Emacs 22) but I don't
think that has been the case before applying this patch.

/Simon

Message #325 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: gzp <at> papp.hu (Gabor Z. Papp), Simon Josefsson <simon <at> josefsson.org>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, ding <at> gnus.org,
        1174 <at> debbugs.gnu.org, Frank Schmitt <ich <at> frank-schmitt.net>
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Wed, 14 Jan 2009 21:12:36 +0100

reopen 1174
quit
----- end of commands for control <at> emacsbugs, bcc-ed -----

[ Gabor, please don't drop the other recipients, especially the bug data
  base 1174 <at> debbugs.gnu.org ]

On Wed, Jan 14 2009, Gabor Z. Papp wrote:

> * Simon Josefsson <simon <at> josefsson.org>:
>
> | I'm testing Emacs 23 now and everything works fine.  So it seems the
> | problem only occurs in Emacs 22.
>
> Same problem here using Emacs 22.
>
> | However, it seems the problem happens on _sending_ which confuses
> | me.
>
> Exactly.
>
> Should I upgrade to Emacs 23 or downgrade gnus cvs to an earlier snapshot?

Neither one.  We need to find a solution that works in all supported
Emacs versions.

Stefan, could you please suggest a fix or give some advice how to
debug this problem?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #332 received at 1174 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Reiner Steib <Reiner.Steib <at> gmx.de>
Cc: gzp <at> papp.hu (Gabor Z. Papp), Simon Josefsson <simon <at> josefsson.org>,
        ding <at> gnus.org, 1174 <at> debbugs.gnu.org,
        Frank Schmitt <ich <at> frank-schmitt.net>
Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23
Date: Wed, 14 Jan 2009 17:08:53 -0500

> Stefan, could you please suggest a fix or give some advice how to
> debug this problem?

I don't know where the problem is, so I can't suggest a fix.
I guess what Simon should do is:
- make sure he can reliably reproduce the problem.
- change imap-demule to string-as-multibyte, make sure that removes
  the problem.
- then try and figure out why/where is imap-demule used in such a way as
  to cause problem.

I must say I can't understand why imap-demule would be called on the
sending side,


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.