GNU bug report logs - #13808
Gnus gratuiously messes with the encoding of attachments

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 13808 in the body.
You can then email your comments to 13808 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: submit <at> debbugs.gnu.org (The Gnus Bugfixing Girls + Boys)
Subject: Gnus gratuiously messes with the encoding of attachments
Date: Mon, 25 Feb 2013 00:25:40 +0100

[Message part 1 (text/plain, inline)]

I append an attachment with C-c C-m f that had been originally encoded
in UTF-8.  Gnus decides to reencode it in Latin-1 for whatever reason.
Attachments should _always_ be a faithful byte-by-byte rendition of the
original file.  Gnus should not tamper with that.  It is acutely
embarrassing if you send "example files" to people in order to let them
check whether they have encoding problems at their site, and the file
arrives with a different encoding.

I'd go so far to state that inline included files should keep the same
encoding.  But at any rate, reencoding an external attachment is plainly
ridiculous.

Gnus v5.13
GNU Emacs 24.3.50.1 (i686-pc-linux-gnu, X toolkit)
 of 2013-01-30 on lola
200 news.gmane.org InterNetNews NNRP server INN 2.5.1 ready (posting ok)
100 Legal commands
  ARTICLE [message-ID|number]
  AUTHINFO USER name|PASS password|GENERIC program [argument ...]
  BODY [message-ID|number]
  CAPABILITIES [keyword]
  DATE
  GROUP newsgroup
  HDR header [message-ID|range]
  HEAD [message-ID|number]
  HELP
  IHAVE message-ID
  LAST
  LIST [ACTIVE [wildmat]|ACTIVE.TIMES [wildmat]|DISTRIB.PATS|DISTRIBUTIONS|HEADERS [MSGID|RANGE]|MODERATORS|MOTD|NEWSGROUPS [wildmat]|OVERVIEW.FMT|SUBSCRIPTIONS]
  LISTGROUP [newsgroup [range]]
  MODE READER
  NEWGROUPS [yy]yymmdd hhmmss [GMT]
  NEWNEWS wildmat [yy]yymmdd hhmmss [GMT]
  NEXT
  OVER [range]
  POST
  QUIT
  STARTTLS
  STAT [message-ID|number]
  XGTITLE [wildmat]
  XHDR header [message-ID|range]
  XOVER [range]
  XPAT header message-ID|range pattern [pattern ...]
Report problems to <usenet <at> ger.gmane.org>.
.
382 Begin TLS negotiation now
100 Legal commands
  ARTICLE [message-ID|number]
  AUTHINFO USER name|PASS password|GENERIC program [argument ...]
  BODY [message-ID|number]
  CAPABILITIES [keyword]
  DATE
  GROUP newsgroup
  HDR header [message-ID|range]
  HEAD [message-ID|number]
  HELP
  IHAVE message-ID
  LAST
  LIST [ACTIVE [wildmat]|ACTIVE.TIMES [wildmat]|DISTRIB.PATS|DISTRIBUTIONS|HEADERS [MSGID|RANGE]|MODERATORS|MOTD|NEWSGROUPS [wildmat]|OVERVIEW.FMT|SUBSCRIPTIONS]
  LISTGROUP [newsgroup [range]]
  MODE READER
  NEWGROUPS [yy]yymmdd hhmmss [GMT]
  NEWNEWS wildmat [yy]yymmdd hhmmss [GMT]
  NEXT
  OVER [range]
  POST
  QUIT
  STARTTLS
  STAT [message-ID|number]
  XGTITLE [wildmat]
  XHDR header [message-ID|range]
  XOVER [range]
  XPAT header message-ID|range pattern [pattern ...]
Report problems to <usenet <at> ger.gmane.org>.
.

-- 
David Kastrup

[xxx.ly (text/x-lilypond, attachment)]

Message #8 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: David Kastrup <dak <at> gnu.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Sat, 06 Jul 2013 18:07:09 +0200

David Kastrup <dak <at> gnu.org> writes:

> I append an attachment with C-c C-m f that had been originally encoded
> in UTF-8.  Gnus decides to reencode it in Latin-1 for whatever reason.
> Attachments should _always_ be a faithful byte-by-byte rendition of the
> original file.

I think MUAs are free to alter the encoding of forwarded messages as
they choose.  Even MTAs can alter encodings according to their whims.

So I don't think it's a bug that Gnus re-encodes stuff, but in this
instance it sounds like it's re-encoding things wrongly.  Or do you have
a preference set somewhere that prefers Latin-1?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Message #11 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 13808 <at> gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Sat, 06 Jul 2013 18:21:59 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> David Kastrup <dak <at> gnu.org> writes:
>
>> I append an attachment with C-c C-m f that had been originally encoded
>> in UTF-8.  Gnus decides to reencode it in Latin-1 for whatever reason.
>> Attachments should _always_ be a faithful byte-by-byte rendition of the
>> original file.
>
> I think MUAs are free to alter the encoding of forwarded messages as
> they choose.

I am not talking about a "forwarded message" but rather an attached
file.  The message part included with C-c m f was specified as

(text/x-lilypond, attachment)

It was _not_ included "inline".  It was not a part of the message.

> Even MTAs can alter encodings according to their whims.

Not in attachments.

> So I don't think it's a bug that Gnus re-encodes stuff, but in this
> instance it sounds like it's re-encoding things wrongly.  Or do you
> have a preference set somewhere that prefers Latin-1?

This was a program file in a programming language that uses utf-8 as its
text encoding.  You can't just reencode the file in Latin-1 and expect
it to magically still work.

If we were talking about an _inline_ text, reencoding it in a different
encoding would already be hard to justify, but this was rather an
_attachment_.

If it is impossible to send attachments without them getting reencoded
unasked for, attaching files become _useless_.

-- 
David Kastrup

Message #14 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Mon, 08 Jul 2013 16:14:40 +0200

First a note and apology: due to a misconfiguration of my mail server,
mail to debbugs.gnu.org has instead been sent (and seen as being sent
to) to gnu.org instead.  As a consequence, a significant part of our
communication might never have made it to the bug tracker.

I apology for that and hope that my system's configuration will now be
fixed in that regard.

Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

> David Kastrup <dak <at> gnu.org> writes:
>
>> You are confusing the technical implementation with the intended
>> semantics.  An inline file is expected to be displayed in the message,
>> but indeed it _can_ be saved conveniently as a single file, so
>> reencoding it is not a good idea either.
>>
>> The only purpose of an attachment is to be saved as a file.  If this
>> file is not identical to the file that has been sent, the purpose is not
>> met.
>
> There is no technical difference between Content-Disposition: inline or
> attachment.  text/plain parts have to adhere to certain standards (be
> encoded with a valid charset etc) to be valid.
>
> If you want to just send a sequence of bytes, you have to send something
> other than text/*.

But we are not talking about "text/plain" here, but rather
"text/x-lilypond".  Gnus offers, among others, the content types

text/cache-manifest 	text/calendar
text/css 	text/csv
text/dns 	text/enriched
text/h323 	text/html
text/iuls 	text/mathml
text/plain 	text/richtext
text/scriptlet 	text/spreadsheet
text/tab-separated-values 	text/texmacs
text/vnd.sun.j2me.app-descriptor 	text/vnd.wap.wml
text/vnd.wap.wmlscript 	text/x-bibtex
text/x-boo 	text/x-c++hdr
text/x-c++src 	text/x-chdr
text/x-component 	text/x-csh
text/x-csrc 	text/x-diff
text/x-dsrc 	text/x-haskell
text/x-java 	text/x-lilypond
text/x-literate-haskell 	text/x-moc
text/x-org 	text/x-pascal
text/x-patch 	text/x-pcs-gcd
text/x-perl 	text/x-python
text/x-scala 	text/x-setext
text/x-sfv 	text/x-sh
text/x-tcl 	text/x-tex
text/x-vcalendar 	text/x-vcard

And when you use C-c m f for attaching a file of extension, say, .ly, it
will _default_ to using text/x-lilypond.  Of the above content types, it
would obviously be a really bad idea to silently reencode an attached
file to have file contents with a different encoding for the
overwhelming majority of the above file types as they either expect a
certain encoding _or_ declare the desired encoding in instructions in
the file that will not get changed by Gnus when it it changes the
original encoding.

>> A file that compiled fine at the site of the sender does no longer
>> compile when received.  Only ASCII characters survive unchanged.  That
>> makes an attachment not better for transfering program files than just
>> pasting them in the middle of the mail message would.
>
> That Gnus re-encoded a text/plain part that was utf-8 on disk to
> Latin-1 sounds like a bug, yes.

I would tend to agree here, but my complaint actually was that Gnus
reencoded a text/x-lilypond part that was utf-8 on disk.

> But that has nothing to do with whether the parts are attachments or
> not.

I think this behavior less defensible when attachments are involved, but
I'd be perfectly happy if it was fixed for both attachments as well as
inline parts.

> Unless you're looking at different standards than what I'm looking at.

Not likely.  I was just trying to create an example of least defensible
behavior by illustrating with an attachment of type text/x-lilypond.

I'd be perfectly content with preserving file encoding for inline
passages of text/plain: the fewer chances there are for things going
wrong or being surprising, the better.

Now part of the scenario for reproducing the problem might be the
following variable:


mm-coding-system-priorities is a variable defined in `mm-util.el'.
Its value is (iso-8859-1 iso-8859-15 utf-8)
Original value was nil

Documentation:
Preferred coding systems for encoding outgoing messages.

More than one suitable coding system may be found for some text.
By default, the coding system with the highest priority is used
to encode outgoing messages (see `sort-coding-systems').  If this
variable is set, it overrides the default priority.

You can customize this variable.

This variable was introduced, or its default value was changed, in
version 21.2 of Emacs.

[back]


I think that this variable should likely not be consulted at all when
attaching/sending a file.  When attaching/sending a _buffer_ that is
_not_ visiting a file, the choice might be more difficult to make.

But I'd consider it an "element of least surprise" strategy to restrict
the effect of mm-coding-system-priorities to the actual original text in
the message buffer.

-- 
David Kastrup

Message #17 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: David Kastrup <dak <at> gnu.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Thu, 30 Jan 2014 17:47:00 -0800

David Kastrup <dak <at> gnu.org> writes:

> I'd be perfectly content with preserving file encoding for inline
> passages of text/plain: the fewer chances there are for things going
> wrong or being surprising, the better.

Yeah.  Attaching files should send the files out the way they are on
disk, byte for byte, I think.

> Now part of the scenario for reproducing the problem might be the
> following variable:
>
> mm-coding-system-priorities is a variable defined in `mm-util.el'.
> Its value is (iso-8859-1 iso-8859-15 utf-8)
> Original value was nil

Mine is nil.

I just tried sending out a utf-8 file, and Message decided it was 8859,
so it's totally garbled:

--=-=-=
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline; filename=foo.txt
Content-Transfer-Encoding: base64

QmzDpWLDpnINCg==
--=-=-=

Yuck.

Hm...  what's determining the charset in the file, anyway?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Message #20 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: David Kastrup <dak <at> gnu.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Thu, 30 Jan 2014 17:55:21 -0800

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> I just tried sending out a utf-8 file, and Message decided it was 8859,
> so it's totally garbled:
>
> --=-=-=
> Content-Type: text/plain; charset=iso-8859-1
> Content-Disposition: inline; filename=foo.txt
> Content-Transfer-Encoding: base64
>
> QmzDpWLDpnINCg==
> --=-=-=
>
> Yuck.
>
> Hm...  what's determining the charset in the file, anyway?

It's calling `find-coding-systems-region', which just returns
`(undecided)'.  Er.  So how is this supposed to work, really?

How Emacs detects coding systems remains a mystery to me...  Anybody
have any insights here?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Message #23 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Fri, 31 Jan 2014 07:46:13 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
>> I just tried sending out a utf-8 file, and Message decided it was 8859,
>> so it's totally garbled:
>>
>> --=-=-=
>> Content-Type: text/plain; charset=iso-8859-1
>> Content-Disposition: inline; filename=foo.txt
>> Content-Transfer-Encoding: base64
>>
>> QmzDpWLDpnINCg==
>> --=-=-=
>>
>> Yuck.
>>
>> Hm...  what's determining the charset in the file, anyway?
>
> It's calling `find-coding-systems-region', which just returns
> `(undecided)'.  Er.  So how is this supposed to work, really?
>
> How Emacs detects coding systems remains a mystery to me...  Anybody
> have any insights here?

Texts don't have a coding system.  What Emacs can detect is what coding
systems an _external_ representation of a multibyte buffer or string
might take.  find-coding-systems-region just returns a list of proposals
all of which will equally work for representing the buffer.  "undecided"
means that there are no non-ASCII characters and _anything_ will work.

If find-coding-systems-region is called on a base64 encoded region, that
will always be its verdict.  That's pretty much the point of base64.

-- 
David Kastrup

Message #26 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: David Kastrup <dak <at> gnu.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Wed, 25 Jan 2017 16:03:00 +0100

The problem stems from mm-encode-body and mm-find-mime-charset-region
and (prefer-coding-system 'iso-8859-1).

The problem is that mm-encode-body inserts the byte from the text file
into a multibyte buffer, and then tries to determine what charset that
is, which is, of course, nonsensical.

That's a very involved piece of code, though, and I'm not quite sure
what the right fix here is...  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #29 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: David Kastrup <dak <at> gnu.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Wed, 25 Jan 2017 16:51:12 +0100

The mml encoding code is headache-inducing, probably because it was
written to support non-Mule and XEmacs and modern Emacs at the same
time.  So you have stuff like:

(defun mm-find-charset-region (b e)
  "Return a list of Emacs charsets in the region B to E."
  (cond
   ((mm-multibyte-p)

[...]

   (t
    ;; We are in a unibyte buffer, so we futz around a bit.

[...]

		      (car (last (assq mail-parse-charset
				       mm-mime-mule-charset-alist)))))
	    (list 'ascii (or charset 'latin-iso8859-1)))))))))

which is part of what's biting us here.

I wonder whether the best solution here is to just rewrite the mml code
completely.  There's so much...  magic in the various code paths.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #36 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: David Kastrup <dak <at> gnu.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Wed, 25 Jan 2017 16:58:05 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> The mml encoding code is headache-inducing, probably because it was
> written to support non-Mule and XEmacs and modern Emacs at the same
> time.  So you have stuff like:
>
> (defun mm-find-charset-region (b e)
>   "Return a list of Emacs charsets in the region B to E."
>   (cond
>    ((mm-multibyte-p)
>
> [...]
>
>    (t
>     ;; We are in a unibyte buffer, so we futz around a bit.
>
> [...]
>
> 		      (car (last (assq mail-parse-charset
> 				       mm-mime-mule-charset-alist)))))
> 	    (list 'ascii (or charset 'latin-iso8859-1)))))))))
>
> which is part of what's biting us here.
>
> I wonder whether the best solution here is to just rewrite the mml code
> completely.  There's so much...  magic in the various code paths.

And I think the answer is...  yes.  I'll start rewriting.  Shouldn't be
... that much code, should it?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #39 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: David Kastrup <dak <at> gnu.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Wed, 25 Jan 2017 17:26:12 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> And I think the answer is...  yes.  I'll start rewriting.  Shouldn't be
> ... that much code, should it?  :-)

Not a lot of rewriting was required, really.  *phew*  It now detect
coding systems reliably, and never rewrites file contents.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #46 received at 13808 <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 13808 <at> debbugs.gnu.org
Subject: Re: bug#13808: Gnus gratuiously messes with the encoding of
 attachments
Date: Wed, 25 Jan 2017 17:56:29 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
>> And I think the answer is...  yes.  I'll start rewriting.  Shouldn't be
>> ... that much code, should it?  :-)
>
> Not a lot of rewriting was required, really.  *phew* It now detect
> coding systems reliably, and never rewrites file contents.

That was quick after the initial delay.  Apparently the partial
handover/retirement of Gmane gave you an opportunity to catch up with
Gnus.

Thanks!

-- 
David Kastrup

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.