GNU bug report logs - #1770
23.0.60; (message-check 'illegible-text ...) fails on eight-bit chars

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 1770 in the body.
You can then email your comments to 1770 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <reinersteib+gmane <at> imap.cc>
To: emacs-pretest-bug <at> gnu.org
Subject: 23.0.60; (message-check 'illegible-text ...) fails on eight-bit chars
Date: Fri, 02 Jan 2009 23:09:41 +0100

Package: emacs,gnus
Version: 23.0.60

When replying to an article with a bogus charset declaration
(e.g. charset="ISO 8859-15" produced by Knode; "ISO-8859-15" would be
correct. An example in gmane.test is
<873ag15s04.not-fsf <at> marauder.physik.uni-ulm.de>) with Gnus, the buffer
contains eight-bit-control characters.

When sending the reply, Gnus asks: "Use ASCII as charset? " (see
`mml-parse-1').

Option 1: Answer `y'.

Result: The reply is sent with charset=us-ascii, but it contains 8bit
        characters.


Option 2: Answer `n'.  The Gnus asks a second time.  Answer `n' again.

Result: The same as above.
        See <87wsddtn9r.fsf <at> marauder.physik.uni-ulm.de> in gmane.test


Expected behavior:

This following code from `message-fix-before-sending' should kick in:
(This is what happens in Emacs 22 with current Gnus CVS trunk,
i.e. the same Gnus code base as Emacs 23.)

  (message-check 'illegible-text
    (let (char found choice)
      (message-goto-body)
      (while (progn
	       (skip-chars-forward mm-7bit-chars)
	       (when (get-text-property (point) 'no-illegible-text)
		 ;; There is a signed or encrypted raw message part
		 ;; that is considered to be safe.
		 (goto-char (or (next-single-property-change
				 (point) 'no-illegible-text)
				(point-max))))
	       (setq char (char-after)))
	(when (or (< (mm-char-int char) 128)
		  (and (mm-multibyte-p)
		       (memq (char-charset char)
			     '(eight-bit-control eight-bit-graphic
						 control-1))
		       (not (get-text-property
			     (point) 'untranslated-utf-8))))
	  (message-overlay-put (message-make-overlay (point) (1+ (point)))
			       'face 'highlight)
	  (setq found t))
	(forward-char))
      (when found
	(setq choice
	      (gnus-multiple-choice
	       "Non-printable characters found.  Continue sending?"
	       `((?d "Remove non-printable characters and send")
		 (?r ,(format
		       "Replace non-printable characters with \"%s\" and send"
		       message-replacement-char))
		 (?i "Ignore non-printable characters and send")
		 (?e "Continue editing"))))
	(if (eq choice ?e)
	  (error "Non-printable characters"))
	(message-goto-body)
	(skip-chars-forward mm-7bit-chars)
	(while (not (eobp))
	  (when (let ((char (char-after)))
		  (or (< (mm-char-int char) 128)
		      (and (mm-multibyte-p)
			   ;; FIXME: Wrong for Emacs 23 (unicode) and for
			   ;; things like undecable utf-8.  Should at least
			   ;; use find-coding-systems-region.
			   (memq (char-charset char)
				 '(eight-bit-control eight-bit-graphic
						     control-1))
			   (not (get-text-property
				 (point) 'untranslated-utf-8)))))
	    (if (eq choice ?i)
		(message-kill-all-overlays)
	      (delete-char 1)
	      (when (eq choice ?r)
		(insert message-replacement-char))))
	  (forward-char)
	  (skip-chars-forward mm-7bit-chars)))))

In Emacs 23, (char-charset char) returns `eight-bit'.  Is adding
eight-bit next to eight-bit-graphic sufficient?  The comment (by Dave
Love, CC-ed if I got X-Debbugs-CC right) seems to suggest that there's
more to be done.

Bye, Reiner.



In GNU Emacs 23.0.60.1 (i686-pc-linux-gnu, GTK+ Version 2.12.9)
 of 2009-01-01 on primula
Windowing system distributor `The X.Org Foundation', version 11.0.10400090
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #10 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Reiner Steib <Reiner.Steib <at> gmx.de>
Cc: 1770 <at> debbugs.gnu.org, emacs-pretest-bug <at> gnu.org
Subject: Re: bug#1770: 23.0.60; (message-check 'illegible-text ...) fails on eight-bit chars
Date: Fri, 02 Jan 2009 22:23:14 -0500

> In Emacs 23, (char-charset char) returns `eight-bit'.  Is adding
> eight-bit next to eight-bit-graphic sufficient?

Not sure if it's sufficient, but it should help, yes.


        Stefan

Message #20 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Dave Love <d.love <at> liverpool.ac.uk>
To: Reiner Steib <Reiner.Steib <at> gmx.de>
Cc: "1770\@debbugs.gnu.org" <1770 <at> debbugs.gnu.org>,
        "emacs-pretest-bug\@gnu.org" <emacs-pretest-bug <at> gnu.org>
Subject: Re: bug#1770: 23.0.60; (message-check 'illegible-text ...) fails on eight-bit chars
Date: Wed, 07 Jan 2009 21:41:58 +0000

Reiner Steib <reinersteib+gmane <at> imap.cc> writes:

> In Emacs 23, (char-charset char) returns `eight-bit'.  Is adding
> eight-bit next to eight-bit-graphic sufficient?  The comment (by Dave
> Love, CC-ed if I got X-Debbugs-CC right) seems to suggest that there's
> more to be done.

You should ask handa about that and other Mule issues.  Experience shows
it's not helpful for me to explain.

There were various things like that I left unfixed for Mule 6 (for
various reasons) five years ago, or whenever it was.

By the way, `undecable' should be `undecodable' in the comment, which
may only apply in Emacs 21 -- I don't know.

I think there are various things wrong with
`message-fix-before-sending'.  The one I remember is it objecting to
stuff in non-text inline MIME parts, e.g. if you try to use
application/octet-stream for a Lisp backtrace.

Message #30 received at 1770 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Reiner Steib <Reiner.Steib <at> gmx.de>
To: Dave Love <d.love <at> liverpool.ac.uk>
Cc: Kenichi Handa <handa <at> m17n.org>, 1770 <at> debbugs.gnu.org
Subject: Re: bug#1770: 23.0.60; (message-check 'illegible-text ...) fails on eight-bit chars
Date: Thu, 08 Jan 2009 21:28:50 +0100

On Wed, Jan 07 2009, Dave Love wrote:

> Reiner Steib <reinersteib+gmane <at> imap.cc> writes:
>
>> In Emacs 23, (char-charset char) returns `eight-bit'.  Is adding
>> eight-bit next to eight-bit-graphic sufficient?  The comment (by Dave
>> Love, CC-ed if I got X-Debbugs-CC right) seems to suggest that there's
>> more to be done.
>
> You should ask handa about that and other Mule issues.  Experience shows
> it's not helpful for me to explain.

Cc-ed.

> There were various things like that I left unfixed for Mule 6 (for
> various reasons) five years ago, or whenever it was.
>
> By the way, `undecable' should be `undecodable' in the comment, 

Fixed.

> which may only apply in Emacs 21 -- I don't know.

Added:

			   ;; FIXME: Wrong for Emacs 23 (unicode) and for
			   ;; things like undecodable utf-8 (in Emacs 21?).
			   ;; Should at least use find-coding-systems-region.
			   ;; -- fx

> I think there are various things wrong with
> `message-fix-before-sending'.  The one I remember is it objecting to
> stuff in non-text inline MIME parts, e.g. if you try to use
> application/octet-stream for a Lisp backtrace.

You can simply say "ignore", can't you?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Message #35 received at 1770 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Kenichi Handa <handa <at> m17n.org>
To: Reiner Steib <Reiner.Steib <at> gmx.de>
Cc: d.love <at> liverpool.ac.uk, 1770 <at> debbugs.gnu.org
Subject: Re: bug#1770: 23.0.60; (message-check 'illegible-text ...) fails on eight-bit chars
Date: Fri, 16 Jan 2009 16:45:56 +0900

In article <871vvdee4d.fsf <at> marauder.physik.uni-ulm.de>, Reiner Steib <Reiner.Steib <at> gmx.de> writes:

> On Wed, Jan 07 2009, Dave Love wrote:
> > Reiner Steib <reinersteib+gmane <at> imap.cc> writes:
> >
>>> In Emacs 23, (char-charset char) returns `eight-bit'.  Is adding
>>> eight-bit next to eight-bit-graphic sufficient?  The comment (by Dave
>>> Love, CC-ed if I got X-Debbugs-CC right) seems to suggest that there's
>>> more to be done.
> >
> > You should ask handa about that and other Mule issues.  Experience shows
> > it's not helpful for me to explain.

> Cc-ed.

Yes.  For Emacs 23, adding eight-bit in the list is ok.
But, I think it is better to catch non-Unicode characters
(#x110000..#x3FFF7F) here too.  For Emacs 23 only, we can
have this simple code:

	(while (not (eobp))
	  (when (not (encode-char (char-after) 'unicode))
                ;; or simply (>= (char-after) #x110000)
	    (if (eq choice ?i)
		(message-kill-all-overlays)
	      (delete-char 1)
	      (when (eq choice ?r)
		(insert message-replacement-char))))
	  (forward-char)
	  (skip-chars-forward mm-7bit-chars))

> Added:

> 			   ;; FIXME: Wrong for Emacs 23 (unicode) and for
> 			   ;; things like undecodable utf-8 (in Emacs 21?).
> 			   ;; Should at least use find-coding-systems-region.
> 			   ;; -- fx

After filtering out those strange characters, how is a
coding system decided?  Is select-message-coding-system
used?

---
Kenichi Handa
handa <at> m17n.org

View this message in rfc822 format

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: d.love <at> liverpool.ac.uk, Reiner Steib <Reiner.Steib <at> gmx.de>, 1770 <at> debbugs.gnu.org
Subject: bug#1770: 23.0.60; (message-check 'illegible-text ...) fails on eight-bit chars
Date: Thu, 30 Sep 2010 19:48:12 +0200

Kenichi Handa <handa <at> m17n.org> writes:

> Yes.  For Emacs 23, adding eight-bit in the list is ok.
> But, I think it is better to catch non-Unicode characters
> (#x110000..#x3FFF7F) here too.  For Emacs 23 only, we can
> have this simple code:
>
> 	(while (not (eobp))
> 	  (when (not (encode-char (char-after) 'unicode))
>                 ;; or simply (>= (char-after) #x110000)
> 	    (if (eq choice ?i)
> 		(message-kill-all-overlays)
> 	      (delete-char 1)
> 	      (when (eq choice ?r)
> 		(insert message-replacement-char))))
> 	  (forward-char)
> 	  (skip-chars-forward mm-7bit-chars))

Was this installed?  If not, was a different fix applied, and the bug
not closed, or is this still a problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi <at> gnus.org * Lars Magne Ingebrigtsen

Message #42 received at 1770 <at> debbugs.gnu.org (full text, mbox):

From: Kenichi Handa <handa <at> m17n.org>
To: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: d.love <at> liverpool.ac.uk, Reiner.Steib <at> gmx.de, 1770 <at> debbugs.gnu.org
Subject: Re: bug#1770: 23.0.60; (message-check 'illegible-text ...) fails on
	eight-bit chars
Date: Thu, 14 Oct 2010 15:37:30 +0900

In article <m3tyl7gm1v.fsf <at> quimbies.gnus.org>, Lars Magne Ingebrigtsen <larsi <at> gnus.org> writes:

> Kenichi Handa <handa <at> m17n.org> writes:
> > Yes.  For Emacs 23, adding eight-bit in the list is ok.
> > But, I think it is better to catch non-Unicode characters
> > (#x110000..#x3FFF7F) here too.  For Emacs 23 only, we can
> > have this simple code:
> >
> > 	(while (not (eobp))
> > 	  (when (not (encode-char (char-after) 'unicode))
> >                 ;; or simply (>= (char-after) #x110000)
> > 	    (if (eq choice ?i)
> > 		(message-kill-all-overlays)
> > 	      (delete-char 1)
> > 	      (when (eq choice ?r)
> > 		(insert message-replacement-char))))
> > 	  (forward-char)
> > 	  (skip-chars-forward mm-7bit-chars))

> Was this installed?

No.

> If not, was a different fix applied, 

Yes.

2009-01-03  Reiner Steib  <Reiner.Steib <at> gmx.de>

	* message.el (message-fix-before-sending): Add `eight-bit' to
	illegible-text check.

> and the bug not closed, or is this still a problem?

As I wrote, non-Unicode characters are still not caught
here.  But I'm not sure it's problem to be solved by
message-fix-before-sending.  I have not yet got a reply to
this question.

> After filtering out those strange characters, how is a
> coding system decided?  Is select-message-coding-system
> used?

---
Kenichi Handa
handa <at> m17n.org

Message #45 received at 1770 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: Reiner.Steib <at> gmx.de, bugs <at> gnus.org, d.love <at> liverpool.ac.uk,
	1770 <at> debbugs.gnu.org
Subject: Re: bug#1770: 23.0.60; (message-check 'illegible-text ...) fails on
	eight-bit chars
Date: Thu, 14 Oct 2010 21:19:33 +0200

Kenichi Handa <handa <at> m17n.org> writes:

> As I wrote, non-Unicode characters are still not caught
> here.  But I'm not sure it's problem to be solved by
> message-fix-before-sending.  I have not yet got a reply to
> this question.

Removing characters that can't be encoded seems like a good idea.  If I
understand the problem correctly.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi <at> gnus.org * Lars Magne Ingebrigtsen

Message #50 received at 1770-close <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Kenichi Handa <handa <at> m17n.org>
Cc: 1770-close <at> debbugs.gnu.org, Reiner.Steib <at> gmx.de, d.love <at> liverpool.ac.uk
Subject: Re: bug#1770: 23.0.60;
	(message-check 'illegible-text ...) fails on eight-bit chars
Date: Sun, 23 Jan 2011 18:55:18 -0800

Kenichi Handa <handa <at> m17n.org> writes:

> As I wrote, non-Unicode characters are still not caught
> here.  But I'm not sure it's problem to be solved by
> message-fix-before-sending.  I have not yet got a reply to
> this question.

Ok.  Well, I think it may (or may not be) nice to warn users about
sending un-encodable bytes.  But they will most likely get a warning of
some kind, since there's probably other eight-bit-chars there, so I
think that's probably sufficient.  So I'm closing this report now,
unless anybody objects...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi <at> gnus.org * Lars Magne Ingebrigtsen

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.