GNU bug report logs - #3640
I can't see all characters in my files anymore

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 3640 in the body.
You can then email your comments to 3640 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: jidanni <at> jidanni.org
To: emacs-pretest-bug <at> gnu.org
Cc: rfrancoise <at> debian.org
Subject: I can't see all characters in my files anymore
Date: Sun, 21 Jun 2009 22:54:54 +0800

Help, I'm now shipping files to people with all kinds of embedded junk
in them like
|        character: ‎ (8206, #o20016, #x200e)
|preferred charset: unicode (Unicode (ISO10646))
|       code point: 0x200E
|           syntax: . 	which means: punctuation
|      buffer code: #xE2 #x80 #x8E
|        file code: #xE2 #x80 #x8E (encoded by coding system utf-8-unix)
|          display: by this font (glyph code)
|    xft:-unknown-FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x8F1)
|
|Character code properties: customize what to show
|  name: LEFT-TO-RIGHT MARK
|  general-category: Cf (Other, Format)
all because I can't see them anymore now that I use emacs23.
I'm like a kid using MicroSoft. I can't see exactly what is in my files
anymore. I can't find the switch to turn it back off.

Message #10 received at 3640 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Chong Yidong <cyd <at> stupidchicken.com>
To: jidanni <at> jidanni.org
Cc: 3640 <at> debbugs.gnu.org
Subject: Re: I can't see all characters in my files anymore
Date: Mon, 22 Jun 2009 15:57:06 -0400

> Help, I'm now shipping files to people with all kinds of embedded junk
> in them like
> |        character: ‎ (8206, #o20016, #x200e)
> |preferred charset: unicode (Unicode (ISO10646))
> |       code point: 0x200E
> |           syntax: . 	which means: punctuation
> |      buffer code: #xE2 #x80 #x8E
> |        file code: #xE2 #x80 #x8E (encoded by coding system utf-8-unix)
> |          display: by this font (glyph code)
> |    xft:-unknown-FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x8F1)
> |
> |Character code properties: customize what to show
> |  name: LEFT-TO-RIGHT MARK
> |  general-category: Cf (Other, Format)
> all because I can't see them anymore now that I use emacs23.

Your font claims to be able to display character code 8206.  So, Emacs
does not get in the way.  This is not a bug.

Message #17 received at 3640 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: jidanni <at> jidanni.org
To: cyd <at> stupidchicken.com
Cc: 3640 <at> debbugs.gnu.org
Subject: Re: I can't see all characters in my files anymore
Date: Tue, 23 Jun 2009 10:43:32 +0800

>>>>> "CY" == Chong Yidong <cyd <at> stupidchicken.com> writes:
>> Help, I'm now shipping files to people with all kinds of embedded junk
>> in them like
>> |        character: ‎ (8206, #o20016, #x200e)
>> |preferred charset: unicode (Unicode (ISO10646))
>> |       code point: 0x200E
>> |           syntax: . 	which means: punctuation
>> |      buffer code: #xE2 #x80 #x8E
>> |        file code: #xE2 #x80 #x8E (encoded by coding system utf-8-unix)
>> |          display: by this font (glyph code)
>> |    xft:-unknown-FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x8F1)
>> |
>> |Character code properties: customize what to show
>> |  name: LEFT-TO-RIGHT MARK
>> |  general-category: Cf (Other, Format)
>> all because I can't see them anymore now that I use emacs23.

CY> Your font claims to be able to display character code 8206.  So, Emacs
CY> does not get in the way.  This is not a bug.

Yes and I'm sure it does display it properly: the proper way is to make
it disappear, as it is a special direction changing character and not a
normal character.

This is where emacs needs to step in and give the user a way to be aware
his file contains such things.

Just like whitespace mode makes one aware one has whitespace.

I'll see if whitespace.el will fill my needs...

Message #22 received at 3640 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: jidanni <at> jidanni.org
To: cyd <at> stupidchicken.com
Cc: 3640 <at> debbugs.gnu.org, emacs-w3m <at> namazu.org
Subject: Re: I can't see all characters in my files anymore
Date: Tue, 23 Jun 2009 10:44:12 +0800

OK, I cooked up this workaround in .emacs:

;;Just to see if there are any zero width >‎<, U+200E LEFT-TO-RIGHT MARK
;;hiding in our files, we do:
(global-whitespace-mode 1)
(setq whitespace-style '(spaces))
;;Set neighbor characters on fire in order to see where the
;;"invisible man" is:
(setq whitespace-space-regexp "\\(.?\x200E+.?\\)")
(put 'whitespace-space-before-tab 'face-alias 'whitespace-space);fire color
;;Works except if it is the only char on a line. (In emacs -nw it is
;;width 1, not zero, for me.)

What a pain. But then to turn it on in emacs-w3m,
(add-hook 'w3m-mode-hook
	  (lambda ()
	    (whitespace-mode 1)))
But that kills all of emacs-w3m's font-locks.

Why can't there be a way to not allow any "invisible man" characters
from hiding, like it was in emacs22. Just make them show up as this
dotted box like in xterm, so we know something is funny there.
Is there some variable that can stop 0 width characters being smuggled
around under the users' noses unaware?

I mean isn't that what separates us from the MicroSoft users, they don't
know what is going on inside their files, but we do -- until now.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.