From eliz@gnu.org Fri Sep 12 03:20:02 2008 X-Spam-Checker-Version: SpamAssassin 3.2.3-bugs.debian.org_2005_01_02 (2007-08-08) on rzlab.ucr.edu X-Spam-Level: X-Spam-Status: No, score=-7.9 required=4.0 tests=BAYES_00,FOURLA, RCVD_IN_DNSWL_MED autolearn=ham version=3.2.3-bugs.debian.org_2005_01_02 Received: (at submit) by emacsbugs.donarmstrong.com; 12 Sep 2008 10:20:03 +0000 Received: from fencepost.gnu.org (fencepost.gnu.org [140.186.70.10]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id m8CAJxgX006119 for ; Fri, 12 Sep 2008 03:20:00 -0700 Received: from eliz by fencepost.gnu.org with local (Exim 4.67) (envelope-from ) id 1Ke5j7-0007ss-1e for emacs-pretest-bug@gnu.org; Fri, 12 Sep 2008 06:18:09 -0400 From: Eli Zaretskii To: emacs-pretest-bug@gnu.org Subject: 23.0.60; Non-ASCII display problems on a tty Message-Id: Sender: Eli Zaretskii Date: Fri, 12 Sep 2008 06:18:09 -0400 emacs -Q C-h H Type C-n several times, and you will see some very strange behavior: for example, some lines are skipped and point never enters them. Also, some non-ASCII characters are displayed incorrectly. For example, the "Bengali" line has only 1 "?" character in the parentheses following the language name, whereas 2 characters are displayed on a graphics display (I tried MS-Windows). On the same line, under "HELLO", there are 2 "?" characters instead of 4, and they are not aligned with the rest of greetings; moving point with C-f skips those "?"s and lands on what is displayed as space, but "C-x =" shows that there are non-ASCII characters in the buffer at those "blank" positions. Etc., etc., it looks like tty display of non-ASCII characters that cannot be displayed by the current terminal-coding-system is very much screwed up. Here's what "locale" reports, in case it's important: eliz@fencepost:~/emacs.cvs/emacs$ locale LANG= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= In GNU Emacs 23.0.60.63 (x86_64-unknown-linux-gnu, X toolkit) of 2008-09-12 on fencepost configured using `configure '--with-jpeg=no' '--with-png=no' '--with-gif=no' '--with-tiff=no'' Important settings: value of $LC_ALL: nil value of $LC_COLLATE: nil value of $LC_CTYPE: nil value of $LC_MESSAGES: nil value of $LC_MONETARY: nil value of $LC_NUMERIC: nil value of $LC_TIME: nil value of $LANG: nil value of $XMODIFIERS: nil locale-coding-system: nil default-enable-multibyte-characters: t Major mode: Fundamental Minor modes in effect: tooltip-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t global-auto-composition-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t line-number-mode: t transient-mark-mode: t view-mode: t Recent input: ESC [ > 0 ; 1 3 6 ; 0 c C-h H ESC O B ESC O B ESC O B ESC O B ESC O B ESC O B ESC O B ESC O B C-n C-n C-n C-n C-n C-n ESC x r e p o r t - e m a TAB TAB RET Recent messages: ("./src/emacs" "-Q") For information about GNU Emacs and the GNU system, type C-h C-a. Loading vc-cvs...done View mode: type C-h for help, h for commands, q to quit. From cyd@stupidchicken.com Thu Sep 18 11:28:16 2008 X-Spam-Checker-Version: SpamAssassin 3.2.3-bugs.debian.org_2005_01_02 (2007-08-08) on rzlab.ucr.edu X-Spam-Level: X-Spam-Status: No, score=-3.8 required=4.0 tests=AWL,BAYES_00 autolearn=ham version=3.2.3-bugs.debian.org_2005_01_02 Received: (at 970) by emacsbugs.donarmstrong.com; 18 Sep 2008 18:28:17 +0000 Received: from cyd.mit.edu (CYD.MIT.EDU [18.115.2.24]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id m8IISENr030230 for <970@emacsbugs.donarmstrong.com>; Thu, 18 Sep 2008 11:28:15 -0700 Received: by cyd.mit.edu (Postfix, from userid 1000) id 07F4957E33A; Thu, 18 Sep 2008 14:32:00 -0400 (EDT) From: Chong Yidong To: Eli Zaretskii Cc: 970@debbugs.gnu.org Subject: Re: 23.0.60; Non-ASCII display problems on a tty Date: Thu, 18 Sep 2008 14:32:00 -0400 Message-ID: <87vdwtz4nz.fsf@cyd.mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii > emacs -Q > C-h H > > Type C-n several times, and you will see some very strange behavior: > for example, some lines are skipped and point never enters them. I think Kenichi Handa's latest composition changes should have fixed this. Can you verify? From eliz@gnu.org Fri Sep 19 01:44:32 2008 X-Spam-Checker-Version: SpamAssassin 3.2.3-bugs.debian.org_2005_01_02 (2007-08-08) on rzlab.ucr.edu X-Spam-Level: X-Spam-Status: No, score=-6.0 required=4.0 tests=AWL,BAYES_00,HAS_BUG_NUMBER, RCVD_IN_DNSWL_LOW,RCVD_IN_NIX1 autolearn=ham version=3.2.3-bugs.debian.org_2005_01_02 Received: (at submit) by emacsbugs.donarmstrong.com; 19 Sep 2008 08:44:32 +0000 Received: from lists.gnu.org (lists.gnu.org [199.232.76.165]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id m8J8iTHt000318 for ; Fri, 19 Sep 2008 01:44:30 -0700 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1KgbbG-0001Uc-6R for bug-gnu-emacs@gnu.org; Fri, 19 Sep 2008 04:44:26 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1KgbbB-0001TZ-Iv for bug-gnu-emacs@gnu.org; Fri, 19 Sep 2008 04:44:22 -0400 Received: from [199.232.76.173] (port=49591 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1KgbbB-0001TI-2a for bug-gnu-emacs@gnu.org; Fri, 19 Sep 2008 04:44:21 -0400 Received: from mtaout3.012.net.il ([84.95.2.7]:62242) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1KgbbA-0001a3-My for bug-gnu-emacs@gnu.org; Fri, 19 Sep 2008 04:44:20 -0400 Received: from HOME-C4E4A596F7 ([77.127.116.246]) by i_mtaout3.012.net.il (HyperSendmail v2004.12) with ESMTPA id <0K7F00IG2PNG5YP0@i_mtaout3.012.net.il> for bug-gnu-emacs@gnu.org; Fri, 19 Sep 2008 11:45:16 +0300 (IDT) Date: Fri, 19 Sep 2008 11:44:19 +0300 From: Eli Zaretskii Subject: Re: bug#970: 23.0.60; Non-ASCII display problems on a tty In-reply-to: <87vdwtz4nz.fsf@cyd.mit.edu> X-012-Sender: halo1@inter.net.il To: Chong Yidong , 970@debbugs.gnu.org Cc: bug-gnu-emacs@gnu.org Reply-to: Eli Zaretskii Message-id: References: <87vdwtz4nz.fsf@cyd.mit.edu> X-detected-operating-system: by monty-python.gnu.org: Solaris 9.1 X-CrossAssassin-Score: 2 > From: Chong Yidong > Date: Thu, 18 Sep 2008 14:32:00 -0400 > Cc: 970@emacsbugs.donarmstrong.com > > > emacs -Q > > C-h H > > > > Type C-n several times, and you will see some very strange behavior: > > for example, some lines are skipped and point never enters them. > > I think Kenichi Handa's latest composition changes should have fixed > this. Can you verify? The ``some lines are skipped'' part is indeed solved. But the other problems mentioned in my bug report are still there. For example, compare the "South Asia" and "Bengali" lines with a graphics display: the number and screen position of the `?' question marks displayed on a tty instead of non-ASCII characters do not match those displayed on a graphics terminal. From eliz@gnu.org Sat Sep 27 07:48:54 2008 X-Spam-Checker-Version: SpamAssassin 3.2.3-bugs.debian.org_2005_01_02 (2007-08-08) on rzlab.ucr.edu X-Spam-Level: X-Spam-Status: No, score=-7.1 required=4.0 tests=AWL,BAYES_00,FOURLA, HAS_BUG_NUMBER,RCVD_IN_DNSWL_LOW autolearn=ham version=3.2.3-bugs.debian.org_2005_01_02 Received: (at submit) by emacsbugs.donarmstrong.com; 27 Sep 2008 14:48:55 +0000 Received: from lists.gnu.org (lists.gnu.org [199.232.76.165]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id m8REmnWo011295 for ; Sat, 27 Sep 2008 07:48:50 -0700 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Kjb6H-0000AY-AM for bug-gnu-emacs@gnu.org; Sat, 27 Sep 2008 10:48:49 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Kjb6F-0000AD-RJ for bug-gnu-emacs@gnu.org; Sat, 27 Sep 2008 10:48:48 -0400 Received: from [199.232.76.173] (port=55480 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Kjb6F-0000AA-JD for bug-gnu-emacs@gnu.org; Sat, 27 Sep 2008 10:48:47 -0400 Received: from mtaout5.012.net.il ([84.95.2.13]:46474) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1Kjb6F-0006VH-9z for bug-gnu-emacs@gnu.org; Sat, 27 Sep 2008 10:48:47 -0400 Received: from HOME-C4E4A596F7 ([77.127.170.116]) by i_mtaout5.012.net.il (HyperSendmail v2004.12) with ESMTPA id <0K7U004EGZTUTBV1@i_mtaout5.012.net.il> for bug-gnu-emacs@gnu.org; Sat, 27 Sep 2008 17:49:07 +0300 (IDT) Date: Sat, 27 Sep 2008 17:48:02 +0300 From: Eli Zaretskii Subject: Re: bug#970: 23.0.60; Non-ASCII display problems on a tty In-reply-to: <87vdwtz4nz.fsf@cyd.mit.edu> X-012-Sender: halo1@inter.net.il To: Kenichi Handa , 970@debbugs.gnu.org Cc: bug-gnu-emacs@gnu.org Reply-to: Eli Zaretskii Message-id: References: <87vdwtz4nz.fsf@cyd.mit.edu> X-detected-operating-system: by monty-python.gnu.org: Solaris 9.1 X-CrossAssassin-Score: 2 I have some more info about this bug. The below is based on displaying a file that is encoded in iso-2022-7bit-unix, and has a single line that is a copy of line 20 from etc/HELLO, which is the entry for the Bengali language. To produce this file, copy line 20 of HELLO, paste it into a new file, type "C-x RET f iso-2022-7bit-unix RET" and save the file. The display problems for this line are directly caused by the fact that tty_write_glyphs is called with its last argument len=22, which means the display engine expects 22 characters to be displayed. And tty_write_glyphs therefore moves cursor by 22 positions to account for that. However, encode_terminal_code returns a string whose length is only 13 characters, and the difference between 13 and 22 is the immediate cause for display problems: the displayed string looks as if it were padded by whitespace, but typing "C-x =" on these ``whitespace'' characters reveals that they are not spaces at all. Looking inside encode_terminal_code, I see that the problem is somehow related to composite characters. The first group of non-ASCII characters (in parentheses) are composite characters whose u.cmp.automatic flag is set. The Lisp object returned by composition_gstring_from_id for this group of characters is a Lisp vector: [[nil 2476 2494 2434 2482 2494] 0 [0 0 2476 2476 1 0 1 1 0 nil] [1 1 2494 2494 1 0 1 1 0 nil] [2 2 2434 2434 1 0 1 1 0 nil] [3 3 2482 2482 1 0 1 1 0 nil] [4 4 2494 2494 1 0 1 1 0 nil]] When this code: if (src->u.cmp.automatic) for (i = src->u.cmp.from; i < src->u.cmp.to; i++) { Lisp_Object g = LGSTRING_GLYPH (gstring, i); int c = LGLYPH_CHAR (g); if (! char_charset (c, charset_list, NULL)) break; buf += CHAR_STRING (c, buf); nchars++; } walks this Lisp vector, it immediately finds that the 1st character cannot be encoded by the current terminal's encoding, and breaks out of the loop. Then the `?' character gets stored in the buffer that is being prepared for encoding: if (i == 0) { /* The first character of the composition is not encodable. */ *buf++ = '?'; nchars++; } This is all as expected, but because of the "if (i == 0)" clause above, the `?' character gets stored only for the first character in this composition, whose codepoint is 2476. For other characters, the u.cmp.from value is greater than 0, so `?' is not stored for them. By contrast, on a graphics terminal, the 5 characters inside the parentheses are displayed as 2 visible glyphs, one (codepoint 2476) for buffer position 10, the other (codepoint 2482) for buffer position 13. Thus, I would expect to see two `?' question marks inside parentheses, not one. Similar problem happens with the second group of non-ASCII characters on this line, the characters that follow the TAB character. Here's the Lisp object returned by composition_gstring_from_id: [[nil 2472 2478 2488 2509 2453 2494 2480] 1 [0 0 2472 2472 1 0 1 1 0 nil] [1 1 2478 2478 1 0 1 1 0 nil] [2 3 2488 2488 1 0 1 1 0 nil] [2 3 2509 2509 0 0 0 1 0 nil] [4 4 2453 2453 1 0 1 1 0 nil] [5 5 2494 2494 1 0 1 1 0 nil] [6 6 2480 2480 1 0 1 1 0 nil]] (Note that in this case, there are elements in this vector whose FROM-IDX and TO-IDX values are not identical, and also the WIDTH value is zero for one of them.) This group of characters is displayed as 4 visible glyphs on a graphics terminal: respectively, for buffer positions 17 (code 2472), 18 (code 2478), 19 (code 2488), and 23 (2480). On a TTY, only one `?' is shown, again for the same reason as described above: the "if (i == 0)" test. My first suspicion would be that the object returned by composition_gstring_from_id gives incorrect data for FROM-IDX and TO-IDX, but I'm not sure I understood the composition machinery enough to draw a definitive conclusion. It is not even clear to me how do we want to display these characters: do we want the number of `?'s to be identical to the number of glyphs displayed by a graphics terminal, or do we want something else? Handa-san, can you please comment on these findings? From eliz@gnu.org Fri Feb 6 07:50:12 2009 Received: (at 970) by emacsbugs.donarmstrong.com; 6 Feb 2009 15:50:13 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5-bugs.debian.org_2005_01_02 (2008-06-10) on rzlab.ucr.edu X-Spam-Level: X-Spam-Bayes: score:0.5 Bayes not run. spammytokens:Tokens not available. hammytokens:Tokens not available. X-Spam-Status: No, score=0.0 required=4.0 tests=none autolearn=ham version=3.2.5-bugs.debian.org_2005_01_02 Received: from mtaout6.012.net.il (mtaout6.012.net.il [84.95.2.16]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id n16Fo4aE006632 for <970@emacsbugs.donarmstrong.com>; Fri, 6 Feb 2009 07:50:05 -0800 Received: from conversion-daemon.i-mtaout6.012.net.il by i-mtaout6.012.net.il (HyperSendmail v2007.08) id <0KEN00500IMEA100@i-mtaout6.012.net.il> for 970@emacsbugs.donarmstrong.com; Fri, 06 Feb 2009 17:50:03 +0200 (IST) Received: from HOME-C4E4A596F7 ([84.229.238.131]) by i-mtaout6.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0KEN001RCINDI220@i-mtaout6.012.net.il>; Fri, 06 Feb 2009 17:50:02 +0200 (IST) Date: Fri, 06 Feb 2009 17:49:38 +0200 From: Eli Zaretskii Subject: Re: Pretest next week In-reply-to: X-012-Sender: halo1@inter.net.il To: Kenichi Handa , 970@debbugs.gnu.org Cc: cyd@stupidchicken.com, emacs-devel@gnu.org Reply-to: Eli Zaretskii Message-id: References: <87y6x4ue2u.fsf@cyd.mit.edu> <87ocxqnnb6.fsf@cyd.mit.edu> > From: Kenichi Handa > CC: eliz@gnu.org, cyd@stupidchicken.com, emacs-devel@gnu.org > Date: Wed, 04 Feb 2009 11:49:19 +0900 > > > > Bug #970 is still not fixed, as of today's CVS. Is someone working on > > > it? I don't think we can release Emacs 23 with this problem. > > > I've just started to work on Bug #970. > > I've just installed fixes. Thank you, I confirm that most of the problems with compositions seem to be solved, at least in the HELLO file display. There are still a few strange phenomena with terminal display, although they seem unrelated to compositions. For example, after typing "C-h H", go to the line that begins with "CJK variety", and type "C-f": you will see that the cursor jumps past some of the characters inside parentheses. Is this a bug? From handa@m17n.org Mon Feb 9 16:44:48 2009 Received: (at 970) by emacsbugs.donarmstrong.com; 10 Feb 2009 00:44:48 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5-bugs.debian.org_2005_01_02 (2008-06-10) on rzlab.ucr.edu X-Spam-Level: * X-Spam-Bayes: score:0.5 Bayes not run. spammytokens:Tokens not available. hammytokens:Tokens not available. X-Spam-Status: No, score=1.3 required=4.0 tests=IMPRONONCABLE_1, MURPHY_WRONG_WORD1,MURPHY_WRONG_WORD2 autolearn=no version=3.2.5-bugs.debian.org_2005_01_02 Received: from mx1.aist.go.jp (mx1.aist.go.jp [150.29.246.133]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id n1A0ii1U020607 for <970@emacsbugs.donarmstrong.com>; Mon, 9 Feb 2009 16:44:46 -0800 Received: from rqsmtp1.aist.go.jp (rqsmtp1.aist.go.jp [150.29.254.115]) by mx1.aist.go.jp with ESMTP id n1A0ig9V002610; Tue, 10 Feb 2009 09:44:42 +0900 (JST) env-from (handa@m17n.org) Received: from smtp3.aist.go.jp by rqsmtp1.aist.go.jp with ESMTP id n1A0igxT029834; Tue, 10 Feb 2009 09:44:42 +0900 (JST) env-from (handa@m17n.org) Received: by smtp3.aist.go.jp with ESMTP id n1A0ifeq014375; Tue, 10 Feb 2009 09:44:41 +0900 (JST) env-from (handa@m17n.org) Received: from handa by etlken with local (Exim 4.69) (envelope-from ) id 1LWgjz-0004sI-0v; Tue, 10 Feb 2009 09:44:43 +0900 From: Kenichi Handa To: Eli Zaretskii CC: 970@debbugs.gnu.org, cyd@stupidchicken.com, emacs-devel@gnu.org In-reply-to: (message from Eli Zaretskii on Fri, 06 Feb 2009 17:49:38 +0200) Subject: Re: Pretest next week References: <87y6x4ue2u.fsf@cyd.mit.edu> <87ocxqnnb6.fsf@cyd.mit.edu> Message-Id: Date: Tue, 10 Feb 2009 09:44:43 +0900 In article , Eli Zaretskii writes: > There are still a few strange phenomena with terminal display, > although they seem unrelated to compositions. > For example, after typing "C-h H", go to the line that begins with > "CJK variety", and type "C-f": you will see that the cursor jumps past > some of the characters inside parentheses. Is this a bug? No. Those CJK characters have width 2, and if they are not supported by the terminal coding system, encode_terminal_code produces two '?'s. --- Kenichi Handa handa@m17n.org From cyd@stupidchicken.com Sun Mar 15 09:06:30 2009 Received: (at control) by emacsbugs.donarmstrong.com; 15 Mar 2009 16:06:30 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5-bugs.debian.org_2005_01_02 (2008-06-10) on rzlab.ucr.edu X-Spam-Level: X-Spam-Bayes: score:0.5 Bayes not run. spammytokens:Tokens not available. hammytokens:Tokens not available. X-Spam-Status: No, score=-2.0 required=4.0 tests=VALID_BTS_CONTROL autolearn=ham version=3.2.5-bugs.debian.org_2005_01_02 Received: from cyd.mit.edu (CYD.MIT.EDU [18.115.2.24]) by rzlab.ucr.edu (8.13.8/8.13.8/Debian-3) with ESMTP id n2FG6RBr005879 for ; Sun, 15 Mar 2009 09:06:29 -0700 Received: by cyd.mit.edu (Postfix, from userid 1000) id A311957E21C; Sun, 15 Mar 2009 12:07:46 -0400 (EDT) From: Chong Yidong To: control@debbugs.gnu.org Subject: tag bugs Date: Sun, 15 Mar 2009 12:07:46 -0400 Message-ID: <87d4cidb1p.fsf@cyd.mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii tags 2413 unreproducible, moreinfo tags 2434 unreproducible, moreinfo close 970 thanks From unknown Sat Aug 16 14:26:45 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: $requester Subject: Internal Control Message-Id: bug archived. Date: Mon, 13 Apr 2009 14:24:11 +0000 User-Agent: Fakemail v42.6.9 # A New Hope # A log time ago, in a galaxy far, far away # something happened. # # Magically this resulted in the following # action being taken, but this fake control # message doesn't tell you why it happened # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator