From unknown Sat Aug 16 11:12:15 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#9771 <9771@debbugs.gnu.org> To: bug#9771 <9771@debbugs.gnu.org> Subject: Status: 24.0.90; Redisplay problems with control characters Reply-To: bug#9771 <9771@debbugs.gnu.org> Date: Sat, 16 Aug 2025 18:12:15 +0000 retitle 9771 24.0.90; Redisplay problems with control characters reassign 9771 emacs submitter 9771 Johan Bockg=C3=A5rd severity 9771 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Sun Oct 16 18:25:37 2011 Received: (at submit) by debbugs.gnu.org; 16 Oct 2011 22:25:37 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFZ9E-0000lL-LP for submit@debbugs.gnu.org; Sun, 16 Oct 2011 18:25:37 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFZ9C-0000l8-01 for submit@debbugs.gnu.org; Sun, 16 Oct 2011 18:25:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RFZ8K-0002xa-7L for submit@debbugs.gnu.org; Sun, 16 Oct 2011 18:24:41 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:45342) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RFZ8K-0002xW-5u for submit@debbugs.gnu.org; Sun, 16 Oct 2011 18:24:40 -0400 Received: from eggs.gnu.org ([140.186.70.92]:46162) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RFZ8J-0005QH-1c for bug-gnu-emacs@gnu.org; Sun, 16 Oct 2011 18:24:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RFZ8H-0002xG-JB for bug-gnu-emacs@gnu.org; Sun, 16 Oct 2011 18:24:38 -0400 Received: from smtprelay-h22.telenor.se ([195.54.99.197]:58457) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RFZ8H-0002x4-9i for bug-gnu-emacs@gnu.org; Sun, 16 Oct 2011 18:24:37 -0400 Received: from iph3.telenor.se (iph3.telenor.se [195.54.127.134]) by smtprelay-h22.telenor.se (Postfix) with ESMTP id 495E6EA954 for ; Mon, 17 Oct 2011 00:24:31 +0200 (CEST) X-SMTPAUTH-B2: [bocjoh] X-SENDER-IP: [85.228.203.146] X-LISTENER: [smtp.bredband.net] X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlVyACZZm05V5MuSPGdsb2JhbABCiTyQSIxpgXQZAQEBATcyghwvXzQBBBgBDAqILpVynkaICASlUg X-IronPort-AV: E=Sophos;i="4.69,354,1315173600"; d="scan'208";a="131935982" Received: from c-92cbe455.04-211-6c6b701.cust.bredbandsbolaget.se (HELO muon.localdomain) ([85.228.203.146]) by iph3.telenor.se with ESMTP; 17 Oct 2011 00:24:31 +0200 Received: by muon.localdomain (Postfix, from userid 1000) id 342FE484673; Mon, 17 Oct 2011 00:24:30 +0200 (CEST) From: =?utf-8?Q?Johan_Bockg=C3=A5rd?= To: bug-gnu-emacs@gnu.org Subject: 24.0.90; Redisplay problems with control characters Date: Mon, 17 Oct 2011 00:24:30 +0200 Message-ID: <871uucr2w1.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.90 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -4.7 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.7 (----) There are several redisplay problems with control characters (GET_FROM_DISPLAY_VECTOR method). 1. Abort in move_it_by_lines (with bidi) emacs -Q C-u 2000 C-q 0 RET M-< M-> Fatal error (6)Aborted (core dumped) (gdb) bt #0 abort () at emacs.c:385 #1 0x0000000000451a48 in move_it_by_lines (it=0x7fffffffc620, dvpos=0) at xdisp.c:8907 #2 0x00000000004c0717 in Frecenter (arg=10) at window.c:5107 #3 0x000000000060ce90 in Ffuncall (nargs=, args=0x7fffffffd460) at eval.c:2974 [...] The following assertion fails: [move_it_by_lines] if (dvpos == 0) { /* DVPOS == 0 means move to the start of the screen line. */ move_it_vertically_backward (it, 0); xassert (it->current_x == 0 && it->hpos == 0); (gdb) p it->current_x $1 = 10 (gdb) p it->hpos $2 = 1 The non-zero values for it->hpos and it->current_x come from move_it_to in this piece of code: [move_it_vertically_backward] /* The above code moves us to some position NLINES down, usually to its first glyph (leftmost in an L2R line), but that's not necessarily the start of the line, under bidi reordering. We want to get to the character position that is immediately after the newline of the previous line. */ if (it->bidi_p && IT_CHARPOS (*it) > BEGV && FETCH_BYTE (IT_BYTEPOS (*it) - 1) != '\n') { EMACS_INT nl_pos = find_next_newline_no_quit (IT_CHARPOS (*it) - 1, -1); move_it_to (it, nl_pos, -1, -1, -1, MOVE_TO_POS); } 2. (Old) problem in BUFFER_POS_REACHED_P The position hpos = 1 above is not just non-zero; it's also in the middle of the ^@ control character (the screen line starts with ^). It's produced by move_it_in_display_line_to: #define BUFFER_POS_REACHED_P() \ [...] && (it->method == GET_FROM_BUFFER \ || (it->method == GET_FROM_DISPLAY_VECTOR \ && it->dpvec + it->current.dpvec_index + 1 >= it->dpend))) According to the condition above, the position in column 0 before the ^ glyph (dpvec_index = 0) is not a possible stop point, but the position between ^ and @ is. Cf. in_display_vector_p: /* Return 1 if IT points into the middle of a display vector. */ in_display_vector_p (struct it *it) { return (it->method == GET_FROM_DISPLAY_VECTOR && it->current.dpvec_index > 0 && it->dpvec + it->current.dpvec_index != it->dpend); } 3. Long lines with display vectors make Emacs really slow (with bidi) emacs -Q C-u 2000 C-q 0 RET M-< Type text... 4. Abort in push_display_prop emacs -Q (setq wrap-prefix "x") C-u 100 C-q 0 RET Fatal error (6)Aborted (core dumped) (gdb) bt #0 abort () at emacs.c:385 #1 0x0000000000477316 in push_display_prop (it=0x7fffffff8cd0, prop=24535585) at xdisp.c:18432 #2 0x0000000000477857 in handle_line_prefix (it=0x7fffffff8cd0) at xdisp.c:18536 #3 0x00000000004782d4 in display_line (it=0x7fffffff8cd0) at xdisp.c:18802 xassert (it->method == GET_FROM_BUFFER || it->method == GET_FROM_STRING); (gdb) p it->method $1 = GET_FROM_DISPLAY_VECTOR From debbugs-submit-bounces@debbugs.gnu.org Mon Oct 17 03:49:21 2011 Received: (at 9771) by debbugs.gnu.org; 17 Oct 2011 07:49:22 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFhwm-0006IB-Vz for submit@debbugs.gnu.org; Mon, 17 Oct 2011 03:49:21 -0400 Received: from fencepost.gnu.org ([140.186.70.10]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFhwl-0006I4-1B for 9771@debbugs.gnu.org; Mon, 17 Oct 2011 03:49:19 -0400 Received: from eliz by fencepost.gnu.org with local (Exim 4.71) (envelope-from ) id 1RFhvv-0002JH-Ql; Mon, 17 Oct 2011 03:48:27 -0400 Date: Mon, 17 Oct 2011 03:48:27 -0400 Message-Id: From: Eli Zaretskii To: Johan =?utf-8?Q?Bockg=C3=A5rd?= In-reply-to: <871uucr2w1.fsf@gnu.org> (message from Johan =?utf-8?Q?Bockg?= =?utf-8?Q?=C3=A5rd?= on Mon, 17 Oct 2011 00:24:30 +0200) Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters References: <871uucr2w1.fsf@gnu.org> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: 9771 Cc: 9771@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) > From: Johan Bockgård > Date: Mon, 17 Oct 2011 00:24:30 +0200 > > 2. (Old) problem in BUFFER_POS_REACHED_P > > The position hpos = 1 above is not just non-zero; it's also in the > middle of the ^@ control character (the screen line starts with ^). It's > produced by move_it_in_display_line_to: > > #define BUFFER_POS_REACHED_P() \ > [...] > && (it->method == GET_FROM_BUFFER \ > || (it->method == GET_FROM_DISPLAY_VECTOR \ > && it->dpvec + it->current.dpvec_index + 1 >= it->dpend))) > > According to the condition above, the position in column 0 before the > ^ glyph (dpvec_index = 0) is not a possible stop point, but the position > between ^ and @ is. Okay, but what is the practical problem with this? > 3. Long lines with display vectors make Emacs really slow (with bidi) It's not the display vectors in general that cause this. It's specifically the display of control characters. E.g., the following test case, which is a small variation of yours, shows no visible slowdown: emacs -Q M-: (require 'disp-table) RET M-: (aset standard-display-table ?x (vconcat "^A")) RET C-u 2000 x M-< Type text... I'll look into this, but is there some real-life use case behind your recipe? Binary nulls in a file generally cause Emacs to make the buffer unibyte, where bidi reordering is disabled. If some optimizations are in order to speed up redisplay in this situation, it would help to have a real-life use case handy, to make sure the optimizations really make a difference. The other parts of the report are more or less clear. Btw, in the future please make a separate bug report for each issue, it makes references to bugs in log messages more straightforward. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Mon Oct 17 10:19:37 2011 Received: (at 9771) by debbugs.gnu.org; 17 Oct 2011 14:19:37 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFo2T-0001Oa-81 for submit@debbugs.gnu.org; Mon, 17 Oct 2011 10:19:37 -0400 Received: from fencepost.gnu.org ([140.186.70.10]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFo2Q-0001OP-D6 for 9771@debbugs.gnu.org; Mon, 17 Oct 2011 10:19:35 -0400 Received: from eliz by fencepost.gnu.org with local (Exim 4.71) (envelope-from ) id 1RFo1a-00079e-Kw; Mon, 17 Oct 2011 10:18:42 -0400 Date: Mon, 17 Oct 2011 10:18:42 -0400 Message-Id: From: Eli Zaretskii To: bojohan@gnu.org, 9771@debbugs.gnu.org In-reply-to: (message from Eli Zaretskii on Mon, 17 Oct 2011 03:48:27 -0400) Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters References: <871uucr2w1.fsf@gnu.org> X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: 9771 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) > Date: Mon, 17 Oct 2011 03:48:27 -0400 > From: Eli Zaretskii > Cc: 9771@debbugs.gnu.org > > > 3. Long lines with display vectors make Emacs really slow (with bidi) > > It's not the display vectors in general that cause this. It's > specifically the display of control characters. The reason for the slowness is the bidirectional property of the control characters, and a peculiarity of the UBA, the Unicode Bidirectional Algorithms, regarding the processing of sequences of characters with that particular property. In the particular case in point, it causes Emacs to search to the end of the long line, when it is about to display the first null character. So now this question becomes very relevant: > is there some real-life use case behind your recipe? Depending on the answer, it may or may not be a good idea to look for some heuristics for this case before 24.1 is released, because AFAICS the current implementation of the UBA requirements is correct. From debbugs-submit-bounces@debbugs.gnu.org Mon Oct 17 13:07:12 2011 Received: (at 9771) by debbugs.gnu.org; 17 Oct 2011 17:07:12 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFqee-0006mN-4v for submit@debbugs.gnu.org; Mon, 17 Oct 2011 13:07:12 -0400 Received: from mtaout23.012.net.il ([80.179.55.175]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFqeb-0006mB-4m for 9771@debbugs.gnu.org; Mon, 17 Oct 2011 13:07:10 -0400 Received: from conversion-daemon.a-mtaout23.012.net.il by a-mtaout23.012.net.il (HyperSendmail v2007.08) id <0LT700F00ZFLJ900@a-mtaout23.012.net.il> for 9771@debbugs.gnu.org; Mon, 17 Oct 2011 19:06:05 +0200 (IST) Received: from HOME-C4E4A596F7 ([77.124.212.197]) by a-mtaout23.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LT700FB5ZI4D560@a-mtaout23.012.net.il>; Mon, 17 Oct 2011 19:06:05 +0200 (IST) Date: Mon, 17 Oct 2011 19:06:04 +0200 From: Eli Zaretskii Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters In-reply-to: <871uucr2w1.fsf@gnu.org> To: Johan =?iso-8859-1?Q?Bockg=E5rd?= Message-id: <838vojwnsz.fsf@gnu.org> MIME-version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: QUOTED-PRINTABLE X-012-Sender: halo1@inter.net.il References: <871uucr2w1.fsf@gnu.org> X-Spam-Score: -1.8 (-) X-Debbugs-Envelope-To: 9771 Cc: 9771@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.8 (-) > From: Johan Bockg=E5rd > Date: Mon, 17 Oct 2011 00:24:30 +0200 >=20 > 4. Abort in push_display_prop >=20 > emacs -Q >=20 > (setq wrap-prefix "x") >=20 > C-u 100 C-q 0 RET >=20 > Fatal error (6)Aborted (core dumped) This part should be fixed in revision 106102 on the trunk. From debbugs-submit-bounces@debbugs.gnu.org Mon Oct 17 15:18:34 2011 Received: (at 9771) by debbugs.gnu.org; 17 Oct 2011 19:18:34 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFshm-00025I-5z for submit@debbugs.gnu.org; Mon, 17 Oct 2011 15:18:34 -0400 Received: from mtaout23.012.net.il ([80.179.55.175]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFshj-000252-EJ for 9771@debbugs.gnu.org; Mon, 17 Oct 2011 15:18:32 -0400 Received: from conversion-daemon.a-mtaout23.012.net.il by a-mtaout23.012.net.il (HyperSendmail v2007.08) id <0LT800F0052NXO00@a-mtaout23.012.net.il> for 9771@debbugs.gnu.org; Mon, 17 Oct 2011 21:16:53 +0200 (IST) Received: from HOME-C4E4A596F7 ([77.124.212.197]) by a-mtaout23.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LT800FGE5JTWZ30@a-mtaout23.012.net.il>; Mon, 17 Oct 2011 21:16:53 +0200 (IST) Date: Mon, 17 Oct 2011 21:16:38 +0200 From: Eli Zaretskii Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters In-reply-to: <871uucr2w1.fsf@gnu.org> To: Johan =?iso-8859-1?Q?Bockg=E5rd?= Message-id: <831uubwhrd.fsf@gnu.org> MIME-version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: QUOTED-PRINTABLE X-012-Sender: halo1@inter.net.il References: <871uucr2w1.fsf@gnu.org> X-Spam-Score: -1.8 (-) X-Debbugs-Envelope-To: 9771 Cc: 9771@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.8 (-) > From: Johan Bockg=E5rd > Date: Mon, 17 Oct 2011 00:24:30 +0200 >=20 > 1. Abort in move_it_by_lines (with bidi) >=20 > emacs -Q > C-u 2000 C-q 0 RET > M-< > M-> >=20 > Fatal error (6)Aborted (core dumped) >=20 > (gdb) bt > #0 abort () at emacs.c:385 > #1 0x0000000000451a48 in move_it_by_lines (it=3D0x7fffffffc620, dv= pos=3D0) > at xdisp.c:8907 > #2 0x00000000004c0717 in Frecenter (arg=3D10) at window.c:5107 > #3 0x000000000060ce90 in Ffuncall (nargs=3D, args= =3D0x7fffffffd460) > at eval.c:2974 > [...] >=20 > The following assertion fails: >=20 > [move_it_by_lines] >=20 > if (dvpos =3D=3D 0) > { > /* DVPOS =3D=3D 0 means move to the start of the screen lin= e. */ > move_it_vertically_backward (it, 0); > xassert (it->current_x =3D=3D 0 && it->hpos =3D=3D 0); >=20 > (gdb) p it->current_x > $1 =3D 10 > (gdb) p it->hpos > $2 =3D 1 This part should be fixed in revision 106106 on the trunk. From debbugs-submit-bounces@debbugs.gnu.org Mon Oct 17 17:05:50 2011 Received: (at 9771) by debbugs.gnu.org; 17 Oct 2011 21:05:50 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFuNa-0005Gi-9p for submit@debbugs.gnu.org; Mon, 17 Oct 2011 17:05:50 -0400 Received: from smtprelay-b11.telenor.se ([62.127.194.20]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RFuNX-0005GV-Rq for 9771@debbugs.gnu.org; Mon, 17 Oct 2011 17:05:49 -0400 Received: from iph3.telenor.se (iph3.telenor.se [195.54.127.134]) by smtprelay-b11.telenor.se (Postfix) with ESMTP id 40D60EA299 for <9771@debbugs.gnu.org>; Mon, 17 Oct 2011 23:04:48 +0200 (CEST) X-SMTPAUTH-B2: [bocjoh] X-SENDER-IP: [85.228.206.157] X-LISTENER: [smtp.bredband.net] X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmNSAOGXnE5V5M6dPGdsb2JhbABDiTyfFxkBAQEBNzKBbgEBBAEnLyMFCwshJQ8BBBgBDAoaE4d/ArdBgyuEXQSlUA X-IronPort-AV: E=Sophos;i="4.69,361,1315173600"; d="scan'208";a="132380956" Received: from c-9dcee455.04-211-6c6b701.cust.bredbandsbolaget.se (HELO muon.localdomain) ([85.228.206.157]) by iph3.telenor.se with ESMTP; 17 Oct 2011 23:04:48 +0200 Received: by muon.localdomain (Postfix, from userid 1000) id 775E148433F; Mon, 17 Oct 2011 23:04:47 +0200 (CEST) From: =?utf-8?Q?Johan_Bockg=C3=A5rd?= To: Eli Zaretskii Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters References: <871uucr2w1.fsf@gnu.org> Date: Mon, 17 Oct 2011 23:04:46 +0200 In-Reply-To: (Eli Zaretskii's message of "Mon, 17 Oct 2011 03:48:27 -0400") Message-ID: <874nz79vo1.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.90 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.4 (--) X-Debbugs-Envelope-To: 9771 Cc: 9771@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.4 (--) Eli Zaretskii writes: >> According to the condition above, the position in column 0 before the >> ^ glyph (dpvec_index = 0) is not a possible stop point, but the position >> between ^ and @ is. > > Okay, but what is the practical problem with this? Nothing, really. But it doesn't seem correct. > Binary nulls in a file generally cause Emacs to make the buffer > unibyte, where bidi reordering is disabled. That doesn't seem to be working, then. If I do emacs -Q /usr/bin/emacs the resulting buffer is multibyte. Type M-> and Emacs hangs. (I waited 5 minutes and had to kill the process.) From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 18 08:57:35 2011 Received: (at 9771) by debbugs.gnu.org; 18 Oct 2011 12:57:35 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RG9Ed-0004e4-K1 for submit@debbugs.gnu.org; Tue, 18 Oct 2011 08:57:35 -0400 Received: from fencepost.gnu.org ([140.186.70.10] ident=Debian-exim) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RG9Eb-0004dx-Cr for 9771@debbugs.gnu.org; Tue, 18 Oct 2011 08:57:34 -0400 Received: from eliz by fencepost.gnu.org with local (Exim 4.71) (envelope-from ) id 1RG9De-0004Zm-5N; Tue, 18 Oct 2011 08:56:34 -0400 Date: Tue, 18 Oct 2011 08:56:34 -0400 Message-Id: From: Eli Zaretskii To: Johan =?utf-8?Q?Bockg=C3=A5rd?= , handa@m17n.org In-reply-to: <874nz79vo1.fsf@gnu.org> (message from Johan =?utf-8?Q?Bockg?= =?utf-8?Q?=C3=A5rd?= on Mon, 17 Oct 2011 23:04:46 +0200) Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters References: <871uucr2w1.fsf@gnu.org> <874nz79vo1.fsf@gnu.org> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: 9771 Cc: 9771@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) > From: Johan Bockgård > Cc: 9771@debbugs.gnu.org > Date: Mon, 17 Oct 2011 23:04:46 +0200 > > Eli Zaretskii writes: > > >> According to the condition above, the position in column 0 before the > >> ^ glyph (dpvec_index = 0) is not a possible stop point, but the position > >> between ^ and @ is. > > > > Okay, but what is the practical problem with this? > > Nothing, really. But it doesn't seem correct. I'm not sure. Maybe I'm missing something (move_it_in_display_line_to is one of the trickiest functions in the display engine), so please bear with me as I explain why I think the test is correct. You wrote: > #define BUFFER_POS_REACHED_P() \ > [...] > && (it->method == GET_FROM_BUFFER \ > || (it->method == GET_FROM_DISPLAY_VECTOR \ > && it->dpvec + it->current.dpvec_index + 1 >= it->dpend))) > > According to the condition above, the position in column 0 before the > ^ glyph (dpvec_index = 0) is not a possible stop point, but the position > between ^ and @ is. A display vector has its glyphs stored at indices 0..it->dpend-1. IOW, the last glyph's index is it->dpend-1. BUFFER_POS_REACHED_P is invoked always after a call to get_next_display_element, but before the call to set_iterator_to_next. The former just consumes the glyph at it->current.dpvec_index; the latter advances the index to the next glyph of the display vector. Therefore, after a call to get_next_display_element, the glyph at dpvec_index was already consumed, but the index was not yet advanced to the next position. Thus, when the index is at dpend-1, as the condition in BUFFER_POS_REACHED_P says, we have already consumed the last glyph. Consuming the last glyph of a display vector means that the next call to set_iterator_to_next will detect that the display vector is exhausted, and will advance to the next buffer position, the one after the position which we just passed by consuming all the glyphs from the display vector used to display the character at that position. Thus, "buffer position reached". Does this make sense? In practice, after correcting the bug that caused the assertion violation, I can no longer reproduce the situation where we stop in the middle of the ^@ character. If you can show me a recipe for winding up in the middle of a display vector under these or similar circumstances, I will have another look. > > Binary nulls in a file generally cause Emacs to make the buffer > > unibyte, where bidi reordering is disabled. > > That doesn't seem to be working, then. > > If I do > > emacs -Q /usr/bin/emacs > > the resulting buffer is multibyte. Indeed. However, buffer-file-coding-system is no-conversion, which is confusingly inconsistent with "C-x RET c no-conversion C-x C-f". Perhaps Handa-san could shed some light on this inconsistency. Anyway, this being so, I found a way to optimize a couple of expensive loops inside bidi.c for the important special case of a plain L2R text in the buffer. With these optimizations, the bidi redisplay of the entire window showing a buffer with 2000 binary nulls is only a few milliseconds slower than the unidirectional one. I will install those changes later today, after some more testing. > Type M-> and Emacs hangs. (I waited 5 minutes and had to kill the > process.) It takes 1.5 seconds now (with a 44MB emacs binary ;-) From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 18 12:59:48 2011 Received: (at 9771) by debbugs.gnu.org; 18 Oct 2011 16:59:48 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RGD11-0001x0-Vt for submit@debbugs.gnu.org; Tue, 18 Oct 2011 12:59:48 -0400 Received: from mtaout21.012.net.il ([80.179.55.169]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RGD0y-0001wn-Mg for 9771@debbugs.gnu.org; Tue, 18 Oct 2011 12:59:46 -0400 Received: from conversion-daemon.a-mtaout21.012.net.il by a-mtaout21.012.net.il (HyperSendmail v2007.08) id <0LT900C00TQQIP00@a-mtaout21.012.net.il> for 9771@debbugs.gnu.org; Tue, 18 Oct 2011 18:58:41 +0200 (IST) Received: from HOME-C4E4A596F7 ([77.124.212.197]) by a-mtaout21.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LT900CYETTSHH40@a-mtaout21.012.net.il>; Tue, 18 Oct 2011 18:58:41 +0200 (IST) Date: Tue, 18 Oct 2011 18:58:41 +0200 From: Eli Zaretskii Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters In-reply-to: X-012-Sender: halo1@inter.net.il To: bojohan@gnu.org Message-id: <83lisiutha.fsf@gnu.org> References: <871uucr2w1.fsf@gnu.org> <874nz79vo1.fsf@gnu.org> X-Spam-Score: -2.1 (--) X-Debbugs-Envelope-To: 9771 Cc: 9771@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.1 (--) > Date: Tue, 18 Oct 2011 08:56:34 -0400 > From: Eli Zaretskii > Cc: 9771@debbugs.gnu.org > > Anyway, this being so, I found a way to optimize a couple of expensive > loops inside bidi.c for the important special case of a plain L2R text > in the buffer. With these optimizations, the bidi redisplay of the > entire window showing a buffer with 2000 binary nulls is only a few > milliseconds slower than the unidirectional one. > > I will install those changes later today, after some more testing. Done (revision 106124 on the trunk). From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 18 21:07:19 2011 Received: (at 9771) by debbugs.gnu.org; 19 Oct 2011 01:07:19 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RGKco-0006g8-Hh for submit@debbugs.gnu.org; Tue, 18 Oct 2011 21:07:18 -0400 Received: from smtprelay-b12.telenor.se ([62.127.194.21]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RGKcl-0006ft-Pg for 9771@debbugs.gnu.org; Tue, 18 Oct 2011 21:07:16 -0400 Received: from iph3.telenor.se (iph3.telenor.se [195.54.127.134]) by smtprelay-b12.telenor.se (Postfix) with ESMTP id 92768EA359 for <9771@debbugs.gnu.org>; Wed, 19 Oct 2011 03:06:10 +0200 (CEST) X-SMTPAUTH-B2: [bocjoh] X-SENDER-IP: [85.228.202.48] X-LISTENER: [smtp.bredband.net] X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtNTAKghnk5V5MowPGdsb2JhbABEiTyfMxkBAQEBNzKBbgEBBAFWIwULCyEUEQ8BBBgBDAoaE4gAArV5iBsEpVU X-IronPort-AV: E=Sophos;i="4.69,368,1315173600"; d="scan'208";a="133002681" Received: from c-30cae455.04-211-6c6b701.cust.bredbandsbolaget.se (HELO muon.localdomain) ([85.228.202.48]) by iph3.telenor.se with ESMTP; 19 Oct 2011 03:06:10 +0200 Received: by muon.localdomain (Postfix, from userid 1000) id 733974841C3; Wed, 19 Oct 2011 03:06:09 +0200 (CEST) From: =?utf-8?Q?Johan_Bockg=C3=A5rd?= To: Eli Zaretskii Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters References: <871uucr2w1.fsf@gnu.org> <874nz79vo1.fsf@gnu.org> Date: Wed, 19 Oct 2011 03:06:09 +0200 In-Reply-To: (Eli Zaretskii's message of "Tue, 18 Oct 2011 08:56:34 -0400") Message-ID: <87ipnlzt6m.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.90 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.4 (--) X-Debbugs-Envelope-To: 9771 Cc: 9771@debbugs.gnu.org, handa@m17n.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.4 (--) Eli Zaretskii writes: > Consuming the last glyph of a display vector means that the next call > to set_iterator_to_next will detect that the display vector is > exhausted, and will advance to the next buffer position, the one after > the position which we just passed by consuming all the glyphs from the > display vector used to display the character at that position. Thus, > "buffer position reached". It seems that BUFFER_POS_REACHED_P at least misses the opportunity to stop immediately before *any* glyph in the display vector has been consumed, i.e. when dpvec_index == 0. (Or the problem is not with BUFFER_POS_REACHED_P but with how it's being used.) [...] > In practice, after correcting the bug that caused the assertion > violation, I can no longer reproduce the situation where we stop in > the middle of the ^@ character. If you can show me a recipe for > winding up in the middle of a display vector under these or similar > circumstances, I will have another look. emacs -Q C-u 2000 C-q 0 RET Notice that the lines visible in the window start with "@^". Press until the top of the buffer just scrolls into view. Notice that lines before point start with "^@", and lines after point start with "@^". Change BUFFER_POS_REACHED_P (or something) so that dpvec_index == 0 is an acceptable stop position and this problem doesn't happen. From debbugs-submit-bounces@debbugs.gnu.org Wed Oct 19 07:51:11 2011 Received: (at 9771-done) by debbugs.gnu.org; 19 Oct 2011 11:51:11 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RGUfs-0006h6-1N for submit@debbugs.gnu.org; Wed, 19 Oct 2011 07:51:10 -0400 Received: from mtaout20.012.net.il ([80.179.55.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RGUfo-0006gZ-Aj for 9771-done@debbugs.gnu.org; Wed, 19 Oct 2011 07:51:06 -0400 Received: from conversion-daemon.a-mtaout20.012.net.il by a-mtaout20.012.net.il (HyperSendmail v2007.08) id <0LTB00900A40S300@a-mtaout20.012.net.il> for 9771-done@debbugs.gnu.org; Wed, 19 Oct 2011 13:49:53 +0200 (IST) Received: from HOME-C4E4A596F7 ([77.124.212.197]) by a-mtaout20.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LTB008X0A74Z6B0@a-mtaout20.012.net.il>; Wed, 19 Oct 2011 13:49:53 +0200 (IST) Date: Wed, 19 Oct 2011 13:49:54 +0200 From: Eli Zaretskii Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters In-reply-to: <87ipnlzt6m.fsf@gnu.org> To: Johan =?iso-8859-1?Q?Bockg=E5rd?= Message-id: <831uu9urod.fsf@gnu.org> MIME-version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: QUOTED-PRINTABLE X-012-Sender: halo1@inter.net.il References: <871uucr2w1.fsf@gnu.org> <874nz79vo1.fsf@gnu.org> <87ipnlzt6m.fsf@gnu.org> X-Spam-Score: -2.1 (--) X-Debbugs-Envelope-To: 9771-done Cc: 9771-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.1 (--) > From: Johan Bockg=E5rd > Cc: handa@m17n.org, 9771@debbugs.gnu.org > Date: Wed, 19 Oct 2011 03:06:09 +0200 >=20 > emacs -Q >=20 > C-u 2000 C-q 0 RET >=20 > Notice that the lines visible in the window start with "@^". >=20 > Press until the top of the buffer just scrolls into view. >=20 > Notice that lines before point start with "^@", and lines after > point start with "@^". Thanks for the test case. > Change BUFFER_POS_REACHED_P (or something) so that dpvec_index = =3D=3D 0 is > an acceptable stop position and this problem doesn't happen. I'm not sure a change in BUFFER_POS_REACHED_P is the right fix for this particular problem. Even if it is, I'm reluctant to make in move_it_in_display_line_to a change that could have effect in many unrelated places, since this function is such a central piece of the display engine. I'd rather make a change that targets this specific problem, certainly while we are in a pretest. I think I found such a change. The problem is that when we start a new redisplay cycle of a window, we begin drawing the window's first line at the last glyph of the display vector used to display the firs= t character position visible in the window. I made a simple change tha= t should fix this (in revision 106131), please see if it resolves the issue. Another problem I found was that one of the redisplay optimizations would redraw too few lines of a window, when the last redrawn line ends in glyphs from a display vector. I fixed that in revision 106133. With all the 4 parts of the original report being addressed now, I'm closing this bug report. If there are any left-overs, please submit separate bug reports about them. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Thu Oct 20 08:45:00 2011 Received: (at 9771) by debbugs.gnu.org; 20 Oct 2011 12:45:00 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RGrzY-0001mM-GU for submit@debbugs.gnu.org; Thu, 20 Oct 2011 08:45:00 -0400 Received: from mtaout22.012.net.il ([80.179.55.172]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1RGrzV-0001mA-QP for 9771@debbugs.gnu.org; Thu, 20 Oct 2011 08:44:59 -0400 Received: from conversion-daemon.a-mtaout22.012.net.il by a-mtaout22.012.net.il (HyperSendmail v2007.08) id <0LTD00D006ZKHH00@a-mtaout22.012.net.il> for 9771@debbugs.gnu.org; Thu, 20 Oct 2011 14:43:43 +0200 (IST) Received: from HOME-C4E4A596F7 ([77.124.212.197]) by a-mtaout22.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LTD00CM07CUO5C0@a-mtaout22.012.net.il>; Thu, 20 Oct 2011 14:43:43 +0200 (IST) Date: Thu, 20 Oct 2011 14:43:45 +0200 From: Eli Zaretskii Subject: Re: bug#9771: 24.0.90; Redisplay problems with control characters In-reply-to: <83lisiutha.fsf@gnu.org> X-012-Sender: halo1@inter.net.il To: bojohan@gnu.org, 9771@debbugs.gnu.org Message-id: <834nz3u932.fsf@gnu.org> References: <871uucr2w1.fsf@gnu.org> <874nz79vo1.fsf@gnu.org> <83lisiutha.fsf@gnu.org> X-Spam-Score: -2.1 (--) X-Debbugs-Envelope-To: 9771 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.1 (--) > Date: Tue, 18 Oct 2011 18:58:41 +0200 > From: Eli Zaretskii > Cc: 9771@debbugs.gnu.org > > > Date: Tue, 18 Oct 2011 08:56:34 -0400 > > From: Eli Zaretskii > > Cc: 9771@debbugs.gnu.org > > > > Anyway, this being so, I found a way to optimize a couple of expensive > > loops inside bidi.c for the important special case of a plain L2R text > > in the buffer. With these optimizations, the bidi redisplay of the > > entire window showing a buffer with 2000 binary nulls is only a few > > milliseconds slower than the unidirectional one. > > > > I will install those changes later today, after some more testing. > > Done (revision 106124 on the trunk). For the record, revision 106151 speeds up the bidi display in a more general class of use cases. (Previously, appending some character, like `a', to the 2000 nulls in the original example would still slow down redisplay, even after the changes in r106124.) From unknown Sat Aug 16 11:12:15 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 18 Nov 2011 12:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator