From unknown Sat Aug 16 23:50:05 2025 X-Loop: help-debbugs@gnu.org Subject: bug#71896: shell-resync-dirs hang Resent-From: Troy Hinckley Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 02 Jul 2024 05:17:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 71896 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: 71896@debbugs.gnu.org X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.171989739416433 (code B ref -1); Tue, 02 Jul 2024 05:17:04 +0000 Received: (at submit) by debbugs.gnu.org; 2 Jul 2024 05:16:34 +0000 Received: from localhost ([127.0.0.1]:35518 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sOVsX-0004Gt-0X for submit@debbugs.gnu.org; Tue, 02 Jul 2024 01:16:33 -0400 Received: from lists.gnu.org ([209.51.188.17]:42064) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sOTAc-0008BP-Fs for submit@debbugs.gnu.org; Mon, 01 Jul 2024 22:23:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sOTAZ-0002EC-BW for bug-gnu-emacs@gnu.org; Mon, 01 Jul 2024 22:23:01 -0400 Received: from sender4-op-o15.zoho.com ([136.143.188.15]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sOTAU-0005oe-Lp for bug-gnu-emacs@gnu.org; Mon, 01 Jul 2024 22:22:57 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1719886968; cv=none; d=zohomail.com; s=zohoarc; b=MlqksOvdRHBJzUKaxKrnzD2Dx85k/sRmYVRwGwWUhGhmC1NBTB6wXlryDxYyMBf+pXOd4jTkfWCkiN91E0tVkKKJnira1CuPKRmkyojS7Ay9vUfsIFCFZvpRi0Kp3Yy1QhjocK/QYGnvcEuk23rI52aoDH1tEy3towZDq2LitMA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1719886968; h=Content-Type:Date:Date:From:From:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=Z1pWzUqgNYF1OqotZ6NaxNjWDOYGh/pXgX+8Wah7kT4=; b=GH5jhtKLnskAnsxv7InxVL7IaM026NRrqbeZOzgjoGhAQcYqaSnwwtfHY8nNsI4+TYS9/rgDj7FsdpVzQDNXyNx+NWrmXSBpWdhAOt7MLeNVpV/vzkwDCUjGEe7ygilxyUdwSgaf/cmeInO0Qt6FzseXrtaF8H7hmKWi2NsiIVc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=dabrev.com; spf=pass smtp.mailfrom=troyhinckley@dabrev.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1719886968; s=zoho; d=dabrev.com; i=troyhinckley@dabrev.com; h=Date:Date:From:From:To:To:Message-ID:References:Subject:Subject:MIME-Version:Content-Type:Message-Id:Reply-To:Cc; bh=Z1pWzUqgNYF1OqotZ6NaxNjWDOYGh/pXgX+8Wah7kT4=; b=hiZkkkSkb5CvaCN6RVSa6gZhMrY/JGzJI1x8gg0lD5SbVrnGVLmxyZB4K9U3ggH3 /J1Ww7LmyOMRx8VxFYvUTiiPeARKP/IW8yOATxJ8F9ZKQ6RUpLLbBfo77AgUBAorMtU K1tS8inGIhCjoYc5YwLhP6pOS8K2d6zqO+H0QRPE= Received: by mx.zohomail.com with SMTPS id 171988696629677.4677691426291; Mon, 1 Jul 2024 19:22:46 -0700 (PDT) Date: Mon, 1 Jul 2024 21:22:35 -0500 From: Troy Hinckley Message-ID: <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> References: <03321175-1a92-4b82-a0cc-d6977b6a733a@Spark> X-Readdle-Message-ID: 83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="66836470_6eb5bd4_524" X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.15; envelope-from=troyhinckley@dabrev.com; helo=sender4-op-o15.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Mailman-Approved-At: Tue, 02 Jul 2024 01:16:24 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --66836470_6eb5bd4_524 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline This is related to =2359804 and =2354384. I will occasionally (about 30% of the time) see hangs when running shell-= resync-dirs in Emacs 29. I tracked this down to two issues: =46irst is in shell-eval-command. =46or Emacs 29 this function was added = to fix =2354384. It has this section in the code: =60=60=60 ;; Wait until we get a prompt (which will be a line without =C2=A0;; a newline). This is far from fool-proof -- if something =C2=A0;; outputs incomplete data and then sleeps, we'll think =C2=A0;; we've received the prompt. =C2=A0(while (not (let* ((lines (string-lines result)) =C2=A0(last (car (last lines)))) =C2=A0(and (length> lines 0) =C2=A0(not (equal last =22=22)) =C2=A0(or (not prev) =C2=A0(not (equal last prev))) =C2=A0(setq prev last)))) =C2=A0(accept-process-output proc 0 100)) =60=60=60 Note that is says that is looking for =E2=80=9Ca line without a newline=E2= =80=9D to determine if we have reached the prompt. However this code does= not actually do that. If the result ends in a newline it will still term= inate the loop and not wait for more input. We can see that by the fact t= hat the following code evaluates to nil. =60=60=60 (let ((result =22dirs=5Cn=22) prev) =C2=A0(not (let* ((lines (string-lines result)) =C2=A0(last (car (last lines)))) =C2=A0(and (length> lines 0) =C2=A0(not (equal last =22=22)) =C2=A0(or (not prev) =C2=A0(not (equal last prev))) =C2=A0(setq prev last))))) =60=60=60 I am not sure what this code is supposed to do, but the issue arrises if = the process output sends anything to this function it will terminate and = not wait for more input. In my case the issue is that the shell is echoin= g the command followed by the result (comint-process-echoes). About 30% o= f the time these two lines get sent as part of two different outputs. Mea= ning the second line (the directory for shell-resync-dirs) does not get c= aptured and instead gets printed to the terminal. This leads us to the hang. The issue is this code in shell-resync-dirs: =60=60=60 (while dlsl =C2=A0(let ((newelt =22=22) =C2=A0tem1 tem2) =C2=A0(while newelt =C2=A0;; We need tem1 because we don't want to prepend =C2=A0;; =60comint-file-name-prefix' repeatedly into newelt via tem2. =C2=A0(setq tem1 (pop dlsl) =C2=A0tem2 (concat comint-file-name-prefix tem1 newelt)) =C2=A0(cond ((file-directory-p tem2) =C2=A0(push tem2 ds) =C2=A0(when (string=3D =22 =22 (car dlsl)) =C2=A0(pop dlsl)) =C2=A0(setq newelt nil)) =C2=A0(t =C2=A0(setq newelt (concat tem1 newelt))))))) =60=60=60 This loop can only terminate if tem2 is a valid directory. Otherwise it w= ill take the default case in the cond and loop forever. And since the bug= in shell-eval-command does not provide the directory when the process ou= tput is split, we get a hang. I believe both of these need to be fixed to properly fix the bug. =46or the shell-eval-command I don=E2=80=99t understand what that loop is= trying to do now, so I am not sure how to fix it without breaking its fu= nctionality. I would just use (string-suffix-p =E2=80=9C=5Cn=E2=80=9D res= ult) to check if the output ends in a newline, but the code is obviously = trying to do something more complex there. If we fix that issue then it will resolve the hang in shell-resync-dirs, = but I think that is just glossing over the problem. That functions should= never hang, no matter what output it get=E2=80=99s from the shell. My re= commendation would be to add =60(and dlsl newelt)=60 as the condition for= the inner while loop with newelt. That way if dlsl is empty, it will ter= minate the loop since there is nothing more to process. This fixed the is= sue for me. -Troy Hinckley --66836470_6eb5bd4_524 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
This is related to =2359804 and =2354384.

I will occasionally (about 30% of the time) see hangs when running shell-= resync-dirs in Emacs 29. I tracked this down to two issues:


=46irst is in shell-eval-command. =46or Emacs 29 this function was added = to fix =2354384. It has this section in the code:

=60=60=60
;; Wait until we get a prompt (which will be a line without
&=23160;;; a newline). This is far from fool-proof -- if something
&=23160;;; outputs incomplete data and then sleeps, we'll think
&=23160;;; we've received the prompt.
&=23160;(while (not (let* ((lines (string-lines result))
&=23160;(last (car (last lines))))
&=23160;(and (length> lines 0)
&=23160;(not (equal last =22=22))
&=23160;(or (not prev)
&=23160;(not (equal last prev)))
&=23160;(setq prev last))))
&=23160;(accept-process-output proc 0 100))
=60=60=60

Note that is says that is looking for =E2=80=9Ca line without a newline=E2= =80=9D to determine if we have reached the prompt. However this code does= not actually do that. If the result ends in a newline it will still term= inate the loop and not wait for more input. We can see that by the fact t= hat the following code evaluates to nil.

=60=60=60
(let ((result =22dirs=5Cn=22) prev)
&=23160;(not (let* ((lines (string-lines result))
&=23160;(last (car (last lines))))
&=23160;(and (length> lines 0)
&=23160;(not (equal last =22=22))
&=23160;(or (not prev)
&=23160;(not (equal last prev)))
&=23160;(setq prev last)))))
=60=60=60

I am not sure what this code is supposed to do, but the issue arrises if = the process output sends anything to this function it will terminate and = not wait for more input. In my case the issue is that the shell is echoin= g the command followed by the result (comint-process-echoes). About 30% o= f the time these two lines get sent as part of two different outputs. Mea= ning the second line (the directory for shell-resync-dirs) does not get c= aptured and instead gets printed to the terminal.


This leads us to the hang. The issue is this code in shell-resync-dirs:
=60=60=60
(while dlsl
&=23160;(let ((newelt =22=22)
&=23160;tem1 tem2)
&=23160;(while newelt
&=23160;;; We need tem1 because we don't want to prepend
&=23160;;; =60comint-file-name-prefix' repeatedly into newelt via tem2. &=23160;(setq tem1 (pop dlsl)
&=23160;tem2 (concat comint-file-name-prefix tem1 newelt))
&=23160;(cond ((file-directory-p tem2)
&=23160;(push tem2 ds)
&=23160;(when (string=3D =22 =22 (car dlsl))
&=23160;(pop dlsl))
&=23160;(setq newelt nil))
&=23160;(t
&=23160;(setq newelt (concat tem1 newelt)))))))
=60=60=60

This loop can only terminate if tem2 is a valid directory. Otherwise it w= ill take the default case in the cond and loop forever. And since the bug= in shell-eval-command does not provide the directory when the process ou= tput is split, we get a hang.

I believe both of these need to be fixed to properly fix the bug.

=46or the shell-eval-command I don=E2=80=99t understand what that loop is= trying to do now, so I am not sure how to fix it without breaking its fu= nctionality. I would just use (string-suffix-p =E2=80=9C=5Cn=E2=80=9D res= ult) to check if the output ends in a newline, but the code is obviously = trying to do something more complex there.

If we fix that issue then it will resolve the hang in shell-resync-dirs, = but I think that is just glossing over the problem. That functions should= never hang, no matter what output it get=E2=80=99s from the shell. My re= commendation would be to add =60(and dlsl newelt)=60 as the condition for= the inner while loop with newelt. That way if dlsl is empty, it will ter= minate the loop since there is nothing more to process. This fixed the is= sue for me.&=23160;

-Troy Hinckley
--66836470_6eb5bd4_524-- From unknown Sat Aug 16 23:50:05 2025 X-Loop: help-debbugs@gnu.org Subject: bug#71896: shell-resync-dirs hang Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 06 Jul 2024 09:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 71896 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: Troy Hinckley Cc: 71896@debbugs.gnu.org Received: via spool by 71896-submit@debbugs.gnu.org id=B71896.17202598972649 (code B ref 71896); Sat, 06 Jul 2024 09:59:02 +0000 Received: (at 71896) by debbugs.gnu.org; 6 Jul 2024 09:58:17 +0000 Received: from localhost ([127.0.0.1]:45606 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sQ2BM-0000ge-CL for submit@debbugs.gnu.org; Sat, 06 Jul 2024 05:58:16 -0400 Received: from eggs.gnu.org ([209.51.188.92]:34378) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sQ2BK-0000gS-53 for 71896@debbugs.gnu.org; Sat, 06 Jul 2024 05:58:15 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sQ2BA-0003DE-OE; Sat, 06 Jul 2024 05:58:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=yDCvcz3q4gBMCMwdcAzsNcSqYPpDcXBBkl6oqXJZ4f4=; b=igKZrV3xoTzgL3TYmHpI 5M/eLuXxptM4o3K4ZKqGC1vPz8aOcc86K0GDwkzDZ7WGfY88ECf9rh8o/SBgMtwz5/yAAakQ5QbUK 3stpFI0MGAydz4rapH7X4LIQQiFKOlAmj8cN49yuq67ajUC/TG/03I8L2+yse8om450CIwQOg7gX8 s5HpT1Rimi3GjxQGqHmgXIpqaeSevAnJ9T4uRgDAMfS+LfL53KkIC3dmRy3Yd6x1jG+bn/qv2Ck+E btuRExNYvjARgH9FPfEHfOTn3YxtJd88PJHWrv3+ZMXeh+4aYzO/t0uKiHMlcxTQWiKtfdFCj4Nyk 2EyPAhT4vz7FjQ==; Date: Sat, 06 Jul 2024 12:58:00 +0300 Message-Id: <86tth24zbr.fsf@gnu.org> From: Eli Zaretskii In-Reply-To: <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> (message from Troy Hinckley on Mon, 1 Jul 2024 21:22:35 -0500) References: <03321175-1a92-4b82-a0cc-d6977b6a733a@Spark> <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > Date: Mon, 1 Jul 2024 21:22:35 -0500 > From: Troy Hinckley > > This is related to #59804 and #54384. > > I will occasionally (about 30% of the time) see hangs when running shell-resync-dirs in Emacs 29. I tracked this > down to two issues: > > First is in shell-eval-command. For Emacs 29 this function was added to fix #54384. It has this section in the > code: > > ``` > ;; Wait until we get a prompt (which will be a line without > ;; a newline). This is far from fool-proof -- if something > ;; outputs incomplete data and then sleeps, we'll think > ;; we've received the prompt. > (while (not (let* ((lines (string-lines result)) > (last (car (last lines)))) > (and (length> lines 0) > (not (equal last "")) > (or (not prev) > (not (equal last prev))) > (setq prev last)))) > (accept-process-output proc 0 100)) > ``` > > Note that is says that is looking for “a line without a newline” to determine if we have reached the prompt. > However this code does not actually do that. If the result ends in a newline it will still terminate the loop and not > wait for more input. We can see that by the fact that the following code evaluates to nil. > > ``` > (let ((result "dirs\n") prev) > (not (let* ((lines (string-lines result)) > (last (car (last lines)))) > (and (length> lines 0) > (not (equal last "")) > (or (not prev) > (not (equal last prev))) > (setq prev last))))) > ``` > > I am not sure what this code is supposed to do, but the issue arrises if the process output sends anything to > this function it will terminate and not wait for more input. In my case the issue is that the shell is echoing the > command followed by the result (comint-process-echoes). About 30% of the time these two lines get sent as > part of two different outputs. Meaning the second line (the directory for shell-resync-dirs) does not get captured > and instead gets printed to the terminal. Does the patch below solve the problem in shell-eval-command? > This leads us to the hang. The issue is this code in shell-resync-dirs: > > ``` > (while dlsl > (let ((newelt "") > tem1 tem2) > (while newelt > ;; We need tem1 because we don't want to prepend > ;; `comint-file-name-prefix' repeatedly into newelt via tem2. > (setq tem1 (pop dlsl) > tem2 (concat comint-file-name-prefix tem1 newelt)) > (cond ((file-directory-p tem2) > (push tem2 ds) > (when (string= " " (car dlsl)) > (pop dlsl)) > (setq newelt nil)) > (t > (setq newelt (concat tem1 newelt))))))) > ``` > > This loop can only terminate if tem2 is a valid directory. Otherwise it will take the default case in the cond and > loop forever. And since the bug in shell-eval-command does not provide the directory when the process output > is split, we get a hang. > > I believe both of these need to be fixed to properly fix the bug. > > For the shell-eval-command I don’t understand what that loop is trying to do now, so I am not sure how to fix it > without breaking its functionality. I would just use (string-suffix-p “\n” result) to check if the output ends in a > newline, but the code is obviously trying to do something more complex there. > > If we fix that issue then it will resolve the hang in shell-resync-dirs, but I think that is just glossing over the > problem. That functions should never hang, no matter what output it get’s from the shell. My recommendation > would be to add `(and dlsl newelt)` as the condition for the inner while loop with newelt. That way if dlsl is > empty, it will terminate the loop since there is nothing more to process. This fixed the issue for me. Thanks, I think I agree with your suggestion for shell-resync-dirs. But please undo the fix you evidently made there to avoid the infloop, and see if the patch below for shell-eval-command makes shell-resync-dirs do its job by correctly resynchronizing shell-dirtrack. diff --git a/lisp/shell.el b/lisp/shell.el index e1936ff..f86156e 100644 --- a/lisp/shell.el +++ b/lisp/shell.el @@ -1629,10 +1629,12 @@ shell-eval-command ;; a newline). This is far from fool-proof -- if something ;; outputs incomplete data and then sleeps, we'll think ;; we've received the prompt. - (while (not (let* ((lines (string-lines result)) - (last (car (last lines)))) + (while (not (let* ((lines (string-lines result nil t)) + (last (car (last lines))) + (last-end (substring last -1))) (and (length> lines 0) - (not (equal last "")) + (not (member last '("" "\n"))) + (not (equal last-end "\n")) (or (not prev) (not (equal last prev))) (setq prev last)))) From unknown Sat Aug 16 23:50:05 2025 X-Loop: help-debbugs@gnu.org Subject: bug#71896: shell-resync-dirs hang Resent-From: Troy Hinckley Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 08 Jul 2024 17:05:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 71896 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: Eli Zaretskii Cc: 71896@debbugs.gnu.org Received: via spool by 71896-submit@debbugs.gnu.org id=B71896.17204582818346 (code B ref 71896); Mon, 08 Jul 2024 17:05:01 +0000 Received: (at 71896) by debbugs.gnu.org; 8 Jul 2024 17:04:41 +0000 Received: from localhost ([127.0.0.1]:51392 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sQrn6-0002AX-Rm for submit@debbugs.gnu.org; Mon, 08 Jul 2024 13:04:41 -0400 Received: from sender4-op-o15.zoho.com ([136.143.188.15]:17559) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sQrn2-0002AL-MF for 71896@debbugs.gnu.org; Mon, 08 Jul 2024 13:04:40 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1720458257; cv=none; d=zohomail.com; s=zohoarc; b=Kjw6A35YWtfjg6QvOe475feXIaTJ8Rk1Lry9P9MuOpr5GkywDBTuDDa5EJEYFIJ/nky1JAm0N2Zk1d6K0/kyOAXj7OgfneUlXBfn6Lee2w6+2rGTieMu0bcC8y7D5CIg9Ifk0s/PTpXxtJJTHdOBAqP9g/zL7Fo8RiMuyzG9t7Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1720458257; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=i8p4EWXji2Nt/3bgTW9LjQ9ecD5AE9AJ9Uf8zs5T1lo=; b=MbFgrHTqBl5u9IcdbA6mrqU4kfg1aSd4JDY+euQnZR/ThA5t7KRC45zbnH6Wh5489ptdtC558tWbvYAvggu8d7XMAWoTr8yY0NlsN4QjYd/U9V1twNdf+JOg75ds5WVrfiyWYV1MST6RbNiJCfuhlTifzddtEimw1UaFAVnGdGE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=dabrev.com; spf=pass smtp.mailfrom=troyhinckley@dabrev.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1720458257; s=zoho; d=dabrev.com; i=troyhinckley@dabrev.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-ID:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Message-Id:Reply-To; bh=i8p4EWXji2Nt/3bgTW9LjQ9ecD5AE9AJ9Uf8zs5T1lo=; b=DY6W+BsGM1tKtu0SulqNtp3tOZAKf14XMd2CBlzv6MmBZxnVLpbDTc6Mj1GQcAmf R4Fu1sHOxLbPpw/2XULYM73GMCka7w8+LXxtd7cmvQKYfjDONVAqqTCPbwzs6oOSJbf uvkwVIIPC1VKUsiJNgwv9wVC07DdTPcUdm29dEHM= Received: by mx.zohomail.com with SMTPS id 1720458255824491.6835186396339; Mon, 8 Jul 2024 10:04:15 -0700 (PDT) Date: Mon, 8 Jul 2024 12:04:07 -0500 From: Troy Hinckley Message-ID: In-Reply-To: <86tth24zbr.fsf@gnu.org> References: <03321175-1a92-4b82-a0cc-d6977b6a733a@Spark> <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> <86tth24zbr.fsf@gnu.org> X-Readdle-Message-ID: ce5f9303-53dd-4f3e-a279-55a3ba4adacd@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="668c1c0c_7545e146_24f" X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --668c1c0c_7545e146_24f Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Thanks Eli, The proposed patch throws an error when =60last=E2=80=99 is =E2=80=9C=E2=80= =9D because (substring =E2=80=9C=E2=80=9D -1) produces (args-out-of-range= =E2=80=9C=E2=80=9D -1 nil). -Troy Hinckley On Jul 6, 2024 at 4:58=E2=80=AFAM -0500, Eli Zaretskii , = wrote: > > Date: Mon, 1 Jul 2024 21:22:35 -0500 > > =46rom: Troy Hinckley > > > > This is related to =2359804 and =2354384. > > > > I will occasionally (about 30% of the time) see hangs when running sh= ell-resync-dirs in Emacs 29. I tracked this > > down to two issues: > > > > =46irst is in shell-eval-command. =46or Emacs 29 this function was ad= ded to fix =2354384. It has this section in the > > code: > > > > =60=60=60 > > ;; Wait until we get a prompt (which will be a line without > > ;; a newline). This is far from fool-proof -- if something > > ;; outputs incomplete data and then sleeps, we'll think > > ;; we've received the prompt. > > (while (not (let* ((lines (string-lines result)) > > (last (car (last lines)))) > > (and (length> lines 0) > > (not (equal last =22=22)) > > (or (not prev) > > (not (equal last prev))) > > (setq prev last)))) > > (accept-process-output proc 0 100)) > > =60=60=60 > > > > Note that is says that is looking for =E2=80=9Ca line without a newli= ne=E2=80=9D to determine if we have reached the prompt. > > However this code does not actually do that. If the result ends in a = newline it will still terminate the loop and not > > wait for more input. We can see that by the fact that the following c= ode evaluates to nil. > > > > =60=60=60 > > (let ((result =22dirs=5Cn=22) prev) > > (not (let* ((lines (string-lines result)) > > (last (car (last lines)))) > > (and (length> lines 0) > > (not (equal last =22=22)) > > (or (not prev) > > (not (equal last prev))) > > (setq prev last))))) > > =60=60=60 > > > > I am not sure what this code is supposed to do, but the issue arrises= if the process output sends anything to > > this function it will terminate and not wait for more input. In my ca= se the issue is that the shell is echoing the > > command followed by the result (comint-process-echoes). About 30% of = the time these two lines get sent as > > part of two different outputs. Meaning the second line (the directory= for shell-resync-dirs) does not get captured > > and instead gets printed to the terminal. > > Does the patch below solve the problem in shell-eval-command=3F > > > This leads us to the hang. The issue is this code in shell-resync-dir= s: > > > > =60=60=60 > > (while dlsl > > (let ((newelt =22=22) > > tem1 tem2) > > (while newelt > > ;; We need tem1 because we don't want to prepend > > ;; =60comint-file-name-prefix' repeatedly into newelt via tem2. > > (setq tem1 (pop dlsl) > > tem2 (concat comint-file-name-prefix tem1 newelt)) > > (cond ((file-directory-p tem2) > > (push tem2 ds) > > (when (string=3D =22 =22 (car dlsl)) > > (pop dlsl)) > > (setq newelt nil)) > > (t > > (setq newelt (concat tem1 newelt))))))) > > =60=60=60 > > > > This loop can only terminate if tem2 is a valid directory. Otherwise = it will take the default case in the cond and > > loop forever. And since the bug in shell-eval-command does not provid= e the directory when the process output > > is split, we get a hang. > > > > I believe both of these need to be fixed to properly fix the bug. > > > > =46or the shell-eval-command I don=E2=80=99t understand what that loo= p is trying to do now, so I am not sure how to fix it > > without breaking its functionality. I would just use (string-suffix-p= =E2=80=9C=5Cn=E2=80=9D result) to check if the output ends in a > > newline, but the code is obviously trying to do something more comple= x there. > > > > If we fix that issue then it will resolve the hang in shell-resync-di= rs, but I think that is just glossing over the > > problem. That functions should never hang, no matter what output it g= et=E2=80=99s from the shell. My recommendation > > would be to add =60(and dlsl newelt)=60 as the condition for the inne= r while loop with newelt. That way if dlsl is > > empty, it will terminate the loop since there is nothing more to proc= ess. This fixed the issue for me. > > Thanks, I think I agree with your suggestion for shell-resync-dirs. > But please undo the fix you evidently made there to avoid the infloop, > and see if the patch below for shell-eval-command makes > shell-resync-dirs do its job by correctly resynchronizing > shell-dirtrack. > > diff --git a/lisp/shell.el b/lisp/shell.el > index e1936ff..f86156e 100644 > --- a/lisp/shell.el > +++ b/lisp/shell.el > =40=40 -1629,10 +1629,12 =40=40 shell-eval-command > ;; a newline). This is far from fool-proof -- if something > ;; outputs incomplete data and then sleeps, we'll think > ;; we've received the prompt. > - (while (not (let* ((lines (string-lines result)) > - (last (car (last lines)))) > + (while (not (let* ((lines (string-lines result nil t)) > + (last (car (last lines))) > + (last-end (substring last -1))) > (and (length> lines 0) > - (not (equal last =22=22)) > + (not (member last '(=22=22 =22=5Cn=22))) > + (not (equal last-end =22=5Cn=22)) > (or (not prev) > (not (equal last prev))) > (setq prev last)))) --668c1c0c_7545e146_24f Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Thanks Eli,
The proposed patch throws an error when =60last=E2=80=99 is =E2=80=9C=E2=80= =9D because (substring =E2=80=9C=E2=80=9D -1) produces (args-out-of-range= =E2=80=9C=E2=80=9D -1 nil).

-Troy Hinckley
On Jul 6, 2024 at 4:58=E2=80=AFAM -= 0500, Eli Zaretskii <eliz=40gnu.org>, wrote:
Date: Mon, 1 Jul 2024 21:22:35 -0500
=46rom: Troy Hinckley <troyhinckley=40dabrev.com>

This is related to =2359804 and =2354384.

I will occasionally (about 30% of the time) see hangs when running shell-= resync-dirs in Emacs 29. I tracked this
down to two issues:

=46irst is in shell-eval-command. =46or Emacs 29 this function was added = to fix =2354384. It has this section in the
code:

=60=60=60
;; Wait until we get a prompt (which will be a line without
;; a newline). This is far from fool-proof -- if something
;; outputs incomplete data and then sleeps, we'll think
;; we've received the prompt.
(while (not (let* ((lines (string-lines result))
(last (car (last lines))))
(and (length> lines 0)
(not (equal last =22=22))
(or (not prev)
(not (equal last prev)))
(setq prev last))))
(accept-process-output proc 0 100))
=60=60=60

Note that is says that is looking for =E2=80=9Ca line without a newline=E2= =80=9D to determine if we have reached the prompt.
However this code does not actually do that. If the result ends in a newl= ine it will still terminate the loop and not
wait for more input. We can see that by the fact that the following code = evaluates to nil.

=60=60=60
(let ((result =22dirs=5Cn=22) prev)
(not (let* ((lines (string-lines result))
(last (car (last lines))))
(and (length> lines 0)
(not (equal last =22=22))
(or (not prev)
(not (equal last prev)))
(setq prev last)))))
=60=60=60

I am not sure what this code is supposed to do, but the issue arrises if = the process output sends anything to
this function it will terminate and not wait for more input. In my case t= he issue is that the shell is echoing the
command followed by the result (comint-process-echoes). About 30% of the = time these two lines get sent as
part of two different outputs. Meaning the second line (the directory for= shell-resync-dirs) does not get captured
and instead gets printed to the terminal.

Does the patch below solve the problem in shell-eval-command=3F

This leads us to the hang. The issue is thi= s code in shell-resync-dirs:

=60=60=60
(while dlsl
(let ((newelt =22=22)
tem1 tem2)
(while newelt
;; We need tem1 because we don't want to prepend
;; =60comint-file-name-prefix' repeatedly into newelt via tem2.
(setq tem1 (pop dlsl)
tem2 (concat comint-file-name-prefix tem1 newelt))
(cond ((file-directory-p tem2)
(push tem2 ds)
(when (string=3D =22 =22 (car dlsl))
(pop dlsl))
(setq newelt nil))
(t
(setq newelt (concat tem1 newelt)))))))
=60=60=60

This loop can only terminate if tem2 is a valid directory. Otherwise it w= ill take the default case in the cond and
loop forever. And since the bug in shell-eval-command does not provide th= e directory when the process output
is split, we get a hang.

I believe both of these need to be fixed to properly fix the bug.

=46or the shell-eval-command I don=E2=80=99t understand what that loop is= trying to do now, so I am not sure how to fix it
without breaking its functionality. I would just use (string-suffix-p =E2= =80=9C=5Cn=E2=80=9D result) to check if the output ends in a
newline, but the code is obviously trying to do something more complex th= ere.

If we fix that issue then it will resolve the hang in shell-resync-dirs, = but I think that is just glossing over the
problem. That functions should never hang, no matter what output it get=E2= =80=99s from the shell. My recommendation
would be to add =60(and dlsl newelt)=60 as the condition for the inner wh= ile loop with newelt. That way if dlsl is
empty, it will terminate the loop since there is nothing more to process.= This fixed the issue for me.

Thanks, I think I agree with your suggestion for shell-resync-dirs.
= But please undo the fix you evidently made there to avoid the infloop, and see if the patch below for shell-eval-command makes
shell-resync-dirs do its job by correctly resynchronizing
shell-dirtrack.

diff --git a/lisp/shell.el b/lisp/shell.el
index e1936ff..f86156e 100644
--- a/lisp/shell.el
+++ b/lisp/shell.el
=40=40 -1629,10 +1629,12 =40=40 shell-eval-command
;; a newline). This is far from fool-proof -- if something
;; outputs incomplete data and then sleeps, we'll think
;; we've received the prompt.
- (while (not (let* ((lines (string-lines result))
- (last (car (last lines))))
+ (while (not (let* ((lines (string-lines result nil t))
+ (last (car (last lines)))
+ (last-end (substring last -1)))
(and (length> lines 0)
- (not (equal last =22=22))
+ (not (member last '(=22=22 =22=5Cn=22)))
+ (not (equal last-end =22=5Cn=22))
(or (not prev)
(not (equal last prev)))
(setq prev last))))
--668c1c0c_7545e146_24f-- From unknown Sat Aug 16 23:50:05 2025 X-Loop: help-debbugs@gnu.org Subject: bug#71896: shell-resync-dirs hang Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 08 Jul 2024 17:46:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 71896 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: Troy Hinckley Cc: 71896@debbugs.gnu.org Received: via spool by 71896-submit@debbugs.gnu.org id=B71896.172046071112299 (code B ref 71896); Mon, 08 Jul 2024 17:46:02 +0000 Received: (at 71896) by debbugs.gnu.org; 8 Jul 2024 17:45:11 +0000 Received: from localhost ([127.0.0.1]:51445 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sQsQJ-0003CI-1e for submit@debbugs.gnu.org; Mon, 08 Jul 2024 13:45:11 -0400 Received: from eggs.gnu.org ([209.51.188.92]:45534) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sQsQH-0003C0-8I for 71896@debbugs.gnu.org; Mon, 08 Jul 2024 13:45:09 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sQsNx-0000tG-OD; Mon, 08 Jul 2024 13:42:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=3fUK9qwR0tml9ftuAmZKb5gDO/ODY2Z5abK+9Iea5tw=; b=c71/lPmxnY/u5wa4f9Km 5vpplY1DzK1P1mAJU8MKELS2NwAraLxHkRza2EMHN8U+ph2MEbKabgJQxVeM+c/q3w99RukUCyXfc NdqyGsO4eWfpbfA9tRviyrzjZ6xY1k69ugheN3dC1ESv5hf+sDYTTnAMhzB/Nbbu1doRqHYcLi3/P 9qbr8Egv1q2M/B6ugbBrh3tb9WRoDcJ+aLmMNUwRDYWaRKAiZOI0uyiGOcxjToK9FqVWOHXvQ26jp lPrDPiueJ/EZMHQhK+MnACTLFC8fWcKOepeT0WGIBWL8SoXU4uC0I9OSehQw2nwLc876BoMCDJyev 7JDIzslKvvjPYg==; Date: Mon, 08 Jul 2024 20:42:11 +0300 Message-Id: <86zfqrzsp8.fsf@gnu.org> From: Eli Zaretskii In-Reply-To: (message from Troy Hinckley on Mon, 8 Jul 2024 12:04:07 -0500) References: <03321175-1a92-4b82-a0cc-d6977b6a733a@Spark> <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> <86tth24zbr.fsf@gnu.org> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > Date: Mon, 8 Jul 2024 12:04:07 -0500 > From: Troy Hinckley > Cc: 71896@debbugs.gnu.org > > The proposed patch throws an error when `last’ is “” because (substring “” -1) produces (args-out-of-range “” - > 1 nil). Is that the only problem with the patch? That is, if this is trivially fixed, does all the rest work correctly and fixes the problems? From unknown Sat Aug 16 23:50:05 2025 X-Loop: help-debbugs@gnu.org Subject: bug#71896: shell-resync-dirs hang Resent-From: Troy Hinckley Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 08 Jul 2024 17:54:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 71896 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: Eli Zaretskii Cc: 71896@debbugs.gnu.org Received: via spool by 71896-submit@debbugs.gnu.org id=B71896.172046121013085 (code B ref 71896); Mon, 08 Jul 2024 17:54:01 +0000 Received: (at 71896) by debbugs.gnu.org; 8 Jul 2024 17:53:30 +0000 Received: from localhost ([127.0.0.1]:51460 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sQsYM-0003Oz-Ap for submit@debbugs.gnu.org; Mon, 08 Jul 2024 13:53:30 -0400 Received: from sender4-op-o15.zoho.com ([136.143.188.15]:17561) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sQsYK-0003Or-JK for 71896@debbugs.gnu.org; Mon, 08 Jul 2024 13:53:29 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1720461197; cv=none; d=zohomail.com; s=zohoarc; b=mB7Jlo+DDtVXqCYwHoBehIH2UwL6b2tt8wj4nQncClZK8hzk5x//T5OHc0raMfWd8SlvlJaqIQJ0wbAWEOwu8wd59TWfyvCJHgDnmKu/YvyUDiLkWmcqpMA2+z5P8X5reYzyxuCibyAqe+KM7mXq+q2HQ6lLBdTFgjUlUOwLIh4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1720461197; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=mzVAeM1a1J8FVDEmDPOKl4jPDeUSYjgHeu6grx6kbWU=; b=Rf7NmIO68mLW0T/10nkPEhi6z26VCH9MLbpzPg8Ah7FNyfwxpCcJG3MCmyiTFN/jtQQGraEiMCG+3KX0i42XwRQ/EOLO9iZaqOSGVVIqkcuN8MaVKkcrTpLT3poOTsNHexJvPj75nQ/HAkG2P6UGS80NDeHOtyRn7QgHdP1ka5Y= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=dabrev.com; spf=pass smtp.mailfrom=troyhinckley@dabrev.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1720461197; s=zoho; d=dabrev.com; i=troyhinckley@dabrev.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-ID:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Message-Id:Reply-To; bh=mzVAeM1a1J8FVDEmDPOKl4jPDeUSYjgHeu6grx6kbWU=; b=LD1VaBPFnl22qTd2dtdptYfr6MdQHajL3ldCaMfq0qisQDWXqYTwF8MpJ9J4p++E lRvfZgRzltuH6kpNcbbQ8UNJLqC8jlH2RJhM2nHSUJxqUxJvAT+PjE61phypj6CuuUG DyQnyCsT44G8PsFCqUd88utrsGEVulFuJ2wCmj0w= Received: by mx.zohomail.com with SMTPS id 1720461195680497.0018159677371; Mon, 8 Jul 2024 10:53:15 -0700 (PDT) Date: Mon, 8 Jul 2024 12:53:06 -0500 From: Troy Hinckley Message-ID: <2f1e46cc-e9c8-4816-830d-81ce74163bda@Spark> In-Reply-To: <86zfqrzsp8.fsf@gnu.org> References: <03321175-1a92-4b82-a0cc-d6977b6a733a@Spark> <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> <86tth24zbr.fsf@gnu.org> <86zfqrzsp8.fsf@gnu.org> X-Readdle-Message-ID: 2f1e46cc-e9c8-4816-830d-81ce74163bda@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="668c2787_216231b_24f" X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --668c2787_216231b_24f Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Yes, if I fix that one issue with the patch then all the rest works corre= ctly and fixes the problem. Thank you. -Troy Hinckley On Jul 8, 2024 at 12:42=E2=80=AFPM -0500, Eli Zaretskii ,= wrote: > > Date: Mon, 8 Jul 2024 12:04:07 -0500 > > =46rom: Troy Hinckley > > Cc: 71896=40debbugs.gnu.org > > > > The proposed patch throws an error when =60last=E2=80=99 is =E2=80=9C= =E2=80=9D because (substring =E2=80=9C=E2=80=9D -1) produces (args-out-of= -range =E2=80=9C=E2=80=9D - > > 1 nil). > > Is that the only problem with the patch=3F That is, if this is > trivially fixed, does all the rest work correctly and fixes the > problems=3F --668c2787_216231b_24f Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Yes, if I fix that one issue with the patch then al= l the rest works correctly and fixes the problem.
Thank you.

-Troy Hinckley
On Jul 8, 2024 at 12:42=E2=80=AFPM = -0500, Eli Zaretskii <eliz=40gnu.org>, wrote:
Date: Mon, 8 Jul 2024 12:04:07 -0500
=46rom: Troy Hinckley <troyhinckley=40dabrev.com>
Cc: 71896=40debbugs.gnu.org

The proposed patch throws an error when =60last=E2=80=99 is =E2=80=9C=E2=80= =9D because (substring =E2=80=9C=E2=80=9D -1) produces (args-out-of-range= =E2=80=9C=E2=80=9D -
1 nil).

Is that the only problem with the patch=3F That is, if this is
trivially fixed, does all the rest work correctly and fixes the
problems=3F
--668c2787_216231b_24f-- From unknown Sat Aug 16 23:50:05 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Troy Hinckley Subject: bug#71896: closed (Re: bug#71896: shell-resync-dirs hang) Message-ID: References: <86sewfksbl.fsf@gnu.org> <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> X-Gnu-PR-Message: they-closed 71896 X-Gnu-PR-Package: emacs Reply-To: 71896@debbugs.gnu.org Date: Fri, 12 Jul 2024 07:02:01 +0000 Content-Type: multipart/mixed; boundary="----------=_1720767721-8453-1" This is a multi-part message in MIME format... ------------=_1720767721-8453-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #71896: shell-resync-dirs hang which was filed against the emacs package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 71896@debbugs.gnu.org. --=20 71896: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D71896 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1720767721-8453-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 71896-done) by debbugs.gnu.org; 12 Jul 2024 07:01:10 +0000 Received: from localhost ([127.0.0.1]:53076 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sSAHG-0002Ax-9B for submit@debbugs.gnu.org; Fri, 12 Jul 2024 03:01:10 -0400 Received: from eggs.gnu.org ([209.51.188.92]:32898) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sSAHB-0002AF-04 for 71896-done@debbugs.gnu.org; Fri, 12 Jul 2024 03:01:08 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSAH5-0002Hs-Eb; Fri, 12 Jul 2024 03:00:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=8KZOlTPuvNgA1YJSt/pYjWcu+ciS1ALpuD9jcP9r1eQ=; b=EYSuRPrFPo0h tdCYTGIqJTj0ArQ0Jh7dMLEWFkGkVRkmcfECKHqt0xr9Uw2Ndb2jAiLl/lp6rCD/Sf9plpid+rXkV 3UYCZsaOU5cYTzD2NGGH/V8kYCRV/BIT4S4WEKiGs+ahESII3dtmRg5tzRKuUtSK477JInk0TkWCY 5evCST+bE+XUQKQQwF6xk6NBbM2UcH8ZP8Qj6Km3qXktnZzfHHBc6xKKWDzlGIY47W7ZCHrOpsXXY mquwWrwC6y9sQkV1AAcZy97HQjhif5HP0ROR3hb6JpN6l3HKrEpwe9aiGb3JXGBiy7tmYjFiwjc3k jTmxePtOH5yoLnNGitSoQA==; Date: Fri, 12 Jul 2024 10:00:46 +0300 Message-Id: <86sewfksbl.fsf@gnu.org> From: Eli Zaretskii To: Troy Hinckley In-Reply-To: <2f1e46cc-e9c8-4816-830d-81ce74163bda@Spark> (message from Troy Hinckley on Mon, 8 Jul 2024 12:53:06 -0500) Subject: Re: bug#71896: shell-resync-dirs hang References: <03321175-1a92-4b82-a0cc-d6977b6a733a@Spark> <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> <86tth24zbr.fsf@gnu.org> <86zfqrzsp8.fsf@gnu.org> <2f1e46cc-e9c8-4816-830d-81ce74163bda@Spark> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 71896-done Cc: 71896-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > Date: Mon, 8 Jul 2024 12:53:06 -0500 > From: Troy Hinckley > Cc: 71896@debbugs.gnu.org > > Yes, if I fix that one issue with the patch then all the rest works correctly and fixes the problem. Thanks, so I've now installed an augmented fix on the emacs-30 release branch, and I'm therefore closing this bug. ------------=_1720767721-8453-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 2 Jul 2024 05:16:34 +0000 Received: from localhost ([127.0.0.1]:35518 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sOVsX-0004Gt-0X for submit@debbugs.gnu.org; Tue, 02 Jul 2024 01:16:33 -0400 Received: from lists.gnu.org ([209.51.188.17]:42064) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sOTAc-0008BP-Fs for submit@debbugs.gnu.org; Mon, 01 Jul 2024 22:23:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sOTAZ-0002EC-BW for bug-gnu-emacs@gnu.org; Mon, 01 Jul 2024 22:23:01 -0400 Received: from sender4-op-o15.zoho.com ([136.143.188.15]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sOTAU-0005oe-Lp for bug-gnu-emacs@gnu.org; Mon, 01 Jul 2024 22:22:57 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1719886968; cv=none; d=zohomail.com; s=zohoarc; b=MlqksOvdRHBJzUKaxKrnzD2Dx85k/sRmYVRwGwWUhGhmC1NBTB6wXlryDxYyMBf+pXOd4jTkfWCkiN91E0tVkKKJnira1CuPKRmkyojS7Ay9vUfsIFCFZvpRi0Kp3Yy1QhjocK/QYGnvcEuk23rI52aoDH1tEy3towZDq2LitMA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1719886968; h=Content-Type:Date:Date:From:From:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=Z1pWzUqgNYF1OqotZ6NaxNjWDOYGh/pXgX+8Wah7kT4=; b=GH5jhtKLnskAnsxv7InxVL7IaM026NRrqbeZOzgjoGhAQcYqaSnwwtfHY8nNsI4+TYS9/rgDj7FsdpVzQDNXyNx+NWrmXSBpWdhAOt7MLeNVpV/vzkwDCUjGEe7ygilxyUdwSgaf/cmeInO0Qt6FzseXrtaF8H7hmKWi2NsiIVc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=dabrev.com; spf=pass smtp.mailfrom=troyhinckley@dabrev.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1719886968; s=zoho; d=dabrev.com; i=troyhinckley@dabrev.com; h=Date:Date:From:From:To:To:Message-ID:References:Subject:Subject:MIME-Version:Content-Type:Message-Id:Reply-To:Cc; bh=Z1pWzUqgNYF1OqotZ6NaxNjWDOYGh/pXgX+8Wah7kT4=; b=hiZkkkSkb5CvaCN6RVSa6gZhMrY/JGzJI1x8gg0lD5SbVrnGVLmxyZB4K9U3ggH3 /J1Ww7LmyOMRx8VxFYvUTiiPeARKP/IW8yOATxJ8F9ZKQ6RUpLLbBfo77AgUBAorMtU K1tS8inGIhCjoYc5YwLhP6pOS8K2d6zqO+H0QRPE= Received: by mx.zohomail.com with SMTPS id 171988696629677.4677691426291; Mon, 1 Jul 2024 19:22:46 -0700 (PDT) Date: Mon, 1 Jul 2024 21:22:35 -0500 From: Troy Hinckley To: bug-gnu-emacs@gnu.org Message-ID: <83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark> References: <03321175-1a92-4b82-a0cc-d6977b6a733a@Spark> Subject: shell-resync-dirs hang X-Readdle-Message-ID: 83b035dd-1ba4-4016-8ba4-cf9bde800175@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="66836470_6eb5bd4_524" X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.15; envelope-from=troyhinckley@dabrev.com; helo=sender4-op-o15.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Tue, 02 Jul 2024 01:16:24 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --66836470_6eb5bd4_524 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline This is related to =2359804 and =2354384. I will occasionally (about 30% of the time) see hangs when running shell-= resync-dirs in Emacs 29. I tracked this down to two issues: =46irst is in shell-eval-command. =46or Emacs 29 this function was added = to fix =2354384. It has this section in the code: =60=60=60 ;; Wait until we get a prompt (which will be a line without =C2=A0;; a newline). This is far from fool-proof -- if something =C2=A0;; outputs incomplete data and then sleeps, we'll think =C2=A0;; we've received the prompt. =C2=A0(while (not (let* ((lines (string-lines result)) =C2=A0(last (car (last lines)))) =C2=A0(and (length> lines 0) =C2=A0(not (equal last =22=22)) =C2=A0(or (not prev) =C2=A0(not (equal last prev))) =C2=A0(setq prev last)))) =C2=A0(accept-process-output proc 0 100)) =60=60=60 Note that is says that is looking for =E2=80=9Ca line without a newline=E2= =80=9D to determine if we have reached the prompt. However this code does= not actually do that. If the result ends in a newline it will still term= inate the loop and not wait for more input. We can see that by the fact t= hat the following code evaluates to nil. =60=60=60 (let ((result =22dirs=5Cn=22) prev) =C2=A0(not (let* ((lines (string-lines result)) =C2=A0(last (car (last lines)))) =C2=A0(and (length> lines 0) =C2=A0(not (equal last =22=22)) =C2=A0(or (not prev) =C2=A0(not (equal last prev))) =C2=A0(setq prev last))))) =60=60=60 I am not sure what this code is supposed to do, but the issue arrises if = the process output sends anything to this function it will terminate and = not wait for more input. In my case the issue is that the shell is echoin= g the command followed by the result (comint-process-echoes). About 30% o= f the time these two lines get sent as part of two different outputs. Mea= ning the second line (the directory for shell-resync-dirs) does not get c= aptured and instead gets printed to the terminal. This leads us to the hang. The issue is this code in shell-resync-dirs: =60=60=60 (while dlsl =C2=A0(let ((newelt =22=22) =C2=A0tem1 tem2) =C2=A0(while newelt =C2=A0;; We need tem1 because we don't want to prepend =C2=A0;; =60comint-file-name-prefix' repeatedly into newelt via tem2. =C2=A0(setq tem1 (pop dlsl) =C2=A0tem2 (concat comint-file-name-prefix tem1 newelt)) =C2=A0(cond ((file-directory-p tem2) =C2=A0(push tem2 ds) =C2=A0(when (string=3D =22 =22 (car dlsl)) =C2=A0(pop dlsl)) =C2=A0(setq newelt nil)) =C2=A0(t =C2=A0(setq newelt (concat tem1 newelt))))))) =60=60=60 This loop can only terminate if tem2 is a valid directory. Otherwise it w= ill take the default case in the cond and loop forever. And since the bug= in shell-eval-command does not provide the directory when the process ou= tput is split, we get a hang. I believe both of these need to be fixed to properly fix the bug. =46or the shell-eval-command I don=E2=80=99t understand what that loop is= trying to do now, so I am not sure how to fix it without breaking its fu= nctionality. I would just use (string-suffix-p =E2=80=9C=5Cn=E2=80=9D res= ult) to check if the output ends in a newline, but the code is obviously = trying to do something more complex there. If we fix that issue then it will resolve the hang in shell-resync-dirs, = but I think that is just glossing over the problem. That functions should= never hang, no matter what output it get=E2=80=99s from the shell. My re= commendation would be to add =60(and dlsl newelt)=60 as the condition for= the inner while loop with newelt. That way if dlsl is empty, it will ter= minate the loop since there is nothing more to process. This fixed the is= sue for me. -Troy Hinckley --66836470_6eb5bd4_524 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
This is related to =2359804 and =2354384.

I will occasionally (about 30% of the time) see hangs when running shell-= resync-dirs in Emacs 29. I tracked this down to two issues:


=46irst is in shell-eval-command. =46or Emacs 29 this function was added = to fix =2354384. It has this section in the code:

=60=60=60
;; Wait until we get a prompt (which will be a line without
&=23160;;; a newline). This is far from fool-proof -- if something
&=23160;;; outputs incomplete data and then sleeps, we'll think
&=23160;;; we've received the prompt.
&=23160;(while (not (let* ((lines (string-lines result))
&=23160;(last (car (last lines))))
&=23160;(and (length> lines 0)
&=23160;(not (equal last =22=22))
&=23160;(or (not prev)
&=23160;(not (equal last prev)))
&=23160;(setq prev last))))
&=23160;(accept-process-output proc 0 100))
=60=60=60

Note that is says that is looking for =E2=80=9Ca line without a newline=E2= =80=9D to determine if we have reached the prompt. However this code does= not actually do that. If the result ends in a newline it will still term= inate the loop and not wait for more input. We can see that by the fact t= hat the following code evaluates to nil.

=60=60=60
(let ((result =22dirs=5Cn=22) prev)
&=23160;(not (let* ((lines (string-lines result))
&=23160;(last (car (last lines))))
&=23160;(and (length> lines 0)
&=23160;(not (equal last =22=22))
&=23160;(or (not prev)
&=23160;(not (equal last prev)))
&=23160;(setq prev last)))))
=60=60=60

I am not sure what this code is supposed to do, but the issue arrises if = the process output sends anything to this function it will terminate and = not wait for more input. In my case the issue is that the shell is echoin= g the command followed by the result (comint-process-echoes). About 30% o= f the time these two lines get sent as part of two different outputs. Mea= ning the second line (the directory for shell-resync-dirs) does not get c= aptured and instead gets printed to the terminal.


This leads us to the hang. The issue is this code in shell-resync-dirs:
=60=60=60
(while dlsl
&=23160;(let ((newelt =22=22)
&=23160;tem1 tem2)
&=23160;(while newelt
&=23160;;; We need tem1 because we don't want to prepend
&=23160;;; =60comint-file-name-prefix' repeatedly into newelt via tem2. &=23160;(setq tem1 (pop dlsl)
&=23160;tem2 (concat comint-file-name-prefix tem1 newelt))
&=23160;(cond ((file-directory-p tem2)
&=23160;(push tem2 ds)
&=23160;(when (string=3D =22 =22 (car dlsl))
&=23160;(pop dlsl))
&=23160;(setq newelt nil))
&=23160;(t
&=23160;(setq newelt (concat tem1 newelt)))))))
=60=60=60

This loop can only terminate if tem2 is a valid directory. Otherwise it w= ill take the default case in the cond and loop forever. And since the bug= in shell-eval-command does not provide the directory when the process ou= tput is split, we get a hang.

I believe both of these need to be fixed to properly fix the bug.

=46or the shell-eval-command I don=E2=80=99t understand what that loop is= trying to do now, so I am not sure how to fix it without breaking its fu= nctionality. I would just use (string-suffix-p =E2=80=9C=5Cn=E2=80=9D res= ult) to check if the output ends in a newline, but the code is obviously = trying to do something more complex there.

If we fix that issue then it will resolve the hang in shell-resync-dirs, = but I think that is just glossing over the problem. That functions should= never hang, no matter what output it get=E2=80=99s from the shell. My re= commendation would be to add =60(and dlsl newelt)=60 as the condition for= the inner while loop with newelt. That way if dlsl is empty, it will ter= minate the loop since there is nothing more to process. This fixed the is= sue for me.&=23160;

-Troy Hinckley
--66836470_6eb5bd4_524-- ------------=_1720767721-8453-1--