From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 25 19:23:05 2023 Received: (at submit) by debbugs.gnu.org; 25 Apr 2023 23:23:05 +0000 Received: from localhost ([127.0.0.1]:53758 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prS01-00074k-3U for submit@debbugs.gnu.org; Tue, 25 Apr 2023 19:23:05 -0400 Received: from lists.gnu.org ([209.51.188.17]:49220) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prRzw-000747-Kx for submit@debbugs.gnu.org; Tue, 25 Apr 2023 19:23:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1prRzw-0005hJ-8A for bug-gnu-emacs@gnu.org; Tue, 25 Apr 2023 19:23:00 -0400 Received: from [78.47.144.35] (helo=metalevel.at) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1prRzr-0001dl-7h for bug-gnu-emacs@gnu.org; Tue, 25 Apr 2023 19:22:59 -0400 Received: from mt-Lenovo-ideapad-120S-11IAP (localhost [127.0.0.1]) by metalevel.at (Postfix) with ESMTP id AA5E59C73F for ; Wed, 26 Apr 2023 01:15:47 +0200 (CEST) Received: by mt-Lenovo-ideapad-120S-11IAP (Postfix, from userid 1000) id DD542140046; Wed, 26 Apr 2023 01:15:36 +0200 (CEST) From: Markus Triska To: bug-gnu-emacs@gnu.org Subject: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode Date: Wed, 26 Apr 2023 01:15:36 +0200 Message-ID: <87wn1zpn8n.fsf@metalevel.at> MIME-Version: 1.0 Content-Type: text/plain X-Host-Lookup-Failed: Reverse DNS lookup failed for 78.47.144.35 (failed) Received-SPF: none client-ip=78.47.144.35; envelope-from=triska@metalevel.at; helo=metalevel.at X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Dear all, to reproduce this issue, please save the following C code in proc.c: #include #include int main() { printf("hello"); exit(0); } and compile it, yielding the program file "./proc": $ gcc proc.c -o proc Further, please store the following Elisp definitions in proc.el, in the same directory as the above program: (add-to-list 'exec-path ".") (defvar p-stopped nil) (defun my-sentinel (proc str) (when (string-match "^\\(?:finished\n\\|exited abnormally\\|killed\n\\)" str) (setq p-stopped t))) (defun wait-for-process-end (proc) (while (not p-stopped) (message "process status is: %s" (process-status proc)) (accept-process-output proc 0.1 nil t))) (defun f () (with-temp-buffer (let ((proc (start-process "p" (current-buffer) "proc"))) (setq p-stopped nil) (set-process-sentinel proc 'my-sentinel) (wait-for-process-end proc)) (message "process said: %s" (buffer-string)))) The following invocation and result shows a case where the program behaves exactly as intended: It waits for the process sentinel to signal that no more output can arrive from the process, and shows the output it got while the process was producing output: $ emacs -Q --script proc.el --eval "(f)" process status is: exit process said: hello In contrast, the following invocation produces, in the echo area, the message "process status is: exit", and Emacs hangs unexpectedly: $ emacs -Q -l proc.el --eval "(f)" The behaviour I expect in this case is analogous to the other invocation: I do expect the message we see, followed by the message "process said: hello", as in the case above. However, this second message only appears if I manually interrupt Emacs with C-g. Regarding the background of this use case: The goal of one of my Emacs scripts is to reliably obtain *all* output from a process, until the process ends (which it always does), and then end the script. A flawed attempt to do this is to check the state of the process (via process-status), and end the script when the the state is no longer 'run. However, process output can (and sometimes does) still arrive even after the state is no longer equal to 'run. Apparently only the sentinel can reliably detect that no more process output can follow. Is there a way to make Emacs invoke the sentinel in the example above? Thank you a lot! Markus In GNU Emacs 26.1 (build 3, x86_64-pc-linux-gnu, X toolkit, Xaw scroll bars) of 2019-04-09 built on mt-laptop Windowing system distributor 'The X.Org Foundation', version 11.0.12201003 System Description: Ubuntu 22.10 Configured using: 'configure --with-tiff=no' Configured features: XPM JPEG GIF PNG SOUND GSETTINGS NOTIFY GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11 THREADS Important settings: value of $LC_MONETARY: en_GB.UTF-8 value of $LC_NUMERIC: en_GB.UTF-8 value of $LC_TIME: en_GB.UTF-8 value of $LANG: en_US.UTF-8 value of $XMODIFIERS: @im=ibus locale-coding-system: utf-8-unix From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 26 02:21:46 2023 Received: (at 63078) by debbugs.gnu.org; 26 Apr 2023 06:21:46 +0000 Received: from localhost ([127.0.0.1]:54107 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prYXB-0002bX-Nr for submit@debbugs.gnu.org; Wed, 26 Apr 2023 02:21:46 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33662) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prYX8-0002bF-UK for 63078@debbugs.gnu.org; Wed, 26 Apr 2023 02:21:45 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1prYX2-0007Ac-JU; Wed, 26 Apr 2023 02:21:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=6hZ6rM9tR6QhvIwpT1wioq9741kt6sYMKUDOht2KF5A=; b=bKtlEgs3Co3Y 0w1N3+asCKAfVTRCeF+KvLQ3JgjViUv8FYJDLYUUFvz7gWTZeUKO3B3+Yet31zut7WsOG2NE2lgaO GOqQFvk8ccH1kxSML2rQA233jhsO0GoaFjp/wv6V7a5iqQHjkAfgShS8EdabvFoxABusyc8TtBHEB PjlNFWiwUqc078g90WqlYOVaA7N+9nC4Gc6we7yBnJ8yUCLcpMXAHbJOAD4P9iePgXYXCsc0Va4Pq bgUVrVzOKKLZEQs8Dcofazg8olQU2G0m/SQskkn8aKogqbIJGqIRwnahgRjYF27+/I6LhUhEu8vio OwrB1KnP2UbKCR/1c4zn4A==; Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1prYWz-0001DE-D4; Wed, 26 Apr 2023 02:21:34 -0400 Date: Wed, 26 Apr 2023 09:22:02 +0300 Message-Id: <837ctzuprp.fsf@gnu.org> From: Eli Zaretskii To: Markus Triska In-Reply-To: <87wn1zpn8n.fsf@metalevel.at> (message from Markus Triska on Wed, 26 Apr 2023 01:15:36 +0200) Subject: Re: bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode References: <87wn1zpn8n.fsf@metalevel.at> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 63078 Cc: 63078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Markus Triska > Date: Wed, 26 Apr 2023 01:15:36 +0200 > > to reproduce this issue, please save the following C code in proc.c: > > #include > #include > int main() { > printf("hello"); > exit(0); > } > > and compile it, yielding the program file "./proc": > > $ gcc proc.c -o proc > > Further, please store the following Elisp definitions in proc.el, in > the same directory as the above program: > > (add-to-list 'exec-path ".") > > (defvar p-stopped nil) > > (defun my-sentinel (proc str) > (when (string-match "^\\(?:finished\n\\|exited abnormally\\|killed\n\\)" > str) > (setq p-stopped t))) > > (defun wait-for-process-end (proc) > (while (not p-stopped) > (message "process status is: %s" (process-status proc)) > (accept-process-output proc 0.1 nil t))) > > (defun f () > (with-temp-buffer > (let ((proc (start-process "p" (current-buffer) "proc"))) > (setq p-stopped nil) > (set-process-sentinel proc 'my-sentinel) > (wait-for-process-end proc)) > (message "process said: %s" (buffer-string)))) > > The following invocation and result shows a case where the program > behaves exactly as intended: It waits for the process sentinel to > signal that no more output can arrive from the process, and shows the > output it got while the process was producing output: > > $ emacs -Q --script proc.el --eval "(f)" > process status is: exit > process said: hello > > In contrast, the following invocation produces, in the echo area, the > message "process status is: exit", and Emacs hangs unexpectedly: > > $ emacs -Q -l proc.el --eval "(f)" > > The behaviour I expect in this case is analogous to the other > invocation: I do expect the message we see, followed by the message > "process said: hello", as in the case above. However, this second > message only appears if I manually interrupt Emacs with C-g. I cannot reproduce the problem with Emacs 28.2: I get "process said: hello" in the echo-area, as expected. Did you try with the -nw command-line switch, and if you do, does it change anything? It could be some timing problem: you start a program that exits almost immediately, so there could be some race condition between the program and your code. From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 26 02:52:31 2023 Received: (at 63078) by debbugs.gnu.org; 26 Apr 2023 06:52:31 +0000 Received: from localhost ([127.0.0.1]:54137 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prZ0w-0003Wl-VM for submit@debbugs.gnu.org; Wed, 26 Apr 2023 02:52:31 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:46700) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prZ0t-0003WX-LV for 63078@debbugs.gnu.org; Wed, 26 Apr 2023 02:52:29 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4Q5qLK6vV0z1sClx; Wed, 26 Apr 2023 08:52:25 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 4Q5qLK5jX8z1qqlS; Wed, 26 Apr 2023 08:52:25 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id 8JSVFTYA1l53; Wed, 26 Apr 2023 08:52:25 +0200 (CEST) X-Auth-Info: bpyuSoz2g3x7S1/mzByles+HUJ1qEOkyIIn3aMJkE6Zm63Zk6bzmCTFj7nubvUPh Received: from tiger.home (aftr-82-135-86-255.dynamic.mnet-online.de [82.135.86.255]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Wed, 26 Apr 2023 08:52:25 +0200 (CEST) Received: by tiger.home (Postfix, from userid 1000) id 909E21987F0; Wed, 26 Apr 2023 08:52:24 +0200 (CEST) From: Andreas Schwab To: Markus Triska Subject: Re: bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode In-Reply-To: <87wn1zpn8n.fsf@metalevel.at> (Markus Triska's message of "Wed, 26 Apr 2023 01:15:36 +0200") References: <87wn1zpn8n.fsf@metalevel.at> X-Yow: Hello. I know the divorce rate among unmarried Catholic Alaskan females!! Date: Wed, 26 Apr 2023 08:52:24 +0200 Message-ID: <87bkjbjftj.fsf@linux-m68k.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.4 (/) X-Debbugs-Envelope-To: 63078 Cc: 63078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.4 (-) On Apr 26 2023, Markus Triska wrote: > In contrast, the following invocation produces, in the echo area, the > message "process status is: exit", and Emacs hangs unexpectedly: > > $ emacs -Q -l proc.el --eval "(f)" If you C-g at this point and look at *Messsages* you will see that the message "process status is: exit" has been repeated a lot of times. So the issue is that the sentinal is being reentered ad infinitum, preventing to make progress. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 26 13:59:14 2023 Received: (at 63078) by debbugs.gnu.org; 26 Apr 2023 17:59:14 +0000 Received: from localhost ([127.0.0.1]:56947 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prjQ8-0003SX-Dx for submit@debbugs.gnu.org; Wed, 26 Apr 2023 13:59:14 -0400 Received: from [78.47.144.35] (port=55342 helo=metalevel.at) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prjPr-0003Rp-R5 for 63078@debbugs.gnu.org; Wed, 26 Apr 2023 13:59:10 -0400 Received: by metalevel.at (Postfix, from userid 1000) id 8595B9C75F; Wed, 26 Apr 2023 19:58:54 +0200 (CEST) From: Markus Triska To: Eli Zaretskii Subject: Re: bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode References: <87wn1zpn8n.fsf@metalevel.at> <837ctzuprp.fsf@gnu.org> Date: Wed, 26 Apr 2023 19:58:54 +0200 In-Reply-To: <837ctzuprp.fsf@gnu.org> (Eli Zaretskii's message of "Wed, 26 Apr 2023 09:22:02 +0300") Message-ID: <87r0s6frtt.fsf@metalevel.at> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Eli Zaretskii writes: > I cannot reproduce the problem with Emacs 28.2: I get "process said: > hello" in the echo-area, as expected. Did you try with the -nw > command-line switch, and if you do, does it change anything? Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 SPF_NONE SPF: sender does not publish an SPF Record 1.3 RDNS_NONE Delivered to internal network by a host with no rDNS X-Debbugs-Envelope-To: 63078 Cc: 63078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) Eli Zaretskii writes: > I cannot reproduce the problem with Emacs 28.2: I get "process said: > hello" in the echo-area, as expected. Did you try with the -nw > command-line switch, and if you do, does it change anything? > It could be some timing problem: you start a program that exits almost > immediately, so there could be some race condition between the program > and your code. I can reproduce the problem also with Emacs 28.8, and yes, I also believe this may be a timing issue: I have now constructed a test case that exhibits the issue also when the "-nw" option is used. To reproduce, please start Emacs with: $ emacs -Q or with: $ emacs -Q -nw and then place the forms below in a buffer, and then evaluate the buffer contents with M-x eval-buffer RET. The bash shell is used as external process, the sample C program I originally included is no longer needed. > The expected output of the forms below is an endless stream of messages of the form: trying to elicit race condition at (25673 25753 287220 854000) process status is: run [2 times] process said: hi failed trying to elicit race condition at (25673 25753 289270 703000) process status is: run [2 times] process said: hi failed i.e., the expected case is that *no* race condition exists, and therefore eliciting the issue *fails*, which is the desired result as long as the program runs. However, after I few seconds, this expected sequence of messages stops, and only the following message is then emitted over and over: process status is: exit When this occurs, it means that the sentinel was unexpectedly not invoked when the process stops, in contrast to all preceding cases where the sentinel was invoked as expected when the process terminates. Running the program in script mode yields the expected result over and over, I cannot elicit the issue with the --script option of Emacs. I hope this helps. Can you reproduce the issue with this example? Thank you and all the best, Markus (defvar p-stopped nil) (defun my-sentinel (proc str) (when (string-match "^\\(?:finished\n\\|exited abnormally\\|killed\n\\)" str) (setq p-stopped t))) (defun wait-for-process-end (proc) (while (not p-stopped) (message "process status is: %s" (process-status proc)) (accept-process-output proc 0.1 nil t))) (defun elicit-issue () (with-temp-buffer (let ((proc (start-process "p" (current-buffer) "bash" "-c" "echo -n hi"))) (setq p-stopped nil) (set-process-sentinel proc 'my-sentinel) (wait-for-process-end proc)) (message "process said: %s" (buffer-string)))) (while t (message "trying to elicit race condition at %s" (current-time)) (elicit-issue) (message "failed")) From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 26 14:00:07 2023 Received: (at 63078) by debbugs.gnu.org; 26 Apr 2023 18:00:07 +0000 Received: from localhost ([127.0.0.1]:56951 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prjR1-0003Ut-DI for submit@debbugs.gnu.org; Wed, 26 Apr 2023 14:00:07 -0400 Received: from [78.47.144.35] (port=55354 helo=metalevel.at) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1prjQz-0003Uh-U6 for 63078@debbugs.gnu.org; Wed, 26 Apr 2023 14:00:06 -0400 Received: by metalevel.at (Postfix, from userid 1000) id 71CFD9C75D; Wed, 26 Apr 2023 20:00:05 +0200 (CEST) From: Markus Triska To: Andreas Schwab Subject: Re: bug#63078: 26.1; Sentinels are sometimes unexpectedly only invoked in script mode References: <87wn1zpn8n.fsf@metalevel.at> <87bkjbjftj.fsf@linux-m68k.org> Date: Wed, 26 Apr 2023 20:00:05 +0200 In-Reply-To: <87bkjbjftj.fsf@linux-m68k.org> (Andreas Schwab's message of "Wed, 26 Apr 2023 08:52:24 +0200") Message-ID: <87mt2ufrru.fsf@metalevel.at> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Andreas Schwab writes: > If you C-g at this point and look at *Messsages* you will see that the > message "process status is: exit" has been repeated a lot of times. So > the issue is that the sentinal is being reentered ad [...] Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 SPF_NONE SPF: sender does not publish an SPF Record 1.3 RDNS_NONE Delivered to internal network by a host with no rDNS X-Debbugs-Envelope-To: 63078 Cc: 63078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) Andreas Schwab writes: > If you C-g at this point and look at *Messsages* you will see that the > message "process status is: exit" has been repeated a lot of times. So > the issue is that the sentinal is being reentered ad infinitum, > preventing to make progress. In the program I posted, this message is not emitted by the sentinel, are you certain that the sentinel is invoked as intended in this case? Thank you and all the best, Markus