From unknown Thu Sep 11 05:48:41 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#79333 <79333@debbugs.gnu.org> To: bug#79333 <79333@debbugs.gnu.org> Subject: Status: 31.0.50; Processes (still) aren't actually locked to threads Reply-To: bug#79333 <79333@debbugs.gnu.org> Date: Thu, 11 Sep 2025 12:48:41 +0000 retitle 79333 31.0.50; Processes (still) aren't actually locked to threads reassign 79333 emacs submitter 79333 Spencer Baugh severity 79333 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Thu Aug 28 15:45:34 2025 Received: (at submit) by debbugs.gnu.org; 28 Aug 2025 19:45:34 +0000 Received: from localhost ([127.0.0.1]:40151 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1uriYv-0005Gk-9s for submit@debbugs.gnu.org; Thu, 28 Aug 2025 15:45:33 -0400 Received: from lists.gnu.org ([2001:470:142::17]:45298) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1uriYr-0005CN-Pd for submit@debbugs.gnu.org; Thu, 28 Aug 2025 15:45:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uriYk-0007jg-QN for bug-gnu-emacs@gnu.org; Thu, 28 Aug 2025 15:45:22 -0400 Received: from mxout5.mail.janestreet.com ([64.215.233.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uriYi-0005R7-OU for bug-gnu-emacs@gnu.org; Thu, 28 Aug 2025 15:45:22 -0400 From: Spencer Baugh To: bug-gnu-emacs@gnu.org Subject: 31.0.50; Processes (still) aren't actually locked to threads X-Debbugs-Cc: Eli Zaretskii , Dmitry Gutov Date: Thu, 28 Aug 2025 15:45:18 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1756410318; bh=KX/nxjp0QQpSoVAhoMFtlfmKx3fY89WndzjhSe/sU6k=; h=From:To:Subject:Date; b=u+dmd1wwZ381HY648LsSXOMoK4zcSEJu/u4W0qbShJ6bL2IMS2W+pmmCIFBMd4SC7 URjKOEWjhdE3xqvBSIApRETEocIK8rxgKT8h5Nu5GTbm5ZreYeLrpgJ69EGgqBKztb YLD1gVgPC63Td87IBhR6h5TDP4LR5OZYuXdjfU0F/jJ8s+jMsfNUdRcQimSnCyxji2 7C8xtXqYg1c4TGSrLlalG5s2Id58H2wL0EZ92vlUqX1kPsaUVCDqpRH+THXqUHnHDK Tb+MoRPVbVlDR+TNLxKzOWqYE0Sipt67hswNdfB1p+Vm42Ktsmt5ETxBIJEt7HvmvZ UF/HqzU0YibOg== Received-SPF: pass client-ip=64.215.233.18; envelope-from=sbaugh@janestreet.com; helo=mxout5.mail.janestreet.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.9 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.1 (/) 1. emacs -Q 2. Eval this: (define-advice shell-command-sentinel (:before (process signal)) (message "process thread: %s, current thread: %s" (process-thread process) (current-thread))) (make-thread (lambda () (async-shell-command "sleep 2") ;; Do nothing. (thread-join (make-thread (lambda () (while t (sit-for 1))))))) (sit-for 3) 3. It hangs due to some thread bug, independent of the main bug I'm reporting here. Just hit C-g. (Notably, it doesn't hang when run in emacs --batch) 4. Observe in *Messages* a message like this: process thread: #, current thread: # The sentinel ran in a thread which is not process-thread. So even after c93be71e45, processes aren't actually locked to threads. (Once again, I think we should take the opportunity here to delete the code for locking processes to threads, since IMO it is not useful, and it is still broken) In GNU Emacs 31.0.50 (build 88, x86_64-pc-linux-gnu, GTK+ Version 3.22.30, cairo version 1.15.12) of 2025-08-28 built on igm-qws-u22796a Repository revision: 1ebb6e8822b5fc635549be14a3d4f2dd6f2d77a4 Repository branch: master Windowing system distributor 'The X.Org Foundation', version 11.0.12011000 System Description: Rocky Linux 8.10 (Green Obsidian) Configured using: 'configure -C --with-gif=no' Configured features: CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XINERAMA XINPUT2 XPM XRANDR GTK3 ZLIB Important settings: value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix Major mode: Lisp Interaction Minor modes in effect: tooltip-mode: t global-eldoc-mode: t eldoc-mode: t show-paren-mode: t electric-indent-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t minibuffer-regexp-mode: t line-number-mode: t indent-tabs-mode: t transient-mark-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t Load-path shadows: None found. Features: (shadow sort mail-extr compile comint ansi-osc ansi-color ring emacsbug lisp-mnt message mailcap yank-media puny dired dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068 epg-config gnus-util text-property-search time-date subr-x mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils comp-run bytecomp byte-compile comp-common rx warnings icons cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd touch-screen tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic indonesian philippine cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5 base64 format env code-pages mule custom widget keymap hashtable-print-readable backquote threads dbusbind inotify dynamic-setting system-font-setting font-render-setting cairo gtk x-toolkit xinput2 x multi-tty move-toolbar make-network-process tty-child-frames native-compile emacs) Memory information: ((conses 16 86231 9338) (symbols 48 6873 0) (strings 32 24499 2161) (string-bytes 1 811121) (vectors 16 15629) (vector-slots 8 186044 7873) (floats 8 30 1) (intervals 56 275 0) (buffers 984 11)) From debbugs-submit-bounces@debbugs.gnu.org Thu Aug 28 16:44:27 2025 Received: (at 79333) by debbugs.gnu.org; 28 Aug 2025 20:44:27 +0000 Received: from localhost ([127.0.0.1]:40276 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1urjTu-00058q-NP for submit@debbugs.gnu.org; Thu, 28 Aug 2025 16:44:27 -0400 Received: from mxout5.mail.janestreet.com ([64.215.233.18]:51519) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1urjTl-00056W-76 for 79333@debbugs.gnu.org; Thu, 28 Aug 2025 16:44:22 -0400 From: Spencer Baugh To: 79333@debbugs.gnu.org Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads In-Reply-To: (Spencer Baugh's message of "Thu, 28 Aug 2025 15:45:18 -0400") References: Date: Thu, 28 Aug 2025 16:44:10 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1756413850; bh=IO+9wbPQpWRlhUpgxlWTi0ppkjqkXp8X9+BijPoseQo=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=z6laOeTsmPeQaQSU7rCzBJl4WtwyidUq9hwfqerEwxgk3Go9CisSVejfMsbu/lK4s LFE33MnCreUJzNdFaKbKHM+3P8kz/0FgaPIlViJCO3Ptk+FsIZ+gtxIIbkSZ4/mby6 zzwyXP6066O1EmFNI5kkaudsgdzo82/F9gjVcP0BAD8hQINKHGw0ep1/q5qFR7DgwO wM/jzxMdq3yrRJOWnCl29DCR2+nwJYxRomTblwNuujWQesY79H7v/wBAdDXa+mpglj VBMjka/vmn0FU3AuoEc2ZUJDrznqyxEFZnOys5wvxmq/BZJ5/jF+W33cdUcsTd36Ii iAQsMzX9eVb8A== X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: Dmitry Gutov , Eli Zaretskii X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Spencer Baugh writes: > 3. It hangs due to some thread bug, independent of the main bug I'm > reporting here. Just hit C-g. (Notably, it doesn't hang when run in > emacs --batch) Extra detail about this: This hang happens for GTK3 Emacs, but doesn't happen for with-x-toolkit=lucid. From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 29 03:35:29 2025 Received: (at 79333) by debbugs.gnu.org; 29 Aug 2025 07:35:29 +0000 Received: from localhost ([127.0.0.1]:41618 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1urtdw-0005Km-SE for submit@debbugs.gnu.org; Fri, 29 Aug 2025 03:35:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46390) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1urtds-0005HY-Je for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 03:35:26 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1urtdm-0003WI-7U; Fri, 29 Aug 2025 03:35:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=ID1VwVDLnf76qkbZ7tZ7MHCP4ng6cAvOLlFVhfSx0Do=; b=Y3+ubbAXrFjA Slt4stwicUmpZ7NnYJ60aL4uP1cPQyUe7jgM4fhw0UlC4wvulw7kkkrALHgjBLsvoPR4W2m6AW6sk UXXGKBUW87Hdlk7XPxgE31noBvmNV2enfbbjCRcRvti3zsRdSTHWXMhFXhKGTH2XfjuOYOcFCXniN +ixhIduHQtV89Thsan3BOlSDskQoAquhLjRqMPQg4iGvFxptw4VCuZUNJ2LLkySZNGYG9oE4BevmG mgv+IdIVvP+j+MbxYjMqngzW/eCKY8P5J6ZhUC5Q/tOssaK54aDBEw8gfq6BLKU6J1hYZwnf7Oent DGpLOubTJY7v1yGMwCJCVQ==; Date: Fri, 29 Aug 2025 10:35:15 +0300 Message-Id: <86h5xqshqk.fsf@gnu.org> From: Eli Zaretskii To: Spencer Baugh In-Reply-To: (message from Spencer Baugh on Thu, 28 Aug 2025 15:45:18 -0400) Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads References: X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > Cc: Eli Zaretskii , Dmitry Gutov > From: Spencer Baugh > Date: Thu, 28 Aug 2025 15:45:18 -0400 > > > 1. emacs -Q > 2. Eval this: > > (define-advice shell-command-sentinel (:before (process signal)) > (message "process thread: %s, current thread: %s" > (process-thread process) > (current-thread))) > (make-thread > (lambda () > (async-shell-command "sleep 2") > ;; Do nothing. > (thread-join (make-thread (lambda () (while t (sit-for 1))))))) > (sit-for 3) > > 3. It hangs due to some thread bug, independent of the main bug I'm > reporting here. Just hit C-g. (Notably, it doesn't hang when run in > emacs --batch) > > 4. Observe in *Messages* a message like this: > > process thread: #, current thread: # > > The sentinel ran in a thread which is not process-thread. I believe that's because of the way we process SIGCHLD on Posix systems: we write to a special file descriptor to wake pselect. That file descriptor is used by all the subprocesses, so it cannot be made thread-specific. See child_signal_init and child_signal_notify. Since this descriptor is shared by all the subprocesses, the sentinel on Posix systems can run in the context of some random thread that succeeds to grab the global lock after it returns from pselect. Making sentinels run in the context of the thread that started the process would require to redesign this part of Emacs. Until then, we will need to document this subtlety. > So even after c93be71e45, processes aren't actually locked to threads. Only the sentinel is not locked. > (Once again, I think we should take the opportunity here to delete the > code for locking processes to threads, since IMO it is not useful, and > it is still broken) I did explain why it is useful, and you haven't brought up any arguments to the contrary. And if "broken" means that the sentinel can run in the context of a random thread, then how come you are asking to leave it in this broken state? From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 29 08:55:51 2025 Received: (at 79333) by debbugs.gnu.org; 29 Aug 2025 12:55:51 +0000 Received: from localhost ([127.0.0.1]:42050 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1urydy-0004Aa-5Y for submit@debbugs.gnu.org; Fri, 29 Aug 2025 08:55:51 -0400 Received: from mxout5.mail.janestreet.com ([64.215.233.18]:53707) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1urydv-00049W-6U for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 08:55:48 -0400 From: Spencer Baugh To: Eli Zaretskii Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads In-Reply-To: <86h5xqshqk.fsf@gnu.org> (Eli Zaretskii's message of "Fri, 29 Aug 2025 10:35:15 +0300") References: <86h5xqshqk.fsf@gnu.org> Date: Fri, 29 Aug 2025 08:55:41 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1756472141; bh=YOAAT7BZX3QiJpPrTxEdv1V7dIAznHQvvjh8PN2zZMk=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=Vu0CDmOSyhZCs00rAk1EXqAYpDk3Uw0FGS0S0cC/tbu//OhGVxK2/8dyGzIqUub1o ZKE4CqS5tpOepgOqdfFIWn50oIfmUqT99y5nWuVuzrEI2+D0V/twlPku0gpGCvvRFN vAg75XNg3DU6TyJI3uogjilsGUxKxCVQ5Txn/QGoy7w95iM84rHcGBESNXeZ5vpWso lpM8mptUGK4BzpRb2Ybz8GucjMYWMWkNFOd+afsXTykkf74NDpZHeNQdchLDGuRUiZ UKEgFrOpz6fzzGyJNEaKb9K+EJEGnzbdpwaOaKXP3xwMh8XkX019ORDdYt4XB3CjF0 x7kwGF07UhZlA== X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Eli Zaretskii writes: >> Cc: Eli Zaretskii , Dmitry Gutov >> From: Spencer Baugh >> Date: Thu, 28 Aug 2025 15:45:18 -0400 >> >> >> 1. emacs -Q >> 2. Eval this: >> >> (define-advice shell-command-sentinel (:before (process signal)) >> (message "process thread: %s, current thread: %s" >> (process-thread process) >> (current-thread))) >> (make-thread >> (lambda () >> (async-shell-command "sleep 2") >> ;; Do nothing. >> (thread-join (make-thread (lambda () (while t (sit-for 1))))))) >> (sit-for 3) >> >> 3. It hangs due to some thread bug, independent of the main bug I'm >> reporting here. Just hit C-g. (Notably, it doesn't hang when run in >> emacs --batch) >> >> 4. Observe in *Messages* a message like this: >> >> process thread: #, current thread: # >> >> The sentinel ran in a thread which is not process-thread. > > I believe that's because of the way we process SIGCHLD on Posix > systems: we write to a special file descriptor to wake pselect. That > file descriptor is used by all the subprocesses, so it cannot be made > thread-specific. See child_signal_init and child_signal_notify. > > Since this descriptor is shared by all the subprocesses, the sentinel > on Posix systems can run in the context of some random thread that > succeeds to grab the global lock after it returns from pselect. > > Making sentinels run in the context of the thread that started the > process would require to redesign this part of Emacs. Until then, we > will need to document this subtlety. Makes sense. >> So even after c93be71e45, processes aren't actually locked to threads. > > Only the sentinel is not locked. The filter can also be run in a different thread, because it's run for any remaining output at the time the process terminates. This code demonstrates that: (define-advice shell-command-sentinel (:before (process signal)) (message "process thread: %s, current thread: %s" (process-thread process) (current-thread))) (define-advice comint-output-filter (:before (process string)) (message "filter: %s, current thread: %s" (process-thread process) (current-thread))) (make-thread (lambda () (async-shell-command "sleep 2 && echo hi") ;; Do nothing. (thread-join (make-thread (lambda () (while t (sit-for 1))))))) (sit-for 3) So, the following are unlocked: - calls into the filter triggered by process state changes - calls into the sentinel And the following are locked: - other calls into the filter Seems hard to document. >> (Once again, I think we should take the opportunity here to delete the >> code for locking processes to threads, since IMO it is not useful, and >> it is still broken) > > I did explain why it is useful, and you haven't brought up any > arguments to the contrary. And if "broken" means that the sentinel > can run in the context of a random thread, then how come you are > asking to leave it in this broken state? I was working on an example to demonstrate how process locking can cause problems for unrelated Lisp code when I found this bug. Here's a finished example of the problem. Suppose I have the following Lisp program which doesn't use threads: (run-at-time .1 nil #'async-shell-command "sleep 1 && echo foobar && sleep inf") (with-current-buffer (get-buffer-create shell-command-buffer-name-async) (while (string-empty-p (buffer-string)) (message "waiting for some process output") (sit-for 1)) (message "buffer contents: %s" (buffer-string))) This is intended to represent some arbitrary package which calls make-process in a timer or a hook. The command "sleep 1 && echo foobar && sleep inf" is chosen to represent some interactive executable like a shell or REPL. This code runs just fine, with the output appearing in the buffer as expected. Now suppose I have some other unrelated package which runs the following code using threads: (make-thread (lambda () (accept-process-output nil 1) (thread-join (make-thread (lambda () (while t (message "doing work") (sit-for 10))))))) This is intended to represent thread 1 waiting for some other thread 2 to complete some long-running task. Crucially, thread 1 is blocked in thread-join, which doesn't run wait_reading_process_output, so thread 1 won't read output from any locked processes. Running the second piece of code hangs the first piece of code, even though neither of them are buggy, and they aren't visibly interacting, and they can be written by totally different authors. Specifically: The timer can be run in the thread created by the second piece of code, and then the process will be created locked to that thread. Then the process's output will never be read by thread 1. So the first piece of code will hang. Here's a complete example, with some logging. (Note that the problem doesn't occur on every run because it's a race condition.) (define-advice start-process (:filter-return (proc)) (message "start-process: %s" (process-thread proc)) proc) (run-at-time .5 nil #'async-shell-command "sleep 1 && echo foobar && sleep inf") (make-thread (lambda () (accept-process-output nil 1) (thread-join (make-thread (lambda () (while t (message "doing work") (sit-for 10))))))) (with-current-buffer (get-buffer-create shell-command-buffer-name-async) (while (string-empty-p (buffer-string)) (message "waiting for some process output") (sit-for 1)) (message "buffer contents: %s" (buffer-string))) This isn't just an academic problem. Most Lisp programs that start processes will run into this problem if there are any Lisp threads running. One way to fix this would be to make it so that thread-join and condition-wait both call wait_reading_process_output. Then there wouldn't be a way to block a thread without calling wait_reading_process_output. Unfortunately, that's difficult to do in a portable way, because we would need to integrate waiting for a condition variable into the wait_reading_process_output event loop, which is impossible on GNU/Linux. Another way to fix this would be to make timers run only in the thread which started them. However, this is insufficient, because the same problem can occur with hooks. Any hook can be run by an unrelated thread, and processes started in that hook may hang if the thread is doing work which doesn't involve calling wait_reading_process_output. As far as I can tell, the only possible fix for this problem is to not lock processes to threads. This problem seems worse than the problems prevented by locking processes to threads, so I think this is the right fix. (Especially because, as the original bug demonstrates, we aren't fully locking processes to threads, neither sentinels nor filters, so we aren't actually getting the benefits of that locking, only the costs) From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 29 09:19:47 2025 Received: (at 79333) by debbugs.gnu.org; 29 Aug 2025 13:19:47 +0000 Received: from localhost ([127.0.0.1]:42122 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1urz17-0007UO-Lw for submit@debbugs.gnu.org; Fri, 29 Aug 2025 09:19:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46714) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1urz11-0007So-Ur for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 09:19:44 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1urz0u-00079Q-Qb; Fri, 29 Aug 2025 09:19:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=EDpax77tW/uh90TzpcJwbQh5NTDvatEIhVDBHsehChE=; b=V6rP5FZYKPBu ml2KDn7Wy0cPZnVX4y06rhSYydmswtxFrnRCBDDGRzcjEJgek3B0+v3uqIeGySgMe/mnwmGZU2Uhx y+Dn78k7+Xo8ggPDWvpjndLNLq8COMFCh2NcAR27utIUn206Uib+tf6p0/irK7QjpN8hB1gxYcSwm bmHhuWCPNWNsAsq71NUbbytR7W40ib3SPaeXQ2ckdZ7c1J28jinsjSxZVOCmS3eaWAz4fx2SvZGic a9Vci9D08+PIltGyJpjwqPLEGn+7GTlMDUh4ELx8LvzppZyN/qJU5DAUWsjmMb4UCZfnZbWVGtow2 CLX4W73bd06+KM1IhfuJ6Q==; Date: Fri, 29 Aug 2025 16:19:30 +0300 Message-Id: <86bjnys1st.fsf@gnu.org> From: Eli Zaretskii To: Spencer Baugh In-Reply-To: (message from Spencer Baugh on Fri, 29 Aug 2025 08:55:41 -0400) Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads References: <86h5xqshqk.fsf@gnu.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Spencer Baugh > Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org > Date: Fri, 29 Aug 2025 08:55:41 -0400 > > Eli Zaretskii writes: > > > Making sentinels run in the context of the thread that started the > > process would require to redesign this part of Emacs. Until then, we > > will need to document this subtlety. > > Makes sense. > > >> So even after c93be71e45, processes aren't actually locked to threads. > > > > Only the sentinel is not locked. > > The filter can also be run in a different thread, because it's run for > any remaining output at the time the process terminates. > > This code demonstrates that: In the discussion of bug#79334, I suggested to make a change in status_notify which should prevent that, as long as the thread to which the process is locked is alive. > >> (Once again, I think we should take the opportunity here to delete the > >> code for locking processes to threads, since IMO it is not useful, and > >> it is still broken) > > > > I did explain why it is useful, and you haven't brought up any > > arguments to the contrary. And if "broken" means that the sentinel > > can run in the context of a random thread, then how come you are > > asking to leave it in this broken state? > > I was working on an example to demonstrate how process locking can cause > problems for unrelated Lisp code when I found this bug. > > Here's a finished example of the problem. > > Suppose I have the following Lisp program which doesn't use threads: > > (run-at-time .1 nil #'async-shell-command "sleep 1 && echo foobar && sleep inf") > (with-current-buffer (get-buffer-create shell-command-buffer-name-async) > (while (string-empty-p (buffer-string)) > (message "waiting for some process output") > (sit-for 1)) > (message "buffer contents: %s" (buffer-string))) > > This is intended to represent some arbitrary package which calls > make-process in a timer or a hook. The command "sleep 1 && echo foobar > && sleep inf" is chosen to represent some interactive executable like a > shell or REPL. > > This code runs just fine, with the output appearing in the buffer as > expected. > > Now suppose I have some other unrelated package which runs the following > code using threads: > > (make-thread > (lambda () > (accept-process-output nil 1) > (thread-join (make-thread (lambda () (while t (message "doing work") (sit-for 10))))))) > > This is intended to represent thread 1 waiting for some other thread 2 > to complete some long-running task. Crucially, thread 1 is blocked in > thread-join, which doesn't run wait_reading_process_output, so thread 1 > won't read output from any locked processes. > > Running the second piece of code hangs the first piece of code, even > though neither of them are buggy, and they aren't visibly interacting, > and they can be written by totally different authors. > > Specifically: The timer can be run in the thread created by the second > piece of code, and then the process will be created locked to that > thread. Then the process's output will never be read by thread 1. So > the first piece of code will hang. I don't understand why one thread starts a process, then another thread waits for its output, and the program which arranges for that doesn't unlock the process so the other thread could do its job. This is what set-process-thread is for, and in this (IMO rather unusual) arrangement, calling it with a nil THREAD argument is exactly what should be done. > As far as I can tell, the only possible fix for this problem is to not > lock processes to threads. No, the fix is for the program to unlock the process using set-process-thread. Did you try that, and if so, did it help? > (Especially because, as the original bug demonstrates, we aren't fully > locking processes to threads, neither sentinels nor filters, so we > aren't actually getting the benefits of that locking, only the costs) If we want programs using threads to be more deterministic and predictable, we need to beef up the locking, not throw it away. At least that is my conclusion from all these discussions, and the above doesn't contradict it, at least not yet. From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 29 09:29:50 2025 Received: (at 79333) by debbugs.gnu.org; 29 Aug 2025 13:29:50 +0000 Received: from localhost ([127.0.0.1]:42149 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1urzAp-0000at-3Z for submit@debbugs.gnu.org; Fri, 29 Aug 2025 09:29:49 -0400 Received: from mxout5.mail.janestreet.com ([64.215.233.18]:49717) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1urzAl-0000Zh-8M for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 09:29:44 -0400 From: Spencer Baugh To: Eli Zaretskii Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads In-Reply-To: <86bjnys1st.fsf@gnu.org> (Eli Zaretskii's message of "Fri, 29 Aug 2025 16:19:30 +0300") References: <86h5xqshqk.fsf@gnu.org> <86bjnys1st.fsf@gnu.org> Date: Fri, 29 Aug 2025 09:29:37 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1756474177; bh=Z2PVNCR5Fl10imehOLqwyJvCBiA5KuS/u88ozclDSgo=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=rn17T5Ml3lLrqMANL6FACGbJIfbubwe3ZQsFHoDDy6WtbW4ztyhJoZBtcVXJicdL0 PrFKEmLCOdWyImuxHty+a6XWRgFoiCJSNoyba/MCZI6d1IB/vblXmBQtXG/l940fIB 0wJzCULvtf+JE6uPy6ZFiy7v6hqosfMntMmu7MxDwpwNWPPpH2axbigFygEJksW8L2 HohqEUYXdU7jR9cRmfSTwl44GD80tUvlC8I/YgXwnDgi6OsKnHpvUCo2jX8BCA1TG0 cgU+Lo4hxMKhJt0RL39KR6xJau5Ky4mLVC5RKBucSP6r59UQtoi3/jZOmLSu4Vi0O1 b5DAAT8VXuk1g== X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Eli Zaretskii writes: >> >> (Once again, I think we should take the opportunity here to delete the >> >> code for locking processes to threads, since IMO it is not useful, and >> >> it is still broken) >> > >> > I did explain why it is useful, and you haven't brought up any >> > arguments to the contrary. And if "broken" means that the sentinel >> > can run in the context of a random thread, then how come you are >> > asking to leave it in this broken state? >> >> I was working on an example to demonstrate how process locking can cause >> problems for unrelated Lisp code when I found this bug. >> >> Here's a finished example of the problem. >> >> Suppose I have the following Lisp program which doesn't use threads: >> >> (run-at-time .1 nil #'async-shell-command "sleep 1 && echo foobar && sleep inf") >> (with-current-buffer (get-buffer-create shell-command-buffer-name-async) >> (while (string-empty-p (buffer-string)) >> (message "waiting for some process output") >> (sit-for 1)) >> (message "buffer contents: %s" (buffer-string))) >> >> This is intended to represent some arbitrary package which calls >> make-process in a timer or a hook. The command "sleep 1 && echo foobar >> && sleep inf" is chosen to represent some interactive executable like a >> shell or REPL. >> >> This code runs just fine, with the output appearing in the buffer as >> expected. >> >> Now suppose I have some other unrelated package which runs the following >> code using threads: >> >> (make-thread >> (lambda () >> (accept-process-output nil 1) >> (thread-join (make-thread (lambda () (while t (message "doing work") (sit-for 10))))))) >> >> This is intended to represent thread 1 waiting for some other thread 2 >> to complete some long-running task. Crucially, thread 1 is blocked in >> thread-join, which doesn't run wait_reading_process_output, so thread 1 >> won't read output from any locked processes. >> >> Running the second piece of code hangs the first piece of code, even >> though neither of them are buggy, and they aren't visibly interacting, >> and they can be written by totally different authors. >> >> Specifically: The timer can be run in the thread created by the second >> piece of code, and then the process will be created locked to that >> thread. Then the process's output will never be read by thread 1. So >> the first piece of code will hang. > > I don't understand why one thread starts a process, then another > thread waits for its output, and the program which arranges for that > doesn't unlock the process so the other thread could do its job. Yes, that's the bug. It is not intentional that the process is started in a thread. That is what causes the bug. > This is what set-process-thread is > for, and in this (IMO rather unusual) arrangement, calling it with a > nil THREAD argument is exactly what should be done. The point is that these are two independent pieces of code, written by different authors. If they just happen to interleave in this way, then the process is *accidentally, unintentionally* started in a thread. How would the author of snippet 1 know to call set-process-thread in this case? From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 29 09:55:43 2025 Received: (at 79333) by debbugs.gnu.org; 29 Aug 2025 13:55:44 +0000 Received: from localhost ([127.0.0.1]:42229 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1urzZt-0004Kn-Fz for submit@debbugs.gnu.org; Fri, 29 Aug 2025 09:55:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52674) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1urzZo-0004Iv-1o for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 09:55:39 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1urzZh-00022p-Qk; Fri, 29 Aug 2025 09:55:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=26bdcOzpHywwI9DzKfkdLYKfmTeaEATVuVkQ+/CCqAY=; b=EWqbLEJHQsPv 1m186Qgx+82d6SiEV/iKLeHQ/Dkd1eL7GBs1o2UDMiIdJ29KDnA2i4JVpZFKXgmev+r7vA8mNNHrw j/m7vr0k434RbhICovBF2STMlOp84hZ6ZetJJ89gEo1fBCA8QQ6tk30ww28GoIgA9IsYl1zhY084P Ia1c6+LE6M9Q8u/tC+cPfGEAdCzqmadH7Tsx4+hCDq64RmEdX7S4Rjgx0bHg0Qyopl8d9ztLbN1nx 6sFtiLQqGU1Bd1unjpCsvI+TTB0XkY5QVympmqhODr0rPupzZCVGy2L95VyCy0X/c4wtg05Ezos3r vsgYhT42jHr7yaPROwiFvg==; Date: Fri, 29 Aug 2025 16:55:19 +0300 Message-Id: <868qj2s054.fsf@gnu.org> From: Eli Zaretskii To: Spencer Baugh In-Reply-To: (message from Spencer Baugh on Fri, 29 Aug 2025 09:29:37 -0400) Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads References: <86h5xqshqk.fsf@gnu.org> <86bjnys1st.fsf@gnu.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Spencer Baugh > Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org > Date: Fri, 29 Aug 2025 09:29:37 -0400 > > > I don't understand why one thread starts a process, then another > > thread waits for its output, and the program which arranges for that > > doesn't unlock the process so the other thread could do its job. > > Yes, that's the bug. > > It is not intentional that the process is started in a thread. That is > what causes the bug. > > > This is what set-process-thread is > > for, and in this (IMO rather unusual) arrangement, calling it with a > > nil THREAD argument is exactly what should be done. > > The point is that these are two independent pieces of code, written by > different authors. > > If they just happen to interleave in this way, then the process is > *accidentally, unintentionally* started in a thread. > > How would the author of snippet 1 know to call set-process-thread in > this case? If you are saying that two arbitrary independently-written pieces of code can get in trouble if they are lumped together to run by the same Lisp program in two separate threads, then I agree. However, having a function that starts a process, but doesn't process its output, and another function that doesn't start any processes, but does accept output from subprocesses, is an unusual thing to do. This could happen as a deliberate design of a program, but then we are not talking about two snippets oblivious to one another, because the person who brings them together like that in the same program does that deliberately, and should understand that for it to work, the process should be either unlocked or locked to the thread which wants to read and process its output. IOW, when making such programs where threads are not independent calls for some adjustments in the code of each thread. What I have in mind is a different case, which I think is much more common, at least at this stage of using Lisp thread in Emacs. It's a case where one takes a single-threaded Lisp program, and runs it from a separate thread so as to avoid blocking the Emacs's main thread. In that case, the same thread will both start the process and expect to be able to process its output (because that's how single-threaded Lisp programs work), and therefore having the process locked by default lets such code work as expected when it is run from a thread. Especially if you take several such programs, each with its own subprocess, and let them all run from several different threads at the same time. From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 29 11:20:41 2025 Received: (at 79333) by debbugs.gnu.org; 29 Aug 2025 15:20:41 +0000 Received: from localhost ([127.0.0.1]:42879 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1us0u9-000805-3F for submit@debbugs.gnu.org; Fri, 29 Aug 2025 11:20:41 -0400 Received: from mxout5.mail.janestreet.com ([64.215.233.18]:48061) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1us0u6-0007z3-8M for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 11:20:39 -0400 From: Spencer Baugh To: Eli Zaretskii Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads In-Reply-To: <868qj2s054.fsf@gnu.org> (Eli Zaretskii's message of "Fri, 29 Aug 2025 16:55:19 +0300") References: <86h5xqshqk.fsf@gnu.org> <86bjnys1st.fsf@gnu.org> <868qj2s054.fsf@gnu.org> Date: Fri, 29 Aug 2025 11:20:32 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1756480832; bh=YsgvnRxDsue5HUefIygZUnxdr3Jm78SJAhPwKLEDVWA=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=TkdvCQIXnjGhBd2iVEy+DiCyQMdgWubC8rziync975QCfLtfqaYkrEJoVjwfx2vt1 AiyqDIwE6AmB3TV1/vTBNNObftlXPpuZOvrl1L3Yn78JYDzmFgyMLyD2TWuMXgqnFH Wyc/rQ6NaVMDyhHn+RNOR+eSckAZRemx5tXOf84yYjp/W8QpYIq8jvlS1C9O8Xkll6 t5B80/qRY1xpooImNbO5yTGOEAq4OQI7WCt4YC6pjzK1hr+twQa0Ow7hLL8U0H+hdx pe5D5cnb0sUmzWDzQ/g7vAKUGokv9D6UYfaPS4v1FxauhkkCfJ8fToycFemj117FO0 8jPT4AE8Xb/5Q== X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Eli Zaretskii writes: >> From: Spencer Baugh >> Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org >> Date: Fri, 29 Aug 2025 09:29:37 -0400 >> >> > I don't understand why one thread starts a process, then another >> > thread waits for its output, and the program which arranges for that >> > doesn't unlock the process so the other thread could do its job. >> >> Yes, that's the bug. >> >> It is not intentional that the process is started in a thread. That is >> what causes the bug. >> >> > This is what set-process-thread is >> > for, and in this (IMO rather unusual) arrangement, calling it with a >> > nil THREAD argument is exactly what should be done. >> >> The point is that these are two independent pieces of code, written by >> different authors. >> >> If they just happen to interleave in this way, then the process is >> *accidentally, unintentionally* started in a thread. >> >> How would the author of snippet 1 know to call set-process-thread in >> this case? > > If you are saying that two arbitrary independently-written pieces of > code can get in trouble if they are lumped together to run by the same > Lisp program in two separate threads, then I agree. I guess that's what I'm saying. But the Lisp program here is just "Emacs". This combination of two independent pieces of code just automatically happens when users is using one package which is using timers, and another package which is using threads. Which of course happens all the time without anyone choosing to do it. For example, one package might add a find-file-hook which starts a subprocess, then another package might add a find-file-hook which starts a thread. Then when the two hooks run in succession, it would cause this problem. > However, having a function that starts a process, but doesn't process > its output, and another function that doesn't start any processes, but > does accept output from subprocesses, is an unusual thing to do. Ah, I guess you're referring to the explicit accept-process-output call. I think that was a confusing part of my example, because it was not necessary to cause the issue. Here's a more refined example: ;; Package 1 (perhaps run in a find-file-hook) (run-at-time .3 nil #'async-shell-command "sleep 1 && echo foobar && sleep inf") ;; Package 2 (perhaps run in a find-file-hook) (make-thread (lambda () (sit-for 1) (thread-join (make-thread (lambda () (while t (sit-for 1))))))) The shell command started by package 1 will sometimes hang forever without producing output. > This could happen as a deliberate design of a program, but then we are > not talking about two snippets oblivious to one another, because the > person who brings them together like that in the same program does > that deliberately, and should understand that for it to work, the > process should be either unlocked or locked to the thread which wants > to read and process its output. > > IOW, when making such programs where threads are not independent calls > for some adjustments in the code of each thread. > > What I have in mind is a different case, which I think is much more > common, at least at this stage of using Lisp thread in Emacs. It's a > case where one takes a single-threaded Lisp program, and runs it from > a separate thread so as to avoid blocking the Emacs's main thread. In > that case, the same thread will both start the process and expect to > be able to process its output (because that's how single-threaded Lisp > programs work), and therefore having the process locked by default > lets such code work as expected when it is run from a thread. > Especially if you take several such programs, each with its own > subprocess, and let them all run from several different threads at the > same time. Yes, I definitely want that real-world case to work right. I agree that that is a very important case. But I think it already works right with processes not locked to threads for any non-buggy program. (I personally have written or used lots of code like that with threads, and the fact that processes were not fully locked to threads did not cause problems. As one public example, diff-hl-mode's diff-hl-update-async) For example, a program like this will work correctly even in a thread: (let ((proc (make-process ...))) (accept-process-output proc)) This would run the filter functions in the same thread, because output from PROC can't be read by another thread until we do a thread switch, which will only happen when we call accept-process-output. If the program was instead something like: (let ((proc (make-process ...))) (sit-for 1) (accept-process-output proc)) then the (accept-process-output proc) might block because the sit-for can thread switch. But this program is already buggy, since sit-for runs wait_reading_process_output which could read the output from PROC. From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 29 11:54:03 2025 Received: (at 79333) by debbugs.gnu.org; 29 Aug 2025 15:54:03 +0000 Received: from localhost ([127.0.0.1]:42941 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1us1QO-0004MI-VR for submit@debbugs.gnu.org; Fri, 29 Aug 2025 11:54:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45314) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1us1QL-0004Kw-MP for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 11:53:58 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1us1QF-0005aR-8p; Fri, 29 Aug 2025 11:53:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=eK2wHDQ/+nXe+3JYxRyLbL8Tqn0h0+rKvZQPim5zoGU=; b=B9MgPHRUm+ql xF4vrqbUxXi7qczvCoXFNADq3/JTCzY5OVOVYaHsEJHlG1lUUFTO9z634kwpA4CELtVp5zBxzeLkh omjFu7VItQcwDQVFrqUNcxjrS6zHUqRj5xdOYUI3unaxplgd48Gcpz+R6I030qXVhallesvedJJDX t6ikQnpUO7C6td2rdBGpiFwF1DHpjVs1hF5qH5SVIYVGpCGbCrgLw01Sc6NMt8wkkp9EzCrSd7JKb SmrHpleKxXGfDYkUrZA/XDYs4sfLy85sPACGIGnf2CzGRnZXtCsCcR6Gc4eDgeBinIz88V7MKiV4V /u+8Ec2DJCbHpHz+dHKMfQ==; Date: Fri, 29 Aug 2025 18:53:49 +0300 Message-Id: <864itqrunm.fsf@gnu.org> From: Eli Zaretskii To: Spencer Baugh In-Reply-To: (message from Spencer Baugh on Fri, 29 Aug 2025 11:20:32 -0400) Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads References: <86h5xqshqk.fsf@gnu.org> <86bjnys1st.fsf@gnu.org> <868qj2s054.fsf@gnu.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Spencer Baugh > Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org > Date: Fri, 29 Aug 2025 11:20:32 -0400 > > Eli Zaretskii writes: > > > If you are saying that two arbitrary independently-written pieces of > > code can get in trouble if they are lumped together to run by the same > > Lisp program in two separate threads, then I agree. > > I guess that's what I'm saying. But the Lisp program here is just > "Emacs". This combination of two independent pieces of code just > automatically happens when users is using one package which is using > timers, and another package which is using threads. Which of course > happens all the time without anyone choosing to do it. > > For example, one package might add a find-file-hook which starts a > subprocess, then another package might add a find-file-hook which starts > a thread. Then when the two hooks run in succession, it would cause > this problem. It's possible that we should have some guidelines for such situations. But this is way far in the future, from where I stand: right now, taking some processing, which works single-threaded and making it run from a separate thread doesn't work well, and we should first make sure that's solved. > > However, having a function that starts a process, but doesn't process > > its output, and another function that doesn't start any processes, but > > does accept output from subprocesses, is an unusual thing to do. > > Ah, I guess you're referring to the explicit accept-process-output call. > I think that was a confusing part of my example, because it was not > necessary to cause the issue. > > Here's a more refined example: > > ;; Package 1 (perhaps run in a find-file-hook) > (run-at-time .3 nil #'async-shell-command "sleep 1 && echo foobar && sleep inf") > ;; Package 2 (perhaps run in a find-file-hook) > (make-thread > (lambda () > (sit-for 1) > (thread-join (make-thread (lambda () (while t (sit-for 1))))))) > > The shell command started by package 1 will sometimes hang forever > without producing output. I believe this is because of that issue with status_notify. At least, we should fix that before we revisit the above and see if anything else needs to be fixed there. > > What I have in mind is a different case, which I think is much more > > common, at least at this stage of using Lisp thread in Emacs. It's a > > case where one takes a single-threaded Lisp program, and runs it from > > a separate thread so as to avoid blocking the Emacs's main thread. In > > that case, the same thread will both start the process and expect to > > be able to process its output (because that's how single-threaded Lisp > > programs work), and therefore having the process locked by default > > lets such code work as expected when it is run from a thread. > > Especially if you take several such programs, each with its own > > subprocess, and let them all run from several different threads at the > > same time. > > Yes, I definitely want that real-world case to work right. I agree that > that is a very important case. But I think it already works right with > processes not locked to threads for any non-buggy program. That's not my experience. If random threads get return values from accept-process-output, you can easily have a thread whose accept-process-output call never returns until timeout, because the output was already read by another thread. > (I personally have written or used lots of code like that with threads, > and the fact that processes were not fully locked to threads did not > cause problems. And locking them does cause problems? > For example, a program like this will work correctly even in a thread: > > (let ((proc (make-process ...))) > (accept-process-output proc)) > > This would run the filter functions in the same thread, because output > from PROC can't be read by another thread until we do a thread switch, > which will only happen when we call accept-process-output. What matters is which thread gets first to the pselect call. That's unpredictable, because it's racy. > If the program was instead something like: > > (let ((proc (make-process ...))) > (sit-for 1) > (accept-process-output proc)) > > then the (accept-process-output proc) might block because the sit-for > can thread switch. But this program is already buggy, since sit-for > runs wait_reading_process_output which could read the output from PROC. A program can easily call sit-for indirectly, because sit-for is called all over the place in Emacs. This is why locking processes is better: it makes the program more predictable. From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 29 12:07:21 2025 Received: (at 79333) by debbugs.gnu.org; 29 Aug 2025 16:07:21 +0000 Received: from localhost ([127.0.0.1]:42965 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1us1dH-0006OQ-MZ for submit@debbugs.gnu.org; Fri, 29 Aug 2025 12:07:20 -0400 Received: from mxout5.mail.janestreet.com ([64.215.233.18]:37287) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1us1dE-0006NH-So for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 12:07:18 -0400 Received: from mail-lj1-f197.google.com ([209.85.208.197]) by mxgoog2.mail.janestreet.com with esmtps (TLS1.3:TLS_AES_128_GCM_SHA256:128) (Exim 4.98.2) id 1us1d8-0000000218H-3GCV for 79333@debbugs.gnu.org; Fri, 29 Aug 2025 12:07:10 -0400 Received: by mail-lj1-f197.google.com with SMTP id 38308e7fff4ca-336a33789edso9627791fa.1 for <79333@debbugs.gnu.org>; Fri, 29 Aug 2025 09:07:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; t=1756483629; x=1757088429; darn=debbugs.gnu.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=xmcqJs2q1UuWK3jUbxBZ0IZA+aHxqW5AQ2gau0V7oxc=; b=PW086ofWvUAPARJv6j9BTP535t7E4MSc7vyVpTYxKl36Q+7Xwv5w9XpTrLtX7pIGmy xL42i0GjhTqWGqUGLQdQva9EKhP6Rtv0AXjntY4bkiNyJ/FIOllbR00uupKDh7LMXJNC Hnf1c2IutxVSxlh2XiQLjiscEmk4V4kkDL0/U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1756483630; bh=xmcqJs2q1UuWK3jUbxBZ0IZA+aHxqW5AQ2gau0V7oxc=; h=References:In-Reply-To:From:Date:Subject:To:Cc; b=aFYxI82qN3BCB4K1xtFtZpgweAQrmeeYltoDUDxJSlaXpsjnpSwoNDflae0xbDGjX SkOzt/cUHNVf7aG4XjR0nX7iWkeXF/GeA5emRHJ0ZcB//djJeZq7uJDhyCWbIVsxkm 7q+X83hlekqLOkxwrXCaUVb7y9y2+DDh+WVRaxaIavkIUCRcfg5FO+wQohEmBZHHlU S0/WNdG9RMigpr13MW6oOKZ0dQ84FuLF4iXXcNXDwDKY3en0tQdt0vW/kCo4jKij1D UVl7Ry8T9thl8TvGXk8A7dHrRY3yLMcqaWLEJ3WD/P/8IprmNTI9Mout3kOWJRIfxF gxYOZULkiPEDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756483629; x=1757088429; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xmcqJs2q1UuWK3jUbxBZ0IZA+aHxqW5AQ2gau0V7oxc=; b=fa6euLIrz+fhD454rD1cnxGc8lfl5WpI0KsPtPL0DaaSOeX6CKLERrk/EC3Jc9qX6t 4YZ3Ljy/+k7ZplTbnf06KOPzA/1tXndHuwg2ZLTFWiSyVyhgwTPKiMGyTw/xby9EAkmP Bl+NY0WQwYEHq98gkPKIcnzcEOP8PF8h9l2zA/iBJa+wKG0gyJ3jcw1wMnj95HlHzBMz gtROpJ2/RJt8p5fR7G6SlwNnCPZ4FIswYgN76An9VXK8wi6RoCtwVWyDBPbHz/g5fcME OKG7hjV7lOX+Rj8XKerTqboGpmqjiST6Ao35gIzjRIbF19/kDpDW6vUbi+Qw5pPw8DRi q7IA== X-Forwarded-Encrypted: i=1; AJvYcCVG81kNpCj9xv90TPLXiOjx2JSQGTrvOL9mD3lHe6J6dfXRTDKyZ6hqIPeH0wfrsvghD3U7ag==@debbugs.gnu.org X-Gm-Message-State: AOJu0YxvTOynyLD0Hz/kKRM5aqWqJVrhc6oVfhmQkH9uAfHscf0vm1dZ uNoqNXvZCmhzrXjJjEfILZTWelQHwSHgGp8YAiJHncD9suJT37Fs0VWA6Cy6hEGnCE6gr4VPGR0 CJ4Cj51UIAFMzOWq1Ic37aj28yS23OfSI5y2DdpiH36tz5xwdrEk9gMZBSUOc8MwBPNlsBt9Jfx T7ViOJuBaecxJadKUGpguCgh/LU50Nfw== X-Gm-Gg: ASbGncthZVsT/JTOnOk0drm1WNp9iJDdIfyxUETVSrHaQDpCCz8x+g0/xDWkLhaBWSj vbOiMDCwpdCfSuiK39r1S3HmdgdXaATuuC6R9RAPidNIW6FaKNcT1eEHVxxSTx34R4txSYx94a7 vxV/sW2T8pwIIQZ/UU8UDl X-Received: by 2002:a05:6512:2284:b0:55a:4fd2:2908 with SMTP id 2adb3069b0e04-55f0cd46a07mr6745154e87.25.1756483629182; Fri, 29 Aug 2025 09:07:09 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHZNrEPToVEJdiq518R9iIG0DFPyh3rt1ztMQ7MyhNDq7KfUbpKLVmpswT0OLzcUXg0xxMFAvCKvZGUhYApjwA= X-Received: by 2002:a05:6512:2284:b0:55a:4fd2:2908 with SMTP id 2adb3069b0e04-55f0cd46a07mr6745146e87.25.1756483628664; Fri, 29 Aug 2025 09:07:08 -0700 (PDT) MIME-Version: 1.0 References: <86h5xqshqk.fsf@gnu.org> <86bjnys1st.fsf@gnu.org> <868qj2s054.fsf@gnu.org> <864itqrunm.fsf@gnu.org> In-Reply-To: <864itqrunm.fsf@gnu.org> From: Spencer Baugh Date: Fri, 29 Aug 2025 12:06:58 -0400 X-Gm-Features: Ac12FXw-VozHVZtjk1Iu3RH4KzloDcj2jlHNbkzrwYB6Krlx3h26GaceXNirviM Message-ID: Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads To: Eli Zaretskii Content-Type: multipart/alternative; boundary="000000000000bd9e4e063d833609" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 79333 Cc: Dmitry Gutov , 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --000000000000bd9e4e063d833609 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Aug 29, 2025, 11:53=E2=80=AFAM Eli Zaretskii wrote: > > From: Spencer Baugh > > Cc: dmitry@gutov.dev, 79333@debbugs.gnu.org > > Date: Fri, 29 Aug 2025 11:20:32 -0400 > > > > Eli Zaretskii writes: > > > > > If you are saying that two arbitrary independently-written pieces of > > > code can get in trouble if they are lumped together to run by the sam= e > > > Lisp program in two separate threads, then I agree. > > > > I guess that's what I'm saying. But the Lisp program here is just > > "Emacs". This combination of two independent pieces of code just > > automatically happens when users is using one package which is using > > timers, and another package which is using threads. Which of course > > happens all the time without anyone choosing to do it. > > > > For example, one package might add a find-file-hook which starts a > > subprocess, then another package might add a find-file-hook which start= s > > a thread. Then when the two hooks run in succession, it would cause > > this problem. > > It's possible that we should have some guidelines for such situations. > But this is way far in the future, from where I stand: right now, > taking some processing, which works single-threaded and making it run > from a separate thread doesn't work well, and we should first make > sure that's solved. > What would the guidelines be? I don't believe there's any way to fix this problem other than by unlocking every process you create. > > However, having a function that starts a process, but doesn't process > > > its output, and another function that doesn't start any processes, bu= t > > > does accept output from subprocesses, is an unusual thing to do. > > > > Ah, I guess you're referring to the explicit accept-process-output call= . > > I think that was a confusing part of my example, because it was not > > necessary to cause the issue. > > > > Here's a more refined example: > > > > ;; Package 1 (perhaps run in a find-file-hook) > > (run-at-time .3 nil #'async-shell-command "sleep 1 && echo foobar && > sleep inf") > > ;; Package 2 (perhaps run in a find-file-hook) > > (make-thread > > (lambda () > > (sit-for 1) > > (thread-join (make-thread (lambda () (while t (sit-for 1))))))) > > > > The shell command started by package 1 will sometimes hang forever > > without producing output. > > I believe this is because of that issue with status_notify. At least, > we should fix that before we revisit the above and see if anything > else needs to be fixed there. > This bug still happens even with my initial fixes for the status_notify issue. But sure, we can fix that first and then come back to this one. (I want to make sure we don't release Emacs 32 with the change that I believe breaks existing thread programs, but as long as we resolve the issues before then, I'm in no rush. I've just reverted the change at my site anyway) > > What I have in mind is a different case, which I think is much more > > > common, at least at this stage of using Lisp thread in Emacs. It's a > > > case where one takes a single-threaded Lisp program, and runs it from > > > a separate thread so as to avoid blocking the Emacs's main thread. I= n > > > that case, the same thread will both start the process and expect to > > > be able to process its output (because that's how single-threaded Lis= p > > > programs work), and therefore having the process locked by default > > > lets such code work as expected when it is run from a thread. > > > Especially if you take several such programs, each with its own > > > subprocess, and let them all run from several different threads at th= e > > > same time. > > > > Yes, I definitely want that real-world case to work right. I agree tha= t > > that is a very important case. But I think it already works right with > > processes not locked to threads for any non-buggy program. > > That's not my experience. If random threads get return values from > accept-process-output, you can easily have a thread whose > accept-process-output call never returns until timeout, because the > output was already read by another thread. > I know you don't have much time to work on this, but it would really help if you could give a concrete example program that demonstrates this. > (I personally have written or used lots of code like that with threads, > > and the fact that processes were not fully locked to threads did not > > cause problems. > > And locking them does cause problems? > Yes. Such as in the example I was describing above. > For example, a program like this will work correctly even in a thread: > > > > (let ((proc (make-process ...))) > > (accept-process-output proc)) > > > > This would run the filter functions in the same thread, because output > > from PROC can't be read by another thread until we do a thread switch, > > which will only happen when we call accept-process-output. > > What matters is which thread gets first to the pselect call. That's > unpredictable, because it's racy. > It is not racy in this example. Even without locking. > If the program was instead something like: > > > > (let ((proc (make-process ...))) > > (sit-for 1) > > (accept-process-output proc)) > > > > then the (accept-process-output proc) might block because the sit-for > > can thread switch. But this program is already buggy, since sit-for > > runs wait_reading_process_output which could read the output from PROC. > > A program can easily call sit-for indirectly, because sit-for is > called all over the place in Emacs. > That's my point. This second example program is buggy whether threads are used or not. > --000000000000bd9e4e063d833609 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Aug 29, 2025, 11:53=E2= =80=AFAM Eli Zaretskii <eliz@gnu.org= > wrote:
> From: Spencer Baug= h <sbaugh@janestreet.com>
> Cc: dmitry@gutov.dev,=C2=A0 79333@debbugs.gnu.org
> Date: Fri, 29 Aug 2025 11:20:32 -0400
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > If you are saying that two arbitrary independently-written pieces= of
> > code can get in trouble if they are lumped together to run by the= same
> > Lisp program in two separate threads, then I agree.
>
> I guess that's what I'm saying.=C2=A0 But the Lisp program her= e is just
> "Emacs".=C2=A0 This combination of two independent pieces of= code just
> automatically happens when users is using one package which is using > timers, and another package which is using threads.=C2=A0 Which of cou= rse
> happens all the time without anyone choosing to do it.
>
> For example, one package might add a find-file-hook which starts a
> subprocess, then another package might add a find-file-hook which star= ts
> a thread.=C2=A0 Then when the two hooks run in succession, it would ca= use
> this problem.

It's possible that we should have some guidelines for such situations.<= br> But this is way far in the future, from where I stand: right now,
taking some processing, which works single-threaded and making it run
from a separate thread doesn't work well, and we should first make
sure that's solved.

<= /div>
What would the guidelines be?=C2=A0 I don't beli= eve there's any way to fix this problem other than by unlocking every p= rocess you create.

> > However, having a function that starts a process, but doesn't= process
> > its output, and another function that doesn't start any proce= sses, but
> > does accept output from subprocesses, is an unusual thing to do.<= br> >
> Ah, I guess you're referring to the explicit accept-process-output= call.
> I think that was a confusing part of my example, because it was not > necessary to cause the issue.
>
> Here's a more refined example:
>
> ;; Package 1 (perhaps run in a find-file-hook)
> (run-at-time .3 nil #'async-shell-command "sleep 1 &&= echo foobar && sleep inf")
> ;; Package 2 (perhaps run in a find-file-hook)
> (make-thread
>=C2=A0 (lambda ()
>=C2=A0 =C2=A0 (sit-for 1)
>=C2=A0 =C2=A0 (thread-join (make-thread (lambda () (while t (sit-for 1)= ))))))
>
> The shell command started by package 1 will sometimes hang forever
> without producing output.

I believe this is because of that issue with status_notify.=C2=A0 At least,=
we should fix that before we revisit the above and see if anything
else needs to be fixed there.

This bug still happens even with my initial fi= xes for the status_notify issue.=C2=A0 But sure, we can fix that first and = then come back to this one.

(I want to make sure we don't release Emacs 32 with the change that= I believe breaks existing thread programs, but as long as we resolve the i= ssues before then, I'm in no rush.=C2=A0 I've just reverted the cha= nge at my site anyway)

<= div class=3D"gmail_quote gmail_quote_container">
> > What I have in mind is a different case, which I think is much mo= re
> > common, at least at this stage of using Lisp thread in Emacs.=C2= =A0 It's a
> > case where one takes a single-threaded Lisp program, and runs it = from
> > a separate thread so as to avoid blocking the Emacs's main th= read.=C2=A0 In
> > that case, the same thread will both start the process and expect= to
> > be able to process its output (because that's how single-thre= aded Lisp
> > programs work), and therefore having the process locked by defaul= t
> > lets such code work as expected when it is run from a thread.
> > Especially if you take several such programs, each with its own > > subprocess, and let them all run from several different threads a= t the
> > same time.
>
> Yes, I definitely want that real-world case to work right.=C2=A0 I agr= ee that
> that is a very important case.=C2=A0 But I think it already works righ= t with
> processes not locked to threads for any non-buggy program.

That's not my experience.=C2=A0 If random threads get return values fro= m
accept-process-output, you can easily have a thread whose
accept-process-output call never returns until timeout, because the
output was already read by another thread.

I know you don't have much ti= me to work on this, but it would really help if you could give a concrete e= xample program that demonstrates this.

> (I personally have written or used lots of code like that with threads= ,
> and the fact that processes were not fully locked to threads did not > cause problems.

And locking them does cause problems?

Yes.=C2=A0 Such as in the example I w= as describing above.


It is not racy in this example.=C2=A0= Even without locking.

<= div class=3D"gmail_quote gmail_quote_container">
> If the program was instead something like:
>
> (let ((proc (make-process ...)))
>=C2=A0 =C2=A0(sit-for 1)
>=C2=A0 =C2=A0(accept-process-output proc))
>
> then the (accept-process-output proc) might block because the sit-for<= br> > can thread switch.=C2=A0 But this program is already buggy, since sit-= for
> runs wait_reading_process_output which could read the output from PROC= .

A program can easily call sit-for indirectly, because sit-for is
called all over the place in Emacs.

That's my point.=C2=A0 This second e= xample program is buggy whether threads are used or not.
--000000000000bd9e4e063d833609-- From debbugs-submit-bounces@debbugs.gnu.org Sun Aug 31 21:24:45 2025 Received: (at 79333) by debbugs.gnu.org; 1 Sep 2025 01:24:45 +0000 Received: from localhost ([127.0.0.1]:55853 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ustHo-00084K-L0 for submit@debbugs.gnu.org; Sun, 31 Aug 2025 21:24:44 -0400 Received: from fout-b1-smtp.messagingengine.com ([202.12.124.144]:44823) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1ustHm-000845-92 for 79333@debbugs.gnu.org; Sun, 31 Aug 2025 21:24:43 -0400 Received: from phl-compute-09.internal (phl-compute-09.internal [10.202.2.49]) by mailfout.stl.internal (Postfix) with ESMTP id 9155B1D0017C; Sun, 31 Aug 2025 21:24:36 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-09.internal (MEProxy); Sun, 31 Aug 2025 21:24:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1756689876; x=1756776276; bh=8MGa8/OmQfzuMkVnEyTIrRDbbgdi5LDCBD3eS+1hQbo=; b= XOse3hEe8Ed1Q3+h54i/mx37i1d1mCL7d7rrjwcZhPxmvTK1p8Fr0JQH7UfveKAr anUmzKWdD2Bkg/3Kla/euLgkuUZcx14pF/JGk+V+ONXCg6CZGs1KUr1J750tKKGR 225axen9jKmdvbdtIHutnj2MbMkoQfEyRSNucRXdUUCLmZt4jlrGPI6bwZkLsk5Y qQXowd7xjRZz2hcjUwNQgBT7BOi7obOVJX/BBs+mQudADU/jA/Vm1nWTKbPuQLEY Umy/F2BL3sHOWr7qz8wgnjMadfUheBRuFt1x+5P0LOVmMjKOakc45o7rzPClmGJb 9yRw1G6DGpQr2isZCU3wbw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1756689876; x= 1756776276; bh=8MGa8/OmQfzuMkVnEyTIrRDbbgdi5LDCBD3eS+1hQbo=; b=l /mEpYbsa3q6mu7nSHUDA84dcty+uWXXsQo939n59b+iPxHhFjSqnuqPNUwI5DR5s P7Arjvv1J8gPvb4vYQII6+7p0mtwHLO1Kn3o6ozVYS1HrLnYaI1sO9F3HDe9fm5y 3hRIQB4/4Vtth6l4zqJF6aUa0imLG0WNLquMldEZ9urdTRTB5iIPsWlJdqnkotLF +rkQYUD9V6w1H2452L7chCZuscy4P4ockux3tw3vhVWnIlyTELhOsx1yGtAjaNWl EyZf1DEs3guBMgVUStHauBrpEP+Ux4HxlFHlYPmwVd/IRQ5TeJWhrhN753mhVpJE oCGFjtDWodTyl6wnfG9hQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdefgdduledtkedvucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu rghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf gurhepkfffgggfuffvvehfhfgjtgfgsehtkeertddtvdejnecuhfhrohhmpeffmhhithhr hicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrthhtvg hrnhepgeelfeetkefghfdvhfdtgeevveevteetgeetveegtedthefhudekteehffeukeek necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepughmih htrhihsehguhhtohhvrdguvghvpdhnsggprhgtphhtthhopeefpdhmohguvgepshhmthhp ohhuthdprhgtphhtthhopehssggruhhghhesjhgrnhgvshhtrhgvvghtrdgtohhmpdhrtg hpthhtohepvghlihiisehgnhhurdhorhhgpdhrtghpthhtohepjeelfeeffeesuggvsggs uhhgshdrghhnuhdrohhrgh X-ME-Proxy: Feedback-ID: i07de48aa:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 31 Aug 2025 21:24:34 -0400 (EDT) Message-ID: Date: Mon, 1 Sep 2025 04:24:27 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads To: Spencer Baugh , Eli Zaretskii References: <86h5xqshqk.fsf@gnu.org> <86bjnys1st.fsf@gnu.org> <868qj2s054.fsf@gnu.org> <864itqrunm.fsf@gnu.org> Content-Language: en-US From: Dmitry Gutov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 79333 Cc: 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) On 29/08/2025 19:06, Spencer Baugh wrote: > > If the program was instead something like: > > > > (let ((proc (make-process ...))) > >   (sit-for 1) > >   (accept-process-output proc)) > > > > then the (accept-process-output proc) might block because the sit-for > > can thread switch.  But this program is already buggy, since sit-for > > runs wait_reading_process_output which could read the output from > PROC. > > A program can easily call sit-for indirectly, because sit-for is > called all over the place in Emacs. > > > That's my point.  This second example program is buggy whether threads > are used or not. I wonder if we would consider comint-proc-query already problematic in this regard. It does: (comint-send-string proc str) ; send the query (accept-process-output proc) ; wait for some output and comint-send-string -> process-send-string -> send_process, which has a 'wait_reading_process_output' call inside. Is it at least theoretically possible that the latter call consumes the output from the process, making the subsequent accept-process-output call in the function hang? From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 01 10:59:22 2025 Received: (at 79333) by debbugs.gnu.org; 1 Sep 2025 14:59:22 +0000 Received: from localhost ([127.0.0.1]:58126 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ut609-0001sJ-Ov for submit@debbugs.gnu.org; Mon, 01 Sep 2025 10:59:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57804) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1ut608-0001s5-9k for 79333@debbugs.gnu.org; Mon, 01 Sep 2025 10:59:20 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ut602-0001he-J2; Mon, 01 Sep 2025 10:59:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=1m/9WRBptWSq3EqBPoS8Xz0VBJip9egH8Cuj2yoSBIA=; b=J0QLplzrTBdnXvxPY3Hx Ozr3d51dA62jl5Z2PeH2iOAXO0BojxmKhC9ve/yhPF7KncPOIgA7yrtHNf7h5fkeiaAscVoyp0oQu 3jqvKSJp5YNhvlaiRdf5QTbfhDtv/U7dElfA6UrOTffxM843hYLvT5WcrKgUFZUAzkrNbHmZTeLFO cohQoecKxHSGYUUp70BHgmsdwlygKKHth/GzHHbowU9CmWqW6TrECFgohu5zqLkcfh4dv0gZ5zWOv XymgAcAcBW3N+jRbSoeylwhxnE2MDpgoOsB4UnFtGNKJijIT6G/uYi4IxdmE1+9VEBuG9twBfnAOx Tzo0usLkqpNYjw==; Date: Mon, 01 Sep 2025 17:59:06 +0300 Message-Id: <86y0qymd6t.fsf@gnu.org> From: Eli Zaretskii To: Dmitry Gutov In-Reply-To: (message from Dmitry Gutov on Mon, 1 Sep 2025 04:24:27 +0300) Subject: Re: bug#79333: 31.0.50; Processes (still) aren't actually locked to threads References: <86h5xqshqk.fsf@gnu.org> <86bjnys1st.fsf@gnu.org> <868qj2s054.fsf@gnu.org> <864itqrunm.fsf@gnu.org> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 79333 Cc: sbaugh@janestreet.com, 79333@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > Date: Mon, 1 Sep 2025 04:24:27 +0300 > Cc: 79333@debbugs.gnu.org > From: Dmitry Gutov > > On 29/08/2025 19:06, Spencer Baugh wrote: > > > If the program was instead something like: > > > > > > (let ((proc (make-process ...))) > > >   (sit-for 1) > > >   (accept-process-output proc)) > > > > > > then the (accept-process-output proc) might block because the sit-for > > > can thread switch.  But this program is already buggy, since sit-for > > > runs wait_reading_process_output which could read the output from > > PROC. > > > > A program can easily call sit-for indirectly, because sit-for is > > called all over the place in Emacs. > > > > > > That's my point.  This second example program is buggy whether threads > > are used or not. > > I wonder if we would consider comint-proc-query already problematic in > this regard. > > It does: > > (comint-send-string proc str) ; send the query > (accept-process-output proc) ; wait for some output > > and comint-send-string -> process-send-string -> send_process, which has > a 'wait_reading_process_output' call inside. Why do you think this could be problematic? > Is it at least theoretically possible that the latter call consumes the > output from the process, making the subsequent accept-process-output > call in the function hang? It shouldn't. send_process only calls wait_reading_process_output if it cannot write the whole string in one go, AFAIR, in which case the last part of the process's output should still be available to accept-process-output call.