Subject: bug#79387: 30.1.90; If a thread exits, other threads in wait_reading_process_output aren't woken up

From: Eli Zaretskii <[email-masked]>
Date: Fri, 05 Sep 2025 09:27:59 +0300

> Cc: Dmitry Gutov <[email-masked]>, Eli Zaretskii <[email-masked]>
> From: Spencer Baugh <[email-masked]>
> Date: Thu, 04 Sep 2025 15:57:18 -0400
> 
> 
> 
> Expl
> Multiple threads can call wait_reading_process_output at the same time, and
> we are careful to avoid having multiple threads monitor the same file
> descriptors.  That's fine and good.
> 
> If thread A calls wait_reading_process_output, then it will monitor all
> the file descriptors it can.  If thread B calls
> wait_reading_process_output after that, it will monitor the remaining
> file descriptors which aren't already being monitored, such as newly
> created processes.  Generally, thread B will be monitoring far fewer
> file descriptors than thread A, which will be monitoring most of them.
> Each thread will handle events on the file descriptors it is monitoring,
> as appropriate.

As an aside: I don't think I agree with the assertion that thread A
monitors most of the descriptors.  If you are talking about I/O
descriptors due to subprocesses, then each thread should monitor only
the subprocesses it started.  Whether A will monitor more or less than
B depends on how many processes each one of them started.

What you describe can only be true if processes are all unlocked.

> However, there's a fundamental problem with the implementation right
> now: if thread A calls wait_reading_process_output and monitors most of
> the fds, and thread B calls wait_reading_process_output and monitors
> just the remaining fds, then what happens if thread A exits?
> 
> Thread B will stay blocked, monitoring just a few file descriptors,
> while most of them stay un-monitored.  This can happen to the main
> thread, for example.
> 
> Thread B will stay in this state until some event happens on the file
> descriptors it *is* monitoring, or it times out; then either
> wait_reading_process_output will return or it will loop.  After looping,
> it will thankfully pick up all the file descriptors which were
> previously being monitored by thread A.

Yes.

Btw, there's also another possibility for thread B to be awoken: if it
was stuck not in pselect, but in the following attempt to take the
global lock.  For example, it could be that its pselect call already
returned for some reason.  In that case, thread B will start running
as soon as thread A exits (assuming there are no other threads
competing for the global lock).

> I think this is a fairly deep issue with the implementation, which has
> probably been causing other weird inconsistencies and bugs.

Why is this a problem?  You are basically describing an abnormal
situation: a thread exited, but the descriptors which only it was
monitoring are somehow left open and alive.  Nominally, this should
not happen: a thread should exit only after all its descriptors are
closed and cleaned up.  So for an abnormal situation, the fact that
some other thread will pick up the orphan descriptors doesn't sound
like a serious problem: Emacs is recovering from something that is
arguably a bug.

> To fix this, we need to wake up threads which are blocked in
> wait_reading_process_output when another thread exits.  This will
> require adding a new mechanism to support this.  I think a good way is
> for thread creation to open a pipe specific to that thread, used for
> this purpose; then when a thread is blocked in
> wait_reading_process_output, it should monitor this pipe along with
> other file descriptors.  Then an exiting thread can write to this pipe
> to wake up a thread which is currently blocked in
> wait_reading_process_output.

I'm not yet convinced we should try to fix this.  It is certainly not
as serious a problem as the one with deleting processes locked to
other threads, something we need to fix ASAP, and might affect what
happens in the situation you describe here.

> Sorry to open up yet another thread bug, but while working on fixes for
> the other ones, I ran into this.  And I think this one is a fairly
> fundamental problem which will require more substantial changes, so
> let's fix this one first.

Yes, let's please fix the other ones first.

> You can reproduce this bug by running this code with "emacs -Q --batch":
> ;; -*- lexical-binding: t; -*-
> (defun my-filter (proc out)
>   (message "%s %s" proc out))
> (make-thread
>  (lambda ()
>    (message "starting thread")
>    (sit-for 1)
>    (message "making process")
>    (make-process
>     :name "true"
>     :filter #'my-filter
>     :command '("sh" "-c" "sleep 1 && echo hello && sleep inf"))
>    (message "thread exiting")))
> (message "main thread sleeping")
> (sit-for 10)
> (message "main thread sleeping again")
> (sit-for 10)
> 
> The process output will not be printed until the first sit-for is
> complete.

First, the results I see are not deterministic, and also
system-dependent.  On MS-Windows, the process output is consistently
printed right after "thread exiting".  On GNU/Linux, this also
happens, but very rarely; in most runs indeed the process output
appears after several seconds.

But I fail to understand why this is a problem.  A single-threaded
Emacs is documented to process output from subprocesses only when it
is idle, so the exact timing of the process output being made
available to Lisp programs is already not deterministic.  In the case
of several threads, the timing also depends on what other threads do
when the process produces its output.  IOW, a program that expects the
output to appear after exactly 1 sec just because of "sleep 1" has
wrong expectations.  Especially since the thread exits without waiting
for the output from its process and without shutting down the process,
which is basically not clean and arguably a bug.

Bottom line: I don't think I see a problem in this case, certainly not
a serious one.