Subject: bug#79367: 31.0.50; magit-commit sometimes doesn't work if diff-hl-update-async is t

From: Eli Zaretskii <[email-masked]>
Date: Thu, 04 Sep 2025 08:18:54 +0300

> From: Spencer Baugh <[email-masked]>
> Cc: [email-masked],  [email-masked],  [email-masked]
> Date: Wed, 03 Sep 2025 14:04:42 -0400
> 
> Eli Zaretskii <[email-masked]> writes:
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=79201#100
> 
> > If you study the code in wait_reading_process_output, you will
> > immediately see what kind of godawful mess will that cause with
> > reading output from several processes started by several threads.
> > 
> > wait_reading_process_output was written under an implicit assumption
> > that there's only one thread that handles all the subprocesses.
> > Therefore, it consistently checks all of the I/O descriptors for
> > available input from the subprocesses, and consumes that as soon as
> > it's available.  That cannot possibly work reliably, let alone
> > predictably, when several threads are involved.  This is so obvious
> > from reading the code that I'm astonished I need to even explain it.
> 
> Note that we need to make sure wait_reading_process_output works
> reliably in this case for the sake of processes which aren't locked to
> threads.  So defaulting to locking processes to threads doesn't actually
> help with this.

Not if we consider unlocked processes a rare and unusual thing to do,
in which case programmers that use this are responsible for the
dealing with the results.  We made such a change in comp.el, but I
cannot say I'm too happy with it, and hope we will revisit that later
and find a better solution.

We do need to make sure nothing disastrous happens in that case, like
crashes or EBADF or Emacs hung and unresponsive, but that's just a
small part of "works reliably", and is not my primary concern here.
My primary concern is that the results of letting processes be
unlocked will cause many unexpected and randomly-unpredictable
results, which will avert people from using threads.  Because who will
want to use an infrastructure that makes working Lisp programs
unpredictable when run from a non-main thread?

> So, to summarize, it sounds like your issue with not locking processes
> to threads is:
> 
>   Without locking processes to the creating thread, a process's output
>   can be read by some other thread during the wait, and then our thread
>   will hang forever if it calls accept-process-output after the wait
>   ended.

Either "hang forever", or fail to work because the expected output is
"stolen" by another thread, or run the process filter in the context
of another thread (where, for example, things like the current buffer
and match-data are different), or some other such unexpected
conditions.

> This is unfortunately not detailed enough to translate into code.  I've
> never seen a real-world program which would be vulnerable to this
> problem, as described here.  I can make up toy programs which meet this
> description which would hang, but I'm not sure they're actually what
> you're concerned about.  Especially because they aren't anything like
> real-world programs.

I toy program that looks valid and runs in a single-threaded Emacs,
but hangs or fails when run from a non-main thread is already a
problem.  Whether it is interesting for us depends on the program.  A
toy program which demonstrates an issue that real-world programs will
meet is interesting.

In any case, it is clear to me, and should be clear to anyone else,
that taking a Lisp program written for a single-threaded Emacs and
running it from a non-main thread will be made easier if the output
from the sub-processes started by that program will be read only by
that thread, because that is closer to the assumptions under which the
original code was written.

> Could you give some more details about the scenario, so I can try to
> write a test demonstrating the problem?  Or could you point to a
> real-world program which is vulnerable to this problem, so I can turn it
> into a test?

I've never used or debugged a real-world program where these aspects
come into play.  My experience is almost exclusively with small "toy"
programs and test programs, the ones we have in the test suite and
also those submitted as part of relevant bug reports.  You may wish
looking at running the Gnus function that fetches articles from a
separate thread (I think someone tried that at some point), and you
may wish talking to Michael Albinus, who tried using threads in Tramp
(I think he has that on a branch somewhere?).