GNU bug report logs -
#79079
31.0.50; Piped command output is sometimes lost in Eshell
Previous Next
To reply to this bug, email your comments to 79079 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
jporterbugs <at> gmail.com, bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Wed, 23 Jul 2025 09:57:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Daniel Mendler <mail <at> daniel-mendler.de>:
New bug report received and forwarded. Copy sent to
jporterbugs <at> gmail.com, bug-gnu-emacs <at> gnu.org.
(Wed, 23 Jul 2025 09:57:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Hello,
sometimes when executing commands in Eshell with pipes the output seems
to get lost, for example:
git log | head -n 10 | cat
Instead of the expected ten lines of output, there is no output. This
problem does not happen always, but from time to time, so maybe there is
some timing issue regarding the process handling? In case the issue does
not occur immediately one can try this:
while true { echo "test"; git log | head -n 1 | cat }
For such loops the following error might occur:
Creating pipe: Too many open files
Cannot redirect stderr: Too many open files, /dev/null
I tested this with emacs -Q on Emacs 31 with a relatively recent commit
from a few days ago (1e3d76af5a7a16682e71f9df46f6788c0880cccd).
Can someone confirm these problems? Thanks!
In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.49, cairo version 1.18.4) of 2025-07-15
Windowing system distributor 'The X.Org Foundation', version 11.0.12101016
System Description: Debian GNU/Linux 13 (trixie)
Configured using:
'configure --prefix=$HOME/.local/share/emacs
--without-compress-install --with-tree-sitter --with-native-compilation
--with-dbus --without-selinux --without-threads --disable-gc-mark-trace
--without-gsettings --without-gpm --with-cairo --with-cairo-xcb
--with-xinput2 --with-x-toolkit=gtk3 --without-toolkit-scroll-bars
'CFLAGS=-O3 -mtune=native -march=native''
Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS HARFBUZZ JPEG LIBOTF LIBSYSTEMD
LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP
SOUND SQLITE3 TIFF TREE_SITTER WEBP X11 XDBE XIM XINERAMA XINPUT2 XPM
XRANDR GTK3 ZLIB
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Wed, 23 Jul 2025 15:53:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/23/2025 2:55 AM, Daniel Mendler via Bug reports for GNU Emacs, the
Swiss army knife of text editors wrote:
> sometimes when executing commands in Eshell with pipes the output seems
> to get lost, for example:
[snip]
> Can someone confirm these problems? Thanks!
I'm not able to reproduce this locally, but if you first run
"eshell-debug process", that will log a bunch of process-related
information to the buffer "*eshell last cmd*". That would probably have
some details that at least show where the I/O went missing.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Wed, 23 Jul 2025 16:38:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 79079 <at> debbugs.gnu.org (full text, mbox):
Jim Porter <jporterbugs <at> gmail.com> writes:
> On 7/23/2025 2:55 AM, Daniel Mendler via Bug reports for GNU Emacs, the Swiss
> army knife of text editors wrote:
>> sometimes when executing commands in Eshell with pipes the output seems
>> to get lost, for example:
> [snip]
>> Can someone confirm these problems? Thanks!
>
> I'm not able to reproduce this locally, but if you first run "eshell-debug
> process", that will log a bunch of process-related information to the buffer
> "*eshell last cmd*". That would probably have some details that at least show
> where the I/O went missing.
Thanks. I tried your suggestion. What stood out is that in the failed
runs, the output from the git process is received, and passed on to the
head process, but no output is received from head. It seems that the
head filter function is never called? This would hint at a lower-level
problem. See the logs of the runs below.
The problem occurs in particular for this command:
git log --color=always --decorate=yes | head -n 1
While it does not happen (or not so often) for this:
git log --color=always --decorate=yes | cat | head -n 1
Did you try the loop with the echo and the command? Do you have any
further suggestions on how to debug this? Maybe it has something to do
with my specific build.
Daniel
### Good run
[process] started external process ‘head’
[process] started external process ‘git’
[process] received output from process ‘git’
[process] forwarding output from process ‘git’
[process] received output from process ‘head’ <===== received output
[process] received output from process ‘git’
[process] forwarding output from process ‘git’
[process] received output from process ‘git’
[process] forwarding output from process ‘git’
[process] sentinel for external process ‘git’: "broken pipe
[process] i/o busy for process ‘git’
[process] sentinel for external process ‘git-stderr’: "finished
[process] finished external process ‘git-stderr’
[process] sentinel for external process ‘head’: "finished <===== finished
[process] finished external process ‘head’
[process] finished external process ‘git’
### Bad run
[process] started external process ‘head’
[process] started external process ‘git’
[process] received output from process ‘git’
[process] forwarding output from process ‘git’
[process] received output from process ‘git’
[process] forwarding output from process ‘git’
[process] received output from process ‘git’
[process] forwarding output from process ‘git’
[process] sentinel for external process ‘head’: "finished
[process] finished external process ‘head’ <===== finished without output
[process] sentinel for external process ‘git’: "broken pipe
[process] i/o busy for process ‘git’
[process] sentinel for external process ‘git-stderr’: "finished
[process] finished external process ‘git-stderr’
[process] finished external process ‘git’
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Wed, 23 Jul 2025 18:07:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 79079 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On 7/23/2025 8:52 AM, Jim Porter wrote:
> I'm not able to reproduce this locally, but if you first run
> "eshell-debug process", that will log a bunch of process-related
> information to the buffer "*eshell last cmd*". That would probably have
> some details that at least show where the I/O went missing.
After a few more tries, I was able to reproduce this very rarely. Does
the following patch help?
Eli: maybe you can help with some of the details here. In process.c,
when we write to a process, we handle EPIPE errors by calling
'deactivate_process'. However, that can lead to us dropping any data
written *by* that process, since we don't call the process's filter
function for any remaining output still in our internal buffer.
I think in this case, we'd just want to let the rest of our code handle
deactivating the process in the usual way. That helps fix this bug, plus
I think it makes sense in general. If a process closes stdin, I believe
we'd get the EPIPE error, but that process might want to continue
working (though it does mean that you could only interact with that
process via signaling it).
Does that make sense?
[epipe.diff (text/plain, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Wed, 23 Jul 2025 18:18:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 79079 <at> debbugs.gnu.org (full text, mbox):
Jim Porter <jporterbugs <at> gmail.com> writes:
> On 7/23/2025 8:52 AM, Jim Porter wrote:
>> I'm not able to reproduce this locally, but if you first run "eshell-debug
>> process", that will log a bunch of process-related information to the buffer
>> "*eshell last cmd*". That would probably have some details that at least show
>> where the I/O went missing.
>
> After a few more tries, I was able to reproduce this very rarely. Does the
> following patch help?
Thanks for looking into this so quickly and coming up with a patch! I
can check this earliest end of next week. Recompiling Emacs always takes
me a little bit longer than simply evaluating some Elisp.
> Eli: maybe you can help with some of the details here. In process.c, when we
> write to a process, we handle EPIPE errors by calling 'deactivate_process'.
> However, that can lead to us dropping any data written *by* that process, since
> we don't call the process's filter function for any remaining output still in
> our internal buffer.
>
> I think in this case, we'd just want to let the rest of our code handle
> deactivating the process in the usual way. That helps fix this bug, plus I think
> it makes sense in general. If a process closes stdin, I believe we'd get the
> EPIPE error, but that process might want to continue working (though it does
> mean that you could only interact with that process via signaling it).
Maybe this also needs some test to avoid regressions? Did you find a way
to reliably reproduce the problem?
> Does that make sense?
>
> diff --git a/src/process.c b/src/process.c
> index e61ec425f7e..258bee21b5e 100644
> --- a/src/process.c
> +++ b/src/process.c
> @@ -6876,7 +6876,6 @@ send_process (Lisp_Object proc, const char *buf, ptrdiff_t len,
> p->raw_status_new = 0;
> pset_status (p, list2 (Qexit, make_fixnum (256)));
> p->tick = ++process_tick;
> - deactivate_process (proc);
> error ("Process %s no longer connected to pipe; closed it",
> SDATA (p->name));
> }
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 05:35:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Wed, 23 Jul 2025 11:06:24 -0700
> From: Jim Porter <jporterbugs <at> gmail.com>
>
> On 7/23/2025 8:52 AM, Jim Porter wrote:
> > I'm not able to reproduce this locally, but if you first run
> > "eshell-debug process", that will log a bunch of process-related
> > information to the buffer "*eshell last cmd*". That would probably have
> > some details that at least show where the I/O went missing.
>
> After a few more tries, I was able to reproduce this very rarely. Does
> the following patch help?
>
> Eli: maybe you can help with some of the details here. In process.c,
> when we write to a process, we handle EPIPE errors by calling
> 'deactivate_process'. However, that can lead to us dropping any data
> written *by* that process, since we don't call the process's filter
> function for any remaining output still in our internal buffer.
>
> I think in this case, we'd just want to let the rest of our code handle
> deactivating the process in the usual way. That helps fix this bug, plus
> I think it makes sense in general. If a process closes stdin, I believe
> we'd get the EPIPE error, but that process might want to continue
> working (though it does mean that you could only interact with that
> process via signaling it).
>
> Does that make sense?
Sounds a somewhat scary change, since this code was last touched 13
years ago. Paul, WDYT?
What about calling the filter with the data we still have? Isn't that
what a Posix shell would do -- flush any buffers?
And why do we get EPIPE in the recipe in this bug, anyway? I'd like
to understand better at least one situation where we get EPIPE while
the data received before that still matters.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 13:53:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 2025-07-23 22:34, Eli Zaretskii wrote:
>> Date: Wed, 23 Jul 2025 11:06:24 -0700
>> From: Jim Porter<jporterbugs <at> gmail.com>
>> In process.c,
>> when we write to a process, we handle EPIPE errors by calling
>> 'deactivate_process'. However, that can lead to us dropping any data
>> written*by* that process, since we don't call the process's filter
>> function for any remaining output still in our internal buffer.
>>
>> I think in this case, we'd just want to let the rest of our code handle
>> deactivating the process in the usual way. That helps fix this bug, plus
>> I think it makes sense in general. If a process closes stdin, I believe
>> we'd get the EPIPE error, but that process might want to continue
>> working (though it does mean that you could only interact with that
>> process via signaling it).
Yes, that makes sense.
> What about calling the filter with the data we still have? Isn't that
> what a Posix shell would do -- flush any buffers?
Not sure what you man by that, but I suspect it wouldn't handle the
scenario Jim describes, where the other process closes its end of the
pipe but keeps running. Emacs shouldn't assume that EPIPE means the
other process has died or has closed any pipes other than the one Emacs
got EPIPE on.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 14:57:01 GMT)
Full text and
rfc822 format available.
Message #26 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Thu, 24 Jul 2025 06:52:39 -0700
> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org,
> Jim Porter <jporterbugs <at> gmail.com>
> From: Paul Eggert <eggert <at> cs.ucla.edu>
>
> >> I think in this case, we'd just want to let the rest of our code handle
> >> deactivating the process in the usual way. That helps fix this bug, plus
> >> I think it makes sense in general. If a process closes stdin, I believe
> >> we'd get the EPIPE error, but that process might want to continue
> >> working (though it does mean that you could only interact with that
> >> process via signaling it).
>
> Yes, that makes sense.
>
>
> > What about calling the filter with the data we still have? Isn't that
> > what a Posix shell would do -- flush any buffers?
>
> Not sure what you man by that, but I suspect it wouldn't handle the
> scenario Jim describes, where the other process closes its end of the
> pipe but keeps running. Emacs shouldn't assume that EPIPE means the
> other process has died or has closed any pipes other than the one Emacs
> got EPIPE on.
I guess I'm confused: if the other process closed its end of the pipe,
why do we care about the stuff we have in our buffers that wasn't yet
piped to that other process?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 15:53:02 GMT)
Full text and
rfc822 format available.
Message #29 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/23/2025 10:34 PM, Eli Zaretskii wrote:
> What about calling the filter with the data we still have? Isn't that
> what a Posix shell would do -- flush any buffers?
>
> And why do we get EPIPE in the recipe in this bug, anyway? I'd like
> to understand better at least one situation where we get EPIPE while
> the data received before that still matters.
In the recipe, we pipe "git log" to "head -n 10", where Eshell handles
the actual piping. (I don't think Eshell's presence here is actually
necessary to see this bug, though it's probably the simplest way to
reproduce it in Emacs.)
"git log" produces a bunch of output that gets sent to Eshell's process
filter. That filter eventually calls 'process-send-string' on the "head"
process, forwarding all of git's stdout to head's stdin. Immediately
after that, head sees that it's already gotten 10 lines of input, so it
exits. "git log" doesn't know that yet, so it sends another batch of
text to Eshell's process filter, which again gets passed to
'process-send-string'. This time around though, the pipe to head's stdin
has been closed, so we get an EPIPE. Currently, Emacs deactivates the
head process, but because of how the timings worked out, all this
happens *before* Eshell's process filter for head ever runs. As a
result, Emacs never see the output that head produced.
Thinking about this more, my previous comment that the output from the
head process is in an internal buffer in Emacs probably isn't accurate.
That *could* happen, but in some cases, we may not have called 'read'
yet, so the data could still be in the OS's buffer. (I'd have to write
some test cases to be 100% sure of all this, but it follows from the
manpages.)
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 16:34:02 GMT)
Full text and
rfc822 format available.
Message #32 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 2025-07-24 07:56, Eli Zaretskii wrote:
>> Not sure what you man by that, but I suspect it wouldn't handle the
>> scenario Jim describes, where the other process closes its end of the
>> pipe but keeps running. Emacs shouldn't assume that EPIPE means the
>> other process has died or has closed any pipes other than the one Emacs
>> got EPIPE on.
> I guess I'm confused: if the other process closed its end of the pipe,
> why do we care about the stuff we have in our buffers that wasn't yet
> piped to that other process?
I could well be the confused one, but the scenario I was concerned about
is when we are running the process for some reason other than the fact
that we are piping to it. For example, we have another pipe to the same
process. In that case we shouldn't deactivate the entire process (i.e.,
turn off all output and input for it) merely because one of its pipes
closed.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 16:48:01 GMT)
Full text and
rfc822 format available.
Message #35 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/24/2025 7:56 AM, Eli Zaretskii wrote:
>> From: Paul Eggert <eggert <at> cs.ucla.edu>
>>
>> Not sure what you man by that, but I suspect it wouldn't handle the
>> scenario Jim describes, where the other process closes its end of the
>> pipe but keeps running. Emacs shouldn't assume that EPIPE means the
>> other process has died or has closed any pipes other than the one Emacs
>> got EPIPE on.
>
> I guess I'm confused: if the other process closed its end of the pipe,
> why do we care about the stuff we have in our buffers that wasn't yet
> piped to that other process?
We could have a situation where a process closes stdin, but we still
haven't read everything (or flushed our buffers) for that process's stdout.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 17:06:02 GMT)
Full text and
rfc822 format available.
Message #38 received at 79079 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On 7/24/2025 6:52 AM, Paul Eggert wrote:
> On 2025-07-23 22:34, Eli Zaretskii wrote:
>>> Date: Wed, 23 Jul 2025 11:06:24 -0700
>>> From: Jim Porter<jporterbugs <at> gmail.com>
>
>>> In process.c,
>>> when we write to a process, we handle EPIPE errors by calling
>>> 'deactivate_process'. However, that can lead to us dropping any data
>>> written*by* that process, since we don't call the process's filter
>>> function for any remaining output still in our internal buffer.
>>>
>>> I think in this case, we'd just want to let the rest of our code handle
>>> deactivating the process in the usual way. That helps fix this bug, plus
>>> I think it makes sense in general. If a process closes stdin, I believe
>>> we'd get the EPIPE error, but that process might want to continue
>>> working (though it does mean that you could only interact with that
>>> process via signaling it).
>
> Yes, that makes sense.
Thanks for checking. Then I think the attached diff would be close to
what we want, with a few caveats. (The original diff was just the
smallest change that seemed to fix the problem so that Daniel could try
it out.)
This diff changes things so that when we get an EPIPE, we close our end
of the pipe (the 'WRITE_TO_SUBPROCESS' fd). I don't know if we need to
do anything special for PTYs or sockets though.
On the Eshell side then, we previously detected EPIPE by checking
whether the process had been deactivated. Since that doesn't happen
anymore, I changed it so that we always treat an error from
'process-send-string' as though the pipe broke. In practice, this just
means that Eshell will think that all future calls to
'process-send-string' would fail too (and so it will send a SIGPIPE
signal to the sending process; "git" in the original test case). I'm not
totally sure that's right though. Could there be a time when we get an
error from 'process-send-string', but we can try calling it again later?
[epipe.diff (text/plain, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 18:51:01 GMT)
Full text and
rfc822 format available.
Message #41 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 2025-07-24 10:05, Jim Porter wrote:
> Could there be a time when we get an error from 'process-send-string',
> but we can try calling it again later?
Not if it's an EPIPE failure. EPIPE means the connection is permanently
down.
I don't offhand know of any failure for writing to a pipe where retrying
would make sense, other than EAGAIN and EWOULDBLOCK which the code is
already dealing with.
Can process-send-string write to other file types? If so, there are
other possibilities where retrying might work, e.g., ENOSPC for write to
a regular file, ENETDOWN for sendto to a network socket.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Thu, 24 Jul 2025 19:07:01 GMT)
Full text and
rfc822 format available.
Message #44 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/24/2025 11:50 AM, Paul Eggert wrote:
> On 2025-07-24 10:05, Jim Porter wrote:
>> Could there be a time when we get an error from 'process-send-string',
>> but we can try calling it again later?
>
> Not if it's an EPIPE failure. EPIPE means the connection is permanently
> down.
Right. I was just wondering if there might be some use in signaling
different types of errors in send_process; one for transient errors (if
there are any we care about), and one for permanent errors (which look
to be most/all of them).
> I don't offhand know of any failure for writing to a pipe where retrying
> would make sense, other than EAGAIN and EWOULDBLOCK which the code is
> already dealing with.
>
> Can process-send-string write to other file types? If so, there are
> other possibilities where retrying might work, e.g., ENOSPC for write to
> a regular file, ENETDOWN for sendto to a network socket.
For Eshell, we only support writing to regular processes, though I
suppose with enough effort you could probably construct an Eshell
command that writes to a socket. Of course, for something like ENETDOWN,
Eshell would probably need some special handling to retry sending the
same data again, instead of just skipping over it.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Fri, 25 Jul 2025 06:43:01 GMT)
Full text and
rfc822 format available.
Message #47 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Thu, 24 Jul 2025 10:05:50 -0700
> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
> From: Jim Porter <jporterbugs <at> gmail.com>
>
> --- a/src/process.c
> +++ b/src/process.c
> @@ -6873,10 +6873,13 @@ send_process (Lisp_Object proc, const char *buf, ptrdiff_t len,
> }
> else if (errno == EPIPE)
> {
> - p->raw_status_new = 0;
> - pset_status (p, list2 (Qexit, make_fixnum (256)));
> - p->tick = ++process_tick;
> - deactivate_process (proc);
> + /* FIXME: Handle PTY too? What about sockets? And
> + remove this debug statement. */
> + AUTO_STRING (format, "[debug] EPIPE: %S");
> + CALLN (Fmessage, format, p->name);
> +
> + close_process_fd(&p->open_fd[WRITE_TO_SUBPROCESS]);
> + p->outfd = -1;
> error ("Process %s no longer connected to pipe; closed it",
> SDATA (p->name));
> }
Why do we need to emit a message about EPIPE, when we will immediately
signal an error with the same information? The message will not be
seen by users except if they look in *Messages*.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Fri, 25 Jul 2025 17:17:02 GMT)
Full text and
rfc822 format available.
Message #50 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/24/2025 11:42 PM, Eli Zaretskii wrote:
> Why do we need to emit a message about EPIPE, when we will immediately
> signal an error with the same information? The message will not be
> seen by users except if they look in *Messages*.
It's just a debug statement for development/testing so that it's clear
when Emacs runs this block. Since this is a timing-related issue, it's
hard to guarantee that we hit this condition. While I'm working on this,
I can then check *Messages* to be sure my new code ran and did the right
thing.
I'll remove the Fmessage call before merging.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sat, 26 Jul 2025 10:16:01 GMT)
Full text and
rfc822 format available.
Message #53 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Thu, 24 Jul 2025 08:52:16 -0700
> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
> From: Jim Porter <jporterbugs <at> gmail.com>
>
> On 7/23/2025 10:34 PM, Eli Zaretskii wrote:
> > What about calling the filter with the data we still have? Isn't that
> > what a Posix shell would do -- flush any buffers?
> >
> > And why do we get EPIPE in the recipe in this bug, anyway? I'd like
> > to understand better at least one situation where we get EPIPE while
> > the data received before that still matters.
>
> In the recipe, we pipe "git log" to "head -n 10", where Eshell handles
> the actual piping. (I don't think Eshell's presence here is actually
> necessary to see this bug, though it's probably the simplest way to
> reproduce it in Emacs.)
>
> "git log" produces a bunch of output that gets sent to Eshell's process
> filter. That filter eventually calls 'process-send-string' on the "head"
> process, forwarding all of git's stdout to head's stdin. Immediately
> after that, head sees that it's already gotten 10 lines of input, so it
> exits. "git log" doesn't know that yet, so it sends another batch of
> text to Eshell's process filter, which again gets passed to
> 'process-send-string'. This time around though, the pipe to head's stdin
> has been closed, so we get an EPIPE. Currently, Emacs deactivates the
> head process, but because of how the timings worked out, all this
> happens *before* Eshell's process filter for head ever runs. As a
> result, Emacs never see the output that head produced.
So calling the filter of the "head"s process when we get EPIPE should
fix that, no?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sat, 26 Jul 2025 17:24:02 GMT)
Full text and
rfc822 format available.
Message #56 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/26/2025 3:15 AM, Eli Zaretskii wrote:
>> Date: Thu, 24 Jul 2025 08:52:16 -0700
>> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
>> From: Jim Porter <jporterbugs <at> gmail.com>
>>
> So calling the filter of the "head"s process when we get EPIPE should
> fix that, no?
It would probably fix this particular case, but I'm not sure it's the
correct way to do it in general. I'll construct some test programs to
test out some of the corner cases here so that we can be sure whatever
implementation we do settle on works correctly in all cases.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sat, 26 Jul 2025 21:14:02 GMT)
Full text and
rfc822 format available.
Message #59 received at 79079 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On 7/26/2025 10:23 AM, Jim Porter wrote:
> On 7/26/2025 3:15 AM, Eli Zaretskii wrote:
>>> Date: Thu, 24 Jul 2025 08:52:16 -0700
>>> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
>>> From: Jim Porter <jporterbugs <at> gmail.com>
>>>
>> So calling the filter of the "head"s process when we get EPIPE should
>> fix that, no?
>
> It would probably fix this particular case, but I'm not sure it's the
> correct way to do it in general. I'll construct some test programs to
> test out some of the corner cases here so that we can be sure whatever
> implementation we do settle on works correctly in all cases.
After examining this more closely, there are three problems with the
current Emacs code, of varying importance:
1. When we receive an EPIPE, we don't call our process-filter one last
time (this is the original bug report). You can see this by running the
following Eshell command repeatedly until it produces no output:
git log | head -n 10 | cat
2. When we receive an EPIPE, we set the exit status to 256; that's not
right, since the exit status should be 0. You can see this by running
the following Eshell command until the prompt shows a non-zero status:
git log | head -n 10 > #<my-buffer>
3. If the child process closes stdin and wants to keep doing some work
(like write to stdout), it can't, since we called 'deactivate_process'.
You can see this with the attached script by running (it's supposed to
print the first line of "git log"):
git log | ./test-script.sh
I've also attached two WIP patches for the different implementations.
"bad-call-filter.diff" uses 'read_process_output' to call our process
filter. It only solves issue #1 above. "good-close-pipe.diff" (which I
posted previously) solves all three issues. While the latter is a
slightly bigger change and requires some more testing on my end to make
sure I've covered 100% of the cases, I think it's the more-correct
solution. If we make the bigger change, it's probably worth calling it
out in NEWS and on emacs-devel so that people can be on the lookout for
any edge cases I haven't identified.
[bad-call-filter.diff (text/plain, attachment)]
[good-close-pipe.diff (text/plain, attachment)]
[test-script.sh (text/plain, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 04:31:01 GMT)
Full text and
rfc822 format available.
Message #62 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Sat, 26 Jul 2025 10:23:48 -0700
> Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
> From: Jim Porter <jporterbugs <at> gmail.com>
>
> On 7/26/2025 3:15 AM, Eli Zaretskii wrote:
> >> Date: Thu, 24 Jul 2025 08:52:16 -0700
> >> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
> >> From: Jim Porter <jporterbugs <at> gmail.com>
> >>
> > So calling the filter of the "head"s process when we get EPIPE should
> > fix that, no?
>
> It would probably fix this particular case, but I'm not sure it's the
> correct way to do it in general. I'll construct some test programs to
> test out some of the corner cases here so that we can be sure whatever
> implementation we do settle on works correctly in all cases.
We should try doing what a Posix shell does in that case. A program
that gets EPIPE generally exits with a failure status (right?),
anything that happens after that is done by the shell itself.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 06:18:02 GMT)
Full text and
rfc822 format available.
Message #65 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On Jul 27 2025, Eli Zaretskii wrote:
> We should try doing what a Posix shell does in that case.
A POSIX shell does nothing. It just connects the processes together and
let them run to exit.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 07:02:01 GMT)
Full text and
rfc822 format available.
Message #68 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> From: Andreas Schwab <schwab <at> linux-m68k.org>
> Cc: Jim Porter <jporterbugs <at> gmail.com>, mail <at> daniel-mendler.de,
> eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
> Date: Sun, 27 Jul 2025 08:17:41 +0200
>
> On Jul 27 2025, Eli Zaretskii wrote:
>
> > We should try doing what a Posix shell does in that case.
>
> A POSIX shell does nothing. It just connects the processes together and
> let them run to exit.
Not useful. I think you know very well what I meant, beyond the
literal meaning of the words.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 07:32:01 GMT)
Full text and
rfc822 format available.
Message #71 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On Jul 27 2025, Eli Zaretskii wrote:
>> From: Andreas Schwab <schwab <at> linux-m68k.org>
>> Cc: Jim Porter <jporterbugs <at> gmail.com>, mail <at> daniel-mendler.de,
>> eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
>> Date: Sun, 27 Jul 2025 08:17:41 +0200
>>
>> On Jul 27 2025, Eli Zaretskii wrote:
>>
>> > We should try doing what a Posix shell does in that case.
>>
>> A POSIX shell does nothing. It just connects the processes together and
>> let them run to exit.
>
> Not useful. I think you know very well what I meant, beyond the
> literal meaning of the words.
But it is the truth. Emacs is not a shell.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 07:41:02 GMT)
Full text and
rfc822 format available.
Message #74 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> From: Andreas Schwab <schwab <at> linux-m68k.org>
> Cc: jporterbugs <at> gmail.com, mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu,
> 79079 <at> debbugs.gnu.org
> Date: Sun, 27 Jul 2025 09:30:59 +0200
>
> On Jul 27 2025, Eli Zaretskii wrote:
>
> >> From: Andreas Schwab <schwab <at> linux-m68k.org>
> >> Cc: Jim Porter <jporterbugs <at> gmail.com>, mail <at> daniel-mendler.de,
> >> eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
> >> Date: Sun, 27 Jul 2025 08:17:41 +0200
> >>
> >> On Jul 27 2025, Eli Zaretskii wrote:
> >>
> >> > We should try doing what a Posix shell does in that case.
> >>
> >> A POSIX shell does nothing. It just connects the processes together and
> >> let them run to exit.
> >
> > Not useful. I think you know very well what I meant, beyond the
> > literal meaning of the words.
>
> But it is the truth. Emacs is not a shell.
Of course it isn't. Which is why we should try to make things happen
which happen when the same command is run from a Posix shell. Some of
that happens in the shell, some in the OS, some in the commands.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 07:53:01 GMT)
Full text and
rfc822 format available.
Message #77 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On Jul 27 2025, Eli Zaretskii wrote:
> Of course it isn't. Which is why we should try to make things happen
> which happen when the same command is run from a Posix shell. Some of
> that happens in the shell, some in the OS, some in the commands.
Emacs does not have the overview that the kernel has. It does not know
when a process closes a stream, it can only discover it when it tries to
read from or write to the other end of the stream. The kernel is the
one who drives all communication, Emacs can only react (because it
itself relies on the kernel to do everything).
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 15:07:02 GMT)
Full text and
rfc822 format available.
Message #80 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 2025-07-26 21:30, Eli Zaretskii wrote:
> We should try doing what a Posix shell does in that case. A program
> that gets EPIPE generally exits with a failure status (right?),
> anything that happens after that is done by the shell itself.
EPIPE typically means exit, yes, as the process reports the error and
often has nothing better to do than to exit. However, it doesn't always
mean exit. For example, an interactive Emacs does not exit with it gets
EPIPE, as can be demonstrated with this shell command on GNU/Linux:
strace -o /tmp/tr emacs -Q -eval \
"(princ (make-string 10000 ?a) 'external-debugging-output)" 2>&1|:
and then type C-x C-c to exit Emacs.
In this case Emacs painstakingly outputs 10000 'a' bytes, one at a time,
and ignores each of the 10000 EPIPE errors that it gets.
Not sure I know what you mean by "doing what a Posix shell does". Are
you talking about commands that a shell implements directly, like the
shell 'echo' command typically is? If so, those commands are supposed to
do what their standalone counterpart executables do, which typically
means they die due to getting a SIGPIPE signal.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 15:39:01 GMT)
Full text and
rfc822 format available.
Message #83 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Sun, 27 Jul 2025 08:06:10 -0700
> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
>
> Not sure I know what you mean by "doing what a Posix shell does". Are
> you talking about commands that a shell implements directly, like the
> shell 'echo' command typically is? If so, those commands are supposed to
> do what their standalone counterpart executables do, which typically
> means they die due to getting a SIGPIPE signal.
I meant wrt data still buffered in the broken pipe(s) and standard
streams.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 16:16:02 GMT)
Full text and
rfc822 format available.
Message #86 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 2025-07-27 08:38, Eli Zaretskii wrote:
>> Date: Sun, 27 Jul 2025 08:06:10 -0700
>> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
>> From: Paul Eggert <eggert <at> cs.ucla.edu>
>>
>> Not sure I know what you mean by "doing what a Posix shell does". Are
>> you talking about commands that a shell implements directly, like the
>> shell 'echo' command typically is? If so, those commands are supposed to
>> do what their standalone counterpart executables do, which typically
>> means they die due to getting a SIGPIPE signal.
>
> I meant wrt data still buffered in the broken pipe(s) and standard
> streams.
Still not quite sure I follow what's being asked, but here goes.
There's no way for any program writing to a pipe to know what's in the
kernel's buffers for the pipe, so there's nothing to do there.
For buffers in user space (e.g., stdout's buffer), once a program has
gotten EPIPE there's no point to any further writes or fflushes; the
program will just get more EPIPEs.
Typically what a program will do in either situation is output an error
message and exit. Of course this exits just the program, not the shell
that invoked the program. Bash, for example, acts like this if bigstring
is long enough:
$ (trap "" PIPE; sleep 1; echo "$bigstring") |:
bash: echo: write error: Broken pipe
$
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 16:55:03 GMT)
Full text and
rfc822 format available.
Message #89 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/26/2025 9:30 PM, Eli Zaretskii wrote:
> We should try doing what a Posix shell does in that case. A program
> that gets EPIPE generally exits with a failure status (right?),
> anything that happens after that is done by the shell itself.
Agreed. Eshell already does its best at the part you mention: in the
pipeline "A | B", if A writes to B and gets an EPIPE error, Eshell sends
a SIGPIPE to process A. (That's not *exactly* what happens in POSIX
shells, but because of the additional indirection Eshell has between
processes in a pipeline, it's as close as I can get without major surgery.)
The remainder is just to make sure that we follow POSIX shell behavior
as closely as we can for process B as well.
As for the rest of the discussion here, my goal isn't exactly to make
Emacs behave like a POSIX shell (which is apples-to-oranges, as Emacs
isn't a shell), but to make *Eshell* behave like a POSIX shell. That
requires some changes to process.c, but I think those changes will be
positive for Emacs in general; the result will be that Emacs responds
more carefully to EPIPE.
(Of course, if there are any changes that would help Eshell but *hurt*
Emacs's process support in other scenarios, I won't make those. But I'm
pretty sure we won't run into this problem.)
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 17:15:01 GMT)
Full text and
rfc822 format available.
Message #92 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Sun, 27 Jul 2025 09:15:09 -0700
> Cc: jporterbugs <at> gmail.com, mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
>
> On 2025-07-27 08:38, Eli Zaretskii wrote:
> >> Date: Sun, 27 Jul 2025 08:06:10 -0700
> >> Cc: mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
> >> From: Paul Eggert <eggert <at> cs.ucla.edu>
> >>
> >> Not sure I know what you mean by "doing what a Posix shell does". Are
> >> you talking about commands that a shell implements directly, like the
> >> shell 'echo' command typically is? If so, those commands are supposed to
> >> do what their standalone counterpart executables do, which typically
> >> means they die due to getting a SIGPIPE signal.
> >
> > I meant wrt data still buffered in the broken pipe(s) and standard
> > streams.
>
> Still not quite sure I follow what's being asked, but here goes.
>
> There's no way for any program writing to a pipe to know what's in the
> kernel's buffers for the pipe, so there's nothing to do there.
>
> For buffers in user space (e.g., stdout's buffer), once a program has
> gotten EPIPE there's no point to any further writes or fflushes; the
> program will just get more EPIPEs.
>
> Typically what a program will do in either situation is output an error
> message and exit. Of course this exits just the program, not the shell
> that invoked the program. Bash, for example, acts like this if bigstring
> is long enough:
>
> $ (trap "" PIPE; sleep 1; echo "$bigstring") |:
> bash: echo: write error: Broken pipe
> $
How ids all this working in the specific pipe command in question, and
why does that piped command yield different results when run from a
shell than when run from Eshell?
That's basically what I'm asking. And as a corollary, what should we
do so that the result in Eshell will be the same as from a shell, and
why?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 19:15:02 GMT)
Full text and
rfc822 format available.
Message #95 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 2025-07-27 10:14, Eli Zaretskii wrote:
>> Date: Sun, 27 Jul 2025 09:15:09 -0700
>> Cc: jporterbugs <at> gmail.com, mail <at> daniel-mendler.de, 79079 <at> debbugs.gnu.org
>> From: Paul Eggert <eggert <at> cs.ucla.edu>
>> ...
>> Typically what a program will do in either situation is output an error
>> message and exit. Of course this exits just the program, not the shell
>> that invoked the program. Bash, for example, acts like this if bigstring
>> is long enough:
>>
>> $ (trap "" PIPE; sleep 1; echo "$bigstring") |:
>> bash: echo: write error: Broken pipe
>> $
>
> How ids all this working in the specific pipe command in question, and
> why does that piped command yield different results when run from a
> shell than when run from Eshell?
Sorry, I don't know.
From Jim's recent email it sounds like Eshell does not actually set up
a pipe when implementing "A | B", but instead does something else that
simulates a pipe. I don't know how the simulation works, but it sounds
like the simulated pipe cannot act like a real pipe when B closes its
end of the simulated pipe, A ignores SIGPIPE, and A writes to the
simulated pipe. With a real pipe A would get an EPIPE error, and I guess
that's not happening with the simulated pipe.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 19:35:02 GMT)
Full text and
rfc822 format available.
Message #98 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/27/2025 12:14 PM, Paul Eggert wrote:
> From Jim's recent email it sounds like Eshell does not actually set up
> a pipe when implementing "A | B", but instead does something else that
> simulates a pipe. I don't know how the simulation works, but it sounds
> like the simulated pipe cannot act like a real pipe when B closes its
> end of the simulated pipe, A ignores SIGPIPE, and A writes to the
> simulated pipe. With a real pipe A would get an EPIPE error, and I guess
> that's not happening with the simulated pipe.
If we're being technical, Eshell does use pipes for its pipe operator,
but there's an additional layer of indirection that makes it difficult
to simulate the usual POSIX shell piping:
process A -> stdout pipe -> process filter -> process-send-string ->
stdin pipe -> process B
If 'process-send-string' fails, we raise a Lisp signal
'eshell-pipe-broken', that tries to be as much like SIGPIPE as we can.
Because of this indirection though, process A won't see an EPIPE/SIGPIPE
at the normally-expected time, hence my comment in esh-proc.el:
;; The output pipe broke, so send SIGPIPE to the
;; process. NOTE: Due to the additional indirection
;; of Emacs process filters, the process will likely
;; see the SIGPIPE later than it would in a regular
;; shell, which could cause problems. For cases
;; where this matters, using an external pipe
;; operator (`*|') may work instead.
Luckily, this bug is really about what happens with process B in the
pipeline, so I won't have to make any further changes to the process A
side of things (at least not yet).
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 19:54:01 GMT)
Full text and
rfc822 format available.
Message #101 received at 79079 <at> debbugs.gnu.org (full text, mbox):
Jim Porter <jporterbugs <at> gmail.com> writes:
> On 7/27/2025 12:14 PM, Paul Eggert wrote:
>> From Jim's recent email it sounds like Eshell does not actually set up a pipe
>> when implementing "A | B", but instead does something else that simulates a
>> pipe. I don't know how the simulation works, but it sounds like the simulated
>> pipe cannot act like a real pipe when B closes its end of the simulated pipe,
>> A ignores SIGPIPE, and A writes to the simulated pipe. With a real pipe A
>> would get an EPIPE error, and I guess that's not happening with the simulated
>> pipe.
>
> If we're being technical, Eshell does use pipes for its pipe operator, but
> there's an additional layer of indirection that makes it difficult to simulate
> the usual POSIX shell piping:
>
> process A -> stdout pipe -> process filter -> process-send-string -> stdin
> pipe -> process B
Only tangentially related to this bug report, but I wonder if it would
make sense to always use actual pipes in Eshell when possible, instead
of the process filter indirection?
There exists already the *| operator from `em-extpipe'. What prevents us
from making it the default (or enable it via customization) if `sh' is
available? The downside is that `sh' is started as an intermediate
process, but are their other downsides? For example piping though an
Elisp function, which transforms the input, would not work, but this is
not yet supported, or is it? As far as I understand *| is often more
efficient, despite starting a separate shell process.
Exposing Unix pipes to directly glue file handles on the Elisp level
would be another option, but it is probably difficult to integrate this
into the existing process API?
Daniel
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 20:10:02 GMT)
Full text and
rfc822 format available.
Message #104 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/27/2025 12:53 PM, Daniel Mendler via Bug reports for GNU Emacs, the
Swiss army knife of text editors wrote:
> Only tangentially related to this bug report, but I wonder if it would
> make sense to always use actual pipes in Eshell when possible, instead
> of the process filter indirection?
>
> There exists already the *| operator from `em-extpipe'. What prevents us
> from making it the default (or enable it via customization) if `sh' is
> available? The downside is that `sh' is started as an intermediate
> process, but are their other downsides? For example piping though an
> Elisp function, which transforms the input, would not work, but this is
> not yet supported, or is it? As far as I understand *| is often more
> efficient, despite starting a separate shell process.
Eshell will support piping to Lisp commands Soon(TM). I'm finishing up a
patch series that I hope to upload for discussion shortly. This has
taken a *long* time to get to the point where I haven't run into any
major issues, but it's finally coming together.
The larger problem is that it's impossible to determine at
command-parsing time whether a particular command is implemented
entirely in Lisp or will create a child process. "cat" is a good
example: even as the leftmost command in a pipeline, sometimes it's a
Lisp command, and other times, it creates a child process by throwing
'eshell-external' to replace the Lisp version of the command. That makes
it challenging to connect the pipes up correctly, since we don't know
which commands have real OS-provided pipes until command execution. I'm
not saying it's impossible, but it's definitely not easy.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 27 Jul 2025 22:04:02 GMT)
Full text and
rfc822 format available.
Message #107 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 2025-07-27 12:34, Jim Porter wrote:
> If we're being technical, Eshell does use pipes for its pipe operator,
> but there's an additional layer of indirection that makes it difficult
> to simulate the usual POSIX shell piping:
>
> process A -> stdout pipe -> process filter -> process-send-string ->
> stdin pipe -> process B
>
> If 'process-send-string' fails, we raise a Lisp signal 'eshell-pipe-
> broken', that tries to be as much like SIGPIPE as we can. Because of
> this indirection though, process A won't see an EPIPE/SIGPIPE at the
> normally-expected time
Thanks for the explanation. Is this documented? Might not hurt to do so
if it's not documented already. Maybe that can be part of your patch
series (and I can certainly sympathize with those patches taking time to
get right!)
> ;; The output pipe broke, so send SIGPIPE to the
> ;; process. NOTE: Due to the additional indirection
> ;; of Emacs process filters, the process will likely
> ;; see the SIGPIPE later than it would in a regular
> ;; shell, which could cause problems.
Suppose we change Emacs to not send SIGPIPE to A in this situation, but
instead to just close Emacs's end of A's output pipe. Then when A
eventually gets around to writing to its output pipe it will get a
SIGPIPE if it is not ignoring that signal, and an EPIPE failure
otherwise, and this will be closer to what happens with ordinary shell
pipes.
An objection to such a change, which is hinted at in the comment, is
that A might get a SIGPIPE later than it would with an ordinary shell
pipe, due to the extra buffering that Emacs supplies. But this objection
isn't a deal breaker, as A can't know how big the kernel pipe buffer is
or how the scheduler works, so it shouldn't rely on that precise of a
schedule anyway.
If I understand things correctly a problem with the current code is that
A can get a SIGPIPE too early, before it writes to its output pipe, or
even if it never writes to its output pipe. That contradicts the POSIX
behavior, and so is a more serious problem than A getting a SIGPIPE
later than usual (which is what the comment is worried about).
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Mon, 28 Jul 2025 01:50:02 GMT)
Full text and
rfc822 format available.
Message #110 received at 79079 <at> debbugs.gnu.org (full text, mbox):
Jim Porter <jporterbugs <at> gmail.com> writes:
> On 7/27/2025 12:53 PM, Daniel Mendler via Bug reports for GNU Emacs, the Swiss
> army knife of text editors wrote:
>> Only tangentially related to this bug report, but I wonder if it would
>> make sense to always use actual pipes in Eshell when possible, instead
>> of the process filter indirection?
>> There exists already the *| operator from `em-extpipe'. What prevents us
>> from making it the default (or enable it via customization) if `sh' is
>> available? The downside is that `sh' is started as an intermediate
>> process, but are their other downsides? For example piping though an
>> Elisp function, which transforms the input, would not work, but this is
>> not yet supported, or is it? As far as I understand *| is often more
>> efficient, despite starting a separate shell process.
>
> Eshell will support piping to Lisp commands Soon(TM). I'm finishing up a patch
> series that I hope to upload for discussion shortly. This has taken a *long*
> time to get to the point where I haven't run into any major issues, but it's
> finally coming together.
Thanks, I am looking forward to that!
> The larger problem is that it's impossible to determine at command-parsing time
> whether a particular command is implemented entirely in Lisp or will create a
> child process. "cat" is a good example: even as the leftmost command in a
> pipeline, sometimes it's a Lisp command, and other times, it creates a child
> process by throwing 'eshell-external' to replace the Lisp version of the
> command. That makes it challenging to connect the pipes up correctly, since we
> don't know which commands have real OS-provided pipes until command execution.
> I'm not saying it's impossible, but it's definitely not easy.
Makes sense.
Daniel
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Mon, 28 Jul 2025 05:36:02 GMT)
Full text and
rfc822 format available.
Message #113 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 7/27/2025 3:03 PM, Paul Eggert wrote:
> Thanks for the explanation. Is this documented? Might not hurt to do so
> if it's not documented already. Maybe that can be part of your patch
> series (and I can certainly sympathize with those patches taking time to
> get right!)
Not anywhere explicitly. Since I'm finishing up some patches to let
Eshell pipe to Lisp commands, it would probably be wise to explain some
of the details of how piping is different in Eshell in the manual.
>> ;; The output pipe broke, so send SIGPIPE to the
>> ;; process. NOTE: Due to the additional indirection
>> ;; of Emacs process filters, the process will likely
>> ;; see the SIGPIPE later than it would in a regular
>> ;; shell, which could cause problems.
>
> Suppose we change Emacs to not send SIGPIPE to A in this situation, but
> instead to just close Emacs's end of A's output pipe. Then when A
> eventually gets around to writing to its output pipe it will get a
> SIGPIPE if it is not ignoring that signal, and an EPIPE failure
> otherwise, and this will be closer to what happens with ordinary shell
> pipes.
That's a good idea. I think that would be a much cleaner solution than
what I wrote originally (which was somewhat limited since at the time I
didn't have much familiarity with src/process.c). While it wouldn't be
*exactly* the same as a POSIX shell, I think it would be close enough
that it would be hard for a program to tell the difference.
Some sort of 'process-close-stream' function that lets the caller close
the stdout or stderr stream for the process (or stdin, but I don't think
I need that here) should do the trick. Even better if we can support
that in Tramp.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Fri, 15 Aug 2025 20:56:02 GMT)
Full text and
rfc822 format available.
Message #116 received at 79079 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Attached is a patch for this with some regression tests (unfortunately,
we can only run them in an interactive session, since Emacs will
terminate itself in batch mode upon receiving a SIGPIPE). I've also
called out the change in NEWS so that users are aware.
On 7/26/2025 2:12 PM, Jim Porter wrote:
> After examining this more closely, there are three problems with the
> current Emacs code, of varying importance:
>
> 1. When we receive an EPIPE, we don't call our process-filter one last
> time (this is the original bug report). You can see this by running the
> following Eshell command repeatedly until it produces no output:
>
> git log | head -n 10 | cat
>
> 2. When we receive an EPIPE, we set the exit status to 256; that's not
> right, since the exit status should be 0. You can see this by running
> the following Eshell command until the prompt shows a non-zero status:
>
> git log | head -n 10 > #<my-buffer>
>
> 3. If the child process closes stdin and wants to keep doing some work
> (like write to stdout), it can't, since we called 'deactivate_process'.
> You can see this with the attached script by running (it's supposed to
> print the first line of "git log"):
>
> git log | ./test-script.sh
>
> I've also attached two WIP patches for the different implementations.
> "bad-call-filter.diff" uses 'read_process_output' to call our process
> filter. It only solves issue #1 above. "good-close-pipe.diff" (which I
> posted previously) solves all three issues. While the latter is a
> slightly bigger change and requires some more testing on my end to make
> sure I've covered 100% of the cases, I think it's the more-correct
> solution. If we make the bigger change, it's probably worth calling it
> out in NEWS and on emacs-devel so that people can be on the lookout for
> any edge cases I haven't identified.
[0001-Allow-child-processes-to-continue-after-EPIPE.patch (text/plain, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sat, 16 Aug 2025 06:05:02 GMT)
Full text and
rfc822 format available.
Message #119 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Fri, 15 Aug 2025 13:55:27 -0700
> From: Jim Porter <jporterbugs <at> gmail.com>
> Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
>
> else if (errno == EPIPE)
> {
> - p->raw_status_new = 0;
> - pset_status (p, list2 (Qexit, make_fixnum (256)));
> - p->tick = ++process_tick;
> - deactivate_process (proc);
> + close_process_fd(&p->open_fd[WRITE_TO_SUBPROCESS]);
^
SPC before the open parenthesis missing there.
> +(defun process-tests/broken-pipe (connection-type)
> + "Test handling of broken pipes; see bug#79079.
> +This test runs a shell script that reads a line of text and closes
> +stdin. We send two lines of text to the script; the second should
> +signal an error indicating that the pipe has been closed. The script
> +should also run to completion, printing out the line of text it read."
> + (with-temp-buffer
> + (let ((saw-error nil)
> + (proc (make-process
> + :name "test" :buffer (current-buffer)
> + :command '("sh" "-c"
> + "read line; exec 0<&-; sleep 1; echo $line")
This will not work on Windows. Can you come up with a test which
doesn't use features specific Posix shells? E.g., can you perhaps use
Emacs itself as the other end of the pipe, running some special Lisp
program?
> + (ert-deftest process-tests/broken-pipe/pty ()
> + (process-tests/broken-pipe 'pty))
> +
> + (ert-deftest process-tests/broken-pipe/pipe-stdin ()
> + (process-tests/broken-pipe '(pipe . pty)))
> +
> + (ert-deftest process-tests/broken-pipe/pty-stdin ()
> + (process-tests/broken-pipe '(pty . pipe))))
> +
The PTY tests should be skipped on Windows, since PTYs aren't
supported there.
Thanks.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sat, 16 Aug 2025 17:20:02 GMT)
Full text and
rfc822 format available.
Message #122 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 8/15/2025 11:04 PM, Eli Zaretskii wrote:
>> +(defun process-tests/broken-pipe (connection-type)
>> + "Test handling of broken pipes; see bug#79079.
>> +This test runs a shell script that reads a line of text and closes
>> +stdin. We send two lines of text to the script; the second should
>> +signal an error indicating that the pipe has been closed. The script
>> +should also run to completion, printing out the line of text it read."
>> + (with-temp-buffer
>> + (let ((saw-error nil)
>> + (proc (make-process
>> + :name "test" :buffer (current-buffer)
>> + :command '("sh" "-c"
>> + "read line; exec 0<&-; sleep 1; echo $line")
>
> This will not work on Windows. Can you come up with a test which
> doesn't use features specific Posix shells? E.g., can you perhaps use
> Emacs itself as the other end of the pipe, running some special Lisp
> program?
I thought that building Emacs on Windows used MSYS2, which provides a
Posix shell implementation. Am I just mistaken here?
It doesn't look like Emacs has the ability to close a file descriptor
from Lisp. I could add a C function to do this from Lisp, but I don't
know if it would be useful outside of this one test.
Another alternative would be to write a small C program to call for this
test. That's probably straightforward if I can figure out the right
place to insert the necessary Makefile logic...
>> + (ert-deftest process-tests/broken-pipe/pty ()
>> + (process-tests/broken-pipe 'pty))
>> +
>> + (ert-deftest process-tests/broken-pipe/pipe-stdin ()
>> + (process-tests/broken-pipe '(pipe . pty)))
>> +
>> + (ert-deftest process-tests/broken-pipe/pty-stdin ()
>> + (process-tests/broken-pipe '(pty . pipe))))
>> +
>
> The PTY tests should be skipped on Windows, since PTYs aren't
> supported there.
The tests aren't especially useful on Windows, but they should pass
(assuming the pipe test does), since on Windows we always use a pipe,
even if the user requests a PTY. Covering these cases would catch any
bugs related to that, and would also mean we don't need to adjust these
later if we one day can support PTYs on Windows (I know this is really
up to Microsoft fixing their bugs though).
That said, I don't really have a strong preference here and don't have a
problem with removing these calls.
(Finally, it'd be nice if we could run these tests within a batch Emacs
session, but then we'd need to arrange things to handle SIGPIPE in batch
Emacs.)
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sat, 16 Aug 2025 19:23:01 GMT)
Full text and
rfc822 format available.
Message #125 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Sat, 16 Aug 2025 10:19:21 -0700
> Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
> From: Jim Porter <jporterbugs <at> gmail.com>
>
> On 8/15/2025 11:04 PM, Eli Zaretskii wrote:
> >> +(defun process-tests/broken-pipe (connection-type)
> >> + "Test handling of broken pipes; see bug#79079.
> >> +This test runs a shell script that reads a line of text and closes
> >> +stdin. We send two lines of text to the script; the second should
> >> +signal an error indicating that the pipe has been closed. The script
> >> +should also run to completion, printing out the line of text it read."
> >> + (with-temp-buffer
> >> + (let ((saw-error nil)
> >> + (proc (make-process
> >> + :name "test" :buffer (current-buffer)
> >> + :command '("sh" "-c"
> >> + "read line; exec 0<&-; sleep 1; echo $line")
> >
> > This will not work on Windows. Can you come up with a test which
> > doesn't use features specific Posix shells? E.g., can you perhaps use
> > Emacs itself as the other end of the pipe, running some special Lisp
> > program?
>
> I thought that building Emacs on Windows used MSYS2, which provides a
> Posix shell implementation. Am I just mistaken here?
Building, yes. But this is a test suite, not a build script.
> It doesn't look like Emacs has the ability to close a file descriptor
> from Lisp. I could add a C function to do this from Lisp, but I don't
> know if it would be useful outside of this one test.
There's nothing wrong in having a function for the benefit of running
tests. If suitably documented, it's okay, I think.
> Another alternative would be to write a small C program to call for this
> test. That's probably straightforward if I can figure out the right
> place to insert the necessary Makefile logic...
That would need a C compiler to run the test suite, something that is
much less desirable.
> >> + (ert-deftest process-tests/broken-pipe/pty ()
> >> + (process-tests/broken-pipe 'pty))
> >> +
> >> + (ert-deftest process-tests/broken-pipe/pipe-stdin ()
> >> + (process-tests/broken-pipe '(pipe . pty)))
> >> +
> >> + (ert-deftest process-tests/broken-pipe/pty-stdin ()
> >> + (process-tests/broken-pipe '(pty . pipe))))
> >> +
> >
> > The PTY tests should be skipped on Windows, since PTYs aren't
> > supported there.
>
> The tests aren't especially useful on Windows, but they should pass
> (assuming the pipe test does), since on Windows we always use a pipe,
> even if the user requests a PTY.
What would be the purpose of running the same tests several times?
> Covering these cases would catch any
> bugs related to that, and would also mean we don't need to adjust these
> later if we one day can support PTYs on Windows (I know this is really
> up to Microsoft fixing their bugs though).
When we have a decent PTY support on Windows, this test will be the
least of our problems, given how much stuff will need to change.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sat, 16 Aug 2025 23:56:02 GMT)
Full text and
rfc822 format available.
Message #128 received at 79079 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On 8/16/2025 12:21 PM, Eli Zaretskii wrote:
>> Date: Sat, 16 Aug 2025 10:19:21 -0700
>> Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
>> From: Jim Porter <jporterbugs <at> gmail.com>
>>
>> It doesn't look like Emacs has the ability to close a file descriptor
>> from Lisp. I could add a C function to do this from Lisp, but I don't
>> know if it would be useful outside of this one test.
>
> There's nothing wrong in having a function for the benefit of running
> tests. If suitably documented, it's okay, I think.
Ok, now done. I've added a new function called 'internal-close-stream'.
Hopefully the documentation is clear enough (it's a pretty simple function).
>> The tests aren't especially useful on Windows, but they should pass
>> (assuming the pipe test does), since on Windows we always use a pipe,
>> even if the user requests a PTY.
>
> What would be the purpose of running the same tests several times?
Just one fewer conditional in the tests, but that's not a big deal. I've
change this to run only the "pipe" case under MS-Windows.
[0001-Allow-child-processes-to-continue-after-EPIPE.patch (text/plain, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 17 Aug 2025 05:04:02 GMT)
Full text and
rfc822 format available.
Message #131 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Sat, 16 Aug 2025 16:54:57 -0700
> Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
> From: Jim Porter <jporterbugs <at> gmail.com>
>
> +static FILE *
> +file_for_stream(Lisp_Object stream)
^^
SPC missing there.
> +{
> + CHECK_SYMBOL (stream);
Calls to CHECK_* should be made by the primitive itself, not by its
subroutine. This is to allow the subroutines to be called where such
tests were already done, or are inappropriate. The idea is that only
stuff exposed to Lisp validates the Lisp objects passed as arguments.
> +DEFUN ("internal-close-stream", Finternal_close_stream,
Nowadays we use "--" in the name of internal functions. I suggest to
call this file--close-stream instead.
> + Sinternal_close_stream, 1, 1, 0,
> + doc: /* Close the file associated with STREAM.
This doesn't close the file, it closes the STREAM. I suggest to
rephrase:
Close the standard STREAM of the Emacs process.
P.S. I'd like to see what this does on Windows, where there's neither
SIGPIPE nor EPIPE, but that's a separate funeral.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 17 Aug 2025 06:34:01 GMT)
Full text and
rfc822 format available.
Message #134 received at 79079 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On 8/16/2025 10:03 PM, Eli Zaretskii wrote:
> Calls to CHECK_* should be made by the primitive itself, not by its
> subroutine. This is to allow the subroutines to be called where such
> tests were already done, or are inappropriate. The idea is that only
> stuff exposed to Lisp validates the Lisp objects passed as arguments.
Good to know, thanks.
> P.S. I'd like to see what this does on Windows, where there's neither
> SIGPIPE nor EPIPE, but that's a separate funeral.
For what it's worth, Gnulib says the following:
/* On native Windows platforms, SIGPIPE does not exist. When write() is
called on a pipe with no readers, WriteFile() fails with error
GetLastError() = ERROR_NO_DATA, and write() in consequence fails with
error EINVAL. */
Gnulib's 'write' (well really, 'rpl_write') checks for ERROR_NO_DATA and
raises SIGPIPE for us, so hopefully we're ok here?
[0001-Allow-child-processes-to-continue-after-EPIPE.patch (text/plain, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 17 Aug 2025 07:58:01 GMT)
Full text and
rfc822 format available.
Message #137 received at 79079 <at> debbugs.gnu.org (full text, mbox):
> Date: Sat, 16 Aug 2025 23:33:49 -0700
> Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 79079 <at> debbugs.gnu.org
> From: Jim Porter <jporterbugs <at> gmail.com>
>
> > P.S. I'd like to see what this does on Windows, where there's neither
> > SIGPIPE nor EPIPE, but that's a separate funeral.
>
> For what it's worth, Gnulib says the following:
>
> /* On native Windows platforms, SIGPIPE does not exist. When write() is
> called on a pipe with no readers, WriteFile() fails with error
> GetLastError() = ERROR_NO_DATA, and write() in consequence fails with
> error EINVAL. */
>
> Gnulib's 'write' (well really, 'rpl_write') checks for ERROR_NO_DATA and
> raises SIGPIPE for us, so hopefully we're ok here?
We don't use Gnulib's replacement for 'write', so if anything is
needed in this area, we'll need to do it in w32.c:sys_write (which
also supports network and serial connections, so it would be
interesting what socket error is returned when the other side closes
the socket.
There's also ERROR_BROKEN_PIPE, which happens when _reading_ from a
pipe whose other end was closed. is that also EPIPE on Posix systems?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#79079; Package
emacs.
(Sun, 17 Aug 2025 17:29:01 GMT)
Full text and
rfc822 format available.
Message #140 received at 79079 <at> debbugs.gnu.org (full text, mbox):
On 8/17/2025 12:57 AM, Eli Zaretskii wrote:
> There's also ERROR_BROKEN_PIPE, which happens when _reading_ from a
> pipe whose other end was closed. is that also EPIPE on Posix systems?
On Posix, I think that would just be EOF.
This bug report was last modified 32 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.