GNU bug report logs - #13400
23.4; overlapping process filter calls

Previous Next

To reply to this bug, email your comments to 13400 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Hendrik Tews <hendrik <at> askra.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 23.4; overlapping process filter calls
Date: Thu, 10 Jan 2013 11:11:42 +0100

Hi,

because Stefan Monnier asked for it
(http://lists.inf.ed.ac.uk/pipermail/proofgeneral-devel/2013/000296.html):

When a process filter does I/O (eg, process-send-string), it
might happen, that the same filter is called again for the same
process, while the first instance blocks. To reproduce, load the
code below and do M-x double-filter.

The problem is real with Coq Proof General and Prooftree. There,
Prooftree is driven from a process filter in Proof General. With
cold disk caches, it takes about 1 sec to start Prooftree (at
least on my laptops). Coq and Proof General easily fill the pipe
in that time, so that the process filter blocks inside
process-send-string. Meanwhile Coq continues to produce output
and the process filter is called again while the previous call
has not finished.

The more I think about it, I believe this is a feature. 

However, the documentation is sometimes wrong and could be
clearer about this feature:

- Section "37.9 Receiving Output from Processes" does not list
  process-send-string. How about other blocking I/O functions?

- Same in "37.9.2. Process Filter Functions"

- Same in "37.4 Creating an Asynchronous Process" ,
  process-send-string is neither waiting for input not time
  delay.

- "37.7 Sending Input to Processes" says that filters can run
  inside process-send-string, but it could be clearer about the
  point that this can also happen inside the same filter for the
  same process.

- "37.9.2 Process Filter Functions" ignores the problem
  completely. There should be a paragraph clearly stating this
  problem. Further, it would be nice, if the filter function
  example could be extended to correctly deal with this problem.


To make it easier in the future to deal with this problem, I
suggest to add a process specific flag
``process-no-concurrent-filters''. When this flag is t for a
process, Emacs accepts output from this process inside a filter
but buffers it without calling the filter. The call to the filter
is delayed until a point where no filter for this process is
running. An error is signaled, if the buffered output exceeds a
certain size.


===========================================================================
(defun double-filter ()
  (interactive)
  (setq df-process (start-process "tee" nil "/bin/cat"))
  (set-process-filter df-process 'df-filter)
  (set-process-sentinel df-process 'df-sentinel)
  (set-process-query-on-exit-flag df-process nil)
  (setq df-buf (get-buffer-create "xxxx"))
  (process-send-string df-process "1\n"))

(setq df-filter-lock nil)
(setq df-error nil)

(defun df-filter (p str)
  (unless df-error
    (with-current-buffer df-buf
      (insert (format "filter received %d bytes lock %s\n"
		      (length str) df-filter-lock)))
    (when df-filter-lock
      (setq df-error t)
      (error "recursive filter"))
    (setq df-filter-lock t)
    (setq long (make-string 4096 65))
    (process-send-string df-process long)
    (process-send-string df-process long)
    (process-send-string df-process long)
    (process-send-string df-process long)
    (process-send-string df-process long)
    (process-send-string df-process long)
    (process-send-string df-process long)
    (process-send-string df-process long)
    (setq df-filter-lock nil)))

(defun df-sentinel (p event)
  (setq df-error t)
  (with-current-buffer df-bug
    (insert "process died\n")))
==============================================================================
 



In GNU Emacs 23.4.1 (i486-pc-linux-gnu, GTK+ Version 2.24.10)
 of 2012-09-09 on murphy, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.11204000
configured using `configure  '--build' 'i486-linux-gnu' '--build' 'i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.4/site-lisp:/usr/share/emacs/site-lisp' '--with-crt-dir=/usr/lib/i386-linux-gnu' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=i486-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -DDEBIAN -O2' 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  show-paren-mode: t
  msb-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Message #8 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Hendrik Tews <hendrik <at> askra.de>
Cc: 13400 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Fri, 26 Jul 2019 23:38:46 -0400

[Message part 1 (text/plain, inline)]

Hendrik Tews <hendrik <at> askra.de> writes:

> because Stefan Monnier asked for it
> (http://lists.inf.ed.ac.uk/pipermail/proofgeneral-devel/2013/000296.html)

[Note: meanwhile the section number has changed to 38 instead of 37.]

> - Section "37.9 Receiving Output from Processes" does not list
>   process-send-string. How about other blocking I/O functions?

In the attached patch, I've added a mention/xref for functions which send
data to processes.

> - Same in "37.9.2. Process Filter Functions"

This section is repeated twice (I addressed the second instance below).

> - Same in "37.4 Creating an Asynchronous Process" ,
>   process-send-string is neither waiting for input not time
>   delay.

I don't see any mention of process-send-string in that section, nor how
it's relevant to the rest of this report.

> - "37.7 Sending Input to Processes" says that filters can run
>   inside process-send-string, but it could be clearer about the
>   point that this can also happen inside the same filter for the
>   same process.

I'm not really convinced that is necessary.

> - "37.9.2 Process Filter Functions" ignores the problem
>   completely. There should be a paragraph clearly stating this
>   problem. Further, it would be nice, if the filter function
>   example could be extended to correctly deal with this problem.

I added a mention of the possibility of recursion.  I'm not sure about
making an example (specifically, what is the best way to deal with this
problem?).

[0001-Note-that-process-filter-can-be-called-recursively-B.patch (text/plain, attachment)]

[Message part 3 (text/plain, inline)]

> To make it easier in the future to deal with this problem, I
> suggest to add a process specific flag
> ``process-no-concurrent-filters''. When this flag is t for a
> process, Emacs accepts output from this process inside a filter
> but buffers it without calling the filter. The call to the filter
> is delayed until a point where no filter for this process is
> running. An error is signaled, if the buffered output exceeds a
> certain size.

I thought of just making a wrapper in Lisp instead, this saves the need
to complicate the process C code with yet another flag; it's already
tricky enough as it is.

;;; -*- lexical-binding: t -*-

(defun make-buffered-filter (filter)
  (let ((filtering nil)
        (buffered nil)
        (process nil))
    (lambda (proc str)
      (if process
          (unless (eq process proc)
            (error "Buffered filter used in different processes: %S, %S"
                   proc process))
        (setq process proc))
      (push str buffered)
      (unless filtering
        (setq filtering t)
        (unwind-protect
            (while buffered
              (setq str (apply #'concat (nreverse buffered)))
              (setq buffered nil)
              (funcall filter proc str))
          (setq filtering nil))))))

;; Can be used like
(set-process-filter my-process (make-buffered-filter #'my-filter-function))

Message #11 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: hendrik <at> askra.de, 13400 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Sat, 27 Jul 2019 11:24:25 +0300

> From: Noam Postavsky <npostavs <at> gmail.com>
> Date: Fri, 26 Jul 2019 23:38:46 -0400
> Cc: 13400 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
> 
> > - Section "37.9 Receiving Output from Processes" does not list
> >   process-send-string. How about other blocking I/O functions?
> 
> In the attached patch, I've added a mention/xref for functions which send
> data to processes.
> 
> > - Same in "37.9.2. Process Filter Functions"
> 
> This section is repeated twice (I addressed the second instance below).
> 
> > - Same in "37.4 Creating an Asynchronous Process" ,
> >   process-send-string is neither waiting for input not time
> >   delay.
> 
> I don't see any mention of process-send-string in that section, nor how
> it's relevant to the rest of this report.
> 
> > - "37.7 Sending Input to Processes" says that filters can run
> >   inside process-send-string, but it could be clearer about the
> >   point that this can also happen inside the same filter for the
> >   same process.
> 
> I'm not really convinced that is necessary.
> 
> > - "37.9.2 Process Filter Functions" ignores the problem
> >   completely. There should be a paragraph clearly stating this
> >   problem. Further, it would be nice, if the filter function
> >   example could be extended to correctly deal with this problem.
> 
> I added a mention of the possibility of recursion.  I'm not sure about
> making an example (specifically, what is the best way to deal with this
> problem?).

I agree with Noam's decisions, and think that his patch is fine.

Thanks.

Message #14 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: hendrik <at> askra.de, 13400 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Sat, 03 Aug 2019 20:02:40 -0400

Eli Zaretskii <eliz <at> gnu.org> writes:

>> > - "37.9.2 Process Filter Functions" ignores the problem
>> >   completely. There should be a paragraph clearly stating this
>> >   problem. Further, it would be nice, if the filter function
>> >   example could be extended to correctly deal with this problem.
>> 
>> I added a mention of the possibility of recursion.  I'm not sure about
>> making an example (specifically, what is the best way to deal with this
>> problem?).
>
> I agree with Noam's decisions, and think that his patch is fine.
>
> Thanks.

Any thoughts about make-buffered-filter?  Leave it up to callers to
write their own most situation-appropriate version of it?  Add it as an
example to the manual?  Add it to Emacs?

Message #17 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: hendrik <at> askra.de, 13400 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Sun, 04 Aug 2019 19:29:59 +0300

> From: Noam Postavsky <npostavs <at> gmail.com>
> Cc: hendrik <at> askra.de,  13400 <at> debbugs.gnu.org,  monnier <at> iro.umontreal.ca
> Date: Sat, 03 Aug 2019 20:02:40 -0400
> 
> Any thoughts about make-buffered-filter?  Leave it up to callers to
> write their own most situation-appropriate version of it?  Add it as an
> example to the manual?  Add it to Emacs?

I'd go with the example in the manual alternative.

Message #20 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13400 <at> debbugs.gnu.org, hendrik <at> askra.de
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Mon, 05 Aug 2019 14:31:39 -0400

>>> > - "37.9.2 Process Filter Functions" ignores the problem
>>> >   completely. There should be a paragraph clearly stating this
>>> >   problem. Further, it would be nice, if the filter function
>>> >   example could be extended to correctly deal with this problem.
>>> 
>>> I added a mention of the possibility of recursion.  I'm not sure about
>>> making an example (specifically, what is the best way to deal with this
>>> problem?).
>>
>> I agree with Noam's decisions, and think that his patch is fine.
>>
>> Thanks.
>
> Any thoughts about make-buffered-filter?  Leave it up to callers to
> write their own most situation-appropriate version of it?  Add it as an
> example to the manual?  Add it to Emacs?

I'm pretty sure that while there might be use-cases for recursive
invocation of filters, this is a very exceptional situation and the
average programmer will not expect it and would be stumped if/when
it happens.

So the default should be to prevent it, with maybe some way to override
it to cater to the exceptional case where recursive invocation
is to be allowed.

I think we could do that by setting a property on process object during
filter invocation to postpone further filter invocations, and then the
process filter could locally unset this property if it wants to allow
recursive invocations.


        Stefan

Message #23 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Hendrik Tews <hendrik <at> askra.de>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: 13400 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Tue, 06 Aug 2019 00:37:19 +0200

Noam,

thanks for addressing this quite old bug report. I do have some
comments:

Noam Postavsky <npostavs <at> gmail.com> writes:

> Hendrik Tews <hendrik <at> askra.de> writes:
>
>> because Stefan Monnier asked for it
>> (http://lists.inf.ed.ac.uk/pipermail/proofgeneral-devel/2013/000296.html)
>
> [Note: meanwhile the section number has changed to 38 instead of 37.]
>
>> - Section "37.9 Receiving Output from Processes" does not list
>>   process-send-string. How about other blocking I/O functions?
>
> In the attached patch, I've added a mention/xref for functions which send
> data to processes.

How about other blocking I/O functions? When output is accepted
inside process-send-string, then it probably is in all
potentially blocking I/O functions? The list of functions in 38.9
after "Output from a subprocess can arrive only" should be
complete. If the programmer cannot rely on the documentation to
be complete here, he or she has no other choice than to assume
that output can arrive anywhere and he/she _is_ plagued with the
usual parallel programming problems.

Therefore, please make a careful investigation to ensure that the
list of functions in which output can arrive is _really_ complete
in the documentation.

>> - Same in "37.9.2. Process Filter Functions"
>
> This section is repeated twice (I addressed the second instance below).

I mentioned this section twice, because there are _two_
documentation problems. I really believe you should address both
and not just one. The first problem in 38.9.2 is the same as
above: The list of functions after "The filter function can only
be called" is far from complete.

The second problem is that it does not document the possibility
of recursive filter calls.

>> - Same in "37.4 Creating an Asynchronous Process" ,
>>   process-send-string is neither waiting for input not time
>>   delay.
>
> I don't see any mention of process-send-string in that section, nor how
> it's relevant to the rest of this report.

Come on, please read that section carefully. "Emacs accepts data
from the process only while waiting for input or for a time
delay" in there implies that Emacs is _not_ accepting data during
process-send-string, because, as I wrote, 

>>   process-send-string is neither waiting for input no[r] time
>>   delay.

Therefore, this section implies that Emacs is _not_ accepting
data during process-send-string. If it does, it is a bug.

>> - "37.7 Sending Input to Processes" says that filters can run
>>   inside process-send-string, but it could be clearer about the
>>   point that this can also happen inside the same filter for the
>>   same process.
>
> I'm not really convinced that is necessary.

It is about a few words making the documentation more precise,
potentially saving somebody a painful debugging session of
several hours - similar to what I went through in 2013. Back
then, I was lucky because clearing the disk caches was enough to
reproduce the problem. Today with SSD's it is probably much
harder...

Adding to the original bug report, I would suggest to restructure
the documentation, such that there is only one section
documenting all the functions in which process output could
arrive and such that all the other sections only refer to that
section. It should really not be the case that different sections
make different and sometimes inconsistent statements about the
same feature.

Hendrik

Message #26 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Hendrik Tews <hendrik <at> askra.de>
Cc: 13400 <at> debbugs.gnu.org, Noam Postavsky <npostavs <at> gmail.com>
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Tue, 06 Aug 2019 03:40:44 -0400

> complete. If the programmer cannot rely on the documentation to
> be complete here, he or she has no other choice than to assume
> that output can arrive anywhere and he/she _is_ plagued with the
> usual parallel programming problems.

While I largely agree with your points, I'll mention that we *are*
plagued, because lots of functions may (in some cases) end up calling
a blocking operation.  E.g. any call to a file function can block if the
file happens to be handled by Tramp, so any call to a function which
(transitively) calls a file function can itself block.

> It is about a few words making the documentation more precise,
> potentially saving somebody a painful debugging session of
> several hours - similar to what I went through in 2013. Back
> then, I was lucky because clearing the disk caches was enough to
> reproduce the problem. Today with SSD's it is probably much
> harder...

I think preventing recursive filter invocations (by default) would be
much better than warning the programmer about that odd possibility.


        Stefan

Message #29 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Hendrik Tews <hendrik <at> askra.de>
Cc: 13400 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Wed, 07 Aug 2019 21:15:49 -0400

[Message part 1 (text/plain, inline)]

Hendrik Tews <hendrik <at> askra.de> writes:

> thanks for addressing this quite old bug report. I do have some
> comments:

Thanks for still following up even though the reply was so delayed.

>>> - Section "37.9 Receiving Output from Processes" does not list
>>>   process-send-string. How about other blocking I/O functions?
>>
>> In the attached patch, I've added a mention/xref for functions which send
>> data to processes.
>
> How about other blocking I/O functions?  When output is accepted
> inside process-send-string, then it probably is in all potentially
> blocking I/O functions?

As far as I know, all the blocking functions are listed in the Input to
Processes node which I xref'd.

>>> - Same in "37.9.2. Process Filter Functions"
>>
>> This section is repeated twice (I addressed the second instance below).
>
> I mentioned this section twice, because there are _two_
> documentation problems.

Aha, missed that, thanks.

>>> - Same in "37.4 Creating an Asynchronous Process" ,
>>>   process-send-string is neither waiting for input not time
>>>   delay.
>>
>> I don't see any mention of process-send-string in that section, nor how
>> it's relevant to the rest of this report.
>
> Come on, please read that section carefully. "Emacs accepts data
> from the process only while waiting for input or for a time
> delay" in there implies that Emacs is _not_ accepting data during
> process-send-string, because, as I wrote, 

Thanks, it's hard to pick out such details on the Nth time reading the
manual.

>>> - "37.7 Sending Input to Processes" says that filters can run
>>>   inside process-send-string, but it could be clearer about the
>>>   point that this can also happen inside the same filter for the
>>>   same process.
>>
>> I'm not really convinced that is necessary.
>
> It is about a few words making the documentation more precise,
> potentially saving somebody a painful debugging session of
> several hours

I'm just not sure anyone is going to notice such details when it really
matters.  In my experience the manual is more about having something
authoritative to point to when folks ask "why does X happen?".  But I've
added a parenthetical to the patch.

> Adding to the original bug report, I would suggest to restructure
> the documentation, such that there is only one section
> documenting all the functions in which process output could
> arrive and such that all the other sections only refer to that
> section. It should really not be the case that different sections
> make different and sometimes inconsistent statements about the
> same feature.

Yes, hence the xrefs.  See attached updated patch.

[0001-Note-that-process-filter-can-be-called-recursively-B.patch (text/plain, attachment)]

Message #32 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13400 <at> debbugs.gnu.org, hendrik <at> askra.de
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Wed, 07 Aug 2019 23:37:29 -0400

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

> I'm pretty sure that while there might be use-cases for recursive
> invocation of filters, this is a very exceptional situation and the
> average programmer will not expect it and would be stumped if/when
> it happens.
>
> So the default should be to prevent it, with maybe some way to override
> it to cater to the exceptional case where recursive invocation
> is to be allowed.
>
> I think we could do that by setting a property on process object during
> filter invocation to postpone further filter invocations, and then the
> process filter could locally unset this property if it wants to allow
> recursive invocations.

Yeah, I suppose that would probably be better.  Although then we have
some potential weird edge cases like what happens when when changing the
property during a recursive invocation.

Message #35 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: hendrik <at> askra.de, 13400 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Thu, 08 Aug 2019 16:36:20 +0300

> From: Noam Postavsky <npostavs <at> gmail.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  13400 <at> debbugs.gnu.org,  hendrik <at> askra.de
> Date: Wed, 07 Aug 2019 23:37:29 -0400
> 
> > So the default should be to prevent it, with maybe some way to override
> > it to cater to the exceptional case where recursive invocation
> > is to be allowed.
> >
> > I think we could do that by setting a property on process object during
> > filter invocation to postpone further filter invocations, and then the
> > process filter could locally unset this property if it wants to allow
> > recursive invocations.
> 
> Yeah, I suppose that would probably be better.  Although then we have
> some potential weird edge cases like what happens when when changing the
> property during a recursive invocation.

Hmm... I'm not sure I understand what would this mean in practice.
Suppose a process filter invokes some blocking API, which then calls
wait_reading_process_output, and 'pselect' tells us that same process
can be read from again.  How will we avoid calling the filter
recursively in this case, and what will we do instead of calling it?

Message #38 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: hendrik <at> askra.de, 13400 <at> debbugs.gnu.org,
 Noam Postavsky <npostavs <at> gmail.com>
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Fri, 09 Aug 2019 17:36:34 -0400

> Hmm... I'm not sure I understand what would this mean in practice.
> Suppose a process filter invokes some blocking API, which then calls
> wait_reading_process_output, and 'pselect' tells us that same process
> can be read from again.

I think we shouldn't pass that process's handle to pselect.


        Stefan

Message #41 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13400 <at> debbugs.gnu.org, hendrik <at> askra.de
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Fri, 09 Aug 2019 21:39:16 -0400

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>> Hmm... I'm not sure I understand what would this mean in practice.
>> Suppose a process filter invokes some blocking API, which then calls
>> wait_reading_process_output, and 'pselect' tells us that same process
>> can be read from again.
>
> I think we shouldn't pass that process's handle to pselect.

That sounds like it would equivalent to setting the process' filter
function to t while running its body.  If you try that with the example
in the OP, you get a deadlock, because the subprocess' pipe becomes full
as Emacs stops reading it, and its input pipe becomes full so Emacs gets
stuck when trying to send data to it.

I thought you meant something more like my make-buffered-filter example,
where Emacs would still read from the process, but not call the filter
function with the new data until the current invocation ends (i.e.,
Emacs would just temporarily save the data).

Message #44 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 13400 <at> debbugs.gnu.org, hendrik <at> askra.de
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Sat, 10 Aug 2019 05:03:24 -0400

> That sounds like it would equivalent to setting the process' filter
> function to t while running its body.

Yes.

> If you try that with the example in the OP, you get a deadlock,
> because the subprocess' pipe becomes full as Emacs stops reading it,
> and its input pipe becomes full so Emacs gets stuck when trying to
> send data to it.

Indeed.  It's a classic problem with popen-like setups.

It's a problem if Emacs itself gets stuck preventing other things to
run, but otherwise I don't see it as a real problem: the OP's example
is artificial.

> I thought you meant something more like my make-buffered-filter example,
> where Emacs would still read from the process, but not call the filter
> function with the new data until the current invocation ends (i.e.,
> Emacs would just temporarily save the data).

That's another option.


        Stefan

Message #47 received at 13400 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Hendrik Tews <hendrik <at> askra.de>
Cc: 13400 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#13400: 23.4; overlapping process filter calls
Date: Tue, 20 Aug 2019 08:19:17 -0400

> See attached updated patch.

I've pushed it to emacs-26.  Leaving the bug open for now, since we
might still want to actually change the behaviour.

615cff4258 2019-08-19T19:49:50-04:00 "Fix process filter documentation (Bug#13400)"
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=615cff42580a3521c1a4ea7c3ec467eb8259e1c7

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.