Subject: Re: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores

From: Dmitry Gutov <[email-masked]>
Date: Tue, 12 Sep 2023 02:06:50 +0300

On 11/09/2023 14:57, Eli Zaretskii wrote:
>> Date: Mon, 11 Sep 2023 03:02:55 +0300
>> Cc: [email-masked], [email-masked], [email-masked],
>>   [email-masked]
>> From: Dmitry Gutov <[email-masked]>
>>
>>> You could record its value in a local variable at the entry to
>>> garbage_collect, and the expose that value to Lisp.
>>
>> That also doesn't seem to give much, given that the condition for
>> entering 'maybe_garbage_collect' is (consing_until_gc < 0). I.e. we wait
>> until it's down to 0, then garbage-collect.
> 
> No, we don't wait until it's zero, we perform GC on the first
> opportunity that we _notice_ that it crossed zero.  So examining how
> negative is the value of consing_until_gc when GC is actually
> performed could tell us whether we checked the threshold with high
> enough frequency, and comparing these values between different runs
> could tell us whether the shorter time spend in GC means really less
> garbage or less frequent checks for the need to GC.

Good point, I'm attaching the same outputs with "last value of 
consing_until_gc" added to every line.

There are some pretty low values in the "read-process-output-max 409600" 
part of the experiment, which probably means runtime staying in C 
accumulating the output into the (now larger) buffer? Not sure.

>> What's in there? First of all, for find-directory-files-recursively-3,
>> there are 0 garbage collections between the beginning of the function
>> and when we start parsing the output (no GCs while the process is
>> writing to the buffer synchronously). I guess inserting output in a
>> buffer doesn't increase consing, so there's nothing to GC?
> 
> No, we just don't count increasing size of buffer text in the "consing
> since GC" counter.  Basically, buffer text is never "garbage", except
> when a buffer is killed.

That makes sense. Perhaps it hints at a faster design for calling 
process asynchronously as well (more on another experiment later).

>> Next: for find-directory-files-recursively-2, the process only finishes
>> at the end, when all GC cycles are done for. I suppose that also means
>> we block the process's output while Lisp is running, and also that
>> whatever GC events occur might coincide with the chunks of output coming
>> from the process, and however many of them turn out to be in total.
> 
> We don't block the process when GC runs.  We do stop reading from the
> process, so if and when the pipe fills, the OS will block the process.

Right. But the effect is almost the same, including the potential 
side-effect that (IIUC) when a process is waiting like that, it's just 
suspended and not rushing ahead using the CPU/disk/etc resources to the 
max. That's an orthogonal train of thought, sorry.