Subject: bug#13188: Whats' the proper senario of par-map? (Was Re: bug#13188: par-map causes VM stack overflow)

From: Mark H Weaver <[email-masked]>
Date: Thu, 28 Mar 2013 14:00:52 -0400

Hi Ludovic,

[email-masked] (Ludovic Courtès) writes:

> Mark H Weaver <[email-masked]> skribis:
>
>> It only makes sense to use 'par-map' when the procedure is fairly
>> expensive to compute.
>
> Indeed.
>
>> There is inevitably a lot of overhead in creating and joining the
>> threads.
>
> We use a thread pool, so there’s no such cost.

Sorry, I was using the term 'threads' not in the sense of OS-level
threads, but in a more general sense.  I should have been more clear.

What I meant is that from the user's perspective, threads are being
created and joined, and even if you build those using a pool of OS-level
threads, this inevitably involves thread synchronization, which is very
expensive on modern architectures.  So I maintain that there _is_ such a
cost, and it can't be avoided.

The point I was really trying to make here, in the simplest possible
terms, is that it will *never* make sense to replace all uses of 'map'
with 'par-map' wherever it is safe to do so.

> But there are other costs.  When delimited continuations are used, we’re
> on the slow path.  Also, Guile’s fat mutexes & co. are terribly
> inefficient.  And finally, there may be contention on the futexes mutex
> (esp. when the computations is too small.)

Indeed, I wouldn't be surprised if we could improve this by an order of
magnitude.  More items for my TODO list :)

> So yes, there’s room for improvement.  Yet, it should be fruitful,
> provided you use it for reasonably long computations, as Mark outlines.

     Regards,
       Mark