Adding a Par construct to Python?

[Nick Craig-Wood]

Steven D'Aprano <steven at REMOVE.THIS.cybersource.com.au> wrote:
>  On Tue, 19 May 2009 05:52:04 -0500, Grant Edwards wrote:
> 
> > On 2009-05-19, Steven D'Aprano <steven at REMOVE.THIS.cybersource.com.au>
> > wrote:
> >> On Mon, 18 May 2009 02:27:06 -0700, jeremy wrote:
> >>
> >>> Let me clarify what I think par, pmap, pfilter and preduce would mean
> >>> and how they would be implemented.
> >> [...]
> >>
> >> Just for fun, I've implemented a parallel-map function, and done a
> >> couple of tests. Comments, criticism and improvements welcome!
> > 
> > My only comment would be that your "slow function" might not be a very
> > simulation for the general-case, since it uses time.sleep() which
> > releases the GIL:
> 
> 
>  I didn't expect my code to magically overcome fundamental limitations of 
>  the CPython interpreter :)
> 
> 
> 
> >> def f(arg):  # Simulate a slow function.
> >>     time.sleep(0.5)
> >>     return 3*arg-2
> > 
> > Any Python function that isn't calling a library function written in C
> > that releases the GIL won't show any speedup will it?
> 
>  Not necessarily. Here's another function, that uses a loop instead of 
>  sleep.
> 
>  def g(arg, SIZE=8*10**6):
>      # Default SIZE is chosen so that on my machine, the loop 
>      # takes approximately 0.5 second.
>      for x in xrange(SIZE):
>          pass
>      return 3*arg-2
> 
> 
> >>> setup = 'from __main__ import pmap, g; data = range(50)'
> >>> min(Timer('map(g, data)', setup).repeat(repeat=5, number=3))
>  65.093590974807739
> >>> min(Timer('pmap(g, data)', setup).repeat(repeat=5, number=3))
>  20.268381118774414

I don't think that can be right - that shows python working without
the GIL contention.  So unless you ran it under IronPython?

Here is what happens when I run it under CPython 2.5 on my dual core
laptop.  I made SIZE=10**6 because I got bored of waiting ;-)

map
9.85280895233
pmap
28.4256689548

So the pmap took nearly 3 times as long.  I expect this is because the
task was divided into 5 sections each competing madly for the GIL.

I ran the same script under the latest jython beta which was very
interesting! pmap showing a slight improvement, and faster than
cPython!

$ jython2.5rc2/jython pmap.py
map
6.242000103
pmap
5.88800001144

-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick