Adding a Par construct to Python?
nick at craig-wood.com
Thu May 21 11:29:55 EDT 2009
Steven D'Aprano <steven at REMOVE.THIS.cybersource.com.au> wrote:
> On Tue, 19 May 2009 05:52:04 -0500, Grant Edwards wrote:
> > On 2009-05-19, Steven D'Aprano <steven at REMOVE.THIS.cybersource.com.au>
> > wrote:
> >> On Mon, 18 May 2009 02:27:06 -0700, jeremy wrote:
> >>> Let me clarify what I think par, pmap, pfilter and preduce would mean
> >>> and how they would be implemented.
> >> [...]
> >> Just for fun, I've implemented a parallel-map function, and done a
> >> couple of tests. Comments, criticism and improvements welcome!
> > My only comment would be that your "slow function" might not be a very
> > simulation for the general-case, since it uses time.sleep() which
> > releases the GIL:
> I didn't expect my code to magically overcome fundamental limitations of
> the CPython interpreter :)
> >> def f(arg): # Simulate a slow function.
> >> time.sleep(0.5)
> >> return 3*arg-2
> > Any Python function that isn't calling a library function written in C
> > that releases the GIL won't show any speedup will it?
> Not necessarily. Here's another function, that uses a loop instead of
> def g(arg, SIZE=8*10**6):
> # Default SIZE is chosen so that on my machine, the loop
> # takes approximately 0.5 second.
> for x in xrange(SIZE):
> return 3*arg-2
> >>> setup = 'from __main__ import pmap, g; data = range(50)'
> >>> min(Timer('map(g, data)', setup).repeat(repeat=5, number=3))
> >>> min(Timer('pmap(g, data)', setup).repeat(repeat=5, number=3))
I don't think that can be right - that shows python working without
the GIL contention. So unless you ran it under IronPython?
Here is what happens when I run it under CPython 2.5 on my dual core
laptop. I made SIZE=10**6 because I got bored of waiting ;-)
So the pmap took nearly 3 times as long. I expect this is because the
task was divided into 5 sections each competing madly for the GIL.
I ran the same script under the latest jython beta which was very
interesting! pmap showing a slight improvement, and faster than
$ jython2.5rc2/jython pmap.py
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick
More information about the Python-list