[Pythonmac-SIG] Is this a reasonable way to do multiprocessing?
Nat Echols
nathaniel.echols at gmail.com
Tue Apr 19 19:39:27 CEST 2011
On Tue, Apr 19, 2011 at 10:11 AM, Lou Pecora <lou_boog2000 at yahoo.com> wrote:
> But as many of you probably know that raises a pickle error. I did a
> little
> research and don't fully understand, but I thought of a way around this
> that is
> pedestrian, but it works on my toy example. I just make (deep) copies of my
> object and let a helper function call mainclass' aMethod. So I change the
> code
(after my class definition) to,
>
This is overkill. I don't understand the precise technical explanation for
the error, but I think it just means that you can't pickle an object's
method - only the object itself (assuming it doesn't have features that
break pickling). So what I do is this:
import multiprocessing as MP
class mainclass (object) :
def __call__ (self, x) :
# do something
return result
myclass=mainclass()
pool= MP.Pool(processes= 3) # 3 CPU workers, for example
theargs= [x1,x2,x3]
resultlist= pool.map(myclass, thargs)
The only time I use a helper function like what you describe is when the
iterable objects contain both the method of interest and the arguments,
e.g.:
class mainclass (object):
def __init__ (self, x) :
self.x = x
def run (self) :
return x**2
def run_many (obj) :
return obj.run()
args = []
for x in [13,29,71] :
args.append(mainclass(x))
p = MP.Pool(processes=3)
p.map(run_many, args)
Which version to use is very case-dependent - my code has both.
Further question, on an 8 CPU MacBook Pro I get a 4X speedup (with 8 CPU
> workers), not 8X (Activity Monitor shows all 8 CPUs are utilized ~100%).
> Is
> there something about dual core processors that I should know about? (since
> 4 =
> 8/2).
>
The top-of-the-line MacBook Pro only has 4 cores - I think you're confusing
Intel's "hyper-threading" with actual cores (not surprising, since software
tends to confuse them too, e.g. Linux kernel). You are very unlikely to see
a significant speedup using 8 processes versus 4 with the kind of crude
parallelization implemented in the Python multiprocessing module. (I assume
that lower-level threading tools like OpenMP will use hyperthreading more
efficiently, but I've never tested this.)
-Nat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20110419/b9954e12/attachment.html>
More information about the Pythonmac-SIG
mailing list