[Pythonmac-SIG] Is this a reasonable way to do multiprocessing?

Nat Echols nathaniel.echols at gmail.com
Tue Apr 19 19:39:27 CEST 2011


On Tue, Apr 19, 2011 at 10:11 AM, Lou Pecora <lou_boog2000 at yahoo.com> wrote:

> But as many of you probably know that raises a pickle error.  I did a
> little
> research and don't fully understand, but I thought of a way around this
> that is
> pedestrian, but it works on my toy example. I just make (deep) copies of my
> object and let a helper function call mainclass'  aMethod. So I change the
> code

(after my class definition) to,
>

This is overkill.  I don't understand the precise technical explanation for
the error, but I think it just means that you can't pickle an object's
method - only the object itself (assuming it doesn't have features that
break pickling).  So what I do is this:

import multiprocessing as MP
class mainclass (object) :
  def __call__ (self, x) :
     # do something
     return result

myclass=mainclass()
pool= MP.Pool(processes= 3)  # 3 CPU workers, for example
theargs= [x1,x2,x3]
resultlist= pool.map(myclass, thargs)

The only time I use a helper function like what you describe is when the
iterable objects contain both the method of interest and the arguments,
e.g.:

class mainclass (object):
  def __init__ (self, x) :
     self.x = x

  def run (self) :
     return x**2

def run_many (obj) :
  return obj.run()

args = []
for x in [13,29,71] :
   args.append(mainclass(x))
p = MP.Pool(processes=3)
p.map(run_many, args)

Which version to use is very case-dependent - my code has both.

Further question, on an 8 CPU MacBook Pro I get a 4X speedup (with 8 CPU
> workers), not 8X (Activity Monitor shows all 8 CPUs are utilized ~100%).
>  Is
> there something about dual core processors that I should know about? (since
> 4 =
> 8/2).
>

The top-of-the-line MacBook Pro only has 4 cores - I think you're confusing
Intel's "hyper-threading" with actual cores (not surprising, since software
tends to confuse them too, e.g. Linux kernel).  You are very unlikely to see
a significant speedup using 8 processes versus 4 with the kind of crude
parallelization implemented in the Python multiprocessing module.  (I assume
that lower-level threading tools like OpenMP will use hyperthreading more
efficiently, but I've never tested this.)

-Nat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20110419/b9954e12/attachment.html>


More information about the Pythonmac-SIG mailing list