Multiprocessing: don't push the pedal to the metal?

[Adam Tauno Williams]

On Mon, 2011-05-23 at 10:32 +1000, Chris Angelico wrote:
> On Mon, May 23, 2011 at 7:06 AM, John Ladasky <ladasky at my-deja.com> wrote:
> > If I spawn N worker sub-processes, my application in fact has N+1
> > processes in all, because there's also the master process itself.
> > I'd still appreciate hearing from anyone else who has more experience
> > with multiprocessing.  If there are general rules about how to do this
> > best, I haven't seen them posted anywhere.  This may not be a Python-
> > specific issue, of course.
> I don't have much experience with Python's multiprocessing model, but
> I've done concurrent programming on a variety of platforms, and there
> are some common issues.

I develop an app that uses multiprocessing heavily.  Remember that all
these processes are processes - so you can use all the OS facilities
regarding processes on them.  This includes setting nice values,
schedular options, CPU pinning, etc...

> Each CPU (or core) has its own execution cache. If you can keep one
> thread running on the same core all the time, it will benefit more
> from that cache than if it has to keep flitting from one to another.

+1

> You undoubtedly will have other processes in the system, too. As well
> as your master, there'll be processes over which you have no control
> (unless you're on a bare-bones system). Some of them may preempt your
> processes.

This is very true.  You get a benefit from dividing work up to the
correct number of processes - but too many processes will quickly take
back all the benefit.  One good trick is to have the parent monitor the
load average and only spawn additional workers when that value is below
a certain value.