multiprocessing vs thread performance
prologic at shortcircuit.net.au
Tue Dec 30 01:05:06 CET 2008
On Tue, Dec 30, 2008 at 12:52 AM, mk <mrkafk at gmail.com> wrote:
> Hello everyone,
> After reading http://www.python.org/dev/peps/pep-0371/ I was under
> impression that performance of multiprocessing package is similar to that of
> thread / threading. However, to familiarize myself with both packages I
> wrote my own test of spawning and returning 100,000 empty threads or
> processes (while maintaining at most 100 processes / threads active at any
> one time), respectively.
> The results I got are very different from the benchmark quoted in PEP 371.
> On twin Xeon machine the threaded version executed in 5.54 secs, while
> multiprocessing version took over 222 secs to complete!
> Am I doing smth wrong in code below? Or do I have to use
> multiprocessing.Pool to get any decent results?
The overhead in starting OS level processes
is quite high. This is why event-driven, single
process servers can perform far better than
ones that fork (spawn multiple processes)
As others have mentioned, it's not suprising
that spawning even 100 processes took some
Bottom line: multiprocessing should not be used this way.
(nor should threading).
More information about the Python-list