multiprocessing vs thread performance
castironpi at gmail.com
Tue Dec 30 02:34:46 CET 2008
On Dec 29, 6:05 pm, "James Mills" <prolo... at shortcircuit.net.au>
> On Tue, Dec 30, 2008 at 12:52 AM, mk <mrk... at gmail.com> wrote:
> > Hello everyone,
> > After readinghttp://www.python.org/dev/peps/pep-0371/I was under
> > impression that performance of multiprocessing package is similar to that of
> > thread / threading. However, to familiarize myself with both packages I
> > wrote my own test of spawning and returning 100,000 empty threads or
> > processes (while maintaining at most 100 processes / threads active at any
> > one time), respectively.
> As others have mentioned, it's not suprising
> that spawning even 100 processes took some
> Bottom line: multiprocessing should not be used this way.
> (nor should threading).
The OP may be interested in Erlang, which Wikipedia (end-all, be-all)
claims is a 'distribution oriented language'.
You might also find it interesting to examine a theoretical OS that is
optimized for process overhead. In other words, what is the minimum
overhead possible? Can processes be as small as threads? Can entire
threads be only a few bytes (words) big?
Also, could generators provide any of the things you need with your
multiple threads? You could, say, call 'next()' on many items in a
list, and just remove them on StopIteration.
More information about the Python-list