Parallelization in Python 2.6

[Jonathan Gardner]

On Aug 18, 11:19 am, Robert Dailey <rcdai... at gmail.com> wrote:
> I'm looking for a way to parallelize my python script without using
> typical threading primitives. For example, C++ has pthreads and TBB to
> break things into "tasks". I would like to see something like this for
> python. So, if I have a very linear script:
>
> doStuff1()
> doStuff2()
>
> I can parallelize it easily like so:
>
> create_task( doStuff1 )
> create_task( doStuff2 )
>
> Both of these functions would be called from new threads, and once
> execution ends the threads would die. I realize this is a simple
> example and I could create my own classes for this functionality, but
> I do not want to bother if a solution already exists.
>

If you haven't heard of the Python GIL, you'll want to find out sooner
rather than later. Short summary: Python doesn't do threading very
well.

There are quite a few parallelization solutions out there for Python,
however. (I don't know what they are off the top of my head, however.)
The way they work is they have worker processes that can be spread
across machines. When you want to parallelize a task, you send off a
function to those worker threads.

There are some serious caveats and problems, not the least of which is
sharing code between the worker threads and the director, so this
isn't a great solution.

If you're looking for highly parallelized code, Python may not be the
right answer. Try something like Erlang or Haskell.