Parallel/Multiprocessing script design question

[A.T.Hofkamp]

On 2007-09-13, Amit N <nospam at 23342.om> wrote:
> Hi guys,
>
> I tend to ramble, and I am afraid none of you busy experts will bother 
> reading my long post, so I will try to summarize it first:

I haven't read the details, but you seem to aim for a single python program
that does 'it'. A single sequential thread is not fast enough, so you want
parallel execution (and yes there are a zillion ways of doing that).

Why don't you start at the other end?

Write a program for each task that you have, then fork/spawn/chain/whatever
enough processes at OS level to eat all your data. The OS is usually much
better at balancing CPU's. The Python module 'subprocess' would be your friend
in that case.

In addition, you can run each program independently, which will come in handy
one day.


Sincerely,
Albert