A matter of queues, tasks and multiprocessing
nagle at animats.com
Fri Nov 12 21:25:21 CET 2010
On 11/12/2010 12:18 PM, John Nagle wrote:
> On 11/10/2010 9:25 AM, Emanuele D'Arrigo wrote:
>> Greetings everybody,
>> I've tried to come up with this message for a couple of weeks now and
>> it doesn't look like I'm getting any clearer in my thoughts so I
>> decided that it's probably best to take the plunge and ask you guys to
>> kindly throw me a rope...
>> What I'm trying to come up with is some kind of simple, dynamically
>> scalable, domain-agnostic, multi-processor capable, tasking system.
>> Maybe I'm already giving myself too many constraints here, i.e. it
>> might all be possible except... the "simple" part.
> No, the problem is vagueness. We can't tell from your description
> what you want to do. A real-time desktop application? Warehouse-scale
> computing? Your own "cloud"?
> Incidentally, bear in mind that once you scale beyond tens of
> machines, error detection and recovery starts to dominate
> the problem.
> John Nagle
Here's how Google does it:
More information about the Python-list