python distributed computing

julien godin t4rtine at
Wed Aug 3 11:37:21 EDT 2011

Hello the list,

I have a, let say 'special' , request.
I would like to make a very specific type of distributed computing.
Let me explain myself :

I have a HUUUUUGE(everything is relative) amount of data coming from UDP/514
( good old syslog), and one process alone could not handle them ( it's
full of regex, comparison and counting. all this at about 100K lines of data
per seconds and each line is about 1500o long. I tried to handle it on one
process, 100% core time in htop. )

So i said to myself : why not distribute the computing ?
I looked for a good library that could do the distribution/process
management job for me (

What i found was not really usable by me ( distributed processing with a lot
of processing but almost no process communication wich is the opposite of
the thing i would like to do: distribute the events in real time so it is
processed in real time.).

My question is : is their any library or API or anything that is able to do
that ?


PS: This is my first time posting on any mailing list, my explanation/syntax
could look a little bite, messy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the Python-list mailing list