python distributed computing
t4rtine at gmail.com
Wed Aug 3 17:37:21 CEST 2011
Hello the list,
I have a, let say 'special' , request.
I would like to make a very specific type of distributed computing.
Let me explain myself :
I have a HUUUUUGE(everything is relative) amount of data coming from UDP/514
( good old syslog), and one process alone could not handle them ( it's
full of regex, comparison and counting. all this at about 100K lines of data
per seconds and each line is about 1500o long. I tried to handle it on one
process, 100% core time in htop. )
So i said to myself : why not distribute the computing ?
I looked for a good library that could do the distribution/process
management job for me (http://wiki.python.org/moin/ParallelProcessing).
What i found was not really usable by me ( distributed processing with a lot
of processing but almost no process communication wich is the opposite of
the thing i would like to do: distribute the events in real time so it is
processed in real time.).
My question is : is their any library or API or anything that is able to do
PS: This is my first time posting on any mailing list, my explanation/syntax
could look a little bite, messy.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-list