Tips for load balancing multiple Python apps on dual/quad core processors?
python at bdurham.com
Mon Apr 7 20:15:29 CEST 2008
I'm looking for tips on how to load balance running multiple Python
applications in multi-CPU environments. My understanding is that Python
applications and their threads are limited to a specific CPU.
Background: I have a Python utility that processes email messages. I
suspect there's a lot of idle time while this utility waits on a remote
email server. I would like to run as many simultaneous copies of this
utility as possible without overwhelming the server these utilities are
running on. My thought is that I should write a dispatcher that monitors
CPU load and launches/cancels multiple instances of my utility with
specific job queues to process.
Is there a cross-platform way to monitor CPU load?
Is there a pre-existing Python module I should be looking at for
building (subclassing) my dispatcher?
More information about the Python-list