Tips for load balancing multiple Python apps on dual/quad core processors?

Malcolm Greene python at bdurham.com
Mon Apr 7 20:15:29 CEST 2008


I'm looking for tips on how to load balance running multiple Python
applications in multi-CPU environments. My understanding is that Python
applications and their threads are limited to a specific CPU.

Background: I have a Python utility that processes email messages. I
suspect there's a lot of idle time while this utility waits on a remote
email server. I would like to run as many simultaneous copies of this
utility as possible without overwhelming the server these utilities are
running on. My thought is that I should write a dispatcher that monitors
CPU load and launches/cancels multiple instances of my utility with
specific job queues to process. 

Is there a cross-platform way to monitor CPU load?

Is there a pre-existing Python module I should be looking at for
building (subclassing) my dispatcher?

Thanks!
Malcolm




More information about the Python-list mailing list