Tips for load balancing multiple Python apps on dual/quad core processors?
steve at holdenweb.com
Mon Apr 7 20:39:07 CEST 2008
Malcolm Greene wrote:
> I'm looking for tips on how to load balance running multiple Python
> applications in multi-CPU environments. My understanding is that Python
> applications and their threads are limited to a specific CPU.
> Background: I have a Python utility that processes email messages. I
> suspect there's a lot of idle time while this utility waits on a remote
> email server. I would like to run as many simultaneous copies of this
> utility as possible without overwhelming the server these utilities are
> running on. My thought is that I should write a dispatcher that monitors
> CPU load and launches/cancels multiple instances of my utility with
> specific job queues to process.
> Is there a cross-platform way to monitor CPU load?
> Is there a pre-existing Python module I should be looking at for
> building (subclassing) my dispatcher?
As a data point for you, I wrote a multi-threaded application that used
100 threads to send about 45,000 emails in less than four hours. The
single-CPU computer that processed this job wasn't using that much CPU
(certainly under 50%).
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
More information about the Python-list