multiprocessing question

MRAB python at mrabarnett.plus.com
Mon Jan 18 21:49:36 EST 2010


Neal Becker wrote:
> I'm using multiprocessing as a crude batch queuing system, like this:
> 
> import my_test_program as prog
> (where my_test_program has a function called 'run')
> 
> def run_test (args):
>     prog.run (args[1:])
> 
> cases = []
> for t in test_conditions:
>   args = [prog.__name__]+[more args...]
> 
>   cases.append (args)
> 
> (leaving out details, but 'cases' will be the list of test cases to run)
> 
> results = pool.map (run_test, cases)
> 
> Problem is, it doesn't seem to keep all my cpus busy, even though there are 
> more test cases than cpus.  Ideas?
> 
If they do a lot of I/O then that could be the bottleneck.



More information about the Python-list mailing list