I would like to do something like:
for i in xrange(len(self.particles)): nv = numpy.random.standard_normal((N,)) launch_on_any_available_processor( self.particles[i] = self.particles[i].random_fork(nv) ) wait_for_completions()
But I don't see a command like "launch_on_any_available_processor". I would be grateful for any advice.
Look more in depth at the multiprocessing library -- it's likely to be what you want... more or less. However, it might be a bit "less" than "more" because, from above, what it looks like you want to do is to launch many (millions?) of very lightweight tasks ("random_fork") on different processors. If you naively start up a fresh python process for each "random_fork" task, the startup costs will dominate (hugely so). So you'll likely need to re-jigger the task you want to dispatch to each processor so that the chunks are larger: processes = [] for i in range(num_processors): processes.append(start_worker_process(num_particles=total_particles// num_processors)) wait_for_processes_to_end() self.particles = numpy.concatenate([process.particles]) Though even this might be a too-granular a task, if you have numerous time-steps for which you need to generate particles. (E.g. if the above would be within another large loop.) Then you'd need to write the worker processes as sort of "particle servers": processes = [] for i in range(num_processors): processes.append(start_worker()) for task in huge_task_list: sub_tasks = divide_tasks(num_processors) for process, sub_task in zip(processes, sub_tasks): process.start(sub_task) wait_for_processes_to_complete_task() self.results = assemble_results(processes) This is of course pretty naive still, but it'll be a start, and the architectures I've (roughly) outlined here fit better with the multiprocessing paradigm. You'll need to do some internet reading and looking at various examples first, to get the details of how to actually implement this stuff. I don't know anything off the top of my head, but perhaps others can chime in? Zach