[Baypiggies] Improving Python+MPI import performance

[Fernando Perez]

On Thu, Jan 12, 2012 at 12:23 PM, Asher Langton <langton2 at llnl.gov> wrote:
> I work on a Python/C++ scientific code that runs as a number of
> independent Python processes communicating via MPI. Unfortunately, as
> some of you may have experienced, module importing does not scale well
> in Python/MPI applications. For 32k processes on BlueGene/P, importing
> 100 trivial C-extension modules takes 5.5 hours, compared to 35
> minutes for all other interpreter loading and initialization. We
> developed a simple pure-Python module (based on knee.py, a
> hierarchical import example) that cuts the import time from 5.5 hours
> to 6 minutes.
>
> The code is available here:
>
> https://github.com/langton/MPI_Import

Excellent!  I suggest you post this to the numpy list, I bet you it
will be of interest to a lot of people...

Cheers,

f