[Baypiggies] Improving Python+MPI import performance

Minesh B. Amin mamin at mbasciences.com
Thu Jan 12 23:28:29 CET 2012

Hi Asher,

Simply amazing :)

Reducing the number of failed open() calls is indeed the key.


On Thu, 2012-01-12 at 12:23 -0800, Asher Langton wrote:
> Hi all,
> I work on a Python/C++ scientific code that runs as a number of
> independent Python processes communicating via MPI. Unfortunately, as
> some of you may have experienced, module importing does not scale well
> in Python/MPI applications. For 32k processes on BlueGene/P, importing
> 100 trivial C-extension modules takes 5.5 hours, compared to 35
> minutes for all other interpreter loading and initialization. We
> developed a simple pure-Python module (based on knee.py, a
> hierarchical import example) that cuts the import time from 5.5 hours
> to 6 minutes.
> The code is available here:
> https://github.com/langton/MPI_Import

More information about the Baypiggies mailing list