[Baypiggies] Improving Python+MPI import performance
Minesh B. Amin
mamin at mbasciences.com
Thu Jan 12 23:28:29 CET 2012
Hi Asher,
Simply amazing :)
Reducing the number of failed open() calls is indeed the key.
Cheers!
Minesh
On Thu, 2012-01-12 at 12:23 -0800, Asher Langton wrote:
> Hi all,
>
> I work on a Python/C++ scientific code that runs as a number of
> independent Python processes communicating via MPI. Unfortunately, as
> some of you may have experienced, module importing does not scale well
> in Python/MPI applications. For 32k processes on BlueGene/P, importing
> 100 trivial C-extension modules takes 5.5 hours, compared to 35
> minutes for all other interpreter loading and initialization. We
> developed a simple pure-Python module (based on knee.py, a
> hierarchical import example) that cuts the import time from 5.5 hours
> to 6 minutes.
>
> The code is available here:
>
> https://github.com/langton/MPI_Import
>
>
More information about the Baypiggies
mailing list