[Numpy-discussion] Improving Python+MPI import performance
langton2 at llnl.gov
Fri Jan 13 17:09:56 EST 2012
On 1/13/12 1:58 PM, Dag Sverre Seljebotn wrote:
>It's actually not too difficult to do something like
>LD_PRELOAD=myhack.so python something.py
>and have myhack.so intercept the filesystem calls Python makes (to libc)
>and do whatever it wants. That's a solution that doesn't interfer with
>how Python does its imports at all, it simply changes how Python
>perceives the world around it ("emulation", though much, much lighter).
>It does require some low-level C code, but there are several examples on
>the net. I know Ondrej Certik just implemented something similar.
One of my colleagues suggested the LD_PRELOAD trick. I asked around here
at LLNL, and I seem to recall hearing that the LD_PRELOAD trick didn't
work on BlueGene/P, which is where the import bottleneck is the worst.
That might have been incorrect though, since LD_PRELOAD is mentioned on
Argonne's BG/P wiki. I'll have to look into this some more.
More information about the NumPy-Discussion