[Numpy-discussion] Improving Python+MPI import performance

[Langton, Asher]

On 1/13/12 1:24 PM, Robert Kern wrote:
>On Fri, Jan 13, 2012 at 21:20, Langton, Asher <langton2 at llnl.gov> wrote:
>
>> 2) More generally, dealing with this as well as other library-loading
>> issues at the system level, perhaps by putting a small disk near a node
>>or
>> small collection of nodes, along with a command to push (broadcast) some
>> portions of the filesystem to these (more-)local disks. Basically, the
>> idea would be to let the user specify those directories or objects that
>> will be accessed by most of the processes and treated as read-only so
>>that
>> those objects can be cached near the node.
>
>Do these systems have a ramdisk capability?

That was another thing we looked at (but didn't implement): broadcasting
the modules to each node and putting them in a ramdisk. The drawback (for
us) is that we're already struggling with the amount of available memory
per core, and according to the vendors, the situation will only get worse
on future systems. The ramdisk approach might work well when there are
lots of small objects that will be accessed.

On 1/13/12 1:42 PM, Sturla Molden wrote:
>Den 13.01.2012 22:24, skrev Robert Kern:
>>Do these systems have a ramdisk capability?
>
>I assume you have seen this as well :)
>
>http://www.cs.uoregon.edu/Research/paracomp/papers/iccs11/iccs_paper_final
>.pdf


I hadn't. Thanks!

-Asher