[Numpy-discussion] Aggregate memmap

[Matthew Turk]

Hi Everyone,

Thanks for your suggestions and replies.  I initially tried what Anne
suggested, modifying the strides in the third dimension to account for
the 8-byte delimiters between slabs, but I couldn't control the
performance as much as I'd like, and I wasn't entirely sure when and
where "real" memory was being allocated or deallocated, or how it was
striding over the disk.  (Memory-mapping also seems to require
allocating the full size of the array as virtual memory.)  In the end
I ended up writing a very short Cython routine hand-tuned for the
problem that accepted the specific 3D slab to read off disk and return
that to the user, which also helped me coordinate better between
multiple MPI processes, as well.

Thanks very much for your suggestions!

-Matt

On Sun, Apr 25, 2010 at 7:57 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Apr 22, 2010 at 21:57, David Huard <david.huard at gmail.com> wrote:
>> Hi Matt,
>> I don't think the memmap code support this. However, you can stack memmaps
>> just as easily as arrays, so if you define individual memmaps for each slice
>> and stack them (numpy.vstack), the resulting array will behave as a regular
>> 3D array.
>
> However, that will make a main-memory copy of everything, presumably
> what he is trying to avoid.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>