memory-mapped Numeric arrays: arrayfrombuffer version 2
Kragen Sitaker
kragen@pobox.com
Thu, 24 Jan 2002 00:39:35 -0500
The 'arrayfrombuffer' package features support for Numerical Python
arrays whose contents are stored in buffer objects, including
memory-mapped files. This has the following advantages:
- loading your array from a file is easy --- a module import and a
single function call --- and doesn't use excessive amounts of
memory.
- loading your array is quick; it doesn't need to be copied from one
part of memory to another in order to be loaded.
- your array gets demand-loaded; parts you aren't using don't need to
be in memory or in swap.
- under memory-pressure conditions, your array doesn't use up swap,
and parts of it you haven't modified can be evicted from RAM without
the need for a disk write
- your arrays can be bigger than your physical memory
- when you modify your array, only the parts you modify get written
back out to disk
This is something that's been requested on the Numpy list a few times
a year since 1999.
arrayfrombuffer lives at
http://pobox.com/~kragen/sw/arrayfrombuffer/
The current version is version 2; it is released under the X11 license
(the BSD license without the advertising clause).
<kragen@pobox.com>
<P><A
HREF="http://pobox.com/~kragen/sw/arrayfrombuffer/">arrayfrombuffer
2</A> - creates Numeric arrays from memory-mapped files. (23-Jan-02)