[Numpy-discussion] memory-mapped Numeric arrays: arrayfrombuffer version 2

[Kragen Sitaker]

I would be very happy if this got included in the
Numpy distribution, so that people don't have to do an extra
download to get this facility.  I just submitted it as a patch
on SourceForge, but my browser failed to upload the file.
http://sourceforge.net/tracker/index.php?func=detail&aid=507847&group_id=1369&atid=301369

There is one thing in arrayfrombuffer.c I'm not sure about, and
I could use some help here.  arrayobj is the return value from
PyArray_FromDimsAndData:
  /* do I need to incref arrayobj?! fromstring does... */
  return (PyObject*)arrayobj;
}

So, do I?  Or not?  It seems to work as it is.

Following is the announcement I posted to comp.lang.python and (hopefully)
comp.lang.python.announce.

The 'arrayfrombuffer' package features support for Numerical Python
arrays whose contents are stored in buffer objects, including
memory-mapped files.  This has the following advantages:

- loading your array from a file is easy --- a module import and a
  single function call --- and doesn't use excessive amounts of
  memory.
- loading your array is quick; it doesn't need to be copied from one
  part of memory to another in order to be loaded.
- your array gets demand-loaded; parts you aren't using don't need to
  be in memory or in swap.
- under memory-pressure conditions, your array doesn't use up swap,
  and parts of it you haven't modified can be evicted from RAM without
  the need for a disk write
- your arrays can be bigger than your physical memory
- when you modify your array, only the parts you modify get written
  back out to disk

This is something that's been requested on the Numpy list a few times
a year since 1999.

arrayfrombuffer lives at
http://pobox.com/~kragen/sw/arrayfrombuffer/
The current version is version 2; it is released under the X11 license
(the BSD license without the advertising clause).