memory-mapped Numeric arrays: arrayfrombuffer version 2
I would be very happy if this got included in the Numpy distribution, so that people don't have to do an extra download to get this facility. I just submitted it as a patch on SourceForge, but my browser failed to upload the file. http://sourceforge.net/tracker/index.php?func=detail&aid=507847&group_id=1369&atid=301369 There is one thing in arrayfrombuffer.c I'm not sure about, and I could use some help here. arrayobj is the return value from PyArray_FromDimsAndData: /* do I need to incref arrayobj?! fromstring does... */ return (PyObject*)arrayobj; } So, do I? Or not? It seems to work as it is. Following is the announcement I posted to comp.lang.python and (hopefully) comp.lang.python.announce. The 'arrayfrombuffer' package features support for Numerical Python arrays whose contents are stored in buffer objects, including memory-mapped files. This has the following advantages: - loading your array from a file is easy --- a module import and a single function call --- and doesn't use excessive amounts of memory. - loading your array is quick; it doesn't need to be copied from one part of memory to another in order to be loaded. - your array gets demand-loaded; parts you aren't using don't need to be in memory or in swap. - under memory-pressure conditions, your array doesn't use up swap, and parts of it you haven't modified can be evicted from RAM without the need for a disk write - your arrays can be bigger than your physical memory - when you modify your array, only the parts you modify get written back out to disk This is something that's been requested on the Numpy list a few times a year since 1999. arrayfrombuffer lives at http://pobox.com/~kragen/sw/arrayfrombuffer/ The current version is version 2; it is released under the X11 license (the BSD license without the advertising clause).
Kragen Sitaker <kragen@pobox.com> writes:
And it is correct, in my opinion. Your routine is the owner of the array (and no one else keeps a reference), so you just pass on ownership. Doing an INCREF here would mean that the array is never freed. (Keep in mind that the data area is never freed anyway!) Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
Kragen Sitaker <kragen@pobox.com> writes:
And it is correct, in my opinion. Your routine is the owner of the array (and no one else keeps a reference), so you just pass on ownership. Doing an INCREF here would mean that the array is never freed. (Keep in mind that the data area is never freed anyway!) Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
participants (2)
-
Konrad Hinsen -
Kragen Sitaker