how-to "put" RAM-based numarray into memmap
Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header up front - which I know how to do as a memmap record) Can I somehow "append" a (standard) numarray to a memmap "as a memmap-slice" ?
I obviously could create the corresponding memslice fresh and then assign the original data elementwise - but that would allocate extra (virtual) memory...
Thanks, Sebastian Haase
Sebastian Haase wrote:
Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header up front
How about:
f = open(filename, 'wb') f.write(MyHeader) A.tofile(f)
To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it.
NOTE: I havn't actaully done this myself, but I've done something bvery similar with fromstring and tostring with Numeric. I'm very happy that Numarray has the *file methods, as they don't require an extra copy in memory.
-Chris
A Divendres 21 Novembre 2003 18:28, Chris Barker va escriure:
Sebastian Haase wrote:
Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header up front
How about:
f = open(filename, 'wb') f.write(MyHeader) A.tofile(f)
To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it.
You can also use pytables for doing that more comfortably:
a=zeros([2,3,3]) import tables fileh=tables.openFile("test.h5", "w") fileh.createArray(fileh.root, "array1", a, "This is a small test array")
/array1 (Array(2, 3, 3)) 'This is a small test array' type = Int32 itemsize = 4 flavor = 'NumArray' byteorder = 'little'
fileh.close()
To read it back you only have to do:
fileh=tables.openFile("test.h5", "r") a=fileh.root.array1.read() a
array([[[0, 0, 0], [0, 0, 0], [0, 0, 0]],
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]])
that's all. All the headers with info about the object you are saving are automatically saved and retrieved behind the scenes.
You can get pytables from http://pytables.sf.net
Cheers,
Sebastian Haase wrote:
Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header up front
How about:
f = open(filename, 'wb') f.write(MyHeader) A.tofile(f)
To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it.
The main reason for my question was just to find out if NUMARRAY supports it, and how ? Also I have many "bookkeeping" functions already implemented for the memmap'd case. (That is, I have a class with member methods operting on a member memmapped array) So if what I descibed is possible I could save myself form duplicating lot's of code.
Essentially I was hoping for the most ellegant solution ;-)
Thanks, Sebastian
On Mon, 2003-11-24 at 14:13, Sebastian Haase wrote:
Sebastian Haase wrote:
Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header up front
How about:
f = open(filename, 'wb') f.write(MyHeader) A.tofile(f)
To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it.
The main reason for my question was just to find out if NUMARRAY supports it, and how ? Also I have many "bookkeeping" functions already implemented for the memmap'd case. (That is, I have a class with member methods operting on a member memmapped array) So if what I descibed is possible I could save myself form duplicating lot's of code.
Essentially I was hoping for the most ellegant solution ;-)
memmap's Memmap class does support an insert() method for adding a new slice to the end of (or anywhere in) an existing map. The new slice, however, will exist as a block of memory allocated on the heap until the memmap is saved to disk.
Thus, two scenarios present themselves: (1) you allocate the new slice ahead of time and create an array from it, avoiding data duplication (2) you create an array and later copy it into (a newly inserted slice of) the memmap, thereby duplicating your data on the heap.
When you close the map, slices on the heap are written to the map file.
Todd
Thanks, Sebastian
This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
On Mon, 2003-11-24 at 14:13, Sebastian Haase wrote:
Sebastian Haase wrote:
Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header
up
front
How about:
f = open(filename, 'wb') f.write(MyHeader) A.tofile(f)
To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it.
The main reason for my question was just to find out if NUMARRAY
supports
it, and how ? Also I have many "bookkeeping" functions already implemented for the memmap'd case. (That is, I have a class with member methods operting on a member
memmapped
array) So if what I descibed is possible I could save myself form duplicating
lot's
of code.
Essentially I was hoping for the most ellegant solution ;-)
memmap's Memmap class does support an insert() method for adding a new slice to the end of (or anywhere in) an existing map. The new slice, however, will exist as a block of memory allocated on the heap until the memmap is saved to disk.
Thus, two scenarios present themselves: (1) you allocate the new slice ahead of time and create an array from it, avoiding data duplication (2) you create an array and later copy it into (a newly inserted slice of) the memmap, thereby duplicating your data on the heap.
When you close the map, slices on the heap are written to the map file.
Todd
Thanks for your reply. Is it possible to define a memmap slice and giving it a (preinitialized !) memory buffer ? I'm thinking: I have a RAM-based numarray, I just take the buffer (pointer) and hand it over to the memmap-slice so that it can make the association between the disk-space and the RAM-space. I guess you are calling "heap" what I call RAM. Is memmap using something inherently different that heap? (I might be missing something...) As I was trying to illustrate in my example, my ram-numarray might already be using most of the available address space.
Thanks, Sebastian
Thanks for your reply. Is it possible to define a memmap slice and giving it a (preinitialized !) memory buffer ?
It should be now. I just added support for a "buffer" parameter to CVS. I tested it manually and it worked OK for me; please try it out or watch for it in numarray-0.8.
I'm thinking: I have a RAM-based numarray, I just take the buffer (pointer) and hand it over to the memmap-slice so that it can make the association between the disk-space and the RAM-space. I guess you are calling "heap" what I call RAM. Is memmap using something inherently different that heap? (I might be missing something...)
Yes and no. When you allocate a slice of an existing memmap, you're getting memory backed on the disk file of your choice; mapped memory generally comes from a different address partition than the memory from malloc(). When you allocate memory using numarray, or insert a slice into a memmap without specifying the buffer, you're getting memory from the heap which is backed on the system swap file. (Of course, this varies a little by system as well; I'm thinking Linux/UNIX here and Win32 may be slightly different).
Todd
participants (4)
-
Chris Barker -
Francesc Alted -
Sebastian Haase -
Todd Miller