how-to "put" RAM-based numarray into memmap
![](https://secure.gravatar.com/avatar/6194b135cba546afa82516de1537de49.jpg?s=120&d=mm&r=g)
Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header up front - which I know how to do as a memmap record) Can I somehow "append" a (standard) numarray to a memmap "as a memmap-slice" ? I obviously could create the corresponding memslice fresh and then assign the original data elementwise - but that would allocate extra (virtual) memory... Thanks, Sebastian Haase
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
Sebastian Haase wrote:
How about: f = open(filename, 'wb') f.write(MyHeader) A.tofile(f) To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it. NOTE: I havn't actaully done this myself, but I've done something bvery similar with fromstring and tostring with Numeric. I'm very happy that Numarray has the *file methods, as they don't require an extra copy in memory. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/6c9110e0401b013d2324fbd6257dc80d.jpg?s=120&d=mm&r=g)
A Divendres 21 Novembre 2003 18:28, Chris Barker va escriure:
You can also use pytables for doing that more comfortably:
To read it back you only have to do:
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]]) that's all. All the headers with info about the object you are saving are automatically saved and retrieved behind the scenes. You can get pytables from http://pytables.sf.net Cheers, -- Francesc Alted
![](https://secure.gravatar.com/avatar/6194b135cba546afa82516de1537de49.jpg?s=120&d=mm&r=g)
The main reason for my question was just to find out if NUMARRAY supports it, and how ? Also I have many "bookkeeping" functions already implemented for the memmap'd case. (That is, I have a class with member methods operting on a member memmapped array) So if what I descibed is possible I could save myself form duplicating lot's of code. Essentially I was hoping for the most ellegant solution ;-) Thanks, Sebastian
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Mon, 2003-11-24 at 14:13, Sebastian Haase wrote:
memmap's Memmap class does support an insert() method for adding a new slice to the end of (or anywhere in) an existing map. The new slice, however, will exist as a block of memory allocated on the heap until the memmap is saved to disk. Thus, two scenarios present themselves: (1) you allocate the new slice ahead of time and create an array from it, avoiding data duplication (2) you create an array and later copy it into (a newly inserted slice of) the memmap, thereby duplicating your data on the heap. When you close the map, slices on the heap are written to the map file. Todd
![](https://secure.gravatar.com/avatar/6194b135cba546afa82516de1537de49.jpg?s=120&d=mm&r=g)
Thanks for your reply. Is it possible to define a memmap slice and giving it a (preinitialized !) memory buffer ? I'm thinking: I have a RAM-based numarray, I just take the buffer (pointer) and hand it over to the memmap-slice so that it can make the association between the disk-space and the RAM-space. I guess you are calling "heap" what I call RAM. Is memmap using something inherently different that heap? (I might be missing something...) As I was trying to illustrate in my example, my ram-numarray might already be using most of the available address space. Thanks, Sebastian
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
It should be now. I just added support for a "buffer" parameter to CVS. I tested it manually and it worked OK for me; please try it out or watch for it in numarray-0.8.
Yes and no. When you allocate a slice of an existing memmap, you're getting memory backed on the disk file of your choice; mapped memory generally comes from a different address partition than the memory from malloc(). When you allocate memory using numarray, or insert a slice into a memmap without specifying the buffer, you're getting memory from the heap which is backed on the system swap file. (Of course, this varies a little by system as well; I'm thinking Linux/UNIX here and Win32 may be slightly different). Todd -- Todd Miller <jmiller@stsci.edu>
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
Sebastian Haase wrote:
How about: f = open(filename, 'wb') f.write(MyHeader) A.tofile(f) To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it. NOTE: I havn't actaully done this myself, but I've done something bvery similar with fromstring and tostring with Numeric. I'm very happy that Numarray has the *file methods, as they don't require an extra copy in memory. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/6c9110e0401b013d2324fbd6257dc80d.jpg?s=120&d=mm&r=g)
A Divendres 21 Novembre 2003 18:28, Chris Barker va escriure:
You can also use pytables for doing that more comfortably:
To read it back you only have to do:
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]]) that's all. All the headers with info about the object you are saving are automatically saved and retrieved behind the scenes. You can get pytables from http://pytables.sf.net Cheers, -- Francesc Alted
![](https://secure.gravatar.com/avatar/6194b135cba546afa82516de1537de49.jpg?s=120&d=mm&r=g)
The main reason for my question was just to find out if NUMARRAY supports it, and how ? Also I have many "bookkeeping" functions already implemented for the memmap'd case. (That is, I have a class with member methods operting on a member memmapped array) So if what I descibed is possible I could save myself form duplicating lot's of code. Essentially I was hoping for the most ellegant solution ;-) Thanks, Sebastian
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Mon, 2003-11-24 at 14:13, Sebastian Haase wrote:
memmap's Memmap class does support an insert() method for adding a new slice to the end of (or anywhere in) an existing map. The new slice, however, will exist as a block of memory allocated on the heap until the memmap is saved to disk. Thus, two scenarios present themselves: (1) you allocate the new slice ahead of time and create an array from it, avoiding data duplication (2) you create an array and later copy it into (a newly inserted slice of) the memmap, thereby duplicating your data on the heap. When you close the map, slices on the heap are written to the map file. Todd
![](https://secure.gravatar.com/avatar/6194b135cba546afa82516de1537de49.jpg?s=120&d=mm&r=g)
Thanks for your reply. Is it possible to define a memmap slice and giving it a (preinitialized !) memory buffer ? I'm thinking: I have a RAM-based numarray, I just take the buffer (pointer) and hand it over to the memmap-slice so that it can make the association between the disk-space and the RAM-space. I guess you are calling "heap" what I call RAM. Is memmap using something inherently different that heap? (I might be missing something...) As I was trying to illustrate in my example, my ram-numarray might already be using most of the available address space. Thanks, Sebastian
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
It should be now. I just added support for a "buffer" parameter to CVS. I tested it manually and it worked OK for me; please try it out or watch for it in numarray-0.8.
Yes and no. When you allocate a slice of an existing memmap, you're getting memory backed on the disk file of your choice; mapped memory generally comes from a different address partition than the memory from malloc(). When you allocate memory using numarray, or insert a slice into a memmap without specifying the buffer, you're getting memory from the heap which is backed on the system swap file. (Of course, this varies a little by system as well; I'm thinking Linux/UNIX here and Win32 may be slightly different). Todd -- Todd Miller <jmiller@stsci.edu>
participants (4)
-
Chris Barker
-
Francesc Alted
-
Sebastian Haase
-
Todd Miller