[Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Chris Barker]

Robert Kern wrote:

> > Thanks, that works, but now I am wondering: what I want is a fast and
> > memory efficient way to get the contents of a file into a NujmPy array,
> > this sure doesn't look any better than:
> >
> > a = fromstring(file.read()))
> 
> Depends on how large the file is. file.read() creates a temporary string the
> size of the file. That string isn't freed until fromstring() finishes and
> returns the array object. For a brief time, both the string and the array have
> duplicates of the same data taking up space in memory.

Exactly. Also is the memory used for hte string guarranteed to be freed
right away? I have no idea how Python internals work.

Anyway, that's why I want a "fromfile()" method, like the one inthe
library array module.
 
> I don't know the details of mmap, so it's certainly possible that the only way
> that fromstring knows how to access the data is to pull all of it into memory
> first, thus recreating the problem. Alas.

I don't understand mmap at all. From the name, it sounds like the entire
contents of the file is mapped into memory, so the memory would get used
as soon as you set it up. If anyone knows, I'd like to hear...


-Chris





-- 
Christopher Barker,
Ph.D.                                                           
ChrisHBarker at home.net                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------