[Numpy-discussion] ragged array implimentation

[Francesc Alted]

A Thursday 10 March 2011 22:25:27 Christopher Barker escrigué:
> On 3/10/11 12:01 PM, Francesc Alted wrote:
> >> 3) when the extra space is used up, it re-allocates the entire
> >> array, with some more extra room
> > 
> > again, carray works exactly the same: the extra room is just a new
> > chunk
> 
> does it re-allocate the entire thing?

Ups.  Of course not, just a new chunk.

> right, is each chunk is a ndarray? Or just a c pointer?

Just a C pointer to a malloc'ed area (it cannot be an ndarray, as the 
data area might be compressed).

> Anyway, the advantage to your approach (IIUC) is that you are not
> continually re-allocating and copying data as the array grows. My
> method reduces how often that happens, but it still does happen, and
> I imagine (haven't tested) that that has performance issues if the
> array gets really large, as it needs a contiguous chunk of memory,
> and can't free the old one until the data has been copied over.

Exactly, carray can use memory by small chunks (typically < 1 MB), 
actually allowing to use fragmented areas across the main memory.  This 
allows for an optimal memory usage (even if compression is not used).

> I never posted it anywhere public. It's not much code (really!), so
> I've enclosed it here.

Ok.  Thanks.  The code is really simple, and that's great.  However, 
carray is quite more sophisticated, as it supports not only enlarging 
arrays, but also shrinking, and since 0.4 on, it also supports n-
dimensional arrays (although you can only resize along the first 
dimension).

-- 
Francesc Alted