[Numpy-discussion] ragged array implimentation

Christopher Barker Chris.Barker at noaa.gov
Thu Mar 10 16:25:27 EST 2011


On 3/10/11 12:01 PM, Francesc Alted wrote:
>> 3) when the extra space is used up, it re-allocates the entire array,
>> with some more extra room
>
> again, carray works exactly the same: the extra room is just a new chunk

does it re-allocate the entire thing?

>> And to keep it really simple, I'm using a numpy array internally, and
>> np.ndarray.resize to grow it. It can only be grown on the first axis
>> (C-order).
>
> uh, very similar, except that due to the chunked nature of carray, it
> does not use `ndarray.resize`, but simply allocates a new chunk.

right, is each chunk is a ndarray? Or just a c pointer?

Anyway, the advantage to your approach (IIUC) is that you are not 
continually re-allocating and copying data as the array grows. My method 
reduces how often that happens, but it still does happen, and I imagine 
(haven't tested) that that has performance issues if the array gets 
really large, as it needs a contiguous chunk of memory, and can't free 
the old one until the data has been copied over.

> Well, that sounds like it could be a start.  Your experience with your
> package would be a nice thing to have.  BTW, can you remember me where
> it is your code?  Maybe I can get some ideas by looking at it.

I never posted it anywhere public. It's not much code (really!), so I've 
enclosed it here.

> As you noticed, chunks and carrays are both native Cython extensions, so
> it should be pretty easy to call them from Cython directly, avoiding the
> Python overhead.

cool. I hope I'll get a chance to poke into this soon.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Accumulator.zip
Type: application/zip
Size: 4703 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110310/93c80ed4/attachment.zip>


More information about the NumPy-Discussion mailing list