[Numpy-discussion] ragged array implimentation
Christopher Barker
Chris.Barker at noaa.gov
Thu Mar 10 16:25:27 EST 2011
On 3/10/11 12:01 PM, Francesc Alted wrote:
>> 3) when the extra space is used up, it re-allocates the entire array,
>> with some more extra room
>
> again, carray works exactly the same: the extra room is just a new chunk
does it re-allocate the entire thing?
>> And to keep it really simple, I'm using a numpy array internally, and
>> np.ndarray.resize to grow it. It can only be grown on the first axis
>> (C-order).
>
> uh, very similar, except that due to the chunked nature of carray, it
> does not use `ndarray.resize`, but simply allocates a new chunk.
right, is each chunk is a ndarray? Or just a c pointer?
Anyway, the advantage to your approach (IIUC) is that you are not
continually re-allocating and copying data as the array grows. My method
reduces how often that happens, but it still does happen, and I imagine
(haven't tested) that that has performance issues if the array gets
really large, as it needs a contiguous chunk of memory, and can't free
the old one until the data has been copied over.
> Well, that sounds like it could be a start. Your experience with your
> package would be a nice thing to have. BTW, can you remember me where
> it is your code? Maybe I can get some ideas by looking at it.
I never posted it anywhere public. It's not much code (really!), so I've
enclosed it here.
> As you noticed, chunks and carrays are both native Cython extensions, so
> it should be pretty easy to call them from Cython directly, avoiding the
> Python overhead.
cool. I hope I'll get a chance to poke into this soon.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker at noaa.gov
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Accumulator.zip
Type: application/zip
Size: 4703 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110310/93c80ed4/attachment.zip>
More information about the NumPy-Discussion
mailing list