[Numpy-discussion] Dynamic array list implementation

[Stephan Hoyer]

We have a type similar to this (a typed list) internally in pandas, although it is restricted to a single dimension and far from feature complete -- it only has .append and a .to_array() method for converting to a 1d numpy array. Our version is written in Cython, and we use it for performance reasons when we would otherwise need to create a list of unknown length:



https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99



In my experience, it's several times faster than using a builtin list from Cython, which makes sense given that it needs to copy about 1/3 the data (no type or reference count for individual elements). Obviously, it uses 1/3 the space to store the data, too. We currently don't expose this object externally, but it could be an interesting project to adapt this code into a standalone project that could be more broadly useful.






Cheers,

Stephan

On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker <chris.barker at noaa.gov>
wrote:

> sorry for being so lazy as to not go look at the project pages, but....
> This sounds like it could be really useful, and maybe supercise a coupl eof
> half-baked projects of mine. But -- what does "dynamic" mean?
> - can you append to these arrays?
> - can it support "ragged arrrays" -- it looks like it does.
>>
>> >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
>> >>> print(L)
>> [[0], [1 2], [3 4 5], [6 7 8 9]]
>>
>> so this looks like a ragged array -- but what do you get when you do:
> for row in L:
>     print row
>> >>> print(L.data)
>> [0 1 2 3 4 5 6 7 8
>>
>> is .data a regular old 1-d numpy array?
>>>> L = ArrayList( np.arange(10), [3,3,4])
>> >>> print(L)
>> [[0 1 2], [3 4 5], [6 7 8 9]]
>> >>> print(L.data)
>> [0 1 2 3 4 5 6 7 8 9]
>>
>> does an ArrayList act like a numpy array in other ways:
> L * 5
> L* some_array
> in which case, how does it do broadcasting???
> Thanks,
> -CHB
>>>> L = ArrayList(["Hello", "world", "!"])
>> >>> print(L[0])
>> 'Hello'
>> >>> L[1] = "brave new world"
>> >>> print(L)
>> ['Hello', 'brave new world', '!']
>>
>>
>>
>> Nicolas
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> -- 
> Christopher Barker, Ph.D.
> Oceanographer
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20151223/471985c1/attachment.html>