[Numpy-discussion] Dynamic array list implementation

Sebastian Berg sebastian at sipsolutions.net
Wed Dec 23 07:31:27 EST 2015


On Mi, 2015-12-23 at 00:34 -0800, Stephan Hoyer wrote:
> We have a type similar to this (a typed list) internally in pandas,
> although it is restricted to a single dimension and far from feature
> complete -- it only has .append and a .to_array() method for
> converting to a 1d numpy array. Our version is written in Cython, and
> we use it for performance reasons when we would otherwise need to
> create a list of unknown length:
> https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99
> 

Probably is a bit orthogonal since I guess you want/need cython, but
pythons buildin array.array should get you there pretty much as well. Of
course it requires the C typecode (though that should not be hard to
get) and does not support strings.

- Sebastian

> 
> In my experience, it's several times faster than using a builtin list
> from Cython, which makes sense given that it needs to copy about 1/3
> the data (no type or reference count for individual elements).
> Obviously, it uses 1/3 the space to store the data, too. We currently
> don't expose this object externally, but it could be an interesting
> project to adapt this code into a standalone project that could be
> more broadly useful.
> 
> 
> Cheers,
> Stephan
> 
> 
> 
> 
> On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
> 
> 
>         sorry for being so lazy as to not go look at the project
>         pages, but....
>         
>         
>         This sounds like it could be really useful, and maybe
>         supercise a coupl eof half-baked projects of mine. But -- what
>         does "dynamic" mean?
>         
>         
>         - can you append to these arrays?
>         - can it support "ragged arrrays" -- it looks like it does.
>                 
>                 
>                 >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
>                 >>> print(L)
>                 [[0], [1 2], [3 4 5], [6 7 8 9]]
>         so this looks like a ragged array -- but what do you get when
>         you do:
>         
>         
>         for row in L:
>             print row
>         
>         
>          
>         
>                 >>> print(L.data)
>                 [0 1 2 3 4 5 6 7 8 
>         is .data a regular old 1-d numpy array?
>         
>         
>                 >>> L = ArrayList( np.arange(10), [3,3,4])
>                 >>> print(L)
>                 [[0 1 2], [3 4 5], [6 7 8 9]]
>                 >>> print(L.data)
>                 [0 1 2 3 4 5 6 7 8 9]
>                 
>         does an ArrayList act like a numpy array in other ways:
>         
>         
>         L * 5
>         
>         
>         L* some_array
>         
>         
>         in which case, how does it do broadcasting???
>         
>         
>         Thanks,
>         
>         
>         -CHB
>         
>         
>                 >>> L = ArrayList(["Hello", "world", "!"])
>                 >>> print(L[0])
>                 'Hello'
>                 >>> L[1] = "brave new world"
>                 >>> print(L)
>                 ['Hello', 'brave new world', '!']
>                 
>                 
>                 
>                 
>                 Nicolas
>                 
>                 _______________________________________________
>                 NumPy-Discussion mailing list
>                 NumPy-Discussion at scipy.org
>                 https://mail.scipy.org/mailman/listinfo/numpy-discussion
>                 
>         
>         
>         
>         
>         -- 
>         
>         Christopher Barker, Ph.D.
>         Oceanographer
>         
>         Emergency Response Division
>         NOAA/NOS/OR&R            (206) 526-6959   voice
>         7600 Sand Point Way NE   (206) 526-6329   fax
>         Seattle, WA  98115       (206) 526-6317   main reception
>         
>         Chris.Barker at noaa.gov
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20151223/37356110/attachment.sig>


More information about the NumPy-Discussion mailing list