<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Dec 24, 2015 at 10:19 AM, Chris Barker <span dir="ltr"><<a href="mailto:chris.barker@noaa.gov" target="_blank">chris.barker@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I'll try to get the code up on gitHub.</div></div></div></div></blockquote><div><br></div><div>Hey look -- it's already there:</div><div><br></div><div><a href="https://github.com/PythonCHB/NumpyExtras">https://github.com/PythonCHB/NumpyExtras</a><br></div><div><br></div><div>too many gitHub accounts.....</div><div><br></div><div>Here is the list/growable array/ accumulator:</div><div><br></div><div><a href="https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/accumulator.py">https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/accumulator.py</a><br></div><div><br></div><div>And here is the ragged array:</div><div><br></div><div><a href="https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/ragged_array.py">https://github.com/PythonCHB/NumpyExtras/blob/master/numpy_extras/ragged_array.py</a><br></div><div><br></div><div>I haven't touched either of these for a while -- not really sure what state they are in.</div><div><br></div><div>-CHB</div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>It would be nice to combine efforts.</div><div><br></div><div>-CHB</div><div><div class="h5"><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
In my case I need to ensure a contiguous storage to allow easy upload onto the GPU.<br>
But my implementation is quite slow, especially when you add one item at a time:<br>
<br>
>>> python benchmark.py<br>
Python list, append 100000 items: 0.01161<br>
Array list, append 100000 items: 0.46854<br>
Array list, append 100000 items at once: 0.05801<br>
Python list, prepend 100000 items: 1.96168<br>
Array list, prepend 100000 items: 12.83371<br>
Array list, append 100000 items at once: 0.06002<br>
<br>
<br>
<br>
I realize I did not answer all Chris' questions:<br>
<span><br>
>>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )<br>
</span>>>> for item in L: print(item)<br>
<span>[0]<br>
[1 2]<br>
[3 4 5]<br>
[6 7 8 9]<br>
<br>
</span>>>> print (type(L.data))<br>
<class 'numpy.ndarray'><br>
>>> print(L.data.dtype)<br>
int64<br>
>>> print(L.data.shape)<br>
(10,)<br>
<br>
<br>
I did not implement operations yet, but it would be a matter for transferring call to the underlying numpy data array.<br>
>>> L._data *= 2<br>
>>> print(L)<br>
[[0], [4 8], [12 16 20], [24 28 32 36]]<br>
<div><div><br>
<br>
<br>
> On 23 Dec 2015, at 09:34, Stephan Hoyer <<a href="mailto:shoyer@gmail.com" target="_blank">shoyer@gmail.com</a>> wrote:<br>
><br>
> We have a type similar to this (a typed list) internally in pandas, although it is restricted to a single dimension and far from feature complete -- it only has .append and a .to_array() method for converting to a 1d numpy array. Our version is written in Cython, and we use it for performance reasons when we would otherwise need to create a list of unknown length:<br>
> <a href="https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99" rel="noreferrer" target="_blank">https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99</a><br>
><br>
> In my experience, it's several times faster than using a builtin list from Cython, which makes sense given that it needs to copy about 1/3 the data (no type or reference count for individual elements). Obviously, it uses 1/3 the space to store the data, too. We currently don't expose this object externally, but it could be an interesting project to adapt this code into a standalone project that could be more broadly useful.<br>
><br>
> Cheers,<br>
> Stephan<br>
><br>
><br>
><br>
> On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker <<a href="mailto:chris.barker@noaa.gov" target="_blank">chris.barker@noaa.gov</a>> wrote:<br>
><br>
> sorry for being so lazy as to not go look at the project pages, but....<br>
><br>
> This sounds like it could be really useful, and maybe supercise a coupl eof half-baked projects of mine. But -- what does "dynamic" mean?<br>
><br>
> - can you append to these arrays?<br>
> - can it support "ragged arrrays" -- it looks like it does.<br>
><br>
> >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )<br>
> >>> print(L)<br>
> [[0], [1 2], [3 4 5], [6 7 8 9]]<br>
><br>
> so this looks like a ragged array -- but what do you get when you do:<br>
><br>
> for row in L:<br>
> print row<br>
><br>
><br>
> >>> print(L.data)<br>
> [0 1 2 3 4 5 6 7 8<br>
><br>
> is .data a regular old 1-d numpy array?<br>
><br>
> >>> L = ArrayList( np.arange(10), [3,3,4])<br>
> >>> print(L)<br>
> [[0 1 2], [3 4 5], [6 7 8 9]]<br>
> >>> print(L.data)<br>
> [0 1 2 3 4 5 6 7 8 9]<br>
><br>
><br>
> does an ArrayList act like a numpy array in other ways:<br>
><br>
> L * 5<br>
><br>
> L* some_array<br>
><br>
> in which case, how does it do broadcasting???<br>
><br>
> Thanks,<br>
><br>
> -CHB<br>
><br>
> >>> L = ArrayList(["Hello", "world", "!"])<br>
> >>> print(L[0])<br>
> 'Hello'<br>
> >>> L[1] = "brave new world"<br>
> >>> print(L)<br>
> ['Hello', 'brave new world', '!']<br>
><br>
><br>
><br>
> Nicolas<br>
><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
> <a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
><br>
><br>
><br>
><br>
> --<br>
><br>
> Christopher Barker, Ph.D.<br>
> Oceanographer<br>
><br>
> Emergency Response Division<br>
> NOAA/NOS/OR&R <a href="tel:%28206%29%20526-6959" value="+12065266959" target="_blank">(206) 526-6959</a> voice<br>
> 7600 Sand Point Way NE <a href="tel:%28206%29%20526-6329" value="+12065266329" target="_blank">(206) 526-6329</a> fax<br>
> Seattle, WA 98115 <a href="tel:%28206%29%20526-6317" value="+12065266317" target="_blank">(206) 526-6317</a> main reception<br>
><br>
> <a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a><br>
><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
> <a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
</div></div></blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R <a href="tel:%28206%29%20526-6959" value="+12065266959" target="_blank">(206) 526-6959</a> voice<br>7600 Sand Point Way NE <a href="tel:%28206%29%20526-6329" value="+12065266329" target="_blank">(206) 526-6329</a> fax<br>Seattle, WA 98115 <a href="tel:%28206%29%20526-6317" value="+12065266317" target="_blank">(206) 526-6317</a> main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div>