<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 29, 2018 at 1:22 PM, Eric Wieser <span dir="ltr"><<a href="mailto:wieser.eric+numpy@gmail.com" target="_blank">wieser.eric+numpy@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I think that there's a lot of confusion going around about recarrays vs structured arrays.<div><br></div><div>[`recarray`](<a href="https://github.com/numpy/numpy/blob/v1.13.0/numpy/core/records.py" target="_blank">https://github.<wbr>com/numpy/numpy/blob/v1.13.0/<wbr>numpy/core/records.py</a>) are a wrapper around structured arrays that provide:<br>* Attribute access to fields as `arr.field` in addition to the normal `arr['field']`</div><div>* Automatic datatype-guessing for nested lists of tuples (which needs a little work, but seems like a justifiable feature)<br>* An undocumented `field` method that behaves like the 1.14 indexing behavior (!)</div><div><br></div><div>Meanwhile, `recfunctions` is a collection of functions that work on normal structured arrays - so is misleadingly named.</div><div>The only link to recarrays is that most of the functions have a `asrecarray` parameter which applies `.view(recarray)` to the result.</div><div><br></div><div>> <span style="color:rgb(33,33,33)">deprecate recarrays</span></div><div><span style="color:rgb(33,33,33)"><br></span></div><div><span style="color:rgb(33,33,33)">Given how thin an abstraction they are over structured arrays, I don't think you mean this.</span></div><div><span style="color:rgb(33,33,33)">Are you advocating for deprecating structured arrays entirely, or just deprecating recfunctions?</span></div></div></blockquote><div><br></div><div>First, statsmodels is in the pandas camp for dataframes, so I don't have any invested interest in recarrays/structured dtypes anymore.</div><div><br></div><div>What I meant was that structured dtypes with implicit (hidden?) padding becomes unintuitive for the recarray/dataframe usecase. (At least I won't try to update my intuition about having extra things in there that are not specified by the main structured dtype.) Also the dataframe_like usage of structured dtypes doesn't seem to be much under consideration anymore.</div><div><br></div><div>So, my **impression** is that the recent changes make the recarray/dataframe usecase for structured dtypes more difficult.</div><div><br></div><div>Given that there is pandas, xarray, dask and more, numpy could as well drop any pretense of supporting dataframe_likes. Or, adjust the recfunctions so we can still work dataframe_like with structured dtypes/recarrays/recfunctions.</div><div><br></div><div><br></div><div>Josef</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class="HOEnZb"><font color="#888888"><div><span style="color:rgb(33,33,33)"><br></span></div><div><span style="color:rgb(33,33,33)">Eric</span></div></font></span></div><br><div class="gmail_quote"><div><div class="h5"><div dir="ltr">On Mon, 29 Jan 2018 at 09:39 Chris Barker <<a href="mailto:chris.barker@noaa.gov" target="_blank">chris.barker@noaa.gov</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Jan 27, 2018 at 8:50 PM, Allan Haldane <span dir="ltr"><<a href="mailto:allanhaldane@gmail.com" target="_blank">allanhaldane@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 01/26/2018 06:01 PM, <a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a> wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
I thought recarrays were pretty cool back in the day, but pandas is<br>
a much better option.<br>
<br>
So I pretty much only use structured arrays for data exchange with C<br>
code....<br>
<br>
My impression is that this turns into a deprecate recarrays and supporting recfunction issue.<br>
<br></span></blockquote></blockquote><div> </div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span></span></blockquote>
*should* we have any dataframe-like functionality in numpy?</blockquote></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
We get requests every once in a while about how to sort rows, or about adding a "groupby" function. I myself have used recarrays in a dataframe-like way, when I wanted a quick multiple-array object that supported numpy indexing. So there is some demand to have minimal "dataframe-like" behavior in numpy itself.<br>
<br>
recarrays play part of this role currently, though imperfectly due to padding and cache issues. I think I'm comfortable with supporting some minor use of structured/recarrays as dataframe-like, with a warning in docs that the user should really look at pandas/xarray, and that structured arrays are primarily for data exchange.<br></blockquote></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"></blockquote><div><br></div><div>Well, I think we should either:</div><div><br></div><div>deprecate recarrays -- i.e. explicitly not support DataFrame-like functionality in numpy, keeping only the data-exchange functionality as maintained.</div><div><br></div><div>or</div><div><br></div><div>Properly support it -- which doesn't mean re-implementing Pandas or xarray, but it would mean addressing any bug-like issues like not dealing properly with padding.</div><div><br></div><div>Personally, I don't need/want it enough to contribute, but if someone does, great.</div><div><br></div><div>This reminds me a bit of the old numpy.Matrix issue -- it was ALMOST there, but not quite, with issues, and there was essentially no overlap between the people that wanted it and the people that had the time and skills to really make it work.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">(If we want to dream, maybe one day we should make a minimal multiple-array container class. I imagine it would look pretty similar to recarray, but stored as a set of arrays instead of a structured array. But maybe recarrays are good enough, and let's not reimplement pandas either.)<br></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Exactly -- we really don't need to re-implement Pandas....</div><div><br></div><div>(except it's CSV reading capability :-) )</div><div><br></div><div>-CHB</div></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div>-- <br><div class="m_-5912940786836457525m_-5279631270231327231gmail_signature" data-smartmail="gmail_signature"></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="m_-5912940786836457525m_-5279631270231327231gmail_signature" data-smartmail="gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="m_-5912940786836457525m_-5279631270231327231gmail_signature" data-smartmail="gmail_signature">NOAA/NOS/OR&R <a href="tel:(206)%20526-6959" value="+12065266959" target="_blank">(206) 526-6959</a> voice<br>7600 Sand Point Way NE <a href="tel:(206)%20526-6329" value="+12065266329" target="_blank">(206) 526-6329</a> fax<br>Seattle, WA 98115 <a href="tel:(206)%20526-6317" value="+12065266317" target="_blank">(206) 526-6317</a> main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div></div></div><span class="">
______________________________<wbr>_________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/numpy-<wbr>discussion</a><br>
</span></blockquote></div>
<br>______________________________<wbr>_________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/numpy-<wbr>discussion</a><br>
<br></blockquote></div><br></div></div>