<div dir="ltr"><div class="markdown-here-wrapper" style="font-size:1em;font-family:Helvetica,arial,freesans,clean,sans-serif;color:rgb(34,34,34);background-color:rgb(255,255,255);border:none;line-height:1.2"><blockquote style="margin:1em 0px;border-left:4px solid rgb(221,221,221);padding:0px 1em;color:rgb(119,119,119);quotes:none">
<p style="margin:1em 0px">Because dtypes were low level with clear memory layout and stayed that way</p>
</blockquote>
<p style="margin:1em 0px">Dtypes have supported padded and out-of-order-fields <a href="https://github.com/numpy/numpy/blob/4772f10191f87a3446f4862de6d4b953e0dd95ff/scipy/base/src/multiarraymodule.c#L2750-L2766" style="color:rgb(51,51,238);text-decoration:none">since at least 2005 (v0.8.4)</a>, and I would guess that the memory layout has not changed since.</p>
<p style="margin:1em 0px">The house has always been made out of glass, it just didn’t look fragile until we showed people where the stones were.</p>
<div title="MDH:Jmd0O8KgPHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoMzMsIDMzLCAzMyk7Ij5CZWNhdXNlIGR0eXBl
cyB3ZXJlIGxvdyBsZXZlbCB3aXRoIGNsZWFyIG1lbW9yeSBsYXlvdXQgYW5kIHN0YXllZCB0aGF0
IHdheTxicj48L3NwYW4+PGJyPkR0eXBlcyBoYXZlIHN1cHBvcnRlZCBwYWRkZWQgYW5kIG91dC1v
Zi1vcmRlci1maWVsZHMgW3NpbmNlIGF0IGxlYXN0IDIwMDUgKHYwLjguNCldKDxhIGhyZWY9Imh0
dHBzOi8vZ2l0aHViLmNvbS9udW1weS9udW1weS9ibG9iLzQ3NzJmMTAxOTFmODdhMzQ0NmY0ODYy
ZGU2ZDRiOTUzZTBkZDk1ZmYvc2NpcHkvYmFzZS9zcmMvbXVsdGlhcnJheW1vZHVsZS5jI0wyNzUw
LUwyNzY2Ij5odHRwczovL2dpdGh1Yi5jb20vbnVtcHkvbnVtcHkvYmxvYi80NzcyZjEwMTkxZjg3
YTM0NDZmNDg2MmRlNmQ0Yjk1M2UwZGQ5NWZmL3NjaXB5L2Jhc2Uvc3JjL211bHRpYXJyYXltb2R1
bGUuYyNMMjc1MC1MMjc2NjwvYT4pLCBhbmQgSSB3b3VsZCBndWVzcyB0aGF0IHRoZSBtZW1vcnkg
bGF5b3V0IGhhcyBub3QgY2hhbmdlZCBzaW5jZS48YnI+PGRpdj48YnI+PC9kaXY+PGRpdj5UaGUg
aG91c2UgaGFzIGFsd2F5cyBiZWVuIG1hZGUgb3V0IG9mIGdsYXNzLCBpdCBqdXN0IGRpZG4ndCBs
b29rIGZyYWdpbGUgdW50aWwgd2Ugc2hvd2VkIHBlb3BsZSB3aGVyZSB0aGUgc3RvbmVzIHdlcmUu
PGJyPjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj4=" style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, 29 Jan 2018 at 20:51 <<a href="mailto:josef.pktd@gmail.com">josef.pktd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jan 29, 2018 at 10:44 PM, Allan Haldane <span dir="ltr"><<a href="mailto:allanhaldane@gmail.com" target="_blank">allanhaldane@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 01/29/2018 05:59 PM, <a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_1760201120190612322gmail-">
<br>
<br>
On Mon, Jan 29, 2018 at 5:50 PM, <<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a> <mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>>> wrote:<br>
<br>
<br>
<br>
On Mon, Jan 29, 2018 at 4:11 PM, Allan Haldane<br></span><span class="m_1760201120190612322gmail-">
<<a href="mailto:allanhaldane@gmail.com" target="_blank">allanhaldane@gmail.com</a> <mailto:<a href="mailto:allanhaldane@gmail.com" target="_blank">allanhaldane@gmail.com</a>>> wrote:<br>
<br>
On 01/29/2018 04:02 PM, <a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a><br></span><span class="m_1760201120190612322gmail-">
<mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>> wrote:<br>
><br>
><br>
> On Mon, Jan 29, 2018 at 3:44 PM, Benjamin Root <<a href="mailto:ben.v.root@gmail.com" target="_blank">ben.v.root@gmail.com</a> <mailto:<a href="mailto:ben.v.root@gmail.com" target="_blank">ben.v.root@gmail.com</a>><br></span><div><div class="m_1760201120190612322gmail-h5">
> <mailto:<a href="mailto:ben.v.root@gmail.com" target="_blank">ben.v.root@gmail.com</a> <mailto:<a href="mailto:ben.v.root@gmail.com" target="_blank">ben.v.root@gmail.com</a>>>> wrote:<br>
><br>
> I <3 structured arrays. I love the fact that I can access data by<br>
> row and then by fieldname, or vice versa. There are times when I<br>
> need to pass just a column into a function, and there are times when<br>
> I need to process things row by row. Yes, pandas is nice if you want<br>
> the specialized indexing features, but it becomes a bear to deal<br>
> with if all you want is normal indexing, or even the ability to<br>
> easily loop over the dataset.<br>
><br>
><br>
> I don't think there is a doubt that structured arrays, arrays with<br>
> structured dtypes, are a useful container. The question is whether they<br>
> should be more or the foundation for more.<br>
><br>
> For example, computing a mean, or reduce operation, over numeric element<br>
> ("columns"). Before padded views it was possible to index by selecting<br>
> the relevant "columns" and view them as standard array. With padded<br>
> views that breaks and AFAICS, there is no way in numpy 1.14.0 to compute<br>
> a mean of some "columns". (I don't have numpy 1.14 to try or find a<br>
> workaround, like maybe looping over all relevant columns.)<br>
><br>
> Josef<br>
<br>
Just to clarify, structured types have always had padding bytes,<br>
that<br>
isn't new.<br>
<br>
What *is* new (which we are pushing to 1.15, I think) is that it<br>
may be<br>
somewhat more common to end up with padding than before, and<br>
only if you<br>
are specifically using multi-field indexing, which is a fairly<br>
specialized case.<br>
<br>
I think recfunctions already account properly for padding bytes.<br>
Except<br>
for the bug in #8100, which we will fix, padding-bytes in<br>
recarrays are<br>
more or less invisible to a non-expert who only cares about<br>
dataframe-like behavior.<br>
<br>
In other words, padding is no obstacle at all to computing a<br>
mean over a<br>
column, and single-field indexes in 1.15 behave identically as<br>
before.<br>
The only thing that will change in 1.15 is multi-field indexing,<br>
and it<br>
has never been possible to compute a mean (or any binary<br>
operation) on<br>
multiple fields.<br>
<br>
<br>
from the example in the other thread<br>
a[['b', 'c']].view(('f8', 2)).mean(0)<br>
<br>
<br>
(from the statsmodels usecase:<br>
read csv with genfromtext to get recarray or structured array<br>
select/index the numeric columns<br>
view them as standard array<br>
do whatever we can do with standard numpy arrays<br>
)<br>
</div></div></blockquote>
<br>
Oh ok, I misunderstood. I see your point: a mean over fields is more difficult than before.<span class="m_1760201120190612322gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Or, to phrase it as a question:<br>
<br>
How do we get a standard array with homogeneous dtype from the corresponding elements of a structured dtype in numpy 1.14.0?<br>
<br>
Josef<br>
</blockquote>
<br></span>
The answer may be that "numpy has never had a way to that",<br>
even if in a few special cases you might hack a workaround using views.<br>
<br>
That's what your example seems like to me. It uses an explicit view, which is an "expert" feature since views depend on the exact memory layout and binary representation of the array. Your example only works if the two fields have exactly the same dtype as each other and as the final dtype, and evidently breaks if there is byte padding for any reason.<br>
<br>
Pandas can do row means without these problems:<br>
<br>
>>> pd.DataFrame(np.ones(10, dtype='i8,f8')).mean(axis=0)<br>
<br>
Numpy is missing this functionality, so you or whoever wrote that example figured out a fragile workaround using views.<br></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Once upon a time (*) this wasn't fragile but the only and recommended way. Because dtypes were low level with clear memory layout and stayed that way, it was easy to check item size or whatever and get different views on it.</div><div>e.g. <a href="https://mail.scipy.org/pipermail/numpy-discussion/2008-December/039340.html" target="_blank">https://mail.scipy.org/pipermail/numpy-discussion/2008-December/039340.html</a></div><div><br></div><div>(*) pre-pandas, pre-stackoverflow on the mailing lists which was for me roughly 2008 to 2012</div><div>but a late thread <a href="https://mail.scipy.org/pipermail/numpy-discussion/2015-October/074014.html" target="_blank">https://mail.scipy.org/pipermail/numpy-discussion/2015-October/074014.html</a></div><div>"What is now the recommended way of converting structured dtypes/recarrays to ndarrays?"</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I suggest that if we want to allow either means over fields, or conversion of a n-D structured array to an n+1-D regular ndarray, we should add a dedicated function to do so in numpy.lib.recfunctions<br>
which does not depend on the binary representation of the array.<br>
<br></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I don't really want to defend an obsolete (?) usecase of structured dtypes.</div><div><br></div><div>However, I think there should be a decision about the future plans for whether dataframe like usages of structure dtypes or through higher level classes or functions are still supported, instead of removing slowly and silently (*) the foundation for this use case, either support this usage or say you will be dropping it. </div><div><br></div><div>(*) I didn't read the details of the release notes</div><div><br></div><div><br></div><div>And another footnote about obsolete:</div><div>Given that I'm the only one arguing about the dataframe_like usecase of recarrays and structured dtypes, I think they are dead <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">for this specific usecase</span> and only my inertia and conservativeness kept them alive in statsmodels.</div><div><br></div><div><br></div><div>Josef</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Allan<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_1760201120190612322gmail-">
Josef<br>
<br>
<br>
Allan<br>
<br>
><br>
> Cheers!<br>
> Ben Root<br>
><br>
> On Mon, Jan 29, 2018 at 3:24 PM, <<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a> <mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>><br></span><span class="m_1760201120190612322gmail-">
> <mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a> <mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>>>> wrote:<br>
><br>
><br>
><br>
> On Mon, Jan 29, 2018 at 2:55 PM, Stefan van der Walt<br>
> <<a href="mailto:stefanv@berkeley.edu" target="_blank">stefanv@berkeley.edu</a> <mailto:<a href="mailto:stefanv@berkeley.edu" target="_blank">stefanv@berkeley.edu</a>><br></span><span class="m_1760201120190612322gmail-">
<mailto:<a href="mailto:stefanv@berkeley.edu" target="_blank">stefanv@berkeley.edu</a> <mailto:<a href="mailto:stefanv@berkeley.edu" target="_blank">stefanv@berkeley.edu</a>>>> wrote:<br>
><br>
> On Mon, 29 Jan 2018 14:10:56 -0500, <a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a> <mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>><br></span>
> <mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a><div><div class="m_1760201120190612322gmail-h5"><br>
<mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>>> wrote:<br>
><br>
> Given that there is pandas, xarray, dask and<br>
more, numpy<br>
> could as well drop<br>
> any pretense of supporting dataframe_likes.<br>
Or, adjust<br>
> the recfunctions so<br>
> we can still work dataframe_like with structured<br>
> dtypes/recarrays/recfunctions.<br>
><br>
><br>
> I haven't been following the duckarray discussion<br>
carefully,<br>
> but could<br>
> this be an opportunity for a dataframe protocol,<br>
so that we<br>
> can have<br>
> libraries ingest structured arrays, record<br>
arrays, pandas<br>
> dataframes,<br>
> etc. without too much specialized code?<br>
><br>
><br>
> AFAIU while not being in the data handling area,<br>
pandas defines<br>
> the interface and other libraries provide pandas<br>
compatible<br>
> interfaces or implementations.<br>
><br>
> statsmodels currently still has recarray support and<br>
usage. In<br>
> some interfaces we support pandas, recarrays and<br>
plain arrays,<br>
> or anything where asarray works correctly.<br>
><br>
> But recarrays became messy to support, one rewrite of<br>
some<br>
> functions last year converts recarrays to pandas,<br>
does the<br>
> manipulation and then converts back to recarrays.<br>
> Also we need to adjust our recarray usage with new numpy<br>
> versions. But there is no real benefit because I<br>
doubt that<br>
> statsmodels still has any recarray/structured dtype<br>
users. So,<br>
> we only have to remove our own uses in the datasets<br>
and unit tests.<br>
><br>
> Josef<br>
><br>
><br>
><br>
><br>
> Stéfan<br>
><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a>><br></div></div>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><span class="m_1760201120190612322gmail-"><br>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a>>><br>
> <a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a>><br>
> <<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a>>><br>
><br>
><br>
><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a>><br></span>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><span class="m_1760201120190612322gmail-"><br>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a>>><br>
> <a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a>><br>
> <<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a>>><br>
><br>
><br>
><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a>><br></span>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><span class="m_1760201120190612322gmail-"><br>
<mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a>>><br>
> <a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a>><br>
> <<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a>>><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a> <mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a>><br>
> <a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a>><br>
><br>
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a> <mailto:<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a>><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a>><br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
<br>
</span></blockquote><div class="m_1760201120190612322gmail-HOEnZb"><div class="m_1760201120190612322gmail-h5">
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
</div></div></blockquote></div></div></div>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
</blockquote></div>