<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 22, 2018 at 10:53 AM, <span dir="ltr"><<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-h5">On Sun, Jan 21, 2018 at 9:48 PM, Allan Haldane <span dir="ltr"><<a href="mailto:allanhaldane@gmail.com" target="_blank">allanhaldane@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello all,<br>
<br>
We are making a decision (again) about what to do about the<br>
behavior of multiple-field indexing of structured arrays: Should<br>
it return a view or a copy, and on what release schedule?<br>
<br>
As a reminder, this refers to operations like (1.13 behavior):<br>
<br>
>>> a = np.zeros(3, dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'f4')])<br>
>>> a[['a', 'c']]<br>
array([(0, 0.), (0, 0.), (0, 0.)],<br>
dtype=[('a', '<i4'), ('c', '<f4')]<br>
<br>
In numpy 1.14.0 we made this return a view instead of a copy, but<br>
downstream test failures suggest we reconsider. In our current<br>
implementation for 1.14.1, we have reverted this change, but<br>
still plan to go through with it in 1.15.<br>
<br>
See here for our discussion the problem and solutions:<br>
<a href="https://github.com/numpy/numpy/pull/10411" rel="noreferrer" target="_blank">https://github.com/numpy/numpy<wbr>/pull/10411</a><br>
<br>
The two main options we have discussed are either to try to make<br>
the change in 1.15, or never make the change at all and always<br>
return a copy.<br>
<br>
Here are some pros and cons:<br>
<br>
Pros (change to view in 1.15)<br>
=============================<br>
<br>
* Views are useful and convenient. Other forms of indexing also<br>
often return views so this is more consistent.<br>
* This change has been planned since numpy 1.7 in 2009,<br>
and there have been visible FutureWarnings about it since<br>
then. Anyone whose code will break should have seen the<br>
warnings. It has been extensively warned about in recent<br>
release notes.<br>
* Past discussions have supported the change. See my comment in<br>
the PR with many links to them and to other history.<br>
* Users have requested the change on the list.<br>
* Possibly a majority of the reported code failures were not<br>
actually caused by the change, but by another bug (#8100)<br>
involving np.load/np.save which this change exposed. If we<br>
push it off to 1.15, we will have time to fix this other bug.<br>
(There were no FutureWarnings for this breakage, of course).<br>
* The code that really will break is of the form<br>
a[['a', 'c']].view('i8')<br>
because the returned itemsize is different. This has<br>
raised FutureWarnings since numpy 1.7, and no users reported<br>
failures due to this change. In the PR we still try to<br>
mitigate this breakage by introducing a new method<br>
`pack_fields`, which converts the result into the 1.13 form,<br>
so that<br>
np.pack_fields(a[['a', 'c']]).view('i8')<br>
will work.<br>
<br>
<br>
Cons (keep returning a copy)<br>
============================<br>
<br>
* The extra convenience is not really that much, and fancy<br>
indexing also returns a copy instead of a view, so there is<br>
a precedent there.<br>
* We want to minimize compatibility breaks with old behavior.<br>
We've had a fair amount of discussion and complaints about<br>
how we break things in general.<br>
* We have lived with a "copy" for 8 years now. At some point the<br>
behavior gets set in stone for compatibility reasons.<br>
* Users have written to the list and github about their code<br>
breaking in 1.14.0. As far as I am aware, they all refer<br>
to the #8100 problem.<br>
* If a new function `pack_fields` is needed to guard against<br>
mishaps with the view behavior, that seems like a sign that<br>
keeping the copy behavior is the best option from an API<br>
perspective.<br>
<br>
My initial vote is go with the change in 1.15: The "view" code<br>
that will ultimately break (not the code related to #8100) has<br>
been sending FutureWarnings for many years, and I am not aware of<br>
any user complaints involving it: All the complaints so far<br>
would be fixed with #8100 in 1.15.<br><br></blockquote><div><br></div></div></div><div>(Note based on a linked mailing list thread, 2012 might be the last time I looked more closely at structured dtypes. </div><div>So some of what I understand might be outdated.)</div><div><br></div><div><br></div><div>views on structured dtypes are very important, but viewing them as standard arrays with standard dtypes is the main part that I had used.</div><div>Essentially structured dtypes are useless for any computation, e.g. just some simple reduce operation. To work with them we need a standard view.<br></div><div><br></div><div>I think the usecase that fails in statsmodels (except there is no test failure anymore because we switched to using pandas in the unit test)</div></div></div></div></blockquote><div><br></div><div><br></div><div>do add a detail here </div><div><br></div><div>results is a recarray created from a csv file with</div><div>results = genfromtxt(open(filename, "rb"), delimiter=",", names=True,dtype=float)<br></div><div><br></div><div>['acvar_lb','acvar<wbr>_ub'] are the last two columns, so this corresponds to my example below where AFAIU no padding is necessary to get a view.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div><br></div><div><div> cls.confint_res = cls.results[['acvar_lb','acvar<wbr>_ub']].view((float,</div><div>> 2))</div><div>E ValueError: Changing the dtype to a subarray type is only supported if the total itemsize is unchanged</div><div><br></div></div><div><br></div><div>This is similar to the above example</div><span class="gmail-"><div><span style="font-size:12.8px">a[['a', 'c']].view('i8')</span><br></div></span><div><span style="font-size:12.8px">but it doesn't try to combine fields. </span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">In many examples where I used structured dtypes a long time ago, switched between consistent views as either a standard array of subsets or as .structured dtypes.</span></div><div><span style="font-size:12.8px">For this usecase it wouldn't matter whether </span><span style="font-size:12.8px">a[['a', 'c']] returns a view or copy, as long as we can get the second view that is consistent with the selected part of the memory. This would also be independent of whether numpy pads internally and adjusts the strides if possible or not.</span></div><div><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px">>>> np.__version__</span></div><div><span style="font-size:12.8px">'1.11.2'</span></div></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">>>> a = np.ones(5, dtype=[('a', 'i8'), ('b', 'f8'), ('c', 'f8')])</span><br></div><div><div><span style="font-size:12.8px">>>> a</span></div><div><span style="font-size:12.8px">array([(1, 1.0, 1.0), (1, 1.0, 1.0), (1, 1.0, 1.0), (1, 1.0, 1.0),</span></div><div><span style="font-size:12.8px"> (1, 1.0, 1.0)], </span></div><div><span style="font-size:12.8px"> dtype=[('a', '<i8'), ('b', '<f8'), ('c', '<f8')])</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">>>> a.mean(0)</span></div><div><span style="font-size:12.8px">Traceback (most recent call last):</span></div><div><span style="font-size:12.8px"> File "<pyshell#15>", line 1, in <module></span></div><div><span style="font-size:12.8px"> a.mean(0)</span></div><div><span style="font-size:12.8px"> File "C:\...\python-3.4.4.amd64\lib<wbr>\site-packages\numpy\core\_met<wbr>hods.py", line 65, in _mean</span></div><div><span style="font-size:12.8px"> ret = umr_sum(arr, axis, dtype, out, keepdims)</span></div><div><span style="font-size:12.8px">TypeError: cannot perform reduce with flexible type</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">>>> a[['b', 'c']].mean(0)</span></div><div><span style="font-size:12.8px">Traceback (most recent call last):</span></div><div><span style="font-size:12.8px"> File "<pyshell#16>", line 1, in <module></span></div><div><span style="font-size:12.8px"> a[['b', 'c']].mean(0)</span></div><div><span style="font-size:12.8px"> File "C:\...\python-3.4.4.amd64\lib<wbr>\site-packages\numpy\core\_met<wbr>hods.py", line 65, in _mean</span></div><div><span style="font-size:12.8px"> ret = umr_sum(arr, axis, dtype, out, keepdims)</span></div><div><span style="font-size:12.8px">TypeError: cannot perform reduce with flexible type</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">>>> a[['b', 'c']].view(('f8', 2)).mean(0)</span></div><div><span style="font-size:12.8px">array([ 1., 1.])</span></div><div><span style="font-size:12.8px">>>> a[['b', 'c']].view(('f8', 2)).dtype</span></div><div><span style="font-size:12.8px">dtype('float64')</span></div></div><div><br></div><div><br></div><div>Aside The plan is that statsmodels will drop all usage and support for rec_arays/structured dtypes </div>in the following release (0.10).<br>Then structured dtypes are free (from our perspective) to provide low level struct support<br>instead of pretending to be dataframe_like.</div><span class="gmail-HOEnZb"><font color="#888888"><div class="gmail_quote"><br></div></font></span><div class="gmail_quote"><span class="gmail-HOEnZb"><font color="#888888">Josef</font></span><span class="gmail-"><br><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Feel free to also discuss the related proposed change, to make<br>
np.diag return a view instead of a copy. That change has<br>
not been implemented yet, only proposed. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Cheers,<br>
Allan<br>
______________________________<wbr>_________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/numpy-discussion</a><br>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>