<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 16, 2015 at 9:31 PM, Allan Haldane <span dir="ltr"><<a href="mailto:allanhaldane@gmail.com" target="_blank">allanhaldane@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">On 10/16/2015 09:17 PM, <a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a> wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">
<br>
<br>
On Fri, Oct 16, 2015 at 8:56 PM, Allan Haldane <<a href="mailto:allanhaldane@gmail.com" target="_blank">allanhaldane@gmail.com</a><br></span><span class="">
<mailto:<a href="mailto:allanhaldane@gmail.com" target="_blank">allanhaldane@gmail.com</a>>> wrote:<br>
<br>
On 10/16/2015 05:31 PM, <a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a><br></span><span class="">
<mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>> wrote:<br>
><br>
><br>
> On Fri, Oct 16, 2015 at 2:21 PM, Charles R Harris<br>
> <<a href="mailto:charlesr.harris@gmail.com" target="_blank">charlesr.harris@gmail.com</a> <mailto:<a href="mailto:charlesr.harris@gmail.com" target="_blank">charlesr.harris@gmail.com</a>><br></span>
<mailto:<a href="mailto:charlesr.harris@gmail.com" target="_blank">charlesr.harris@gmail.com</a><span class=""><br>
<mailto:<a href="mailto:charlesr.harris@gmail.com" target="_blank">charlesr.harris@gmail.com</a>>>> wrote:<br>
><br>
><br>
><br>
> On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris<br>
> <<a href="mailto:charlesr.harris@gmail.com" target="_blank">charlesr.harris@gmail.com</a> <mailto:<a href="mailto:charlesr.harris@gmail.com" target="_blank">charlesr.harris@gmail.com</a>><br></span>
<mailto:<a href="mailto:charlesr.harris@gmail.com" target="_blank">charlesr.harris@gmail.com</a><span class=""><br>
<mailto:<a href="mailto:charlesr.harris@gmail.com" target="_blank">charlesr.harris@gmail.com</a>>>> wrote:<br>
><br>
><br>
><br>
> On Fri, Oct 16, 2015 at 11:58 AM, <<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a> <mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>><br></span>
> <mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a><div><div class="h5"><br>
<mailto:<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>>>> wrote:<br>
><br>
> was there a change with reduce operations with<br>
recarrays in<br>
> 1.10 or 1.10.1?<br>
><br>
> Travis shows a new test failure in the statsmodels<br>
testsuite<br>
> with 1.10.1:<br>
><br>
> ERROR: test suite for <class<br>
> 'statsmodels.base.tests.test_data.TestRecarrays'><br>
><br>
> File<br>
><br>
"/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",<br>
> line 131, in _handle_constant<br>
> const_idx = np.where(self.exog.ptp(axis=0) ==<br>
> 0)[0].squeeze()<br>
> TypeError: cannot perform reduce with flexible type<br>
><br>
><br>
> Sorry for asking so late.<br>
> (statsmodels is short on maintainers, and I'm distracted)<br>
><br>
><br>
> statsmodels still has code to support recarrays and<br>
> structured dtypes from the time before pandas became<br>
> popular, but I don't think anyone is using them together<br>
> with statsmodels anymore.<br>
><br>
><br>
> There were several commits dealing both recarrays and<br>
ufuncs, so<br>
> this might well be a regression.<br>
><br>
><br>
> A bisection would be helpful. Also, open an issue.<br>
><br>
><br>
><br>
> The reason for the test failure might be somewhere else hiding behind<br>
> several layers of statsmodels, but only started to show up with<br>
numpy 1.10.1<br>
><br>
> I already have the reduce exception with my currently installed numpy<br>
> '1.9.2rc1'<br>
><br>
>>>> x = np.random.random(9*3).view([('const', 'f8'),('x_1', 'f8'),<br>
> ('x_2', 'f8')]).view(np.recarray)<br>
><br>
>>>> np.ptp(x, axis=0)<br>
> Traceback (most recent call last):<br>
> File "<stdin>", line 1, in <module><br>
> File<br>
><br>
"C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\core\fromnumeric.py",<br>
> line 2047, in ptp<br>
> return ptp(axis, out)<br>
> TypeError: cannot perform reduce with flexible type<br>
><br>
><br>
> Sounds like fun, and I don't even know how to automatically bisect.<br>
><br>
> Josef<br>
<br>
That example isn't the problem (ptp should definitely fail on structured<br>
arrays), but I've tracked down what is - it has to do with views of<br>
record arrays.<br>
<br>
The fix looks simple, I'll get it in for the next release.<br>
<br>
<br>
Thanks,<br>
<br>
I realized that at that point in the statsmodels code we should have<br>
only regular ndarrays, so the array conversion fails somewhere.<br>
<br>
AFAICS, the main helper function to convert is<br>
<br>
def struct_to_ndarray(arr):<br>
return arr.view((float, len(arr.dtype.names)))<br>
<br>
which doesn't look like it will handle other dtypes than float64. Nobody<br>
ever complained, so maybe our test suite is the only user of this.<br>
<br>
What is now the recommended way of converting structured<br>
dtypes/recarrays to ndarrays?<br>
<br>
Josef<br>
</div></div></blockquote>
<br>
Yes, that's the code I narrowed it down to as well. I think the code in statsmodels is fine, the problem is actually a bug I must admit I introduced in changes to the way views of recarrays work.<br>
<br>
If you are curious, the bug is in this line:<br>
<br>
<a href="https://github.com/numpy/numpy/blob/master/numpy/core/records.py#L467" rel="noreferrer" target="_blank">https://github.com/numpy/numpy/blob/master/numpy/core/records.py#L467</a><br>
<br>
This line was intended to fix the problem that accessing a nested record array field would lose the 'np.record' dtype. I only considered void structured arrays, and had forgotten about sub-arrays which statsmodels uses.<br>
<br>
I think the fix is to replace `issubclass(val.type, nt.void)` with `val.names` or something similar. I'll take a closer look soon.<div class=""><div class="h5"><br></div></div></blockquote><div><br></div><div>Another example fresh from Travis that might have the same source</div><div><br></div><div>and I didn't even know statsmodels uses recarrays in the models</div><div><br></div><div><div>AssertionError: </div><div>Arrays are not almost equal to 7 decimals</div><div>(shapes (6,), (6, 3) mismatch)</div><div> x: recarray([�?, �;�:B�ѿ](�D����������,</div><div> ��L�������� ƿC�3Y�?, O�����N;�j���8���H��,</div><div> �N�A���� ���T��B;��pٿ, 9m�;_���J��...</div><div> y: array([[ 1. , 0. , 0. ],</div><div> [-0.2794347, -0.100468 , -1.9709737],</div><div> [-0.0469873, -0.1728197, 0.0436493],...</div></div><div><br></div><div><br></div><div>Josef</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><div class="h5">
<br>
Allan<br>
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
</div></div></blockquote></div><br></div></div>