[Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

[josef.pktd at gmail.com]

On Fri, Oct 16, 2015 at 8:56 PM, Allan Haldane <allanhaldane at gmail.com>
wrote:

> On 10/16/2015 05:31 PM, josef.pktd at gmail.com wrote:
> >
> >
> > On Fri, Oct 16, 2015 at 2:21 PM, Charles R Harris
> > <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
> >
> >
> >
> >     On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris
> >     <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>>
> wrote:
> >
> >
> >
> >         On Fri, Oct 16, 2015 at 11:58 AM, <josef.pktd at gmail.com
> >         <mailto:josef.pktd at gmail.com>> wrote:
> >
> >             was there a change with reduce operations with recarrays in
> >             1.10 or 1.10.1?
> >
> >             Travis shows a new test failure in the statsmodels testsuite
> >             with 1.10.1:
> >
> >             ERROR: test suite for <class
> >             'statsmodels.base.tests.test_data.TestRecarrays'>
> >
> >               File
> >
>  "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
> >             line 131, in _handle_constant
> >                 const_idx = np.where(self.exog.ptp(axis=0) ==
> >             0)[0].squeeze()
> >             TypeError: cannot perform reduce with flexible type
> >
> >
> >             Sorry for asking so late.
> >             (statsmodels is short on maintainers, and I'm distracted)
> >
> >
> >             statsmodels still has code to support recarrays and
> >             structured dtypes from the time before pandas became
> >             popular, but I don't think anyone is using them together
> >             with statsmodels anymore.
> >
> >
> >         There were several commits dealing both recarrays and ufuncs, so
> >         this might well be a regression.
> >
> >
> >     A bisection would be helpful. Also, open an issue.
> >
> >
> >
> > The reason for the test failure might be somewhere else hiding behind
> > several layers of statsmodels, but only started to show up with numpy
> 1.10.1
> >
> > I already have the reduce exception with my currently installed numpy
> > '1.9.2rc1'
> >
> >>>> x = np.random.random(9*3).view([('const', 'f8'),('x_1', 'f8'),
> > ('x_2', 'f8')]).view(np.recarray)
> >
> >>>> np.ptp(x, axis=0)
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> >   File
> >
> "C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\core\fromnumeric.py",
> > line 2047, in ptp
> >     return ptp(axis, out)
> > TypeError: cannot perform reduce with flexible type
> >
> >
> > Sounds like fun, and I don't even know how to automatically bisect.
> >
> > Josef
>
> That example isn't the problem (ptp should definitely fail on structured
> arrays), but I've tracked down what is - it has to do with views of
> record arrays.
>
> The fix looks simple, I'll get it in for the next release.
>

Thanks,

I realized that at that point in the statsmodels code we should have only
regular ndarrays, so the array conversion fails somewhere.

AFAICS, the main helper function to convert is

def struct_to_ndarray(arr):
    return arr.view((float, len(arr.dtype.names)))

which doesn't look like it will handle other dtypes than float64. Nobody
ever complained, so maybe our test suite is the only user of this.

What is now the recommended way of converting structured dtypes/recarrays
to ndarrays?

Josef



>
> Allan
>
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20151016/5726dab0/attachment.html>