[Numpy-discussion] numpy 1.10.1 reduce operation on recarrays

[Allan Haldane]

On 10/16/2015 05:31 PM, josef.pktd at gmail.com wrote:
> 
> 
> On Fri, Oct 16, 2015 at 2:21 PM, Charles R Harris
> <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
> 
> 
> 
>     On Fri, Oct 16, 2015 at 12:20 PM, Charles R Harris
>     <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
> 
> 
> 
>         On Fri, Oct 16, 2015 at 11:58 AM, <josef.pktd at gmail.com
>         <mailto:josef.pktd at gmail.com>> wrote:
> 
>             was there a change with reduce operations with recarrays in
>             1.10 or 1.10.1?
> 
>             Travis shows a new test failure in the statsmodels testsuite
>             with 1.10.1:
> 
>             ERROR: test suite for <class
>             'statsmodels.base.tests.test_data.TestRecarrays'>
> 
>               File
>             "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py",
>             line 131, in _handle_constant
>                 const_idx = np.where(self.exog.ptp(axis=0) ==
>             0)[0].squeeze()
>             TypeError: cannot perform reduce with flexible type
> 
> 
>             Sorry for asking so late.
>             (statsmodels is short on maintainers, and I'm distracted)
> 
> 
>             statsmodels still has code to support recarrays and
>             structured dtypes from the time before pandas became
>             popular, but I don't think anyone is using them together
>             with statsmodels anymore.
> 
> 
>         There were several commits dealing both recarrays and ufuncs, so
>         this might well be a regression.
> 
> 
>     A bisection would be helpful. Also, open an issue.
> 
> 
> 
> The reason for the test failure might be somewhere else hiding behind
> several layers of statsmodels, but only started to show up with numpy 1.10.1
> 
> I already have the reduce exception with my currently installed numpy
> '1.9.2rc1'
> 
>>>> x = np.random.random(9*3).view([('const', 'f8'),('x_1', 'f8'),
> ('x_2', 'f8')]).view(np.recarray)
> 
>>>> np.ptp(x, axis=0)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File
> "C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\core\fromnumeric.py",
> line 2047, in ptp
>     return ptp(axis, out)
> TypeError: cannot perform reduce with flexible type
> 
> 
> Sounds like fun, and I don't even know how to automatically bisect.
> 
> Josef

That example isn't the problem (ptp should definitely fail on structured
arrays), but I've tracked down what is - it has to do with views of
record arrays.

The fix looks simple, I'll get it in for the next release.

Allan