[Numpy-discussion] NA masks for NumPy are ready to test

Bruce Southey bsouthey at gmail.com
Fri Aug 19 10:15:56 EDT 2011 

On 08/18/2011 04:43 PM, Mark Wiebe wrote:
> It's taken a lot of changes to get the NA mask support to its current 
> point, but the code ready for some testing now. You can read the 
> work-in-progress release notes here:
>
> https://github.com/m-paradox/numpy/blob/missingdata/doc/release/2.0.0-notes.rst
>
> To try it out, check out the missingdata branch from my github 
> account, here, and build in the standard way:
>
> https://github.com/m-paradox/numpy
>
> The things most important to test are:
>
> * Confirm that existing code still works correctly. I've tested 
> against SciPy and matplotlib.
> * Confirm that the performance of code not using NA masks is the same 
> or better.
> * Try to do computations with the NA values, find places they don't 
> work yet, and nominate unimplemented functionality important to you to 
> be next on the development list. The release notes have a preliminary 
> list of implemented/unimplemented functions.
> * Report any crashes, build problems, or unexpected behaviors.
>
> In addition to adding the NA mask, I've also added features and done a 
> few performance changes here and there, like letting reductions like 
> sum take lists of axes instead of being a single axis or all of them. 
> These changes affect various bugs like 
> http://projects.scipy.org/numpy/ticket/1143 and 
> http://projects.scipy.org/numpy/ticket/533.
>
> Thanks!
> Mark
>
> Here's a small example run using NAs:
>
> >>> import numpy as np
> >>> np.__version__
> '2.0.0.dev-8a5e2a1'
> >>> a = np.random.rand(3,3,3)
> >>> a.flags.maskna = True
> >>> a[np.random.rand(3,3,3) < 0.5] = np.NA
> >>> a
> array([[[NA, NA,  0.11511708],
>         [ 0.46661454,  0.47565512, NA],
>         [NA, NA, NA]],
>
>        [[NA,  0.57860351, NA],
>         [NA, NA,  0.72012669],
>         [ 0.36582123, NA,  0.76289794]],
>
>        [[ 0.65322748,  0.92794386, NA],
>         [ 0.53745165,  0.97520989,  0.17515083],
>         [ 0.71219688,  0.5184328 ,  0.75802805]]])
> >>> np.mean(a, axis=-1)
> array([[NA, NA, NA],
>        [NA, NA, NA],
>        [NA,  0.56260412,  0.66288591]])
> >>> np.std(a, axis=-1)
> array([[NA, NA, NA],
>        [NA, NA, NA],
>        [NA,  0.32710662,  0.10384331]])
> >>> np.mean(a, axis=-1, skipna=True)
> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2474: 
> RuntimeWarning: invalid value encountered in true_divide
>   um.true_divide(ret, rcount, out=ret, casting='unsafe')
> array([[ 0.11511708,  0.47113483,         nan],
>        [ 0.57860351,  0.72012669,  0.56435958],
>        [ 0.79058567,  0.56260412,  0.66288591]])
> >>> np.std(a, axis=-1, skipna=True)
> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2707: 
> RuntimeWarning: invalid value encountered in true_divide
>   um.true_divide(arrmean, rcount, out=arrmean, casting='unsafe')
> /home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/core/fromnumeric.py:2730: 
> RuntimeWarning: invalid value encountered in true_divide
>   um.true_divide(ret, rcount, out=ret, casting='unsafe')
> array([[ 0.        ,  0.00452029,         nan],
>        [ 0.        ,  0.        ,  0.19853835],
>        [ 0.13735819,  0.32710662,  0.10384331]])
> >>> np.std(a, axis=(1,2), skipna=True)
> array([ 0.16786895,  0.15498008,  0.23811937])
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
That is great news!
(Python2.x will be another email.)

Python3.1 and Python3.2 failed with building 
'multiarraymodule_onefile.o' but I could not see any obvious reason.

I had removed my build directory and then 'python3 setup.py build' but I 
saw this message:
Running from numpy source directory.
numpy/core/setup_common.py:86: MismatchCAPIWarning: API mismatch 
detected, the C API version numbers have to be updated. Current C api 
version is 6, with checksum ef5688af03ffa23dd8e11734f5b69313, but 
recorded checksum for C API version 6 in codegen_dir/cversions.txt is 
e61d5dc51fa1c6459328266e215d6987. If functions were added in the C API, 
you have to update C_API_VERSION  in numpy/core/setup_common.py.
   MismatchCAPIWarning)

Upstream of the build log is below.

Bruce

In file included from 
numpy/core/src/multiarray/multiarraymodule_onefile.c:53:0:
numpy/core/src/multiarray/na_singleton.c: At top level:
numpy/core/src/multiarray/na_singleton.c:708:25: error: 
‘Py_TPFLAGS_CHECKTYPES’ undeclared here (not in a function)
numpy/core/src/multiarray/common.c:48:1: warning: ‘_use_default_type’ 
defined but not used
numpy/core/src/multiarray/ctors.h:93:1: warning: ‘_arrays_overlap’ 
declared ‘static’ but never defined
numpy/core/src/multiarray/scalartypes.c.src:2251:1: warning: 
‘gentype_getsegcount’ defined but not used
numpy/core/src/multiarray/scalartypes.c.src:2269:1: warning: 
‘gentype_getcharbuf’ defined but not used
numpy/core/src/multiarray/mapping.c:110:1: warning: ‘_array_ass_item’ 
defined but not used
numpy/core/src/multiarray/number.c:266:1: warning: ‘array_divide’ 
defined but not used
numpy/core/src/multiarray/number.c:464:1: warning: 
‘array_inplace_divide’ defined but not used
numpy/core/src/multiarray/buffer.c:25:1: warning: ‘array_getsegcount’ 
defined but not used
numpy/core/src/multiarray/buffer.c:58:1: warning: ‘array_getwritebuf’ 
defined but not used
numpy/core/src/multiarray/buffer.c:71:1: warning: ‘array_getcharbuf’ 
defined but not used
numpy/core/src/multiarray/na_mask.c:681:1: warning: 
‘PyArray_GetMaskInversionFunction’ defined but not used
In file included from numpy/core/src/multiarray/scalartypes.c.src:25:0,
                  from 
numpy/core/src/multiarray/multiarraymodule_onefile.c:10:
numpy/core/src/multiarray/_datetime.h:9:1: warning: function declaration 
isnât a prototype
In file included from 
numpy/core/src/multiarray/multiarraymodule_onefile.c:13:0:
numpy/core/src/multiarray/datetime.c:33:1: warning: function declaration 
isnât a prototype
In file included from 
numpy/core/src/multiarray/multiarraymodule_onefile.c:17:0:
numpy/core/src/multiarray/arraytypes.c.src: In function âVOID_getitemâ:
numpy/core/src/multiarray/arraytypes.c.src:643:9: warning: passing 
argument 2 of âPyArray_SetBaseObjectâ from incompatible pointer type
build/src.linux-x86_64-3.2/numpy/core/include/numpy/__multiarray_api.h:763:12: 
note: expected âstruct PyObject *â but argument is of type âstruct 
PyArrayObject *â
In file included from 
numpy/core/src/multiarray/multiarraymodule_onefile.c:44:0:
numpy/core/src/multiarray/nditer_pywrap.c: In function ânpyiter_subscriptâ:
numpy/core/src/multiarray/nditer_pywrap.c:2395:29: warning: passing 
argument 1 of âPySlice_GetIndicesâ from incompatible pointer type
/usr/local/include/python3.2m/sliceobject.h:38:5: note: expected âstruct 
PyObject *â but argument is of type âstruct PySliceObject *â
numpy/core/src/multiarray/nditer_pywrap.c: In function 
ânpyiter_ass_subscriptâ:
numpy/core/src/multiarray/nditer_pywrap.c:2440:29: warning: passing 
argument 1 of âPySlice_GetIndicesâ from incompatible pointer type
/usr/local/include/python3.2m/sliceobject.h:38:5: note: expected âstruct 
PyObject *â but argument is of type âstruct PySliceObject *â
In file included from 
numpy/core/src/multiarray/multiarraymodule_onefile.c:53:0:
numpy/core/src/multiarray/na_singleton.c: At top level:
numpy/core/src/multiarray/na_singleton.c:708:25: error: 
âPy_TPFLAGS_CHECKTYPESâ undeclared here (not in a function)
numpy/core/src/multiarray/common.c:48:1: warning: â_use_default_typeâ 
defined but not used
numpy/core/src/multiarray/ctors.h:93:1: warning: â_arrays_overlapâ 
declared âstaticâ but never defined
numpy/core/src/multiarray/scalartypes.c.src:2251:1: warning: 
âgentype_getsegcountâ defined but not used
numpy/core/src/multiarray/scalartypes.c.src:2269:1: warning: 
âgentype_getcharbufâ defined but not used
numpy/core/src/multiarray/mapping.c:110:1: warning: â_array_ass_itemâ 
defined but not used
numpy/core/src/multiarray/number.c:266:1: warning: âarray_divideâ 
defined but not used
numpy/core/src/multiarray/number.c:464:1: warning: 
âarray_inplace_divideâ defined but not used
numpy/core/src/multiarray/buffer.c:25:1: warning: âarray_getsegcountâ 
defined but not used
numpy/core/src/multiarray/buffer.c:58:1: warning: âarray_getwritebufâ 
defined but not used
numpy/core/src/multiarray/buffer.c:71:1: warning: âarray_getcharbufâ 
defined but not used
numpy/core/src/multiarray/na_mask.c:681:1: warning: 
âPyArray_GetMaskInversionFunctionâ defined but not used
error: Command "gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes -fPIC -Inumpy/core/include 
-Ibuild/src.linux-x86_64-3.2/numpy/core/include/numpy 
-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core 
-Inumpy/core/src/npymath -Inumpy/core/src/multiarray 
-Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include 
-I/usr/local/include/python3.2m 
-Ibuild/src.linux-x86_64-3.2/numpy/core/src/multiarray 
-Ibuild/src.linux-x86_64-3.2/numpy/core/src/umath -c 
numpy/core/src/multiarray/multiarraymodule_onefile.c -o 
build/temp.linux-x86_64-3.2/numpy/core/src/multiarray/multiarraymodule_onefile.o" 
failed with exit status 1



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110819/4d8c9a93/attachment.html>

More information about the NumPy-Discussion mailing list