Mailman 3 November 2018 - NumPy-Discussion

Implementations of ndarray.__array_function__ (and ndarray.__array_ufunc__)
by Marten van Kerkwijk 05 Nov '18

05 Nov '18

Hi again, Another thought about __array_function__, this time about the implementation for ndarray. In it, we currently check whether any of the types define a (different) __array_function__, and, if so, give up. This seems too strict: I think that, at least in principle, subclasses should be allowed through even if they override __array_function__. This thought was triggered by Travis pointing to the Liskov substitution principle [1], that code written for a given type should just work on a (properly written) subclass. This suggests `ndarray` should not exclude subclasses even if they override __array_function__, since if the subclass does not work that way, it can already ensure an error is raised since it knows it is called first. Indeed, this is also how python itself works: if, eg., I subclass list as follows: ``` class MyList(list): def __radd__(self, other): return NotImplemented ``` then any `list + mylist` will just concatenate the lists, even though `MyList.__radd__` explicitly tells it cannot do it (it returning `NotImplemented` means that `list.__add__` gets a change). The reason that we do not already follow this logic may be that currently `ndarray.__array_function__` ends by calling the public function, which will lead to infinite recursion if there is a subclass that overrides __array_function__ and returns NotImplemented. However, inside ndarray.__array_function__, there is no real reason to call the public function - one might as well just call the implementation, in which case this is not a problem. Does the above make sense? I realize that the same would be true for `__array_ufunc__`, though there the situation is slightly trickier since it is not as easy to bypass any further override checks. Nevertheless, it does seem like it would be correct to do the same there. (And if we agree this is the case, I'd quite happily implement it -- with the merger of multiarray and umath it has become much easier to do.) All the best, Marten [1] https://en.wikipedia.org/wiki/Liskov_substitution_principle

2 2

NumPy 1.15.4 release
by Charles R Harris 04 Nov '18

04 Nov '18

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.15.4. This is a bugfix release for bugs and regressions reported following the 1.15.3 release. The most noticeable fix is probably having a boolean type fill value for masked arrays after the use of `==` and `!=`. The Python versions supported by this release are 2.7, 3.4-3.7. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.15.4/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.4>. Compatibility Note ================== The NumPy 1.15.x OS X wheels released on PyPI no longer contain 32-bit binaries. That will also be the case in future releases. See `#11625 < https://github.com/numpy/numpy/issues/11625>`__ for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 4 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Matti Picus - Sebastian Berg - bbbbbbbbba + Pull requests merged ==================== A total of 4 pull requests were merged for this release. - `#12296 <https://github.com/numpy/numpy/pull/12296>`__: BUG: Dealloc cached buffer info - `#12297 <https://github.com/numpy/numpy/pull/12297>`__: BUG: Fix fill value in masked array '==' and '!=' ops. - `#12307 <https://github.com/numpy/numpy/pull/12307>`__: DOC: Correct the default value of `optimize` in `numpy.einsum` - `#12320 <https://github.com/numpy/numpy/pull/12320>`__: REL: Prepare for the NumPy 1.15.4 release Cheers, Charles Harris

1 0

asanyarray vs. asarray
by Matti Picus 01 Nov '18

01 Nov '18

12 24

Attribute hiding APIs for PyArrayObject
by Matti Picus 01 Nov '18

01 Nov '18

TL;DR - should we revert the attribute-hiding constructs in ndarraytypes.h and unify PyArrayObject_fields with PyArrayObject? Background NumPy 1.8 deprecated direct access to PyArrayObject fields. It made PyArrayObject "opaque", and hid the fields behind a PyArrayObject_fields structure https://github.com/numpy/numpy/blob/v1.15.3/numpy/core/include/numpy/ndarra… with a comment about moving this to a private header. In order to access the fields, users are supposed to use PyArray_FIELDNAME functions, like PyArray_DATA and PyArray_NDIM. It seems there were thoughts at the time that numpy might move away from a C-struct based underlying data structure. Other changes were also made to enum names, but those are relatively painless to find-and-replace. NumPy has a mechanism to manage deprecating APIs, C users define NPY_NO_DEPRICATED_API to a desired level, say NPY_1_8_API_VERSION, and can then access the API "as if" they were using NumPy 1.8. Users who do not define NPY_NO_DEPRICATED_API get a warning when compiling, and default to the pre-1.8 API (aliasing of PyArrayObject to PyArrayObject_fields and direct access to the C struct fields). This is convenient for downstream users, both since the new API does not provide much added value, and it is much easier to write a->nd than PyArray_NDIM(a). For instance, pandas uses direct assignment to the data field for fast json parsing https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/src/ujson/pyt… via chunks. Working around the new API in pandas would require more engineering. Also, for example, cython has a mechanism to transpile python code into C, mapping slow python attribute lookup to fast C struct field access https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#… In a parallel but not really related universe, cython recently upgraded the object mapping so that we can quiet the annoying "size changed" runtime warning https://github.com/numpy/numpy/issues/11788 without requiring warning filters, but that requires updating the numpy.pxd file provided with cython, and it was proposed that NumPy actually vendor its own file rather than depending on the cython one (https://github.com/numpy/numpy/issues/11803). The problem We have now made further changes to our API. In NumPy 1.14 we changed UPDATEIFCOPY to WRITEBACKIFCOPY, and in 1.16 we would like to deprecate PyArray_SetNumericOps and PyArray_GetNumericOps. The strange warning when NPY_NO_DEPRICATED_API is annoying. The new API cannot be supported by cython without some deep surgery (https://github.com/cython/cython/pull/2640). When I tried dogfooding an updated numpy.pxd for the only cython code in NumPy, mtrand.pxy, I came across some of these issues (https://github.com/numpy/numpy/pull/12284). Forcing the new API will require downstream users to refactor code or re-engineer constructs, as in the pandas example above. The question Is the attribute-hiding effort worth it? Should we give up, revert the PyArrayObject/PyArrayObject_fields division and allow direct access from C to the numpy internals? Is there another path forward that is less painful? Matti

7 6