NumPy-Discussion October 2020

numpy-discussion@python.org

30 participants
27 discussions

NumPy 1.19.3 release
by Charles R Harris 28 Oct '20

28 Oct '20

Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.19.3 has been released. NumPy 1.19.3 is a small maintenance release with two major improvements: - Python 3.9 binary wheels on all supported platforms, - OpenBLAS fixes for Windows 10 version 2004 fmod bug. This release supports Python 3.6-3.9 and is linked with OpenBLAS 3.12 to avoid some of the fmod problems on Windows 10 version 2004. Microsoft is aware of the problem and users should upgrade when the fix becomes available, the fix here is limited in scope. NumPy Wheels for this release can be downloaded from the PyPI <https://pypi.org/project/numpy/1.19.3/>, source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.19.3>. Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 8 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Chris Brown + - Daniel Vanzo + - E. Madison Bray + - Hugo van Kemenade + - Ralf Gommers - Sebastian Berg - @danbeibei + *Pull requests merged* A total of 10 pull requests were merged for this release. - #17298: BLD: set upper versions for build dependencies - #17336: BUG: Set deprecated fields to null in PyArray_InitArrFuncs - #17446: ENH: Warn on unsupported Python 3.10+ - #17450: MAINT: Update test_requirements.txt. - #17522: ENH: Support for the NVIDIA HPC SDK nvfortran compiler - #17568: BUG: Cygwin Workaround for #14787 on affected platforms - #17647: BUG: Fix memory leak of buffer-info cache due to relaxed strides - #17652: MAINT: Backport openblas_support from master. - #17653: TST: Add Python 3.9 to the CI testing on Windows, Mac. - #17660: TST: Simplify source path names in test_extending. Cheers, Charles Harris

2 1

NumPy Community Meeting Wednesday (1h difference for Europe due to daylight saving ending)
by Sebastian Berg 27 Oct '20

27 Oct '20

Hi all, There will be a NumPy Community meeting Wednesday October 27th at 1pm Pacific Time (20:00 UTC). Everyone is invited and encouraged to join in and edit the work-in-progress meeting topics and notes at: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

Accepting NEP 42 — New and extensible DTypes
by Sebastian Berg 27 Oct '20

27 Oct '20

Hi all, after another thorough revision of NEP 42 (much thanks to Ben!), I propose accepting the NEP, with the note that details are expected change. I am always happy to clarify and review the document based on feedback, but I feel the important technical points should be very clear and settled. Exposing all of the proposed API may need additional detailed API discussion. My focus is still a bit on the big picture design choices that the NEP makes need to move forward and settle the implementation internal to NumPy, although I am happy to discuss the details! The title of the NEP is: NEP 42 — New and extensible DTypes And available at: https://numpy.org/neps/nep-0042-new-dtypes.html While enabling new user-defined DTypes is the main goal, the main work is the internal restructure of NumPy's own DTypes necessary to allow that. I have pasted the "Abstract" and "Motivation and scope" section below, which give a good overview of the issues and we are trying to address. It is followed by the "Usage and impact" section which gives a big- picture overview of the design. I will refer to the full NEP for more detailed technical decisions and explanations. Cheers, Sebastian PS: In some places NEP 42 references NEP 43, for which I hope to merge the draft soon, the current status is here: https://github.com/numpy/numpy/pull/16723 However, this should be mainly interested for those wishing to go into more technical details. *********************************************************************** ******* Abstract *********************************************************************** ******* NumPy's dtype architecture is monolithic -- each dtype is an instance of a single class. There's no principled way to expand it for new dtypes, and the code is difficult to read and maintain. As :ref:`NEP 41 <NEP41>` explains, we are proposing a new architecture that is modular and open to user additions. dtypes will derive from a new ``DType`` class serving as the extension point for new types. ``np.dtype("float64")`` will return an instance of a ``Float64`` class, a subclass of root class ``np.dtype``. This NEP is one of two that lay out the design and API of this new architecture. This NEP addresses dtype implementation; NEP 43 addresses universal functions. .. note:: Details of the private and external APIs may change to reflect user comments and implementation constraints. The underlying principles and choices should not change significantly. *********************************************************************** ******* Motivation and scope *********************************************************************** ******* Our goal is to allow user code to create fully featured dtypes for a broad variety of uses, from physical units (such as meters) to domain- specific representations of geometric objects. :ref:`NEP 41 <NEP41>` describes a number of these new dtypes and their benefits. Any design supporting dtypes must consider: - How shape and dtype are determined when an array is created - How array elements are stored and accessed - The rules for casting dtypes to other dtypes In addition: - We want dtypes to comprise a class hierarchy open to new types and to subhierarchies, as motivated in :ref:`NEP 41 <NEP41>`. And to provide this, - We need to define a user API. All these are the subjects of this NEP. - The class hierarchy, its relation to the Python scalar types, and its important attributes are described in `nep42_DType class`_. - The functionality that will support dtype casting is described in `Casting`_. - The implementation of item access and storage, and the way shape and dtype are determined when creating an array, are described in :ref:`nep42_array_coercion`. - The functionality for users to define their own DTypes is described in `Public C-API`_. The API here and in NEP 43 is entirely on the C side. A Python-side version will be proposed in a future NEP. A future Python API is expected to be similar, but provide a more convenient API to reuse the functionality of existing DTypes. It could also provide shorthands to create structured DTypes similar to Python's `dataclasses <https://docs.python.org/3.8/library/dataclasses.html>`_. *********************************************************************** ******* Usage and impact *********************************************************************** ******* We believe the few structures in this section are sufficient to consolidate NumPy's present functionality and also to support complex user-defined DTypes. The rest of the NEP fills in details and provides support for the claim. Again, though Python is used for illustration, the implementation is a C API only; a future NEP will tackle the Python API. After implementing this NEP, creating a DType will be possible by implementing the following outlined DType base class, that is further described in `nep42_DType class`_: class DType(np.dtype): type : type # Python scalar type parametric : bool # (may be indicated by superclass) @property def canonical(self) -> bool: raise NotImplementedError def ensure_canonical(self : DType) -> DType: raise NotImplementedError For casting, a large part of the functionality is provided by the "methods" stored in ``_castingimpl`` @classmethod def common_dtype(cls : DTypeMeta, other : DTypeMeta) -> DTypeMeta: raise NotImplementedError def common_instance(self : DType, other : DType) -> DType: raise NotImplementedError # A mapping of "methods" each detailing how to cast to another DType # (further specified at the end of the section) _castingimpl = {} For array-coercion, also part of casting: def __dtype_setitem__(self, item_pointer, value): raise NotImplementedError def __dtype_getitem__(self, item_pointer, base_obj) -> object: raise NotImplementedError @classmethod def __discover_descr_from_pyobject__(cls, obj : object) -> DType: raise NotImplementedError # initially private: @classmethod def _known_scalar_type(cls, obj : object) -> bool: raise NotImplementedError Other elements of the casting implementation is the ``CastingImpl``: casting = Union["safe", "same_kind", "unsafe"] class CastingImpl: # Object describing and performing the cast casting : casting def resolve_descriptors(self, Tuple[DType] : input) -> (casting, Tuple[DType]): raise NotImplementedError # initially private: def _get_loop(...) -> lowlevel_C_loop: raise NotImplementedError which describes the casting from one DType to another. In NEP 43 this ``CastingImpl`` object is used unchanged to support universal functions.

1 1

Proposal: split numpy.distutils into it's own package
by Dustin Spicuzza 26 Oct '20

26 Oct '20

Hey all, [copied from https://github.com/numpy/numpy/issues/17620, as requested by the feature request guidelines] Cross-compiling scipy and other projects that depend on numpy's distutils is a huge pain right now, because to do it [in addition to lots of other details that you have to get right] you have to have both a native and cross-compiled version of numpy installed. It seems pretty unreasonable that I need a native version of numpy installed to compile scipy. One might ask, why is this needed? Well, scipy's setup.py uses numpy's distutils fork for the fortran support (I think). Because numpy.distutils is a subpackage of numpy, if you're working with a cross-compiled version of numpy, it eventually tries to import some .so that wasn't compiled for your host and everything dies -- thus you have to have a native numpy installed to allow numpy.distutils to import correctly. As far as I can tell, numpy's distutils fork is pure python and doesn't actually use anything else in numpy itself, and so is completely separable from numpy. If it were its own top-level package that scipy et al could use, then cross-compiling would be significantly less tricky. The mechanics of this would work like so: * contents of numpy.distutils would be moved to 'numpy_distutils' package * importing numpy.distutils would raise a deprecation warning and redirect to numpy_distutils, probably using import hook magic * scipy and other packages would now utilize numpy_distutils instead of numpy.distutils at build time Initially, I considered proposing creating a separate pypi package for numpy_distutils, but I expect that would be more trouble than it's worth. One could also propose creating a new PEP 517 build backend, or move to cmake, or other huge changes, but those also seem more trouble than they're worth. Thanks for your consideration. Dustin

5 10

help needed
by Amardeep Singh 24 Oct '20

24 Oct '20

Hi All I am a new joiner.Using macbook. Can someone please guide me how to debug numpy using clion? I am able to build not sure about debugging the c code. ProductName: Mac OS X ProductVersion: 10.15.7 BuildVersion: 19H2 thx

2 3

ENH: Changed repr of np.bool_ #17592 (requesting confirmation)
by Ganesh Kathiresan 23 Oct '20

23 Oct '20

Hi All, I have raised a PR <https://github.com/numpy/numpy/pull/17592>to change the np.bool_ string representation to avoid confusion with python's True and False as mentioned in this issue <https://github.com/numpy/numpy/issues/12950>. This change makes string representation of np.bool_ into numpy.True_ and numpy.False_ As Eric mentioned, I wanted your feedback on the change and if we can go ahead with this idea. Regards, Ganesh

1 0

NumPy Development Meeting Wednesday - Triage Focus
by Sebastian Berg 20 Oct '20

20 Oct '20

Hi all, Our bi-weekly triage-focused NumPy development meeting is today (Wednesday, October 21st) at 11 am Pacific Time (18:00 UTC). Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized, discussed, or reviewed. Best regards Sebastian

1 0

ANN: SciPy 1.5.3
by Tyler Reddy 19 Oct '20

19 Oct '20

Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.5.3, which is a bug fix release that includes Linux ARM64 wheels for the first time. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.5.3 One of a few ways to install this release with pip: pip install scipy==1.5.3 ========================== SciPy 1.5.3 Release Notes ========================== SciPy 1.5.3 is a bug-fix release with no new features compared to 1.5.2. In particular, Linux ARM64 wheels are now available and a compatibility issue with XCode 12 has been fixed. Authors ====== * Peter Bell * CJ Carey * Thomas Duvernay + * Gregory Lee * Eric Moore * odidev * Dima Pasechnik * Tyler Reddy * Simon Segerblom Rex + * Daniel B. Smith * Will Tirone + * Warren Weckesser A total of 12 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.5.3 ------------------------------ * `#9611 <https://github.com/scipy/scipy/issues/9611>`__: Overflow error with new way of p-value calculation in kendall... * `#10069 <https://github.com/scipy/scipy/issues/10069>`__: scipy.ndimage.watershed_ift regression in 1.0.0 * `#11260 <https://github.com/scipy/scipy/issues/11260>`__: BUG: DOP853 with complex data computes complex error norm, causing... * `#11479 <https://github.com/scipy/scipy/issues/11479>`__: RuntimeError: dictionary changed size during iteration on loading... * `#11972 <https://github.com/scipy/scipy/issues/11972>`__: BUG (solved): Error estimation in DOP853 ODE solver fails for... * `#12543 <https://github.com/scipy/scipy/issues/12543>`__: BUG: Picture rotated 180 degrees and rotated -180 degrees should... * `#12613 <https://github.com/scipy/scipy/issues/12613>`__: Travis X.4 and X.7 failures in master * `#12654 <https://github.com/scipy/scipy/issues/12654>`__: scipy.stats.combine_pvalues produces wrong results with method='mudholkar_george' * `#12819 <https://github.com/scipy/scipy/issues/12819>`__: BUG: Scipy Sparse slice indexing assignment Bug with zeros * `#12834 <https://github.com/scipy/scipy/issues/12834>`__: BUG: ValueError upon calling Scipy Interpolator objects * `#12836 <https://github.com/scipy/scipy/issues/12836>`__: ndimage.median can return incorrect values for integer inputs * `#12860 <https://github.com/scipy/scipy/issues/12860>`__: Build failure with Xcode 12 Pull requests for 1.5.3 ----------------------------- * `#12611 <https://github.com/scipy/scipy/pull/12611>`__: MAINT: prepare for SciPy 1.5.3 * `#12614 <https://github.com/scipy/scipy/pull/12614>`__: MAINT: prevent reverse broadcasting * `#12617 <https://github.com/scipy/scipy/pull/12617>`__: MAINT: optimize: Handle nonscalar size 1 arrays in fmin_slsqp... * `#12623 <https://github.com/scipy/scipy/pull/12623>`__: MAINT: stats: Loosen some test tolerances. * `#12638 <https://github.com/scipy/scipy/pull/12638>`__: CI, MAINT: pin pytest for Azure win * `#12668 <https://github.com/scipy/scipy/pull/12668>`__: BUG: Ensure factorial is not too large in mstats.kendalltau * `#12705 <https://github.com/scipy/scipy/pull/12705>`__: MAINT: \`openblas_support\` added sha256 hash * `#12706 <https://github.com/scipy/scipy/pull/12706>`__: BUG: fix incorrect 1d case of the fourier_ellipsoid filter * `#12721 <https://github.com/scipy/scipy/pull/12721>`__: BUG: use special.sindg in ndimage.rotate * `#12724 <https://github.com/scipy/scipy/pull/12724>`__: BUG: per #12654 adjusted mudholkar_george method to combine p... * `#12726 <https://github.com/scipy/scipy/pull/12726>`__: BUG: Fix DOP853 error norm for complex problems * `#12730 <https://github.com/scipy/scipy/pull/12730>`__: CI: pin xdist for Azure windows * `#12786 <https://github.com/scipy/scipy/pull/12786>`__: BUG: stats: Fix formula in the \`stats\` method of the ARGUS... * `#12795 <https://github.com/scipy/scipy/pull/12795>`__: CI: Pin setuptools on windows CI * `#12830 <https://github.com/scipy/scipy/pull/12830>`__: [BUG] sparse: Avoid using size attribute in LIL __setitem__ * `#12833 <https://github.com/scipy/scipy/pull/12833>`__: BUG: change list of globals items to list of a copy * `#12842 <https://github.com/scipy/scipy/pull/12842>`__: BUG: Use uint16 for cost in NI_WatershedElement * `#12845 <https://github.com/scipy/scipy/pull/12845>`__: BUG: avoid boolean or integer addition error in ndimage.measurements.median * `#12864 <https://github.com/scipy/scipy/pull/12864>`__: BLD: replace the #include of libqull_r.h with with this of qhull_ra.h... * `#12867 <https://github.com/scipy/scipy/pull/12867>`__: BUG: Fixes a ValueError yielded upon calling Scipy Interpolator... * `#12902 <https://github.com/scipy/scipy/pull/12902>`__: CI: Remove 'env' from pytest.ini * `#12913 <https://github.com/scipy/scipy/pull/12913>`__: MAINT: Ignore pytest's PytestConfigWarning Checksums ========= MD5 ~~~ fd75941594b22a322f63a27d53c1bdda scipy-1.5.3-cp36-cp36m-macosx_10_9_x86_64.whl 85d547117ae6d9fd447120c1768ff2d0 scipy-1.5.3-cp36-cp36m-manylinux1_i686.whl 8e8ca0d9123c4f4bf59f74ec474fc936 scipy-1.5.3-cp36-cp36m-manylinux1_x86_64.whl 855db93424cf97e5b4685c4cf74be346 scipy-1.5.3-cp36-cp36m-manylinux2014_aarch64.whl 20405ee4157858d33e38c62b873b6420 scipy-1.5.3-cp36-cp36m-win32.whl be3f3ba1018e7b1f3a30738aa502a4b5 scipy-1.5.3-cp36-cp36m-win_amd64.whl 12be163517b748a025cd399e6e9ce7df scipy-1.5.3-cp37-cp37m-macosx_10_9_x86_64.whl b2d761876d1f1d27e289b08f44034ede scipy-1.5.3-cp37-cp37m-manylinux1_i686.whl 1cd15a513ad76c64a4353a63eee6b3a8 scipy-1.5.3-cp37-cp37m-manylinux1_x86_64.whl 00bb4f63f4ee40869193c8e639da3274 scipy-1.5.3-cp37-cp37m-manylinux2014_aarch64.whl f783a76397dfe2a73752ac86f32fa474 scipy-1.5.3-cp37-cp37m-win32.whl 3c2927e6adf9b522f7f1745308b4d1f2 scipy-1.5.3-cp37-cp37m-win_amd64.whl faaa0cab95b6f352508120b8f628aaae scipy-1.5.3-cp38-cp38-macosx_10_9_x86_64.whl 77e8cb222c1868f01f048a444ef49260 scipy-1.5.3-cp38-cp38-manylinux1_i686.whl a2a035b15f78106090b9550f1383b40f scipy-1.5.3-cp38-cp38-manylinux1_x86_64.whl 86d52a963596a4cf6e1930ac5cf79a03 scipy-1.5.3-cp38-cp38-manylinux2014_aarch64.whl 8dd29aceb8dae5b5cc4f8100d8b35423 scipy-1.5.3-cp38-cp38-win32.whl 14f617e43a37827a29e8ee9ad97eda4b scipy-1.5.3-cp38-cp38-win_amd64.whl ecf5c58e4df1d257abf1634d51cb9205 scipy-1.5.3.tar.gz 61bcc473115587a66acfb42d3021479b scipy-1.5.3.tar.xz a9c549f19c661ab1b44e5b2cc707d4c1 scipy-1.5.3.zip SHA256 ~~~~~~ f574558f1b774864516f3c3fe072ebc90a29186f49b720f60ed339294b7f32ac scipy-1.5.3-cp36-cp36m-macosx_10_9_x86_64.whl e527c9221b6494bcd06a17f9f16874406b32121385f9ab353b8a9545be458f0b scipy-1.5.3-cp36-cp36m-manylinux1_i686.whl b9751b39c52a3fa59312bd2e1f40144ee26b51404db5d2f0d5259c511ff6f614 scipy-1.5.3-cp36-cp36m-manylinux1_x86_64.whl d5e3cc60868f396b78fc881d2c76460febccfe90f6d2f082b9952265c79a8788 scipy-1.5.3-cp36-cp36m-manylinux2014_aarch64.whl 1fee28b6641ecbff6e80fe7788e50f50c5576157d278fa40f36c851940eb0aff scipy-1.5.3-cp36-cp36m-win32.whl ffcbd331f1ffa82e22f1d408e93c37463c9a83088243158635baec61983aaacf scipy-1.5.3-cp36-cp36m-win_amd64.whl 07b083128beae040f1129bd8a82b01804f5e716a7fd2962c1053fa683433e4ab scipy-1.5.3-cp37-cp37m-macosx_10_9_x86_64.whl e2602f79c85924e4486f684aa9bbab74afff90606100db88d0785a0088be7edb scipy-1.5.3-cp37-cp37m-manylinux1_i686.whl aebb69bcdec209d874fc4b0c7ac36f509d50418a431c1422465fa34c2c0143ea scipy-1.5.3-cp37-cp37m-manylinux1_x86_64.whl bc0e63daf43bf052aefbbd6c5424bc03f629d115ece828e87303a0bcc04a37e4 scipy-1.5.3-cp37-cp37m-manylinux2014_aarch64.whl 8cc5c39ed287a8b52a5509cd6680af078a40b0e010e2657eca01ffbfec929468 scipy-1.5.3-cp37-cp37m-win32.whl 0edd67e8a00903aaf7a29c968555a2e27c5a69fea9d1dcfffda80614281a884f scipy-1.5.3-cp37-cp37m-win_amd64.whl 66ec29348444ed6e8a14c9adc2de65e74a8fc526dc2c770741725464488ede1f scipy-1.5.3-cp38-cp38-macosx_10_9_x86_64.whl 12fdcbfa56cac926a0a9364a30cbf4ad03c2c7b59f75b14234656a5e4fd52bf3 scipy-1.5.3-cp38-cp38-manylinux1_i686.whl a1a13858b10d41beb0413c4378462b43eafef88a1948d286cb357eadc0aec024 scipy-1.5.3-cp38-cp38-manylinux1_x86_64.whl 5163200ab14fd2b83aba8f0c4ddcc1fa982a43192867264ab0f4c8065fd10d17 scipy-1.5.3-cp38-cp38-manylinux2014_aarch64.whl 33e6a7439f43f37d4c1135bc95bcd490ffeac6ef4b374892c7005ce2c729cf4a scipy-1.5.3-cp38-cp38-win32.whl a3db1fe7c6cb29ca02b14c9141151ebafd11e06ffb6da8ecd330eee5c8283a8a scipy-1.5.3-cp38-cp38-win_amd64.whl ddae76784574cc4c172f3d5edd7308be16078dd3b977e8746860c76c195fa707 scipy-1.5.3.tar.gz cab92f8dab54a3be66525ea23e4f6568145abd1e94681cce19258a140f4de416 scipy-1.5.3.tar.xz 32dd070e203363e3a63bc184afb1d8e165c0a36a49b83b097642f78ef4da6077 scipy-1.5.3.zip

2 1

Attempting to wrap a Fortran-77 subroutine using f2py but I haven't been able to understand what is causing the error
by Samuel Dupree 19 Oct '20

19 Oct '20

I'm attempting to wrap a Fortran-77 source member using f2py. I'm running he Anaconda distribution for Python 3.7.6 on a Mac Pro (2019) under Mac OS X Catalina (ver. 10.15.6). The version of NumPy I'm running is 1.18.3. I've attached a copy of the Fortran source code to this note (see rkfn78.for). The command I'm using to wrap this code is f2py3 -c rkfn78.for --fcompiler=gfortran --f77flags="-c -O -Wall" -m rkfn78 The output I get is captured in the file rkfn78_build_output.txt. I don't understand the cause behind the error message I get, so any advice would be welcomed. Sam Dupree.

2 6

new function: broadcast_shapes
by Madhulika Jain Chambers 16 Oct '20

16 Oct '20

Hello all, I opened a PR to add a function which returns the broadcasted shape from a given set of shapes: https://github.com/numpy/numpy/pull/17535 As this is a proposed change to the API, I wanted to see if there was any feedback from the list. Thanks so much, Madhulika

4 3

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion October 2020