Mailman 3 August 2019 - NumPy-Discussion

NumPy 1.16.5 released
by Charles R Harris 28 Aug '19

28 Aug '19

Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.16.5 has been released. This release fixes bugs reported against the 1.16.4 release and backports several enhancements from master that seem appropriate for the LTS release series that is the last to support Python 2.7. Downstream developers should use Cython >= 0.29.2 and OpenBLAS >= 3.7 to avoid wrong results on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.16.5/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.16.5>. *Contributors* A total of 18 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Alexander Shadchin - Allan Haldane - Bruce Merry + - Charles Harris - Colin Snyder + - Dan Allan + - Emile + - Eric Wieser - Grey Baker + - Maksim Shabunin + - Marten van Kerkwijk - Matti Picus - Peter Andreas Entschev + - Ralf Gommers - Richard Harris + - Sebastian Berg - Sergei Lebedev + - Stephan Hoyer *Pull requests merged* A total of 23 pull requests were merged for this release. - gh-13742: ENH: Add project URLs to setup.py - gh-13823: TEST, ENH: fix tests and ctypes code for PyPy - gh-13845: BUG: use npy_intp instead of int for indexing array - gh-13867: TST: Ignore DeprecationWarning during nose imports - gh-13905: BUG: Fix use-after-free in boolean indexing - gh-13933: MAINT/BUG/DOC: Fix errors in _add_newdocs - gh-13984: BUG: fix byte order reversal for datetime64[ns] - gh-13994: MAINT,BUG: Use nbytes to also catch empty descr during allocation - gh-14042: BUG: np.array cleared errors occured in PyMemoryView_FromObject - gh-14043: BUG: Fixes for Undefined Behavior Sanitizer (UBSan) errors. - gh-14044: BUG: ensure that casting to/from structured is properly checked. - gh-14045: MAINT: fix histogram*d dispatchers - gh-14046: BUG: further fixup to histogram2d dispatcher. - gh-14052: BUG: Replace contextlib.suppress for Python 2.7 - gh-14056: BUG: fix compilation of 3rd party modules with Py_LIMITED_API... - gh-14057: BUG: Fix memory leak in dtype from dict contructor - gh-14058: DOC: Document array_function at a higher level. - gh-14084: BUG, DOC: add new recfunctions to `__all__` - gh-14162: BUG: Remove stray print that causes a SystemError on python 3.7 - gh-14297: TST: Pin pytest version to 5.0.1. - gh-14322: ENH: Enable huge pages in all Linux builds - gh-14346: BUG: fix behavior of structured_to_unstructured on non-trivial... - gh-14382: REL: Prepare for the NumPy 1.16.5 release. Cheers, Charles Harris

1 0

NumPy logo refresh
by Ralf Gommers 28 Aug '19

28 Aug '19

Hi all, At https://github.com/numpy/numpy.org/issues/37 we have an update to the NumPy logo, changing the colors for more contrast / a more modern look. Please have a look! What would also be helpful is to know who made this logo, and with what tool. We'd like to produce a new SVG, but doing that from the old SVG turned out to be problematic. So for now it's PNG only. Cheers, Ralf

1 0

Calling BLAS functions from Python
by Jens Jørgen Mortensen 27 Aug '19

27 Aug '19

Hi! I'm trying to use dgemm, zgemm and friends from scipy.linalg.blas to multiply matrices efficiently. As an example, I'd like to do: c += a.dot(b) using whatever BLAS scipy is linked to and I want to avoid copies of large matrices. This works the way I want it: >>> import numpy as np >>> from scipy.linalg.blas import dgemm >>> a = np.ones((2, 3), order='F') >>> b = np.ones((3, 4), order='F') >>> c = np.zeros((2, 4), order='F') >>> dgemm(1.0, a, b, 1.0, c, 0, 0, 1) array([[3., 3., 3., 3.], [3., 3., 3., 3.]]) >>> print(c) [[3. 3. 3. 3.] [3. 3. 3. 3.]] but if c is not contiguous, then c is not overwritten: >>> c = np.zeros((7, 4), order='F')[:2, :] >>> dgemm(1.0, a, b, 1.0, c, 0, 0, 1) array([[3., 3., 3., 3.], [3., 3., 3., 3.]]) >>> print(c) [[0. 0. 0. 0.] [0. 0. 0. 0.]] Which is also what the docs say, but I think the raw BLAS function dgemm could do the update of c in-place by setting LDC=7. See here: http://www.netlib.org/lapack/explore-html/d7/d2b/dgemm_8f.html Is there a way to call the raw BLAS function from Python? I found this capsule thing, but I don't know if there is a way to call that (maybe using ctypes): >>> from scipy.linalg import cython_blas >>> cython_blas.__pyx_capi__['dgemm'] <capsule object "void (char *, char *, int *, int *, int *, __pyx_t_5scipy_6linalg_11cython_blas_d *, __pyx_t_5scipy_6linalg_11cython_blas_d *, int *, __pyx_t_5scipy_6linalg_11cython_blas_d *, int *, __pyx_t_5scipy_6linalg_11cython_blas_d *, __pyx_t_5scipy_6linalg_11cython_blas_d *, int *)" at 0x7f06fe1d2ba0> Best, Jens Jørgen

3 4

NumPy Community Meeting Wednesday, Aug. 28
by Sebastian Berg 27 Aug '19

27 Aug '19

Hi all, There will be a NumPy Community meeting Wednesday August 28 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

PyData/Sparse 0.8.0
by Hameer Abbasi 27 Aug '19

27 Aug '19

(Apologies for the cross-posting) Hello, everyone! PyData/Sparse 0.8.0 has been released. It is available on PyPI (and a source distribution and a universal binary) and conda-forge. Changelog with credits: https://sparse.pydata.org/en/0.8.0/changelog.html Documentation: https://sparse.pydata.org/en/0.8.0/ Source: https://github.com/pydata/sparse/releases/tag/0.8.0

1 0

NumPy 1.17.1 released
by Charles R Harris 27 Aug '19

27 Aug '19

Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.17.1 has been released. This release contains a number of fixes for bugs reported against NumPy 1.17.0 along with a few documentation and build improvements. The Python versions supported are 3.5-3.7, note that Python 2.7 has been dropped. Python 3.8b3 should work with the released source packages, but there are no future guarantees. Downstream developers should use Cython >= 0.29.13 for Python 3.8 support and OpenBLAS >= 3.7 to avoid wrong results on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.17.1/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.17.1>. *Contributors* A total of 17 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Alexander Jung + - Allan Haldane - Charles Harris - Eric Wieser - Giuseppe Cuccu + - Hiroyuki V. Yamazaki - Jérémie du Boisberranger - Kmol Yuan + - Matti Picus - Max Bolingbroke + - Maxwell Aladago + - Oleksandr Pavlyk - Peter Andreas Entschev - Sergei Lebedev - Seth Troisi + - Vladimir Pershin + - Warren Weckesser *Pull requests merged* A total of 24 pull requests were merged for this release. - gh-14156: TST: Allow fuss in testing strided/non-strided exp/log loops - gh-14157: BUG: avx2_scalef_ps must be static - gh-14158: BUG: Remove stray print that causes a SystemError on python 3.7. - gh-14159: BUG: Fix DeprecationWarning in python 3.8. - gh-14160: BLD: Add missing gcd/lcm definitions to npy_math.h - gh-14161: DOC, BUILD: cleanups and fix (again) 'build dist' - gh-14166: TST: Add 3.8-dev to travisCI testing. - gh-14194: BUG: Remove the broken clip wrapper (Backport) - gh-14198: DOC: Fix hermitian argument docs in svd. - gh-14199: MAINT: Workaround for Intel compiler bug leading to failing test - gh-14200: TST: Clean up of test_pocketfft.py - gh-14201: BUG: Make advanced indexing result on read-only subclass writeable... - gh-14236: BUG: Fixed default BitGenerator name - gh-14237: ENH: add c-imported modules for freeze analysis in np.random - gh-14296: TST: Pin pytest version to 5.0.1 - gh-14301: BUG: Fix leak in the f2py-generated module init and `PyMem_Del`... - gh-14302: BUG: Fix formatting error in exception message - gh-14307: MAINT: random: Match type of SeedSequence.pool_size to DEFAULT_POOL_SIZE. - gh-14308: BUG: Fix numpy.random bug in platform detection - gh-14309: ENH: Enable huge pages in all Linux builds - gh-14330: BUG: Fix segfault in `random.permutation(x)` when x is a string. - gh-14338: BUG: don't fail when lexsorting some empty arrays (#14228) - gh-14339: BUG: Fix misuse of .names and .fields in various places (backport... - gh-14345: BUG: fix behavior of structured_to_unstructured on non-trivial... - gh-14350: REL: Prepare 1.17.1 release Cheers, Charles Harris

1 0

first Season of Docs blog post
by Ralf Gommers 26 Aug '19

26 Aug '19

Hi all, Anne has written a very nice blog post about the start of the Season of Docs projects: https://towardsdatascience.com/what-do-you-want-to-see-in-the-numpy-docs-de… It even includes a pineapple with sunglasses and a party hat, so a good read:) Cheers, Ralf

1 0

NumPy cross-compilation on macOS host
by Jakub Piotr Cłapa 24 Aug '19

24 Aug '19

Hi, I spent some time trying to cross-compile NumPy for Linux ARM64 on a macOS host as part of an OpenWRT compilation [1]. It was not so bad but certainly not trivial and I was wondering if we could use my experience to help streamline the process. I encountered two challenges: 1. Python distutils did not allow me to provide a proper cross-ranlib program path. I opened an issue for this on the Python bug tracker [2]. 2. numpy distutils scripts check sys.platform in many places. sys.platform returns the platform on which compilation occurs (darwin in my case) which in case of cross-compilation is not the one on which the code will be run (linux). This of course results in invalid compiler and linker flags. I made a quick'n'dirty fix [3] and got numpy to compile and run correctly (but I have yet to run the full test suite). Of course this is not a proper way to solve this so I was hoping to get some feedback on the preferred/best way. I did not find any info about an official way to do this with distutils. One idea (from @commodo on GitHub) would be to just change my hardcoded string ("Linux") to something like this: target_platform = os.getenviron(“TARGET_PLATFORM”, sys.platform) It could also be further refactored by moving it to a single place but while certainly more elegant and "proper" it may ultimately make it less readable and harder to follow. I am also unsure where would be the best place to put it and how to pass it around to other modules. [1]: https://github.com/openwrt/packages/pull/9797 [2]: https://bugs.python.org/issue37916 [3]: https://github.com/openwrt/packages/pull/9797/commits/e701e122fb9c78d5cecff…

2 1

NumFOCUS Summit and 4 Nov sprint
by Ralf Gommers 22 Aug '19

22 Aug '19

Hi all, On 2-3 November the NumFOCUS Summit will be held in NYC, and as a sponsored project NumPy can send two people to it (with NumFOCUS covering travel and hotel costs). To choose those two people we asked the members of the Steering Council first. It seems that we may have an open slot left, so we're asking anyone else who is a part of the NumPy project if they're interested in going. Details from the invitation: We will kick things off Friday, November 1st with a Reception Dinner at Havana Central <https://numfocus-dot-yamm-track.appspot.com/Redirect?ukey=1my4O8gi_yIWFTKq0…> . The Summit program (Nov 2-3) will be aimed at addressing both the challenge of securing funding and navigating the challenges of project governance and leadership. You can view the schedule in more detail here <https://numfocus-dot-yamm-track.appspot.com/Redirect?ukey=1my4O8gi_yIWFTKq0…> . If you're interested, please say so either here or off-list. Also note that we'll likely organize a sprint on 4 Nov, so if there are people who would be interested in that (I imagine there's quite a few people on this list from near NYC?), that'd be good to know as well. Cheers, Ralf

1 0

Releasing the GIL in ufuncs dealing with object arrays
by Joris Van den Bossche 21 Aug '19

21 Aug '19

Hi all, PyGEOS (https://github.com/caspervdw/pygeos) is an experimental package implementing a set of numpy ufuncs to provide vectorized geometry functionality (wrapping the C++ GEOS library). The way it does this is by implementing a Python extension type (pygeos.Geometry) that wraps an actual GEOSGeometry object by storing a pointer to it in the PyObject struct of the extension type. This way, we can store those objects in an object dtype array in numpy, but still access the pointer in the ufunc inner loop without needing the python interpreter. The single threaded performance of the ufuncs with the approach above is very good. There doesn't seem to be an overhead of using the object array approach. However, as far as I can find in the docs, the GIL is only released in ufuncs for non-object dtypes. So the question here is: is there a way to let numpy release the GIL in such a case nonetheless? Although the array holds python objects, the ufunc inner loop only accesses a static attribute of the object struct, not needing any explicit Python interaction. Best, Joris

2 3