Mailman 3 November 2022 - NumPy-Discussion

Release of NumPy 1.23.5
by Charles R Harris 20 Nov '22

20 Nov '22

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.23.5. NumPy 1.23.5 is a maintenance release that fixes bugs discovered after the 1.23.4 release and keeps the build infrastructure current. The Python versions supported for this release are 3.8-3.11. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.23.5>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.23.5>. *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @DWesl - Aayush Agrawal + - Adam Knapp + - Charles Harris - Navpreet Singh + - Sebastian Berg - Tania Allard *Pull requests merged* A total of 10 pull requests were merged for this release. - #22489: TST, MAINT: Replace most setup with setup_method (also teardown) - #22490: MAINT, CI: Switch to cygwin/cygwin-install-action@v2 - #22494: TST: Make test_partial_iteration_cleanup robust but require leak... - #22592: MAINT: Ensure graceful handling of large header sizes - #22593: TYP: Spelling alignment for array flag literal - #22594: BUG: Fix bounds checking for ``random.logseries`` - #22595: DEV: Update GH actions and Dockerfile for Gitpod - #22596: CI: Only fetch in actions/checkout - #22597: BUG: Decrement ref count in gentype_reduce if allocated memory... - #22625: BUG: Histogramdd breaks on big arrays in Windows Cheers, Charles Harris

1 0

new time for community meetings
by Inessa Pawson 20 Nov '22

20 Nov '22

It’s that time of the year again when some of us have to adjust our clocks. The fun doesn’t stop there as we have to adjust our work schedules too. Let’s see if we can find a new time slot on Wednesday for bi-weekly NumPy community meetings that works for everyone: http://whenisgood.net/numpy-community-meeting. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 1

deprecating the function np.round_
by Inessa Pawson 20 Nov '22

20 Nov '22

In NumPy, there are several functions that allow even rounding to a given number of decimals: np.around, np.round, np.round_. The function np.round_ is a disrecommended backwards-compatibility alias of the first two. I propose deprecating np.round_ to avoid further user confusion. To provide feedback on the proposed, please leave your comments in the issue on GitHub: https://github.com/numpy/numpy/issues/22617. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

NEP 50: deprecating np.find_common_type
by Inessa Pawson 18 Nov '22

18 Nov '22

As part of the work outlined in NEP 50, the function np.find_common_type will be deprecated soon after the release of NumPy 1.24. To provide feedback on this decision, please respond in the Conversation section of the pull request: https://github.com/numpy/numpy/pull/22539. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

New methods exposing UFunc dtype resolution/promotion and internals
by Sebastian Berg 16 Nov '22

16 Nov '22

Hi all, I would like to expose more of the ufunc internals in the following PR: https://github.com/numpy/numpy/pull/22422/ There are three new proposed functions. I hope the first one can be generally useful while the last two are very specific (and thus underscored), but will hopefully become useful e.g. for Numba or numexpr. ufunc.resolve_dtypes(dtypes, *, signature=None, casting=None, reduction=False) ----------------------------------------------------------- Allows you to find out what dtypes NumPy's implementation will use without executing a ufunc. For the full docs, see: https://output.circle-artifacts.com/output/job/c8f72dd5-f8fb-448c-8fd8-d618… Example from the docs: >>> int32 = np.dtype("int32") >>> float32 = np.dtype("float32") The typical ufunc call does not pass an output dtype. `np.add` has two inputs and one output, so leave the output as ``None`` (not provided): >>> np.add.resolve_dtypes((int32, float32, None)) (dtype('float64'), dtype('float64'), dtype('float64')) The loop found uses "float64" for all operands (including the output), the first input would be cast. ``resolve_dtypes`` supports "weak" handling for Python scalars by passing ``int``, ``float``, or ``complex``: >>> np.add.resolve_dtypes((float32, float, None)) (dtype('float32'), dtype('float32'), dtype('float32')) Where the Python ``float`` behaves samilar to a Python value ``0.0`` in a ufunc call. (See :ref:`NEP 50 <NEP50>` for details.) ufunc._resolve_dtypes_and_context(dtypes, *, signature=None, casting=None, reduction=False) ------------------------------------------------------------- Identical to the above, but it additionally returns a "call_info" which allows access to the actual ufunc implmentation. ufunc._get_loop(call_info, /, *, fixed_strides=None) ---------------------------------------------------- Second function that is passed the `call_info` from the previous one. Both would normally be called (this is because it is the way NumPy must do it internally and allows most flexibility). After doing both calls, `call_info` can be used from C to directly access the C implementation. Flux in the C-API are expected (for now). But for example Numba already releases new versions when NumPy releases a new version. Cheers, Sebastian

1 1

help wanted to review Portuguese translation
by Inessa Pawson 15 Nov '22

15 Nov '22

Angélica Cardozo took initiative to translate into Portuguese the subtitles for the video “Find your way in the NumPy codebase” ( https://www.youtube.com/watch?v=mTWpBf1zewc) posted on the official NumPy YouTube channel (https://www.youtube.com/c/NumPy_team). We are currently looking for a contributor who could review the translation. If you are available, please respond to this message in a thread. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

A Strange Phenomenon with respect to Multi-threading
by 腾刘 15 Nov '22

15 Nov '22

Hello~, I'm here again. Recently I came up with a really weird and confusing phenomenon. I used to dig deep into Numpy's C code to find out how np.add(a, b) is implemented and I thought I've figured it out. As far as I know, there is no multi-threading in Numpy for addition, instead, we use SIMD to speed it up. However, I found that for some certain situation with certain version of Numpy, there is multi-threading. Here is the screenshot (where we can see multi-threading clearly): [image: image.png] The environment is: Python 3.6.5 :: Anaconda, Inc. Numpy 1.14.3 np.show_config(): mkl_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/home/SIXIE/tliu/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/SIXIE/tliu/anaconda3/include'] blas_mkl_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/home/SIXIE/tliu/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/SIXIE/tliu/anaconda3/include'] blas_opt_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/home/SIXIE/tliu/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/SIXIE/tliu/anaconda3/include'] lapack_mkl_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/home/SIXIE/tliu/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/SIXIE/tliu/anaconda3/include'] lapack_opt_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/home/SIXIE/tliu/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/SIXIE/tliu/anaconda3/include'] However, when I try to repeat the multi-threading with my built-from-source Numpy, it failed (the cpu percentage is at most 100%): [image: image.png] And from the screenshot we can see that we also have mkl_rt and pthread when building the Numpy since I added it to *site.cfg*. The complete config info is: blas_armpl_info: NOT AVAILABLE blas_mkl_info: libraries = ['mkl_rt', 'pthread', 'mkl_rt'] library_dirs = ['/home/tliu/anaconda3/envs/numpy-dev/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/tliu/anaconda3/envs/numpy-dev/include'] blas_opt_info: libraries = ['mkl_rt', 'pthread', 'mkl_rt'] library_dirs = ['/home/tliu/anaconda3/envs/numpy-dev/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/tliu/anaconda3/envs/numpy-dev/include'] lapack_armpl_info: NOT AVAILABLE lapack_mkl_info: libraries = ['mkl_rt', 'pthread', 'mkl_rt'] library_dirs = ['/home/tliu/anaconda3/envs/numpy-dev/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/tliu/anaconda3/envs/numpy-dev/include'] lapack_opt_info: libraries = ['mkl_rt', 'pthread', 'mkl_rt'] library_dirs = ['/home/tliu/anaconda3/envs/numpy-dev/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/home/tliu/anaconda3/envs/numpy-dev/include'] Supported SIMD extensions in this NumPy install: baseline = SSE,SSE2,SSE3 found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2 not found = AVX512F,AVX512CD,AVX512_KNL,AVX512_KNM,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL I want to repeat multi-threading because I really want to know in which part of C code Numpy dispatches this computation to an outer library which utilize multi-threading (Numpy itself doesn't have multi-threading, if we see multi-threads, it must be created by outer library like BLAS, mkl, etc. But we know BLAS doesn't have functions for adding two vectors, so there must be mkl. BLAS is for matrix multiplication). But with pip-installed Numpy*, *I can't debug with it; I could only use *gdb *with my own built-from-source Numpy. I must miss something so that multi-threading doesn't recur. Any instructions or suggestions? Thank to all you guys in advance~

2 2

plan for moving to Meson
by Ralf Gommers 12 Nov '22

12 Nov '22

Hi all, With distutils now removed from the stdlib in the Python 3.12 release cycle, the clock is ticking a bit for dealing with our build system situation. With SciPy's move to Meson now basically complete - there are always loose ends & improvements, but the 1.9 releases have gone well - it's time to look at NumPy doing the same thing. Scikit-image has also merged support for Meson, and Pandas is about to. So all the generic & Python packaging related things we need should be in place. NumPy does have some hairy stuff of course that those other projects don't have (lots of config checks and platform-specific behavior, extensive SIMD support). It shouldn't be too difficult to get a baseline build - Linux/macOS with baseline SIMD flags - working, but there'll be a long tail of things that are hard to test or will need to be upstreamed to Meson. Here is a tracking issue where I wrote up a plan for how to approach the transition: https://github.com/numpy/numpy/issues/22546. Linked from there is a GitHub project board that we plan to use to divide up the work. And there is a `meson` branch in the repo with a start to the implementation. NumPy should default to building with Meson in the 1.25.0 release, which should also still build with `numpy.distutils`. And `numpy.distutils` will continue to be shipped for Python <3.12 for a while after that, until we determine that it's no longer needed. If anyone has ideas or concerns regarding the current plan, it'd be great to hear them - here or on the issue. Cheers, Ralf

4 7

Re: New feature: binary (arbitrary base) rounding
by Stefano Miccoli 12 Nov '22

12 Nov '22

On 8 Nov 2022, at 15:32, numpy-discussion-request(a)python.org<mailto:numpy-discussion-request@python.org> wrote: Thanks for the proposal. I don't have much of an opinion on this and right now I am mainly wondering whether there is prior art which can inform us that this is relatively widely useful? Base-2 (bit) rounding is implemented in Numcodecs <https://numcodecs.readthedocs.io/en/stable/bitround.html#module-numcodecs.b…> in the context of data compression. As pointed out by M. Klöwer et al. in <https://doi.org/10.1038/s43588-021-00156-2> "Rounding bits without real information to zero facilitates lossless compression algorithms and encodes the uncertainty within the data itself." I'm not an expert, but I never encountered rounding floating point numbers in bases different from 2 and 10. Stefano

5 9

New feature: binary (arbitrary base) rounding
by Oscar Gustafsson 10 Nov '22

10 Nov '22

Hi all, I hope this is the correct way to propose a new feature. https://github.com/numpy/numpy/issues/22522 Currently, the around-function supports rounding to a given number of decimal digits. It is often quite convenient to be able to round to a given number of binary digits to mimic fixed-point representations. Currently this can be readily achieved using e.g. fractional_bits = 5 scale = 2**fractional_bits x_round = np.around(x*scale)/scale However, it would be more convenient (and probably faster) to provide dedicated support for it. I see a few different ways to obtain this: 1. Provide a separate function (binaryround?) 2. Provide a base argument to around which defaults to 10. 3. Provide a quant(ization) function where the argument is the step-size. (For completeness, one may think of having multiple quantization modes, not just rounding) Any opinions? BR Oscar Gustafsson

3 3