Mailman 3 November 2022 - NumPy-Discussion

next NumPy community meeting
by Inessa Pawson 10 Sep '23

10 Sep '23

The next NumPy community meeting will be held this Wednesday, August 17th at 18:00 (6 pm) UTC. Join us via Zoom: https://us06web.zoom.us/j/83278611437?pwd=ekhoLzlHRjdWc0NOY2FQM0NPemdkZz09 Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both For the notes from the previous meeting, visit: https://github.com/numpy/archive/blob/main/community_meetings/community-202… Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 28

reminder: put an upper bound on setuptools if you use numpy.distutils!
by Ralf Gommers 31 Aug '23

31 Aug '23

Hi all, After setuptools 65.0 was released a few days ago, all users of numpy.distutils had their builds broken. This is already fixed in setuptools 65.0.2 because the breakage was particularly bad. However, the next breakage may not be fixed anymore (and more breakages *are* expected). So this is a good time to remind you all that you should put an upper bound on the setuptools version you allow in the releases of your package - to the last version that is known to work with your package. Our official stance here is that setuptools versions >=60 are not supported - see the "deprecating numpy.distutils" thread: https://mail.python.org/archives/list/numpy-discussion@python.org/message/P…. Newer versions may work for you, depending on what features you use. They don't for NumPy and for SciPy; both projects pin to 59.X to avoid problems. For the recent issue with setuptools 65.0.0, see https://github.com/numpy/numpy/issues/22135. We have also made the warnings about this topic in our docs more explicit, see https://github.com/numpy/numpy/pull/22154. Cheers, Ralf

2 2

Adding bit_count ufunc
by Doug Turnbull 13 Aug '23

13 Aug '23

👋Long time numpy user, and big fan of all your work TL; DR - there's a `bit_count` method on numpy scalars, and Python ints, I'm advocating for a `ufunc` `bit_count` as has been implemented in this PR: https://github.com/numpy/numpy/pull/21429 A number of people have requested this as a numpy feature and there's a lot of great discussion at this issue https://github.com/numpy/numpy/issues/16325 A bit count comes up in certain similarity situations (like hamming distance). This involves an xor with numpy arrays and a bit count, where the bit count is currently the bottleneck by a few orders of magnitude in my local benchmarking. Currently there's a number of not particularly fast, workarounds for doing this with numpy arrays, like the bit-twiddling solutions here https://stackoverflow.com/a/68943135/8123 https://stackoverflow.com/a/109025/8123 So I'd love to be able to continue the work of bit_count -> numpy array ufunc to get performance gains at the array level 🙏

5 7

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Representation of NumPy scalars
by Sebastian Berg 09 Jun '23

09 Jun '23

TL;DR: NumPy scalars representation is e.g. `34.3` instead of `float32(34.3)`. So the representation is missing the type information. What are your thoughts on changing that? Hi all, I am thinking about the next steps for NEP 50 (The NEP wants to fix the NumPy promotion rules, especially with respect to scalars): https://numpy.org/neps/nep-0050-scalar-promotion.html In relation to that, there was one point that Stéfan brought up previously. The NumPy scalars (representation) currently print as numbers: >>> np.float32(34.3) 34.3 >>> np.uint8(5) 5 That can already be confusing now. However, it gets more problematic if NEP 50 is introduced since the behavior between a Python `34.3` and `np.float32(34.3)` would differ more than it does now (please refer to the NEP). The change would be that we should print as: float64(34.3) (or similar?) This Email is mainly to ask for any feedback or concern on such a change. I suspect we may have to write a very brief NEP about it. If there is little concern, maybe we could move forward such a change promptly. Otherwise it could be moved forward together with NEP 50 and take effect in a "major" release [1]. Cheers, Sebastian [1] Note that for me, even a major release would hopefully not affect the majority of users or be very disruptive.

11 21

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Introducing Arm Optimized Routines
by Chris Sidebottom 08 Feb '23

08 Feb '23

Hello, Here at Arm, we've been investigating how we can improve performance on AArch64. One way in which we can improve performance is by integrating some existing optimized routines (https://github.com/ARM-software/optimized-routines), similar to the SVML methods for AVX512 that are currently included as a git submodule. Our intent is to include the optimized routines repository as an additional submodule which we can then use to provide routines on AArch64 for ASIMD, SVE and beyond. Currently, we're targeting 4-ULP as this aligns with libmvec (https://sourceware.org/glibc/wiki/libmvec) and the SVML integration (https://github.com/numpy/numpy/pull/19478). This is alongside adding sufficient error handling to pass the Numpy test suite, meeting the test requirements highlighted in the SVML integration (https://github.com/numpy/numpy/pull/19478#issuecomment-893001722). We've already started curating the necessary functions, let us know if you have any feedback. Cheers, Chris IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

3 4

status of long double support and what to do about it
by Ralf Gommers 14 Dec '22

14 Dec '22

Hi all, We have to do something about long double support. This is something I wanted to propose a long time ago already, and moving build systems has resurfaced the pain yet again. This is not a full proposal yet, but the start of a discussion and gradual plan of attack. The problem ----------- The main problem is that long double support is *extremely* painful to maintain, probably far more than justified. I could write a very long story about that, but instead I'll just illustrate with some of the key points: (1) `long double` is the main reason why we're having such a hard time with building wheels on Windows, for SciPy in particular. This is because MSVC makes long double 64-bit, and Mingw-w64 defaults to 80-bit. So we have to deal with Mingw-w64 toolchains, proposed compiler patches, etc. This alone has been a massive time sink. A couple of threads: https://github.com/numpy/numpy/issues/20348 https://discuss.scientific-python.org/t/releasing-or-not-32-bit-windows-whe… The first issue linked above is one of the key ones, with a lot of detail about the fundamental problems with `long double`. The Scientific Python thread focused more on Fortran, however that Fortran + Windows problem is at least partly the fault of `long double`. And Fortran may be rejuvenated with LFortran and fortran-lang.org; `long double` is a dead end. (2) `long double` is not a well-defined format. We support 9 specific binary representations, and have a ton of code floating around to check for those, and manually fiddle with individual bits in long double numbers. Part of that is the immediate pain point for me right now: in the configure stage of the build we consume object files produced by the compiler and parse them, matching some known bit patterns. This check is so weird that it's the only one that I cannot implement in Meson (short of porting the hundreds of lines of Python code for it to C), see https://github.com/mesonbuild/meson/issues/11068 for details. To get an idea of the complexity here: https://github.com/numpy/numpy/blob/9e144f7c1598221510d49d8c6b79c66dc000edf… https://github.com/numpy/numpy/blob/9e144f7c1598221510d49d8c6b79c66dc000edf… https://github.com/numpy/numpy/blob/main/numpy/core/src/npymath/npy_math_co… https://github.com/numpy/numpy/blob/main/numpy/core/src/multiarray/dragon4.… Typically `long double` has multiple branches, and requires more code than float/double. (3) We spend a lot of time dealing with issues and PR to keep `long double` working, as well as dealing with hard-to-diagnose build issues. Which sometimes even stops people from building/contributing, especially on Windows. Some recent examples: https://github.com/numpy/numpy/pull/20360 https://github.com/numpy/numpy/pull/18536 https://github.com/numpy/numpy/pull/21813 https://github.com/numpy/numpy/pull/22405 https://github.com/numpy/numpy/pull/19950 https://github.com/numpy/numpy/pull/18330/commits/aa9fd3c7cb https://github.com/scipy/scipy/issues/16769 https://github.com/numpy/numpy/issues/14574 (4) `long double` isn't all that useful. On both Windows and macOS `long double` is 64-bit, which means it is just a poor alias to `double`. So it does literally nothing for the majority of our users, except confuse them and take up extra memory. On Linux, `long double` is 80-bit precision, which means it doesn't do all that much there either, just a modest bump in precision. Let me also note that it's not just the user-visible dtypes that we have to consider; long double types are also baked into the libnpymath static library that we ship with NumPy. That's a thing we have to do something about anyway (shipping static libraries is not the best idea, see https://github.com/numpy/numpy/issues/20880). We just have to make sure to not forget about it when thinking about solutions here. Potential solutions ------------------- (A) The ideal solution would be to have a proper, portable quad-precision (128 bits of precision) dtype. It's now possible to write that externally, after all the work that Sebastian and others have put into the dtype infrastructure. The dtype itself already exists ( https://github.com/peytondmurray/quaddtype, maybe there are more implementations floating around). It just need the people who have an actual need for it to drive that. It's still a significant amount of work, so I'll not go into this one more right now. (B) Full deprecation and removal of all `long double` support from NumPy (and SciPy), irrespective of whether the quad dtype comes to life. Given the current state, I'm personally convinced that that is easily justified. However, I know some folks are going to be hesitant, given that we don't know how many remaining users we have or what use cases they have. So let's see if we can find more granular solutions (note: these are ideas, not all fully researched solutions that we can pick from and are guaranteed to work out well). (C) Only support `long double` where it has proper compiler support (C99/C++11), so using it "just works". And remove all the support for old formats and accessing bit representations directly. This also implies making some optional functions mandatory. For example, the issue I ran into today showed up at runtime in a fallback path for systems that don't have `strtold_l`. We don't test such fallback paths in CI, so they're going to be fragile anyway. (D) Add a build mode with a command-line flag, where we typedef `long double` to `double`. I'll note that we already touched on that once (see https://github.com/numpy/numpy/issues/20348); I'm not sure though if it's fundamentally not a good idea or that we just didn't do it intentionally enough. Next steps ---------- First, it would be great to hear from folks who have use cases for long double support in NumPy. So far we have very little knowledge of that, we only see the problems and work it causes us. Second, I'm going to have to add support for (C) or (D) temporarily to the Meson build anyway, as we run into things. It can be worked around if we really have to by implementing support for long double format detection in Meson, or by rewriting all the detection logic so it's all-in-one in C. But that takes a significant amount of time. Third, let's figure out which way we'd like to go. Do you see alternative solutions? Or like any of the ones I listed more than others? Are you willing to jump in and work on a quad dtype? Cheers, Ralf

10 15

Documentation Team meeting - Monday June 8th
by Melissa Mendonça 04 Dec '22

04 Dec '22

Hi all! A reminder that on Monday, June 8, we have another documentation team meeting at 3PM UTC**. If you wish to join on Zoom, you need to use this link https://zoom.us/j/420005230 Here's the permanent hackmd document with the meeting notes: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg <https://www.google.com/url?q=https%3A%2F%2Fhackmd.io%2FoB_boakvRqKR-_2jRV-Q…> Hope to see you around (especially if you want to introduce yourself or discuss ideas for Google Season of Docs). ** You can click this link to get the correct time at your timezone: https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Documentati… - Melissa

6 74

Create `np.exceptions` for new exceptions in NumPy?
by Sebastian Berg 02 Dec '22

02 Dec '22

Hi all, I want to add a new exception or two. It is a longer story, that you can find at the bottom :). Lets create a namespace for custom errors! I don't want to propose new exceptions that just get dumped in to the main namespace, so why not make one like `errors` in pandas or `exceptions` in scikit-learn. I would suggest introducing `np.exceptions`. We already have custom errors and warnings: * AxisError * TooHardError (used by `np.shares_memory()`) * ComplexWarning * RankWarning * VisibleDeprecationWarning * ModuleDeprecationWarning (not sure what this is) And a few private ones around ufunc "no loops" or casting failures (for delayed printing and formatting convenience). No need to move them all now, but maybe it is time to create a module where we put them all? With the intention that when the stars align, we will deprecate their main namespace aliases (either soon or in years). Beyond the error I just wanted, there were things brought up before, such as either `BroadcastError` or `ShapeMismatch`. Adding the namespace would make them more discoverable and just remove an annoying road-block for adding new ones. I will argue that the cost is practically zero. I do not want custom exceptions for too many things, but there are probably good reasons to have more than we do have right now, and even the ones we have seem enough for a namespace. Cheers, Sebastian The long story is that following one of those many threads, I decided that it looks worthwhile to introduce a new error class: InvalidPromotion I would want to use this for any/most promotion related failures. That means: * `np.result_type` or `np.promote_types` will give this if there is no valid way to promote * UFuncs will either give this error when there is no implementation or use it to raise a reliable error for "operation not defined for the inputs". [0] This would inherit from `TypeError` "of course". The why is a ball of yarn, that includes having a better shot at *finally* getting rid of the annoying comparison deprecation/future warning [1], eventually allowing more informative promotion errors, and that it might just be useful. Cheers, Sebastian [0] I first thought we should use the same error, but you can argue that `InvalidPromotion` doesn't include "this ufunc only works for floating point values". And yes, "no loop" can also mean "not implemented", but that may be need to be distinguished explicitly if really needed. [1] e.g. `np.array(["asdf"]) == 0`

4 4