Mailman 3 May 2024 - NumPy-Discussion

Adding bfill() to numpy.
by Raquel Braunschweig 21 May '24

21 May '24

Hello everyone, My colleague and I will be opening a Pull Request (PR) about adding bfill() (backward fill) function to NumPy. This function is designed to fill NaN values in an array by propagating the next valid observation backward along a specified axis. We believe this addition will be highly useful for data preprocessing and manipulation tasks. Here are some key points regarding our proposed implementation: Function Explanation: The bfill() function identifies NaN values in an array and replaces them by copying the next valid value in the array backwards. Users can specify the axis along which the filling should be performed, providing flexibility for different data structures. Use Cases: This function is particularly beneficial in time series analysis, data cleaning, and preparing datasets for machine learning models. We are looking forward to your feedback and suggestions. Thank you for your attention and we appreciate your support. Best regards, Raquel and Gonçalo

4 4

cp313 wheels on scientific-python-nightly-wheels/numpy
by Andrew Nelson 20 May '24

20 May '24

Hi all, in case you weren't aware there are now cp313 numpy nightly wheels available on https://anaconda.org/scientific-python-nightly-wheels/numpy. This should help with downstream testing pipelines. A. -- _____________________________________ Dr. Andrew Nelson _____________________________________

1 0

next NumPy community meeting
by Inessa Pawson 19 May '24

19 May '24

The next NumPy community meeting will be held this Wednesday, November 8th at 5pm UTC. Join us via Zoom: https://numfocus-org.zoom.us/s/83278611437?pwd=ekhoLzlHRjdWc0NOY2FQM0NPemdk… . Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both. For the notes from the previous meetings, visit: https://github.com/numpy/archive/tree/main/community_meetings. -- Cheers, Inessa Inessa Pawson GitHub: inessapawson

1 14

.npy MIME type?
by cj-python＠zougloub.eu 17 May '24

17 May '24

Hi, As I wanted to have a textual preview of .npy files, I noticed that there is no support (partial support in magic, none in xdg-mime upstream), and simply no documented MIME type (official or not), for the .npy file format. Would it be reasonable to consider application/x-numpy (which is the placeholder I use for now in my code [1] so I'm only suggesting it by lazyness and lack of imagination) and perhaps even registration of application/numpy at https://www.iana.org/assignments/media-types/media-types.xhtml ? Best regards, -- Jérôme [1] Links to code: - xdg-mime XML info at https://gitlab.com/cJ/xdg-shared-mime-info-packages/-/blob/main/numpy.xml ; could be upstreamed - text preview at https://gitlab.com/exmakhina/lessopen/-/blob/main/render/xm_lessopen_render…

1 0

ERROR: Module "features" does not exist
by Thomas Mansencal 16 May '24

16 May '24

Hello, We have a custom and unusual build environment that is making things a bit difficult to transition from the old Setuptools based build process to Meson. I'm currently blocked at the CPU feature set detection: ``` [tasks] Build command: call vcvarsall amd64 && cd /d "[...]\numpy" && pip install -e . --no-build-isolation ********************************************************************** ** Visual Studio 2022 Developer Command Prompt v17.9.6 ** Copyright (c) 2022 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x64' Defaulting to user installation because normal site-packages is not writeable Obtaining file:///[...]/numpy Checking if build backend supports build_editable: started Checking if build backend supports build_editable: finished with status 'done' Preparing editable metadata (pyproject.toml): started Preparing editable metadata (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error Ã— Preparing editable metadata (pyproject.toml) did not run successfully. â”‚ exit code: 1 â•°â”€> [24 lines of output] + [...]\meson\Scripts\meson.exe setup [...]\numpy [...]\numpy\build\cp311 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=[...]\numpy\build\cp311\meson-python-native-file.ini The Meson build system Version: 1.4.0 Source dir: [...]\numpy Build dir: [...]\numpy\build\cp311 Build type: native build Project name: NumPy Project version: 1.26.5 C compiler for the host machine: cl (msvc 19.39.33523 "Microsoft (R) C/C++ Optimizing Compiler Version 19.39.33523 for x64") C linker for the host machine: link link 14.39.33523.0 C++ compiler for the host machine: cl (msvc 19.39.33523 "Microsoft (R) C/C++ Optimizing Compiler Version 19.39.33523 for x64") C++ linker for the host machine: link link 14.39.33523.0 Cython compiler for the host machine: [...]\python-3.11\cython\bin\cython.bat (cython 3.0.9) Host machine cpu family: x86_64 Host machine cpu: x86_64 Program python found: YES (C:\Program Files\Python311\python.exe) Run-time dependency python found: YES 3.11 Has header "Python.h" with dependency python: YES Compiler for C supports arguments -fno-strict-aliasing: NO Message: [...]\numpy ..\..\meson_cpu\x86\meson.build:3:15: ERROR: Module "features" does not exist A full log can be found at [...]\numpy\build\cp311\meson-logs\meson-log.txt [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed Ã— Encountered error while generating package metadata. â•°â”€> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. 14:27:24 ERROR BuildError: The custom build system failed. ``` So the build fails whilst trying to import the "features" module here: https://github.com/numpy/numpy/blob/main/meson_cpu/x86/meson.build#L2C1-L3C1 Where is that module meant to be located? Cheers, Thomas

2 1

Build NumPy with Debugging Symbols with Meson
by Robert McLeod 16 May '24

16 May '24

Hi everyone, Is there a gist or similar guide anywhere on the steps required to build NumPy with debugging symbols on the Windows platform using the new Meson build system? It seems a bit difficult because NumPy seems to expect a `conda`-like file structure whereas if you are cloning `cpython` directly it is different. In particular `cpython` puts all the Windows files under the `PC` and `PCBuild` subdirectories. Also meson doesn't seem to have a means to call `python_d.exe` directly which may be causing problems. I've ended up building both `python.exe` and `python_d.exe` but run into some problem finding Python in `meson.build`. This describes my efforts to date: # Build CPython with Debugging Symbols on Windows ```shell git clone https://github.com/python/cpython.git cd cpython git switch 3.11 PCbuild\get_externals.bat # Build the debug version `python_d.exe` PCbuild\build.bat -e -d # Meson calls `python.exe` and doesn't seem to have an argument to change this. # We could either build it, or create a symlink PCbuild\build.bat # ln -s PCbuild/amd64/python_d.exe PCbuild/amd64/python.exe ``` The built Python will then be located in `<prefix>/cpython/PCBuild/amd64/python_d.exe`. ```batch set PREFIX=C:/Users/Robert/dev/cpython set PATH=%PREFIX%;%PREFIX%/PCBuild/amd64;%PREFIX%/Scripts;%PATH% ``` Next we have to install pip: https://docs.python.org/3/library/ensurepip.html, meson, and cython. ```shell python_d -m ensurepip python_d -mpip install meson meson-python ninja cython ``` NOTE: produces `pip3.exe`, not `pip`. # Build NumPy with debug Python ```shell git clone https://github.com/numpy/numpy.git cd numpy git switch maintenance/1.26.x git submodule update --init ``` https://mesonbuild.com/meson-python/how-to-guides/debug-builds.html Next try and build with meson/ninja: ```shell mkdir build-dbg cd build-dbg meson .. setup --buildtype=debug --includedir=C:/Users/Robert/dev/cpython/PC --libdir=C:/Users/Robert/dev/cpython/PCbuild/amd64 ninja ninja install ``` `meson .. setup <...>` fails and complains that, ''' Run-time dependency python found: NO (tried sysconfig) ..\meson.build:41:12: ERROR: Python dependency not found ''' which corresponds to: ```meson.build py = import('python').find_installation(pure: false) py_dep = py.dependency() ``` I tried also editing `pyproject.toml` to add the section: ```toml [tool.meson-python.args] setup = ['-Dbuildtype=debug', '-Dincludedir=C:/Users/Robert/dev/cpython/PC', '-Dlibdir=C:/Users/Robert/dev/cpython/PCbuild/amd64'] ``` and then build NumPy with pip using the debug python build: ```shell python_d -mpip install . --no-build-isolation -Cbuild-dir=build-dbg ``` But it fails in the same fashion. Searching for issues I find this one but it's likely in this case something related to the debug Python build is the problem. https://github.com/mesonbuild/meson/issues/12547 Meson installed version is 1.4.0. Any advice would be appreciated. I'm happy to write a gist if I can get it working. -- Robert McLeod robbmcleod(a)gmail.com robert.mcleod(a)hitachi-hightech.com

3 2

ANN: APyTypes 0.1
by Oscar Gustafsson 15 May '24

15 May '24

Dear all, as a consequence of a discussion here https://mail.python.org/archives/list/numpy-discussion@python.org/message/W… we are now please to announce the first release of APyTypes - algorithmic data types, both fixed- and floating-point, in Python. The library aims to be a drop-in replacement for NumPy, but with the difference that you can exactly specify the numerical format (and rounding mode). It is fair to say that there is a bit of work left until that, but the scalar and array data types and basic arithmetic operations are working. The library can be installed using: pip install apytypes (please let us know if your platform is not supported). For a comparison with other libraries see: https://apytypes.github.io/apytypes/comparison.html (general documentation at https://apytypes.github.io/ repo at https://github.com/apytypes/apytypes ). It should be noted that this is not a suitable replacement if you just want to use a different, existing, dtype, e.g. from ml_dtypes, this is a completely flexible solution for any dtype with complete control of rounding/quantization, including stochastic quantization. BR Oscar Gustafsson (I will not post additional announcements here, but as this was an outcome of a discussion on the mailing list, I thought it may be of interest to at least some members.)

1 0

Fastpathing lexsort for integers
by ml＠pietrobattiston.it 14 May '24

14 May '24

Hi all (my first message in this list), I have written some (Python) code to make lexsort on (lists of arrays of) integers more efficient, by "merging" all levels in a single array. I'm asking for some feedback between doing a PR. Compared to the current code, there is an improvement as long as: - the arrays (levels) are long at least 10^3 - the improvement becomes significant (~3x) above 10^4 - the sum of bits required for the representation of each level is no larger than the platform uint's bits; otherwise, my code falls back to Python int, and becomes way less performant. My rough guess it that the second condition holds in the vast majority of use cases (including a planned use in scipy - see https://github.com/scipy/scipy/issues/20151 for context). The code is here: https://github.com/toobaz/numpy/commit/aa5c6948 I know that the current form is not well written - at least _int_lexsort seems out of place, and imports placed inside the function - but I didn't know in which file it would belong (and anyway, this version makes a quick glance easy). In any case, feedback is welcome as to what should go where. For the records, the approach is similar to that employed in pandas MultiIndexes: https://github.com/pandas-dev/pandas/pull/19074 API-wise, there are various possibilities to do this: - the current status is that a new parameter "int_path" is added, and this determines which path to take; so nothing changes unless the user wants to use the new path - alternatively, I could check if all arrays are int (O(1)) and their maxima/minima (O(n)) and use the optimized code path only if it relies on uint, not Python int. The downside is that I would (slightly?) slow down those cases which do fallback to the old code. - moreover, in the current version "int_path" can take as value either a bool or a sorting algorithm, as understood by np.argsort. But an alternative is that we add a new argument "kind" to lexsort (consistently with argsort), which at the moment would raise NotImplementedError in the "old" path. By the way, my C is too rusty to really understand what goes on here: https://github.com/numpy/numpy/blob/main/numpy/_core/src/multiarray/item_se… ... but maybe it wouldn't be so difficult to support switching "kind" also in the "general" lexsort (old code path). - note that lexsort is guaranteed to be stable, while the new code path is stable only if "kind" is set to e.g. 'stable' (the default for argsort being 'quicksort') The current code misses tests (I'll be happy to add some once someone confirms that there is interest for this contribution), but some (performance) tests I ran are here: https://github.com/toobaz/int_lexsort/blob/main/Show%20benchmarks.ipynb Thanks for your attention, Pietro Battiston

2 3

3.13 wheels
by Andrew Nelson 14 May '24

14 May '24

I'm not sure if we've discussed this yet. I propose we start making nightly wheels for cp313 just after numpy2.0 gets officially released. That way there's no churn in the wheel infra. Note that there's already a 3.13dev job in CI.

2 1

Please consider dropping Python 3.9 support for Numpy 2.0
by Mark Harfouche 13 May '24

13 May '24

I know this is somewhat the 11th hour for the numpy 2.0 release but as downstream packager and user of numpy, I would really like to ask that Numpy strives to adhere to NEP29, and SPEC0 SEPC0 makes it pretty clear that Python 3.9 should not be included in any "future" versions. Leading by example in this case will help downstream make the hard decision to adhere to the compatibility guidelines from SPEC0. Thank you for considering this last minute request. I know it adds work at this stage. Mark

13 26