Mailman 3 November 2021 - NumPy-Discussion

Deprecating PyDataMem_SetEventHook
by Matti Picus 18 Nov '21

18 Nov '21

NumPy has had a way to add a call to a user-supplied function whenever allocating via PyDataMem_SetEventHook [0]. Python added the tracemalloc [1] functionality which make this redundant. NumPy also added memory alloc/free overriding in NEP 49 [2] which will be part of the 1.22 release. I propose deprecating the PyDataMem_SetEventHook function, and have submitted a PR to do so for 1.23 [3]. Thoughts? Is anyone using the hook for something that tracemalloc cannot do? Matti [0] https://numpy.org/devdocs/reference/c-api/data_memory.html#c.PyDataMem_SetE… [1] https://docs.python.org/3/library/tracemalloc.html [2] https://numpy.org/neps/nep-0049.html [3] https://github.com/numpy/numpy/pull/20394

1 0

request for feedback on cross-library design proposal
by Ralf Gommers 17 Nov '21

17 Nov '21

(Apologies for the cross-posting, if you follow more than one of these lists, you may see this multiple times.) Hi all, I'd like to solicit the feedback of maintainers of NumPy, SciPy, scikit-learn, and scikit-image, and other interested community members on this blog post and proposed design to enable multiple array types in SciPy, scikit-learn, scikit-image and other libraries in this ecosystem: https://labs.quansight.org/blog/2021/11/pydata-extensibility-vision/. Because this design is meant to be shared across projects, it would be great to discuss it in a central place. The new https://discuss.scientific-python.org/ seems like a good venue for that. Here are two separate forum threads for this: - for the big picture: https://discuss.scientific-python.org/t/a-proposed-design-for-supporting-mu… - for the most tricky design decision (user-facing switch & default behavior): https://discuss.scientific-python.org/t/default-dispatching-behavior-for-su… Thanks in advance for any input! Cheers, Ralf

1 0

installing numpy on OpenIndiana
by asyropoulos＠aol.com 17 Nov '21

17 Nov '21

Hello, I am trying to install numpy on OpenIndiana (OpenSolaris) with no luck.The command "pip3.9 install numpy" fails because it cannot detect theFORTRAN compiler. I downloacded the zip file from git and I tried to build.First, I had to touch setup.py because the following command FULLVERSION = versioneer.get_version() assigns to the value '0+unknown' to FULLVERSION. Thus I replaced it with: FULLVERSION = "1.22.0+True" Next, I entered the command $ ./setup.py build_src and what I got follows. I am using gcc7.0 and gfortran is included in my system. Also, I have python3.9 install in my system. Can someone suggest any solution thisproblem of mine? Thanks in advance Apostolos Running from numpy source directory.Cythonizing sources numpy/random/_bounded_integers.pxd.in has not changed numpy/random/_bounded_integers.pyx.in has not changed numpy/random/_common.pyx has not changed numpy/random/_generator.pyx has not changed numpy/random/_mt19937.pyx has not changed numpy/random/_pcg64.pyx has not changed numpy/random/_philox.pyx has not changed numpy/random/_sfc64.pyx has not changed numpy/random/bit_generator.pyx has not changed numpy/random/mtrand.pyx has not changed numpy/random/_bounded_integers.pyx has not changed blas_opt_info: blas_mkl_info: customize UnixCCompiler libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE blis_info: libraries blis not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE openblas_info: C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -m64 -O2 -fPIC -DPIC -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6 -D_REENTRANT -m64 -O2 -fPIC -DPIC -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6 -fPIC creating /tmp/tmp57blbggg/tmp creating /tmp/tmp57blbggg/tmp/tmp57blbggg compile options: '-c' gcc: /tmp/tmp57blbggg/source.c gcc /tmp/tmp57blbggg/tmp/tmp57blbggg/source.o -L/usr/lib -lopenblas -o /tmp/tmp57blbggg/a.out ld: fatal: file /tmp/tmp57blbggg/tmp/tmp57blbggg/source.o: wrong ELF class: ELFCLASS64 ld: fatal: file processing errors. No output written to /tmp/tmp57blbggg/a.out collect2: error: ld returned 1 exit status NOT AVAILABLE accelerate_info: NOT AVAILABLE atlas_3_10_blas_threads_info: Setting PTATLAS=ATLAS libraries tatlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE atlas_3_10_blas_info: libraries satlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE /tmp/numpy-main/numpy/distutils/system_info.py:2024: UserWarning: Optimized (vendor) Blas libraries are not found. Falls back to netlib Blas library which has worse performance. A better performance should be easily gained by switching Blas library. if self._calc_info(blas): blas_info: libraries blas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE /tmp/numpy-main/numpy/distutils/system_info.py:2024: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. if self._calc_info(blas): blas_src_info: NOT AVAILABLE /tmp/numpy-main/numpy/distutils/system_info.py:2024: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. if self._calc_info(blas): NOT AVAILABLE non-existing path in 'numpy/distutils': 'site.cfg' lapack_opt_info: lapack_mkl_info: libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE openblas_lapack_info: C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -m64 -O2 -fPIC -DPIC -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6 -D_REENTRANT -m64 -O2 -fPIC -DPIC -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6 -fPIC creating /tmp/tmpydsl_vpn/tmp creating /tmp/tmpydsl_vpn/tmp/tmpydsl_vpn compile options: '-c' gcc: /tmp/tmpydsl_vpn/source.c gcc /tmp/tmpydsl_vpn/tmp/tmpydsl_vpn/source.o -L/usr/lib -lopenblas -o /tmp/tmpydsl_vpn/a.out ld: fatal: file /tmp/tmpydsl_vpn/tmp/tmpydsl_vpn/source.o: wrong ELF class: ELFCLASS64 ld: fatal: file processing errors. No output written to /tmp/tmpydsl_vpn/a.out collect2: error: ld returned 1 exit status NOT AVAILABLE openblas_clapack_info: libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE flame_info: libraries flame not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE atlas_3_10_threads_info: Setting PTATLAS=ATLAS libraries tatlas,tatlas not found in /usr/local/lib libraries tatlas,tatlas not found in /usr/lib libraries tatlas,tatlas not found in /usr/lib/ <class 'numpy.distutils.system_info.atlas_3_10_threads_info'> NOT AVAILABLE atlas_3_10_info: libraries satlas,satlas not found in /usr/local/lib libraries satlas,satlas not found in /usr/lib libraries satlas,satlas not found in /usr/lib/ <class 'numpy.distutils.system_info.atlas_3_10_info'> NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/ <class 'numpy.distutils.system_info.atlas_threads_info'> NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib libraries f77blas,cblas,atlas not found in /usr/lib/ <class 'numpy.distutils.system_info.atlas_info'> NOT AVAILABLE lapack_info: libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/'] NOT AVAILABLE /tmp/numpy-main/numpy/distutils/system_info.py:1856: UserWarning: Lapack (http://www.netlib.org/lapack/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [lapack]) or by setting the LAPACK environment variable. return getattr(self, '_calc_info_{}'.format(name))() lapack_src_info: NOT AVAILABLE /tmp/numpy-main/numpy/distutils/system_info.py:1856: UserWarning: Lapack (http://www.netlib.org/lapack/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [lapack_src]) or by setting the LAPACK_SRC environment variable. return getattr(self, '_calc_info_{}'.format(name))() NOT AVAILABLE numpy_linalg_lapack_lite: FOUND: language = c define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')] Warning: attempted relative import with no known parent package /usr/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros' warnings.warn(msg) running build_src build_src building py_modules sources building library "npymath" sources gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: file processing errors. No output written to _configtest collect2: error: ld returned 1 exit status Traceback (most recent call last): File "/tmp/numpy-main/./setup.py", line 451, in <module> setup_package() File "/tmp/numpy-main/./setup.py", line 443, in setup_package setup(**metadata) File "/tmp/numpy-main/numpy/distutils/core.py", line 169, in setup return old_setup(**new_attr) File "/usr/lib/python3.9/vendor-packages/setuptools/__init__.py", line 153, in setup return distutils.core.setup(**attrs) File "/usr/lib/python3.9/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/tmp/numpy-main/numpy/distutils/command/build_src.py", line 144, in run self.build_sources() File "/tmp/numpy-main/numpy/distutils/command/build_src.py", line 155, in build_sources self.build_library_sources(*libname_info) File "/tmp/numpy-main/numpy/distutils/command/build_src.py", line 288, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/tmp/numpy-main/numpy/distutils/command/build_src.py", line 378, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 684, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program ----------------------Apostolos Syropoulos Xanthi, Greece

2 2

NumPy Development Meeting Wednesday - Triage Focus
by Sebastian Berg 17 Nov '21

17 Nov '21

Hi all, Our bi-weekly triage-focused NumPy development meeting is Wednesday, November 17rd at 16:30 UTC (8:30am Pacific Time). Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized, discussed, or reviewed. Best regards Sebastian PS: Sorry for the late reminder.

1 0

Tensor typing meeting (15 Nov) slides & video
by Stefan van der Walt 17 Nov '21

17 Nov '21

I thought this might be of interest to Bas and others. ----- Original message ----- From: Matthew Rahtz via Typing-sig Date: Tuesday, November 16, 2021 02:31 Thanks to everyone for coming! Recording is here <https://drive.google.com/file/d/1ywE7sHM--9QEZ_LhcALFBVF_nsO601NB/view?usp=…>, and the slides are at Dealing with DTypes <https://docs.google.com/presentation/d/14hHWlpMOuijNEhF7fwvANa8Ainu4R2LtBt3…>. Details of this and previous tensor typing meetings are available here <https://docs.google.com/document/d/1oaG0V2ZE5BRDjd9N-Tr1N0IKGwZQcraIlZ0N8ay…>. Our next tensor typing meeting will tentatively be on *Monday the 13th of December*. See you then! A quick summary of the talk is below. * How should we handle data type promotion in stubs? * Option 1: One overload for each possible combination of dtypes * Option 2: One overload for each result dtype * Option 3: Don't handle type promotion * Option 4: Use a dtype class hierarchy and exploit Union type operator behaviour * Option 5: Propose a type promotion operator * Option 6: Propose a 'nearest common parent' operator * Option 7: Propose a type lookup table operator * Consensus during discussion was that since it looks like a new type operator *would* be required, we should probably hold off on dealing with this until the community shows a strong desire for this feature, and in the meantime just not handle data type promotion in stubs.

2 1

NumPy 1.22.x has been branched
by Charles R Harris 16 Nov '21

16 Nov '21

Hi All, Just as the subject line says. Chuck

1 0

spam on the mailing lists
by Matti Picus 16 Nov '21

16 Nov '21

The list default for new subscribers is to hold posts for approval before sending to the mailing list. Hopefully that will cut down the number of spam messages, at the cost of delay of maybe a day for new subscriber posts. In order to make the policy clear, I submitted a PR to numpy.org [0]. Comments and suggestions welcome there. Matti [0] https://github.com/numpy/numpy.org/pull/487

2 2

[RFC] - numpy/SVML appears to be poorly optimized
by Noah Goldstein 15 Nov '21

15 Nov '21

The numpy SVML library: https://github.com/numpy/SVML appears to be poorly optimized. Since its just the raw assembly dump this also makes it quite difficult to improve (with either a better compiler or by hand). Some of the glaring issues are: 1. register allocation / spilling 2. rodata layouts / const-propagation of the values. 3. Very odd use of internal functions that really ought to be inlined. Are these functions meant to be heavily optimized? If so, are people open to patches that optimize them (either with new C implementations are in the current assembly implementations).

5 5

new time for community meetings
by Inessa Pawson 14 Nov '21

14 Nov '21

It’s that time of the year again when most of us have to adjust our clocks. For some of us the fun doesn’t stop there as we have to adjust our work schedules too. Let’s see if we can find a new time slot on Wednesday for bi-weekly NumPy community meetings that works for everyone: http://whenisgood.net/hhy4hk8 -- Every good wish, Inessa Pawson

1 1

Revert the return of a single NaN for `np.unique` with floating point numbers?
by Sebastian Berg 14 Nov '21

14 Nov '21

Hi all, In NumPy 1.21, the output of `np.unique` changed in the presence of multiple NaNs. Previously, all NaNs were returned when we now only return one (all NaNs were considered unique): a = np.array([1, 1, np.nan, np.nan, np.nan]) Before 1.21: >>> np.unique(a) array([ 1., nan, nan, nan]) After 1.21: array([ 1., nan]) This change was requested in an old issue: https://github.com/numpy/numpy/issues/2111 And happened here: https://github.com/numpy/numpy/pull/18070 While, it has a release note. I am not sure the change got the attention it deserved. This would be especially worrying if it is a regression for anyone? Cheers, Sebastian PS: One additional note, is that this does not work for object arrays (it cannot reasonable): >>> np.unique(a.astype(object)) array([1.0, nan, nan, nan], dtype=object)

4 6