Mailman 3 February 2024 - SciPy-Dev

SciPy Community Meeting: Wednesday November 29th at 8pm UTC
by Melissa Mendonça 05 Mar '24

05 Mar '24

Hi all, The next SciPy Community Meeting will happen on* Wednesday November 29th at 8pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 Our community calendar is also available here: https://scientific-python.org/calendars/ Cheers, Melissa

1 7

welcome Lucas and Daniel to the SciPy maintainers team
by Ralf Gommers 29 Feb '24

29 Feb '24

Hi all, On behalf of the SciPy developers I'd like to welcome Lucas Colley and Daniel Schmitz as new members of the maintainers team. Lucas has done a lot of general maintenance work and has been one of the main contributors to our array API standard support over the past 7 months: https://github.com/scipy/scipy/pulls/lucascolley. Daniel has been contributing steadily for about 2 years, mainly to scipy.stats and scipy.optimize: https://github.com/scipy/scipy/pulls/dschmitz89. Both have already had a significant impact on SciPy, and we are looking forward to their continued contributions. Welcome Daniel and Lucas! Cheers, Ralf

8 7

Next SciPy newcomers' meeting - TODAY October 13th at 8pm UTC
by Melissa Mendonça 29 Feb '24

29 Feb '24

Hello, folks! Our next newcomers meeting happens TODAY, October 13th, at 8pm UTC. Feel free to join to ask questions about contributing, our community, follow ups on existing issues or pull requests, or to just chat. If you are a maintainer or a regular contributor, you are also welcome to join as well to connect with other members of our community :) Here's the hackmd we use to take notes, which includes the meeting details: https://hackmd.io/@melissawm/scipy-newcomers And here's the link if you wish to join us on Zoom: https://us06web.zoom.us/j/89085469494?pwd=dVFwK3dVczRyRUowZWZkckFtTmloUT09 Hope to see you around! Cheers, Melissa

1 2

RFC: decouple doctesting from refguide-check
by Evgeni Burovski 23 Feb '24

23 Feb '24

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Question: Where can I place my Euler diagram parser library?
by Bruno Peixoto 19 Feb '24

19 Feb '24

Hi all, I wonder: where can I place my Euler diagram parser library? The tool has been useful to me, and I would like to have a broader audience. I got a suggestion on [1], scipy may not be the most suitable place for it. However, I still want to offer it somewhere else prestigious. Where do you recommend? https://pypi.org/project/eule/ [1] https://github.com/scipy/scipy/issues/20109#issuecomment-1952907750

1 0

Question: Can LowLevelCallable functions be beneficial to scipy.optimize performance?
by d.lenard80＠gmail.com 19 Feb '24

19 Feb '24

Hello SciPy Team, Recently, I came across a blog article titled "Julia: 17X Faster than Python's SciPy" which discussed performance benchmarks comparing Julia and Python for scientific computing tasks. In the article, the author highlighted significant performance gains in Julia compared to Python's SciPy library, particularly in the context of using scipy.optimize.curve_fit as an example. The article can be found here: https://www.matecdev.com/posts/julia-17x-faster-vs-python-scipy.html Coincidentally, just a few days ago, I discovered LowLevelCallable functions as a way to speed up quadrature operations. I speculated that the overhead of Python functions might explain the performance difference described in the mentioned article. However, upon further investigation, I realized that LowLevelCallable functions are not currently utilized for optimizers in scipy.optimize.curve_fit. In my search for more information on this topic, I stumbled upon the GitHub issue titled "Passing LowLevelCallable to Optimizers" (#11783), where Mr. Gommers from SciPy expressed reservations about the implementation of LowLevelCallable functions as inputs to minimization problems in SciPy. He noted that there is not much overhead in using Python functions, and the potential benefits are likely to be very small. I conducted some experiments with scipy.integrate.quad and observed speed-ups ranging from 10-20x when using LowLevelCallable functions compared to Python functions for quadrature tasks. These findings suggest that LowLevelCallable functions can offer performance improvements in certain scenarios. This leads me to wonder: are the benefits of LowLevelCallable specific to the scipy.integrate subpackage, or could it be beneficial for the scipy.optimize subpackage as well? I wanted to share these findings and seek your expert opinion on the matter. Best regards, Oyibo Below is the code of the performance check. I used Numba to convert the Python function into a LowLevelCallable. # **************************************************************************** import numpy as np import numba as nb import numba.types as nbt import scipy.integrate as si from scipy import LowLevelCallable import timeit # **************************************************************************** # Case 1: Integrate with Python function def py_integrand(t): return np.exp(-t) / t**2 result = si.quad(py_integrand, 1, np.inf) print('Python:') print('Integral Result:', result[0]) # Integral Result: 0.14849550677592208 # **************************************************************************** # Case 2: Integrate with LowLevelCallable (use Numba to convert) llc_integrand = nb.cfunc("float64(float64)")(py_integrand).ctypes result = si.quad(llc_integrand, 1, np.inf) print('\nLowLevelCallable:') print('Integral Result:', result[0]) # Integral Result: 0.14849550677592208 # **************************************************************************** # Case 3: Integrate with LowLevelCallable for multiple args # (A little bit more code but more versatile and faster) def LowLevelCallable_integrand(integrand_func): jitted_function = nb.njit(integrand_func) # @cfunc('float64(int32, float64*)') @nb.cfunc(nbt.float64(nbt.intc, nbt.CPointer(nbt.float64))) def wrapped(n, xx): values = nb.carray(xx, n) return jitted_function(values) return LowLevelCallable(wrapped.ctypes) @LowLevelCallable_integrand def llc_integrand_gen(args): t = args[0] return np.exp(-t) / t**2 result = si.quad(llc_integrand_gen, 1, np.inf) print('\nLowLevelCallable (generalized, can be used for additional args):') print('Integral Result:', result[0]) # Integral Result: 0.14849550677592208 # **************************************************************************** # Measuring execution time num_repeats = 10_000 num_runs = 10 time_py = np.mean(timeit.repeat(lambda: si.quad(py_integrand, 1, np.inf), repeat=num_repeats, number=num_runs))*1000 time_llc = np.mean(timeit.repeat(lambda: si.quad(llc_integrand, 1, np.inf), repeat=num_repeats, number=num_runs))*1000 time_llc_gen = np.mean(timeit.repeat(lambda: si.quad(llc_integrand_gen, 1, np.inf), repeat=num_repeats, number=num_runs))*1000 print(f'\nExecution Time (mean of {num_runs} runs, {num_repeats} repeats):') print(f'Python Callable: {time_py:.3f} ms') print(f'LowLevelCallable: {time_llc:.3f} ms') print(f'LowLevelCallable_gen: {time_llc_gen:.3f} ms') print(f'time_py/time_llc: {time_py/time_llc:.1f}x') print(f'time_py/time_llc_gen: {time_py/time_llc_gen:.1f}x') # Execution Time (mean of 10 runs, 10000 repeats): # Python Callable: 1.218 ms # LowLevelCallable: 0.114 ms # LowLevelCallable_gen: 0.065 ms # time_py/time_llc: 10.6x # time_py/time_llc_gen: 18.7x

5 7

Expose NonConvergence exception in scipy.optimize
by Jake Bowhay 18 Feb '24

18 Feb '24

Hi all, In [1] it was pointed out that the NonConvergence exception is only available through scipy.optimize.nonlin which is deprecated. This is annoying if you want to catch this exception. I have opened [2] to expose NonConvergence in the main scipy.optimize namespace so that this exception is accessible without deprecation warning and in the future when optimize.nonlin is removed. Please feel free to raise any thoughts/objections here or on the pr. Thanks, Jake [1] https://github.com/scipy/scipy/issues/20100 [2] https://github.com/scipy/scipy/pull/20103

1 0

Developer request.
by Gaurav soni 17 Feb '24

17 Feb '24

My name is gaurav soni, I'm from jodhpur, rajasthan, india. I am 25 years old and I want to help in developing the scipy for its betterment. I am a student pursuing masters in data science and artificial intelligence from Woolf University.

2 1

Request for Review: Addition of average_time Metric to Benchmark Suite
by Tessa van der Heiden 09 Feb '24

09 Feb '24

Hey SciPy Developers, I've submitted a PR adding an average_time metric to the global optimizers' Benchmark. You can find it here: https://github.com/scipy/scipy/pull/20006. It's received positive initial feedback but seems to have been sidelined. Could someone take a look and push it forward? Thanks, Tessa

1 0

SciPy tests silently output NaNs as test statistics and p-value if there are NaNs present in the input
by Mikolaj Buchwald 08 Feb '24

08 Feb '24

This is a general usability/UX remark: when there are NaNs in the input, the SciPy tests return NaNs as a statistic and p-value. This is an behavior inherited from NumPy reduction functions (such as np.sum), however, in the context of statistical functions this is very unintuitive, potentially misleading (easily can be interpreted as non-significant or highly significant result), and can cost some time to debug. See for example: * https://stackoverflow.com/questions/77087907/kruskal-wallis-test-always-giv… * https://github.com/scipy/scipy/issues/20056 A simple solution that I propose is: add a warning message that will appear if there are NaNs in the input, so that the user could be alerted/immediately notified (tipped) on what is the reason for the observed behaviour and how to proceed with debugging such an issue. I am not sure about it, and have not tested it, but knowing the general philosophy of R or Excel, they would just disregard NaNs (implicit .dropna()) and returned the user with the desired statistics. Such radical solution is out of scope, I believe, in Python, but please consider adding at least a warning. Kind Regards, Mikolaj

1 0