Mailman 3 November 2022 - SciPy-Dev

Units of scipy.signal.stft
by Servant Mathieu Nov. 27, 2022

Nov. 27, 2022

Hi community, I am currently applying a short-term Fourier transform to some electrophysiological signal (variable 'my_signal', in mV/m^2). I would like to get the spectral amplitude in the same unit (mV/m^2/Hz). I did: sf = 1024 window_size_step = 323 hop_size = 1 f, t, Zxx = sg.stft(my_signal, fs = sf, window = 'boxcar', nperseg = window_size_step, noverlap = (window_size_step-hop_size),return_onesided = True) amp_spectrum = np.abs(Zxx) I have a doubt on the unit of variable 'amp_spectrum'. Do I need to apply some additional scaling? All the best, Mathieu I have a doubt on the unit of variable 'amp_spectrum'. Do I need to apply some additional scaling?

1 0

stats.goodness_of_fit: a generalized univariate goodness-of-fit test
by Matt Haberland Nov. 27, 2022

Nov. 27, 2022

Hi team, gh-16967 (https://github.com/scipy/scipy/pull/16967) added `stats.goodness_of_fit`, a general goodness-of-fit test that can replicate/improve the results of `ks_1samp`, `cramervonmises`, and `anderson` as special cases, but extends the functionality to arbitrary choices of known and unknown parameter values (and arbitrary distributions and p-values, which `anderson` lacked before). If you're interested, please head to https://github.com/scipy/scipy/pull/16967 to review the addition, and let me know if there's anything that needs to be changed before 1.10 branches. This email is a little late, since the PR already merged, but there is still time to incorporate changes (or revert the PR) if needed. Matt -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

1 0

NumPy 1.24.0rc1 released
by Charles R Harris Nov. 24, 2022

Nov. 24, 2022

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.24.0rc1. The NumPy 1.24.0 release continues the ongoing work to improve the handling and promotion of dtypes, increase execution speed, and clarify the documentation. There are also a large number of new and expired deprecations due to changes in promotion and cleanups. This might be called a deprecation release. Highlights are - Many new deprecations. - Many expired deprecations. - New f2py features and fixes. - New "dtype" and "casting" keywords for stacking functions. The Python versions supported in this release are 3.8-3.11 Note that 32 bit wheels are only provided for Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.24.0rc1/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.24.0rc1>. The Python 3.8 aarch64 wheel is missing from this pre-release due to build problems. *Contributors* A total of 174 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @DWesl - @h-vetinari - Aaron Meurer - Aayush Agrawal + - Adam Knapp + - Adarsh Singh + - Al-Baraa El-Hag - Alban Desmaison + - Ales Crocaro + - Alex + - Alexander Grund + - Alexander Kanavin + - Andras Deak - Angéllica Araujo + - Anselm Hahn + - Ari Cooper Davis + - Axel Huebl + - Bas van Beek - Bhavuk Kalra - Bram Ton + - Brent Tan + - Brigitta Sipőcz - Callum O'Riley + - Charles Harris - Chiara Marmo - Christoph Reiter - Damien Caliste - Dan Schult + - Daniel da Silva - DanielHabenicht + - Dave Goel + - David Gilbertson + - Developer-Ecosystem-Engineering - Digya Acharya + - Dimitri Papadopoulos Orfanos - Eric-Shang + - Evgeni Burovski - Ewout ter Hoeven - Felix Hirwa Nshuti + - Frazer McLean + - Ganesh Kathiresan - Gavin Zhang + - Gaëtan de Menten - Greg Lucas - Greg Sadetsky + - Gregory P. Smith [Google LLC] + - Hameer Abbasi - Hannah Aizenman + - Hood Chatham - I-Shen Leong - Iantra Solari + - Ikko Ashimine + - Inessa Pawson - Jake Bowhay + - Jake Close + - Jarrod Millman - Jason Thai - Jessé Pires + - Jhonatan Cunha + - Jingxuan He + - Johnathon Cusick + - Jon Morris - Jordy Williams + - Josh Wilson - João Fontes Gonçalves + - Juan Luis Cano Rodríguez - Jyn Spring 琴春 + - KIU Shueng Chuan - Karl Otness + - Kevin Sheppard - Kinshuk Dua + - Leo Fang - Leona Taric + - Lev Maximov + - Lillian Zha + - Loïc Estève - M. Eric Irrgang + - Mariusz Felisiak - Mark Harfouche - Martin Zacho + - Matheus Santana Patriarca + - Matt Williams + - Matteo Raso + - Matthew Barber - Matthew Brett - Matthew Sterrett + - Matthias Bussonnier - Matthias Koeppe + - Matti Picus - Meekail Zain + - Melissa Weber Mendonça - Michael Osthege + - Michael Siebert + - Mike Toews - Miki Watanabe + - Miles Cranmer + - MilesCranmer + - Monika Kubek + - Muhammad Jarir Kanji + - Mukulika Pahari - Namami Shanker - Nathan Goldbaum + - Nathan Rooy + - Navpreet Singh + - Noritada Kobayashi + - Oleksiy Kononenko + - Omar + - Pal Barta + - Pamphile Roy - Patrick Hoefler + - Pearu Peterson - Pedro Nacht + - Petar Mlinarić + - Pey Lian Lim - Pieter Eendebak - Pradipta Ghosh - Pranab Das + - Precision Wordcraft LLC + - PrecisionWordcraft + - Rafael CF Sousa + - Rafael Cardoso Fernandes Sousa - Rafael Sousa + - Raghuveer Devulapalli - Ralf Gommers - Rin Cat (鈴猫) + - Robert Kern - Rohit Davas + - Rohit Goswami - Ross Barnowski - Ruth Comer + - Sabiha Tahsin Soha + - Sachin Krishnan T V + - Sanjana M Moodbagil + - Sanya Sinha + - Sarah Coffland + - Saransh Chopra + - Satish Kumar Mishra + - Satish Mishra + - Sayed Adel - Schrijvers Luc + - Sebastian Berg - Serge Guelton - Seva Lotoshnikov + - Shashank Gupta + - Shoban Chiddarth + - Shreya Singh + - Shreyas Joshi + - Sicheng Zeng + - Simran Makandar + - Srimukh Sripada + - Stefan van der Walt - Stefanie Molin + - Stuart Archibald - Tania Allard - Thomas A Caswell - Thomas Kastl + - Thomas Mansencal + - Tony Newton / Teng Liu + - Toshiki Kataoka - Tyler Reddy - Vardhaman Kalloli + - Warren Weckesser - Will Ayd + - William Stein + - XinRu Lin + - Yin Li + - Yulv-git + - Yunika Bajracharya + - Zachary Blackwood + - amagicmuffin + - dg3192 + - juztamau5 + - swagatip + - 渡邉美希 + Cheers, Charles Harris

1 0

Release of NumPy 1.23.5
by Charles R Harris Nov. 20, 2022

Nov. 20, 2022

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.23.5. NumPy 1.23.5 is a maintenance release that fixes bugs discovered after the 1.23.4 release and keeps the build infrastructure current. The Python versions supported for this release are 3.8-3.11. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.23.5>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.23.5>. *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @DWesl - Aayush Agrawal + - Adam Knapp + - Charles Harris - Navpreet Singh + - Sebastian Berg - Tania Allard *Pull requests merged* A total of 10 pull requests were merged for this release. - #22489: TST, MAINT: Replace most setup with setup_method (also teardown) - #22490: MAINT, CI: Switch to cygwin/cygwin-install-action@v2 - #22494: TST: Make test_partial_iteration_cleanup robust but require leak... - #22592: MAINT: Ensure graceful handling of large header sizes - #22593: TYP: Spelling alignment for array flag literal - #22594: BUG: Fix bounds checking for ``random.logseries`` - #22595: DEV: Update GH actions and Dockerfile for Gitpod - #22596: CI: Only fetch in actions/checkout - #22597: BUG: Decrement ref count in gentype_reduce if allocated memory... - #22625: BUG: Histogramdd breaks on big arrays in Windows Cheers, Charles Harris

1 0

Parameter format pass in log_pdf for the stats.fit method
by Julian CHAMBRIER Nov. 17, 2022

Nov. 17, 2022

Dear all, I have created a Pert method that can be used in the stats.fit function to optimize the parameters low, peak, high and lambda. The problem is that the optimizer tests very close but negligible parameter values with a high number of numbers after the decimal point. For example, it tests with lambda = 1.1111111111 and then it tests 1.1111111113. For the purpose of optimization, I would like to know how to force the parameters to be only integers (or trim 2 digits after the decimal point) so that it tests less possibilities and converges faster to the optimal optimization. I tried to modify my _argcheck method but the problem is that it first generates the values of the parameters low, peak, high and lambda and then checks if the conditions in the _argcheck method are valid. Whereas I want the parameters to respect a format before going into _argcheck. Also, I don't want it to generate the parameter values and artificially trim them (with round function for example) before passing them to the pdf method (or log_pdf). Indeed, if I do that, it will still generate the values of the parameters and with the round function we risk to test several times a combination. Do you have any idea which method to modify/add in order to constrain the parameters a bit when they are generated to be tested as a candidate for the best optimization. P.S : My question is also generalized to the other distributions in order to avoid that the numbers after the decimal point are too big and therefore test too many possibilities. Thanks in advance, Sincerely, Julian

3 9

the interpolate project
by Evgeni Burovski Nov. 16, 2022

Nov. 16, 2022

Hi, I'd like to announce the conclusion of the 'clear the scipy.interpolate backlog' project [1], and take a moment to reflect on its results. The intended scope of the project is archived at [2], and the project board is at [3]. So the main idea of the project was to work on stale issues and pull requests which were bitrotting in the backlog and which were difficult to act on in our usual workflow. The main goals of the project are achieved. The subpackage is definitely not finished (and unlikely to ever be), but it does look much healthier than before. To summarize the current status of the issue tracker: - there are several PRs in flight, which can and will be acted upon in our usual pace, likely before branching 1.10; - after these are done, we will have left five issues in total: two are ILP64 Fortran support; two are in Qhull (an integer overflow and a threading issue), and a possible build difference on Windows vs Linux (also qhull-related). - one stalled pull request which proposes an interpolant in the form of a ratio of two polynomials. This has a couple of open questions about the math, so if you are familiar with rational interpolators, please take a look! - some 20-odd enhancement requests in the 'triaged, waiting-for-contributor' column of [3]. Working on any of these will be most welcome. So what was not archived as planned in the original proposal [2]. First, the timeline. The originally planned timeline slipped horribly, for various reasons. Of the work items, two last items were achieved partially at best. First, "Contact fortran-lang organization and other interested parties on possible maintenance of FITPACK and other Fortran components." We are watching the fortran-lang progress with barely restricted excitement, and we hope to make use of it when it matures, across various scipy subpackages. For FITPACK specifically, our evolved vision is documented in the SciPy roadmap [4]. In a nutshell, we think it useful to grow an independent feature set for smoothing splines; once this is reasonably feature-complete, we can revisit the question of the future of FITPACK in scipy. BTW, I'd like to take an opportunity to draw attention to [5] which starts bringing non-FITPACK smoothing splines to SciPy. The other item which we did not get to is "Clean up interpolate usage in other subpackages, primarily in signal." This work remains to be done still. In closing, I'd like to stress that working on SciPy is always a team effort. Over the last months, multiple people did amazing work. Matt was doing great reviews, Jake went down a rabbit hole of smoking out a bunch of unexpected issues, Kai was improving performance of multivariate interpolators, Atsushi saw stalled PRs across the finish line and refactored clunky C/Fortran glue, Nikita fixed possibly the most obscure UB bug in 40-year old Fortran code, Carl and Matthew and Peter showed an exemplary handling of integer overflows --- and this is just the tip of an iceberg. It is a real pleasure working with members of the vibrant SciPy community, and I'm looking forward to the continuing collaboration with fellow devs. Cheers, Evgeni [1] It is a bit awkward since the project was discussed internally in the dev team, but was in fact not circulated to the mailing list. This is very much not intentional, and the responsibility for this error is squarely with me. [2] https://github.com/scipy/archive/blob/main/proposals/use_scipy_funds/2022_0… [3] https://github.com/orgs/scipy/projects/1 [4] https://scipy.github.io/devdocs/dev/roadmap-detailed.html [5] https://github.com/scipy/scipy/pull/16653

2 1

SciPy CZI Grant award
by Pamphile Roy Nov. 16, 2022

Nov. 16, 2022

8 7

_fitstart method doesn't work
by Julian CHAMBRIER Nov. 15, 2022

Nov. 15, 2022

Hello, I can't get the _fitstart method to work. Indeed, the optimizer is supposed to take the values returned by _fitstart, but when I display the values, the first values don't correspond at all to the values I entered in _fitstart. Could you help me? Thanks a lot, Julian

2 2

Implementation of new, faster and simpler method of conversion for Rotation.as_euler() method
by Evandro Nov. 11, 2022

Nov. 11, 2022

Hey there everyone! I'm posting here following advice on a pull request I made about the implementation of an algorithm I recently published. I recently (just today!) published an article about a method of the direct conversion between a quaternion variable to Euler angles in any sequence. I tested my new method against the already implemented method in two different computers, and got up to 30 times less execution time, while having the same results. - I implemented a function called "_compute_euler_from_quat", heavily based on the already implemented "_compute_euler_from_matrix" for ease of comparison. - I added a new optional argument to the as_euler() method called algorithm: --- Setting algorithm = "from_matrix" (default) uses the legacy algorithm. --- Setting algorithm = "from_quat" uses the new algorithm. This method gives explicit simple formulas for all of the angles and takes way less steps to calculate. The method implemented in the Rotation class first transforms the quaternion into a rotation Matrix, then performs complicated matrix multiplications to transform the axes for the desired sequence and finally calculates the value of the angles using inverse trigonometric functions. My method, in comparison, only does a permutation of the quaternion elements and then applies the inverse trigonometric functions. Even though the time execution is better, the main advantage of my algorithm is that it is arguably a lot simpler to understand, implement and maintain. --- Links: --- Paper (Open access): https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0276302 Pull request: https://github.com/scipy/scipy/pull/17392 The code used for the test: https://github.com/evbernardes/quaternion_to_euler

1 0

RFC: a proposal to deprecate interpolate.interp2d
by Evgeni Burovski Nov. 8, 2022

Nov. 8, 2022

Hi, TL;DR : I propose to deprecate and eventually remove the `interp2d` routine from `scipy.interpolate` I believe it has both interface and structural problems which cannot be fixed in a satisfactory backwards compatible way; all its functionality is available otherwise via equivalent and more predictable routines (*BivariateSpline and bisplrep); for interpolation specifically we now have better behaved routines (interpn and griddata). Cons for deprecation: - The routine is a direct analog of the MATLAB routine of the same name. - It has been there "forever". - It is probably widely used. This list of reasons make it sound like there is no way we can deprecate it; please read on however. Pros for deprecation: 1. The internal implementation of `interp2d` is not really meant for interpolation, and is known to produce wrong answers in some cases. 2. If it's gone, no functionality is missing from scipy.interpolate: it is a third equivalent way of wrapping FITPACK routines, along with bisplrep/bisplev and *BivariateSpline classes. 3. We now have better, more controllable, more predictable, and easier-to-understand routines for almost all interp2d functionality. 4. `interp2d` interface is confusing to users and error-prone and cannot be fixed while keeping backwards compatibility 5. Being a MATLAB name clone, it naturally attracts new users, who would otherwise use better, more suitable or more predictable routines. Interp2d has at least five open issues: see the FITPACK column of if it's gone, no functionality is missing from scipy.interpolate: it is a third equivalent way of wrapping FITPACK routines, along with bisplrep and *BivariateSpline classes. `interp2d` has at least five open issues, see e.g. the FITPACK column of https://github.com/orgs/scipy/projects/1. I'll now expand on the "pros" points above and try to summarize the open issues. 1. `interp2d` is a wrapper around the FITPACK spline-fitting routines. These are mainly targeted for spline *smoothing*, i.e. fitting a smoothing spline to noisy data. They do so by automagically selecting the spline knots to minimize a certain quality metric which tells how close the fit is to the data. There is a single control knob, the "smoothing parameter s", which can be set to zero to require the interpolation conditions, which influences the knot selection. When this knot construction procedure fails --- and it's most prone to failing for linear interpolation, which is the default for interp2d --- it returns a wrong result, either silently or with an extremely obscure warning. For details on this, see the discussion https://github.com/scipy/scipy/issues/2167. While that issue is about `BivariateSpline`s, interp2d has exactly the same set of problems and no way for a user to control the result. Also note https://github.com/scipy/scipy/issues/11309 , which shows essentially the same problem, only compounded by peculiarities of the interp2d interface. In passing, note a helpful code comment, "# TODO: surfit is really not meant for interpolation!", https://github.com/scipy/scipy/blob/main/scipy/interpolate/_interpolate.py#…, which git blame attributes to commit 7adfbbda57e from 2012 2. We ship at least three ways of wrapping 2D FITPACK routines: the functional interface, `bisplrep / bisplev`, the OOP interface `*BivariateSpline`s and interp2d. The main difference is the set of defaults--- which adds to user confusion, and that interp2d switches between essentially SmoothBivariateSpline and RectBivariateSpline, depending on the shapes of the input arrays. All in all, exactly zero functionality is missing from scipy.interpolate if interp2d is gone. 3. One confusing aspect of the FITPACK procedure is that there is no guarantee that the spline knots coincide with the data points. This makes sense for fitting and smoothing; for interpolation however, most users (I'm speculating here) expect that these coincide. Elsewhere in scipy.interpolate, the currently recommended way is - for regular grids in N-D, use RegularGridInterpolator. This does guarantee that knots=data points. - for scattered data in N-D, use `griddata`. Internally it is triangulation-bases via QHull. In 2D specifically, it allows constructing C1-continuous splines in addition to linear and nearest-neighbor interpolators. 4. First, as mentioned above, `interp2d` switches between different algorithms depending on the input shapes, https://github.com/scipy/scipy/issues/11309. For 1D input arrays it does scattered data fitting, equivalent to the SmoothBivariateSpline; for meshgrid inputs, it's equivalent to RectBivariateSpline. Second, also as mentioned, there is no guarantee that the spline knots coincide with the data. This breaks e.g. a natural expectation that a linear interpolator is bilinear in data. Third, it has an unusual behavior with respect to scalar inputs, see Warren's detailed comment in https://github.com/scipy/scipy/issues/16396#issuecomment-1154175268, which contains the note "I suspect this breaks anyone's mental model for what should happen" Fourth, it does not handle data well with increasing/decreasing ordering, see https://github.com/scipy/scipy/issues/16492. There is also an ample opportunity to silently get wrong results. Concluding, The routine's name seems to hint that it is _the_ way for 2D interpolation, even more so for users with prior Matlab experience. It is however not the recommended routine, it has too many problems, and scipy.interpolate would, I believe, be better with it gone. So the proposal is to deprecate it and add ample documentation for users to migrate to either equivalent or better functionality. I'm ready to do the legwork, and I'm planning to be around to help with user queries. Thoughts? Evgeni

6 18