Mailman 3 May 2023 - SciPy-Dev

Idiomatic import for SciPy
by Pamphile Roy May 26, 2023

May 26, 2023

Hi everyone, Since SciPy 1.9, we now have a lazy import mechanism which allows us to do `import scipy` and then work on any submodules without waiting time due to the import itself. NumPy and Pandas, just to name these, have set a precedence for an idiomatic way of importing their library. This is arguably an advantage for introspection, debugging, teaching, consistency or even communication purposes. Hence, I would like to propose that we also adopt an idiomatic import style for SciPy. If we would want to move forward with that idea, I would propose to make a brainstorming session with interested people to discuss the topic. This could for instance be a topic for next week’s community call. Cheers, Pamphile

3 2

Re: Linux build with Python 3.9
by Christoph Baumgarten May 25, 2023

May 25, 2023

Thanks Andrew, creating the venv outside of the scipy folder solved it. @Ralf I have tried building with the latest main-branch and the venv inside the scipy folder, that worked, so #18006 has indeed fixed this problem. Christoph On Wed, May 24, 2023 at 6:01 PM <scipy-dev-request(a)python.org> wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman3/lists/scipy-dev.python.org/ > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > Today's Topics: > > 1. Re: Linux build with Python 3.9 (Ralf Gommers) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 24 May 2023 10:25:49 +0200 > From: Ralf Gommers <ralf.gommers(a)gmail.com> > Subject: [SciPy-Dev] Re: Linux build with Python 3.9 > To: SciPy Developers List <scipy-dev(a)python.org> > Message-ID: > < > CABL7CQiJspqbfPQ5hHXGeEOf8y8hkm6aWpQ0uejbFshu0Viobg(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="00000000000004ea8205fc6c4157" > > On Wed, May 24, 2023 at 10:08 AM Andrew Nelson <andyfaff(a)gmail.com> wrote: > > > Is the venv located in the scipy source directory? Try installing the > venv > > elsewhere. > > > > That should fix it indeed. But also, this error should not be happening > anymore with in-tree venvs after https://github.com/scipy/scipy/pull/18006 > and this is the first report I've seen since that. Chris, if you see this > error with an up-to-date main branch, can you please open an issue with a > reproducer and assign it to me? > > The `export LD_LIBRARY_PATH=/usr/local/lib` part is also an indication that > something is off, that should never be needed. I'll have a look at removing > that from the CI job. > > Cheers, > Ralf > > > > > On Wed, May 24, 2023, 07:41 Christoph Baumgarten < > > christoph.baumgarten(a)gmail.com> wrote: > > > >> > >> Hi all, > >> > >> I wanted to replicate a test failure in the Meson build "Linux - 32 bit" > >> of my PR #17955. I tried to follow the steps in the following yml file > >> (L335): > >> > >> scipy/linux_meson.yml at main · scipy/scipy · GitHub > >> < > https://github.com/scipy/scipy/blob/main/.github/workflows/linux_meson.yml > > > >> > >> So essentially I did the following: > >> > >> - pull docker image and run it > >> - clone my repository > >> - run "python3.9 tools/openblas_support.py" > >> - copy the downloaded files to the indicated directories > >> - set PKG_CONFIG_PATH > >> - create venv and install packages > >> > >> These steps worked fine. When I then run "python dev.py build", I get an > >> error: > >> > >> scipy/meson.build:48:9: ERROR: Tried to form an absolute path to a dir > in > >> the source tree. You should not do that but use relative paths instead, > for > >> directories that are part of your project. [...] > >> > >> The detailed output is at the end of the email. The last two lines in > the > >> yml file are: > >> > >> python dev.py build && \ > >> LD_LIBRARY_PATH=/usr/local/lib python dev.py test > >> > >> I first wanted to build scipy before running certain tests. I am not > sure > >> if the error is related to not using LD_LIBRARY_PATH, I just ran "python > >> dev.py build". > >> > >> I hope that someone can help. Thanks > >> > >> Regards > >> > >> Christoph > >> > >> (test) [root@c73b212ff29a scipy]# python dev.py build > >> meson setup /home/scipy/build --prefix /home/scipy/build-install > >> The Meson build system > >> Version: 1.1.0 > >> Source dir: /home/scipy > >> Build dir: /home/scipy/build > >> Build type: native build > >> Project name: SciPy > >> Project version: 1.11.0.dev0 > >> C compiler for the host machine: cc (gcc 10.2.1 "cc (GCC) 10.2.1 > 20210130 > >> (Red Hat 10.2.1-11)") > >> C linker for the host machine: cc ld.bfd 2.35-5 > >> C++ compiler for the host machine: c++ (gcc 10.2.1 "c++ (GCC) 10.2.1 > >> 20210130 (Red Hat 10.2.1-11)") > >> C++ linker for the host machine: c++ ld.bfd 2.35-5 > >> Cython compiler for the host machine: cython (cython 0.29.33) > >> Host machine cpu family: x86 > >> Host machine cpu: i686 > >> Compiler for C supports arguments -Wno-unused-but-set-variable: YES > >> Compiler for C supports arguments -Wno-unused-function: YES > >> Compiler for C supports arguments -Wno-conversion: YES > >> Compiler for C supports arguments -Wno-misleading-indentation: YES > >> Compiler for C supports arguments -Wno-incompatible-pointer-types: YES > >> Library m found: YES > >> Fortran compiler for the host machine: gfortran (gcc 10.2.1 "GNU Fortran > >> (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)") > >> Fortran linker for the host machine: gfortran ld.bfd 2.35-5 > >> Compiler for Fortran supports arguments -Wno-conversion: YES > >> Checking if "-Wl,--version-script" : links: YES > >> Program cython found: YES (/home/scipy/test/bin/cython) > >> Program python3 found: YES (/home/scipy/test/bin/python) > >> Found pkg-config: /usr/bin/pkg-config (0.27.1) > >> Program pythran found: YES (/home/scipy/test/bin/pythran) > >> Run-time dependency threads found: YES > >> > >> scipy/meson.build:48:9: ERROR: Tried to form an absolute path to a dir > in > >> the source tree. > >> You should not do that but use relative paths instead, for > >> directories that are part of your project. > >> > >> _______________________________________________ > >> SciPy-Dev mailing list -- scipy-dev(a)python.org > >> To unsubscribe send an email to scipy-dev-leave(a)python.org > >> https://mail.python.org/mailman3/lists/scipy-dev.python.org/ > >> Member address: andyfaff(a)gmail.com > >> > > _______________________________________________ > > SciPy-Dev mailing list -- scipy-dev(a)python.org > > To unsubscribe send an email to scipy-dev-leave(a)python.org > > https://mail.python.org/mailman3/lists/scipy-dev.python.org/ > > Member address: ralf.gommers(a)googlemail.com > > >

1 0

Linux build with Python 3.9
by Christoph Baumgarten May 24, 2023

May 24, 2023

Hi all, I wanted to replicate a test failure in the Meson build "Linux - 32 bit" of my PR #17955. I tried to follow the steps in the following yml file (L335): scipy/linux_meson.yml at main · scipy/scipy · GitHub <https://github.com/scipy/scipy/blob/main/.github/workflows/linux_meson.yml> So essentially I did the following: - pull docker image and run it - clone my repository - run "python3.9 tools/openblas_support.py" - copy the downloaded files to the indicated directories - set PKG_CONFIG_PATH - create venv and install packages These steps worked fine. When I then run "python dev.py build", I get an error: scipy/meson.build:48:9: ERROR: Tried to form an absolute path to a dir in the source tree. You should not do that but use relative paths instead, for directories that are part of your project. [...] The detailed output is at the end of the email. The last two lines in the yml file are: python dev.py build && \ LD_LIBRARY_PATH=/usr/local/lib python dev.py test I first wanted to build scipy before running certain tests. I am not sure if the error is related to not using LD_LIBRARY_PATH, I just ran "python dev.py build". I hope that someone can help. Thanks Regards Christoph (test) [root@c73b212ff29a scipy]# python dev.py build meson setup /home/scipy/build --prefix /home/scipy/build-install The Meson build system Version: 1.1.0 Source dir: /home/scipy Build dir: /home/scipy/build Build type: native build Project name: SciPy Project version: 1.11.0.dev0 C compiler for the host machine: cc (gcc 10.2.1 "cc (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)") C linker for the host machine: cc ld.bfd 2.35-5 C++ compiler for the host machine: c++ (gcc 10.2.1 "c++ (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)") C++ linker for the host machine: c++ ld.bfd 2.35-5 Cython compiler for the host machine: cython (cython 0.29.33) Host machine cpu family: x86 Host machine cpu: i686 Compiler for C supports arguments -Wno-unused-but-set-variable: YES Compiler for C supports arguments -Wno-unused-function: YES Compiler for C supports arguments -Wno-conversion: YES Compiler for C supports arguments -Wno-misleading-indentation: YES Compiler for C supports arguments -Wno-incompatible-pointer-types: YES Library m found: YES Fortran compiler for the host machine: gfortran (gcc 10.2.1 "GNU Fortran (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)") Fortran linker for the host machine: gfortran ld.bfd 2.35-5 Compiler for Fortran supports arguments -Wno-conversion: YES Checking if "-Wl,--version-script" : links: YES Program cython found: YES (/home/scipy/test/bin/cython) Program python3 found: YES (/home/scipy/test/bin/python) Found pkg-config: /usr/bin/pkg-config (0.27.1) Program pythran found: YES (/home/scipy/test/bin/pythran) Run-time dependency threads found: YES scipy/meson.build:48:9: ERROR: Tried to form an absolute path to a dir in the source tree. You should not do that but use relative paths instead, for directories that are part of your project.

3 2

Nightly builds on Scientific Python org
by Pamphile Roy May 23, 2023

May 23, 2023

Hi everyone, Currently we are uploading our nightlies to: https://pypi.anaconda.org/scipy-wheels-nightly This organisation is going to be retired (there is no timeline at the moment) and there is an effort from Scientific Python (in collaboration with the previous team hosting wheels for us) to move to a new organisation: https://pypi.anaconda.org/scientific-python-nightly-wheels As you can see, a few packages already migrated and there are open PRs in most of the remaining libraries in our ecosystem. We are proposing to do the same for SciPy and Jarrod opened a PR to that end: https://github.com/scipy/scipy/pull/18504 You are welcome to join the discussion on the PR. Cheers, Pamphile

1 0

New keyword name for linalg.lu
by Ilhan Polat May 23, 2023

May 23, 2023

Dear all, In [1], I have proposed a new keyword for scipy.linalg.lu function, with prospective name "p_indices" as a bool switch. This will have the effect of having the permutation matrix P in A = P L U decomposition returned as a vector of indices such as [3, 1, 2, 0] meaning first and fourth row is swapped. With this you can still compute A through L[P, :] @ U without allocating a very costly 2D matrix of 1s and 0s. In tall arrays, say 400x10, the resulting 2D array P is 400x400 and in the vector form this reduces to 400x1. Hence, the switch to control this behavior is done by a new keyword and I don't know if "p_indices" is a nice one (doesn't feel like it). I considered, return_indices, perm_as_1d, p_as_1d, p_as_vec and bunch more but either too many underscores or too verbose. Thus, I'd like to ask for either validation or new ideas. We should maybe ping the NumPy audience for this too as this has no counterpart in NumPy. Thanks, ilhan [1] : https://github.com/scipy/scipy/pull/18358

7 15

Updates for the incomplete beta functions in scipy.special
by Warren Weckesser May 22, 2023

May 22, 2023

Hey all, In the pull request https://github.com/scipy/scipy/pull/17697, I propose the following changes for functions in `scipy.special` that are related to the incomplete beta function: * Use boost's `ibeta` as the backend for `special.betainc` and boost's `ibeta_inv` as the backend for `special.betaincinv`. This replaces the Cephes-based backend for these functions. * Add `special.betaincc` and `special.betainccinv` as wrappers of boost's `ibetac` and `ibetac_inv`, resp. These are the "complementary" versions of the functions. * Make the existing names `special.btdtr` and `special.btdtri` aliases of `special.betainc` and `special.betaincinv`, resp. The names for the new functions `betaincc` and `betainccinv` follow the pattern of the names of the existing incomplete gamma functions. That is, the names betainc, betaincinv, betaincc and betainccinv are analogous to the existing names gammainc, gammaincinv, gammaincc and gammainccinv. Because this introduces the new functions `betaincc` and `betainccinv`, it is more than a basic maintenance change, so feedback from the mailing list would be appreciated. Warren

2 2

optimize.Bounds - does lb need to be less than ub?
by Andrew Nelson May 18, 2023

May 18, 2023

Hi all, in https://github.com/scipy/scipy/pull/18483 `Bounds` is being modified slightly. Currently there's a check in there: ``` if (self.lb > self.ub).any(): raise ValueError("An upper bound is less than the corresponding lower bound.") ``` I would personally find it useful if Bounds was more robust in this area, and was able to deal with lb entries that were greater than the ub. I would like Bounds to have a clear up code with something like: ``` _lb = np.minimum(self.lb, self.ub) _ub = np.maximum(self.lb, self.ub) self.lb = _lb self.ub = _ub ``` What are peoples thoughts on this? A. _____________________________________ Dr. Andrew Nelson _____________________________________

4 3

tol, atol and rtol behavior of scipy.sparse.linalg solvers
by Ilhan Polat May 13, 2023

May 13, 2023

Dear all, Yet another mail about upcoming changes, so apologies for the seemingly repeating message lately. In [1], we started to remove all the FORTRAN code and rewrote all solvers in pure Python for various reasons too long to enumerate here. As a long waiting issue we want to also address the suboptimal state of providing tolerances for these solvers. As I wrote in [2], I'd like to mention it again to ask for feedback in case anyone has better ideas Currently, we have the following situation; We have a tol and a strange looking atol keywords. What internally happens is that all the inputs are now passed to a helper function called get_atol https://github.com/scipy/scipy/blob/maintenance/1.10.x/scipy/sparse/linalg/… 1. User didn't provide atol then it first maps to None and that causes a warning and internally atol becomes legacy. This means: a. tol value is checked against the residual norm Ax - b. Note that matvec operation is costly. If the residual norm is already less than tol we exit. b. Then b norm is checked against 0. Since tol*norm(b) won't do any good we return tol c. And the typical case is to return tol*norm(b). 2. User passed legacy all the same as above except the warning. 3. If atol is actually set to something, then we return max(atol, tol*norm(b)). Why max is used I don't understand. Typically, atol, rtol pair is used as tol = atol + rtol*<some metric>. This is more or less what everybody expects from this pair and their naming is chosen to reflect this. But I'm a bit lost in all the issues I could read. What I propose is the following, 1. Make default atol=0. 2. If atol is set to string, fire a warning saying this is going away in version 1.13 (hence make the current deprecation on a definite schedule and assuming I can make it to 1.11 release). These are the users probably wanted to keep old behavior. Document how it can be done to match old behavior via atol, rtol. 3. Introduce a new keyword rtol and map current tol to rtol. Set tol=None as the default to weed out the ones who actually set tol value and fire deprecation warnings for them to use rtol. 4. Use the formula tol = atol + rtol*norm(b) All feedback is welcome. [1] : https://github.com/scipy/scipy/pull/18391 [2] : https://github.com/scipy/scipy/issues/15738

4 10

Re: ENH: Add scipy.stats.multivariate_laplace #18186
by Daniel Schmitz May 13, 2023

May 13, 2023

1 0

Added Shuffled Complex Evolution (SCE) to SciPy.optimize
by Matthias Cuntz May 10, 2023

May 10, 2023

Dear all, Andrea Gavana reported results for global optimisation algorithms in 2013 (http://infinity77.net/global_optimization/) and in 2021 (http://infinity77.net/go_2021/). I had provided him with a version of the Shuffled Complex Evolution (SCE) optimiser (Duan et al., Water Resour Res 1992, doi: 10.1029/91WR02985). It performed exceptional well in the 2013 exercise and there was interest on the scipy-dev mailing list. While SCE did not score as high anymore in the 2021 exercise, it is still our first algorithm of choice for our hydrologic models and our land-surface models (written in Fortran and C). We routinely optimise more than 50 parameters (e.g. Cuntz et al., Water Resour Res 2015, doi: 10.1002/2015WR016907). So I implemented it in scipy.optimize, following very closely the implementation of differential evolution: https://github.com/scipy/scipy/pull/18436 However, the code is developed since 2013 so might have some “historic” code in it. But at least lots of bugs were removed since then ;-) I ran the global benchmark suite of scipy (results are in the PR). SCE performs reasonably well, comparable with differential evolution but needing only about 1/10-th of model runs. Dual Annealing and SHGO had higher success rates. But DA also needed about 10 times more model runs. SHGO had high success rates and needed little model evaluations. I will try this on our environmental models next. The implementation of SCE has some extras that we really like: - It can write out restart files for SCE. When optimising expensive models on computing clusters, it happens that the job used its allocated time but optimisation hasn’t finished yet. One can simply launch the job again setting `restart=True` in SCE. - Parameters can be sampled not only with a uniform distribution but also logarithmic. Sampling a parameter with a uniform distribution gives suboptimal solutions if the parameter can vary orders of magnitude. Imagine you have bounds [1e-9, 1e-3]. All numbers below 1e-4 are very unlikely if sampled uniformly. It is much better to sample the parameter in log-space then (e.g. Mai, J Hydrolo 2023, 10.1016/j.jhydrol.2023.129414). - One can mask parameters. We have to write config files for our environmental models (e.g. Fortran namelists), which need, of course, all parameters and not only the parameters to optimise. One might have determined already non-influential (insensitive) parameters and just wants to mask them from the optimisation. Or one wants to fix a parameter that is correlated with another parameter. - args and kwargs can be passed to the objective function. Thanks in advance for your feedback, Matthias

1 0