Mailman 3 September 2020 - SciPy-Dev

proposal to drop Python 3.6 and NumPy 1.14 / 1.15
by Ralf Gommers 15 Nov '20

15 Nov '20

Hi all, Python 3.9 is coming out before the next SciPy release, and NEP 29 recommends to support only Python >=3.7 and NumPy >= 1.16 from last month onwards [1]. I think supporting 3 Python versions and 5 NumPy versions in SciPy 1.6.0 is easily enough. That would also bring us back in line with SciPy on conda-forge, which built against NumPy 1.16 for 1.5.2. Any objections? Cheers, Ralf [1] https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table

10 21

Adding non-parametric methods to scipy.stats
by Romain Jacob 27 Oct '20

27 Oct '20

Hello everyone, I have been working for some time on the implementation of non-parametric methods to compute confidence intervals for percentiles. There are some very interesting results in the literature (see e.g. a nice pitch in [1]) which I think it would be great to add to SciPy to make them more readily available. It also seems to be rather in line with "recent" discussions of the roadmap for scipy.stats [2]. I would be interested in contributing this. What do you think? Cheers, -- Romain [1] https://ieeexplore.ieee.org/document/6841797 [2] https://github.com/scipy/scipy/issues/10577 -- Romain Jacob Postdoctoral Researcher ETH Zurich - Computer Engineering and Networks Laboratory www.romainjacob.net <https://www.romainjacob.net/> @RJacobPartner <https://twitter.com/RJacobPartner> Gloriastrasse 35, ETZ G75 8092 Zurich +41 7 68 16 88 22

6 18

Re: [SciPy-Dev] Discontinued Rackspace Open Compute Discount
by Robert Lucente 05 Oct '20

05 Oct '20

Several CI services are free for open source projects although there are throughput limits https://about.gitlab.com https://github.com/features/actions https://travis-ci.org https://www.appveyor.com -----Original Message----- >From: Olivier Grisel <olivier.grisel(a)ensta.org> >Sent: Feb 8, 2020 11:30 AM >To: SciPy Developers List <scipy-dev(a)python.org> >Subject: [SciPy-Dev] Discontinued Rackspace Open Compute Discount > >Hi all, > >I received an email end of January that I just discovered today >telling me that the Rackspace Open Compute Discount grant we use to >host http://wheels.scipy.org for nightly builds and release management >of our community has been discontinued on December 31st 2019. > >I would like to take down the Rackspace hosted cloudfiles as soon as >possible but we need to find a replacement as this will break all the >CIs that rely on nightly wheels and also all the wheel release >infrastructure for several project in the ecosystem. > >Apparently the fees for January and the beginning of February already >sum to $936.51. This is a large cost because there were a bunch of >unused left-over block volumes that I already deleted. > >To host wheels for nightly builds and release automation, a possible >alternatives include using the github release pages but they are not >really designed for this kind of workflow. > >Alternatively there is a PyPI repo hosting feature in Azure Pipelines >in public preview but I am not sure how it works, if it's free for >open source projects and if we could use it as a replacement for the >Rackspace cloud storage we currently use to host >http://wheels.scipy.org . > >https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/pypi?view… > >Important: if you have uploaded any important files to one of the >cloud file storage / blob containers on Rackspace, please download a >copy as soon as possible. > >But please do not download things you do not really want to keep to >avoid adding any unnecessary bandwidth costs. If you have file write >permission on one of those containers, please delete the files you own >and do not need anymore. > >Feel free to join this room to discuss the matter interactively: >https://gitter.im/scikit-learn/nightly-build-hosting > >Feel free to reply to this email on the scipy-dev mailing list if you >have suggestions for setting up a free replacement. > >-- >Olivier >http://twitter.com/ogrisel - http://github.com/ogrisel >_______________________________________________ >SciPy-Dev mailing list >SciPy-Dev(a)python.org >https://mail.python.org/mailman/listinfo/scipy-dev

12 25

Issues Inserting Graphical Overlay Using Matplotlib Patches
by Stephen Malcolm 01 Oct '20

01 Oct '20

Hello All, I'm having some trouble adding a graphical overlay i.e. an ellipse onto my plot. I wish to do this, as I need to explain/ portray the mean, standard deviation and outliers. And hence evaluate the suitability of the dataset. Could you please let me know what code I'm missing/ or need to add, in order to insert this ellipse? I have no trouble plotting the data points and the mean using this code, however, the ellipse (width and height/ standard deviation) doesn't appear. I have no errors, instead, I'm getting a separate graph (without data points or ellipse) below the plotted one. Please find my code below: #pandas used to read dataset and return the data #numpy and matplotlib to represent and visualize the data #sklearn to implement kmeans algorithm import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.cluster import KMeans #import the data data = pd.read_csv('banknotes.csv') #extract values x=data['V1'] y=data['V2'] #print range to determine normalization print ("X_max : ",x.max()) print ("X_min : ",x.min()) print ("Y_max : ",y.max()) print ("Y_min : ",y.min()) #normalize values mean_x=x.mean() mean_y=y.mean() max_x=x.max() max_y=y.max() min_x=x.min() min_y=y.min() for i in range(0,x.size): x[i] = (x[i] - mean_x) / (max_x - min_x) for i in range(0,y.size): y[i] = (y[i] - mean_y) / (max_y - min_y) #statistical analyis using mean and standard deviation import matplotlib.patches as patches mean = np.mean(data, 0) std_dev = np.std(data, 0) ellipse = patches.Ellipse([mean[0], mean [1]], std_dev[0]*2, std_dev[1]*2, alpha=0.25) plt.xlabel('V1') plt.ylabel('V2') plt.title('Visualization of raw data'); plt.scatter(data.iloc[:, 0], data.iloc[:, 1]) plt.scatter(mean[0],mean[1]) plt.figure(figsize=(6, 6)) fig,graph = plt.subplots() graph.add_patch(ellipse) Kind Regards, Stephen

3 5

How to compile Scipy without optimisations -O0?
by Blair Azzopardi 29 Sep '20

29 Sep '20

Hi I'm trying to debug some scipy C code between two platforms and I'm struggling with each platform optimizing away different variables while stepping through. I'd like to add -O0 flag and rebuild but when I do something along the lines of: OPT=-O0 python setup.py build_ext --inplace I see the flag added but it's superseded by later flags (default optimization -O2), ie. .. extra options: '-std=c++14' x86_64-linux-gnu-g++: /tmp/tmpnf9pc9rw/main.c C compiler: x86_64-linux-gnu-g++ -pthread -Wno-unused-result -Wsign-compare -O0 -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC .. I've also tried creating a site.cfg with the following: [ALL] extra_compile_args = -O0 where I see the buildlog picking this option up but it appears to have no effect on the compiler flags. FOUND: libraries = ['openblas', 'openblas'] library_dirs = ['/usr/lib/x86_64-linux-gnu'] language = c define_macros = [('HAVE_CBLAS', None)] extra_compile_args = ['-O0'] Finally, I also tried building via runtests.py but this appears to have the same effect as setting the OPT environment variable above (eg python runtests.py -g -b). Short of sifting through the numpy distutils package and hacking the flags in, I was wondering if there's a recommended way to do this in Scipy? I'm sure I'm just being daft. Kind regards, Blair

3 4

Re: [SciPy-Dev] SciPy-Dev Digest, Vol 203, Issue 9
by frank.home＠sbcglobal.net 28 Sep '20

28 Sep '20

Fair enough, we don't have to present an example that would then be a little bloated with details of catching infeasibilities. But it seems like a concise statement in the documentation that says the objective function is allowed to return np.inf, maybe a dozen words or so, might help some people. Otherwise, one only knows by reading the source code. Thank you for the reference, I am enjoying reading it :) I am also realizing that I overloaded the meaning of "infeasible" in my use of the term, which means I should come up with another term. For what it is worth, in the problems I am currently working on, the objective function is actually the solution of a set of coupled differential equations, and the variables over which the optimization takes place are parameters in the differential equations and other parameters in the boundary conditions. In an inner loop, the system of differential equations are solved, and then a figure of merit is reported back to the differential evolution algorithm. For some sets of the parameters, there is no solution to the set of equations. So "infeasibility" in the sense I am using it means the objective function does not exist for the given set of parameters, not that the objective function can be evaluated but constraints are violated. Sorry for the overloading of this term. One could argue that the "infeasible" space in the sense I am using the term should be mapped out before an optimization, and then encoded as bounds on the independent variables. But for sufficiently complicated problems, that is difficult to do. That is why it was great to figure out how to do it using the powerful scipy differential evolution optimizer. Best regards,Frank Torres On Saturday, September 26, 2020, 09:00:25 AM PDT, <scipy-dev-request(a)python.org> wrote: Send SciPy-Dev mailing list submissions to scipy-dev(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/scipy-dev or, via email, send a message with subject or body 'help' to scipy-dev-request(a)python.org You can reach the person managing the list at scipy-dev-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of SciPy-Dev digest..." Today's Topics: 1. Re: SciPy-Dev Digest, Vol 203, Issue 6 (Andrew Nelson) The documentation does say that if "If callback returns True, then the minimization is halted (any polishing is still carried out)", so the documentation already indicates0 that the user can control when the minimisation halts. However, it's not mentioned in the documentation that the objective function is allowed to return `np.inf` if the solution is impossible.When constraints are employed the solver will attempt to reduce the magnitude of the constraint violation. It's worth reading the Lampinen paper (Lampinen, J., A constraint handling approach for the differential evolution algorithm. Proceedings of the 2002 Congress on Evolutionary Computation. CEC’02 (Cat. No. 02TH8600). Vol. 2. IEEE, 2002.) to see how that's applied. > We did learn to turn those warnings into errors and catch them with try-except blocks, and then return np.inf if the exceptions happened. Even that might be useful for some people to see described in documentation :) Hmm, that's a reasonably simple programming approach to take, and I don't think it would aid the documentation because its clarity would be degraded._______________________________________________ SciPy-Dev mailing list SciPy-Dev(a)python.org https://mail.python.org/mailman/listinfo/scipy-dev

1 0

Re: [SciPy-Dev] SciPy-Dev Digest, Vol 203, Issue 6
by frank.home＠sbcglobal.net 26 Sep '20

26 Sep '20

Hello, My colleague and I discussed this and now agree that this should be handled in the function being optimized, not in the scipy differential_evolution code. We do feel, however, that it would be good to add documentation for handling such cases, especially when the optimum winds up being at a boundary where np.inf occurs. We would be happy to submit draft documentation to add to https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differe… if that is of interest. Let me explain some of the details of issues we ran across. Interestingly, in a side conversation we learned that a similar thing came up in issue #7799/7819. In our case, the function is actually a solution of a set of coupled boundary value ode's. For some values of the boundary condition, there are no feasible solutions, but it would be difficult in many cases to explicitly map out boundaries with all infeasible regions and add those as bounds. In our initial attempts to use the differential evolution solver, the scipy code failed when our solver of the ode's gave warnings due to infeasibilities. We did learn to turn those warnings into errors and catch them with try-except blocks, and then return np.inf if the exceptions happened. Even that might be useful for some people to see described in documentation :) This is a case of infeasible solutions. But another problem we encountered is that sometimes the optimum then winds up being very near or actually at the boundary with an infeasible region. In such a case, the differential evolution code does not converge or behave well because each time it picks a new population, some points are in the infeasible region (since the "answer" is at the border), and therefore there are np.inf values that continue to get returned and thereby interfere with the concept of shrinking the population towards the answer. We handled this by using a callback function that gave early termination if the "best value" is not changing. I feel that would also be useful to document. Initially we were proposing to add to the scipy code itself to recognize this is happening, and do something about it internally in scipy. But we now agree that having this different convergence criterion is better handled with a callback function, as it is specific to the problem being encountered and is consistent with the availability of the callback function. A similar problem also occurred in issue #7799/7819 for basinhopping optimization. In that case, there was not a boundary where solutions for the function became infeasible, but rather there was an explicit constraint that wound up causing the solution to be exactly at the constraint boundary. The effect was the same as above because the solution was no longer a local minimum. Returning np.inf when appropriate and adding a callback function that provides early termination when the objective function is not changing works for that case as well. Would it be of interest to have this added to the documention? Best regards,Frank On Wednesday, September 16, 2020, 02:59:09 PM PDT, scipy-dev-request(a)python.org <scipy-dev-request(a)python.org> wrote: Send SciPy-Dev mailing list submissions to scipy-dev(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/scipy-dev or, via email, send a message with subject or body 'help' to scipy-dev-request(a)python.org You can reach the person managing the list at scipy-dev-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of SciPy-Dev digest..." Today's Topics: 1. feature proposal for scipy.optimize.differential_evolution (frank.home(a)sbcglobal.net) 2. Re: feature proposal for scipy.optimize.differential_evolution (Matt Haberland) 3. Re: feature proposal for scipy.optimize.differential_evolution (Andrew Nelson) Hello, In using scipy.optimize.differential_evolution, we found that it is useful to modify the behavior for problems where solutions of the equations being optimized are not feasible in some of the space being explored. Rather than having the code throw exceptions and then abort, we use the approach of returning np.inf for the objective function in infeasible regions. We are interested in turning this approach into a feature of scipy.optimize.differential_evolution and submitting a pull request, and would like to know if people think this would be a useful contribution. Our approach would be to add a boolean parameter to scipy.optimize.differential_evolution for selecting whether to handle lack of ability to find a solution in the above manner. The default for the parameter would be to not do this, i.e. run scipy.optimize.differential_evolution as it now works. Please let us know what you think, and if you would be interested in reviewing code, please let us know that as well! Thank you,Frank Torres Sounds useful to me to have that option. Do you know of other software with a similar option? On Wed, Sep 16, 2020 at 10:09 AM frank.home(a)sbcglobal.net <frank.home(a)sbcglobal.net> wrote: Hello, In using scipy.optimize.differential_evolution, we found that it is useful to modify the behavior for problems where solutions of the equations being optimized are not feasible in some of the space being explored. Rather than having the code throw exceptions and then abort, we use the approach of returning np.inf for the objective function in infeasible regions. We are interested in turning this approach into a feature of scipy.optimize.differential_evolution and submitting a pull request, and would like to know if people think this would be a useful contribution. Our approach would be to add a boolean parameter to scipy.optimize.differential_evolution for selecting whether to handle lack of ability to find a solution in the above manner. The default for the parameter would be to not do this, i.e. run scipy.optimize.differential_evolution as it now works. Please let us know what you think, and if you would be interested in reviewing code, please let us know that as well! Thank you,Frank Torres _______________________________________________ SciPy-Dev mailing list SciPy-Dev(a)python.org https://mail.python.org/mailman/listinfo/scipy-dev -- Matt HaberlandAssistant ProfessorBioResource and Agricultural Engineering08A-3K, Cal Poly On Thu, 17 Sep 2020 at 03:09, frank.home(a)sbcglobal.net <frank.home(a)sbcglobal.net> wrote: In using scipy.optimize.differential_evolution, we found that it is useful to modify the behavior for problems where solutions of the equations being optimized are not feasible in some of the space being explored. Rather than having the code throw exceptions and then abort, we use the approach of returning np.inf for the objective function in infeasible regions. We are interested in turning this approach into a feature of scipy.optimize.differential_evolution and submitting a pull request, and would like to know if people think this would be a useful contribution. Our approach would be to add a boolean parameter to scipy.optimize.differential_evolution for selecting whether to handle lack of ability to find a solution in the above manner. The default for the parameter would be to not do this, i.e. run scipy.optimize.differential_evolution as it now works. As the code currently exists the user's objective `func` is allowed to return `np.inf` if it finds itself within a region that is infeasible, differential_evolution doesn't throw exceptions or abort if `func` does that. One can also mark regions as infeasible with constraint functions. So it's not clear to me what feature is missing. Do you have a demonstration? _______________________________________________ SciPy-Dev mailing list SciPy-Dev(a)python.org https://mail.python.org/mailman/listinfo/scipy-dev

2 1

Pythran 0.9.8 - memes tra
by Serge Guelton 24 Sep '20

24 Sep '20

Hi Folks, New Pythran release, tagged on github, available on PyPI and conda, and described here: http://serge-sans-paille.github.io/pythran-stories/pythran-097-memes-tra.ht… Enjoy!

1 0

feature proposal for scipy.optimize.differential_evolution
by frank.home＠sbcglobal.net 17 Sep '20

17 Sep '20

Hello, In using scipy.optimize.differential_evolution, we found that it is useful to modify the behavior for problems where solutions of the equations being optimized are not feasible in some of the space being explored. Rather than having the code throw exceptions and then abort, we use the approach of returning np.inf for the objective function in infeasible regions. We are interested in turning this approach into a feature of scipy.optimize.differential_evolution and submitting a pull request, and would like to know if people think this would be a useful contribution. Our approach would be to add a boolean parameter to scipy.optimize.differential_evolution for selecting whether to handle lack of ability to find a solution in the above manner. The default for the parameter would be to not do this, i.e. run scipy.optimize.differential_evolution as it now works. Please let us know what you think, and if you would be interested in reviewing code, please let us know that as well! Thank you,Frank Torres

3 2

Re: [SciPy-Dev] Disjoint set / union find data structure
by Peter Larsen 13 Sep '20

13 Sep '20

>> On Mon, Jul 20, 2020 at 2:37 PM Peter Larsen <peter.mahler.larsen at gmail.com>wrote: >> In many projects I find myself needing a disjoint set data structure >> https://en.wikipedia.org/wiki/Disjoint-set_data_structure. >> It would be very convenient if we had an implementation in SciPy. >> > >That seems like a reasonable and useful thing to add. > > >> The codebase already contains two implementations, albeit not publicly >> accessible ones: >> >> https://github.com/scipy/scipy/blob/1d8b34b47d2b881ea904f4c9bf4c49b3fa36b29… >> >> https://github.com/scipy/scipy/blob/8dba340293fe20e62e173bdf2c10ae208286692… >> >> If there are no objections to its inclusion in SciPy I will write a PR. I >> tentatively propose to put it in scipy.cluster but other suggestions are >> welcome. >> > >Cluster, spatial and sparse all could make sense. If we foresee using this >internally in all those modules, I think we'd have to put it in `sparse` or >in `_lib`. The reason: no further dependencies between modules. cluster >already depends on spatial, and spatial depends on sparse. > >Cheers, >Ralf I made a pull request for a disjoint set here: https://github.com/scipy/scipy/pull/12600 Thanks to help from reviewers it is in quite good shape. The only thing missing is someone with sufficient "ownership" of the `scipy.cluster.hierarchy` module to sign off on its inclusion there. To allow internal reuse without creating further dependencies, the actual disjoint set code is in `scipy._lib` but with the public interface in `scipy.cluster.hierarchy`. Peter

2 1