Mailman 3 June 2015 - SciPy-Dev

On deprecating `stats.threshold`
by Abraham Escalante July 6, 2015

July 6, 2015

Hello all, As part of the ongoing scipy.stats improvements we are pondering the deprecation of `stats.threshold` (and its masked array counterpart: `mstats.threshold`) for the following reasons. - The functionality it provides is nearly identical to `np.clip`. - Its usage does not seem to be common (Ralf made a search with searchcode <https://searchcode.com/>; it is not used in scipy as a helper function either). Of course, before we deprecate anything, we would like to know if anyone in the community is a regular user of this function and/or if you guys may have a use case where it may be preferable to use `stats.threshold` over `np.clip`. Please reply if you have any objections to this deprecation. You can find the corresponding PR here: gh-4976 <https://github.com/scipy/scipy/pull/4976> Regards, Abraham. PS. For reference, both `np.clip` and `stats.threshold` replace the values outside a threshold from an array_like input. The difference is that `stats.threshold` replaces all values below the minimum or above the maximum with the same new value whereas `np.clip` uses the minimum to replace those below and the maximum for those above. Example: >>> a = np.arange(10) >>> a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> np.clip(a, 3, 7) array([3, 3, 3, 3, 4, 5, 6, 7, 7, 7]) >>> stats.threshold(a, 3, 7, -1) array([-1, -1, -1, 3, 4, 5, 6, 7, -1, -1])

6 11

ENH: API: DEP: Added parallelization for differential evolution
by Pavel Ponomarev June 26, 2015

June 26, 2015

Hello Here are the files which enable parallelization for DE using MPI or joblib. The parallelization is realized in the same way as in emcee. Please check https://github.com/scipy/scipy/compare/master...pavelponomarev:DE_with_para… Discussion started in https://github.com/scipy/scipy/issues/4864 BR, Pavel Ponomarev

2 1

scipy.optimize.fmin_cg
by Phillip Feldman June 24, 2015

June 24, 2015

The optimization algorithm of scipy.optimize.fmin_cg appears to be quite powerful, but the interface is rather limited. In particular, it would be great if one could specify the `xtol` and `ftol` convergence criteria that most of the other SciPy optimizers accept. Phillip

1 0

Weekly Summary 2015/06/23
by Abraham Escalante June 23, 2015

June 23, 2015

Hello all, This week we had a lot of merged PRs and closed issues for `scipy.stats` so the project is seeing some progress and I'm feeling good with the decision to change the approach. Here is a summary of the most important things: *Week 4 topics:* - Trimmed statistics functions PR has been merged. - nan checks: An agreement seems close. - Deprecation of (M)ANOVA `f_value*` functions. - Deprecation of `threshold` is still in discussion. - 'alternative' keyword addition to `binom_test` and `mannwhitneu`. *Week 5 topics:* - nan checks discussion. - Deprecation of `threshold` discussion. - Resume some previously started tasks: - ppcc_max. - 'alternative' keyword addition. - New batch of PRs: - `fligner`, `bartlett`, `ansari`, `shapiro`, `variation`, `moment`, `kruskal` and `kendalltau` are the likeliest prospects. I will start with as many as possible. As usual, for your convenience here are some links of interest so that you can contribute and/or follow the progress of the project. - Open Pull Requests <https://github.com/scipy/scipy/pulls/aeklant> - Open StatisticsCleanup issues <https://github.com/scipy/scipy/issues?page=2&q=is%3Aopen+is%3Aissue+milesto…> Regards, Abraham.

1 0

Fwd: Increasing the maximum number of points in the spatial.Delaunay triangulation
by Cantwell Carson June 22, 2015

June 22, 2015

I originally wrote this to the scipy-user group, but I think it might be more appropriate here. I am working on an image processing algorithm that relies on the Delaunay triangulation of a 3d image. Unfortunately, I need to work on images with more than 2**24 points that go into the Delaunay triangulation. This is problematic because the Qhull routine labels the input vertices with a 24-bit label. I have tried editing the source code for qhull here in libqhull.h, lines 352, 380 and 663 - 664: unsigned id:32; /* unique identifier, =>room for 8 flags, bit field matches qh.ridge_id */ ... unsigned id:32; /* unique identifier, bit field matches qh.vertex_id */ ... unsigned ridge_id:32; /* ID of next, new ridge from newridge() */ unsigned vertex_id:32; /* ID of next, new vertex from newvertex() */ poly.c, lines 569, 1019 - 1023: long ridgeid; ... if (qh ridge_id == 0xFFFFFFFF) { qh_fprintf(qh ferr, 7074, "\ qhull warning: more than %d ridges. ID field overflows and two ridges\n\ may have the same identifier. Otherwise output ok.\n", 0xFFFFFFFF); and poly2.c, lines 2277 - 2279: if (qh vertex_id == 0xFFFFFFFF) { qh_fprintf(qh ferr, 6159, "qhull error: more than %d vertices. ID field overflows and two vertices\n\ may have the same identifier. Vertices will not be sorted correctly.\n", 0xFFFFFFFF); This compiles fine, but so far, this produces a memory leak (as far as I can tell) when I put more than 2**24 points into the algorithm. I have emailed Brad Barber, who indicated that this changing the ridgeid field was possible. I am also working on a divide and conquer approach, but am having trouble deleting the old Delaunay object before instantiating the next one. I can send that code along if there is no good way to increase the number of points in the algorithm. I think it would be useful to have an algorithm that can accept more points, given the increased speed and processing power of modern computers. Thanks for your help.

2 3

Is SciPy-User down?
by Joseph C. Slater June 22, 2015

June 22, 2015

I apologize for using the wrong list- but why I'm not using the correct list is self-explanatory. I don't know how to contact the maintainer of scipy-user. I sent to the list Friday, and twice since. gmane (http://news.gmane.org/gmane.comp.python.scientific.user) doesn't show any activity, and I haven't received any emails from the list since last week. Thank you, Joe

2 3

(M)ANOVA, and deprecating stats.f_value* functions
by Ralf Gommers June 22, 2015

June 22, 2015

Hi all, In scipy.stats there are three functions that calculate various F-statistics for inputs obtained from univariate or multivariate ANOVA. These are f_value, f_value_multivariate and f_value_wilks_lambda: https://github.com/scipy/scipy/blob/master/scipy/stats/stats.py#L4603-L4683 The problem with those is that they're not very useful standalone. f_value implements a statistic that's also calculated and given as a return by f_oneway (which does one-way ANOVA). The other two functions are related to multivariate ANOVA, for which scipy.stats doesn't provide any functionality. At the moment Statsmodels provides a lot more ANOVA functionality than scipy.stats does, and I agree with Josef [1, 2] that adding new functionality in this area to Statsmodels would fit better than adding it to Scipy. There's also a recent proposal [3] for M-way repeated ANOVA to be added to scipy.stats. That could be added to Statsmodels instead (my preference). If we do want to add it to Scipy, we need to have a clear list of what else is needed to create a coherent set of functions in this area. Thoughts? Ralf [1] https://github.com/scipy/scipy/issues/650 [2] https://github.com/scipy/scipy/issues/660 [3] https://github.com/scipy/scipy/issues/4913

4 4

Signal Smooth
by Nicolas Petitclerc June 21, 2015

June 21, 2015

Hi, Coming from IDL, I've always wondered why there was no 'smooth' function in SciPy, seems like a standard/common operation. But I found an implementation in the Cookbook: http://wiki.scipy.org/Cookbook/SignalSmooth that does what I would expect such a function to do. But I found an indexing bug and a few usability and coding style improvement to the Cookbook. My version of the function is in attachment. Any reason why this function is not part of Scipy? Thanks, Nicolas

6 11

`trim1` and 'trimboth' backwards incompatible change
by Abraham Escalante June 20, 2015

June 20, 2015

Hello all, `trim1` and `trimboth` currently trim items from one or both tails (respectively) of an array_like input but they do so without sorting the items. It has been discussed that a function such as that does not make much sense so that behaviour is being changed. Now the items will be sorted prior to trimming. As any other backwards incompatible change, we would like to hear if anyone has an opinion on this matter before we commit to it. You can find the changes in PR 4910 <https://github.com/scipy/scipy/pull/4910> Regards, Abraham.

4 4

Clarification sought on Scipy Numpy version requirements.
by Charles R Harris June 19, 2015

June 19, 2015

Hi All, I'm looking to change some numpy deprecations into errors as well as remove some deprecated functions. The problem I see is that SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really, old. So the question is, does "support" mean compiles with earlier versions of Numpy ? If that is the case there is very little that can be done about deprecation. OTOH, if it means Scipy can be compiled with more recent numpy versions but used with earlier Numpy versions (which is a good feat), I'd like to know. I'd also like to know what the interface requirements are, as I'd like to remove old_defines.h Chuck

1 0