Mailman 3 June 2015 - SciPy-Dev

Weekly Summary 2015/06/17
by Abraham Escalante 18 Jun '15

18 Jun '15

Hello all, I am a little late for this week's weekly report, I know. This past week I realised I needed to change the course of action because the bulk of the project is comprised of issues and the regular PR can usually take two weeks even if they go smoothly; and a lot longer if not so smooth. Long story short, I decided to start working on as many PRs as possible to keep the waiting time between feedback cycles to a minimum. Otherwise, the project's completion could be jeopardised. Since there will be many open PRs at a time and I expect to be working on several issues through several weeks, I will try to mention here only the most relevant ones and I encourage you to check the current status of: - Open Pull Requests <https://github.com/scipy/scipy/pulls/aeklant> - Open StatisticsCleanup issues <https://github.com/scipy/scipy/issues?page=2&q=is%3Aopen+is%3Aissue+milesto…> *Week 3 topics:* - Trimmed statistics functions. - nan checks: How to deal with nan values in scipy.stats? - Deprecation of `find_repeats`. - 'alternative' keyword addition to `binom_test` and `mannwhitneu`. *This week's topics:* - Resolution of nan handling discussion. - Deprecation of (M)ANOVA `f_value*` functions. - Deprecation of `threshold`. Again, this is not a comprehensive list. For more information regarding any of the topics feel free to browse through the open issues or PRs. You can also contact me and I will be more than happy to let you know the details of whichever of those topics you might find interesting (or at least point you in the direction of someone who knows what he's talking about). Regards, Abraham.

1 0

Re: [SciPy-Dev] [Numpy-discussion] How to limit cross correlation window width in Numpy?
by Honi Sanders 17 Jun '15

17 Jun '15

I will also repeat what I said in response on Github (discussions at: https://github.com/scipy/scipy/issues/4940, https://github.com/numpy/numpy/issues/5954): I do want a function that computes cross-correlograms, however the implementation is exactly the same for cross-correlograms as convolution. Not only that, is the numpy.correlate() function not for computing cross-correlograms? Maxlag and lagstep still make sense in the context of convolution. Say you have a time series (this is not the best example) of rain amounts and you have a kernel for plant growth given rain in the recent past. Your time series is the entire year, but you are only interested in the plant growth during the months of April through August. Not only that, you do not need daily readout of plant growth; weekly resolution is enough for your needs. You wouldn’t want to compute the convolution for the entire time series, instead you would do: numpy.convolve(rain, growth_kernel, (april, september, 7), lagvec) and get lagvec back with the indices of the sundays in april through august, and a return vector with the amount of plant growth on those days. I don’t really think it would be good to add an entirely new function to scipy.signal. It was already hard enough as a new user trying to figure out which of the five seemingly identical functions in numpy, scipy, and matplotlib that I should be using. Besides, if all of these functions are essentially doing the same computation, there should only be a single base implementation that they all use, so that 1) the learning curve is decreased and 2) that any optimizations be passed on to all of the functions instead of having to be independently reimplemented several times. So, even if we do decide that scipy.signal should have a new correlogram command, it should be a wrapper for numpy.correlate. But why wouldn't one just use scipy.signal.correlate for the 1d case as well? Also, see https://github.com/numpy/numpy/pull/5978 for the pull request with a list of specific issues in my implementation that may need attention. Honi > On Jun 17, 2015, at 6:13 PM, Sturla Molden <sturla.molden(a)gmail.com> wrote: > > On 17/06/15 04:38, Honi Sanders wrote: > >> I have now implemented this functionality in numpy.correlate() and numpy.convolve(). https://github.com/bringingheavendown/numpy. The files that were edited are: >> numpy/core/src/multiarray/multiarraymodule.c >> numpy/core/numeric.py >> numpy/core/tests/test_numeric.py >> Please look over the code, my design decisions, and the unit tests I have written. This is my first time contributing, so I am not confident about any of these and welcome feedback. > > I'll just repeat here what I already said on Github. > > I think this stems from the need to compute cross-correlograms as used > in statistical signal analysis, whereas numpy.correlate and > scipy.signal.correlate are better suited for matched filtering. > > I think the best solution would be to add a function called > scipy.signal.correlogram, which would return a cross-correlation and an > array of time lags. It could take minlag and maxlag as optional arguments. > > Adding maxlag and minlag arguments to numpy.convolve makes very little > sense, as far as I am concerned. > > Sturla > > > > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion

1 0

Numerical differentiation in scipy.optimize
by Nikolay Mayorov 17 Jun '15

17 Jun '15

Hi, I made a PR which adds a more solid implementation of numerical differentiation than `optimize.approx_fprime`, and also the replacement for `optimize.check_grad`. Here is the link https://github.com/scipy/scipy/pull/4884 I ask you to take a look and make your notes or suggestions, here or at github. Things you might want to change: - Method names. Perhaps `numerical_derivative` is a better name.- Parameter names and API in general.- Addition of higher order differentiation schemes, 5-point scheme can be added easily.- Anything else you think is important. Best regards,Nikolay Mayorov.

5 8

How to limit cross correlation window width in Numpy?
by Honi Sanders 17 Jun '15

17 Jun '15

I am learning numpy/scipy, coming from a MATLAB background. The xcorr function in Matlab has an optional argument "maxlag" that limits the lag range from –maxlag to maxlag. This is very useful if you are looking at the cross-correlation between two very long time series but are only interested in the correlation within a certain time range. The performance increases are enormous considering that cross-correlation is incredibly expensive to compute. In numpy/scipy it seems there are several options for computing cross-correlation. numpy.correlate, numpy.convolve, scipy.signal.fftconvolve. If someone wishes to explain the difference between these, I'd be happy to hear, but mainly what is troubling me is that none of them have a maxlag feature. This means that even if I only want to see correlations between two time series with lags between -100 and +100 ms, for example, it will still calculate the correlation for every lag between -20000 and +20000 ms (which is the length of the time series). This gives a 200x performance hit! Is it possible that I could contribute this feature?

3 9

`ranksums` and `mannwhitneyu` deprecation from `scipy.stats`
by Alex Wagner 15 Jun '15

15 Jun '15

Hello all, The `mannwhitneyu` and `ranksums` functions are to be deprecated in 17.0 with the introduction of the new `mww` function (for Mann-Whitney-Wilcoxon). Details follow. The `mannwhitneyu` function has been rewritten to include alternative tests (‘two-sided’, ‘less’, ‘greater’) and to do exact p-value computations on small input sets. This function has been renamed `mww` to avoid breaking backwards compatibility of `mannwhitneyu`, while fixing the reported p-value (`mannwhitneyu` claims to report a ‘one-sided p-value', but in fact reports 1/2 the p-value of a two-sided test). In addition, `mww` returns a Bunch object instead of a named tuple to support future additions to the function (see PR <https://github.com/scipy/scipy/pull/4933#issuecomment-108137084>). The `ranksums` function has been completely subsumed by the `mww` function, and is to be deprecated for this reason. If there are any objections to deprecating these functions, or the addition of the `mww` function, please voice them here. Best, Alex

4 6

Need logos for SciPy Central
by Surya 14 Jun '15

14 Jun '15

Hello everyone, Since we are in the process of updating the servers with new code, it would be great to have below logos on the site soon. 1. Favicon image (sadly, we don't have one yet) 2. Header logo The header current logo we plan to deploy is: https://github.com/scipy/SciPyCentral/blob/master/deploy/media/img/scipycen… But, it would be great if anyone can help us make better. The new header logo should be approximately of the same size as in the above link. Thanks Surya

1 0

ODE solvers
by Anne Archibald 12 Jun '15

12 Jun '15

Hi folks, Over on PR 4904 there has been much discussion about the right interface for ODE solvers. I have some experience with ODE solvers under various conditions, including one fairly heavy-duty problem, but it would be good to have a wider discussion about how ODE solvers ought to work from within scipy. https://github.com/scipy/scipy/pull/4904 Please let me know if you think this is a good design, or if you think I've left out anything important. I can probably get a pure-python version driving one of the existing solvers this weekend. My suggestion has two layers: Each solver gets wrapped in an object that does the following: * Setup and initialization of the solver: RHS, initial conditions, error tolerance, direction of t, solver parameters * Manage reentrancy constraints (some FORTRAN solvers store state in global variables) * Advance one step (size set by solver, or shorter if requested) * Know t, y, yprime at beginning and end of current step * Report an estimate of the accumulated error * Evaluate y (maybe yprime and higher derivatives) anywhere within the current step Most existing scipy solvers, and most modern implementations, support all of these basic operations. Then there is one user-level driver object that uses an underlying solver object to provide a friendlier interface that can: * Advance in a single step to any requested time. * Take an ordered array of times and return y values (and/or derivatives of various orders, perhaps) for each time. * Return the same for a list of times chosen by the solver to adequately represent the solution. * Use additional stopping criteria: for example, stop when any of a list of "event functions" changes sign or passes one of a list of values of interest. Some of these would actually stop the solver even if the solver would otherwise have tried to return more values, others would simply indicate places to report the solution state. Concerns for implementation: * Users should be able to provide compiled callables of some kind and have them called without passing through the python interpreter. This presumably needs to apply to stopping conditions/event functions too. * For fast solutions, a python driver loop may be too slow. Interesting but out-of-scope ideas: * Symplectic solvers * ODE solutions as function objects evaluatable anywhere * Solvers using long doubles or quad precision internally For concreteness about the "stopping time"/"event functions", let me mention two situations I have encountered. The first was I needed to detect when the ray I was integrating got too close to a black hole and switch coordinate systems. The second is where I need to integrate a solution not to a particular list of times, but until a function of the time (time plus position-dependent propagation delay) hits a particular list of values. Anne

6 9

SciPy 2015 Birds of a Feather Submission
by Matthew McCormick 12 Jun '15

12 Jun '15

Members of the scipy community, As one of the co-chairs in charge of organizing the birds-of-a-feather sessions at the SciPy conference this year, I wanted to reach out to your community to encourage you to submit a BoF proposal. A BoF session is an opportunity to open up a discussion on topics related to scipy development, future or just general questions. Please let us know if there is anything we can help with in terms of organization. More details can be found here: http://scipy2015.scipy.org/ehome/115969/259291/?& Kyle Mandli and Matt McCormick

1 0

[scipy.stats improvements] Weekly Summary 2015/06/07
by Abraham Escalante 08 Jun '15

08 Jun '15

Hello all, Here is the summary scipy.stats improvements this week. *Week 2* - [WIP] Trimmed statistics functions have inconsistent API - [WIP] statistics review: trim_mean - [WIP] statistics review: trim1 - [WIP] statistics review: trimboth - [WIP] statistics review: find_repeats *Plan for week 3* - statistics review: _chk_asarray - statistics review: ppcc_max - statistics review: threshold - statistics review: describe - statistics review: relfreq - statistics review: cumfreq You can find the open PRs at any given moment here <https://github.com/scipy/scipy/pulls/aeklant>. Cheers, Abraham.

1 0

Cython 0.23 will monkey patch the standard lib
by Sturla Molden 06 Jun '15

06 Jun '15

The next major release of Cython (0.23) will begin to monkey patch the Python standard lib. Personally I consider this behavior borderline evil, and I don't think we need any of the features that Cython patches into the standard lib. So should we add the off-switches to SciPy's "cythonize" scipt? https://github.com/cython/cython/blob/master/CHANGES.rst It appears the off-switches are: -D CYTHON_PATCH_ASYNCIO=0 -D CYTHON_PATCH_INSPECT=0 -D CYTHON_PATCH_ABC=0 Sturla

2 2