Mailman 3 November 2013 - SciPy-Dev

Re: [SciPy-Dev] removing umfpack wrapper
by Robert Cimrman Nov. 24, 2014

Nov. 24, 2014

On 11/12/2013 11:22 PM, Ralf Gommers wrote: > On Tue, Nov 12, 2013 at 10:20 AM, Robert Cimrman <cimrman3(a)ntc.zcu.cz>wrote: > >> Hi Ralf, >> >> On 11/10/2013 04:36 PM, Ralf Gommers wrote: >>> Hi all, >>> >>> This is long overdue, but I think it's time to remove the UMFPACK wrapper >>> before the next release. I'd like some feedback on how to do that and to >>> what package (if any) to point existing users. >>> >>> As for why to remove the wrapper, see: >>> https://github.com/scipy/scipy/issues/3002 >>> http://permalink.gmane.org/gmane.comp.python.scientific.user/20451 >>> Short summary: UMFPACK changed from a BSD-compatible license to GPL at >> some >>> point. >>> >>> The deprecation warning in sparse.linalg has been referring people to >>> scikits.umfpack until now, however that package has disappeared >> completely >>> as far as I can tell. I suspect it was in the old scikits svn repo and >> was >>> never moved before that was killed. The alternatives seems to be >> Pysparse ( >> >> I missed that the scikit was lost in transition... >> >> How much work would be to move it to the new site - move from SVN to git, >> and? >> I would be willing to do it, if there is interest. >> > > Hi Robert, I think the moving wouldn't be a lot of work (assuming svn > access can still be arranged). If you'd revive the scikit it's not a > one-time effort though - it's only useful if the code is being maintained > imho. That can still be low effort perhaps, but with reviewing some PRs, > maintenance releases, putting it on pypi, etc. I'd expect at least an hour > a week or so. > > Are there scikits.umfpack users now? Would it make sense to salvage > whatever is useful from the scikit and contribute it to pysparse instead? We are using it in sfepy as the default direct solver. It seems to me that the pysparse interface requires the input matrix to be in the LL (linked list) format, which is unfortunate for us, as we use CSR. The scipy (and former scikit) umfpack wrappers have used CSR, so no copies were necessary. (After all, I have created the original scipy wrappers to be used from sfepy in the first place...) So in case others are not interested in having the scikit, I can see two possible solutions from my perspective: either enhance pysparse interface to allow CSR as well, or move the wrappers to sfepy (which I maintain anyway, but swig -> cython conversion would be needed). Not sure yet which solution I prefer. Cheers, r. > Cheers, > Ralf >> >> Cheers, >> r. >> >>> http://pysparse.sourceforge.net) and PyUBlasExt ( >>> https://pypi.python.org/pypi/PyUblasExt). Scikits.sparse is dead(-ish) >> and >>> doesn't wrap umfpack it looks like. PyUBlasExt has something that looks >>> like a wrapper but for C++ (?). So can we refer people to pysparse? >>> >>> Other question: do we call Pysparse instead of scikits.umfpack when >>> available? And if so, by default or not? Or do we rip out everything that >>> sits under useUmfpack=True? >>> >>> Cheers, >>> Ralf

2 5

Linalg: function for diagonalize hermitian commuting matrices simultaneously
by Lars-Hendrik Frahm Oct. 7, 2014

Oct. 7, 2014

Dear SciPy Developers, I like to contribute a function for simultaneously diagonalize a set of hermitian commuting matrices. Especially for matrices with degenerate eigenvalues. Is there need for something like that in SciPy ? I think it could be helpful for someone else and I didn't find it in recent SciPy versions or on the mailing list. Hope to hear from you.

3 3

Bootstrap confidence limits code
by Constantine Evans Oct. 5, 2014

Oct. 5, 2014

Hello everyone, A few years ago I implemented a scikit for bootstrap confidence limits (https://github.com/cgevans/scikits-bootstrap). I didn’t think much about it after that until recently, when I realized that some people are actually using it, and that there’s apparently been some talk about implementing this functionality in either scipy.stats or statsmodels (I should thank Randal Olson for discussing this and bringing it to my attention). As such I’ve rewritten most of the code, and written up some docstrings. The current code can do confidence intervals with basic percentile interval, bias-corrected accelerated, and approximate bootstrap confidence methods, and can also provide bootstrap and jackknife indexes. Most of it is implemented from the descriptions in Efron and Tibshirani’s Introduction to the Bootstrap, but the ABC code at the moment is a port from the modified-BSD-licensed bootstrap package for R (not the boot package) as I’m not entirely confident in my understanding of the method. And so, I have a few questions for everyone: * Is there any interest in including this sort of code in either scipy.stats or statsmodels? If so, where do people think would be the better place? The code is relatively small; at the moment it is less than 200 lines, with docstrings probably making up 100 of those lines. * Also, if so, what would need to be changed, added, and improved beyond what is mentioned in the Contributing to Scipy part of the reference guide? I’m never a fan of my own code, and imagine quite a bit would need to be fixed; I know tests will need to be added too. In addition, I have a few questions about what would be better practice for the API, and I haven’t really found a guide on best practices for Scipy: * When I started writing the code, I wrote a single function ci for confidence intervals, with a method argument to choose the method. This is easy for users, especially so that they don’t have to look through documentation to realize that BCA is the most generally useful method (at least from everything I’ve read) and that there really isn’t any reason to use many of the simpler methods. However, ABC takes different paramenters, and needs a statistic function that takes weights, which makes this single-function organization trickier. At the moment, I have a separate function for ABC. Would it be better to split up all the methods to their own functions? * ABC requires a statistic function that takes weights. I’ve noticed that things like np.average takes a weights= argument. Would it be better to require input of a stat(data,weights) function, or input of a stat(data,weights=) with weights as a named argument? The latter would be nice in terms of allowing the same function to be used for all methods, but would make it impossible to use a lambda for the function. Is there some other method of doing this entirely? * Are there any missing features that anyone thinks should be added? I apologize if much of this is answered elsewhere, I just haven’t found any of it; I also apologize if this is far too long-winded and confusing! Regards, Constantine Evans

4 3

1.0 roadmap: weave
by Arnd Baecker May 6, 2014

May 6, 2014

Hi, > On Sat, Sep 21, 2013 at 8:54 PM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: [...] > weave > ````` > This is the only module that was not ported to Python 3. Effectively it's > deprecated (not recommended to use for new code). In the future it should be > removed from scipy (can be made into a separate module). Let me just mention that I am worried about the removal of weave: a substantial amount of code developed in our group over the years uses weave. Porting this to cython seems not trivial, in particular, because many of the original authors (diploma students, PhD students, ...) have left academia by now. Well, as porting weave to Python 3 is way out of my capabilities, this is just a comment in the hope that someone could give this a try (if at some point testing would be needed, I could try to help, of course). Best, Arnd

9 12

thLib: package for 3D kinematics and data analysis (including Savitzky-Golay filter)
by Haslwanter Thomas Dec. 4, 2013

Dec. 4, 2013

I have compiled a package that should be useful for many researchers, especially in the field of movement analysis. It can be installed with pip install thLib and documentation is provided under http://work.thaslwanter.at/thLib/html/ I would be happy to contribute any of the functions in this package to SciPy. Especially, I believe that a Savitzky-Golay filter is overdue for Scipy. Please provide feedback, which of the functions may be relevant for SciPy, and which functions are faulty or missing. Thanks thomas --- Prof. (FH) PD Dr. Thomas Haslwanter School of Applied Health and Social Sciences University of Applied Sciences Upper Austria FH OÖ Studienbetriebs GmbH Garnisonstraße 21 4020 Linz/Austria Tel.: +43 (0)5 0804 -52170 Fax: +43 (0)5 0804 -52171 E-Mail: Thomas.Haslwanter(a)fh-linz.at<mailto:Thomas.Haslwanter@fh-linz.at> Web: me-research.fh-linz.at<http://work.thaslwanter.at> or work.thaslwanter.at

5 8

adding linear fitting routine
by David Pine Nov. 29, 2013

Nov. 29, 2013

I have written a function called linfit for linear least square fitting that I am proposing to have added to one of the numpy or scipy libraries. linfit performs a full least squares chi-squared fit. Thus, it can handle data with error estimates (aka error bars), weighting the data accordingly. linfit provides estimates of the uncertainties of the fitted parameters, the slope and y-intercept. linfit allows one to optionally: (1) use no weighting, or (2) to weight data according the residuals, often called relative weighting (the way it's often done in work in the social sciences), or (3) to use the absolute measure of uncertainties either for each data point or for all the data points at once (the way it's often done in the physical sciences). These options were included with the recent discussion on weighted least squares fitting in mind. See scipy/scipy#448. I am not sure where linfit best belongs in the numpy/scipy universe. The most reasonable places would seem to be either the polynomial package (a straight line is the simplest polynomial) or perhaps the scipy.optimize package along with curve_fit, which fits nonlinear functions to data. I wrote the function because there really is nothing like it in numpy or scipy, and it is so basic that in my opinion, something like it should be available. I tried to write it so that it is useful to a very wide range of users that cross all branches of social and physical sciences as well as engineering. I have added linfit to a cloned version of the numpy.polynomial module, which can be found at https://github.com/numpy/numpy/pull/4080. The standalone linfit function can be found at https://github.com/djpine/linfit. An ipython notebook demonstrating various ways of using linfit is available at the same site; its output can be viewed at http://nbviewer.ipython.org/github/djpine/linfit/blob/master/linfit.ipynb. I have in included a unit test test_linfit.py that, among other things, compares the speed of linfit to numpy.polyfit, scipy.linalg.lstsq, and scipy.stats.linregress. linfit is faster for all cases I tested, typically by several times. Finally, I am new to using Github and to contributing to numpy/scipy, so I am not sure if I have submitted everything properly, but I hope this gets the process going. David Pine

1 0

broken link to chm documentation
by Pierre Haessig Nov. 28, 2013

Nov. 28, 2013

Hi, I don't know how widely is the CHM documentation used, but the link to the CHM version of the Reference Guide on http://docs.scipy.org/doc/ is broken. It points to http://docs.scipy.org/doc/scipy/scipy-chm.zip which is "Not Found" (there is also a CHM version for v0.8 which does work) best, Pierre

2 1

(Possible) new optimization routine (2) - scipy.optimize
by Andrea Gavana Nov. 27, 2013

Nov. 27, 2013

Hi All, since I posted the last time about (possible) new optimization routines in scipy.optimize, I have been working here and there in making the code for AMPGO (Adaptive Memory Programming for Global Optimization) a bit more robust and in expanding the benchmark test suite. Since recently I saw a few messages about optimization methods flying around in the mailing list, I thought I may share some more findings and possibly (finally) start integrating AMPGO into scipy. First things first: I would love to see AMPGO into scipy, but there are still a couple of issues to be solved: 1. Some of the local optimization methods AMPGO can use (like L-BFGS-B, TNC and so on) can take advantage of gradient information, and sometimes people do actually have access to the gradient of the objective function. The problem is, given the definition of the Tunnelling function used by AMPGO: T(x) = [f(x) - aspiration)**2.0] / prod(dist(s, x)) for s in tabu_list Where "dist" is the euclidean distance between the current point "x" and one of the previous local optima "s" ("tabu_list" is a list containing 2 or more of these local optima). (or see page 4 at http://leeds-faculty.colorado.edu/glover/fred%20pubs/416%20-%20AMP%20(TS)%2… a clearer formula for the Tunnelling function). I have absolutely no idea of how to get the analytical expression of the gradient of the Tunnelling function, given the gradient of the objective function. I'm sure it's very much doable but my calculus skills are way too inadequate. 2. As the current code for AMPGO supports local solvers from scipy.optimize and OpenOpt, it turns out to be a pain to generalize its interface in order for AMPGO to be compatible with the minimize() general API. Not to mention the general PR process. In any case, I'll try to gather enough willpower to get AMPGO up to scipy standards and I'll contribute the benchmarks I created as well. So, mentioning the benchmarks and the results, I managed to expand the multi-dimensional test suite from 85 to 184 test functions. The test suite is now one of the most complete and comprehensive in the Open Source world, and it's not in Fortran 77 (luckily enough). The test suite currently contains: - 18 one-dimensional test functions with multiple local/global minima; - 184 multivariate problems (where the number of independent variables ranges from 2 to 17), again with multiple local/global minima. The main page describing the rules, algorithms and motivation is here: http://infinity77.net/global_optimization/index.html A fairly in-depth summary page on AMPGO and sensitivities on its input parameters (local solver, tunnelling strategy, etc...): http://infinity77.net/global_optimization/ampgo.html Algorithms comparisons: http://infinity77.net/global_optimization/multidimensional.html http://infinity77.net/global_optimization/univariate.html Test functions and how they rank in a "difficult-to-solve" context: http://infinity77.net/global_optimization/test_functions.html The overall conclusion is that AMPGO is superior to all the other algorithms I have tried, leaving the second-best (pyasa) behind by a full 20% of number of solved problems. It's also the fastest (function-evaluation-wise), as it is able to outperform all the other algorithms' best results within 200 function evaluations or less (even though the other algorithms limit is 2,000). However, to be fair, it is an algorithm designed for low-dimensional optimization problems (i.e., 1-20 variables). If anyone has any suggestion on how to implement my point (1) above, please feel free to share your thoughts. I have no clue whatsoever. Any comment or requests to add additional benchmarks, please give me a shout. Enjoy :-) . Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net # ------------------------------------------------------------- # def ask_mailing_list_support(email): if mention_platform_and_version() and include_sample_app(): send_message(email) else: install_malware() erase_hard_drives() # ------------------------------------------------------------- #

2 1

Supporting numpy/scipy in an IDE
by Sven Brauch Nov. 26, 2013

Nov. 26, 2013

Hi! I'm writing a plugin for an IDE which does things like code completion on Python code. Usually, it gets the necessary information from static analysis of Python code (which works fairly well for many libraries); however, in the case of numpy, this obviously does not work at all because numpy is not written in pure Python. For improving numpy support, I would especially need the following information: * What are the arguments of a particular function? Python's inspect won't tell me; it seems like this information is just available in the prosa text docstring. The website has it in a somewhat machine-readable way, is there a method to get the actual list of parameters which does not involve parsing that? Or is the table on the website based on parsing the docstring? * What kind of object does a function return? Of course this is not always well-defined, but in most cases it is. This information, too, only seems to be available in the prosa docstrings, if at all -- and different from the argument specification, it's usually just mentioned somewhere in a not even remotely machine-readable form. My question is thus: Do you see any reasonable (automatable) way to extract this information from any resource (code, docs, ...)? I think having this information in a machine-readable form would also be beneficial for various other projects. I'm happy about every suggestion. I dislike solutions based on inspecting things at runtime; it is scary to run code a user types into the editor, and it will also be slow in a variety of cases. Best regards, Sven

7 17

any chance of enabling sort option in scipy.linalg.qz?
by Ricardo Mayer Nov. 26, 2013

Nov. 26, 2013

Dear all, As of release 0.13 of scipy the sorting option for the generalized schur decomposition (aka QZ) it's been disabled because some failure wrapping the corresponding function from LAPACK. Is there are any chance of fixing this for the next release (or sooner)? Doing the QZ decomposition in such a way that generalized eigenvalues with absolute values greater than one appear last is required for a pretty common algorithm in (macro)economics solving linear rational expectation models. I'd loathe to have to go back to matlab to do just this :-) (which I won't!!! but still ...) best, R PS: browsing the current documentation for 0.14 it seems that this situation has not changed, yet.

2 1