Mailman 3 December 2013 - SciPy-Dev

Re: [SciPy-Dev] removing umfpack wrapper
by Robert Cimrman 24 Nov '14

24 Nov '14

On 11/12/2013 11:22 PM, Ralf Gommers wrote: > On Tue, Nov 12, 2013 at 10:20 AM, Robert Cimrman <cimrman3(a)ntc.zcu.cz>wrote: > >> Hi Ralf, >> >> On 11/10/2013 04:36 PM, Ralf Gommers wrote: >>> Hi all, >>> >>> This is long overdue, but I think it's time to remove the UMFPACK wrapper >>> before the next release. I'd like some feedback on how to do that and to >>> what package (if any) to point existing users. >>> >>> As for why to remove the wrapper, see: >>> https://github.com/scipy/scipy/issues/3002 >>> http://permalink.gmane.org/gmane.comp.python.scientific.user/20451 >>> Short summary: UMFPACK changed from a BSD-compatible license to GPL at >> some >>> point. >>> >>> The deprecation warning in sparse.linalg has been referring people to >>> scikits.umfpack until now, however that package has disappeared >> completely >>> as far as I can tell. I suspect it was in the old scikits svn repo and >> was >>> never moved before that was killed. The alternatives seems to be >> Pysparse ( >> >> I missed that the scikit was lost in transition... >> >> How much work would be to move it to the new site - move from SVN to git, >> and? >> I would be willing to do it, if there is interest. >> > > Hi Robert, I think the moving wouldn't be a lot of work (assuming svn > access can still be arranged). If you'd revive the scikit it's not a > one-time effort though - it's only useful if the code is being maintained > imho. That can still be low effort perhaps, but with reviewing some PRs, > maintenance releases, putting it on pypi, etc. I'd expect at least an hour > a week or so. > > Are there scikits.umfpack users now? Would it make sense to salvage > whatever is useful from the scikit and contribute it to pysparse instead? We are using it in sfepy as the default direct solver. It seems to me that the pysparse interface requires the input matrix to be in the LL (linked list) format, which is unfortunate for us, as we use CSR. The scipy (and former scikit) umfpack wrappers have used CSR, so no copies were necessary. (After all, I have created the original scipy wrappers to be used from sfepy in the first place...) So in case others are not interested in having the scikit, I can see two possible solutions from my perspective: either enhance pysparse interface to allow CSR as well, or move the wrappers to sfepy (which I maintain anyway, but swig -> cython conversion would be needed). Not sure yet which solution I prefer. Cheers, r. > Cheers, > Ralf >> >> Cheers, >> r. >> >>> http://pysparse.sourceforge.net) and PyUBlasExt ( >>> https://pypi.python.org/pypi/PyUblasExt). Scikits.sparse is dead(-ish) >> and >>> doesn't wrap umfpack it looks like. PyUBlasExt has something that looks >>> like a wrapper but for C++ (?). So can we refer people to pysparse? >>> >>> Other question: do we call Pysparse instead of scikits.umfpack when >>> available? And if so, by default or not? Or do we rip out everything that >>> sits under useUmfpack=True? >>> >>> Cheers, >>> Ralf

2 5

Linalg: function for diagonalize hermitian commuting matrices simultaneously
by Lars-Hendrik Frahm 07 Oct '14

07 Oct '14

Dear SciPy Developers, I like to contribute a function for simultaneously diagonalize a set of hermitian commuting matrices. Especially for matrices with degenerate eigenvalues. Is there need for something like that in SciPy ? I think it could be helpful for someone else and I didn't find it in recent SciPy versions or on the mailing list. Hope to hear from you.

3 3

Bootstrap confidence limits code
by Constantine Evans 05 Oct '14

05 Oct '14

Hello everyone, A few years ago I implemented a scikit for bootstrap confidence limits (https://github.com/cgevans/scikits-bootstrap). I didn’t think much about it after that until recently, when I realized that some people are actually using it, and that there’s apparently been some talk about implementing this functionality in either scipy.stats or statsmodels (I should thank Randal Olson for discussing this and bringing it to my attention). As such I’ve rewritten most of the code, and written up some docstrings. The current code can do confidence intervals with basic percentile interval, bias-corrected accelerated, and approximate bootstrap confidence methods, and can also provide bootstrap and jackknife indexes. Most of it is implemented from the descriptions in Efron and Tibshirani’s Introduction to the Bootstrap, but the ABC code at the moment is a port from the modified-BSD-licensed bootstrap package for R (not the boot package) as I’m not entirely confident in my understanding of the method. And so, I have a few questions for everyone: * Is there any interest in including this sort of code in either scipy.stats or statsmodels? If so, where do people think would be the better place? The code is relatively small; at the moment it is less than 200 lines, with docstrings probably making up 100 of those lines. * Also, if so, what would need to be changed, added, and improved beyond what is mentioned in the Contributing to Scipy part of the reference guide? I’m never a fan of my own code, and imagine quite a bit would need to be fixed; I know tests will need to be added too. In addition, I have a few questions about what would be better practice for the API, and I haven’t really found a guide on best practices for Scipy: * When I started writing the code, I wrote a single function ci for confidence intervals, with a method argument to choose the method. This is easy for users, especially so that they don’t have to look through documentation to realize that BCA is the most generally useful method (at least from everything I’ve read) and that there really isn’t any reason to use many of the simpler methods. However, ABC takes different paramenters, and needs a statistic function that takes weights, which makes this single-function organization trickier. At the moment, I have a separate function for ABC. Would it be better to split up all the methods to their own functions? * ABC requires a statistic function that takes weights. I’ve noticed that things like np.average takes a weights= argument. Would it be better to require input of a stat(data,weights) function, or input of a stat(data,weights=) with weights as a named argument? The latter would be nice in terms of allowing the same function to be used for all methods, but would make it impossible to use a lambda for the function. Is there some other method of doing this entirely? * Are there any missing features that anyone thinks should be added? I apologize if much of this is answered elsewhere, I just haven’t found any of it; I also apologize if this is far too long-winded and confusing! Regards, Constantine Evans

4 3

conv2 and normxcorr2
by awebster＠falsecolour.com 12 May '14

12 May '14

I noticed a couple of popular matlab functions - conv2 and normxcorr2 were not present in the scipy.signal packages. I would like to submit them for addition. Can anyone point me on instructions on how to write such a thing? Below are examples. # 2D convolution using the convolution's FFT property def conv2(a,b): ma,na = a.shape mb,nb = b.shape return fft.ifft2(fft.fft2(a,[2*ma-1,2*na-1])*fft.fft2(b,[2*mb-1,2*nb-1])) # compute a normalized 2D cross correlation using convolutions # this will give the same output as matlab, albeit in row-major order def normxcorr2(b,a): c = conv2(a,flipud(fliplr(b))) a = conv2(a**2, ones(b.shape)) b = sum(b.flatten()**2) c = c/sqrt(a*b) return c -- Aaron Webster

5 8

1.0 roadmap: weave
by Arnd Baecker 06 May '14

06 May '14

Hi, > On Sat, Sep 21, 2013 at 8:54 PM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: [...] > weave > ````` > This is the only module that was not ported to Python 3. Effectively it's > deprecated (not recommended to use for new code). In the future it should be > removed from scipy (can be made into a separate module). Let me just mention that I am worried about the removal of weave: a substantial amount of code developed in our group over the years uses weave. Porting this to cython seems not trivial, in particular, because many of the original authors (diploma students, PhD students, ...) have left academia by now. Well, as porting weave to Python 3 is way out of my capabilities, this is just a comment in the hope that someone could give this a try (if at some point testing would be needed, I could try to help, of course). Best, Arnd

9 12

Add a "Wrapping Code From Other Languages" section to topical-software
by mir amicitas 07 Jan '14

07 Jan '14

Dear Scipy-dev, I would like to add a new section to the topical-software page for wrapping code in Matlab, IDL etc. Some of these tools are currently listed in Miscellaneous, while they would naturally fall under "Running Code Written In Other Languages". As part of this I will add a new link to the package pyidlrpc (https://bitbucket.org/amicitas/pyidlrpc/) which can be used to wrap IDL code. My specific suggestion is to add a section under: "Running Code Written In Other Languages" With the title: "Wrapping/Executing Code in Other Languages" Under this heading I will add the MATLAB and IDL excecution/wrapping libraries. Please reply with any comments. Otherwise I will make a pull request with the above changes. Sincerely, Novimir Pablant

2 2

ANN: NumPy 1.7.2 release
by Julian Taylor 01 Jan '14

01 Jan '14

Hello, I'm happy to announce the of Numpy 1.7.2. This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 - 3.3. More than 42 issues were fixed, the most important issues are listed in the release notes: https://github.com/numpy/numpy/blob/v1.7.2/doc/release/1.7.2-notes.rst Compared to the last release candidate four additional minor issues have been fixed and compatibility with python 3.4b1 improved. Source tarballs, installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.2 Cheers, Julian Taylor

1 0

ENH: add distance gufuncs to `scipy.spatial`
by Jaime Fernández del Río 19 Dec '13

19 Dec '13

Hi, I have just submitted a quite bulky pull request ( https://github.com/scipy/scipy/pull/3163), that deserves a lengthier explanation... It basically adds a new submodule, umath_distance, to scipy.spatial. This module reproduces some of the functionality in the distance submodule, using gufuncs, and thus adding all the broadcasting sweetness that comes with them. Some examples ============= >>> import numpy as np >>> import scipy.spatial.umath_distance as udist >>> a = np.random.rand(3, 10) >>> b = np.random.rand(4, 10) Compute the distance between corresponding vectors in two multidimensional arrays: >>> udist.euclidean(a, b[:3]) array([ 1.25080976, 1.00788142, 0.89158743]) Compute all distances between to sets of vectors, i.e. what distance.cdist does, using broadcasting... >>> udist.euclidean(a[:, np.newaxis, :], b) array([[ 1.25080976, 1.20893157, 1.63674343, 1.62432255], [ 1.36665537, 1.00788142, 1.10137183, 1.29196888], [ 1.24924713, 1.27379914, 0.89158743, 1.317478 ]]) ...or a convenience argument in the Python wrapper: >>> udist.euclidean(a, b, cdist=True) array([[ 1.25080976, 1.20893157, 1.63674343, 1.62432255], [ 1.36665537, 1.00788142, 1.10137183, 1.29196888], [ 1.24924713, 1.27379914, 0.89158743, 1.317478 ]]) If a single input is given, compute all pairwise distances, i.e. what distance.pdist does: >>> udist.euclidean(a) array([ 1.59655565, 1.73136646, 0.99252024]) And check the result is correct with a cdist-style call: >>> udist.euclidean(a, a, cdist=True) array([[ 0. , 1.59655565, 1.73136646], [ 1.59655565, 0. , 0.99252024], [ 1.73136646, 0.99252024, 0. ]]) What's included =============== So far only the distance functions of only two inputs are included, which are all in scipy.spatial.distance except mahalanobis, minkowski, seuclidean and wminkowski. If you people like this gufunc approach, I will happily expand it to include those as well. The code so far is very modular, both at the C and Python level, and makes it relatively easy to add a new distance function from a C function doing a single distance calculation between two vectors. I haven't figured out all the details, but I think it would be possible to move the gufunc kernels to a header file, and make it easy(er) to add a new distance gufunc, perhaps even from Cython, in a similar way to how it can be done with PyUFunc_dd_d and the like. I have also spent a fair amount of time checking the definition of the implemented distances from original sources, and I think I have found some errors there, both in the definitions (Sokal-Michener is wrong), and in the way pathological edge cases are handled. The C code is heavily commented on the subject, starting around line 154. Performance =========== For the existing cdist and pdist functions, replacing inline functions with function pointers does take a toll on the performance of the gufunc approach, which is worse the smaller the dimension of the vectors being compared, i.e. the less work each function call does: >>> import timeit >>> import scipy.spatial.distance as dist >>> a = np.random.rand(1000, 3) >>> b = np.random.rand(2000, 3) >>> min(timeit.repeat("dist.cdist(a, b, metric='euclidean')", 'from __main__ import dist, a, b', number=1, repeat=100)) 0.017471377426346635 >>> min(timeit.repeat("udist.euclidean(a, b, cdist=True)", 'from __main__ import udist, a, b', number=1, repeat=100)) 0.023448978561347644 >>> a = np.random.rand(1000, 100) >>> b = np.random.rand(2000, 100) >>> min(timeit.repeat("dist.cdist(a, b, metric='euclidean')", 'from __main__ import dist, a, b', number=1, repeat=100)) 0.20550328478645952 >>> min(timeit.repeat("udist.euclidean(a, b, cdist=True)", 'from __main__ import udist, a, b', number=1, repeat=100)) 0.22972938023184497 On the other hand, the C implementations in distance of most of the boolean functions are grossly inefficient, so there are significant speed-ups there despite the extra overhead: >>> a = np.random.rand(1000, 100) < 0.5 >>> b = np.random.rand(2000, 100) < 0.5 >>> min(timeit.repeat("dist.cdist(a, b, metric='sokalsneath')", 'from __main__ import dist, a, b', number=1, repeat=100)) 1.43636283024901 >>> min(timeit.repeat("udist.sokalsneath(a, b, cdist=True)", 'from __main__ import udist, a, b', number=1, repeat=100)) 0.3878355139542009 -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

1 0

Compiling scipy 0.13.2 with Python 3.4 rc 1
by Stephen Simmons 14 Dec '13

14 Dec '13

Hi all, In case anyone wants to try scipy with the new Python 3.4, there is a problem with importlib in rc1 that results in import errors while building scipy. Basically f2py can't do its imports properly and some autogenerated code comes out with missing symbols. It can be hard to see this amidst the stream of build messages and warnings. They show up clearly though when starting python3.4 and running scipy.test(). First time through I got 174 failing tests. If you don't want to wait for Python 3.4 rc2, you can download the fixed version of importlib.__init__.py from here (updated on 12 Dec): http://hg.python.org/cpython/file/c1a7ba57b4ff/Lib/importlib/__init__.py After doing this, I removed numpy and scipy (sudo pip3.4 uninstall numpy etc), recompiled numpy and scipy, and got no test failures (using Linux Mint 14). Regards Stephen

2 1

A command line interface for submissions in scipy central
by Surya Kasturi 13 Dec '13

13 Dec '13

The idea is to build an interface that lets to submit snippets scipycentral using command line. How it could be for example: $ spc submit ./code/filter.py -- snippet Its a new idea. So, your opinions and suggestions are all welcome. If there are more +1 I will proceed with building. Thanks Surya

1 0