Mailman 3 July 2014 - SciPy-Dev

Re: [SciPy-Dev] removing umfpack wrapper
by Robert Cimrman 23 Nov '14

23 Nov '14

On 11/12/2013 11:22 PM, Ralf Gommers wrote: > On Tue, Nov 12, 2013 at 10:20 AM, Robert Cimrman <cimrman3(a)ntc.zcu.cz>wrote: > >> Hi Ralf, >> >> On 11/10/2013 04:36 PM, Ralf Gommers wrote: >>> Hi all, >>> >>> This is long overdue, but I think it's time to remove the UMFPACK wrapper >>> before the next release. I'd like some feedback on how to do that and to >>> what package (if any) to point existing users. >>> >>> As for why to remove the wrapper, see: >>> https://github.com/scipy/scipy/issues/3002 >>> http://permalink.gmane.org/gmane.comp.python.scientific.user/20451 >>> Short summary: UMFPACK changed from a BSD-compatible license to GPL at >> some >>> point. >>> >>> The deprecation warning in sparse.linalg has been referring people to >>> scikits.umfpack until now, however that package has disappeared >> completely >>> as far as I can tell. I suspect it was in the old scikits svn repo and >> was >>> never moved before that was killed. The alternatives seems to be >> Pysparse ( >> >> I missed that the scikit was lost in transition... >> >> How much work would be to move it to the new site - move from SVN to git, >> and? >> I would be willing to do it, if there is interest. >> > > Hi Robert, I think the moving wouldn't be a lot of work (assuming svn > access can still be arranged). If you'd revive the scikit it's not a > one-time effort though - it's only useful if the code is being maintained > imho. That can still be low effort perhaps, but with reviewing some PRs, > maintenance releases, putting it on pypi, etc. I'd expect at least an hour > a week or so. > > Are there scikits.umfpack users now? Would it make sense to salvage > whatever is useful from the scikit and contribute it to pysparse instead? We are using it in sfepy as the default direct solver. It seems to me that the pysparse interface requires the input matrix to be in the LL (linked list) format, which is unfortunate for us, as we use CSR. The scipy (and former scikit) umfpack wrappers have used CSR, so no copies were necessary. (After all, I have created the original scipy wrappers to be used from sfepy in the first place...) So in case others are not interested in having the scikit, I can see two possible solutions from my perspective: either enhance pysparse interface to allow CSR as well, or move the wrappers to sfepy (which I maintain anyway, but swig -> cython conversion would be needed). Not sure yet which solution I prefer. Cheers, r. > Cheers, > Ralf >> >> Cheers, >> r. >> >>> http://pysparse.sourceforge.net) and PyUBlasExt ( >>> https://pypi.python.org/pypi/PyUblasExt). Scikits.sparse is dead(-ish) >> and >>> doesn't wrap umfpack it looks like. PyUBlasExt has something that looks >>> like a wrapper but for C++ (?). So can we refer people to pysparse? >>> >>> Other question: do we call Pysparse instead of scikits.umfpack when >>> available? And if so, by default or not? Or do we rip out everything that >>> sits under useUmfpack=True? >>> >>> Cheers, >>> Ralf

2 5

Linalg: function for diagonalize hermitian commuting matrices simultaneously
by Lars-Hendrik Frahm 07 Oct '14

07 Oct '14

Dear SciPy Developers, I like to contribute a function for simultaneously diagonalize a set of hermitian commuting matrices. Especially for matrices with degenerate eigenvalues. Is there need for something like that in SciPy ? I think it could be helpful for someone else and I didn't find it in recent SciPy versions or on the mailing list. Hope to hear from you.

3 3

Bootstrap confidence limits code
by Constantine Evans 04 Oct '14

04 Oct '14

Hello everyone, A few years ago I implemented a scikit for bootstrap confidence limits (https://github.com/cgevans/scikits-bootstrap). I didn’t think much about it after that until recently, when I realized that some people are actually using it, and that there’s apparently been some talk about implementing this functionality in either scipy.stats or statsmodels (I should thank Randal Olson for discussing this and bringing it to my attention). As such I’ve rewritten most of the code, and written up some docstrings. The current code can do confidence intervals with basic percentile interval, bias-corrected accelerated, and approximate bootstrap confidence methods, and can also provide bootstrap and jackknife indexes. Most of it is implemented from the descriptions in Efron and Tibshirani’s Introduction to the Bootstrap, but the ABC code at the moment is a port from the modified-BSD-licensed bootstrap package for R (not the boot package) as I’m not entirely confident in my understanding of the method. And so, I have a few questions for everyone: * Is there any interest in including this sort of code in either scipy.stats or statsmodels? If so, where do people think would be the better place? The code is relatively small; at the moment it is less than 200 lines, with docstrings probably making up 100 of those lines. * Also, if so, what would need to be changed, added, and improved beyond what is mentioned in the Contributing to Scipy part of the reference guide? I’m never a fan of my own code, and imagine quite a bit would need to be fixed; I know tests will need to be added too. In addition, I have a few questions about what would be better practice for the API, and I haven’t really found a guide on best practices for Scipy: * When I started writing the code, I wrote a single function ci for confidence intervals, with a method argument to choose the method. This is easy for users, especially so that they don’t have to look through documentation to realize that BCA is the most generally useful method (at least from everything I’ve read) and that there really isn’t any reason to use many of the simpler methods. However, ABC takes different paramenters, and needs a statistic function that takes weights, which makes this single-function organization trickier. At the moment, I have a separate function for ABC. Would it be better to split up all the methods to their own functions? * ABC requires a statistic function that takes weights. I’ve noticed that things like np.average takes a weights= argument. Would it be better to require input of a stat(data,weights) function, or input of a stat(data,weights=) with weights as a named argument? The latter would be nice in terms of allowing the same function to be used for all methods, but would make it impossible to use a lambda for the function. Is there some other method of doing this entirely? * Are there any missing features that anyone thinks should be added? I apologize if much of this is answered elsewhere, I just haven’t found any of it; I also apologize if this is far too long-winded and confusing! Regards, Constantine Evans

4 3

computing pairwise distance of vectors with missing (nan) values
by Moritz Emanuel Beber 12 Aug '14

12 Aug '14

Dear all, My basic problem is that I would like to compute distances between vectors with missing values. You can find more detail in my question on SO (http://stackoverflow.com/questions/24781461/compute-the-pairwise-distance-i…). Since it seems this is not directly possible with scipy at the moment, I started to Cythonize my function. Currently, the below function is not much faster than my pure Python implementation, so I thought I'd ask the experts here. *Note that even though I'm computing the euclidean distance, I'd like to make use of different distance metrics. * So my current attempt at Cythonizing is: import numpy cimport numpy cimport cython from numpy.linalg import norm numpy.import_array() @cython.boundscheck(False) @cython.wraparound(False) def masked_euclidean(numpy.ndarray[numpy.double_t, ndim=2] data): cdef Py_ssize_t m = data.shape[0] cdef Py_ssize_t i = 0 cdef Py_ssize_t j = 0 cdef Py_ssize_t k = 0 cdef numpy.ndarray[numpy.double_t] dm = numpy.zeros(m * (m - 1) // 2, dtype=numpy.double) cdef numpy.ndarray[numpy.uint8_t, ndim=2, cast=True] mask = numpy.isfinite(data) # boolean for i in range(m - 1): for j in range(i + 1, m): curr = numpy.logical_and(mask[i], mask[j]) u = data[i][curr] v = data[j][curr] dm[k] = norm(u - v) k += 1 return dm Maybe the lack of speed-up is due to the Python function 'norm'? So my question is, how to improve the Cython implementation? Or is there a completely different way of approaching this problem? Thanks in advance, Moritz

3 4

ANN: NumPy 1.9.0 beta 2 release
by Julian Taylor 30 Jul '14

30 Jul '14

Hello, The source packages and binaries got numpy 1.9.0 beta 2 have just been uploaded to sourceforge. https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b2 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 - 3.4. Unfortunately we have disabled the new __numpy_ufunc__ feature for overriding ufuncs in subclasses for now. There are still some unresolved issues with its behavior regarding python operator precedence and subclasses. If you have a stake in the issue please read Pauli's summary of the remaining issues: http://mail.scipy.org/pipermail/numpy-discussion/2014-July/070737.html When the issues are resolved to everyones satisfaction we hope to enable the feature for 1.10 in its final form. We have restored the indexing edge case that broke matplotlib with numpy 1.9.0 beta 1 but some of the other test failures in other packages are deemed bugs in their code and not reasonable to support in numpy anymore. Most projects have fixed the issues in their latest stable or development versions. Depending on how bad the broken functionality is you may need to update your third party packages when updating numpy to 1.9.0b2. An attempt was made to update the windows binary toolchain to the latest mingw/mingw64 version and an up to date ATLAS version but this turned up a few ugly test failures. Help in resolving these issues is appreciated, no core developer has Windows debugging experience. Please see this issue for details: https://github.com/numpy/numpy/issues/4909 The changelog is mostly the same as in beta1. Please read it carefully there have been many small changes that could affect your code. https://github.com/numpy/numpy/blob/maintenance/1.9.x/doc/release/1.9.0-not… Please also take special note of the future changes section which will apply to the following release 1.10.0 and make sure to check if your applications would be affected by them. Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b2 Cheers, Julian Taylor

1 0

convolution using numpy/scipy using MKL libraries
by Sai Rajeshwar 27 Jul '14

27 Jul '14

hi all, Im trying to implement 3d convolutional networks.. for which I wanted to use convolve function from scipy.signal.convolve or fftconvolve.. but looks like both of them doesnot use MKL libraries.. is there any implementation of convolutoin which uses MKL libraries or MKL-threaded so that code runs faster. thanks a lot in advance *with regards..* *M. Sai Rajeswar* *M-tech Computer Technology* *IIT Delhi----------------------------------Cogito Ergo Sum---------*

2 1

scipy improve performance by parallelizing
by Sai Rajeshwar 25 Jul '14

25 Jul '14

hi all, im trying to optimise a python code takes huge amount of time on scipy functions such as scipy.signa.conv. Following are some of my queries regarding the same.. It would be great to hear from you.. thanks.. ---------------------------------------------------- 1) Can Scipy take advantage of multi-cores.. if so how 2)what are ways we can improve the performance of scipy/numpy functions eg: using openmp, mpi etc 3)If scipy internally use blas/mkl libraries can we enable parallelism through these? looks like i have to work on internals of scipy.. thanks a lot.. *with regards..* *M. Sai Rajeswar* *M-tech Computer Technology* *IIT Delhi----------------------------------Cogito Ergo Sum---------*

12 23

scipy.sparse versus pysparse
by nicky van foreest 25 Jul '14

25 Jul '14

Hi, I am doing some testing between scipy.sparse and pysparse on my ubuntu machine. Some testing reveals that pysparse is about 9 times faster in matrix-vector multiplication that scipy.sparse. Might there be anything specific I forgot to do during scipy's installation (I just ran apt-get install python-scipy)? Is there another simple explanation for this difference? I prefer to use scipy.sparse for its cleaner api, but a factor 9 in speed is considerable. thanks Nicky

4 8

__numpy_ufunc__ and 1.9 release
by Julian Taylor 24 Jul '14

24 Jul '14

hi, as you may know we want to release numpy 1.9 soon. We should have solved most indexing regressions the first beta showed. The remaining blockers are finishing the new __numpy_ufunc__ feature. This feature should allow for alternative method to overriding the behavior of ufuncs from subclasses. It is described here: https://github.com/numpy/numpy/blob/master/doc/neps/ufunc-overrides.rst The current blocker issues are: https://github.com/numpy/numpy/issues/4753 https://github.com/numpy/numpy/pull/4815 I'm not to familiar with all the complications of subclassing so I can't really say how hard this is to solve. My issue is that it there still seems to be debate on how to handle operator overriding correctly and I am opposed to releasing a numpy with yet another experimental feature that may or may not be finished sometime later. Having datetime in infinite experimental state is bad enough. I think nobody is served well if we release 1.9 with the feature prematurely based on a not representative set of users and the later after more users showed up see we have to change its behavior. So I'm wondering if we should delay the introduction of this feature to 1.10 or is it important enough to wait until there is a consensus on the remaining issues?

3 3

Pull Request: Dirichlet Distribution
by Johannes Kulick 21 Jul '14

21 Jul '14

Hi, I sent a pull request, that implements a Dirichlet distribution. Code review would be appreciated! https://github.com/scipy/scipy/pull/3815 Best, Johannes Kulick -- Question: What is the weird attachment to all my emails? Answer: http://en.wikipedia.org/wiki/Digital_signature

1 0