Mailman 3 September 2014 - SciPy-Dev

Re: [SciPy-Dev] removing umfpack wrapper
by Robert Cimrman 23 Nov '14

23 Nov '14

On 11/12/2013 11:22 PM, Ralf Gommers wrote: > On Tue, Nov 12, 2013 at 10:20 AM, Robert Cimrman <cimrman3(a)ntc.zcu.cz>wrote: > >> Hi Ralf, >> >> On 11/10/2013 04:36 PM, Ralf Gommers wrote: >>> Hi all, >>> >>> This is long overdue, but I think it's time to remove the UMFPACK wrapper >>> before the next release. I'd like some feedback on how to do that and to >>> what package (if any) to point existing users. >>> >>> As for why to remove the wrapper, see: >>> https://github.com/scipy/scipy/issues/3002 >>> http://permalink.gmane.org/gmane.comp.python.scientific.user/20451 >>> Short summary: UMFPACK changed from a BSD-compatible license to GPL at >> some >>> point. >>> >>> The deprecation warning in sparse.linalg has been referring people to >>> scikits.umfpack until now, however that package has disappeared >> completely >>> as far as I can tell. I suspect it was in the old scikits svn repo and >> was >>> never moved before that was killed. The alternatives seems to be >> Pysparse ( >> >> I missed that the scikit was lost in transition... >> >> How much work would be to move it to the new site - move from SVN to git, >> and? >> I would be willing to do it, if there is interest. >> > > Hi Robert, I think the moving wouldn't be a lot of work (assuming svn > access can still be arranged). If you'd revive the scikit it's not a > one-time effort though - it's only useful if the code is being maintained > imho. That can still be low effort perhaps, but with reviewing some PRs, > maintenance releases, putting it on pypi, etc. I'd expect at least an hour > a week or so. > > Are there scikits.umfpack users now? Would it make sense to salvage > whatever is useful from the scikit and contribute it to pysparse instead? We are using it in sfepy as the default direct solver. It seems to me that the pysparse interface requires the input matrix to be in the LL (linked list) format, which is unfortunate for us, as we use CSR. The scipy (and former scikit) umfpack wrappers have used CSR, so no copies were necessary. (After all, I have created the original scipy wrappers to be used from sfepy in the first place...) So in case others are not interested in having the scikit, I can see two possible solutions from my perspective: either enhance pysparse interface to allow CSR as well, or move the wrappers to sfepy (which I maintain anyway, but swig -> cython conversion would be needed). Not sure yet which solution I prefer. Cheers, r. > Cheers, > Ralf >> >> Cheers, >> r. >> >>> http://pysparse.sourceforge.net) and PyUBlasExt ( >>> https://pypi.python.org/pypi/PyUblasExt). Scikits.sparse is dead(-ish) >> and >>> doesn't wrap umfpack it looks like. PyUBlasExt has something that looks >>> like a wrapper but for C++ (?). So can we refer people to pysparse? >>> >>> Other question: do we call Pysparse instead of scikits.umfpack when >>> available? And if so, by default or not? Or do we rip out everything that >>> sits under useUmfpack=True? >>> >>> Cheers, >>> Ralf

2 5

Expose Lapack for Cython
by Ian Henriksen 20 Nov '14

20 Nov '14

Hi all, I have put together pxd files that allow direct access from Cython to the the blas and lapack functions wrapped in scipy.linalg.blas and scipy.linalg.lapack. The idea is to make these low level functions available in Cython without the corresponding Python level function call that is necessary when using the wrappers already in SciPy. They are currently in my github repository https://github.com/insertinterestingnamehere/cylinalg. The approach is based on the gist https://gist.github.com/pv/5437087. I think this would be a good addition to SciPy since it is really only another interface to the wrappers that are already there. Where would be the best place to put this? Also, what sorts of tests should be included? Any other input is welcome too. Thanks! -Ian Henriksen

7 17

0.14.1 and 0.15.0 release schedule
by Ralf Gommers 04 Nov '14

04 Nov '14

Hi all, It's about time for a couple of releases. 0.14.1 fixes some regressions as well as a lot of test failures against numpy 1.9.0 (due to __numpy_ufunc__) removal. All known regressions are fixed now, so I plan to tag it this weekend. If there's anything really urgent that still needs to go in, please speak up. For 0.15.0 there are still 17 issues/PRs, but most of those we should be able to merge/fix relatively quickly. I propose the following release schedule: - beta 1: 16 Sep - release candidate 1: 4 Oct - release candidate 2: 18 Oct (if needed) - final release: 25 Oct Does that work for everyone? Are there issues/PRs that aren't under the 0.15.0 milestone but have to go in? Cheers, Ralf

7 13

Linalg: function for diagonalize hermitian commuting matrices simultaneously
by Lars-Hendrik Frahm 07 Oct '14

07 Oct '14

Dear SciPy Developers, I like to contribute a function for simultaneously diagonalize a set of hermitian commuting matrices. Especially for matrices with degenerate eigenvalues. Is there need for something like that in SciPy ? I think it could be helpful for someone else and I didn't find it in recent SciPy versions or on the mailing list. Hope to hear from you.

3 3

Bootstrap confidence limits code
by Constantine Evans 04 Oct '14

04 Oct '14

Hello everyone, A few years ago I implemented a scikit for bootstrap confidence limits (https://github.com/cgevans/scikits-bootstrap). I didn’t think much about it after that until recently, when I realized that some people are actually using it, and that there’s apparently been some talk about implementing this functionality in either scipy.stats or statsmodels (I should thank Randal Olson for discussing this and bringing it to my attention). As such I’ve rewritten most of the code, and written up some docstrings. The current code can do confidence intervals with basic percentile interval, bias-corrected accelerated, and approximate bootstrap confidence methods, and can also provide bootstrap and jackknife indexes. Most of it is implemented from the descriptions in Efron and Tibshirani’s Introduction to the Bootstrap, but the ABC code at the moment is a port from the modified-BSD-licensed bootstrap package for R (not the boot package) as I’m not entirely confident in my understanding of the method. And so, I have a few questions for everyone: * Is there any interest in including this sort of code in either scipy.stats or statsmodels? If so, where do people think would be the better place? The code is relatively small; at the moment it is less than 200 lines, with docstrings probably making up 100 of those lines. * Also, if so, what would need to be changed, added, and improved beyond what is mentioned in the Contributing to Scipy part of the reference guide? I’m never a fan of my own code, and imagine quite a bit would need to be fixed; I know tests will need to be added too. In addition, I have a few questions about what would be better practice for the API, and I haven’t really found a guide on best practices for Scipy: * When I started writing the code, I wrote a single function ci for confidence intervals, with a method argument to choose the method. This is easy for users, especially so that they don’t have to look through documentation to realize that BCA is the most generally useful method (at least from everything I’ve read) and that there really isn’t any reason to use many of the simpler methods. However, ABC takes different paramenters, and needs a statistic function that takes weights, which makes this single-function organization trickier. At the moment, I have a separate function for ABC. Would it be better to split up all the methods to their own functions? * ABC requires a statistic function that takes weights. I’ve noticed that things like np.average takes a weights= argument. Would it be better to require input of a stat(data,weights) function, or input of a stat(data,weights=) with weights as a named argument? The latter would be nice in terms of allowing the same function to be used for all methods, but would make it impossible to use a lambda for the function. Is there some other method of doing this entirely? * Are there any missing features that anyone thinks should be added? I apologize if much of this is answered elsewhere, I just haven’t found any of it; I also apologize if this is far too long-winded and confusing! Regards, Constantine Evans

4 3

last call for numpy 1.8.2 bugfixes
by Julian Taylor 01 Oct '14

01 Oct '14

hi, as numpy 1.9 is going to be a relative hard upgrade as indexing changes expose a couple bugs in third party packages and the large amount of small little incompatibilities I will create a numpy 1.8.2 release tomorrow with a couple of important or hard to work around bugfixes. The most important bugfix is fixing the wrong result partition with multiple selections could produce if selections ended up in an equal range, see https://github.com/numpy/numpy/issues/4836 (if the crash is still unreproducable, help appreciated). the rest of the fixes are small ones listed below. If I have missed one or you consider one of the fixes to invasive for a bugfix release please speak up now. As the number of fixes is small I will skip a release candidate. Make fftpack._raw_fft threadsafe https://github.com/numpy/numpy/issues/4656 Prevent division by zero https://github.com/numpy/numpy/issues/650 Fix lack of NULL check in array_richcompare https://github.com/numpy/numpy/issues/4613 incorrect argument order to _copyto in in np.nanmax, np.nanmin https://github.com/numpy/numpy/issues/4628 Hold GIL for types with fields, fixes https://github.com/numpy/numpy/issues/4642 svd ufunc typo https://github.com/numpy/numpy/issues/4733 check alignment of strides for byteswap https://github.com/numpy/numpy/issues/4774 add missing elementsize alignment check for simd reductions https://github.com/numpy/numpy/issues/4853 ifort has issues with optimization flag /O2 https://github.com/numpy/numpy/issues/4602

6 9

unicode_literals for string arguments
by Andreas Hilboll 26 Sep '14

26 Sep '14

Hi, I'm running into some issues which I attribute to not using from __future__ import unicode_literals. I'm running Python2.7, and am writing a script where I use from __future__ import unicode_literals. This raises ValueErrors e.g. in stats.binned_statistic_2d: ValueError: statistic not understood Do you see a way to prevent these errors from happening? Obviously, scipy should work no matter if a Py2 user uses from __future__ import unicode_literals or not. Any ideas? Cheers, Andreas

3 2

Hamming etc. windows are wrong
by Jerry 26 Sep '14

26 Sep '14

I’ve noticed that the Hamming window (and I suppose other windows) provided by Octave, SciPy****, and NumPy***** is wrong, returning a window that is symmetric rather than one that is “DFT symmetric.” This is easily spotted by looking at the first and last points of the window; if they are the same values, then the window is incorrect. These windows are normally applied to data that are equally spaced, thus implying that their DFTs are periodic. Let’s assume that we are windowing N data points, indexed from 0 to N-1 as is normal in signal processing. Then the DFT is also N points long, also indexed from 0 to N-1. The periodic extension principal implies that the point indexed by N in the DFT domain has the same value as the point indexed by 0. The window can also be imagined to have the same kind of periodic extension, so that what would be the N-th point of the window matches the N-point of the data with the same weight as was applied to the 0-th point of the data but without duplicating that point at N-1. This mistake is widespread, infecting many software packages. (I am sending this same note to the NumPy, SciPy, and Octave lists.) It is certainly wrong in most textbooks, including the classic, Oppenheim & Schafer, 1975. However, the definitive and widely referenced paper on windows by Fredric J. Harris, “On the Use of Windows for Harmonic Analysis with the Discrete Fourier Transform,” Proceedings of the IEEE, vol. 66, NO. 1 , January, 1978 * discusses this problem, both its roots and its correction. For example, quoting from the paper: “Since the DFT essentially considers sequences to be periodic, we can consider the missing end point to be the beginning of the next period of the periodic extension of this sequence. In fact, under the periodic extension, the next sample (at 16 s in Fig. 1.) is indistinguishable from the sample at zero seconds. “This apparent lack of symmetry due to the missing (but implied) end point is a source of confusion in sampled window design. This can be traced to the early work related to convergence factors for the partial sums of the Fourier series. The partial sums (or the finite Fourier transform) always include an odd number of points and exhibit even symmetry about the origin. Hence much of the literature and many software libraries incorporate windows designed with true even symmetry rather than the implied symmetry with the missing end point!” ...and later... “We will now catalog some well-known (and some not well-known windows. For each window we will comment on the justification for its use and identify its significant parameters. All the windows will be presented as even (about the origin) sequences with an odd number of points. To convert the window to DFT even, the right endpoint will be discarded and the sequence will be shifted so that the left end point coincides with the origin.” Note that he limits himself therefore to even-numbered "DFT even" windows, but the rule remains--honor the implied periodicity even for windows of odd length. Lest one consider this a trivial difference, consider the following Octave-Matlab code (corresponding NumPy and SciPy code would yield the same results). x = ones(8, 1); # An even sequence; its DFT should be real. hWrong = hamming(8) h9 = hamming(9); hRight = h9(1:8) xWrong = x .* hWrong; # Multiplication by "ones" xRight = x .* hRight; # shown for clarity. XWrong = fft(xWrong) # Contains nonzero imaginary part. XRight = fft(xRight) # Real, as expected from evenness of x. XWrongMag = abs(XWrong) # The magnitudes are XRightMag = abs(XRight) # also different. This causes the following output: hWrong = 0.080000 0.253195 0.642360 0.954446 0.954446 0.642360 0.253195 0.080000 hRight = 0.080000 0.214731 0.540000 0.865269 1.000000 0.865269 0.540000 0.214731 XWrong = 3.86000 + 0.00000i -1.76795 - 0.73231i 0.13889 + 0.13889i 0.01906 + 0.04602i 0.00000 + 0.00000i 0.01906 - 0.04602i 0.13889 - 0.13889i -1.76795 + 0.73231i XRight = 4.32000 + 0.00000i -1.84000 + 0.00000i 0.00000 + 0.00000i 0.00000 + 0.00000i 0.00000 + 0.00000i 0.00000 - 0.00000i 0.00000 - 0.00000i -1.84000 - 0.00000i XWrongMag = 3.86000 1.91362 0.19642 0.04981 0.00000 0.04981 0.19642 1.91362 XRightMag = 4.32000 1.84000 0.00000 0.00000 0.00000 0.00000 0.00000 1.84000 This news will likely upset many and I will be interested to see what arguments will flow in favor of keeping the status quo--I’m sure one will be, “we’ve always done it this way” and others will be more substantial, possibly expressing decent arguments why the current situation is useful for some applications. In any case, I refer again to the Harris paper. So rather than host an argument on this list, this is what I propose: Do what Matlab** does. Acknowledge both uses by making a flag to handle the “periodic” or the “symmetric” cases. The Matlab default is “symmetric” which is of course unfortunate but at least such inclusion in Octave, NumPy, and SciPy would retain compatibility with the existing usage. Then it’s up to the user whether to shoot him/herself in the foot, assuming that such a decision is guided by actually referring to the documentation for the package being used and not blindly using the default. Jerry Bauck * http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=1455106&url=http%3A%2… ** http://www.mathworks.com/help/signal/ref/hamming.html *** http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.signal.ham… **** I am just learning Python-NumPy-SciPy but it appears as though the SciPy situation is that the documentation page above *** mentions the flag but the flag has not been implemented into SciPy itself. I would be glad to stand corrected. ***** http://docs.scipy.org/doc/numpy/reference/generated/numpy.hamming.html

2 1

ANN: SfePy 2014.3
by Robert Cimrman 25 Sep '14

25 Sep '14

I am pleased to announce release 2014.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (preliminary support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: http://groups.google.com/group/sfepy-devel Git (source) repository, issue tracker, wiki: http://github.com/sfepy Highlights of this release -------------------------- - isogeometric analysis (IGA) speed-up by C implementation of NURBS basis evaluation - generalized linear combination boundary conditions that work between different fields/variables and support non-homogeneous periodic conditions - non-constant essential boundary conditions given by a function in IGA - reorganized and improved documentation For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladimir Lukes, Matyas Novak, Zhihua Ouyang, Jaroslav Vondrejc

1 0

Fast symmetric rank-one Cholesky decomposition update
by Danilo Horta 19 Sep '14

19 Sep '14

Hi folks, Let L be the Cholesky decomposition of A. There are O(n^2) methods for updating L such that L is now the Cholesky decomposition of A + numpy.outer(v, v) (and also for A - np.outer(v, v)). Is this something adequate to scipy? If so, I'm willing to implement. (I need this kind of method...) Best, Danilo

3 2