Mailman 3 September 2013 - SciPy-Dev

Linalg: function for diagonalize hermitian commuting matrices simultaneously
by Lars-Hendrik Frahm 07 Oct '14

07 Oct '14

Dear SciPy Developers, I like to contribute a function for simultaneously diagonalize a set of hermitian commuting matrices. Especially for matrices with degenerate eigenvalues. Is there need for something like that in SciPy ? I think it could be helpful for someone else and I didn't find it in recent SciPy versions or on the mailing list. Hope to hear from you.

3 3

Bootstrap confidence limits code
by Constantine Evans 04 Oct '14

04 Oct '14

Hello everyone, A few years ago I implemented a scikit for bootstrap confidence limits (https://github.com/cgevans/scikits-bootstrap). I didn’t think much about it after that until recently, when I realized that some people are actually using it, and that there’s apparently been some talk about implementing this functionality in either scipy.stats or statsmodels (I should thank Randal Olson for discussing this and bringing it to my attention). As such I’ve rewritten most of the code, and written up some docstrings. The current code can do confidence intervals with basic percentile interval, bias-corrected accelerated, and approximate bootstrap confidence methods, and can also provide bootstrap and jackknife indexes. Most of it is implemented from the descriptions in Efron and Tibshirani’s Introduction to the Bootstrap, but the ABC code at the moment is a port from the modified-BSD-licensed bootstrap package for R (not the boot package) as I’m not entirely confident in my understanding of the method. And so, I have a few questions for everyone: * Is there any interest in including this sort of code in either scipy.stats or statsmodels? If so, where do people think would be the better place? The code is relatively small; at the moment it is less than 200 lines, with docstrings probably making up 100 of those lines. * Also, if so, what would need to be changed, added, and improved beyond what is mentioned in the Contributing to Scipy part of the reference guide? I’m never a fan of my own code, and imagine quite a bit would need to be fixed; I know tests will need to be added too. In addition, I have a few questions about what would be better practice for the API, and I haven’t really found a guide on best practices for Scipy: * When I started writing the code, I wrote a single function ci for confidence intervals, with a method argument to choose the method. This is easy for users, especially so that they don’t have to look through documentation to realize that BCA is the most generally useful method (at least from everything I’ve read) and that there really isn’t any reason to use many of the simpler methods. However, ABC takes different paramenters, and needs a statistic function that takes weights, which makes this single-function organization trickier. At the moment, I have a separate function for ABC. Would it be better to split up all the methods to their own functions? * ABC requires a statistic function that takes weights. I’ve noticed that things like np.average takes a weights= argument. Would it be better to require input of a stat(data,weights) function, or input of a stat(data,weights=) with weights as a named argument? The latter would be nice in terms of allowing the same function to be used for all methods, but would make it impossible to use a lambda for the function. Is there some other method of doing this entirely? * Are there any missing features that anyone thinks should be added? I apologize if much of this is answered elsewhere, I just haven’t found any of it; I also apologize if this is far too long-winded and confusing! Regards, Constantine Evans

4 3

1.0 roadmap: weave
by Arnd Baecker 06 May '14

06 May '14

Hi, > On Sat, Sep 21, 2013 at 8:54 PM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: [...] > weave > ````` > This is the only module that was not ported to Python 3. Effectively it's > deprecated (not recommended to use for new code). In the future it should be > removed from scipy (can be made into a separate module). Let me just mention that I am worried about the removal of weave: a substantial amount of code developed in our group over the years uses weave. Porting this to cython seems not trivial, in particular, because many of the original authors (diploma students, PhD students, ...) have left academia by now. Well, as porting weave to Python 3 is way out of my capabilities, this is just a comment in the hope that someone could give this a try (if at some point testing would be needed, I could try to help, of course). Best, Arnd

9 12

SLSQP Constrained Optimizer Status
by Rob Falck 24 Nov '13

24 Nov '13

I'm currently implementing the Sequential Least Squares Quadratic Programming (SLSQP) optimizer, by Dieter Kraft, for use in Scipy. The Fortran code being wrapped with F2PY is here: http://www.netlib.org/toms/733 (its use within Scipy has been cleared) SLSQP provides for bounds on the independent variables, as well as equality and inequality constraint functions, which is a capability that doesn't exist in scipy.optimize. Currently the code works, although the constraint normals are being generated by approximation. I'm working on a way to pass these in. I think the most elegant way will be a single function that returns the matrix of constraint normals. For a demonstration of what the code can do, here is an optimization of f(x,y) = 2xy + 2x - x**2 - 2y**2 Example 14.2 in Chapra & Canale gives the maximum as x=2.0, y=1.0. The unbounded optimization tests find this solution. As expected, its faster when derivatives are provided rather than approximated. Unbounded optimization. Derivatives approximated. Elapsed time: 1.45792961121 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] Unbounded optimization. Derivatives provided. Elapsed time: 1.03211402893 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] The following example uses an equality constraint to find the optimal when x=y. Bound optimization. Derivatives approximated. Elapsed time: 1.384973526 ms Results [[0.99999996845920858, 0.99999996845920858], -0.99999999999999889, 4, 0, 'Optimization terminated successfully.'] I've tried to conform to the syntax used by the other optimizers in scipy.optimize. The function definition and doc string are below. If anyone is interested in testing it out, let me know. def fmin_slsqp( func, x0 , eqcons=[], f_eqcons=None, ieqcons=[], f_ieqcons=None, bounds = [], fprime = None, fprime_cons=None,args = (), iter = 100, acc = 1.0E-6, iprint = 1, full_output = 0, epsilon = _epsilon ): """ Minimize a function using Sequential Least SQuares Programming Description: Python interface function for the SLSQP Optimization subroutine originally implemented by Dieter Kraft. Inputs: func - Objective function (in the form func(x, *args)) x0 - Initial guess for the independent variable(s). eqcons - A list of functions of length n such that eqcons[j](x0,*args) == 0.0 in a successfully optimized problem f_eqcons - A function of the form f_eqcons(x, *args) that returns an array in which each element must equal 0.0 in a successfully optimized problem. If f_eqcons is specified, eqcons is ignored. ieqcons - A list of functions of length n such that ieqcons[j](x0,*args) >= 0.0 in a successfully optimized problem f_ieqcons - A function of the form f_ieqcons(x0, *args) that returns an array in which each element must be greater or equal to 0.0 in a successfully optimized problem. If f_ieqcons is specified, ieqcons is ignored. bounds - A list of tuples specifying the lower and upper bound for each independent variable [ (xl0, xu0), (xl1, xu1), ...] fprime - A function that evaluates the partial derivatives of func fprime_cons - A function of the form f(x, *args) that returns the m by n array of constraint normals. If not provided, the normals will be approximated. Equality constraint normals precede inequality constraint normals. args - A sequence of additional arguments passed to func and fprime iter - The maximum number of iterations (int) acc - Requested accuracy (float) iprint - The verbosity of fmin_slsqp. iprint <= 0 : Silent operation iprint == 1 : Print summary upon completion (default) iprint >= 2 : Print status of each iterate and summary full_output - If 0, return only the minimizer of func (default). Otherwise, output final objective function and summary information. epsilon - The step size for finite-difference derivative estimates. Outputs: ( x, { fx, gnorm, its, imode, smode }) x - The final minimizer of func. fx - The final value of the objective function. its - The number of iterations. imode - The exit mode from the optimizer, as an integer. smode - A string describing the exit mode from the optimizer. Exit modes are defined as follows: -1 : Gradient evaluation required (g & a) 0 : Optimization terminated successfully. 1 : Function evaluation required (f & c) 2 : Number of equality constraints larger than number of independent variables 3 : More than 3*n iterations in LSQ subproblem 4 : Inequality constraints incompatible 5 : Singular matrix E in LSQ subproblem 6 : Singular matrix C in LSQ subproblem 7 : Rank-deficient equality constraint subproblem HFTI 8 : Positive directional derivative for linesearch 9 : Iteration limit exceeded -- - Rob Falck

4 16

Vector Strength function
by Todd 21 Nov '13

21 Nov '13

I am interested in implementing a function for scipy. The function is called "vector strength". It is basically a measure of how reliably a set of events occur at a particular phase. It was originally developed for neuroscience research, to determine how well a set of neural events sync up with a periodic stimulus like a sound waveform. However, it is useful for determining how periodic a supposedly periodic set of events really are, for example: 1. Determining whether crime is really more common during a full moon and by how much 2. Determining how concentrated visitors to a coffee shop are during rush hour 3. Determining exactly how concentrated hurricanes are during hurricane season My thinking is that this could be implemented in stages: First would be a Numpy function that would add a set of vectors in polar coordinates. Given a number of magnitude/angle pairs it would provide a summed magnitude/angle pair. This would probably be combined with a cartesian<->polar conversion functions. Making use of this function would be a scipy function that would actually implement the vector strength calculation. This is done by treating each event as a unit vector with a phase, then taking the average of the vectors. If all events have the same phase, the result will have an amplitude of 1. If they all have a different phases, the result will have an amplitude of 0. It may even be worth having a dedicated polar dtype, although that may be too much. What does everyone think of this proposal?

4 12

Quadpack conversion to Fortran
by Brian Lee Newsom 09 Oct '13

09 Oct '13

Hello, My name is Brian Newsom and I am just beginning to use/work on the scipy library, specifically the quadpack within the integration pack. Because of the recursion and wrapping methods used now, running nquad (scipy 0.13) function is slow-the fortran and python are nested in a way that requires switching back and forth between the two often. Ideally this inefficiency could be overcome by translating the main interface (quadpack.py) into fortran and just having a minimal python wrapper that is not used in the recursion. First off, has anyone attempted (or is currently attempting) this? Are there any major difficulties I should be wary of or is this unrealistic for any reason? Additionally, I was curious if there are any computational advantages to using C or Fortran for this purpose within this library, either is a possibility in the case that one is strongly preferred. Otherwise, I am sure I will be full of questions in the near future with specific regards to the building and setup of these files within the scipy library. Thank you, Brian Newsom Brian.Newsom(a)colorado.edu

3 5

1.8.0rc1
by Charles R Harris 02 Oct '13

02 Oct '13

Hi All, NumPy 1.8.0rc1 is up now on sourceforge<http://sourceforge.net/projects/numpy/files/NumPy/1.8.0rc1/>.The binary builds are included except for Python 3.3 on windows, which will arrive later. Many thanks to Ralf for the binaries, and to those who found and fixed the bugs in the last beta. Any remaining bugs are all my fault ;) I hope this will be the last release before final, so please test it thoroughly. Chuck

6 9

ImportError: cannot import name oldnumeric
by xavier.gnata＠gmail.com 30 Sep '13

30 Sep '13

Hi, I have compiled numpy1.9.0.dev-e051ff following http://osdf.github.io/blog/numpyscipy-with-openblas-for-ubuntu-1204-second-… to unable Openblas. It looks ok (run=5277 errors=0 failures=0 and Openblas is used) but now scipy git complains like this: "python setup.py build Cythonizing sources scipy/interpolate/interpnd.pyx has not changed scipy/ndimage/src/_ni_label.pyx has not changed scipy/stats/vonmises_cython.pyx has not changed scipy/stats/_rank.pyx has not changed scipy/signal/_spectral.pyx has not changed scipy/special/_ufuncs_cxx.pyx has not changed scipy/special/_ufuncs.pyx has not changed scipy/cluster/_vq_rewrite.pyx has not changed scipy/sparse/csgraph/_shortest_path.pyx has not changed scipy/sparse/csgraph/_traversal.pyx has not changed scipy/sparse/csgraph/_min_spanning_tree.pyx has not changed scipy/sparse/csgraph/_tools.pyx has not changed scipy/io/matlab/mio5_utils.pyx has not changed scipy/io/matlab/mio_utils.pyx has not changed scipy/io/matlab/streams.pyx has not changed scipy/spatial/ckdtree.pyx has not changed scipy/spatial/qhull.pyx has not changed /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_suite' warnings.warn(msg) blas_opt_info: blas_mkl_info: Disabled blas_mkl_info: (MKL is None) Disabled blas_mkl_info: (MKL is None) libraries mkl,vml,guide not found in [] NOT AVAILABLE openblas_info: FOUND: libraries = ['openblas'] library_dirs = ['/usr/local/lib'] language = f77 FOUND: libraries = ['openblas'] library_dirs = ['/usr/local/lib'] language = f77 Traceback (most recent call last): File "setup.py", line 230, in <module> setup_package() File "setup.py", line 227, in setup_package setup(**metadata) File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/core.py", line 135, in setup config = configuration() File "setup.py", line 170, in configuration config.add_subpackage('scipy') File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 966, in add_subpackage caller_level = 2) File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 935, in get_subpackage caller_level = caller_level + 1) File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 872, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/setup.py", line 15, in configuration config.add_subpackage('lib') File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 966, in add_subpackage caller_level = 2) File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 935, in get_subpackage caller_level = caller_level + 1) File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 872, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/lib/setup.py", line 9, in configuration config.add_subpackage('blas') File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 966, in add_subpackage caller_level = 2) File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 935, in get_subpackage caller_level = caller_level + 1) File "/usr/local/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 872, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/lib/blas/setup.py", line 22, in configuration from scipy._build_utils import get_g77_abi_wrappers File "/usr/local/src/scipy/scipy/__init__.py", line 77, in <module> from numpy import oldnumeric ImportError: cannot import name oldnumeric " What's going on with oldnumeric? xavier

2 2

Scipy 1.0 roadmap - wavelets
by Daπid 28 Sep '13

28 Sep '13

On 21 September 2013 20:57, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: > *Wavelets*: add proper wavelets, including discrete wavelet transform. > What's > there now doesn't make much sense. > There is a package on Wavelets under MIT license, but it looks abandoned for more than a year. Perhaps it could be reintegrated into Scipy, perhaps it would be better to create a scikit-wavelets while we get something of enough quality. http://www.pybytes.com/pywavelets I can step in, if someone helps and guides me. /David.

3 4

Re: [SciPy-Dev] [Numpy-discussion] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence"
by Nathaniel Smith 27 Sep '13

27 Sep '13

On Fri, Sep 27, 2013 at 8:27 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > 27.09.2013 22:15, Nathaniel Smith kirjoitti: > [clip] >> 3) The issue of how to make an in-place like ndarray += sparse >> continue to work in the brave new __numpy_ufunc__ world. >> >> For this last issue, I think we disagree. It seems to me that the >> right answer is that csc_matrix.__numpy_ufunc__ needs to step up and >> start supporting out=! If I have a large dense ndarray and I try to += >> a sparse array to it, this operation should take no temporary memory >> and nnz time. Right now it sounds like it actually copies the large >> dense ndarray, which takes time and space proportional to its size. >> AFAICT the only way to avoid that is for scipy.sparse to implement >> out=. It shouldn't be that hard...? > > Sure, scipy.sparse can easily support also the output argument. Great! I guess solving this somehow will be release-critical, to avoid a regression in this case when __numpy_ufunc__ gets released. If the solution should be in scipy, I guess we should file the bug there? > But I still maintain that the implementation of __iadd__ in Numpy is > wrong. Oh yeah totally. > What it does now is: > > def __iadd__(self, other): > return np.add(self, other, out=self) > > But since we know this raises a TypeError if the input is of a type that > cannot be dealt with, it should be: > > def __iadd__(self, other): > try: > return np.add(self, other, out=self) > except TypeError: > return NotImplemented > > Of course, it's written in C so it's a bit more painful to write this. > > I think this will have little performance impact, since the check would > be only a NULL check in the inplace op methods + subsequent handling. I > can take a look at some point at this... I'm a little uncertain about the "swallow all TypeErrors" aspect of this -- e.g. this could have really weird effects for object arrays, where ufuncs may raise arbitrary user exceptions. One possibility in the long run is to just say, if you want to override ndarray __iadd__ or whatever, then you have to use __numpy_ufunc__. Not really much point in having *two* sets of implementations of the NotImplemented dance for the same operation. -n

2 1

Re: [SciPy-Dev] [Numpy-discussion] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence"
by Nathaniel Smith 27 Sep '13

27 Sep '13

On Fri, Sep 27, 2013 at 7:34 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > 27.09.2013 19:33, Nathaniel Smith kirjoitti: > [clip] >> I really don't understand what arcane magic is used to make ndarray += >> csc_matrix work at all, but my question is, is it going to break when >> we complete the casting transition described above? It was just >> supposed to catch things like int += float. > > This maybe clarifies it: > >>>> import numpy >>>> import scipy.sparse >>>> x = numpy.ones((2,2)) >>>> y = scipy.sparse.csr_matrix(x) >>>> z = x >>>> z += y >>>> x > array([[ 1., 1.], > [ 1., 1.]]) >>>> z > matrix([[ 2., 2.], > [ 2., 2.]]) > > The execution flows like this: > > ndarray.__iadd__(arr, sparr) > np.add(arr, sparr, out=???) > return NotImplemented # wtf > return NotImplemented > Python does arr = sparr.__radd__(arr) > > Since Scipy master sparse matrices now have __numpy_ufunc__, but it > doesn't handle out= arguments, the second step currently raises a > TypeError (for Numpy master + Scipy master). > > And this is actually the correct thing to do, as having np.add return > NotImplemented is just broken. Only ndarray.__iadd__ has the authority > to return the NotImplemented. > > To make the in-place ops work again, it seems Numpy needs some > additional fixes in its binary op machinery, before __numpy_ufunc__ > business works fully as intended. Namely, the binary op routines will > need to catch TypeErrors and convert them to NotImplemented. > > The code paths where Numpy ufuncs currently return NotImplemented could > also be changed to raise TypeErrors, but I'm not sure if someone > somewhere relies on this behavior (I hope not). Okay, so I see three separate issues: 1) My original concern, that the upcoming casting change for in-place operations will cause some horrible interaction. Tentatively this seems like it might be okay since even after the "cast" succeeds, np.add is still just refusing to do the operation, so hopefully we can set it up so that it will continue to fail once the casting rule becomes more strict. 2) The issue that ufuncs return NotImplemented and it makes baby Guido cry. This is completely broken, agreed. Not sure when someone will get around to clearing this stuff up. 3) The issue of how to make an in-place like ndarray += sparse continue to work in the brave new __numpy_ufunc__ world. For this last issue, I think we disagree. It seems to me that the right answer is that csc_matrix.__numpy_ufunc__ needs to step up and start supporting out=! If I have a large dense ndarray and I try to += a sparse array to it, this operation should take no temporary memory and nnz time. Right now it sounds like it actually copies the large dense ndarray, which takes time and space proportional to its size. AFAICT the only way to avoid that is for scipy.sparse to implement out=. It shouldn't be that hard...? -n

1 0

Re: [SciPy-Dev] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence"
by Nathaniel Smith 27 Sep '13

27 Sep '13

[CC'ing scipy-dev because see below] On Tue, Sep 24, 2013 at 6:45 PM, Nathaniel Smith <njs(a)pobox.com> wrote: > Hi all, > > I'm getting a very strange error, and unfortunately I can't seem to > reproduce it *except* when running on Travis-CI, so my debugging tools > are really limited. I'm wondering if anyone else has seen anything > like this? > > The offending line of code is: > > File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pyrerp/incremental_ls.py", > line 323, in append_bottom_half > self.xtx += xtx > ValueError: setting an array element with a sequence. > > And debug prints reveal that in the case that causes the error, > 'self.xtx' is an ndarray that prints as: > > [[ 0. 0.] > [ 0. 0.]] > > while 'xtx' is a scipy.sparse.csc.csc_matrix that prints as: > > (1, 0) 45.0 > (0, 0) 10.0 > (1, 1) 285.0 > (0, 1) 45.0 In accordance with the cosmic law governing such things, this turns out to be a bug in numpy that I introduced in 2397c9d4, specifically this line, which introduces an unchecked DEPRECATE: https://github.com/numpy/numpy/commit/2397c9d4#L5R528 (Plus some complicated nonsense involving virtualenvs that sometimes fall back on system libraries even though the virtualenv contains a perfectly good version of the same library etc. to make reproduction more confusing.) My test code was setting warnings to raise errors by default, and apparently ndarray += csc_matrix goes through some circuitous path that (AFAICT) involves creating an object ndarray containing the csc_matrix and calling the add ufunc, which somehow trips on the cast warning above. Then later on, at line 159 of arraytypes.c.src, the the @TYPE@_setitem function does: if (PyErr_Occurred()) { if (PySequence_Check(op) && !PyString_Check(op) && !PyUnicode_Check(op)) { PyErr_Clear(); PyErr_SetString(PyExc_ValueError, "setting an array element with a sequence."); } return -1; } and the PyErr_Occurred here catches the real error and replaces it with some nonsense. So three points: 1) Maybe this will be useful to someone googling later. 2) I guess I'll file some horrible patch against numpy that just throws away exceptions caused by that deprecation, because there is no way for PyArray_CanCastTypeTo to raise an error :-(. 3) This script raises a DeprecationWarning with numpy 1.7.1 and scipy 0.12.0: import warnings, numpy, scipy.sparse warnings.filterwarnings("always") a = numpy.zeros((2, 2)) b = scipy.sparse.csc_matrix([[0.0, 0.0], [0.0, 0.0]]) a += b test-script3.py:5: DeprecationWarning: Implicitly casting between incompatible kinds. In a future numpy release, this will raise an error. Use casting="unsafe" if this is intentional. I really don't understand what arcane magic is used to make ndarray += csc_matrix work at all, but my question is, is it going to break when we complete the casting transition described above? It was just supposed to catch things like int += float. -n

1 0

Scipy 1.0 roadmap
by Ralf Gommers 24 Sep '13

24 Sep '13

Hi all, At EuroScipy Pauli, David and I sat together and drafted a roadmap for Scipy 1.0. We then discussed this offline with some of the other currently most active core devs, to get it into a state that's ready for discussion on this list. So here it is: https://github.com/scipy/scipy/pull/2908 Our aim is for this roadmap to help guide us towards a 1.0 version, which will contain only code that we consider to be "of sufficient quality". Also, it will help to communicate to new and potential developers where their contributions are especially needed. In order to discuss/review this roadmap without generating a monster thread, I propose the following: - topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread. - things in the General section (API changes, documentation/test/build guidelines, etc.), are discussed on this thread as well. - for discussion of module-specific content, start a new thread and name it "1.0 roadmap: <module_name>". - for minor things, comment on the PR. Cheers, Ralf

14 25

Comment on scipy.interpolate.UnivariateSpline error
by Padarn Wilson 23 Sep '13

23 Sep '13

Hi, I (well actually a friend) was recently trying to fit a UnivariateSpline to some data and then evaluate the derivative. The spline generated fine and could be evaluated (it looked reasonable) but the derivative would give the error: File "/usr/lib/pytho n2.7/dist-packages/scipy/interpolate/fitpack2.py", line 258, in derivatives raise ValueError("Error code returned by spalde: %s" % ier) ValueError: Error code returned by spalde: 10 It turned out that the x-array input was not in ascending order, as the documentation says is required. The exact data I was using lead to non-ascending knots. I recognise this error is not encountered if the documentation is followed, but the error is uninformative, and it seems like an easy mistake to make. Especially given the spline still looks fine otherwise. In case anyone wants to reproduce the example, the data are: 2.651,1.421 2.581,1.36 2.494,1.256 2.398,1.186 2.328,1.116 2.232,1.046 2.11,0.951 2.041,0.881 1.945,0.837 1.84,0.794 1.727,0.733 1.631,0.689 1.57,0.663 1.483,0.602 1.413,0.567 1.291,0.515 1.203,0.488 1.108,0.445 0.994,0.392 0.898,0.349 0.802,0.314 0.724,0.279 0.619,0.253 0.523,0.218 0.419,0.183 0.314,0.14 0.253,0.096 0.157,0.078 0.087,0.044 Thanks, Padarn

1 0

Voting functionality on scipy central
by Surya Kasturi 22 Sep '13

22 Sep '13

Hi there, At this time, we thought of giving a relook to Reputation system we built for SciPy Central. There are some of design decisions that have to be looked into 1. Should new revisions carry on reputation of previous revision? or 2. Should we attach reputation to "Submission" as a whole or each "Revision" individually? Thanks

3 6

how to turn internal names into API and freeze them forever
by josef.pktd＠gmail.com 22 Sep '13

22 Sep '13

The scipy.stats.distributions allow now keyword access to the shape parameters. Before, the names of shape parameters where just part of the signature, but couldn't be used in code, only positional argument where possible. Now, users can actually use them in their code. hypergeom uses `M,n,N` where I usually need to read the description for several minutes to remember what they are. most distributions use one letter parameters: Since I just checked numpy.random.triangular uses `mode` scipy.triang just uses `c` Josef

2 1

Re: [SciPy-Dev] Scipy 1.0 roadmap - stats
by josef.pktd＠gmail.com 22 Sep '13

22 Sep '13

On Sun, Sep 22, 2013 at 4:52 AM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: > > > > On Sun, Sep 22, 2013 at 7:52 AM, Christopher Jordan-Squire <cjordan1(a)uw.edu> > wrote: >> >> For scipy stats, is there anything on the table regarding somehow >> unifying the sampling in numpy.random and the distributions in >> scipy.stats? I'm specifically thinking of two issues: >> >> (1) There's a lot of duplication between numpy.random and scipy.stats >> but with different interfaces. This seems like something that ideally >> would be reduced. > > > numpy.random only provides sampling and only has about half the > distributions of scipy.stats. Sampling is really only a small part of what > scipy.stats provides (pdf, cdf, moments, fitting a distribution, etc.). So > I'm not bothered by that duplication. If we'd want to reduce it I think it > would have to be removed from numpy, which doesn't sound like a good idea. There is no code duplication, so it also never bothered me. scipy.stats distributions has quite a bit more overhead than numpy random if you call random number generation repeatedly instead of requesting one big array. There are some naming inconsistencies between scipy.stats and numpy.random, but I never looked systematically for that, and there is no open issue. The distributions in scipy.stats have restrictions on the choice of paramaterization because of the generic use of loc and scale. One issue that should be added to the roadmap is fixing the broadcasting of loc and scale in the scipy random numbers. I don't see a way to fix this in a backwards compatible way. --- Some functions like nanmean, nanstd and others can be removed from scipy.stats because they will be available in numpy. (when scipy requires a minimum version of numpy that contains those.) > >> >> (2) The interface for the distributions in scipy.stats seems to >> explicitly be for scalar random variables, so there's no multivariate >> normals, multinomials, dirichlet, wishart, etc.. Instead the sampling >> is in numpy.random, and pdf's aren't there. > > > Two days ago PR-2726 was merged, which adds a multivariate normal > distribution. Others can be added. IIRC there has been an enhancement ticket > for wishhart somewhere and there's a Python implementation floating around > somewhere. > > Cheers, > Ralf > >> >> Has this been discussed elsewhere? No, all the discussions for scipy.stats are inn github issues (including PRs) or on the mailing list. Josef >> >> On Sat, Sep 21, 2013 at 8:03 PM, Blake Griffith >> <blake.a.griffith(a)gmail.com> wrote: >> > >> >> sparse >> >> `````` >> >> >> >> Don't emulate np.matrix behavior, drop 2-D? >> > >> > >> > What is meant by this? Emulate np.array instead? >> > >> > _______________________________________________ >> > SciPy-Dev mailing list >> > SciPy-Dev(a)scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-dev >> > >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev >

1 0

stats test coverage
by josef.pktd＠gmail.com 21 Sep '13

21 Sep '13

fresh from our CI and maintenance team an excerpt 85% build/testenv/lib/python2.7/site-packages/scipy/stats/_binned_statistic.py 393122104180.9 100% build/testenv/lib/python2.7/site-packages/scipy/stats/contingency.py 272373701.0 93% build/testenv/lib/python2.7/site-packages/scipy/stats/distributions.py 7960335631282280.9 100% build/testenv/lib/python2.7/site-packages/scipy/stats/__init__.py 346131301.0 79% build/testenv/lib/python2.7/site-packages/scipy/stats/kde.py 51312599260.8 61% build/testenv/lib/python2.7/site-packages/scipy/stats/morestats.py 16155803552250.6 67% build/testenv/lib/python2.7/site-packages/scipy/stats/mstats_basic.py 20508795912880.7 66% build/testenv/lib/python2.7/site-packages/scipy/stats/mstats_extras.py 466162107550.7 100% build/testenv/lib/python2.7/site-packages/scipy/stats/mstats.py 823301.0 94% build/testenv/lib/python2.7/site-packages/scipy/stats/_multivariate.py 49312511780.9 35% build/testenv/lib/python2.7/site-packages/scipy/stats/rv.py 72176110.4 83% build/testenv/lib/python2.7/site-packages/scipy/stats/stats.py 446310328541780.8 Josef

3 2

welcome Evgeni Burovski to the scipy team
by Ralf Gommers 19 Sep '13

19 Sep '13

Hi all, On behalf of the scipy developers I'd like to welcome Evgeni Burovski as a member of the core dev team. Evgeni has (among other things) spent a lot of effort over the last months on making the statistical distributions more user-friendly and accurate: http://www.ohloh.net/p/scipy/contributors/21026013018665. I'm sure more good stuff will follow:) Cheers, Ralf

6 5

ANN: SfePy 2013.3
by Robert Cimrman 18 Sep '13

18 Sep '13

I am pleased to announce release 2013.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Home page: http://sfepy.org Downloads, mailing list, wiki: http://code.google.com/p/sfepy/ Git (source) repository, issue tracker: http://github.com/sfepy Highlights of this release -------------------------- - implementation of Mesh topology data structures in C - implementation of regions based on C Mesh (*) - MultiProblem solver for conjugate solution of subproblems - new advanced examples (vibro-acoustics, Stokes flow with slip conditions) (*) Warning: region selection syntax has been changed in a principal way, see [1]. Besides the simple renaming, all regions meant for boundary conditions or boundary/surface integrals need to have their kind set explicitly to 'facet' (or 'edge' in 2D, 'face' in 3D). [1] http://sfepy.org/doc-devel/users_guide.html#regions For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladimír Lukeš

1 0

ANN: 1.8.0b2 release.
by Charles R Harris 14 Sep '13

14 Sep '13

Hi all, I'm happy to announce the second beta release of Numpy 1.8.0. This release should solve the Windows problems encountered in the first beta. Many thanks to Christolph Gohlke and Julian Taylor for their hard work in getting those issues settled. It would be good if folks running OS X could try out this release and report any issues on the numpy-dev mailing list. Unfortunately the files still need to be installed from source as dmg files are not avalable at this time. Source tarballs and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b2/. The Windows and OS X installers will follow when the infrastructure issues are dealt with. Chuck

2 1

reported error for numpy 1.8
by Charles R Harris 14 Sep '13

14 Sep '13

Hi all, I don't know if this is known or not, so posting here before opening an issue. This shows up in scipy master as well as 0.12, along with a ton of (expected) deprecation warnings for the latter. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/charris/.local/lib/python2.7/site-packages/scipy/io/matlab/tests/test_mio.py", line 301, in _load_check_case matdict = loadmat(file_name, struct_as_record=True) File "/home/charris/.local/lib/python2.7/site-packages/scipy/io/matlab/mio.py", line 126, in loadmat matfile_dict = MR.get_variables(variable_names) File "/home/charris/.local/lib/python2.7/site-packages/scipy/io/matlab/mio5.py", line 288, in get_variables res = self.read_var_array(hdr, process) File "/home/charris/.local/lib/python2.7/site-packages/scipy/io/matlab/mio5.py", line 248, in read_var_array return self._matrix_reader.array_from_header(header, process) File "mio5_utils.pyx", line 616, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy/io/matlab/mio5_utils.c:5889) File "mio5_utils.pyx", line 657, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy/io/matlab/mio5_utils.c:5446) File "mio5_utils.pyx", line 814, in scipy.io.matlab.mio5_utils.VarReader5.read_char (scipy/io/matlab/mio5_utils.c:7270) TypeError: buffer is too small for requested array Chuck

1 0

tol parameter to minimize_scalar is inconsistent [without HTML this time]
by Micky Latowicki 13 Sep '13

13 Sep '13

[sorry for the HTML in the first try] Hi, I'm a longtime user of scipy and greatly appreciate your work. I'd like to ask/suggest something about minimize_scalar. It seems to me that the tol parameter is relative tolerance when the method is 'brent' and absolute tolerance when the method is 'bounded'. Isn't this confusing? Perhaps this parameter should be deprecated, until both methods can support either relative or absolute tolerance? Meanwhile, parameters with names such as 'abs_xtol' and 'rel_xtol' can be introduced, and invalid combinations of parameters and method would be rejected with a ValueError? Cheers. This mail was sent via Mail-SeCure system. ************************************************************************************ This footnote confirms that this email message has been scanned by PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. ************************************************************************************

4 3

recent scipy.stats work
by josef.pktd＠gmail.com 12 Sep '13

12 Sep '13

Evgeni (mainly) has been busy to improve stats.distributions stats.distributions is on the way to losing it's remaining bugs in higher moments and entropy. :( and is getting a reliable test suite for them. Here is a, maybe incomplete, list to see if we want to backport any of them to 0.13 https://github.com/scipy/scipy/pull/2801 fix beta_gen.fit (Warren) https://github.com/scipy/scipy/pull/2845 open, numerical improvemnt (but also vonmises_line) https://github.com/scipy/scipy/pull/2774 open, bugfix for entropy docs https://github.com/scipy/scipy/pull/2814 not bugfix https://github.com/scipy/scipy/pull/2822 speed improvement https://github.com/scipy/scipy/pull/2841 numerical improvements not ready https://github.com/scipy/scipy/pull/2821 fixing moments https://github.com/scipy/scipy/pull/2790 Josef

2 1

tol parameter to minimize_scalar is inconsistent
by Micky Latowicki 11 Sep '13

11 Sep '13

1 0

Akima Interpolation
by Andreas Hilboll 10 Sep '13

10 Sep '13

I'd like to have Akima interpolation in scipy.interpolate, as I couldn't find any. 1.) Did I miss something and there actually *is* an Akima interpolation in scipy? 2.) Is there interest in having Akima interpolation in Scipy? 3.) Does anyone have a good recommendation which implementation I should consider for pulling into Scipy? -- Andreas.

7 8

Consensus on the future of integrate.ode
by boyfarrell＠gmail.com 10 Sep '13

10 Sep '13

Dear list, I think it would be interesting to start a discussion regarding the direction of integrate.ode. Reading the list, there seems to be many people interested seeing improvements. Is there a roadmap for development? If not can we come to a consensus about how best to contribute to this part of scipy? I think it is fair to say that other projects have overtaken scipy in terms integrating new solvers and general development in this area. Which is a shame. Personally, speaking as a user, I would like to see the use of more modern solvers. For example the code could be updated to use CVODE. This has many improvements/fixes over the Fortran VODE solver that is currently used. This would be a nice place to start, clearly that is still a nontrivial task. Also, is there a way that developers could be encourage to contribute additional solvers? For example by making an elegant low level API that makes this easier. Moreover, say that in order for a developer to connect a new solver to scipy all that had to be done was implement a particular, well defined, interface for their wrapper. Once implemented the new solver could be called with the exciting integrate.ode interface. Has anybody got experience is such design? Surely something like this has been attempted? The GNU Scientific Library interface springs to mind as a good example, http://www.gnu.org/software/gsl/manual/html_node/Ordinary-Differential-Equa… Finally, I have read some discussion here regarding changing the API and or unifying the odeint and ode. I don't want to get too distracted with this as my main point is outlined above. Personally the current API doesn't bother me too much, I find both easy to use. But if this is being considered for an update I think the crucial thing is to have the interface have no reliance on any particular solver. Abstraction is key. Happy to know your thoughts and if we can make a push together for some improvements in this area then all the better. Best regard, Dan

7 16

Fwd: [Numpy-discussion] ANN: 1.8.0b2 release.
by Daπid 08 Sep '13

08 Sep '13

On 8 September 2013 21:14, Charles R Harris <charlesr.harris(a)gmail.com>wrote: > It would be good if folks running OS X could try out this release and > report any issues on the numpy-dev mailing list. Unfortunately the files > still need to be installed from source as dmg files are not avalable at > this time. > Hi, I have tried on a intel Mac with Python2.7 and gcc installed with Macports and it builds fine. The FORTRAN compiler is gfortran. There are a bunch of compiling warnings and errors* (attachment log27.log), but it works fine. Running the tests it gets printed: .......Parameter 4 to routine DGETRF was incorrect....... but all of them pass. I did not specify which BLAS to use, and it choose Accelerate, but ATLAS is also installed. >>> numpy.show_config() atlas_threads_info: NOT AVAILABLE blas_opt_info: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'] define_macros = [('NO_ATLAS_INFO', 3)] atlas_blas_threads_info: NOT AVAILABLE openblas_info: NOT AVAILABLE lapack_opt_info: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] extra_compile_args = ['-msse3'] define_macros = [('NO_ATLAS_INFO', 3)] Running SciPy (from Macports) test does raise two errors and a bunch of Deprecation warnings. See attachment scipy.log. The first two warnings were present before upgrading NumPy, but not the rest. Let me know if you need me to run it again, or change the configuration in any way. Regards, David. ------- * Errors such as: _configtest.c:1:20: error: endian.h: No such file or directory _configtest.c:5: error: size of array ‘test_array’ is negative _configtest.c:7: error: ‘Py_UNICODE_WIDE’ undeclared (first use in this function)

2 2

Consensus on the future of integrate.ode
by Mario Mulansky 06 Sep '13

06 Sep '13

> Please, please don't implement anything in the SciPy ODE solvers with Boost. > odeint unless it's a separate interface that requires extreme generality. > > Boost.odeint uses Boost.uBLAS for its linear algebra (see > http://headmyshoulder.github.io/odeint-v2/doc/boost_numeric_odeint/concepts/ > implicit_system.html), and Boost.uBLAS is slow because it tries to be > general (everything is templated), and it has debugging features added by > default. > > A lot of users (myself included) don't need that generality, and would > prefer the extra performance that you'd get from a code that can use a > faster dense linear algebra library (like an optimized LAPACK, Eigen, > Elemental, maybe MTL4). I agree with the concerns but I would like to note that the current implementation of implicit routines in Boost.odeint is preliminary. We are currently working on a generalization of these routines to allow the usage of arbitrary backends that should be done by the end of the year. However, Boost.odeint misses some more of the functionality of CVODE, e.g. automatic detection of stiffness or numerical Jacobi approximation. But I think that using Boost.odeint could be benefitial for scipy.odeint (e.g. parallelization) and we would be happy to assist scipy developers, e.g. by providing missing functionality, if you want to use Boost.odeint. Best regards, Mario

1 0

contributing to scipy.spatial.distance
by Michael Yang 04 Sep '13

04 Sep '13

hi - I'm new to this development list so please excuse me if I'm not doing things right (like top-posting or bottom-posting, etc.) I would like to contribute an additional vector distance function to the absolutely wonderful scipy.spatial.distance package written by Damian Eads. I have a patch for scipy/spatial/distance.py written for scipy version 0.7.1 a while back. Could someone please advise me on how to incorporate the patch? I don't know if the repo is in github or somewhere else. Also (not to be self-aggrandizing!) but would it be possible to add my name somewhere to this file if the patch gets makes it in successfully? Thanks, Michael Yang

3 2

matrix_rank scipy version ?
by josef.pktd＠gmail.com 03 Sep '13

03 Sep '13

random question, while catching up with some linalg (matrix decomposition and least squares with reduced rank) scipy has rank-revealing (pivoting) QR for a few versions now. Would it be worth it to add a scipy version of matrix_rank that uses QR, instead of SVD as in numpy (IIRC) What's the cutoff for a QR version? Josef

1 0

fortran docstrings
by josef.pktd＠gmail.com 03 Sep '13

03 Sep '13

playing with parsing fortran documentation in special cdflib https://gist.github.com/josef-pkt/6426618 some fortran files don't work example ################# cumpoi(s, xlam, cum, ccum) CUMulative POIsson distribution Returns the probability of S or fewer events in a Poisson distribution with mean XLAM. Parameters ---------- s : array_like Upper limit of cumulation of the Poisson. xlam : array_like Mean of the Poisson distribution. Returns ------- cum : array_like Cumulative poisson distribution. ccum : array_like Compliment of Cumulative poisson distribution. Notes ----- Uses formula 26.4.21 of Abramowitz and Stegun, Handbook of Mathematical Functions to reduce the cumulative Poisson to the cumulative chi-square distribution. ################# cumt(t, df, cum, ccum) CUMulative T-distribution Computes the integral from -infinity to T of the t-density. Parameters ---------- t : array_like Upper limit of integration of the t-density. df : array_like Degrees of freedom of the t-distribution. Returns ------- cum : array_like Cumulative t-distribution. ccum : array_like Compliment of Cumulative t-distribution. Notes ----- Formula 26.5.27 of Abramowitz and Stegun, Handbook of Mathematical Functions is used to reduce the t-distribution to an incomplete beta. ################# cumtnc(t, df, pnonc, cum, ccum) CUMulative Non-Central T-distribution Computes the integral from -infinity to T of the non-central t-density. Parameters ---------- t : array_like Upper limit of integration of the non-central t-density. df : array_like Degrees of freedom of the non-central t-distribution. pnonc : array_like Non-centrality parameter of the non-central t distibutio Returns ------- cum : array_like Cumulative t-distribution. ccum : array_like Compliment of Cumulative t-distribution. Notes ----- Upper tail of the cumulative noncentral t using formulae from page 532 of Johnson, Kotz, Balakrishnan, Coninuous Univariate Distributions, Vol 2, 2nd Edition. Wiley (1995) This implementation starts the calculation at i = lambda, which is near the largest Di. It then sums forward and backward.

1 0

Re: [SciPy-Dev] [Numpy-discussion] ANN: Scipy 0.13.0 beta 1 release
by David Cournapeau 03 Sep '13

03 Sep '13

On Mon, Sep 2, 2013 at 5:46 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi, > > 02.09.2013 18:30, David Cournapeau kirjoitti: > [clip] > > Ok, I think the commit 5935030f8cced33e433804a21bdb15572d1d38e8 is > > quite wrong. > > > > It conflates the issue of dealing with Accelerate brokenness and > > using g77 ABI. I would suggest reverting it for 0.13.0 (and > > re-disable single precision), as fixing this correctly may require > > quite some time/testing. > > I'm -1 on returning to the previous situation where many routines on > OSX are simply broken (which was the situation previously --- > "disabling single precision" left several things still broken). I'd > rather just postpone the 0.13.0 release until this issue is solved > properly. > I see, I missed that it was more than just reverting to slower versions. > > Can you say what exactly is wrong -- as far as I know, Accelerate on > OSX simply uses g77 ABI. There were some bugs previously where it in > places did things differently, but on recent OSX releases this is no > longer the case? > > I guess you are trying to link with MKL? In that case, I would rather > propose restoring the previous MKL wrappers (with trivial extensions > for the missing w* symbols). > The pb is specific to MKL, yes, but I've found a workaround: in the case of MKL, you just need to workaround the g77 ABI of the MKL vs gfortran ABI, but the LAPACK interface is the actual LAPACK, not CLAPACK. So for MKL, you need non dummy wrappers iff the function returns a complex. A dirty patch seems to confirm that this fixes the issue, I will prepare an actual patch tomorrow, David > > - -- > Pauli Virtanen > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > > iEYEARECAAYFAlIkwPQACgkQ6BQxb7O0pWBnkwCfeKj32CYYCPdEWVcYYMtq/OZM > 32wAoIn9yG/HNtUeh+XwqAm2uAS9sVQ5 > =CKBN > -----END PGP SIGNATURE----- > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

2 2

ANN: Scipy 0.13.0 beta 1 release
by Ralf Gommers 02 Sep '13

02 Sep '13

Hi all, I'm happy to announce the availability of the first beta release of Scipy 0.13.0. Please try this beta and report any issues on the scipy-dev mailing list. Source tarballs and release notes can be found at https://sourceforge.net/projects/scipy/files/scipy/0.13.0b1/. Windows and OS X installers will follow later (we have a minor infrastructure issue to solve, and I'm at EuroScipy now). Cheers, Ralf

6 14

ANN: Numpy 1.8.0 beta 1 release
by Charles R Harris 01 Sep '13

01 Sep '13

Hi all, I'm happy to announce the first beta release of Numpy 1.8.0. Please try this beta and report any issues on the numpy-dev mailing list. Source tarballs and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/. The Windows and OS X installers will follow when the infrastructure issues are dealt with. Chuck

1 0