Mailman 3 May 2013 - SciPy-Dev

Bootstrap confidence limits code
by Constantine Evans 05 Oct '14

05 Oct '14

Hello everyone, A few years ago I implemented a scikit for bootstrap confidence limits (https://github.com/cgevans/scikits-bootstrap). I didn’t think much about it after that until recently, when I realized that some people are actually using it, and that there’s apparently been some talk about implementing this functionality in either scipy.stats or statsmodels (I should thank Randal Olson for discussing this and bringing it to my attention). As such I’ve rewritten most of the code, and written up some docstrings. The current code can do confidence intervals with basic percentile interval, bias-corrected accelerated, and approximate bootstrap confidence methods, and can also provide bootstrap and jackknife indexes. Most of it is implemented from the descriptions in Efron and Tibshirani’s Introduction to the Bootstrap, but the ABC code at the moment is a port from the modified-BSD-licensed bootstrap package for R (not the boot package) as I’m not entirely confident in my understanding of the method. And so, I have a few questions for everyone: * Is there any interest in including this sort of code in either scipy.stats or statsmodels? If so, where do people think would be the better place? The code is relatively small; at the moment it is less than 200 lines, with docstrings probably making up 100 of those lines. * Also, if so, what would need to be changed, added, and improved beyond what is mentioned in the Contributing to Scipy part of the reference guide? I’m never a fan of my own code, and imagine quite a bit would need to be fixed; I know tests will need to be added too. In addition, I have a few questions about what would be better practice for the API, and I haven’t really found a guide on best practices for Scipy: * When I started writing the code, I wrote a single function ci for confidence intervals, with a method argument to choose the method. This is easy for users, especially so that they don’t have to look through documentation to realize that BCA is the most generally useful method (at least from everything I’ve read) and that there really isn’t any reason to use many of the simpler methods. However, ABC takes different paramenters, and needs a statistic function that takes weights, which makes this single-function organization trickier. At the moment, I have a separate function for ABC. Would it be better to split up all the methods to their own functions? * ABC requires a statistic function that takes weights. I’ve noticed that things like np.average takes a weights= argument. Would it be better to require input of a stat(data,weights) function, or input of a stat(data,weights=) with weights as a named argument? The latter would be nice in terms of allowing the same function to be used for all methods, but would make it impossible to use a lambda for the function. Is there some other method of doing this entirely? * Are there any missing features that anyone thinks should be added? I apologize if much of this is answered elsewhere, I just haven’t found any of it; I also apologize if this is far too long-winded and confusing! Regards, Constantine Evans

4 3

SLSQP Constrained Optimizer Status
by Rob Falck 25 Nov '13

25 Nov '13

I'm currently implementing the Sequential Least Squares Quadratic Programming (SLSQP) optimizer, by Dieter Kraft, for use in Scipy. The Fortran code being wrapped with F2PY is here: http://www.netlib.org/toms/733 (its use within Scipy has been cleared) SLSQP provides for bounds on the independent variables, as well as equality and inequality constraint functions, which is a capability that doesn't exist in scipy.optimize. Currently the code works, although the constraint normals are being generated by approximation. I'm working on a way to pass these in. I think the most elegant way will be a single function that returns the matrix of constraint normals. For a demonstration of what the code can do, here is an optimization of f(x,y) = 2xy + 2x - x**2 - 2y**2 Example 14.2 in Chapra & Canale gives the maximum as x=2.0, y=1.0. The unbounded optimization tests find this solution. As expected, its faster when derivatives are provided rather than approximated. Unbounded optimization. Derivatives approximated. Elapsed time: 1.45792961121 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] Unbounded optimization. Derivatives provided. Elapsed time: 1.03211402893 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] The following example uses an equality constraint to find the optimal when x=y. Bound optimization. Derivatives approximated. Elapsed time: 1.384973526 ms Results [[0.99999996845920858, 0.99999996845920858], -0.99999999999999889, 4, 0, 'Optimization terminated successfully.'] I've tried to conform to the syntax used by the other optimizers in scipy.optimize. The function definition and doc string are below. If anyone is interested in testing it out, let me know. def fmin_slsqp( func, x0 , eqcons=[], f_eqcons=None, ieqcons=[], f_ieqcons=None, bounds = [], fprime = None, fprime_cons=None,args = (), iter = 100, acc = 1.0E-6, iprint = 1, full_output = 0, epsilon = _epsilon ): """ Minimize a function using Sequential Least SQuares Programming Description: Python interface function for the SLSQP Optimization subroutine originally implemented by Dieter Kraft. Inputs: func - Objective function (in the form func(x, *args)) x0 - Initial guess for the independent variable(s). eqcons - A list of functions of length n such that eqcons[j](x0,*args) == 0.0 in a successfully optimized problem f_eqcons - A function of the form f_eqcons(x, *args) that returns an array in which each element must equal 0.0 in a successfully optimized problem. If f_eqcons is specified, eqcons is ignored. ieqcons - A list of functions of length n such that ieqcons[j](x0,*args) >= 0.0 in a successfully optimized problem f_ieqcons - A function of the form f_ieqcons(x0, *args) that returns an array in which each element must be greater or equal to 0.0 in a successfully optimized problem. If f_ieqcons is specified, ieqcons is ignored. bounds - A list of tuples specifying the lower and upper bound for each independent variable [ (xl0, xu0), (xl1, xu1), ...] fprime - A function that evaluates the partial derivatives of func fprime_cons - A function of the form f(x, *args) that returns the m by n array of constraint normals. If not provided, the normals will be approximated. Equality constraint normals precede inequality constraint normals. args - A sequence of additional arguments passed to func and fprime iter - The maximum number of iterations (int) acc - Requested accuracy (float) iprint - The verbosity of fmin_slsqp. iprint <= 0 : Silent operation iprint == 1 : Print summary upon completion (default) iprint >= 2 : Print status of each iterate and summary full_output - If 0, return only the minimizer of func (default). Otherwise, output final objective function and summary information. epsilon - The step size for finite-difference derivative estimates. Outputs: ( x, { fx, gnorm, its, imode, smode }) x - The final minimizer of func. fx - The final value of the objective function. its - The number of iterations. imode - The exit mode from the optimizer, as an integer. smode - A string describing the exit mode from the optimizer. Exit modes are defined as follows: -1 : Gradient evaluation required (g & a) 0 : Optimization terminated successfully. 1 : Function evaluation required (f & c) 2 : Number of equality constraints larger than number of independent variables 3 : More than 3*n iterations in LSQ subproblem 4 : Inequality constraints incompatible 5 : Singular matrix E in LSQ subproblem 6 : Singular matrix C in LSQ subproblem 7 : Rank-deficient equality constraint subproblem HFTI 8 : Positive directional derivative for linesearch 9 : Iteration limit exceeded -- - Rob Falck

4 16

Vector Strength function
by Todd 22 Nov '13

22 Nov '13

I am interested in implementing a function for scipy. The function is called "vector strength". It is basically a measure of how reliably a set of events occur at a particular phase. It was originally developed for neuroscience research, to determine how well a set of neural events sync up with a periodic stimulus like a sound waveform. However, it is useful for determining how periodic a supposedly periodic set of events really are, for example: 1. Determining whether crime is really more common during a full moon and by how much 2. Determining how concentrated visitors to a coffee shop are during rush hour 3. Determining exactly how concentrated hurricanes are during hurricane season My thinking is that this could be implemented in stages: First would be a Numpy function that would add a set of vectors in polar coordinates. Given a number of magnitude/angle pairs it would provide a summed magnitude/angle pair. This would probably be combined with a cartesian<->polar conversion functions. Making use of this function would be a scipy function that would actually implement the vector strength calculation. This is done by treating each event as a unit vector with a phase, then taking the average of the vectors. If all events have the same phase, the result will have an amplitude of 1. If they all have a different phases, the result will have an amplitude of 0. It may even be worth having a dedicated polar dtype, although that may be too much. What does everyone think of this proposal?

4 12

new tests in sparse
by Blake Griffith 04 Jun '13

04 Jun '13

Hello scipy, I'm in the process of adding new tests for sparse matrices with dtype=bool, while I'm at it, I thought it would be a good idea to add tests for other dtypes. Currently all the tests are for int64 and float64. There is some canonical data defined in test_base.py. I use this to make data with all the supported dtypes. Then I change the tests that use this canonical data to iterate over this data with different dtypes (including bool). Does this sound ok? It will make running test_base.py a bit slower. You can see my work here: https://github.com/cowlicks/scipy/commits/official-bool-support

3 7

sparse bool dtype support
by Blake Griffith 31 May '13

31 May '13

Hello, I'm working on adding support for the bool dtype for sparse matrices. But I'm having some problems I'm adding a wrapper for the npy_bool type. I've included a file called bool_ops.h which basically contains: typedef npy_int8 npy_bool_wrapper; ( https://github.com/cowlicks/scipy/blob/swig-sandbox/scipy/sparse/sparsetool… ) Then I included bool_ops.h and the wrapper npy_bool_wrapper in sparsetools.i and numpy.i where I thought it was necessary. So that the typemaps for the npy_bool type are generated. You can see this in my branch: in sparsetools.i https://github.com/cowlicks/scipy/blob/swig-sandbox/scipy/sparse/sparsetool… https://github.com/cowlicks/scipy/blob/swig-sandbox/scipy/sparse/sparsetool… https://github.com/cowlicks/scipy/blob/swig-sandbox/scipy/sparse/sparsetool… in numpy.i https://github.com/cowlicks/scipy/blob/swig-sandbox/scipy/sparse/sparsetool… https://github.com/cowlicks/scipy/blob/swig-sandbox/scipy/sparse/sparsetool… However I still get the same error seen before in ticket #1533 ( https://github.com/scipy/scipy/issues/2058) TypeError: Array of type 'byte' required. Array of type 'bool' given Which is not what I expected because the typemaps for bool type should now be generated. Could the order in INSTANTIATE_ALL be the problem? ( https://github.com/cowlicks/scipy/blob/swig-sandbox/scipy/sparse/sparsetool… ) Why is order important here? I included npy_bool_wrapper where I did because the Numpy C api says npy_bool is an unsigned char. I'm currently trying to reorder INSTANTIATE_ALL to make things work. Thanks in advance,

2 4

Re: [SciPy-Dev] New scipy.org site
by Matti Pastell 31 May '13

31 May '13

Hello all, I did some work in converting the Cookbook to Sphinx: I scraped the Cookbook from the archive Robert sent to the list earlier and used a couple of scripts to make the conversion, it's not perfect but not that bad either. You can see the results at: http://mpastell.github.io/ I put the various stages of conversion to GitHub with a brief explanation: https://github.com/mpastell/SciPy-CookBook Conversion scripts currently require a forked version of Pweave http://mpastell.com/pweave (I'm the author) that only I have, but of course I can share it. I'd like to get some feedback before I do any more work and I think a lot of the pages will require a bit of manual editing in the end. Also what do you think about using Pweave for the Cookbook? It can be used to capture figures and code and can be intgrated to sphinx build process. An example (http://mpastell.com/pweave/examples.html). Matti

2 2

Error when compiling numpy
by Wim R. Cardoen 30 May '13

30 May '13

Hello, I tried to compile the latest stable version of numpy version 1.7.1 on RHEL5 using gfortran version 4.1.2 Very soon I got the following error (see below) Suggestions to resolve this issue are fully appreciated. Thanks, Wim Error: running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build/src.linux-x86_64-2.7 creating build/src.linux-x86_64-2.7/numpy creating build/src.linux-x86_64-2.7/numpy/distutils building library "npymath" sources customize Gnu95FCompiler Traceback (most recent call last): File "setup.py", line 214, in <module> setup_package() File "setup.py", line 207, in setup_package configuration=configuration ) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/uufs/ chpc.utah.edu/sys/pkg/python/2.7.3/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/uufs/ chpc.utah.edu/sys/pkg/python/2.7.3/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/uufs/ chpc.utah.edu/sys/pkg/python/2.7.3/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/command/build…", line 37, in run old_build.run(self) File "/uufs/ chpc.utah.edu/sys/pkg/python/2.7.3/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "/uufs/ chpc.utah.edu/sys/pkg/python/2.7.3/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/uufs/ chpc.utah.edu/sys/pkg/python/2.7.3/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/command/build…", line 152, in run self.build_sources() File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/command/build…", line 163, in build_sources self.build_library_sources(*libname_info) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/command/build…", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/command/build…", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 646, in get_mathlib_info st = config_cmd.try_link('int main(void) { return 0;}') File "/uufs/ chpc.utah.edu/sys/pkg/python/2.7.3/lib/python2.7/distutils/command/config.py", line 248, in try_link self._check_compiler() File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/command/confi…", line 77, in _check_compiler self.fcompiler.customize(self.distribution) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/fcompiler/__i…", line 500, in customize get_flags('opt', oflags) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/fcompiler/__i…", line 491, in get_flags flags.extend(getattr(self.flag_vars, tag)) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/environment.py", line 37, in __getattr__ return self._get_var(name, conf_desc) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/environment.py", line 51, in _get_var var = self._hook_handler(name, hook) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/fcompiler/__i…", line 698, in _environment_hook return hook() File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/fcompiler/gnu…", line 348, in get_flags_opt return GnuFCompiler.get_flags_opt(self) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/fcompiler/gnu…", line 195, in get_flags_opt v = self.get_version() File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/fcompiler/__i…", line 430, in get_version version = CCompiler.get_version(self, force=force, ok_status=ok_status) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/ccompiler.py", line 458, in CCompiler_get_version status, output = exec_command(version_cmd,use_tee=0) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/exec_command.…", line 219, in exec_command **env) File "/uufs/ chpc.utah.edu/sys/src/pylib/2.7.3/numpy/1.7.1/numpy/distutils/exec_command.…", line 269, in _exec_command_posix status = int(status_text) ValueError: invalid literal for int() with base 10: ''

2 1

scipy.org seems to be down
by Anthony Scopatz 29 May '13

29 May '13

Be Well Anthony

2 2

Strongly Connected Components
by Tim Leslie 26 May '13

26 May '13

Hi All, In the latest version of scipy I updated the algorithm used to find strongly connected components of a directed graph. The code lives in scipy.sparse.csgraph.connected_components. I've written up a blog post which explains how this algorithm works and also the memory optimisations which were made to greatly improve the capacity of the scipy implementation. http://www.timl.id.au/?p=327 If noone objects I'll update the documentation in the code to point to this page, as I think it provides a useful explanation of the algorithm which is not available elsewhere. Cheers, Tim

1 0

Reusing docstrings of the parent class in overridden methods of a subclass
by Warren Weckesser 25 May '13

25 May '13

In scipy.stats.distributions, some of the distribution classes (subclasses of rv_continuous) override, say, the `fit` method. See, for example, gamma_gen or beta_gen. Is there a standard way for the `fit` method of the subclass to use the docstring of the `fit` method in the parent class? In python 2.7, this monkey patching seems to work: beta_gen.fit.__func__.__doc__ = rv_continuous.fit.__doc__ (This is what I did in https://github.com/scipy/scipy/pull/2519) In python 3.3, however, beta_gen.fit does not have the __func__ attribute. Is there a recommended way to do this? The docstring is rather long, and I'd rather not repeat it in the subclass. Warren

3 5