Mailman 3 August 2013 - SciPy-User

scipy sparse efficiency questions
by John Floyd Aug. 13, 2013

Aug. 13, 2013

Scipy sparse direct solvers use superlu as the backend by default. Superlu supports using previous reordering data which is particularly useful when time stepping through solutions for systems where the matrix structure does not change with time. Looking through the scipy code indicates that the internal variables do keep the c_perm and r_perm variables. I cannot see a way to use this feature with this wrapper. Have I missed something? Can I rerun the solver without re-calculating the reordering? This also applies if I use the spilu to use as a preconditioner - it appears to recalc the reordering for call. This is numerically expensive. I realise that petsc and trilinos offer a more complete solution but for the time being just want a simplified approach that I can move easily from linux/unix to windows. The latter causes plenty of problems when I have to compile my own modules - I develop on linux. Thanks John John Floyd Water, Wetlands and Coasts Science Office of Environment and Heritage Department of Premier and Cabinet john.floyd(a)environment.nsw.gov.au <mailto:e@environment.nsw.gov.au> Ph: 02-9995-5656 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- This email is intended for the addressee(s) named and may contain confidential and/or privileged information. If you are not the intended recipient, please notify the sender and then delete it immediately. Any views expressed in this email are those of the individual sender except where the sender expressly and with authority states them to be the views of the Office of Environment and Heritage, NSW Department of Premier and Cabinet. PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL

2 1

problem with the leastsq function
by Frédéric Parrenin Aug. 13, 2013

Aug. 13, 2013

Dear all, I am currently building a code which uses the leastsq function for a minimization problem. I apply leastsq to a function named residuals which takes as entry a vector named variables. For some configurations of the problem, I get the following error message: error: Error occurred while calling the Python function named residuals This is not very meaningful. Is there a way to debug this leastsq function and to really now where the problem is? Best regards, Frédéric

2 1

vti file animation
by Matthew Dornfeld Aug. 12, 2013

Aug. 12, 2013

Hi, I have a bunch of vti files. Each one corresponds to the output of a simulation for one time step. I want to combine these all together and make an animation. I'm trying to do this with mayavi, but I'm having trouble finding documentation on how. Can anyone help? Matt

1 0

Built-in FAP for Lomb-Scargle?
by Nicholas Earl Aug. 11, 2013

Aug. 11, 2013

Hello everyone, I was just curious if there are any built-in functions to provide the false alarm probability of the Lomb-Scargle routine in SciPy? Thanks for your time! Nicholas Earl Graduate Research Assistant San Diego State University

2 1

scipy.io.loadmat throws TypeError with large files
by Richard Llewellyn Aug. 11, 2013

Aug. 11, 2013

Hi, I get this or similar (different integer than 75724 in error) exceptions when loading a sparse matrix (CSC) saved with savemat, all default options. >>> m = loadmat('my_large_mat.mat') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio.py", line 176, in loadmat matfile_dict = MR.get_variables(variable_names) File "/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio5.py", line 274, in get_variables hdr, next_position = self.read_var_header() File "/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio5.py", line 236, in read_var_header raise TypeError('Expecting miMATRIX type here, got %d' % mdtype) TypeError: Expecting miMATRIX type here, got 75724 here the matrix was: > matrix <400000x4176 sparse matrix of type '<class 'numpy.uint8'>' with 934099575 stored elements in Compressed Sparse Column format> and looks fine before saving. It looks as if this only occurs when the saved matrix file size is > 4GB -- at least I haven't seen it with files in the 3GB range. 64 bit Linux. Not a crisis, as I am chunking anyway, so I can just chunk smaller, but when I get more RAM would be nice to bump it up to 8 GB files or so. Thanks.

2 5

[ANN] patsy version 0.2.0 released
by Nathaniel Smith Aug. 10, 2013

Aug. 10, 2013

Hi all, I've just released version 0.2.0 of patsy, a Python library for describing statistical models (especially linear models, or models that have a linear component) and building design matrices. Patsy brings the convenience of R "formulas" to Python. -- New in this release (for more details: https://patsy.readthedocs.org/en/latest/changes.html#v0-2-0) New support for automatically detecting and (optionally) removing missing values. New stateful transform for B-spline regression: bs(). (Requires scipy.) Added a core API to make it possible to run predictions on only a subset of model terms. (This is particularly useful for e.g. plotting the isolated effect of a single fitted spline term.) See DesignMatrixBuilder.subset(). LookupFactor now allows users to mark variables as categorical directly. pandas.Categorical objects are now recognized as representing categorical data and handled appropriately. Better error reporting for exceptions raised by user code inside formulas. We now, whenever possible, tag the generated exception with information about which factor’s code raised it, and use this information to give better error reporting. EvalEnvironment.capture() now takes a reference argument, to make it easier to implement new dmatrix()-like functions. Miscellaneous doc improvements and bug fixes. -- General information Documentation: http://patsy.readthedocs.org/ Downloads: http://pypi.python.org/pypi/patsy/ Dependencies: numpy Python 2.5 or later (Python 3 works) Optional dependencies: nose: needed to run tests scipy: needed for spline-related functions like 'bs' Install: pip install patsy or python setup.py install Code and bug tracker: https://github.com/pydata/patsy Mailing list: pydata(a)googlegroups.com http://groups.google.com/group/pydata http://news.gmane.org/gmane.comp.python.pydata License: 2-clause BSD, see LICENSE.txt for details. Share and enjoy, -n

1 0

Optimize.Leastsq Issues
by Suraj Mirpuri Aug. 9, 2013

Aug. 9, 2013

Hello All, I think I've been having some difficulty with optimize.leastsq the past few hours because for the life of me I cannot seem to get it to work. Specifically, my issue is that when I use it without a Jacobian function it estimates a Jacobian of all 0's, and when I give it a real Jacobian and use full_output = 1 it also returns all 0's. However this cannot be the case as at least at the initial point that I feed in, the Jacobian is definitely non zero. It just ends up giving me the initial parameters each time. I am not quite sure I have it set up right so I have attached a paste bin link to my code if anyone could take a look at it, and the output of leastsq is at the end of it. Also it's been sort of cobbled together late at night so I recognize it's not exactly efficient code, I just need to get it working I guess. Also I tried both col_deriv=1 and 0 with appropriate return value in grad_func as I found a thread that suggests this makes a difference. I couldn't see that it did. Code: http://pastebin.com/qa3gw1AD Thanks in advance! -- Raj Mirpuri Research Assistant Roberts Group Caltech

2 2

[ANN] IPython 1.0 is finally released, nearly 12 years in the making!
by Fernando Perez Aug. 9, 2013

Aug. 9, 2013

Hi all, I am incredibly thrilled, on behalf of the amazing IPython Dev Team, to announce the official release of IPython 1.0 today, an effort nearly 12 years in the making. The previous version (0.13) was released on June 30, 2012, and in this development cycle we had: ~12 months of work. ~700 pull requests merged. ~600 issues closed (non-pull requests). contributions from ~150 authors. ~4000 commits. # A little context What does "1.0" mean for IPython? Obviously IPython has been a staple of the scientific Python community for years, and we've made every effort to make it a robust and production ready tool for a long time, so what exactly do we mean by tagging this particular release as 1.0? Basically, we feel that the core design of IPython, and the scope of the project, is where we want it to be. What we have today is what we consider a reasonably complete, design- and scope-wise, IPython 1.0: an architecture for interactive computing, that can drive kernels in a number of ways using a well-defined protocol, and rich and powerful clients that let users control those kernels effectively. Our different clients serve different needs, with the old workhorse of the terminal still being very useful, but much of our current development energy going into the Notebook, obviously. The Notebook enables interactive exploration to become Literate Computing, bridging the gaps from individual work to collaboration and publication, all with an open file format that is a direct record of the underlying communication protocol. There are obviously plenty of open issues (many of them very important) that need fixing, and large and ambitious new lines of development for the years to come. But the work of the last four years, since the summer of 2009 when Brian Granger was able to devote a summer (thanks to funding from the NiPy project - nipy.org) to refactoring the old IPython core code, finally opened up or infrastructure for real innovation. By disentangling what was a useful but impenetrable codebase, it became possible for us to start building a flexible, modern system for interactive computing that abstracted the old REPL model into a generic protocol that kernels could use to talk to clients. This led at first to the creation of the Qt console, and then to the Notebook and out-of-process terminal client. It also allowed us to (finally!) unify our parallel computing machinery with the rest of the interactive system, which Min Ragan-Kelley pulled off in a development tour de force that involved rewriting in a few weeks a huge and complex Twisted-based system. We are very happy with how the Notebook work has turned out, and it seems the entire community agrees with us, as the uptake has been phenomenal. Back from the very first "IPython 0.0.1" that I started in 2001: https://gist.github.com/fperez/1579699 there were already hints of tools like Mathematica: it was my everyday workhorse as a theoretical physicist and I found its Notebook environment invaluable. But as a grad student trying out "just an afternoon hack" (IPython was my very first Python program as I was learning the language), I didn't have the resources, skills or vision to attempt building an entire notebook system, and to be honest the tools of the day would have made that enterprise a miserable one. But those ideas were always driving our efforts, and as IPython started becoming a project with a team, we made multiple attempts to get a good Notebook built around IPython. Those interested can read an old blog post of mine with the history (http://blog.fperez.org/2012/01/ipython-notebook-historical.html). The short story is that in 2011, on our sixth attempt, Brian was again able to devote a focused summer into using our client-server architecture and, with the stack of the modern web (Javascript, CSS, websockets, Tornado, ...), finally build a robust system for Literate Computing across programming languages. Today, thanks to the generous support and vision of Josh Greenberg at the Alfred P. Sloan Foundation, we are working very hard on building the notebook infrastructure, and this release contains major advances on that front. We have high hopes for what we'll do next; as a glimpse of the future that this enables, now there is a native Julia kernel that speaks to our clients, notebook included: https://github.com/JuliaLang/IJulia.jl. # Team I can't stress enough how impressed I am with the work people are doing in IPython, and what a privilege it is to work with colleagues like these. Brian Granger and Min Ragan-Kelley joined IPython around 2005, initially working on the parallel machinery, but since ~ 2009 they have become the heart of the project. Today Min is our top committer and knows our codebase better than anyone else, and I can't imagine better partners for an effort like this. And from regulars in our core team like Thomas Kluyver, Matthias Bussonnier, Brad Froehle and Paul Ivanov to newcomers like Jonathan Frederic and Zach Sailer, in addition to the many more whose names are in our logs, we have a crazy amount of energy being poured into IPython. I hope we'll continue to harness it productively! The full list of contributors to this release can be seen here: http://ipython.org/ipython-doc/rel-1.0.0/whatsnew/github-stats-1.0.html # Release highlights * nbconvert: this is the major piece of new functionality in this cycle, and was an explicit part of our roadmap (https://github.com/ipython/ipython/wiki/Roadmap:-IPython). nbconvert is now an IPython subcommand to convert notebooks into other formats such as HTML or LaTeX, but more importantly, it's a very flexible system that lets you write custom templates to generate new output with arbitrary control over the formatting and transformations that are applied to the input. We want to stress that despite the fact that a huge amount of work went into nbconvert, this should be considered a *tech preview* release. We've come to realize how complex this problem is, and while we'll make every effort to keep the high-level command-line syntax and APIs as stable as possible, it is quite likely that the internals will continue to evolve, possibly in backwards-incompatible ways. So if you start building services and libraries that make heavy use of the nbconvert internals, please be prepared for some turmoil in the months to come, and ping us on the dev list with questions or concerns. * Notebook improvements: there has been a ton of polish work in the notebook at many levels, though the file format remains unchanged from 0.13, so you shouldn't have any problems sharing notebooks with colleagues still using 0.13. - Autosave: probably the most oft-requested feature, the notebook server now autosaves your files! You can still hit Ctrl-S to force a manual save (which also creates a special 'checkpoint' you can come back to). - The notebook supports raw_input(), and thus also %debug. This was probably the main deficiency of the notebook as a client compared to the terminal/qtconsole, and it has been finally fixed. - Add %%html, %%svg, %%javascript, and %%latex cell magics for writing raw output in notebook cells. - Fix an issue parsing LaTeX in markdown cells, which required users to type \\\, instead of \\. -Images support width and height metadata, and thereby 2x scaling (retina support). - %%file has been renamed %%writefile (%%file) is deprecated. * The input transofrmation code has been updated and rationalized. This is a somewhat specialized part of IPython, but of importance to projects that build upon it for custom environments, like Sympy and Sage. Our full release notes are here: http://ipython.org/ipython-doc/rel-1.0.0/whatsnew/version1.0.html and the gory details are here: http://ipython.org/ipython-doc/rel-1.0.0/whatsnew/github-stats-1.0.html # Installation Installation links and instructions are at: http://ipython.org/install.html And IPython is also on PyPI: http://pypi.python.org/pypi/ipython # Requirements IPython 1.0 requires Python ≥ 2.6.5 or ≥ 3.2.1. It does not support Python 3.0, 3.1, or 2.5. # Acknowledgments Last but not least, we'd like to acknowledge the generous support of those who make it possible for us to spend our time working on IPython. In particular, the Alfred P. Sloan Foundation today lets us have a solid team working full-time on the project, and without the support of Enthought Inc at multiple points in our history, we wouldn't be where we are today. The full list of our support is here: http://ipython.org/index.html#support Thanks to everyone! Please enjoy IPython 1.0, and report all bugs as usual! Fernando, on behalf of the IPython Dev Team. -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

1 0

Pdf for multivariate normal distribution
by Joris Vankerschaver Aug. 3, 2013

Aug. 3, 2013

Hi all, Is there an implementation of the PDF/CDF/etc for the multivariate normal distribution somewhere in SciPy? If not, how useful would it be to implement this? I have some snippets that I could put in the distribution framework used in scipy.stats. All the best, Joris

1 0

optimize_leastsq : error calling the function
by Delphine t Aug. 3, 2013

Aug. 3, 2013

Hello everyone, I'm having trouble using the leastsq optimisation function, could anyone help me ? I have a list of drillings, each one whith its own parameters. All this parameters are concatenated in the array called VARIABLES the dictionnary dico_taille_variable associate to each drillings the size of the its vetcor "parameters" in order to deconcatenate the VARIABLES array and apply a function calculating the residuals of each drilling in the function residuals_global that I'm trying to optimize. Residual_global calls the function residuals which is in a Class Drilling and has this aspect : def residuals(self,vars) : m=self.model(vars) f=interpolate.interp1d(self.depth, self.age) resi_age=(f(self.icemarkers_depth)-self.icemarkers_age)/self.icemarkers_sigma k=interpolate.interp1d(self.depth, self.gage) resi_gage=(k(self.gasmarkers_depth)-self.gasmarkers_age)/self.gasmarkers_sigma l=interpolate.interp1d(self.depth, self.Ddepth) resi_Ddepth=(l(self.Ddepth_depth)-self.Ddepth_Ddepth)/self.Ddepth_sigma resi_corr_tau=self.corr_tau resi_corr_a=self.corr_a resi_corr_LIDIE=self.corr_LIDIE self.retresi=np.concatenate((resi_age,resi_gage, resi_Ddepth, resi_corr_tau, resi_corr_a, resi_corr_LIDIE)) return self.retresi _______The function to optimize_____ def residuals_global(VARIABLES): index=0 vecteur=[] RESIDUALS=np.array([]) for key in dico_taille_variable: t=dico_taille_variable[key] vecteur=VARIABLES[index+0:index+t] D.residuals(vecteur) RESIDUALS=np.concatenate((RESIDUALS, D.retresi)) index=index+t print 'RESIDUALS', RESIDUALS return RESIDUALS _______The call_________ def optimisation() : vars2, hess,infodict,mesg,ier=leastsq(residuals_global, VARIABLES, full_output=1) print mesg print 'vars',vars2 print 'hess',hess return vars2,hess optimisation() _______The result_______ error Traceback (most recent call last) /usr/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 176 else: 177 filename = fname --> 178 __builtin__.execfile(filename, *where) /home/dtardif/Téléchargements/python/Main.py in <module>() 64 return vars2,hess 65 ---> 66 optimisation() 67 68 D.sigma(hess, vars2) /home/dtardif/Téléchargements/python/Main.py in optimisation() 58 59 def optimisation() : ---> 60 vars2, hess,infodict,mesg,ier=leastsq(residuals_global, VARIABLES, full_output=1) 61 print mesg 62 print 'vars',vars2 /usr/lib/python2.7/dist-packages/scipy/optimize/minpack.pyc in leastsq(func, x0, args, Dfun, full_output, col_deriv, ftol, xtol, gtol, maxfev, epsfcn, factor, diag) 281 maxfev = 200*(n + 1) 282 retval = _minpack._lmdif(func, x0, args, full_output, ftol, xtol, --> 283 gtol, maxfev, epsfcn, factor, diag) 284 else: 285 if col_deriv: error: Error occurred while calling the Python function named residuals_global _______________________________ I hope I put all the data needed to understand my problem Any idea ? Thank's ! -- View this message in context: http://scipy-user.10969.n7.nabble.com/optimize-leastsq-error-calling-the-fu… Sent from the Scipy-User mailing list archive at Nabble.com.

2 1