Mailman 3 August 2013 - SciPy-Dev

Discussion on a new interface for multidimensional quadrature
by Nathan Woods Aug. 15, 2013

Aug. 15, 2013

I'm copying the most relevant parts over from a discussion on GitHub, having to do with a new n-dimensional integration tool. The discussion is whether or not to make nquad's new interface consistent with dbl-and tplquad, even though their precedent is arguably more confusing to users. nquad is a recursively defined wrapper to scipy.integrate.quad that allows for n-dimensional integration with almost the entire list of arguments allowed by quad. It is therefore more general than either dbl- or tplquad, and encompasses their use cases, while allowing the user greater control over the integration. The interface is given: nquad( f(x0,x1,…xn,t0,…tm) ,[[x0_lo,x0_hi],…[xn_lo,xn_hi]],[t0,…tm],[opts0,…optsn]). The ranges of integration are given in the same order as the arguments in the function to be integrated. This ordering is consistent with both Matlab and Octave, and it is similar to what is encountered in other SciPy packages such as fft. It is also what makes sense to new users. dbl- and tplquad, however, reverse the order of arguments in f. From the documentation for tplquad: Return the triple integral of func(z, y, x) from x=a..b, y=gfun(x)..hfun(x), and z=qfun(x,y)..rfun(x,y) This is confusing, and contrary to common practice. It is proposed that nquad's calling sequence be left as is, and that the use of dbl- and tplquad be deprecated in some future version of SciPy, and that nquad be recommended for multidimensional integration in new code. dbl- and tplquad would of course be retained, though not recommended, in order to avoid breaking existing code. Wow, that came out all legalistic. Anyway, what do you all think about this change? I have a horse in the race (I wrote most of nquad), but I really think that maintaining the backwards style from dbl- and tplquad and enshrining it in new code is a bad idea. This is a perfect time to get away from it, if that's what we want to do. > Good question. I think this needs some discussion on the scipy-dev list, because it would be a fairly large change (dblquad/tplquad are old and widely used I think). Could you please bring it up there? > >> Yeah, I remember wondering how to handle that. It's not a very difficult fix, and most of the work will be in changing over the examples and tests so they work with the new ordering. However, do we really want to stick with the old order? >> >> If we want to change it, then now is a perfect time, since nquad duplicates and extends the functionality of dbl- and tplquad. Those two could be deprecated and retained for compatibility, while nquad is recommended for new code. Since the calling sequence for nquad is so different, new users will have to adjust anyway. >> >> The current argument list for tplquad is confusing. The order of arguments in the function does not match with the order of the range functions that must be provided, which is a source of confusion. The ordering was presumably chosen to make things look as much like quad as possible, but is that really important now? >> >> As a reference, Matlab's integral3 and triplequad and GNU Octave's triple quad both order things (x,y,z). SciPy's fft2 also appears to do things the same way. > >>> There's one thing that needs discussion: the integration order is reversed compared to dblquad and tplquad. Where you'd use def func3d(z, y, x) in tplquad you have to use def func3d(x, y, z) here. I was never a fan of the order in dbp/tplquad, but we're stuck with that and I thinknquad should use that same order for consistency.

4 15

Confused about scipy.interpolate
by Juan Luis Cano Aug. 14, 2013

Aug. 14, 2013

Hello all, I was doing some experiments for the first time with scipy.interpolate today with the purpose of writing a brief article/tutorial on the matter and, as a user, I got a bit confused with the current state of the package and the "best practices". Let me explain. The most straight-forward way I found to make a polynomial interpolation was the funcion interp1d. It worked well, and I could even specify a higher polynomial order so I could replicate and explain the Runge phenomenon. Except for the fact that it refuses to work outside of the boundaries. I had imagined that interp1d would give me some sort of polynomial representation that could be naturally extended outside of the domain specified by the interpolation points. Moreover, it seems to use some "older" functions (see below). So I read "a more recent wrapper of the FITPACK routines", and I tried UnivariateSpline. Which naturally works inside of the domain, but cannot have a degree higher than 5, probably because it's geared to create splines (as the name points out). So still not valid (I wanted a 10th degree polynomial to show the oscillations). splmake and company are advertised as an "older" wrapping, so I felt like not using them (for the sake of using the newer things). For example, splmake was gone from docs (http://mail.scipy.org/pipermail/scipy-user/2010-March/024489.html) and besides it has an issue pending, with the word "confusing" in the title (https://github.com/scipy/scipy/issues/1408). And finally I found barycentric_interpolate and krogh_interpolate, which gave me the same results but it seems one is useful if I'm adding new points and the other is useful if I want the derivatives. But now I wonder which one is better. And this leads me to a question - what if I just want a simple, arbitrary degree polynomial interpolation? I thought that maybe something could be done similar to what was introduced in 0.11.0 with minimize and minimize_scalar, which I think it was a great improvement because the user usually doesn't have to worry about the method used, unless they want to specify a certain one (even if the naming was IMHO a bit unfortunate regarding the latter, but that's another story). So is there a way an `interpolate` function can be created, with a string argument specifying the method (with default "barycentric", for example)? Sorry for the long email, but I tried to explain in detail what was my "path" as a user trying to figure out what to do. Hope it is useful for the devs. Regards Juan Luis Cano

2 1

Re: [SciPy-Dev] Multivariate normal distribution in scipy.stats
by Joris Vankerschaver Aug. 14, 2013

Aug. 14, 2013

Hi all, I've made a PR for this; see https://github.com/scipy/scipy/pull/2726 I used the design of stats.rv_continuous as my inspiration, so that the calling conventions agree with the 1D case. It should also be relatively straightforward to extend this setup with other methods or incorporate other multivariate random variables. This is my first contribution to scipy! This was fun, but please let me know what needs to be improved. With best wishes, Joris >>>> +1 to add it without a framework. Can be either a new module or in the >>> scipy.stats namespace - there aren't that many multivariate distributions >>> that it would necessarily warrant a new submodule. >>> >>> I only meant implementation-wise. I think we should have a new >>> scipy/stats/multivariate.py file and import its public names into the >>> scipy.stats namespace. distributions.py is ... crowded and is >> (rightfully, >>> IMO) dedicated to the rv_continuous/rv_discrete framework. >>> >> >> Ah OK. I fully agree then (except name it _multivariate.py).

1 0

Testing SciPy against NumPy master.
by Blake Griffith Aug. 12, 2013

Aug. 12, 2013

Would it be possible to have TravisCI test against NumPy master? Currently it only tests against the latest stable version of NumPy, 1.7.x with various versions of python. If this is possible, it would probably only need to be done with Python 3.3. When I test SciPy with NumPy master it seems like something new always broken.

3 4

[ANN] IPython 1.0 is finally released, nearly 12 years in the making!
by Fernando Perez Aug. 9, 2013

Aug. 9, 2013

Hi all, I am incredibly thrilled, on behalf of the amazing IPython Dev Team, to announce the official release of IPython 1.0 today, an effort nearly 12 years in the making. The previous version (0.13) was released on June 30, 2012, and in this development cycle we had: ~12 months of work. ~700 pull requests merged. ~600 issues closed (non-pull requests). contributions from ~150 authors. ~4000 commits. # A little context What does "1.0" mean for IPython? Obviously IPython has been a staple of the scientific Python community for years, and we've made every effort to make it a robust and production ready tool for a long time, so what exactly do we mean by tagging this particular release as 1.0? Basically, we feel that the core design of IPython, and the scope of the project, is where we want it to be. What we have today is what we consider a reasonably complete, design- and scope-wise, IPython 1.0: an architecture for interactive computing, that can drive kernels in a number of ways using a well-defined protocol, and rich and powerful clients that let users control those kernels effectively. Our different clients serve different needs, with the old workhorse of the terminal still being very useful, but much of our current development energy going into the Notebook, obviously. The Notebook enables interactive exploration to become Literate Computing, bridging the gaps from individual work to collaboration and publication, all with an open file format that is a direct record of the underlying communication protocol. There are obviously plenty of open issues (many of them very important) that need fixing, and large and ambitious new lines of development for the years to come. But the work of the last four years, since the summer of 2009 when Brian Granger was able to devote a summer (thanks to funding from the NiPy project - nipy.org) to refactoring the old IPython core code, finally opened up or infrastructure for real innovation. By disentangling what was a useful but impenetrable codebase, it became possible for us to start building a flexible, modern system for interactive computing that abstracted the old REPL model into a generic protocol that kernels could use to talk to clients. This led at first to the creation of the Qt console, and then to the Notebook and out-of-process terminal client. It also allowed us to (finally!) unify our parallel computing machinery with the rest of the interactive system, which Min Ragan-Kelley pulled off in a development tour de force that involved rewriting in a few weeks a huge and complex Twisted-based system. We are very happy with how the Notebook work has turned out, and it seems the entire community agrees with us, as the uptake has been phenomenal. Back from the very first "IPython 0.0.1" that I started in 2001: https://gist.github.com/fperez/1579699 there were already hints of tools like Mathematica: it was my everyday workhorse as a theoretical physicist and I found its Notebook environment invaluable. But as a grad student trying out "just an afternoon hack" (IPython was my very first Python program as I was learning the language), I didn't have the resources, skills or vision to attempt building an entire notebook system, and to be honest the tools of the day would have made that enterprise a miserable one. But those ideas were always driving our efforts, and as IPython started becoming a project with a team, we made multiple attempts to get a good Notebook built around IPython. Those interested can read an old blog post of mine with the history (http://blog.fperez.org/2012/01/ipython-notebook-historical.html). The short story is that in 2011, on our sixth attempt, Brian was again able to devote a focused summer into using our client-server architecture and, with the stack of the modern web (Javascript, CSS, websockets, Tornado, ...), finally build a robust system for Literate Computing across programming languages. Today, thanks to the generous support and vision of Josh Greenberg at the Alfred P. Sloan Foundation, we are working very hard on building the notebook infrastructure, and this release contains major advances on that front. We have high hopes for what we'll do next; as a glimpse of the future that this enables, now there is a native Julia kernel that speaks to our clients, notebook included: https://github.com/JuliaLang/IJulia.jl. # Team I can't stress enough how impressed I am with the work people are doing in IPython, and what a privilege it is to work with colleagues like these. Brian Granger and Min Ragan-Kelley joined IPython around 2005, initially working on the parallel machinery, but since ~ 2009 they have become the heart of the project. Today Min is our top committer and knows our codebase better than anyone else, and I can't imagine better partners for an effort like this. And from regulars in our core team like Thomas Kluyver, Matthias Bussonnier, Brad Froehle and Paul Ivanov to newcomers like Jonathan Frederic and Zach Sailer, in addition to the many more whose names are in our logs, we have a crazy amount of energy being poured into IPython. I hope we'll continue to harness it productively! The full list of contributors to this release can be seen here: http://ipython.org/ipython-doc/rel-1.0.0/whatsnew/github-stats-1.0.html # Release highlights * nbconvert: this is the major piece of new functionality in this cycle, and was an explicit part of our roadmap (https://github.com/ipython/ipython/wiki/Roadmap:-IPython). nbconvert is now an IPython subcommand to convert notebooks into other formats such as HTML or LaTeX, but more importantly, it's a very flexible system that lets you write custom templates to generate new output with arbitrary control over the formatting and transformations that are applied to the input. We want to stress that despite the fact that a huge amount of work went into nbconvert, this should be considered a *tech preview* release. We've come to realize how complex this problem is, and while we'll make every effort to keep the high-level command-line syntax and APIs as stable as possible, it is quite likely that the internals will continue to evolve, possibly in backwards-incompatible ways. So if you start building services and libraries that make heavy use of the nbconvert internals, please be prepared for some turmoil in the months to come, and ping us on the dev list with questions or concerns. * Notebook improvements: there has been a ton of polish work in the notebook at many levels, though the file format remains unchanged from 0.13, so you shouldn't have any problems sharing notebooks with colleagues still using 0.13. - Autosave: probably the most oft-requested feature, the notebook server now autosaves your files! You can still hit Ctrl-S to force a manual save (which also creates a special 'checkpoint' you can come back to). - The notebook supports raw_input(), and thus also %debug. This was probably the main deficiency of the notebook as a client compared to the terminal/qtconsole, and it has been finally fixed. - Add %%html, %%svg, %%javascript, and %%latex cell magics for writing raw output in notebook cells. - Fix an issue parsing LaTeX in markdown cells, which required users to type \\\, instead of \\. -Images support width and height metadata, and thereby 2x scaling (retina support). - %%file has been renamed %%writefile (%%file) is deprecated. * The input transofrmation code has been updated and rationalized. This is a somewhat specialized part of IPython, but of importance to projects that build upon it for custom environments, like Sympy and Sage. Our full release notes are here: http://ipython.org/ipython-doc/rel-1.0.0/whatsnew/version1.0.html and the gory details are here: http://ipython.org/ipython-doc/rel-1.0.0/whatsnew/github-stats-1.0.html # Installation Installation links and instructions are at: http://ipython.org/install.html And IPython is also on PyPI: http://pypi.python.org/pypi/ipython # Requirements IPython 1.0 requires Python ≥ 2.6.5 or ≥ 3.2.1. It does not support Python 3.0, 3.1, or 2.5. # Acknowledgments Last but not least, we'd like to acknowledge the generous support of those who make it possible for us to spend our time working on IPython. In particular, the Alfred P. Sloan Foundation today lets us have a solid team working full-time on the project, and without the support of Enthought Inc at multiple points in our history, we wouldn't be where we are today. The full list of our support is here: http://ipython.org/index.html#support Thanks to everyone! Please enjoy IPython 1.0, and report all bugs as usual! Fernando, on behalf of the IPython Dev Team. -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

1 0

Re: [SciPy-Dev] Multivariate normal distribution in scipy.stats
by J Vankerschaver Aug. 6, 2013

Aug. 6, 2013

Hi Ralf and Robert, Thanks for your suggestions. This makes the task a lot more manageable; I'll start right away. With best wishes, Joris > > > +1 to add it without a framework. Can be either a new module or in the > > scipy.stats namespace - there aren't that many multivariate distributions > > that it would necessarily warrant a new submodule. > > > > I only meant implementation-wise. I think we should have a new > > scipy/stats/multivariate.py file and import its public names into the > > scipy.stats namespace. distributions.py is ... crowded and is > (rightfully, > > IMO) dedicated to the rv_continuous/rv_discrete framework. > > > > Ah OK. I fully agree then (except name it _multivariate.py). > >

1 0

Multivariate normal distribution in scipy.stats
by Joris Vankerschaver Aug. 5, 2013

Aug. 5, 2013

Dear all, I was wondering if there's any interest to have the multivariate normal distribution integrated into scipy.stats. Its PDF is easily calculated, for instance by something like def multinormal_pdf(r, mean, cov): """ Probability density function for a multidimensional Gaussian distribution.""" dim = r.shape[-1] dev = r - mean maha = np.einsum('...k,...kl,...l->...', dev, np.linalg.pinv(cov), dev) return (2 * np.pi)**(-0.5 * dim) * np.linalg.det(cov)**(0.5) * np.exp(-0.5 * maha) Here, r is an array-like of position vectors, with the last axis labeling the components. While all of this would be useful and not too hard to code up, it doesn't seem to fit within the scipy.stats.rv_continuous framework, which seems to be explicitly geared to 1D random variables. Is there a natural place where this code could be fitted in, or is this functionality already somewhere else in SciPy? With best wishes, Joris

3 4

Contributing to the Documentation
by Clemens Novak Aug. 4, 2013

Aug. 4, 2013

Hi, i've been working with scipy for some time and am a great fan of the tutorial pages. I've found some spots where some additional information / an example would be nice (at least in my opinion); e.g. the scipy.signal part explains a lot of concepts, but some examples would be helpful; scipy.fftpack would require some attention... On the contribution page you encourage enhancements to scipy's documentation - is still so, and - if yes - do you prefer pull requests (via Github) or is it better to use the documentation wiki? Regards - Clemens

5 9

labeling issues as easy-fix
by Ralf Gommers Aug. 4, 2013

Aug. 4, 2013

Hi, At https://github.com/scipy/scipy/issues I've just added an easy-fix label, to guide newcomers to issues that don't require too much scipy background or hard debugging. I had planned to do that for a while, and the upcoming EuroSciPy sprint is a good motivation to get it done. Any help would be much appreciated. Maybe module maintainers could go through their module? Adding labels requires commit rights, but anyone without those who wants to help out can either just leave a comment on the ticket or send me the issue ID. Ralf

3 7

welcome Alex Griffing to the scipy team
by Ralf Gommers Aug. 2, 2013

Aug. 2, 2013

Hi all, I'd like to welcome Alex Griffing to the scipy development team. He was effectively already a part of it as many of you may have noticed but we've made it official by giving him commit rights. Alex has done quite some work on linalg over the last months, and more maintenance work and PR reviewing in other modules: see: https://www.ohloh.net/p/scipy/contributors/21026012405206). Cheers, Ralf

3 2