Mailman 3 March 2014 - NumPy-Discussion

ANN: Bokeh 0.4.2
by Bryan Van de Ven March 10, 2014

March 10, 2014

I am happy to announce the release of Bokeh version 0.4.2! Bokeh is a Python library for visualizing large and realtime datasets on the web. Its goal is to provide elegant, concise construction of novel graphics in the style of Protovis/D3, while delivering high-performance interactivity to thin clients. Bokeh includes its own Javascript library (BokehJS) that implements a reactive scenegraph representation of the plot, and renders efficiently to HTML5 Canvas. Bokeh works well with IPython Notebook, but can generate standalone graphics that embed into regular HTML. Check out the full documentation, interactive gallery, and tutorial at http://bokeh.pydata.org If you are using Anaconda, you can install with conda: conda install bokeh Alternatively, you can install with pip: pip install bokeh Some of the new features in this release include: * Additional Matplotlib and Seaborn compatibility (PolyCollection) * Extensive tutorial with exercises and solutions added to docs * new %bokeh magic for improved IPython notebook integration * Windows support for bokeh-server with two new storage backends (in-memory and shelve) Also, we've fixed lots of little bugs - see the CHANGELOG for full details. BokehJS is also available by CDN for use in standalone javascript applications: http://cdn.pydata.org/bokeh-0.4.2.js http://cdn.pydata.org/bokeh-0.4.2.css http://cdn.pydata.org/bokeh-0.4.2.min.js http://cdn.pydata.org/bokeh-0.4.2.min.css Some examples of BokehJS use can be found on the Bokeh JSFiddle page: http://jsfiddle.net/user/bokeh/fiddles/ The release of Bokeh 0.5 is planned for late March. Some notable features we plan to include are: * Abstract Rendering for semantically meaningful downsampling of large datasets * Better grid-based layout system, using Cassowary.js * More MPL/Seaborn/ggplot.py compatibility and examples * Additional tools, improved interactions, and better plot frame * Touch support Issues, enhancement requests, and pull requests can be made on the Bokeh Github page: https://github.com/continuumio/bokeh Questions can be directed to the Bokeh mailing list: bokeh(a)continuum.io Special thanks to recent contributors: Melissa Gymrek, Amy Troschinetz, Ben Zaitlen, Damian Avila, and Terry Jones Regards, Bryan Van de Ven Continuum Analytics http://continuum.io

1 0

SVG Logo of NumPy
by Christophe Bal March 10, 2014

March 10, 2014

Hello, is there a SVG version of the NumPy logo ? This would be to be used on my website. Christophe BAL

2 2

numpy gsoc ideas (was: numpy gsoc topic idea: configurable algorithm precision and vector math library integration)
by Nathaniel Smith March 7, 2014

March 7, 2014

On Mon, Mar 3, 2014 at 7:20 PM, Julian Taylor <jtaylor.debian(a)googlemail.com> wrote: > hi, > > as the numpy gsoc topic page is a little short on options I was thinking > about adding two topics for interested students. But as I have no > experience with gsoc or mentoring and the ideas are not very fleshed out > yet I'd like to ask if it might make sense at all: > > 1. configurable algorithm precision [...] > with np.precmode(default="fast"): > np.abs(complex_array) > > or fast everything except sum and hypot > > with np.precmode(default="fast", sum="kahan", hypot="standard"): > np.sum(d) [...] Not a big fan of this one -- it seems like the biggest bulk of the effort would be in figuring out a non-horrible API for exposing these things and getting consensus around it, which is not a good fit to the SoC structure. I'm pretty nervous about the datetime proposal that's currently on the wiki, for similar reasons -- I'm not sure it's actually doable in the SoC context. > 2. vector math library integration This is a great suggestion -- clear scope, clear benefit. Two more ideas: 3. Using Cython in the numpy core The numpy core contains tons of complicated C code implementing elaborate operations like indexing, casting, ufunc dispatch, etc. It would be really nice if we could use Cython to write some of these things. However, there is a practical problem: Cython assumes that each .pyx file generates a single compiled module with its own Cython-defined API. Numpy, however, contains a large number of .c files which are all compiled together into a single module, with its own home-brewed system for defining the public API. And we can't rewrite the whole thing. So for this to be viable, we would need some way to compile a bunch of .c *and .pyx* files together into a single module, and allow the .c and .pyx files to call each other. This might involve changes to Cython, some sort of clever post-processing or glue code to get existing cython-generated source code to play nicely with the rest of numpy, or something else. So this project would have the following goals, depending on how practical this turns out to be: (1) produce a hacky proof-of-concept system for doing the above, (2) turn the hacky proof-of-concept into something actually viable for use in real life (possibly this would require getting changes upstream into Cython, etc.), (3) use this system to actually port some interesting numpy code into cython. 4. Pythonic dtypes The current dtype system is klugey. It basically defines its own class system, in parallel to Python's, and unsurprisingly, this new class system is not as good. In particular, it has limitations around the storage of instance-specific data which rule out a large variety of interesting user-defined dtypes, and causes us to need some truly nasty hacks to support the built-in dtypes we do have. And it makes defining a new dtype much more complicated than defining a new Python class. This project would be to implement a new dtype system for numpy, in which np.dtype becomes a near-empty base class, different dtypes (e.g., float64, float32) are simply different subclasses of np.dtype, and dtype objects are simply instances of these classes. Further enhancements would be to make it possible to define new dtypes in pure Python by subclassing np.dtype and implementing special methods for the various dtype operations, and to make it possible for ufunc loops to see the dtype objects. This project would provide the key enabling piece for a wide variety of interesting new features: missing value support, better handling of strings and categorical data, unit handling, automatic differentiation, and probably a bunch more I'm forgetting right now. If we get someone who's up to handling the dtype thing then I can mentor or co-mentor. What do y'all think? (I don't think I have access to update that wiki page -- or maybe I'm just not clever enough to figure out how -- so it would be helpful if someone who can, could?) -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

5 7

numpy apply_along_axis named arguments
by Albert Jornet Puig March 7, 2014

March 7, 2014

Hi All, I am working with *apply_along_axis* method and I would like to apply a method that requires to pass named arguments (scipy.stats.mstats.mquantiles with prob[]). But currently, it is not possible with *apply_along_axis*. I wonder if it would make sense to add the possibility to pass named arguments. I am also aware that It could be implement using other ways (a loop for each row). That's why I would like to ask whether it makes sense or not to ask for this. Even though, I managed to modify the code, which is simple. http://pastebin.com/pBn0TbgK

2 2

Adding weights to cov and corrcoef
by Sebastian Berg March 7, 2014

March 7, 2014

Hi all, in Pull Request https://github.com/numpy/numpy/pull/3864 Neol Dawe suggested adding new parameters to our `cov` and `corrcoef` functions to implement weights, which already exists for `average` (the PR still needs to be adapted). The idea right now would be to add a `weights` and a `frequencies` keyword arguments to these functions. In more detail: The situation is a bit more complex for `cov` and `corrcoef` than `average`, because there are different types of weights. The current plan would be to add two new keyword arguments: * weights: Uncertainty weights which causes `N` to be recalculated accordingly (This is R's `cov.wt` default I believe). * frequencies: When given, `N = sum(frequencies)` and the values are weighted by their frequency. Because it appeared that the uncertainty type of weights are not obvious, while other types of weights should be pretty easily implemented by scaling `frequencies` (i.e. one may want `sum(frequencies) == len(data)`). However, we may have missed something obvious, or maybe it is already getting too statistical for NumPy, or the keyword argument might be better `uncertainties` and `frequencies`. So comments and insights are very welcome :). Regards, Sebastian

4 5

Re: [Numpy-discussion] Adding weights to cov and corrcoef (Sebastian Berg)
by David Goldsmith March 6, 2014

March 6, 2014

Date: Wed, 05 Mar 2014 17:45:47 +0100 > From: Sebastian Berg <sebastian(a)sipsolutions.net> > Subject: [Numpy-discussion] Adding weights to cov and corrcoef > To: numpy-discussion(a)scipy.org > Message-ID: <1394037947.21356.20.camel@sebastian-t440> > Content-Type: text/plain; charset="UTF-8" > > Hi all, > > in Pull Request https://github.com/numpy/numpy/pull/3864 Neol Dawe > suggested adding new parameters to our `cov` and `corrcoef` functions to > implement weights, which already exists for `average` (the PR still > needs to be adapted). > Do you mean adopted? > However, we may have missed something obvious, or maybe it is already > getting too statistical for NumPy, or the keyword argument might be > better `uncertainties` and `frequencies`. So comments and insights are > very welcome :). > +1 for it being "too baroque" for NumPy--should go in SciPy (if it isn't already there): IMHO, NumPy should be kept as "lean and mean" as possible, embellishments are what SciPy is for. (Again, IMO.) DG

5 8

Re: [Numpy-discussion] Adding weights to cov and corrcoef
by David Goldsmith March 6, 2014

March 6, 2014

Date: Thu, 06 Mar 2014 13:40:40 +0100 > From: Sebastian Berg <sebastian(a)sipsolutions.net> > Subject: Re: [Numpy-discussion] Adding weights to cov and corrcoef > (Sebastian Berg) > To: numpy-discussion(a)scipy.org > Message-ID: <1394109640.9122.13.camel@sebastian-t440> > Content-Type: text/plain; charset="UTF-8" > > On Mi, 2014-03-05 at 10:21 -0800, David Goldsmith wrote: > > +1 for it being "too baroque" for NumPy--should go in SciPy (if it > > isn't already there): IMHO, NumPy should be kept as "lean and mean" as > > possible, embellishments are what SciPy is for. (Again, IMO.) > > > > Well, on the other hand, scipy does not actually have a `std` function > of its own, I think. Oh, well, in that case forget I said anything. (Though I think it's "interesting" that no one else has chimed in: if you're the only one that needs it (at this time), perhaps it would be best to "roll your own" and then offer to "pass it around." :-)) DG > So if it is quite useful I think this may be an > option (I don't think I ever used weights with std, so I can't argue > strongly for inclusion myself). Unless adding new functions to > `scipy.stats` (or just statsmodels) which implement different types of > weights is the longer term plan, then things might bite... > > > DG > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > ------------------------------ > > Message: 5 > Date: Thu, 6 Mar 2014 13:45:36 +0000 > From: Nathaniel Smith <njs(a)pobox.com> > Subject: Re: [Numpy-discussion] numpy gsoc ideas (was: numpy gsoc > topic idea: configurable algorithm precision and vector math > library > integration) > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAPJVwBm=no71WvC9Zjh7DXNaGn0jpAGmvrOyoVvmHK-MW= > R3Fw(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Thu, Mar 6, 2014 at 5:17 AM, Sturla Molden <sturla.molden(a)gmail.com> > wrote: > > Nathaniel Smith <njs(a)pobox.com> wrote: > > > >> 3. Using Cython in the numpy core > >> > >> The numpy core contains tons of complicated C code implementing > >> elaborate operations like indexing, casting, ufunc dispatch, etc. It > >> would be really nice if we could use Cython to write some of these > >> things. > > > > So the idea of having a NumPy as a pure C library in the core is > abandoned? > > This question doesn't make sense to me so I think I must be missing > some context. > > Nothing is abandoned: This is one email by one person on one mailing > list suggesting a project to the explore the feasibility of something. > And anyway, Cython is just a C code generator, similar in principle to > (though vastly more sophisticated than) the ones we already use. It's > not like we've ever promised our users we'll keep stable which kind of > code generators we use internally. > > >> However, there is a practical problem: Cython assumes that > >> each .pyx file generates a single compiled module with its own > >> Cython-defined API. Numpy, however, contains a large number of .c > >> files which are all compiled together into a single module, with its > >> own home-brewed system for defining the public API. And we can't > >> rewrite the whole thing. So for this to be viable, we would need some > >> way to compile a bunch of .c *and .pyx* files together into a single > >> module, and allow the .c and .pyx files to call each other. > > > > Cython takes care of that already. > > > > http://docs.cython.org/src/userguide/sharing_declarations.html#cimport > > > > > http://docs.cython.org/src/userguide/external_C_code.html#using-cython-decl… > > Linking multiple .c and .pyx files together into a single .so/.dll is > much more complicated than just using 'cimport'. Try it if you don't > believe me :-). > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > > > ------------------------------ > > Message: 6 > Date: Thu, 6 Mar 2014 13:59:30 +0000 > From: Nathaniel Smith <njs(a)pobox.com> > Subject: Re: [Numpy-discussion] numpy gsoc ideas (was: numpy gsoc > topic idea: configurable algorithm precision and vector math > library > integration) > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAPJVwB=PmquKm5j4-oquCkLkcK8G1pipB8XLbq5= > 26izjbJPjg(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Thu, Mar 6, 2014 at 9:11 AM, David Cournapeau <cournape(a)gmail.com> > wrote: > > > > On Wed, Mar 5, 2014 at 9:11 PM, Nathaniel Smith <njs(a)pobox.com> wrote: > >> So this project would have the following goals, depending on how > >> practical this turns out to be: (1) produce a hacky proof-of-concept > >> system for doing the above, (2) turn the hacky proof-of-concept into > >> something actually viable for use in real life (possibly this would > >> require getting changes upstream into Cython, etc.), (3) use this > >> system to actually port some interesting numpy code into cython. > > > > > > Having to synchronise two projects may be hard for a GSoC, no ? > > Yeah, if someone is interested in this it would be nice to get someone > from Cython involved too. But that's why the primary goal is to > produce a proof-of-concept -- even if all that comes out is that we > learn that this cannot be done in an acceptable manner, then that's > still a succesful (albeit disappointing) result. > > > Otherwise, I am a bit worried about cython being used on the current C > code > > as is, because core and python C API are so interwined (especially > > multiarray). > > I don't understand this objection. The whole advantage of Cython is > that it makes it much, much easier to write code that involves > intertwining complex algorithms and heavy use of the Python C API :-). > There's tons of bug-prone spaghetti in numpy for doing boring things > like refcounting, exception passing, and argument parsing. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > > > ------------------------------ > > Message: 7 > Date: Thu, 6 Mar 2014 10:35:15 -0700 > From: Charles R Harris <charlesr.harris(a)gmail.com> > Subject: Re: [Numpy-discussion] 1.8.1rc1 on sourceforge. > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > < > CAB6mnx+btuF3vKxvebfBZKybggp+C6mtnb7zD1ck1bqk9VXV2w(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On Wed, Mar 5, 2014 at 7:28 PM, Matthew Brett <matthew.brett(a)gmail.com > >wrote: > > > Hi, > > > > On Wed, Mar 5, 2014 at 3:29 PM, Matthew Brett <matthew.brett(a)gmail.com> > > wrote: > > > Hi, > > > > > > I built (and tested) some numpy wheels for the rc1: > > > > > > http://nipy.bic.berkeley.edu/numpy-dist/ > > > > Now building, installing, testing, uploading wheels nightly on OSX 10.9: > > > > http://nipy.bic.berkeley.edu/builders/numpy-bdist-whl-osx-2.7 > > http://nipy.bic.berkeley.edu/builders/numpy-bdist-whl-osx-3.3 > > > > and downloading, testing built wheels on OSX 10.6: > > > > http://nipy.bic.berkeley.edu/builders/numpy-bdist-whl-osx-2.7-downloaded > > http://nipy.bic.berkeley.edu/builders/numpy-bdist-whl-osx-3.3-downloaded > > > > Chuck - are you release manager for this cycle? Would you mind > > sending me your public ssh key so I can give you access to the > > buildbots for custom builds and so on? > > > > Cheers, > > > > > Julian has done most of the work for 1.8.1. I did the 1.8.0 release because > it needed doing, but building releases isn't my strong point and Ralf > actually did the builds for that. So I'll happily send you my ssh, but > either Ralph or Julian might be a better bet for getting the work done :) > > Chuck >

1 0

1.8.1 release
by Charles R Harris March 6, 2014

March 6, 2014

Hi All, A lot of fixes have gone into the 1.8.x branch and it looks about time to do a bugfix release. There are a couple of important bugfixes still to backport, but if all goes well next weekend, March 1, looks like a good target date. So give the current 1.8.x branch a try so as to check that it covers your most urgent bugfix needs. Chuck

9 18

numpy gsoc topic idea: configurable algorithm precision and vector math library integration
by Julian Taylor March 6, 2014

March 6, 2014

hi, as the numpy gsoc topic page is a little short on options I was thinking about adding two topics for interested students. But as I have no experience with gsoc or mentoring and the ideas are not very fleshed out yet I'd like to ask if it might make sense at all: 1. configurable algorithm precision some functions in numpy could be implemented in different ways depending on requirements on speed and numerical precision. Two examples that come to my mind are hypot and sum hypot (or abs(complex)) use the C99 hypot function which guarantees 1ulp precision but is very slow compared to a simple sqrt(a**2 +b**2). This precision might not be required for all applications, overflow safety might be enough. summation in numpy 1.9 is performed via pairwise summation which has O(log(n)*e) error properties, but only for the fast axis. An alternative O(e) approach would be kahan summation which works for all axis but is 4 time slower than normal summation (a bit can be regained via vectorization thought) My idea is have an option to change the algorithms used in numpy depending on the set requirements. E.g. with np.precmode(default="fast"): np.abs(complex_array) or fast everything except sum and hypot with np.precmode(default="fast", sum="kahan", hypot="standard"): np.sum(d) I have not though much about implementation, it might be tricky to get this threadsafe in the current ufunc model. 2. vector math library integration some operations like powers, sin, cos etc are relatively slow in numpy depending on the c library used. There are now a few free libraries available that make use of modern hardware to speed these operations up, e.g. sleef and yeppp (also mkl but I have no interest in supporting non-free software) It might be interesting to investigate if these libraries can be integrated with numpy. This also somewhat ties in with the configurable precision mode as the vector math libraries often have different options depending on precision and speed requirements. Do those sound like topics we could add to our wiki?

4 3

GSoC: ideas & finding mentors
by Ralf Gommers March 5, 2014

March 5, 2014

Hi students, There is quite a bit of interest in GSoC ideas for Scipy and Numpy, which is great to see. The official application period to submit proposals opens next week and closes on the 21st, which is in two weeks and a bit. So now is the time to start discussing draft proposals on the list. There have been a few ideas posted on the list which haven't gotten enough feedback yet (FFTs, cluster, ODEs). This may reflect the lack of an active maintainer of those modules, so it will be harder to find a suitable mentor. I want to point out that this is also a chicken-and-egg problem: if you're actively posting and improving your draft and sending some pull requests to fix some small issues, it shows both your willingness to work with the community and how you work with core devs to get your PRs merged, which helps find an interested mentor. To tackle the student-mentor matchmaking from another angle, I've added on https://github.com/scipy/scipy/wiki/GSoC-project-ideas a "potential mentors" field to the idea I know the names for (Stefan and me, for wavelets). If other potential mentors could do the same for other ideas, that would be very helpful. I can take some guesses (Pauli, Evgeni for splines? Chuck, Chris Barker for datetime?) but I haven't added any names. So please do add your name, keeping in mind that this is to get the process going and not yet a full commitment. Final note: you don't necessarily have to be a core developer to be a co-mentor. If you're an expert on a topic that a student is interested in and would like to see that project happen, please indicate you're willing to help. Cheers, Ralf

1 0