Mailman 3 February 2014 - NumPy-Discussion

ANN: Bokeh 0.4 Release
by Bryan Van de Ven Feb. 5, 2014

Feb. 5, 2014

I am pleased to announce the release of Bokeh version 0.4! Bokeh is a Python library for visualizing large and realtime datasets on the web. Its goal is to provide elegant, concise construction of novel graphics in the style of Protovis/D3, while delivering high-performance interactivity to thin clients. Bokeh includes its own Javascript library (BokehJS) that implements a reactive scenegraph representation of the plot, and renders efficiently to HTML5 Canvas. Bokeh works well with IPython Notebook, but can generate standalone graphics that embed into regular HTML. Check out the full documentation and interactive gallery at http://bokeh.pydata.org If you are using Anaconda, you can install with conda: conda install bokeh Alternatively, you can install with pip: pip install bokeh Some of the new features in this release include: * Preliminary work on Matplotlib support: convert MPL figures to Bokeh plots * Free public beta of Bokeh plot hosting at http://bokehplots.com * Tool improvements: - "always on" pan tool and wheel zoom tool (with shift key) - box zoom tool - viewport reset tool * Enhanced datetime axis, with better performance and nicer ticking * Expanded testing, including TravisCI integrations and static image output using PhantomJS * RGBA and color mapped image plots now available from Python * Python 3 supported * Vastly improved documentation for glyphs, with inline examples and JSFiddle integration Also, we've fixed lots of little bugs - see the CHANGELOG for full details. Bokeh will be having a free "Office Hours" later this week! Join us this Thursday at 2pm CST on EngineHereathttps://www.enginehere.com/stream/437/bokeh-04-release/ for a live informational session about the latest release. We'll be covering all the newest features and updates through a combination of live lecture, Q&A, and pair programming. It's all free, just sign up to the EngineHere learning platform. BokehJS is also available by CDN for use in standalone javascript applications: http://cdn.pydata.org/bokeh-0.4.js http://cdn.pydata.org/bokeh-0.4.css http://cdn.pydata.org/bokeh-0.4.min.js http://cdn.pydata.org/bokeh-0.4.min.css Some examples of BokehJS use can be found on the Bokeh JSFiddle page: http://jsfiddle.net/user/bokeh/fiddles/ The release of Bokeh 0.5 is planned for late March. Some notable features we plan to include are: * Abstract Rendering for semantically meaningful downsampling of large datasets * Better grid-based layout system, using Cassowary.js * Selection tools, tooltips, etc. Issues, enhancement requests, and pull requests can be made on the Bokeh Github page: https://github.com/continuumio/bokeh Questions can be directed to the Bokeh mailing list: bokeh(a)continuum.io Special thanks to recent contributors: Janek Klawe, Samantha Hughes, Rebecca Paz, and Benedikt Sauer. Regards, Bryan Van de Ven Continuum Analytics http://continuum.io

1 0

striding through arbitrarily large files
by RayS Feb. 5, 2014

Feb. 5, 2014

I was struggling with methods of reading large disk files into numpy efficiently (not FITS or .npy, just raw files of IEEE floats from numpy.tostring()). When loading arbitrarily large files it would be nice to not bother reading more than the plot can display before zooming in. There apparently are no built in methods that allow skipping/striding... With a 2GB file, I want to read n (like 4096) evenly sampled points out of it. I tried making a dtype, and other tricks, to read "Pythonically", but failed. I broke down and used a for loop with fh.seek() and fromfile() The file will be open()ed once, but data read many times. num_channels = 9 desired_len = 4096 bytes_per_val = numpy.dtype(numpy.float32).itemsize f_obj = open(path, 'rb') f_obj.seek(0,2) file_length = f_obj.tell() f_obj.seek(0,0) bytes_per_smp = num_channels * bytes_per_val num_samples = file_length / bytes_per_smp stride_smps = num_samples / desired_len ## an int stride_bytes = (stride_smps - 1) * bytes_per_smp arr = numpy.zeros((desired_len, 9)) for i in range(0, desired_len, 1): f_obj.seek(i*stride_bytes, 0) arr[i] = numpy.fromfile(f_obj, dtype='f32', count=9) So, is there a better way to move the pointer through the file without a for loop? A generator? The dtype and other methods like mmap fail with memoryError since they still try to load the whole file, although apparently you can mmap with 64bit systems, which I might try soon with a new 64bit install. - Ray

4 4

Re: [Numpy-discussion] Fast decrementation of indices
by Rick White Feb. 4, 2014

Feb. 4, 2014

I think you'll find the algorithm below to be a lot faster, especially if the arrays are big. Checking each array index against the list of included or excluded elements is must slower than simply creating a secondary array and looking up whether the elements are included or not. b = np.array([ [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 7, 8, 9, 10, 11], [5, 6, 1, 0, 2, 7, 3, 8, 1, 4, 9, 2, 10, 5, 3, 4, 11, 0, 0, 1, 2, 3, 4, 5] ]) a = [1,2,3,7,8] keepdata = np.ones(12, dtype=np.bool) keepdata[a] = False w = np.where(keepdata[b[0]] & keepdata[b[1]]) newindex = keepdata.cumsum()-1 c = newindex[b[:,w[0]]] Cheers, Rick On 2 February 2014 20:58, Mads Ipsen <mads.ipsen(a)gmail.com> wrote: > Hi, > > I have run into a potential 'for loop' bottleneck. Let me outline: > > The following array describes bonds (connections) in a benzene molecule > > b = [[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 7, > 8, 9, 10, 11], > [5, 6, 1, 0, 2, 7, 3, 8, 1, 4, 9, 2, 10, 5, 3, 4, 11, 0, 0, 1, > 2, 3, 4, 5]] > > ie. bond 0 connects atoms 0 and 5, bond 1 connects atom 0 and 6, etc. In > practical examples, the list can be much larger (N > 100.000 connections. > > Suppose atoms with indices a = [1,2,3,7,8] are deleted, then all bonds > connecting those atoms must be deleted. I achieve this doing > > i_0 = numpy.in1d(b[0], a) > i_1 = numpy.in1d(b[1], a) > b_i = numpy.where(i_0 | i_1)[0] > b = b[:,~(i_0 | i_1)] > > If you find this approach lacking, feel free to comment. > > This results in the following updated bond list > > b = [[0, 0, 4, 4, 5, 5, 5, 6, 10, 11] > [5, 6, 10, 5, 4, 11, 0, 0, 4, 5]] > > This list is however not correct: Since atoms [1,2,3,7,8] have been > deleted, the remaining atoms with indices larger than the deleted atoms > must be decremented. I do this as follows: > > for i in a: > b = numpy.where(b > i, bonds-1, bonds) (*) > > yielding the correct result > > b = [[0, 0, 1, 1, 2, 2, 2, 3, 5, 6], > [2, 3, 5, 2, 1, 6, 0, 0, 1, 2]] > > The Python for loop in (*) may easily contain 50.000 iteration. Is there > a smart way to utilize numpy functionality to avoid this? > > Thanks and best regards, > > Mads > > -- > +---------------------------------------------------------+ > | Mads Ipsen | > +----------------------+----------------------------------+ > | G?seb?ksvej 7, 4. tv | phone: +45-29716388 | > | DK-2500 Valby | email: mads.ipsen(a)gmail.com | > | Denmark | map : www.tinyurl.com/ns52fpa | > +----------------------+----------------------------------+

2 1

Indexing changes in 1.9
by Charles R Harris Feb. 3, 2014

Feb. 3, 2014

Sebastian has done a lot of work to refactor/rationalize numpy indexing. The changes are extensive enough that it would be good to have more public review, so here is the release note. The NumPy indexing has seen a complete rewrite in this version. This makes > most advanced integer indexing operations much faster and should have no > other implications. > However some subtle changes and deprecations were introduced in advanced > indexing operations: > > * Boolean indexing into scalar arrays will always return a new 1-d array. > This means that ``array(1)[array(True)]`` gives ``array([1])`` and > not the original array. > * Advanced indexing into one dimensional arrays used to have > (undocumented) > special handling regarding repeating the value array in assignments > when the shape of the value array was too small or did not match. > Code using this will raise an error. For compatibility you can use > ``arr.flat[index] = values``, which uses the old code branch. > * The iteration order over advanced indexes used to be always C-order. > In NumPy 1.9. the iteration order adapts to the inputs and is not > guaranteed (with the exception of a *single* advanced index which is > never reversed for compatibility reasons). This means that the result > is > undefined if multiple values are assigned to the same element. > An example for this is ``arr[[0, 0], [1, 1]] = [1, 2]``, which may > set ``arr[0, 1]`` to either 1 or 2. > * Equivalent to the iteration order, the memory layout of the advanced > indexing result is adapted for faster indexing and cannot be predicted. > * All indexing operations return a view or a copy. No indexing operation > will return the original array object. > * In the future Boolean array-likes (such as lists of python bools) > will always be treated as Boolean indexes and Boolean scalars > (including > python `True`) will be a legal *boolean* index. At this time, this is > already the case for scalar arrays to allow the general > ``positive = a[a > 0]`` to work when ``a`` is zero dimensional. > * In NumPy 1.8 it was possible to use `array(True)` and `array(False)` > equivalent to 1 and 0 if the result of the operation was a scalar. > This will raise an error in NumPy 1.9 and, as noted above, treated as a > boolean index in the future. > * All non-integer array-likes are deprecated, object arrays of custom > integer like objects may have to be cast explicitly. > * The error reporting for advanced indexing is more informative, however > the error type has changed in some cases. (Broadcasting errors of > indexing arrays are reported as `IndexError`) > * Indexing with more then one ellipsis (`...`) is deprecated. > Thoughts? Chuck

4 7

Fast decrementation of indices
by Mads Ipsen Feb. 3, 2014

Feb. 3, 2014

Hi, I have run into a potential 'for loop' bottleneck. Let me outline: The following array describes bonds (connections) in a benzene molecule b = [[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 7, 8, 9, 10, 11], [5, 6, 1, 0, 2, 7, 3, 8, 1, 4, 9, 2, 10, 5, 3, 4, 11, 0, 0, 1, 2, 3, 4, 5]] ie. bond 0 connects atoms 0 and 5, bond 1 connects atom 0 and 6, etc. In practical examples, the list can be much larger (N > 100.000 connections. Suppose atoms with indices a = [1,2,3,7,8] are deleted, then all bonds connecting those atoms must be deleted. I achieve this doing i_0 = numpy.in1d(b[0], a) i_1 = numpy.in1d(b[1], a) b_i = numpy.where(i_0 | i_1)[0] b = b[:,~(i_0 | i_1)] If you find this approach lacking, feel free to comment. This results in the following updated bond list b = [[0, 0, 4, 4, 5, 5, 5, 6, 10, 11] [5, 6, 10, 5, 4, 11, 0, 0, 4, 5]] This list is however not correct: Since atoms [1,2,3,7,8] have been deleted, the remaining atoms with indices larger than the deleted atoms must be decremented. I do this as follows: for i in a: b = numpy.where(b > i, bonds-1, bonds) (*) yielding the correct result b = [[0, 0, 1, 1, 2, 2, 2, 3, 5, 6], [2, 3, 5, 2, 1, 6, 0, 0, 1, 2]] The Python for loop in (*) may easily contain 50.000 iteration. Is there a smart way to utilize numpy functionality to avoid this? Thanks and best regards, Mads -- +---------------------------------------------------------+ | Mads Ipsen | +----------------------+----------------------------------+ | Gåsebæksvej 7, 4. tv | phone: +45-29716388 | | DK-2500 Valby | email: mads.ipsen(a)gmail.com | | Denmark | map : www.tinyurl.com/ns52fpa | +----------------------+----------------------------------+

5 4

De Bruijn sequence
by Vincent Davis Feb. 1, 2014

Feb. 1, 2014

I happen to be working with De Bruijn sequences. Is there any interest in this being part of numpy/scipy? https://gist.github.com/vincentdavis/8588879 Vincent Davis

3 3