Mailman 3 July 2014 - NumPy-Discussion

parallel distutils extensions build? use gcc -flto
by Julian Taylor 16 Jul '14

16 Jul '14

hi, I have been playing around a bit with gccs link time optimization feature and found that using it actually speeds up a from scratch build of numpy due to its ability to perform parallel optimization and linking. As a bonus you also should get faster binaries due to the better optimizations lto allows. As compiling with lto does require some possibly lesser know details I wanted to share it. Prerequesits are a working gcc toolchain of at least gcc-4.8 and binutils > 2.21, gcc 4.9 is better as its faster. First of all numpy checks the long double representation by compiling a file and looking at the binary, this won't work as the od -b reimplementation here does not understand lto objects, so on x86 we must short circuit that: --- a/numpy/core/setup_common.py +++ b/numpy/core/setup_common.py @@ -174,6 +174,7 @@ def check_long_double_representation(cmd): # We need to use _compile because we need the object filename src, object = cmd._compile(body, None, None, 'c') try: + return 'IEEE_DOUBLE_LE' type = long_double_representation(pyod(object)) return type finally: Next we build numpy as usual but override the compiler, linker and ar to add our custom flags. The setup.py call would look like this: CC='gcc -fno-fat-lto-objects -flto=4 -fuse-linker-plugin -O3' \ LDSHARED='gcc -fno-fat-lto-objects -flto=4 -fuse-linker-plugin -shared -O3' AR=gcc-ar \ python setup.py build_ext Some explanation: The ar override is needed as numpy builds a static library and ar needs to know about lto objects. gcc-ar does exactly that. -flto=4 the main flag tell gcc to perform link time optimizations using 4 parallel processes. -fno-fat-lto-objects tells gcc to only build lto objects, normally it builds both an lto object and a normal object for toolchain compatibilty. If our toolchain can handle lto objects this is just a waste of time and we skip it. (The flag is default in gcc-4.9 but not 4.8) -fuse-linker-plugin directs it to run its link time optimizer plugin in the linking step, the linker must support plugins, both bfd (> 2.21) and gold linker do so. This allows for more optimizations. -O3 has to be added to the linker too as thats where the optimization occurs. In general a problem with lto is that the compiler options of all steps much match the flags used for linking. If you are using c++ or gfortran you also have to override that to use lto (CXX and FF(?)) See https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html for a lot more details. For some numbers on my machine a from scratch numpy build with no caching takes 1min55s, with lto on 4 it only takes 55s. Pretty neat for a much more involved optimization process. Concerning the speed gain we get by this, I ran our benchmark suite with this build, there were no really significant gains which is somewhat expected as numpy is simple C code with most function bottlenecks already inlined. So conclusion: flto seems to work well with recent gccs and allows for faster builds using the limited distutils. While probably not useful for development where compiler caching (ccache) is of utmost importance it is still interesting for projects doing one shot uncached builds (travis like CI) and have huge objects (e.g. swig or cython) and don't want to change to proper parallel build systems like bento. PS: So far I know clang also supports lto but I never used it PPS: using NPY_SEPARATE_COMPILATION=0 crashes gcc-4.9, time for a bug report. Cheers, Julian

1 0

Rounding float to integer while minizing the difference between the two arrays?
by Chao YUE 16 Jul '14

16 Jul '14

Dear all, I have two arrays with both float type, let's say X and Y. I want to round the X to integers (intX) according to some decimal threshold, at the same time I want to limit the following difference as small: diff = np.sum(X*Y) - np.sum(intX*Y) I don't have to necessarily minimize the "diff" variable (If with this demand the computation time is too long). But I would like to limit the "diff" to, let's say ten percent within np.sum(X*Y). I have tried to write some functions, but I don't know where to start the opitimization. def convert_integer(x,threshold=0): """ This fucntion converts the float number x to integer according to the threshold. """ if abs(x-0) < 1e5: return 0 else: pdec,pint = math.modf(x) if pdec > threshold: return int(math.ceil(pint)+1) else: return int(math.ceil(pint)) def convert_arr(arr,threshold=0): out = arr.copy() for i,num in enumerate(arr): out[i] = convert_integer(num,threshold=threshold) return out In [147]: convert_arr(np.array([0.14,1.14,0.12]),0.13) Out[147]: array([1, 2, 0]) Now my problem is, how can I minimize or limit the following? diff = np.sum(X*Y) - np.sum(convert_arr(X,threshold=?)*Y) Because it's the first time I encounter such kind of question, so please give me some clue to start :p Thanks a lot in advance. Best, Chao -- please visit: http://www.globalcarbonatlas.org/ *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

1 2

Bug in np.cross for 2D vectors
by Neil Hodgson 15 Jul '14

15 Jul '14

Hi, We came across this bug while using np.cross on 3D arrays of 2D vectors. The first example shows the problem and we looked at the source for np.cross and believe we found the bug - an unnecessary swapaxes when returning the output (comment inserted in the code). Thanks Neil # Example shape = (3,5,7,2) # These are effectively 3D arrays (3*5*7) of 2D vectors data1 = np.random.randn(*shape) data2 = np.random.randn(*shape) # The cross product of data1 and data2 should produce a (3*5*7) array of scalars cross_product_longhand = data1[:,:,:,0]*data2[:,:,:,1]-data1[:,:,:,1]*data2[:,:,:,0] print 'longhand output shape:',cross_product_longhand.shape # and it does cross_product_numpy = np.cross(data1,data2) print 'numpy output shape:',cross_product_numpy.shape # It seems to have transposed the last 2 dimensions if (cross_product_longhand == np.transpose(cross_product_numpy, (0,2,1))).all(): print 'Unexpected transposition in numpy.cross (numpy version %s)'%np.__version__ # np.cross L1464if axis is not None: axisa, axisb, axisc=(axis,)*3 a = asarray(a).swapaxes(axisa, 0) b = asarray(b).swapaxes(axisb, 0) msg = "incompatible dimensions for cross product\n"\ "(dimension must be 2 or 3)" if (a.shape[0] not in [2, 3]) or (b.shape[0] not in [2, 3]): raise ValueError(msg) if a.shape[0] == 2: if (b.shape[0] == 2): cp = a[0]*b[1] - a[1]*b[0] if cp.ndim == 0: return cp else: ## WE SHOULD NOT SWAPAXES HERE! ## For 2D vectors the first axis has been ## collapsed during the cross product return cp.swapaxes(0, axisc)

3 2

plt.show() and plt.draw() doesnt work
by Josè Luis Mietta 14 Jul '14

14 Jul '14

Hi experts! I have a numpy array M. I generate a graph using NetworkX and then I want to draw this graph: import networkx as nx import matplotlib.pyplot as plt G=nx.graph(M) nx.draw(G) plt.draw() Doing this, no picture appears. In addition, if I do `plt.show()` no picture appears. Please help! Best regards

3 2

np.zeros of structured array of array of objects
by Julian Taylor 11 Jul '14

11 Jul '14

Hi, looking at https://github.com/numpy/numpy/issues/4857 I noticed that np.zeros of a structured array of array of objects only initializes the first element of if the embedded array to zero and leaves the rest None: In [1]: a = numpy.zeros(10, dtype=[('multiple objects', object, 2)]); a Out[1]: array([([0, None],), ([0, None],), ([0, None],), ([0, None],), ([0, None],), ([0, None],), ([0, None],), ([0, None],), ([0, None],), ([0, None],)], dtype=[('multiple objects', 'O', (2,))]) Is this the intented behavior? I would have expected all fields to be set to an int-object 0. If not can we change it or is it too likely people rely on this behavior?

2 1

ANN: pandas 0.14.1 released
by Jeff Reback 11 Jul '14

11 Jul '14

Hello, We are proud to announce v0.14.1 of pandas, a minor release from 0.14.0. This release includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. This was 1.5 months of work with 244 commits by 45 authors encompassing 306 issues. We recommend that all users upgrade to this version. *Highlights:* - New method select_dtypes() <http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sele…> to select columns based on the dtype - New method sem() <http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.sem.htm…> to calculate the standard error of the mean. - Support for dateutil timezones (see *docs* <http://pandas.pydata.org/pandas-docs/stable/timeseries.html#timeseries-time…> ). - Support for ignoring full line comments in the read_csv() <http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html#…>text parser. - New documentation section on *Options and Settings* <http://pandas.pydata.org/pandas-docs/stable/options.html#options>. - Lots of bug fixes For a more a full description of Whatsnew for v0.14.1 here: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html *What is it:* *pandas* is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. Documentation: http://pandas.pydata.org/pandas-docs/stable/ Source tarballs, windows binaries are available on PyPI: https://pypi.python.org/pypi/pandas windows binaries are courtesy of Christoph Gohlke and are built on Numpy 1.8 macosx wheels will be available soon, courtesy of Matthew Brett Please report any issues here: https://github.com/pydata/pandas/issues Thanks The Pandas Development Team Contributors to the 0.14.1 release - Andrew Rosenfeld - Andy Hayden - Benjamin Adams - Benjamin M. Gross - Brian Quistorff - Brian Wignall - bwignall - clham - Daniel Waeber - David Bew - David Stephens - DSM - dsm054 - helger - immerrr - Jacob Schaer - jaimefrio - Jan Schulz - John David Reaver - John W. O’Brien - Joris Van den Bossche - jreback - Julien Danjou - Kevin Sheppard - K.-Michael Aye - Kyle Meyer - lexual - Matthew Brett - Matt Wittmann - Michael Mueller - Mortada Mehyar - onesandzeroes - Phillip Cloud - Rob Levy - rockg - sanguineturtle - Schaer, Jacob C - seth-p - sinhrks - Stephan Hoyer - Thomas Kluyver - Todd Jennings - TomAugspurger - unknown - yelite

1 0

The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows)
by Nathaniel Smith 11 Jul '14

11 Jul '14

On Fri, Apr 11, 2014 at 12:21 PM, Carl Kleffner <cmkleffner(a)gmail.com> wrote: > a discussion about OpenBLAS on the octave maintainer list: > http://article.gmane.org/gmane.comp.gnu.octave.maintainers/38746 I'm getting the impression that OpenBLAS is being both a tantalizing opportunity and a practical thorn-in-the-side for everyone -- Python, Octave, Julia, R. How crazy would it be to get together an organized effort to fix this problem -- "OpenBLAS for everyone"? E.g., by collecting patches to fix the bits we don't like (like unhelpful build system defaults), applying more systematic QA, etc. Ideally this could be done upstream, but if upstream is MIA or disagrees about OpenBLAS's goals, then it could be maintained as a kind of "OpenBLAS++" that merges regularly from upstream (compare to [1][2][3] for successful projects handled in this way). If hardware for testing is a problem, then I suspect NumFOCUS would be overjoyed to throw a few kilodollars at buying one instance of each widely-distributed microarchitecture released in the last few years as a test farm... I think the goal is pretty clear: a modern optionally-multithreaded BLAS under a BSD-like license with a priority on correctness, out-of-the-box functionality (like runtime configurability and feature detection), speed, and portability, in that order. I unfortunately don't have the skills to actually lead such an effort (I've never written a line of asm in my life...), but surely our collective communities have people who do? -n [1] http://www.openssh.com/portable.html [2] http://www.eglibc.org/mission (a "friendly fork" of glibc holding stuff that Ulrich Drepper got cranky about, which eventually was merged back) [3] https://en.wikipedia.org/wiki/Go-oo -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

10 37

numpy.partition and the ICC compiler
by Julian Taylor 10 Jul '14

10 Jul '14

hi, there seems to be some issue with the newish selection code when compiling numpy with the ICC compiler. See this issue: https://github.com/numpy/numpy/issues/4836 I cannot reproduce the problem even when compiling with ICC myself. I have also tried valgrind and GCC's undefined behavior sanitizer without any results. Can somebody with debugging experience please try the posted testcase and if it is reproduce-able provide the information needed to fix this. It should also affect 1.8.0 if you replace the percentile call with np.partition(imc[i], (0, 1, 1027603, 1027604)) I would need a backtrace, the current register state, the local variables, disassembly and ideally the few steps until the crash/wrong result with variable state. Cheers, Julian

1 0

Number of elements in a intersection graph
by Josè Luis Mietta 10 Jul '14

10 Jul '14

Hi experts!! I am studying the intersection between line segments (sticks). I have an Numpy array (M) corresponding to the intersection graph of the system (the element Mij = 1 if the sticks' i 'and' 'j' intersect, and Mij = 0 if not intersect). I want to determine the number of elements that form the path that connects two sticks (N and K), i.e.: the number of sticks that form the spanning cluster between stick N and K. How I can do?? Please explain step by step. Best regards! Thanks a lot. José Luis

3 2

ANN: Bokeh 0.5 released
by Bryan Van de Ven 09 Jul '14

09 Jul '14

I am very happy to announce the release of Bokeh version 0.5! (http://continuum.io/blog/bokeh-0.5) Bokeh is a Python library for visualizing large and realtime datasets on the web. This release includes many new features: weekly dev releases, a new plot frame, a click tool, "always on" hover tool, multiple axes, log axes, minor ticks, gears and gauges glyphs, and an NPM BokehJS package. Several usability enhancements have been made to the plotting.py interface to make it even easier to use. The Bokeh tutorial also now includes exercises in IPython notebook form. Of course, we've made many little bug fixes - see the CHANGELOG for full details. The biggest news is all the long-term and architectural goals landing in Bokeh 0.5: * Widgets! Build apps and dashboards with Bokeh * Very high level bokeh.charts interface * Initial Abstract Rendering support for big data visualizations * Tighter Pandas integration * Simpler, easier plot embedding options Expect dynamic, data-driven layouts, including ggplot style auto-faceting in upcoming releases, as well as R language bindings, more statistical plot types in bokeh.charts, and cloud hosting for Bokeh apps. Check out the full documentation, interactive gallery, and tutorial at http://bokeh.pydata.org as well as the new Bokeh IPython notebook nbviewer index (including all the tutorials) at: http://nbviewer.ipython.org/github/ContinuumIO/bokeh-notebooks/blob/master/… If you are using Anaconda, you can install with conda: conda install bokeh Alternatively, you can install with pip: pip install bokeh BokehJS is also available by CDN for use in standalone javascript applications: http://cdn.pydata.org/bokeh-0.5.min.js http://cdn.pydata.org/bokeh-0.5.min.css Issues, enhancement requests, and pull requests can be made on the Bokeh Github page: https://github.com/continuumio/bokeh Questions can be directed to the Bokeh mailing list: bokeh(a)continuum.io If you have interest in helping to develop Bokeh, please get involved! Special thanks to recent contributors: Tabish Chasmawala, Samuel Colvin, Christina Doig, Tarun Gaba, Maggie Mari, Amy Troschinetz, Ben Zaitlen. Bryan Van de Ven Continuum Analytics http://continuum.io

1 0