Mailman 3 August 2013 - NumPy-Discussion

[ANN] numexpr 2.2 released
by Francesc Alted Aug. 31, 2013

Aug. 31, 2013

========================== Announcing Numexpr 2.2 ========================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's VML library (included in Intel MKL), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational kernel for projects that don't want to adopt other solutions that require more heavy dependencies. What's new ========== This release is mainly meant to fix a problem with the license the numexpr/win32/pthread.{c,h} files emulating pthreads on Windows. After persmission from the original authors is granted, these files adopt the MIT license and can be redistributed without problems. See issue #109 for details (https://code.google.com/p/numexpr/issues/detail?id=110). Another important improvement is the new algorithm to decide the initial number of threads to be used. This was necessary because by default, numexpr was using a number of threads equal to the detected number of cores, and this can be just too much for moder systems where this number can be too high (and counterporductive for performance in many cases). Now, the 'NUMEXPR_NUM_THREADS' environment variable is honored, and in case this is not present, a maximum number of *8* threads are setup initially. The new algorithm is fully described in the Users Guide, in the note of 'General routines' section: https://code.google.com/p/numexpr/wiki/UsersGuide#General_routines. Closes #110. In case you want to know more in detail what has changed in this version, see: http://code.google.com/p/numexpr/wiki/ReleaseNotes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at Google code in: http://code.google.com/p/numexpr/ You can get the packages from PyPI as well: http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

1 0

Structured array dtype
by Nicolas Rougier Aug. 31, 2013

Aug. 31, 2013

Hi, I'm a bit lost with the following example (numpy 1.7.1, osx 10.8): >>> Z = np.zeros(10, [('a', np.float32, 3), ('b', np.float32, 4)]) >>> Z['a'].dtype dtype('float32') >>> Z.dtype['a'] dtype(('<f4', (3,))) Does that mean that dtype['a'] is the dtype of field 'a' when in Z, while Z['a'].dtype is the dtype of field 'a' when "extracted" or my way of thinking is totally wrong ? What bothers me the most is that I cannot do: >>> Z['a'].view(Z.dtype['a']) ValueError: new type not compatible with array. Nicolas

2 2

_PyADt
by Charles R Harris Aug. 30, 2013

Aug. 30, 2013

Anyone know what _PyADt is? It turns up in ndarraytypes.h #define PyDataType_ISBOOL(obj) PyTypeNum_ISBOOL(_PyADt(obj)) and only there. It's not in the build directory, google yields nothing. I suspect it is an historical artifact turned bug and should be replaced by ((PyArray_Descr*)(obj))->type_num. Chuck

2 1

Relative speed
by Anubhab Baksi Aug. 30, 2013

Aug. 30, 2013

Hi, I need to know about the relative speed (i.e., which one is faster) of the followings: 1. list and numpy array, tuples and numpy array 2. list of tuples and numpy matrix (first one is rectangular) 3. random.randint() and numpy.random.random_integers() Thank you.

8 10

Matrix peculiarities
by Colin J. Williams Aug. 30, 2013

Aug. 30, 2013

1 0

Stick (line segments) percolation algorithm - graph theory?
by Josè Luis Mietta Aug. 29, 2013

Aug. 29, 2013

Hi experts! I wrote an algorithm for study stick percolation (i.e.: networks between line segments that intersect between them). In my algorithm N sticks (line segments) are created inside a rectanglar box of sides 'b' and 'h' and then, one by one, the algorithm explores the intersection between all line segments. This is a Monte Carlo simulation, so the 'experiment' is executed many times (no less than 100 times). Writen like that, very much RAM is consumed: array_x1=uniform.rvs(loc=-b/2,scale=b,size=N) array_y1=uniform.rvs(loc=-h/2,scale=h,size=N) array_x2=uniform.rvs(loc=-b/2,scale=b,size=N) array_y2=uniform.rvs(loc=-h/2,scale=h,size=N) M =np.zeros([N,N]) foru inxrange(100): ---->This'100'isthe number of experiments. forj inxrange(N): ifj>0: x_A1B1 =array_x2[j]-array_x1[j] y_A1B1 =array_y2[j]-array_y1[j] x_A1A2 =array_x1[0:j]-array_x1[j] y_A1A2 =array_y1[0:j]-array_y1[j] x_A2A1 =-1*x_A1A2 y_A2A1 =-1*y_A1A2 x_A2B2 =array_x2[0:j]-array_x1[0:j] y_A2B2 =array_y2[0:j]-array_y1[0:j] x_A1B2 =array_x2[0:j]-array_x1[j] y_A1B2 =array_y2[0:j]-array_y1[j] x_A2B1 =array_x2[j]-array_x1[0:j] y_A2B1 =array_y2[j]-array_y1[0:j] p1 =x_A1B1*y_A1A2 -y_A1B1*x_A1A2 p2 =x_A1B1*y_A1B2 -y_A1B1*x_A1B2 p3 =x_A2B2*y_A2B1 -y_A2B2*x_A2B1 p4 =x_A2B2*y_A2A1 -y_A2B2*x_A2A1 condition_1=p1*p2 condition_2=p3*p4 fork inxrange (j): ifcondicion_1[k]<=0andcondicion_2[k]<=0: M[j,k]=1 ifj+1<N+4: x_A1B1 =array_x2[j]-array_x1[j] y_A1B1 =array_y2[j]-array_y1[j] x_A1A2 =array_x1[j+1:]-array_x1[j] y_A1A2 =array_y1[j+1:]-array_y1[j] x_A2A1 =-1*x_A1A2 y_A2A1 =-1*y_A1A2 x_A2B2 =array_x2[j+1:]-array_x1[j+1:] y_A2B2 =array_y2[j+1:]-array_y1[j+1:] x_A1B2 =array_x2[j+1:]-array_x1[j] y_A1B2 =array_y2[j+1:]-array_y1[j] x_A2B1 =array_x2[j]-array_x1[j+1:] y_A2B1 =array_y2[j]-array_y1[j+1:] p1 =x_A1B1*y_A1A2 -y_A1B1*x_A1A2 p2 =x_A1B1*y_A1B2 -y_A1B1*x_A1B2 p3 =x_A2B2*y_A2B1 -y_A2B2*x_A2B1 p4 =x_A2B2*y_A2A1 -y_A2B2*x_A2A1 condicion_1=p1*p2 condicion_2=p3*p4 fork inxrange (N-j-1): ifcondicion_1[k]<=0andcondicion_2[k]<=0: M[j,k+j+1]=1 Here, the element Mij=1 if stick i intersect stick j and Mij=0 if not. How can i optimize my algorithm? Graph theory is usefull in this case? Waiting for your answers. Thanks a lot! Best regards

2 2

Use numpy.distutils to build fortran with BLAS
by Matt Hoffman Aug. 29, 2013

Aug. 29, 2013

I have some fortran code that I would like to build and make accessible from python which needs to call a few BLAS routines. Building it in a one-off manner is not a problem, but I can't seem to find a good way to distribute it in a way that just works. So really I'm wondering if there is a "correct" way use numpy.distutils in order to link against the same BLAS libraries used by numpy. I'll paste my current approach below, but I'm not sure that this is the best/most-portable way to do it. (And it certainly breaks on MacOS due to the weird way that macs use rpath.) Anyway my setup contains something like: from numpy.distutils.core import setup from numpy.distutils.system_info import get_info, NotFoundError from numpy.distutils.misc_util import Configuration sources = ['foo.f'] extra_info = {} extra_link_args = [] try: extra_info = get_info('mkl', 2) extra_link_args = ['-Wl,-rpath,'+path for path in extra_info['library_dirs']], except NotFoundError: try: extra_info = get_info('blas', 2) extra_link_args = ['-Wl,-rpath,'+path for path in extra_info['library_dirs']], except NotFoundError: # couldn't find anything. just fall back to building the functions we need ourselves. sources += ['ddot.f'] config = Configuration() config.add_extension( sources=sources, extra_info=extra_info, extra_link_args=extra_link_args ) setup(**config.todict()) Thanks! -matt

2 1

Array addition inconsistency
by Martin Luethi Aug. 29, 2013

Aug. 29, 2013

Dear all, After some surprise, I noticed an inconsistency while adding array slices: > a = np.arange(5) > a[1:] = a[1:] + a[:-1] > a array([0, 1, 3, 5, 7]) versus inplace > a = np.arange(5) > a[1:] += a[:-1] > a array([ 0, 1, 3, 6, 10]) My suspicition is that the second variant does not create intermediate storage, and thus works on the intermediate result, effectively performing a.cumsum(). This behaviour is certainly surprising, and leads to unexpected errors if used without testing. Best, Martin -- Dr. Martin Lüthi luethi(a)vaw.baug.ethz.ch VAW Glaciology http://www.vaw.ethz.ch/gz ETH Zürich http://people.ee.ethz.ch/~luethim 8093 Zürich Tel: +41 44 632 40 93

4 3

lots of warnings with python3
by Neal Becker Aug. 28, 2013

Aug. 28, 2013

I tried running python2 -3 on some code, and found numpy produces a lot of warnings. Many like: python -3 -c 'import numpy' ... /usr/lib64/python2.7/site-packages/numpy/lib/polynomial.py:928: DeprecationWarning: Overriding __eq__ blocks inheritance of __hash__ in 3.x But also: /usr/lib64/python2.7/site-packages/numpy/lib/shape_base.py:838: DeprecationWarning: classic int division n /= max(dim_in,1)

3 4

Right way to build Fortran 90 module using numpy.distutils
by Juan Luis Cano Aug. 27, 2013

Aug. 27, 2013

I'm having some problems to properly build a module written in Fortran 90 using numpy.distutils. It only contains one subroutine, but: * If I write a .f90 file with the single subroutine (no module .. contains), then f2py tries to compile it using a f77 compiler, and that's not what I want. * If I put it inside a module definition, then at the final compilation stage modulef2pywrapper.c is missing the corresponding .mod file and I have to make the build fail, then copy the .mod from the temp directory and rebuild. Weird. On the other hand, with the second approach f2py creates a unnecessary level of nesting in the corresponding python module and I don't feel quite ok with that. What is the right way to do this? I find the documentation on numpy.distutils a bit lacking, I almost always find the solution by trial and error. I can push the code, it's a short file (part of the rewrite of odeint I'm doing in SciPy). Regards Juan Luis Cano

2 4