Mailman 3 February 2011 - NumPy-Discussion

using loadtxt() for given number of rows?
by Robert Cimrman Feb. 14, 2011

Feb. 14, 2011

Hi, I work with text files which contain several arrays separated by a few lines of other information, for example: POINTS 4 float -5.000000e-01 -5.000000e-01 0.000000e+00 5.000000e-01 -5.000000e-01 0.000000e+00 5.000000e-01 5.000000e-01 0.000000e+00 -5.000000e-01 5.000000e-01 0.000000e+00 CELLS 2 8 3 0 1 2 3 2 3 0 (yes, that's the legacy VTK format, but take it just as an example) I have used custom Python code with loops to read similar files, so the speed was not too good. Now I wonder if it would be possible to use the numpy.loadtxt() function for the "array-like" parts. It supports passing an open file object in, which is good, but it wants to read the entire file, which does not work in this case. It seems to me, that an additional parameter to loadtxt(), say "nrows" or "numrows", would do the job, so that the function does not try reading the entire file. Another possibility would be to raise an exception as it is now, but also to return the data succesfully read so far. What do you think? Is this worth a ticket? r.

3 5

FYI: experimental waf support for numpy
by David Feb. 14, 2011

Feb. 14, 2011

Hi there, Following recent release of waf 1.6 and its adoption by the samba project, as well as my own work on integrating waf and bento, I have spent some time to build numpy with it. Although this is experimental, it should be possible for people to play with it: https://github.com/cournape/numpy/tree/waf_build You need a recent waf checkout (trunk), and you build numpy as follows: $waf_checkout/waf-light configure $waf_checkout/waf-light build Waf is python build system, a bit like scons, but smaller, cleaner, much faster and somehow easier to integrate as a library The added code in numpy is ~ 1200 lines of code, of which almost half is just to get missing configuration from waf, IOW, the whole numpy build is described in ~ 600 lines of code, and works on linux, mac os x and windows (Visual Studio only for now). Besides the maintenance advantage, the waf build has a few interesting features: - < 1 sec no-op build with dependencies tracking (header changes, file content change, etc...) - ~ 10 sec debug build on a 2 years old machine - very easy to customize with alternative blas/lapack libraries, or use alternative compilers like clang. - nice output and full log written when a configuration error has occured, for easier build debugging. Note that this cannot be used to install numpy, and is of interest only for developers ATM - the goal is to integrate this with bento to get a robust build and deployment framework for numpy, scipy and beyond. cheers, David

4 4

odd performance of sum?
by eat Feb. 13, 2011

Feb. 13, 2011

Hi, Observing following performance: In []: m= 1e5 In []: n= 1e2 In []: o= ones(n) In []: M= randn(m, n) In []: timeit M.sum(1) 10 loops, best of 3: 38.3 ms per loop In []: timeit dot(M, o) 10 loops, best of 3: 21.1 ms per loop In []: m= 1e2 In []: n= 1e5 In []: o= ones(n) In []: M= randn(m, n) In []: timeit M.sum(1) 100 loops, best of 3: 18.3 ms per loop In []: timeit dot(M, o) 10 loops, best of 3: 21.2 ms per loop One would expect sum to outperform dot with a clear marginal. Does there exixts any 'tricks' to increase the performance of sum? In []: sys.version Out[]: '2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)]' # installed binaries from http://python.org/ In []: np.version.version Out[]: '1.5.1' # installed binaries from http://scipy.org/ Regards, eat

8 22

mkl 10.3 linking - undefined symbol: i_free?
by Andrzej Giniewicz Feb. 13, 2011

Feb. 13, 2011

Hello, I'd like to ask if anyone got around the undefined symbol i_free issue? What I did was that I used link advisor from file:///home/giniu/Downloads/MKL_Linking_Adviser-1.03.htm and it told me to use -L$MKLPATH $MKLPATH/libmkl_solver_lp64.a -Wl,--start-group -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -Wl,--end-group -fopenmp -lpthread So I created site.cfg like: library_dirs = /opt/intel/composerxe-2011.2.137/mkl/lib/intel64 include_dirs = /opt/intel/composerxe-2011.2.137/mkl/include mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core lapack_libs = mkl_lapack95_lp64 and added -fopenmp to LDFLAGS. Numpy built, then I was able to install. I started python and imported numpy, and till this point it worked. When I tried to run numpy.test() and got: Running unit tests for numpy NumPy version 1.5.1 NumPy is installed in /usr/lib/python2.7/site-packages/numpy Python version 2.7.1 (r271:86832, Feb 7 2011, 19:39:54) [GCC 4.5.2 20110127 (prerelease)] nose version 1.0.0 . *** libmkl_def.so *** failed with error : /opt/intel/composerxe-2011.2.137/mkl/lib/intel64/libmkl_def.so: undefined symbol: i_free MKL FATAL ERROR: Cannot load libmkl_def.so Anyone who got this working could give me some hint about how to get around it? I would be grateful Cheers, Andrzej.

2 1

Installing numpy
by Jonathan Tu Feb. 13, 2011

Feb. 13, 2011

Hi, I need to reinstall numpy because the cluster I am using was recently overhauled. I am wondering if numpy works with Python 2.7 now. Also, I would like numpy to run as fast as possible. The last time I did this, I was advised to install ATLAS by hand, as the one that comes with RHEL is not suitable. The first time I tried this, I kept running into problems that I think were due to mismatched fortran compilers. Is there a good resource for how to do this? I am fairly new to Linux. Thanks, Jonathan Tu

5 8

in1d enhancement ticket
by Neil Feb. 12, 2011

Feb. 12, 2011

Hi, If someone with commit access has the chance, could they take a look at ticket 1603: http://projects.scipy.org/numpy/ticket/1603 and apply it if it looks ok? It speeds up in1d(a, b) a lot for the common use case where len(b) << len(a). Thanks, Neil

1 0

ANN: carray 0.4 released
by Francesc Alted Feb. 11, 2011

Feb. 11, 2011

Hi, A new release of my new toy project is out. Enjoy! Announcing carray 0.4 ===================== What's new ---------- The most prominent feature in 0.4 is the support of multidimensional carrays. Than means that, for example, you can do:: >>> a = ca.arange(6).reshape((2,3)) Now, you can access any element in any dimension:: >>> a[:] array([[0, 1, 2], [3, 4, 5]]) >>> a[1] array([3, 4, 5]) >>> a[1,::2] array([3, 5]) >>> a[1,1] 4 Also, all the iterators in carray have received a couple of new parameters that allows to `limit` or `skip` selected elements in queries. Finally, many performance improvements have been implemented (mainly related with efficient zero-detection code). This allows for improved query times when using iterators. See: https://github.com/FrancescAlted/carray/wiki/query-compress for an example on how fast the new iterators can do. For more detailed info, see the release notes in: https://github.com/FrancescAlted/carray/wiki/Release-0.4 What it is ---------- carray is a chunked container for numerical data. Chunking allows for efficient enlarging/shrinking of data container. In addition, it can also be compressed for reducing memory needs. The compression process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. carray comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Resources --------- Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://carray.pytables.org/download Manual: http://carray.pytables.org/docs/manual Home of Blosc compressor: http://blosc.pytables.org User's mail list: carray(a)googlegroups.com http://groups.google.com/group/carray -- Francesc Alted

1 0

Get the index of a comparison of two lists
by FRENK Andreas Feb. 11, 2011

Feb. 11, 2011

Hi, I need to create a construct that returns the index of entries of the first list, if values in the first and second list are equal. Take valA = [1,2,3,4,20,21,22,23,24] valB = [1,2,3,4, 5,21,22,23] The correct solution is: [0,1,2,3,5,6,7] A potential loop can be: takeList=[] for j,a in enumerate(valA): if a in valB: takeList.append(j) Please note, valA can have entries like [1,10000000,1000000001,.....], i.e. it can be very sparse. I also thought about using bincount, but due to the sparse nature the return values from bincount would allocate too much memory. Any idea how to do it fast using numpy? Thanks, Andreas

3 2

Rationale for simple_capsule_dtor to be static but non-inline ?
by David Feb. 10, 2011

Feb. 10, 2011

Hi, in npy3_compat.h, one function simple_capsule_dtor is defined as static but non-inline. AFAIK, there is no reason not to put inline (if supported by the compiler of course) for a static function defined in a header. Unless I hear someone justify it, I will change it, cheers, David

2 2

2d*3d matrix dot product
by Davide Lasagna Feb. 10, 2011

Feb. 10, 2011

Hi, I want to compute the following dot product: P = np.array( [[ p11, p12 ], [p21, p22]] ) C = np.array( [c1, c2] ) where c1 and c2 are m*m matrices, so that C.shape = (2,m,m) I want to compute: A = np.array([a1, a2]) where a1 and a2 are two matrices m*m, from the dot product of P and C. I would expect: a1 = p11*c1 + p12*c2 a2 = p21*c1 + p22*c2 The code should be general so that i can multiply any P and C with shapes: P.shape = (n, n) C.shape = (n, m, l) and with a result as: A.shape = (n, m, l) I had a look at np.dot? but i can't sort out how to transpose/reshape the C array. Any help is greatly appreciated. Ciao

3 3