Mailman 3 November 2010 - NumPy-Discussion

about SIMD (SSE2 & SSE3)
by qihua wu 07 Nov '10

07 Nov '10

I used the following command to install the numpy to enable the SSE3 numpy-1.5.1rc1-win32-superpack-python3.1.exe /arch sse3 Then how can I know whether numpy is running with SSE or not? I have a program to process the data from sql server using java to process 600M rows, it takes 7 hours to complete, about 4 hours is eating the cpu. I am wondering whether I can port the java to numpy to cut the 4 hours to 2hours or even less by enabling the SSE3. Any comment?

4 5

Optimize Floyd-Wallshall algorithm with Numpy
by K. Sun 06 Nov '10

06 Nov '10

Hello, I wrote the following code with numpy to implement the Floyd-Wallshall algorithm to compute the pair-wise shortest path in a undirected weighted graph. It is really slow when N ~ 10k, while the same implementation in matlab is much faster. I am sorry I don't want to run it again to present some accurate comparison. Is there any suggestions to optimize this code without destroying the elegant coding style of python? Thank you very much. def floyd( dis ): ''' dis is the pair-wise distance matrix. return and update dis as the shortest path matrix w_{ij}.''' N = dis.shape[0] for k in range( N ): route = np.kron( np.ones( (N, 1) ), dis[k, :] ) dis = np.minimum( dis, route + route.T ) return dis

3 4

scipy.linalg.solve()'s overwrite option does not work
by braingateway 06 Nov '10

06 Nov '10

Hi everyone, I believe the overwrite option is used for reduce memory usage. But I did following test, and find out it does not work at all. Maybe I misunderstood the purpose of overwrite option. If anybody could explain this, I shall highly appreciate your help. >>> a=npy.random.randn(20,20) >>> x=npy.random.randn(20,4) >>> a=npy.matrix(a) >>> x=npy.matrix(x) >>> b=a*x >>> import scipy.linalg as sla >>> a0=npy.matrix(a) >>> a is a0 False >>> b0=npy.matrix(b) >>> b is b0 False >>> X=sla.solve(a,b,overwrite_b=True,debug=True) solve:overwrite_a= False solve:overwrite_b= True >>> X is b False >>> (X==b).all() False >>> (b0==b).all() True >>> sla.solve(a,b,overwrite_a=True,overwrite_b=True,debug=True) solve:overwrite_a= True solve:overwrite_b= True >>> (a0==a).all() True >>> help(sla.solve) Help on function solve in module scipy.linalg.basic: solve(a, b, sym_pos=False, lower=False, overwrite_a=False, overwrite_b=False, debug=False) Solve the equation a x = b for x Parameters ---------- a : array, shape (M, M) b : array, shape (M,) or (M, N) sym_pos : boolean Assume a is symmetric and positive definite lower : boolean Use only data contained in the lower triangle of a, if sym_pos is true. Default is to use upper triangle. overwrite_a : boolean Allow overwriting data in a (may enhance performance) overwrite_b : boolean Allow overwriting data in b (may enhance performance) Returns -------

1 0

ANN: PyTables 2.2.1 released
by Francesc Alted 05 Nov '10

05 Nov '10

=========================== Announcing PyTables 2.2.1 =========================== This is maintenance release. The upgrade is recommended for all that are running PyTables in production environments. What's new ========== Many fixes have been included, as well as a fair bunch of performance improvements. Also, the Blosc compression library has been updated to 1.1.2, in order to prevent locks in some scenarios. Finally, the new evaluation version of PyTables Pro is based on the previous Pro 2.2. In case you want to know more in detail what has changed in this version, have a look at: http://www.pytables.org/moin/ReleaseNotes/Release_2.2.1 You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://www.pytables.org/download/stable For an on-line version of the manual, visit: http://www.pytables.org/docs/manual-2.2.1 What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team -- Francesc Alted

1 0

problem with a binary file on OS X 10.6
by Olli Sipilä 05 Nov '10

05 Nov '10

Hello, I have a problem reading a binary file on OS X 10.6. On my desktop mac (OS X 10.4.11, Python 2.5.4, Numpy 1.3.0), the command CELLS, NSIZE, NE = fromfile(fp, int32, 3), where fp is the filename, correctly prints "21 20 300". However, when I try the above on my laptop using Snow Leopard (Python 2.6.6, Numpy 1.5.0), I get the numbers "352321536 335544320 738263040". This results in a failure when trying to read the data that comes afterwards (Numpy fails with "array too big"). I assume this error may have something to do with 10.6 being 64-bit and 10.4 being 32-bit; however, the Python 2.6.6. distribution is also 32-bit. Any thoughts on this? Thanks,Olli Sipilä

5 5

create an object array with named dtype
by Skipper Seabold 04 Nov '10

04 Nov '10

I just ran into this and am kind of baffled. There are other ways I could do this so it's not a huge deal, but I'm wondering if this is a bug. I want a (named) structured array with an object dtype. Is this possible? For example In [68]: import numpy as np In [69]: A = np.array((np.arange(10)[:,None], 3, 4.5), dtype=[("prob", float, (1 0,1)), ....: ("m1", float),("m2", float)]) In [70]: B = np.array((np.arange(10,20)[:,None], 3, 4.5), dtype=[("prob", float, (10,1)), ....: ("m1", float),("m2", float)]) This works fine. In [71]: C = np.empty(2, dtype=object) In [72]: C[0] = A In [73]: C[1] = B In [74]: np.all(C[0] == A) Out[74]: True In [75]: np.all(C[1] == B) Out[75]: True This doesn't. Looks like it takes a bad view on the array part (?) In [76]: D = np.empty(2, dtype=[("n1",object),("n2", object)]) In [77]: D["n1"] = A In [78]: D["n2"] = B In [79]: np.all(D["n1"] == A) Out[79]: False In [80]: np.all(D["n2"] == B) Out[80]: False Skipper

2 1

ANN: python-blosc 1.0.2
by Francesc Alted 04 Nov '10

04 Nov '10

==================================================== Announcing python-blosc 1.0.2 A Python wrapper for the Blosc compression library ==================================================== What is it? =========== Blosc (http://blosc.pytables.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc is a Python package that wraps it. What is new? ============ Updated to Blosc 1.1.2. Fixes some bugs when dealing with very small buffers (typically smaller than specified typesizes). Closes #1. Basic Usage =========== [Using IPython shell and a 2-core machine below] # Create a binary string made of int (32-bit) elements >>> import array >>> a = array.array('i', range(10*1000*1000)) >>> bytes_array = a.tostring() # Compress it >>> import blosc >>> bpacked = blosc.compress(bytes_array, typesize=a.itemsize) >>> len(bytes_array) / len(bpacked) 110 # 110x compression ratio. Not bad! # Compression speed? >>> timeit blosc.compress(bytes_array, typesize=a.itemsize) 100 loops, best of 3: 12.8 ms per loop >>> len(bytes_array) / 0.0128 / (1024*1024*1024) 2.9103830456733704 # wow, compressing at ~ 3 GB/s, that's fast! # Decompress it >>> bytes_array2 = blosc.decompress(bpacked) # Check whether our data have had a good trip >>> bytes_array == bytes_array2 True # yup, it seems so # Decompression speed? >>> timeit blosc.decompress(bpacked) 10 loops, best of 3: 21.3 ms per loop >>> len(bytes_array) / 0.0213 / (1024*1024*1024) 1.7489625814375185 # decompressing at ~ 1.7 GB/s is pretty good too! More examples showing other features (and using NumPy arrays) are available on the python-blosc wiki page: http://github.com/FrancescAlted/python-blosc/wiki Documentation ============= Please refer to docstrings. Start by the main package: >>> import blosc >>> help(blosc) and ask for more docstrings in the referenced functions. Download sources ================ Go to: http://github.com/FrancescAlted/python-blosc and download the most recent release from here. Blosc is distributed using the MIT license, see LICENSES/BLOSC.txt for details. Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc ---- **Enjoy data!** -- Francesc Alted

1 0

flattened index for Sparse Matrix?
by braingateway 04 Nov '10

04 Nov '10

Hi Everyone, I am trying sparse matrix these days. I am wondering is there any way I can access the sparse matrix with flattened index? For example: a=numpy.matrix([[0,1,2],[3,4,5]) matrix([[0, 1, 2], [3, 4, 5]]) >>> >>>print a.flat[3] >>> 3 >>> >>> a.flat[3]=10 >>> >>> print a >>> [[ 0 1 2] [10 4 5]] How could I do the similar indexing for sparse matrix? Thanks ahead, LittleBigBrain

1 0

please test/review: Scipy on OS X with Python 2.7
by Ralf Gommers 04 Nov '10

04 Nov '10

Hi, If you had an issue recently trying to compile scipy on OS X, can you please try to install numpy from http://github.com/rgommers/numpy/commits/farchs and then compile scipy? A quick review from a numpy.distutils expert would also be very welcome. Related (long) discussion at http://projects.scipy.org/numpy/ticket/1399. Thanks, Ralf

3 7

Error in API docs?
by Henry Gomersall 04 Nov '10

04 Nov '10

There is an inconsistency in the documentation for NPY_INOUT_ARRAY. cf. http://docs.scipy.org/doc/numpy/user/c-info.how-to-extend.html#NPY_INOUT_AR… http://docs.scipy.org/doc/numpy/reference/c-api.array.html#NPY_INOUT_ARRAY The first link includes the flag NPY_UPDATEIFCOPY. Checking the code seems to confirm that the correct version is that in the first link, and also that NPY_OUT_ARRAY is wrong in the API docs. I haven't checked NPY_INOUT_FARRAY or NPY_OUT_FARRAY. Cheers, Henry

2 2

NumPy-Discussion November 2010