Mailman 3 February 2011 - NumPy-Discussion

Largest possible numpy array
by Jaidev Deshpande 26 Feb '11

26 Feb '11

Hi How can I know the size of the largest possible 2-D array in numpy, given a specific 'dtype' and my system memory? How can one play around with this? Would it help to note that the array might be say *m* megabytes on disk, say, *n* % sparse? Also, is there some good literature about the largest possible 2-D arrays on different computing platforms? Google isn't helping much. Thanks

2 1

How recreated a integer list with a generator?
by Mario Moura 26 Feb '11

26 Feb '11

Hi Folks How recreated a integer list with generator? import random tmp = [] for x in range(0,600): tmp.append(random.randint(0,5000)) tmp = list(set(tmp)) tmp.sort() Should be possible recreated this list using same magic math formula or numpy array? Can I create a generator? How? What is my target? I need store this list im my database, the list itself get big size (resources) so I want save resources and save the generator object with pickle. Is it possible? What I have: size = len(tmp) min = tmp[1] max = tmp[-1] All numbers are integer and it is a random serie so each list will have your own generator. Again tmp is a static list and must be recreated with a generator the target is pickle the generator to be smaller than the list itself. Best Regards Mario

1 0

Porting numpy to Python3
by Algis Kabaila 25 Feb '11

25 Feb '11

Are there plans to port numpy to Python3? In particular, when will the packages of Linear Algebra (viz matrix inversion) be available in Python 3 compatible modules. Because of the importance of numpy in many scientific endeavours is so great, information of the availability in Python 3 mode is very important and will be greatly appreciated. OldAl. -- Algis http://akabaila.pcug.org.au/StructuralAnalysis.pdf

4 9

Memory error with numpy.loadtxt()
by Jaidev Deshpande 25 Feb '11

25 Feb '11

Hi Is it possible to load a text file 664 MB large with integer values and about 98% sparse? numpy.loadtxt() shows a memory error. If it's not possible, what alternatives could I have? The usable RAM on my machine running Windows 7 is 3.24 GB. Thanks.

3 2

When memory access is a bottleneck
by Keith Goodman 25 Feb '11

25 Feb '11

A topic that often comes up on the list is that arr.sum(axis=-1) is faster than arr.sum(axis=0). For C ordered arrays, moving along the last axis moves the smallest amount in memory. And moving small amounts in memory keeps the data in cache longer. Can I use that fact to speed up calculations along axis=0? As a test I used the nanmean function from the Bottleneck package. I created a second version of the function that calculates the nanmean of two columns at a time (code below). I figured that would be a more efficient use of cache. It is faster for many array sizes but not for all. Is there anything I can do about that? Is the speed up really due to cache? The smallest arrays I tried should fit entirely in cache, so perhaps unrolling the loop is helping the compiler be more efficient? Yeah, so obviously, I don't understand what is going on. Faster: >> a = np.random.rand(10,10) >> timeit nanmean_2d_float64_axis0(a) 1000000 loops, best of 3: 1.24 us per loop >> timeit nanmean_2d_float64_axis0_double(a) 1000000 loops, best of 3: 1.17 us per loop >> a = np.random.rand(100,100) >> timeit nanmean_2d_float64_axis0(a) 100000 loops, best of 3: 16 us per loop >> timeit nanmean_2d_float64_axis0_double(a) 100000 loops, best of 3: 15.7 us per loop >> a = np.random.rand(1000,1000) >> timeit nanmean_2d_float64_axis0(a) 100 loops, best of 3: 8.57 ms per loop >> timeit nanmean_2d_float64_axis0_double(a) 100 loops, best of 3: 5.02 ms per loop >> a = np.random.rand(10000,100) >> timeit nanmean_2d_float64_axis0(a) 100 loops, best of 3: 3.52 ms per loop >> timeit nanmean_2d_float64_axis0_double(a) 100 loops, best of 3: 3.35 ms per loop Slower: >> a = np.random.rand(100,10000) >> timeit nanmean_2d_float64_axis0(a) 100 loops, best of 3: 2.22 ms per loop >> timeit nanmean_2d_float64_axis0_double(a) 100 loops, best of 3: 2.57 ms per loop Code (Bottleneck simplified BSD license): @cython.boundscheck(False) @cython.wraparound(False) def nanmean_2d_float64_axis0(np.ndarray[np.float64_t, ndim=2] a): "Mean of 2d array with dtype=float64 along axis=0 ignoring NaNs." cdef int count = 0 cdef np.float64_t asum = 0, ai cdef Py_ssize_t i0, i1 cdef np.npy_intp *dim dim = PyArray_DIMS(a) cdef int n0 = dim[0] cdef int n1 = dim[1] cdef np.npy_intp *dims = [n1] cdef np.ndarray[np.float64_t, ndim=1] y = PyArray_EMPTY(1, dims, NPY_float64, 0) for i1 in range(n1): asum = 0 count = 0 for i0 in range(n0): ai = a[i0, i1] if ai == ai: asum += ai count += 1 if count > 0: y[i1] = asum / count else: y[i1] = NAN return y @cython.boundscheck(False) @cython.wraparound(False) def nanmean_2d_float64_axis0_double(np.ndarray[np.float64_t, ndim=2] a): "Mean of 2d array with dtype=float64 along axis=0 ignoring NaNs." cdef int count = 0, count2 = 0 cdef np.float64_t asum = 0, asum2 = 0, ai cdef Py_ssize_t i0, i1, i11 cdef np.npy_intp *dim dim = PyArray_DIMS(a) cdef int n0 = dim[0] cdef int n1 = dim[1] cdef np.npy_intp *dims = [n1] cdef np.ndarray[np.float64_t, ndim=1] y = PyArray_EMPTY(1, dims, NPY_float64, 0) for i1 in range(0,n1,2): asum = 0 count = 0 asum2 = 0 count2 = 0 i11 = i1 + 1 for i0 in range(n0): ai = a[i0, i1] if ai == ai: asum += ai count += 1 ai = a[i0, i11] if ai == ai: asum2 += ai count2 += 1 if count > 0: y[i1] = asum / count else: y[i1] = NAN if count2 > 0: y[i11] = asum / count else: y[i11] = NAN return y

1 0

Problems with numpy
by Mustapha BOUIKHIF 25 Feb '11

25 Feb '11

Hello all, I am having problems with numpy (version 1.5.1) when executing this command as root, evry thing is OK: python -Wd -c 'import numpy; numpy.test()' Running unit tests for numpy NumPy version 1.5.1 NumPy is installed in /usr/local/lib64/python2.6/site-packages/numpy Python version 2.6 (r26:66714, Nov 9 2010, 01:31:57) [GCC 4.3.4 [gcc-4_3-branch revision 152973]] nose version 1.0.0 ...................... Ran 3006 tests in 14.088s OK (KNOWNFAIL=4) But as a normal user (non root), the same command show errors: python -Wd -c 'import numpy; numpy.test()' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib64/python2.6/site-packages/numpy/__init__.py", line 127, in <module> raise ImportError(msg) ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python intepreter from there. Python 2.6 Linux <machine> 2.6.32.27-0.2-default x86_64 x86_64 x86_64 GNU/Linux modules are in /usr/local/lib64/python2.6/site-packages/ Any help will be appreciated. -- Cordialement Mustapha Bouikhif CNRS - IDRIS Tel: +33 1 69 35 85 27 Fax: +33 1 69 35 37 75 E-mail: mustapha.bouikhif(a)idris.fr http://www.idris.fr

3 6

NumPy speed tests by NASA
by Sturla Molden 24 Feb '11

24 Feb '11

I came accross some NumPy performance tests by NASA. Comparisons against pure Python, Matlab, gfortran, Intel Fortran, Intel Fortran with MKL, and Java. For those that are interested, it is here: https://modelingguru.nasa.gov/docs/DOC-1762 Sturla

11 15

blank values '--'
by Bryan Woods 24 Feb '11

24 Feb '11

Hello, I am loading in data from the GRIB2 files using PyNIO. Once I load them into numpy arrays I am left with a halo of values near the array boundaries that appear to be empty. When I print them or use them in conditional statements I see values of '--'. I'd like to turn these values into zeroes at the very least. As an example if I print the array I see: [[-- -- -- ..., -- -- --] [-- -- -- ..., -- -- --] [-- -- -- ..., -- -- --] ..., [-- -- -- ..., -- -- --] [-- -- -- ..., -- -- --] [-- -- -- ..., -- -- --]] though I know that there are valid values in the interior of the domain which plot properly. I can't find anything in the numpy documentation that addresses this type of missing value. Has anyone dealt with this before? Thanks, Bryan

4 3

ANN: inSCIght, The Scientific Computing Podcast
by Anthony Scopatz 24 Feb '11

24 Feb '11

Hello All, I am very pleased to announce inSCIght, a new scientific computing podcast (press release below). I apologize for those of you in the intersection of these lists that may receive this message multiple times. As I mention in the press release, we are very open to your contributions! Be Well Anthony inSCIght: The Scientific Computing Podcast ========================================== 'inSCIght' is a podcast that focuses on scientific computing in all of its various forms. Every week we have a few panelists engage head-to-head on poignant and interesting topics. The panelists are drawn from all across the scientific computing community. From embedded systems experts to very high level language gurus, biologists and nuclear engineers, the hosts of inSCIght use computers to solve science and engineering problems everyday. This podcast throws people, ideas, and opinions into an audio-blender hoping to educate and entice each other and the world. You can find us at: * inSCIght.org (http://inscight.org/), * Twitter (http://twitter.com/inscight/), * Convore (https://convore.com/inscight/), * and GitHub (https://github.com/inscight/). Furthermore, we are are always looking to supplement our current repertoire of hosts and special guests. So if you would like to contribute to inSCIght or have something interesting to present on a show, feel free to email us at info_AT_inscight.org. We'd love to have you join the conversation! The inSCIght podcast is a co-production of Enthought, Software Carpentry, and The Hacker Within. Thanks for listening! The inSCIght podcast is licensed under the Creative Commons Attribution 3.0 Unported (CC BY 3.0) license.

1 0

Problems with installation (wrong Python version)
by Dumka Dumka 23 Feb '11

23 Feb '11

Dear numpy Community, I try to install numpy on my Windows 7. I take the code from here: http://sourceforge.net/projects/numpy/files/NumPy/ (I click on "Download numpy-1.5.1-win32-superpack-python2.6.exe (5.4 MB)"). After I start the installation, the installer tells me: Python version 2.6 (Found in registry). I also see the link to the "C:\Python26" directory in the installation window. However, I want to use Python2.7 (I had it on my computer together with the Python 2.6). So, I deleted the "C:\Python26" directory an set a new value to the "Path" environment variable to make a link to the newer version of the Python (version 2.7). I also restarted the computer. However, it does not help. When I try to run the numpy installer again I see the same: Python version 2.6 (Found in registry). I cannot understand where the installer find the link to the old version. If I just continue the installation, it fails (and it started to fail after I deleted the "C:\Python26" directory). So, to summarize, numpy installer "thinks" that I have Python 2.6 (and I don't) and then it fails because it is wrong. How can I force the numpy installer to use Python 2.7? Can I? Thank you in advance for any help, Dumka

3 3

NumPy-Discussion February 2011