Mailman 3 May 2010 - NumPy-Discussion

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

SVD errors
by mtrumpis＠berkeley.edu 04 Jun '13

04 Jun '13

Hello list.. I've run into two SVD errors over the last few days. Both errors are identical in numpy/scipy. I've submitted a ticket for the 1st problem (numpy ticket #990). Summary is: some builds of the lapack_lite module linking against system LAPACK (not the bundled dlapack_lite.o, etc) give a "LinAlgError: SVD did not converge" exception on my matrix. This error does occur using Mac's Accelerate framework LAPACK, and a coworker's Ubuntu LAPACK version. It does not seem to happen using ATLAS LAPACK (nor using Octave/Matlab on said Ubuntu) Just today I've come across a negative singular value cropping up in an SVD of a different matrix. This error does occur on my ATLAS LAPACK based numpy, as well as on the Ubuntu setup. And once again, it does not happen in Octave/Matlab. I'm using numpy 1.3.0.dev6336 -- don't know what the Ubuntu box is running. Here are some npy files for the two different cases: https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/noconverge_operator.npy https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/negsval_operator.npy Mike

6 7

example reading binary Fortran file
by David Froger 13 Feb '13

13 Feb '13

Hy, My question is about reading Fortran binary file (oh no this question again...) Until now, I was using the unpack module like that : def lread(f,fourBeginning,fourEnd,*tuple): from struct import unpack """Reading a Fortran binary file in litte-endian""" if fourBeginning: f.seek(4,1) for array in tuple: for elt in xrange(array.size): transpose(array).flat[elt] = unpack(array.dtype.char,f.read(array.itemsize))[0] if fourEnd: f.seek(4,1) After googling, I read that fopen and npfille was deprecated, and we should use numpy.fromfile and ndarray.tofile, but despite of the documentaion, the cookbook, the mailling list and google, I don't succed in making a simple example. Considering the simple Fortran code below what is the Python script to read the four arrrays? What about if my pc is little endian and the file big endian? I think it will be a good idea to put the Fortran writting-arrays code and the Python reading-array script in the cookbook and maybe a page to help people comming from Fortran to start with Python ? Best, David Froger program makeArray implicit none integer,parameter:: nx=10,ny=20 real(4),dimension(nx,ny):: ux,uy,p integer :: i,j open(11,file='uxuyp.bin',form='unformatted') do i = 1,nx do j = 1,ny ux(i,j) = real(i*j) uy(i,j) = real(i)/real(j) p (i,j) = real(i) + real(j) enddo enddo write(11) ux,uy write(11) p close(11) end program makeArray

8 22

dtype '|S0' not understood
by David Warde-Farley 07 Oct '12

07 Oct '12

Howdy, It seems it's possible using e.g. In [25]: dtype([('foo', str)])Out[25]: dtype([('foo', '|S0')]) to get yourself a zero-length string. However dtype('|S0') results in a TypeError: data type not understood. I understand the stupidity of creating a 0-length string field but it's conceivable that it's accidental. For example, it could lead to a situation where you've created that field, are missing all the data you had meant to put in it, serialize with np.save, and upon np.load aren't able to get _any_ of your data back because the dtype descriptor is considered bogus (can you guess why I thought of this scenario?). It seems that either dtype(str) should do something more sensible than zero-length string, or it should be possible to create it with dtype('| S0'). Which should it be? David

3 3

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 17 May '11

17 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

3d plane to point cloud fitting using SVD
by Peter Schmidtke 26 Jan '11

26 Jan '11

Dear Numpy Users, I want to fit a 3d plane into a 3d point cloud and I saw that one could use svd for this purpose. So as I am very fond of numpy I saw that svd was implementented in the linalg module. Currently I have a numpy array called xyz with n lines (number of points) and 3 columns (x,y,z). I calculated the centroid as : xyz0=npy.mean(xyz, axis=0) #calculate the centroid Next I shift the centroid of the point cloud to the origin with. M=xyz-xyz0 next I saw by matlab analogy (http://www.mathworks.co.jp/matlabcentral/newsreader/view_thread/262996) that I can write this : u,s,vh=numpy.linalg.linalg.svd(M) Then in the matlab analog they use the last column of vh to get the a,b,c coefficients for the equation a,b,c=vh[:, -1] in numpy The problem is that the equation ax+by+cz=0 does not represent the plan through my point cloud at all. What am I doing wrong, how can I get the a,b and c coefficients? Thanks in advance. -- Peter Schmidtke ---------------------- PhD Student at the Molecular Modeling and Bioinformatics Group Dep. Physical Chemistry Faculty of Pharmacy University of Barcelona

3 2

Math Library
by Charles R Harris 17 Jan '11

17 Jan '11

Hi All, David Cournapeau has mentioned that he would like to have a numpy math library that would supply missing functions and I'm wondering how we should organise the source code. Should we put a mathlib directory in numpy/core/src? Inside that directory would be functions for single/double/extended/quad precision. Should they be in separate directories? What about complex versions? I'm thinking that a good start would be to borrow the msun functions for doubles. We should also make a list of what functions would go into the library and what interface the complex functions present. Thoughts? Chuck

10 23

A faster median (Wirth's method)
by Sturla Molden 01 Dec '10

01 Dec '10

We recently has a discussion regarding an optimization of NumPy's median to average O(n) complexity. After some searching, I found out there is a selection algorithm competitive in speed with Hoare's quick select. It has the advantage of being a lot simpler to implement. In plain Python: import numpy as np def wirthselect(array, k): """ Niklaus Wirth's selection algortithm """ a = np.ascontiguousarray(array) if (a is array): a = a.copy() l = 0 m = a.shape[0] - 1 while l < m: x = a[k] i = l j = m while 1: while a[i] < x: i += 1 while x < a[j]: j -= 1 if i <= j: tmp = a[i] a[i] = a[j] a[j] = tmp i += 1 j -= 1 if i > j: break if j < k: l = i if k < i: m = j return a Now, the median can be obtained in average O(n) time as: def median(x): """ median in average O(n) time """ n = x.shape[0] k = n >> 1 s = wirthselect(x, k) if n & 1: return s[k] else: return 0.5*(s[k]+s[:k].max()) The beauty of this is that Wirth select is extremely easy to migrate to Cython: import numpy ctypedef numpy.double_t T # or whatever def wirthselect(numpy.ndarray[T, ndim=1] array, int k): cdef int i, j, l, m cdef T x, tmp cdef T *a _array = np.ascontiguousarray(array) if (_array is array): _array = _array.copy() a = <T *> _array.data l = 0 m = <int> a.shape[0] - 1 with nogil: while l < m: x = a[k] i = l j = m while 1: while a[i] < x: i += 1 while x < a[j]: j -= 1 if i <= j: tmp = a[i] a[i] = a[j] a[j] = tmp i += 1 j -= 1 if i > j: break if j < k: l = i if k < i: m = j return _array For example, we could have a small script that generates withselect for all NumPy dtypes (T as template), and use a dict as jump table. Chad, you can continue to write quick select using NumPy's C quick sort in numpy/core/src/_sortmodule.c.src. When you are done, it might be about 10% faster than this. :-) Reference: http://ndevilla.free.fr/median/median.pdf Best regards, Sturla Molden

12 28

Test failures on 2.6
by T J 18 Nov '10

18 Nov '10

Hi, I'm getting a couple of test failures with Python 2.6, Numpy 1.2.0, Nose 0.10.4: nose version 0.10.4 ..........................................................................................................................................................................................................................................................................................................................................................................................................................................F................K........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:68: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(c.readlines(), ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1315: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1322: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1989: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(test.mask, [0,1,0,0,0,0,0,0,0,0]) ...............................................E................................................................................................................................................................................ ====================================================================== ERROR: Tests the min/max functions with explicit outputs ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py", line 653, in test_minmax_funcs_with_output result = npfunc(xm,axis=0,out=nout) File "/home/me/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", line 1525, in amin return amin(axis, out) File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/core.py", line 2978, in min np.putmask(out, newmask, np.nan) ValueError: cannot convert float NaN to integer ====================================================================== FAIL: test_umath.TestComplexFunctions.test_against_cmath ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/me/usr/lib/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/me/usr/lib/python2.6/site-packages/numpy/core/tests/test_umath.py", line 268, in test_against_cmath assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b) AssertionError: arcsin 2: (1.57079632679-1.31695789692j); cmath: (1.57079632679+1.31695789692j) ---------------------------------------------------------------------- Ran 1726 tests in 8.856s FAILED (KNOWNFAIL=1, errors=1, failures=1) <nose.result.TextTestResult run=1726 errors=1 failures=1>

2 1

BUG: NumPy exposes the wrong include directory
by Stéfan van der Walt 26 Aug '10

26 Aug '10

Hi all, Under Ubuntu, the NumPy headers are dumped under /usr/include/python2.6. This is not the location that should be exposed when another copy of NumPy is installed. The attached patch lets from numpy.distutils.system_info import get_info print get_info['numpy'] return the correct path. However, it's a quick hack, and I'd like someone familiar with numpy.distutils to have a look and suggest a proper fix. Regards Stéfan

3 4

building numpy against Cray xt-libsci
by Amir 01 Jul '10

01 Jul '10

I am trying to build numpy against Cray's xt-libsci library on a Cray XT5. I am getting an error I am hoping for hints on how to resolve: In [1]: import numpy <snip> 20 isfinite, size 21 from numpy.lib import triu ---> 22 from numpy.linalg import lapack_lite 23 from numpy.matrixlib.defmatrix import matrix_power 24 ImportError: /opt/xt-libsci/10.4.0/gnu/lib/libsci.so: undefined symbol: fftw_version These are the symbols in libsci: % nm /opt/xt-libsci/10.4.0/gnu/lib/libsci.so | grep fftw_version 00000000010f9a30 B __crafft_internal__crafft_fftw_version_num U fftw_version 00000000005aa8a4 T get_fftw_version I first built numpy with no custom site.cfg file. It built correctly and all tests ran. But it was too slow. Then I tried building numpy against libsci, which has BLAS, LAPACK, FFTW3 among other things. I had to build a libcblas.a from the netlib src as libsci does not have cblas (using gcc, gfortran 4.3.3). Here is my site.cfg, accumulated from several nice tutorials on how to build numpy on these machines, which for some reason don't work for me. The instructions were based on numpy 1.2. [blas] blas_libs = cblas library_dirs = /global/homes/amir/local/lib [lapack] lapack_libs = sci library_dirs = /opt/xt-libsci/10.4.0/gnu/lib [blas_opt] blas_libs = cblas, sci libraries = cblas, sci [lapack_opt] libraries = sci [fftw] libraries = fftw3 Here is what is linked to lapack_lite.so: % ldd ./numpy/linalg/lapack_lite.so libsci.so => /opt/xt-libsci/10.4.0/gnu/lib/libsci.so (0x00002b4493325000) libgfortran.so.3 => /opt/gcc/4.3.3/snos/lib64/libgfortran.so.3 (0x00002b44a4579000) libm.so.6 => /lib64/libm.so.6 (0x00002b44a4770000) libgcc_s.so.1 => /opt/gcc/4.3.3/snos/lib64/libgcc_s.so.1 (0x00002b44a48c6000) libc.so.6 => /lib64/libc.so.6 (0x00002b44a49dd000) /lib64/ld-linux-x86-64.so.2 (0x0000555555554000) Thanks, Amir.

3 2

[Job] research developer job opening at AQR Capital
by Wes McKinney 10 Jun '10

10 Jun '10

We are looking for a scientific Python developer to work on the design and implementation of AQR’s proprietary research and production systems. In this role you will work closely with researchers and portfolio managers to identify problems and create solutions addressing speed, scale, and usability. Most of our work goes into building back-testing infrastructure and other tools for quantitative portfolio analysis. As part of this you will become intimately familiar with financial concepts and AQR’s investment philosophy. Requirements ------------ * Degree in computer science, mathematics, or other technical discipline * Experience level: recent graduates up to 5 years * Experience using NumPy and/or SciPy libraries * Effective problem solving and quantitative skills * Well organized, detail oriented and able to focus in a dynamic environment * Ability to discuss and explain involved concepts in both verbal and written form * Able to work independently, as well as part of a team About AQR Capital Management ---------------------------- The founders of AQR were still Ph.D. candidates when they initially developed the complex financial models that are now at AQR’s core. Today, AQR Capital Management is a $24 billion asset management firm employing a disciplined multi-asset, global research process (AQR stands for Applied Quantitative Research). AQR's investment products are provided through a limited set of collective investment vehicles and separate accounts that deploy all or a subset of AQR's investment strategies. These investment products span from aggressive high volatility market-neutral hedge funds, to low volatility benchmark-driven traditional products. AQR is located in Greenwich, CT, a short commute from New York City. * If you are interested in applying for this position please submit your resume to matt.gombos(a)aqr.com and put "GAARD Candidate" in the subject line. * We are an affirmative action / equal opportunity employer.

2 2

Bug in nanmin called with unsigned integers
by Tony S Yu 07 Jun '10

07 Jun '10

I got bit again by this bug with unsigned integers. (My original changes got overwritten when I updated from svn and, unfortunately, merged conflicts without actually looking over the changes.) In any case, I thought it'd be a good time to bump the issue (with patch). Cheers, -Tony PS: Just for context, this issue comes up when displaying images with Chaco (which converts images to unsigned integer arrays and calls nanmin).

2 3

Finding Star Images on a Photo (Video chip) Plate?
by Wayne Watson 02 Jun '10

02 Jun '10

Suppose I have a 640x480 pixel video chip and would like to find star images on it, possible planets and the moon. A possibility of noise exits, or bright pixels. Is there a known method for finding the centroids of these astro objects? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39° 15' 7" N, 121° 2' 32" W, 2700 feet There are no statues or memorials dedicated to Thomas Paine for his substantial part in the American Revolution. -- An observation in The Science of Liberty by Timoth Ferris Web Page:<www.speckledwithstars.net/>

7 12

Introduction to Scott, Jason, and (possibly) others from Enthought
by Travis Oliphant 31 May '10

31 May '10

Hi everyone, There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works. I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work). While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated). Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do. Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy. I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility. Best regards, -Travis

12 46

linalg.det throws IndexError on 2.6.5
by Ilya Sterin 31 May '10

31 May '10

I'm not sure what's causing this at this point and before I dig deeper thought someone can shed some light... On my Mac OS X python 2.6.1, numpy.linalg.det functions properly... >>> numpy.linalg.det(numpy.array([[1, 2], [3, 4]])) -2.0 On Centos 5 python 2.6.5, I get and IndexError (out of bounds)... >>> numpy.linalg.det(numpy.array([[1, 2], [3, 4]])) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/python2.6/lib/python2.6/site-packages/numpy/linalg/linalg.py", line 1507, in det return (1.-2.*sign)*multiply.reduce(diagonal(a), axis=-1) File "/opt/python2.6/lib/python2.6/site-packages/numpy/core/fromnumeric.py", line 949, in diagonal return asarray(a).diagonal(offset, axis1, axis2) IndexError: index 12884901888 out of bounds 0<=index<4 Any ideas? Thanks. Ilya

2 4

ix_ and copies
by Keith Goodman 30 May '10

30 May '10

Will making changes to arr2 never change arr1 if arr2 = arr1[np.ix_(*lists)] where lists is a list of (index) lists? np.ix_ returns a tuple of arrays so I'm guessing (and hoping) the answer is yes.

3 4

shuffle a slice
by Keith Goodman 29 May '10

29 May '10

np.random.shuffle: "Modify a sequence in-place by shuffling its contents." Matches doc string: >> a = np.arange(10) >> np.random.shuffle(a[:-1]) >> a array([0, 7, 8, 4, 3, 6, 2, 1, 5, 9]) Doesn't match doc string: >> l = range(10) >> np.random.shuffle(l[:-1]) >> l [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] Is there any way for numpy to catch this?

2 2

assert_almost_equal and inf
by josef.pktd＠gmail.com 29 May '10

29 May '10

I'm getting some test failures in my code with infs in numpy 1.4, nan treatment has been corrected in numpy.testing >>> assert_almost_equal(np.array([-5.23722324, -np.nan, -5.23722324]), np.array([-5.23722324, -np.nan, -5.23722324])) but it seems that inf doesn't compare almost equal >>> assert_almost_equal(np.array([-5.23722324, -np.inf, -5.23722324]), np.array([-5.23722324, -np.inf, -5.23722324])) Traceback (most recent call last): ... ValueError: Arrays are not almost equal x: array([-5.23722324, -Inf, -5.23722324]) y: array([-5.23722324, -Inf, -5.23722324]) numpy 1.4.0 has this been changed already? or is it not desired? Josef

1 0

Gauss-Newton Method in Python?
by Wayne Watson 29 May '10

29 May '10

Is Subject method available in Python? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39° 15' 7" N, 121° 2' 32" W, 2700 feet There are no statues or memorials dedicated to Thomas Paine for his substantial part in the American Revolution. -- An observation in The Science of Liberty by Timoth Ferris Web Page:<www.speckledwithstars.net/>

3 3

Re: [Numpy-discussion] Installing audiolab on python 2.6
by arthur de conihout 28 May '10

28 May '10

Hi I reinstall it on a different machine i got the same error with the nsdfile no warning anymore with audiolab . I followed these points: Requirements¶<http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/audiolab/sphinx/i…> audiolab requires the following softwares: - a python interpreter. - libsndfile - numpy (any version >= 1.2 should work). - setuptools On Ubuntu, you can install the dependencies as follow: sudo apt-get install python-dev python-numpy python-setuptools libsndfile-dev Optional¶<http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/audiolab/sphinx/i…> Audiolab can optionally install audio backends. For now, only alsa (Linux) and Core Audio (Mac OS X) are supported. On Linux, you need alsa headers for this to work; on Ubuntu, you can install them with the following command: sudo apt-get install libasound2-dev For Mac OS X, you need the CoreAudio framework, available on the Apple website. Build¶<http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/audiolab/sphinx/i…>For unix users, if libsndfile is installed in standart location (eg /usr/lib, /usr/local/lib), the installer should be able to find them automatically, and you only need to do a “python setup.py install” Concerning the eventual site.cfg file i have no idea what to do with this it might be the problem maybe you can give advice? Thank you 2010/5/28 David Cournapeau <cournape(a)gmail.com> > On Fri, May 28, 2010 at 1:58 AM, arthur de conihout > <arthurdeconihout(a)gmail.com> wrote: > > Thank you for the answer > > i would love trying audiolab but i got this error while importing i m > > running python2.6 > > > >>>> import audiolab > > > /usr/local/lib/python2.6/dist-packages/audiolab-0.0.0-py2.6-linux-i686.egg/audiolab/soundio/play.py:48: > > UserWarning: Could not import alsa backend; most probably, you did not > have > > alsa headers when building audiolab > > How did you install it ? Your installation looks bogus (if you build > it by yourself, please give the *exact* instructions you used) > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

1 1

curious about how people would feel about moving to github
by Jarrod Millman 28 May '10

28 May '10

Hello, I changed the subject line for this thread, since I didn't want to hijack another thread. Anyway, I am not proposing that we actually decide whether to move to git and github now, but I am just curious how people would feel. We had a conversation about this a few years ago and it was quite contentious at the time. Since then, I believe a number of us have started using git and github for most of our work. And there are a number of developers using git-svn to develop numpy now. So I was curious to get a feeling for what people would think about it, if we moved to git. (I don't want to rehash the arguments for the move.) Anyway, Chuck listed the main concerns we had previously when we discussed moving from svn to git. See the discussion below. Are there any other concerns? Am I right in thinking that most of the developers would prefer git at this point? Or are there still a number of developers who would prefer using svn still? On Wed, May 26, 2010 at 12:54 PM, Charles R Harris <charlesr.harris(a)gmail.com> wrote: > I think the main problem has been windows compatibility. Git is best from > the command line whereas the windows command line is an afterthought. > Another box that needs a check-mark is the buildbot. If svn clients are > supported then it may be that neither of those are going to be a problem. I was under the impression that there were a number of decent git clients for Windows now, but I don't know anyone who develops on Windows. Are there any NumPy developers who use Windows who could check out the current situation? Pulling from github with an svn client works very well, so buildbot could continue working as is: http://github.com/blog/626-announcing-svn-support And if it turns out the Windows clients are still not good enough, we could look into the recently add svn write support to github: http://github.com/blog/644-subversion-write-support No need for us to make any changes immediately. I am just curious how people would feel about it at this point. Jarrod

14 41

crash in np.poly
by josef.pktd＠gmail.com 28 May '10

28 May '10

I was tracking a test failure/crash in scipy.signal.ltisys numpy 1.4.0: >>> np.poly(np.zeros((0,0))) ** On entry to DGEEV parameter number 5 had an illegal value Josef

2 3

Re: [Numpy-discussion] Help Convolution with binaural filters(HRTFs)
by arthur de conihout 28 May '10

28 May '10

> > > > Hi, >> i try to implement a real-time convolution module refreshed by head >> listener location (angle from a reference point).The result of the >> convolution by binaural flters(HRTFs) allows me to spatialize a monophonic >> wavfile. I got trouble with this as long as my convolution doesnt seem to >> work properly: >> np.convolve() doesnt convolve the entire entry signal >> >> ->trouble with extracting numpyarrays from the audio wav. filters and >> monophonic entry >> ->trouble then with encaspulating the resulting array in a proper wav >> file...it is not read by audacity >> >> Do you have any idea of how this could work or do you have any >> implementation of stereo filtering by impulse response to submit me >> Thank you very much >> >> Arthur de Conihout >> >> > > > >

3 9

NotImplemented returns
by josef.pktd＠gmail.com 27 May '10

27 May '10

A while ago we had a brief discussion about this. Is this a feature? or should there be a ticket for this >>> np.sqrt('5') NotImplemented >>> a = np.sqrt('5') >>> a NotImplemented >>> type(a) <type 'NotImplementedType'> Josef

3 5

StarCluster 0.91 - NumPy/SciPy Clusters on EC2
by Justin Riley 27 May '10

27 May '10

This is a one-time message to announce the availability of version 0.91 of the StarCluster package. Why should you care? StarCluster allows you to create NumPy/SciPy clusters configured with NFS-shared filesystems and the Sun Grid Engine queueing system out of the box on Amazon's Elastic Compute Cloud (EC2). The NumPy/SciPy installations have been compiled against a custom-compiled ATLAS for the larger EC2 instances. About ----- There is an article about StarCluster on www.hpcinthecloud.com: http://www.hpcinthecloud.com/features/StarCluster-Brings-HPC-to-the-Amazon-… There is also a screencast of installing, configuring, launching, and terminating an HPC cluster on Amazon EC2: http://www.hpcinthecloud.com/blogs/MITs-StarCluster-An-Update-with-Screenca… Project description from PyPI: StarCluster is a utility for creating and managing scientific computing clusters hosted on Amazon's Elastic Compute Cloud (EC2). StarCluster utilizes Amazon's EC2 web service to create and destroy clusters of Linux virtual machines on demand. To get started, the user creates a simple configuration file with their AWS account details and a few cluster preferences (e.g. number of machines, machine type, ssh keypairs, etc). After creating the configuration file and running StarCluster's "start" command, a cluster of Linux machines configured with the Sun Grid Engine queuing system, an NFS-shared /home directory, and OpenMPI with password-less ssh is created and ready to go out-of-the-box. Running StarCluster's "stop" command will shutdown the cluster and stop paying for service. This allows the user to only pay for what they use. StarCluster provides a Ubuntu-based Amazon Machine Image (AMI) in 32bit and 64bit architectures. The AMI contains an optimized NumPy/SciPy/Atlas/Blas/Lapack installation compiled for the larger Amazon EC2 instance types. The AMI also comes with Sun Grid Engine (SGE) and OpenMPI compiled with SGE support. The public AMI can easily be customized by launching a single instance of the public AMI, installing additional software on the instance, and then using StarCluster can also utilize Amazon's Elastic Block Storage (EBS) volumes to provide persistent data storage for a cluster. EBS volumes allow you to store large amounts of data in the Amazon cloud and are also easy to back-up and replicate in the cloud. StarCluster will mount and NFS-share any volumes specified in the config. StarCluster's "createvolume" command provides the ability to automatically create, format, and partition new EBS volumes for use with StarCluster. Download -------- StarCluster is available on PyPI (http://pypi.python.org/pypi/StarCluster) and also on the project's website: http://web.mit.edu/starcluster You will find the docs as well as links to the StarCluster mailing list on the website. New in this version: -------------------- * support for launching and managing multiple clusters on EC2 * added "listclusters" command for showing all active clusters on EC2 * support for attaching and NFS-sharing multiple EBS volumes * added createimage and createvolume commands for easily creating new AMIs and EBS volumes for use with StarCluster * experimental support for launching clusters using spot instances * added support for StarCluster "plugins" that provide the ability to perform additional configuration/setup routines on top of StarCluster's default cluster configuration * added "listpublic" command for listing all available public StarCluser AMIs that can be used with StarCluster * bash/zsh command line completion for StarCluster's command line interface

1 0

error during config on Ubuntu powerpc64, numpy-1.4.1
by Chris Kees 27 May '10

27 May '10

Hi, I'm getting an error in check_long_double_representation on a linux/powerpc64 box. Has anybody seen this before/know a fix? -Chris > python -V Python 2.6.5 > uname -a Linux chl-29-200 2.6.32-21-powerpc64-smp #32-Ubuntu SMP Fri Apr 16 10:28:57 UTC 2010 ppc64 GNU/Linux >python setup.py install ... C compiler: gcc -m32 -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/home/cekees/src/proteus/linux /include/python2.6 -c' gcc: _configtest.c removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 187, in <module> setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/home/cekees/src/proteus/externalPackages/numpy-1.4.1/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/home/cekees/src/proteus/linux/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/home/cekees/src/proteus/linux/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/home/cekees/src/proteus/linux/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/cekees/src/proteus/externalPackages/numpy-1.4.1/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/home/cekees/src/proteus/linux/lib/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/home/cekees/src/proteus/linux/lib/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/home/cekees/src/proteus/linux/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/cekees/src/proteus/externalPackages/numpy-1.4.1/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/home/cekees/src/proteus/externalPackages/numpy-1.4.1/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/home/cekees/src/proteus/externalPackages/numpy-1.4.1/numpy/distutils/command/build_src.py", line 328, in build_extension_sources sources = self.generate_sources(sources, ext) File "/home/cekees/src/proteus/externalPackages/numpy-1.4.1/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 413, in generate_config_h rep = check_long_double_representation(config_cmd) File "numpy/core/setup_common.py", line 136, in check_long_double_representation type = long_double_representation(pyod(object)) File "numpy/core/setup_common.py", line 244, in long_double_representation raise ValueError("Unrecognized format (%s)" % saw) ValueError: Unrecognized format (['001', '043', '105', '147', '211', '253', '315', '357', '301', '235', '157', '064', '124', '000', '000', '000', '000', '000', '000', '000', '000', '000', '000', '000', '376', '334', '272', '230', '166', '124', '062', '020'])

1 0

Extending documentation to c code
by Charles R Harris 27 May '10

27 May '10

Hi All, I'm wondering if we could extend the current documentation format to the c source code. The string blocks would be implemented something like /**NpyDoc """The Answer. Answer the Ultimate Question of Life, the Universe, and Everything. Parameters ---------- We don't need no stinkin' parameters. Notes ----- The run time of this routine may be excessive. """ */ int answer_ultimate_question(void) { return 42; } and the source scanned to generate the usual documentation. Thoughts? Chuck

7 20

How to distinguish between number and string dypes
by Vincent Davis 27 May '10

27 May '10

How do I determine if an array's (or column in a structured array) dtype is a number or a string. I see how to determine the actual dtype but all I want to know is if it is a string or a number. *Vincent Davis 720-301-3003 * vincent(a)vincentdavis.net my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>

6 8

zeros_like and friends shouldn't use ndarray.__new__(type(a), ...)
by Jim Porter 27 May '10

27 May '10

Ok, let's try sending this message again, since it looks like I can't send from gmane... (See discussion on python-list at http://permalink.gmane.org/gmane.comp.python.general/661328 for context) numpy.zeros_like contains the following code: def zeros_like(a): if isinstance(a, ndarray): res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) res.fill(0) return res ... This is a problem because basetype.__new__(subtype, ...) raises an exception when subtype is defined from C (specifically, when Py_TPFLAGS_HEAPTYPE is not set). There's a check in Objects/typeobject.c in tp_new_wrapper that disallows this (you can grep for "is not safe" to find there the exception is raised). The end result is that it's impossible to use zeros_like, ones_like or empty_like with ndarray subtypes defined in C. While I'm still not sure why Python needs this check in general, Robert Kern pointed out that the problem can be fixed pretty easily in NumPy by changing zeros_like and friends to something like this (with some modifications from me): def zeros_like(a): if isinstance(a, ndarray): res = numpy.zeros(a.shape, a.dtype, order=a.flags.fnc) res = res.view(type(a)) res.__array_finalize__(a) return res ... - Jim

2 1

Bug in frompyfunc starting at 10000 elements?
by James Bergstra 26 May '10

26 May '10

Hi all, I'm wondering if this is a bug... Something strange happens with my ufunc as soon as I use 10000 elements. As the test shows, the ufunc computes the correct result for either the first or last 9999 elements, but both at the same time is no good. Turns out I'm only running numpy 1.3.0 with Python 2.6.4... could someone with a more recent installation maybe check to see if this has been fixed? Thanks, def test_ufunc(): np = numpy rng = np.random.RandomState(2342) a = rng.randn(10000, 2) b = rng.randn(10000, 1) f = lambda x,y:x*y ufunc = np.frompyfunc(lambda *x:numpy.prod(x), 2, 1) def g(x,y): return np.asarray(ufunc(x,y), dtype='float64') assert numpy.allclose(f(a[:-1],b[:-1]), g(a[:-1],b[:-1])) # PASS assert numpy.allclose(f(a[1:],b[1:]), g(a[1:],b[1:])) # PASS assert numpy.allclose(f(a,b), g(a,b)) # FAIL -- http://www-etud.iro.umontreal.ca/~bergstrj

3 2

numpy and the Google App Engine
by Christopher Hanley 26 May '10

26 May '10

Greetings, Google provides a product called App Engine. The description from their site follows, "Google App Engine enables you to build and host web apps on the same systems that power Google applications. App Engine offers fast development and deployment; simple administration, with no need to worry about hardware, patches or backups; and effortless scalability. " You can deploy applications written in either Python or JAVA. There are free and paid versions of the service. The Google App Engine would appear to be a powerful source of CPU cycles for scientific computing. Unfortunately this is currently not the case because numpy is not one of the supported libraries. The Python App Engine allows only the installation of user supplied pure Python code. I have recently returned from attending the Google I/O conference in San Francisco. While there I inquired into the possibility of getting numpy added. The basic response was that there doesn't appear to be much interest from the community given the amount of work it would take to vet and add numpy. I would like to ask your help in changing this perception. The quickest and easiest thing you can do would be to add your "me too" to this feature request (item #190) on the support site: http://code.google.com/p/googleappengine/issues/detail?id=190 If this issue is important to you could also consider raising this issue in the related Google Group: http://groups.google.com/group/google-appengine Letting Google know how you will use numpy would be helpful. If you or your institution would be willing to pay for service if you could deploy cloud applications that required numpy would be helpful to let them know as well. Finally, if you run into any App Engine developers (Guido included) let them know that you would like to see numpy added. Thank you for your time and consideration. Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338

4 5

[Patch] Fix memmap pickling
by Gael Varoquaux 26 May '10

26 May '10

Memmapped arrays don't pickle right. I know that to get them to really pickle and restore identically, we would need some effort. However, in the current status, pickling and restoring a memmapped array leads to tracebacks that seem like they could be avoided. I am attaching a patch with a test that shows the problem, and a fix. Should I create a ticket, or is this light-enough to be applied immediatly? Cheers, Gaël

2 3

FW: Numpy python build
by Padma TAN 26 May '10

26 May '10

Hi, Can I just install numpy and scipy without ATLAS? And what does this means " gnu: no Fortran 90 compiler found"? Im installing on RHEL Thanks in advance! [root@giswk002 numpy-1.3.0]# python setup.py build Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/Python-2.6.2/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/Python-2.6.2/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2 libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in /usr/local/Python-2.6.2/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib/sse2 libraries f77blas,cblas,atlas not found in /usr/lib NOT AVAILABLE /usr/local/numpy-1.3.0/numpy/distutils/system_info.py:1383: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: libraries blas not found in /usr/local/Python-2.6.2/lib libraries blas not found in /usr/local/lib FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/Python-2.6.2/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/Python-2.6.2/lib libraries lapack_atlas not found in /usr/local/Python-2.6.2/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2 libraries lapack_atlas not found in /usr/lib/sse2 libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /usr/local/Python-2.6.2/lib libraries lapack_atlas not found in /usr/local/Python-2.6.2/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib/sse2 libraries lapack_atlas not found in /usr/lib/sse2 libraries f77blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info NOT AVAILABLE /usr/local/numpy-1.3.0/numpy/distutils/system_info.py:1290: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: libraries lapack not found in /usr/local/Python-2.6.2/lib libraries lapack not found in /usr/local/lib FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src building py_modules sources building library "npymath" sources building extension "numpy.core._sort" sources adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h' to sources. numpy/core/code_generators/genapi.py:9: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h' to sources. numpy.core - nothing done with h_files = ['build/src.linux-i686-2.6/numpy/core/include/numpy/config.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h'] building extension "numpy.core.multiarray" sources adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files = ['build/src.linux-i686-2.6/numpy/core/src/scalartypes.inc', 'build/src.linux-i686-2.6/numpy/core/src/arraytypes.inc', 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__ufunc_api.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files = ['build/src.linux-i686-2.6/numpy/core/src/scalartypes.inc', 'build/src.linux-i686-2.6/numpy/core/src/arraytypes.inc', 'build/src.linux-i686-2.6/numpy/core/src/umath_funcs.inc', 'build/src.linux-i686-2.6/numpy/core/src/umath_loops.inc', 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__ufunc_api.h'] building extension "numpy.core.scalarmath" sources adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/config.h' to sources. adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.linux-i686-2.6/numpy/core/include/numpy/__ufunc_api.h' to sources. numpy.core - nothing done with h_files = ['build/src.linux-i686-2.6/numpy/core/include/numpy/config.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/numpyconfig.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__multiarray_api.h', 'build/src.linux-i686-2.6/numpy/core/include/numpy/__ufunc_api.h'] building extension "numpy.core._dotblas" sources building extension "numpy.core.umath_tests" sources building extension "numpy.lib._compiled_base" sources building extension "numpy.numarray._capi" sources building extension "numpy.fft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. adding 'numpy/linalg/python_xerbla.c' to sources. building extension "numpy.random.mtrand" sources /usr/local/numpy-1.3.0/numpy/distutils/command/config.py:39: DeprecationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of try_run is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/local/Python-2.6.2/include/python2.6 -c' gcc: _configtest.c gcc -pthread _configtest.o -o _configtest _configtest failure. removing: _configtest.c _configtest.o _configtest building data_files sources running build_py copying numpy/version.py -> build/lib.linux-i686-2.6/numpy copying build/src.linux-i686-2.6/numpy/__config__.py -> build/lib.linux-i686-2.6/numpy copying build/src.linux-i686-2.6/numpy/distutils/__config__.py -> build/lib.linux-i686-2.6/numpy/distutils running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using build_ext running scons running build_scripts adding 'build/scripts.linux-i686-2.6/f2py' to scripts ------ End of Forwarded Message

2 1

__eq__ with str and object
by Keith Goodman 26 May '10

26 May '10

I don't understand this: >> a1 = np.array(['a', 'b'], dtype=object) >> a2 = np.array(['a', 'b']) >> >> a1 == a2 array([ True, True], dtype=bool) # Looks good >> a2 == a1 False # Should I have expected this? This works like I expected: >> a1 = np.array([1, 2], dtype=object) >> a2 = np.array([1, 2]) >> >> a1 == a2 array([ True, True], dtype=bool) >> a2 == a1 array([ True, True], dtype=bool)

3 3

Reading and writing binary data to/from file objects
by Christoph Bersch 26 May '10

26 May '10

Hi, I want to read binary data from a file using fromfile. This works as long as I use a file name as argument to fromfile. With a file object the data is wrong! Consider the following example: from numpy import * fname='file.bin' fname2='file2.bin' a = arange(1, 30) print type(a[0]) print "\noriginal data" print a # write to file name a.tofile(fname) # write to file object f = open(fname2, 'w') a.tofile(f) f.close() print "\nWritten to file name, read from file name" b = fromfile(fname, dtype=int32) print b print "\nWritten to file name, read from file object" f = open(fname, 'r') b = fromfile(f, dtype=int32) f.close() print b print "\nWritten to file object, read from file name" b = fromfile(fname2, dtype=int32) print b print "\nWritten to file object, read from file object" f = open(fname2, 'r') b = fromfile(f, dtype=int32) f.close() print b This prints: <type 'numpy.int32'> original data [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29] Written to file name, read from file name [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29] Written to file name, read from file object [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 0 0 0] Written to file object, read from file name [ 1 2 3 4 5 6 7 8 9 2573 2816 3072 3328 3584 3840 4096 4352 4608 4864 5120 5376 5632 5888 6144 6400 6656 6912 7168 7424] Written to file object, read from file object [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 0 0 0] Size of 'file.bin' (written to file name) is 116 Bytes, size of 'file2.bin' (written to file object) is 117 Bytes! So the only correct data is when writing to a file name and reading from a file name. All other variants yield wrong data! This happens with Numpy 1.4.1 with Python 2.6 under Windows XP. For comparison I only have a 1.1.0 Numpy with Python 2.5 under Linux Debian where in all cases I get the same, correct arrays! Am I doing something substantially wrong or is this a bug? Christoph

2 2

Geometric, negative binomial and poisson fail for extreme arguments
by Charles R Harris 26 May '10

26 May '10

Josef, This is ticket #896 from two years ago. IIRC, there was some more recent discussion on the list of some of these. Do you know what the current state of these distributions is? Chuck

2 2

Re: [Numpy-discussion] Introduction to Scott, Jason, and (possibly) others from Enthought
by Joe Harrington 26 May '10

26 May '10

On Tue, 25 May 2010 15:54:26 -0500, Travis Oliphant <oliphant(a)enthought.com> wrote: >On May 25, 2010, at 2:50 PM, Charles R Harris wrote: > >> >> >> On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant <oliphant(a)enthought.com >> > wrote: >> >> Hi everyone, >> >> There has been some talk about re-factoring NumPy to separate out the >> Python C-API layer and make NumPy closer to a C-library. I know >> there are a few different ideas about what this means, and also that >> people are very busy. I also know there is a NumPy 2.0 release that >> is in the works. >> >> I'm excited to let everyone know that we (at Enthought) have been able >> to find resources (about 3 man months) to work on this re-factoring >> project and Scott and Jason (both very experienced C and Python >> programmers) are actively pursuing it. My hope is that NumPy 2.0 >> will contain this re-factoring (which should be finished just after >> SciPy 2010 --- where I'm going to organize a Sprint on NumPy which >> will include at least date-time improvements and re-factoring work). >> >> While we have specific goals for the re-factoring, we want this >> activity to be fully integrated with the NumPy community and Scott and >> Jason want to interact with the community as much as feasible as they >> suggest re-factoring changes (though they both have more experience >> with phone-conversations to resolve concerns than email chains and so >> some patience from everybody will be appreciated). >> >> Because Jason and Scott are new to this mailing list (but not new to >> NumPy), I wanted to introduce them so they would feel more >> comfortable posting questions and people would have some context as to >> what they were trying to do. >> >> Scott and Jason are both very proficient and skilled programmers and I >> have full confidence in their abilities. That said, we very much >> want the input of as many people as possible as we pursue the goal of >> grouping together more tightly the Python C-API interface layer to >> NumPy. >> >> I will be involved in some of the discussions, but am currently on a >> different project which has tight schedules and so I will only be able >> to provide limited "mailing-list" visibility. >> >> >> I think 2.0 would be a bit early for this. Is there any reason it >> couldn't be done in 2.1? What is the planned policy with regards to >> the visible interface for extensions? It would also be nice to have >> a rough idea of how the resulting code would be layered, i.e., what >> is the design for this re-factoring. Simply having a design would be >> a major step forward. > >The problem with doing it in 2.1 is that this re-factoring will >require extensions to be re-built. The visible interface to >extensions will not change, but there will likely be ABI >incompatibility. It seems prudent to do this in NumPy 2.0. >Perhaps we can also put in place the ABI-protecting indirection >approaches that David C. was suggesting earlier. > >Some aspects of the design are still being fleshed out, but the basic >idea is to separate out a core library that is as independent of the >Python C-API as possible. There will likely be at least some >dependency on the Python C-API (reference counting and error handling >and possibly others) which any interface would have to provide in a >very simple Python.h -- equivalent, for example. > >Our purpose is to allow NumPy to be integrated with other languages or >other frameworks systems without explicitly relying on CPython. >There are a lot of questions as to how this will work, and so much of >that is being worked out. Part of the reason for this mail is to >help ensure that as much of this discussion as possible takes place in >public. > >-Travis > > >> >> In any case, I think the primary goal for 2.0 should remain the >> python3k port. >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >-- >Travis Oliphant >Enthought Inc. >1-512-536-1057 >http://www.enthought.com >oliphant(a)enthought.com > Chuck wants to make 2.0 be the py3k release. I expressed a desire some time ago to hold 2.0 until we had reviewed docs, on the basis that this should have been done for 1.0, and 2.0 is our "real 1.0" for this and other reasons. Now Travis wants to refactor the C code in a way that breaks extensions (another "real 1.0" item). But Chuck wants to get his project out the door soon, and not without justification. NumPy is not a small project anymore and it would make sense not only to have an open community process (which we sort-of do, except when we don't), but to define that process and to conduct more of it in plain view, on the web site, rather than hiding it here in the list archives where only subscribers are likely to see it. There are many interested parties (users) who are not on the lists. It's fine (even good) not to commit to specific dates until we know them for sure, but the process and rough timeline should be laid out publicly. I hope we can agree on one item, which is that the 2.0 release should be one that we are not retroactively embarrassed about. This means it should be well tested and that the docs should be up to date for the new stuff, including examples. Whether the docs need to be reviewed or not is a different question; they should at least not be wrong. It's May 25. SciPy 2010 starts 28 June, just one month out. Chuck and Travis, would you be amenable to making a 2.0 release page on the web site and share there your timelines for making these major changes, including testing and documenting them? It would also be good to see any other planned changes listed out, the 2.0 release cutoff date, and a timeline for the release testing process. Thanks, --jh--

1 0

ensuring docstrings in new code
by Joe Harrington 25 May '10

25 May '10

Over on [Numpy-discussion] Extending documentation to c code, David G. gave voice to a frustration he and I share about the status of documentation in the new-code development process. I don't want to paint with a broad brush, yet in recent months there have been a number of checkins, unanimously passed off by the entire core development group/Steering Committee, without a single mention that this code entered SVN with deficient or even completely absent docstrings. I'm not pointing out the offenses simply because I don't think public embarrassment is a good policy, but David has a list that we can share with developers privately. It is really hard to do much without docs on anything you didn't write yourself. When I learned how to program, it was beaten into me to document as I went along, that good projects always did, and that major problems up to and including a complete implosion and abandonment of the software could and sometimes did result if docs were not kept current. The Doc Project is not the writing arm of the developers. So far, we have only worked on the existing software, not anything new, and most of us don't even have SVN access. Rather, the Doc Project is cleaning up what others, in their haste, did not do, and it frustrates us to see others still not doing it. What change of procedures would ensure that no code, possibly including bug fixes, enters SVN without a ready-for-review docstring? Perhaps new code could be reviewed according to a checklist that includes "has a ready-for-review docstring". Or perhaps announcements to the lists of pending code checkins should include [CHECKIN] in the subject, to attract the attention of the Doc Project's Editor (i.e., David), so that he can act as an outside cop. Or maybe the SVN can somehow be made to check for docstrings, or to ask for them on checkin. Or maybe something in the build could make such a list and compare it to the last list and complain if there are new routines on it. We've called for a rule of no new code without docs many times. I'm not sure of the best solution, but we need something or things won't change. It's not like these are hard to write. The lack of a ready-for-review docstring is a major bug in itself. --jh--

1 0

calling C function from Python via f2py
by Matt Fearon 25 May '10

25 May '10

Hello, I am trying to use f2py to generate a wrapped C function that I can call from Python (passing arguments to and from). I have this almost working, but I receive trouble with "exp and pow" related to C and some "pos (2) error" with one of my passed variables. My f2py syntax is: f2py -c -lm FFMCcalc.pyf FFMCcalc.c Also, my 3 scripts are short and attached. 1. FFMCcalc.c, C function 2. FFMCcalc.pyf, wrapper file 3. test.py, short python code that calls C function Any advice would greatly appreciated to get this working. thanks, Matt

2 4

loadtxt raises an exception on empty file
by Maria Liukis 25 May '10

25 May '10

Hello everybody, I'm using numpy V1.3.0 and ran into a case when numpy.loadtxt('foo.txt') raised an exception: >>>import numpy as np >>>np.loadtxt('foo.txt') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/io.py", line 456, in loadtxt raise IOError('End-of-file reached before encountering data.') IOError: End-of-file reached before encountering data. >>> if provided file 'foo.txt' is empty. Would anybody happen to know if it's a feature or a bug? I would expect it to return an empty array. numpy.fromfile() handles empty text files: >>> np.fromfile('foo.txt', sep='\t\n ') array([], dtype=float64) >>> Would anybody suggest a graceful way of handling empty files with numpy.loadtxt() (except for catching an IOError exception)? Many thanks, Masha -------------------- liukis(a)usc.edu

3 2

Tickets for review involving build problems, attn Ralf Gommers
by Charles R Harris 25 May '10

25 May '10

Hi Ralf, As one of the official build gurus in training, could you look over the tickets for review that fix build problems on various platforms? TIA, Chuck

2 1

Can not compile numpy with python2.7 onl linux
by Nadav Horesh 24 May '10

24 May '10

I think that line 3405 in _capi.c (svn 8386) should be: #if PY_VERSION_HEX >= 0x03010000 (At least it looks reasonable considering line 3375, and it solves my problem) Nadav

2 2

Summation of large float32/float64 arrays
by Matthew Turk 24 May '10

24 May '10

Hi all, I have a possibly naive question. I don't really understand this particular set of output: In [1]: import numpy In [2]: a1 = numpy.random.random((512,512,512)).astype("float32") In [3]: a1.sum(axis=0).sum(axis=0).sum(axis=0) Out[3]: 67110312.0 In [4]: a1.sum() Out[4]: 16777216.0 I recognize that the intermediate sums may accumulate error differently than a single call to .sum(), but I guess my concern is that it's accumulating a lot faster than I anticipated. (Interesting to note that a1.sum() returns 0.5*512^3, down to the decimal; is it summing up the mean, which should be ~0.5?) However, with a 256^3 array: In [1]: import numpy In [2]: a1 = numpy.random.random((256,256,256)).astype("float32") In [3]: a1.sum(axis=0).sum(axis=0).sum(axis=0) Out[3]: 8389703.0 In [4]: a1.sum() Out[4]: 8389245.0 The errors are much more reasonable. Is there an overflow or something that occurs with the 512^3? These problems all go completely away with a float64 array, but the issue originally showed up when trying to normalize an on-disk float32 array of size 512^3, where the normalization factor was off by a substantial factor (>2x) depending on the mechanism used to sum. My suspicion is that perhaps I have a naive misconception about intermediate steps in summations, or there is a subtlety I'm missing here. I placed a sample script I used to test this here: http://pastebin.com/dGbHwFPK Thanks for any insight anybody can provide, Matt

4 7

The labeled array package at a glance
by Keith Goodman 23 May '10

23 May '10

For those not familiar with the la package and its labeled array, larry, here's a table that gives you a quick overview: http://bazaar.launchpad.net/~kwgoodman/larry/trunk/annotate/head:/doc/sourc…

1 0

Build failure at rev8246
by T J 21 May '10

21 May '10

Hi, I tried upgrading today and had trouble building numpy (after rm -rf build). My full build log is here: http://www.filedump.net/dumped/build1274454454.txt If someone can point me in the right direction, I'd appreciate it very much. To excerpts from the log file: Running from numpy source directory.numpy/core/setup_common.py:86: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 4, with checksum 59750b518272c8987f02d66445afd3f1, but recorded checksum for C API version 4 in codegen_dir/cversions.txt is 3d8940bf7b0d2a4e25be4338c14c3c85. If functions were added in the C API, you have to update C_API_VERSION in numpy/core/setup_common.pyc. MismatchCAPIWarning) In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:36: numpy/core/src/multiarray/buffer.c: At top level: numpy/core/src/multiarray/buffer.c:715: error: conflicting types for ‘_descriptor_from_pep3118_format’ numpy/core/src/multiarray/common.c:221: note: previous implicit declaration of ‘_descriptor_from_pep3118_format’ was here numpy/core/src/multiarray/buffer.c: In function ‘_descriptor_from_pep3118_format’: numpy/core/src/multiarray/buffer.c:751: warning: assignment from incompatible pointer type In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:45: numpy/core/src/multiarray/multiarraymodule.c: In function ‘initmultiarray’: numpy/core/src/multiarray/multiarraymodule.c:3062: warning: implicit declaration of function ‘_numpymemoryview_init’ error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-i686-2.6/numpy/core/src/multiarray -Ibuild/src.linux-i686-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o build/temp.linux-i686-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1 Hope this helps.

2 2

Numpy doc string license
by Keith Goodman 21 May '10

21 May '10

I'd like to include modified numpy doc strings in my package. Do I just put a note in my license file that says my package contains numpy doc strings and then paste in the numpy license? My package is distributed under a Simplifed BSD license, if that matters.

2 2

astype None
by Keith Goodman 21 May '10

21 May '10

While automating some unit tests for my labeled array class, larry, I assumed that np.array([1, 2], dtype=dtype) would give the same result as np.array([1, 2]).astype(dtype) But it doesn't for dtype=None: >> np.array([1, 2, 3], dtype=None) array([1, 2, 3]) >> np.array([1, 2, 3]).astype(None) array([ 1., 2., 3.]) I prefer the behavior of array where dtype=None is a no-op.

2 2

Problem in swig
by Steven Nien 21 May '10

21 May '10

I want to pass 1 integer and 2 numpy array into the C function as followings: void update_incident_field(int k, float *a, int na, float *b, int nb) { for (i=0; i<na; i ++) { a[i] = a[i] + b[i] * k; } } But I don't know how to write the interface code (***.i) Can someone help me? Thanks! The swig interface code I written(did't work, strange output) %module example %{ #define SWIG_FILE_WITH_INIT #include "example.h" %} %include "numpy.i" %init %{ import_array(); %} %apply (float* INPLACE_ARRAY1, int DIM1) {(float *a, int na), (float *b, int nb)}; %include "example.h"

3 3

dtype and array creation
by Keith Goodman 21 May '10

21 May '10

Why do the follow expressions give different dtype? >> np.array([1, 2, 3], dtype=str) array(['1', '2', '3'], dtype='|S1') >> np.array(np.array([1, 2, 3]), dtype=str) array(['1', '2', '3'], dtype='|S8')

3 5

question about creating numpy arrays
by Darren Dale 20 May '10

20 May '10

I have a question about creation of numpy arrays from a list of objects, which bears on the Quantities project and also on masked arrays: >>> import quantities as pq >>> import numpy as np >>> a, b = 2*pq.m,1*pq.s >>> np.array([a, b]) array([ 12., 1.]) Why doesn't that create an object array? Similarly: >>> m = np.ma.array([1], mask=[True]) >>> m masked_array(data = [--], mask = [ True], fill_value = 999999) >>> np.array([m]) array([[1]]) This has broader implications than just creating arrays, for example: >>> np.sum([m, m]) 2 >>> np.sum([a, b]) 13.0 Any thoughts? Thanks, Darren

5 10

Runtime error in numpy.polyfit
by William Carithers 20 May '10

20 May '10

I'm trying to do a simple 2nd degree polynomial fit to two arrays of 5 entries. I get a runtime error: RuntimeError: more argument specifiers than keyword list entries (remaining format:'|:calc_lwork.gelss') in the lstsq module inside numpy.polyfit. Here's the code snippet: def findPeak(self, ydex, xdex): # take a vertical slice vslice = [] for i in range(-1,10,1) : vslice.append(arcImage[ydex+i][xdex]) vdex = n.where(vslice == max(vslice)) ymax = ydex -1 + vdex[0][0] # approximate gaussian fit by parabolic fit to logs yvalues = n.array([ymax-2, ymax-1, ymax, ymax+1, ymax+2]) svalues=n.array([arcImage[ymax-2][xdex],arcImage[ymax-1][xdex],arcImage[ymax ][xdex],arcImage[ymax+1][xdex], arcImage[ymax+2][xdex]]) avalues = n.log(svalues) ypoly = n.polyfit(yvalues, avalues, 2) And the traceback: File "/Users/williamcarithers/BOSS/src/calibrationModel.py", line 345, in findPeak ypoly = n.polyfit(yvalues, avalues, 2) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/ numpy/lib/polynomial.py", line 503, in polyfit c, resids, rank, s = _lstsq(v, y, rcond) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/ numpy/lib/polynomial.py", line 46, in _lstsq return lstsq(X, y, rcond) File "/Library/Python/2.6/site-packages/scipy-0.7.1-py2.6-macosx-10.6-universal.e gg/scipy/linalg/basic.py", line 545, in lstsq lwork = calc_lwork.gelss(gelss.prefix,m,n,nrhs)[1] RuntimeError: more argument specifiers than keyword list entries (remaining format:'|:calc_lwork.gelss') This is such a simple application of polyfit and the error occurs in the guts of lstsq, so I'm completely stumped. Any help would be greatly appreciated. Thanks, Bill Carithers

4 10

Converting None to NULL using ndpointer
by Lutz Maibaum 18 May '10

18 May '10

Hello, I am trying to use a C library function from Python using numpy and ctypes. Assume the function has the signature void foo(int* bar) so it usually takes an integer array as a parameter. In python, I define foo.argtypes=[ndpointer(dtype="intc",flags="C_CONTIGUOUS")] and can then call the function foo(x), where x is an appropriate ndarray. This works very well. The problem is that I need to be able to call the function foo with a NULL pointer as an argument. Ideally, I would like to be able to use foo(None), but that doesn't work because the argument conversion throws a TypeError. Therefore my questions: 1. Is there a ndarray that is converted to NULL by ndpointer? I was hoping that maybe ndarray(None) or an array of size 0 would work, but they don't. 2. Is there a way to tell ndpointer that None is an acceptable parameter, which should be converted to NULL? 3. I could also change the argument specification in python to foo.argtypes=[POINTER(c_int)], but then I would lose the type checking features of ndpointer. Can I invoke the ndpointer conversion manually? For example, does a ndpointer object have a method that takes a ndarray as an argument and returns a POINTER(c_int) if appropriate? Any suggestions would be much appreciated. Thanks, Lutz

1 0

Saving an array on disk to free memory - Pickling
by Jean-Baptiste Rudant 18 May '10

18 May '10

Hello, I tried to create an object : - which behave just like a numpy array ; - which can be saved on disk in an efficient way (numpy.save in my example but with pytables in my real program) ; - which can be "unloaded" (if it is saved) to free memory : it can exsit has an empty stuff which knows how to retrieve real values ; it will be loaded only when we need to work with it ; - which unloads itself before being pickled (values are already saved and don't have to be pickled). It can't, at least I think so, inherit from ndarray because sometimes (for example juste after being unpickled and before being used) it is juste an empty shell. I don't think memmap can be helpful (I want to use pytables to save it on disk and I want it to be flexible : if I use it in a temporary way, I just need it in memory and I will never save it on disk). My problems are : - this code is ugly ; - I have to define explicitely all special methods (__add__, __mul__...) of ndarrays because: * __getattr__ don't retrieve them ; * even if it does, I have to define explicitely the type of the return value (if I well understand, if it inherits from ndarray __array_wrap__ do all the stuff). Thank you for the help. Regards. import numpy import numpy class PersistentArray(object): def __init__(self, values): ''' values is a numpy array ''' self.values = values self.filename = None self.is_loaded = True self.is_saved = False def save(self, filename): self.filename = filename numpy.save(self.filename, self.values) self.is_saved = True def load(self): self.values = numpy.load(self.filename) self.is_loaded = True def unload(self): if not self.is_saved: raise Exception, "PersistentArray must be saved before being unloaded" del self.values self.is_loaded = False def __getitem__(self, index): return self.values[index] def __getattr__(self, key): if key == 'values': if not self.is_loaded: self.load() return self.values elif key == '__array_interface__': #I can't remember why I wrote this code, but I think it's necessary to make pickling work properly raise AttributeError, key else: try: #to emulate ndarray inheritance return self.values.__getattribute__(key) except AttributeError: raise AttributeError, key def __setstate__(self, dict): self.__dict__.update(dict) if self.is_loaded and self.is_saved: self.load() def __getstate__(self): if not self.is_saved: raise Exception, "persistent array must be saved before being pickled" odict = self.__dict__.copy() if self.is_saved: if self.is_loaded: odict['is_loaded'] = False del odict['values'] return odict filename = 'persistent_test.npy' a = PersistentArray(numpy.arange(10e6)) a.save(filename) a.sum() a.unload() # a still exists, knows how to retrieve values if needed, but don't use space in memory

3 4

Wrong Eigenvalue (Approximation?)
by Gabriel Mihalache 18 May '10

18 May '10

Hello, all! I'm new to Numpy and Python so please tolerate by ignorance on this but I'm having problems with some weird behavior. Consider the session: >>> import numpy as np >>> import numpy.linalg as la >>> x = np.array([[0.3, 0.2, 0.5], [0.2, 0.1, 0.7], [0.9, 0.05, 0.05]]).transpose() >>> esystem = la.eig(x) >>> esystem[0] array([ 1. , -0.3 , -0.25]) >>> esystem[0][0] 1.0000000000000004 The eigenvalue should be 1 exactly. In fact, later on I want to be able to do np.where(x == 1) which fails. The way I set up the matrix, I know for sure that there must be one eigenvalue exactly equal to 1. Any help is greatly appreciated! This is all on Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32 Regards, Gabriel

6 5

savetxt not working with python3.1
by Nadav Horesh 17 May '10

17 May '10

in module npyio.py lines 794,796 "file" should be replaced by "_file" Nadav

3 2

faster code
by Davide Lasagna 17 May '10

17 May '10

Hi all, What is the fastest and lowest memory consumption way to compute this? y = np.arange(2**24) bases = y[1:] + y[:-1] Actually it is already quite fast, but i'm not sure whether it is occupying some temporary memory is the summation. Any help is appreciated. Cheers Davide

5 8

newbie: convert recarray to floating-point ndarray with mixed types
by Gregory, Matthew 17 May '10

17 May '10

Apologies for what is likely a simple question and I hope it hasn't been asked before ... Given a recarray with a dtype consisting of more than one type, e.g. >>> import numpy as n >>> a = n.array([(1.0, 2), (3.0, 4)], dtype=[('x', float), ('y', int)]) >>> b = a.view(n.recarray) >>> b rec.array([(1.0, 2), (3.0, 4)], dtype=[('x', '<f8'), ('y', '<i4')]) Is there a simple way to convert 'b' to a floating-point ndarray, casting the integer field to a floating-point? I've tried the naïve: >>> c = b.view(dtype='float').reshape(b.size,-1) but that fails with: ValueError: new type not compatible with array. I understand why this would fail (as it is a view and not a copy), but I'm lost on a method to do this conversion simply. thanks, matt

3 2

[sympy] EuroScipy abstract submission deadline extended
by Gael Varoquaux 16 May '10

16 May '10

Given that we have been able to turn on registration only very late, the EuroScipy conference committee is extending the deadline for abstract submission for the 2010 EuroScipy conference. On Thursday May 20th, at midnight Samoa time, we will turn off the abstract submission on the conference site. Up to then, you can modify the already-submitted abstract, or submit new abstracts. We are very much looking forward to your submissions to the conference. Gaël Varoquaux Nicolas Chauvat -- EuroScipy 2010 is the annual European conference for scientists using Python. It will be held July 8-11 2010, in ENS, Paris, France. Links: Conference website: http://www.euroscipy.org/conference/euroscipy2010 Call for papers: http://www.euroscipy.org/card/euroscipy2010_call_for_papers Practical information: http://www.euroscipy.org/card/euroscipy2010_practical_information

1 0

Array input
by lumtegis＠gmail.com 16 May '10

16 May '10

Hi all, Am creating a script to do least square adjustment of levelling data. How do I get user input into a 1D array of intergers. Thanks in advance. Allan maungu.

2 1

EuroScipy abstract submission deadline extended
by Gael Varoquaux 16 May '10

16 May '10

Given that we have been able to turn on registration only very late, the EuroScipy conference committee is extending the deadline for abstract submission for the 2010 EuroScipy conference. On Thursday May 20th, at midnight Samoa time, we will turn off the abstract submission on the conference site. Up to then, you can modify the already-submitted abstract, or submit new abstracts. We are very much looking forward to your submissions to the conference. Ga�l Varoquaux Nicolas Chauvat -- EuroScipy 2010 is the annual conference for scientists using Python. It will be held July 8-11 2010, in ENS, Paris, France. Links: Conference website: http://www.euroscipy.org/conference/euroscipy2010 Call for papers: http://www.euroscipy.org/card/euroscipy2010_call_for_papers Practical information: http://www.euroscipy.org/card/euroscipy2010_practical_information

1 0

Bug with how numpy.distutils.system_info handles the site.cfg
by Chris Colbert 15 May '10

15 May '10

I had this problem back in 2009 when building Enthought Enable, and was happy with a work around. It just bit me again, and I finally got around to drilling down to the problem. On linux, if one uses the numpy/site.cfg [default] section when building from source to specifiy local library directories, the x11 libs won't be found by NumPy. The relevant section of the site.cfg.example reads as follows: # Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # #[DEFAULT] #library_dirs = /usr/local/lib #include_dirs = /usr/local/include Now, I build NumPy with Atlas and my Atlas libs are installed in /usr/local, so my [default] section of site.cfg looks like this (as suggested by the site.cfg.example): # Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # [DEFAULT] library_dirs = /usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/local/include NumPy builds and works fine with this. The problem occurs when other libraries use numpy.distutils.system_info.get_info('x11') (ala Enthought Enable). That function eventually calls numpy.distutils.system_info.system_info.parse_config_files which has the following definition: def parse_config_files(self): self.cp.read(self.files) if not self.cp.has_section(self.section): if self.section is not None: self.cp.add_section(self.section) When self.cp is instantiated (when looking for the x11 libs), it is provided the following defaults: {'libraries': '', 'src_dirs': '.:/usr/local/src', 'search_static_first': '0', 'library_dirs': '/usr/X11R6/lib64:/usr/X11R6/lib:/usr/X11/lib64:/usr/X11/lib:/usr/lib64:/usr/lib', 'include_dirs': '/usr/X11R6/include:/usr/X11/include:/usr/include'} As is clearly seen, the 'library_dirs' contains the proper paths to find the x11 libs. But since the config file has [default] section, these paths get trampled and replaced with whatever is contained in the site.cfg [default] section. In my case, this is /usr/local/lib:/usr/local/lib/atlas. Thus, my x11 libs aren't found and the Enable build fails. The workaround is to include an [x11] section in site.cfg with the appropriate paths, but I don't really feel this should be necessary. Would the better behavior be to look for a [default] section in the config file in the parse_config_files method and add those paths to the already specified defaults? Changing the site.cfg [default] section to read as follows: [DEFAULT] library_dirs = /usr/lib:/usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/include:/usr/local/include is not an option because then NumPy will find and use the system atlas, which in my case is not threaded nor optimized for my machine. If you want me to patch the parse_config_files method, just let me know. Cheers, Chris

2 2

Problems creating numpy.array with a dtype
by Jimmie Houchin 15 May '10

15 May '10

Hello, I am really liking Numpy a lot. It is wonderful to be able to do the things that it does in a language as friendly as Python, and with the performance Numpy delivers over standard Python. Thanks. I am having a problem with creation of Numpy arrays with my generated dtypes. I am creating a dataset of a weeks worth of financial instruments data, in order to explore and test relationships with various Technical Analysis functions. My basic data is simply a list (or tuple) of lists (or tuples). ((startdate, bidopen, bidhigh, bidlow, bidclose, askopen, askhigh, asklow, askclose), ...) Nothing unusual. However I am creating arrays which have many, many more columns to allow storing the data generated from applying the functions to the original data. I have created two functions. One to dynamically create the dtype based on data I want to create for the exploration. And another to create the array and populate it with the initial data from a database. Code slightly modified, not tested. #examples taFunctions = (smva, wmva) inputColumns = (bidclose, ohlcavg) def createDType(): """Will create a dtype based on the pattern for naming and the parameters of those items being stored in the array. """ dttypes = [('startdate','object'), ('bidopen','f8'), ('bidhigh','f8'), ('bidlow','f8'), ('bidclose','f8'), ('askopen','f8'), ('askhigh','f8'), ('asklow','f8'), ('askclose','f8'), ('ocavg','f8'), ('hlavg','f8'), ('ohlavg','f8'), ('ohlcavg','f8'), ('direction','i1'), ('volatility', 'f8'), ('spread', 'f8'), ('pivot', 'S4')] for f in taFunctions: for i in inputColumns: dttypes.append((f+"-"+i,'f8')) dtminute = np.dtype(dttypes) return dtminute, dttypes def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() wdata = [] lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') for m in weekData: data = list(m)+lst[9:] wdata.append(data) return np.array(wdata,dtype=dtminute) The createDType() function works fine. The getArray() function fails with: ValueError: Setting void-array with object members using buffer. However changing the getArray() function to this works just fine. def getArray(instrument, weekString=None): ... cur.execute(sql) weekData = cur.fetchall() arrayLength = len(weekData) lst = [] dtminute, dttypes = createDType() for i in dttypes: if i[1] == 'f8': lst.append(0.0) elif i[1] == 'i1': lst.append(0) else: lst.append('') listLength = len(lst) weekArray = np.zeros(arrayLength, dtype=dtminute) for i in range(arrayLength): for j in range(listLength): if j < 9: weekArray[i][j] = weekData[i][j] else: weekArray[i][j] = lst[j] return weekArray After I finally worked out getArray number two I am back in business writing the rest of my app. But I banged my head on version number one for quite some time trying to figure out what I am doing wrong. I still don't know. I find no errors in my data length or types. I would thing that either would cause version two to fail also. In help in understanding is greatly appreciated. This is using Numpy 1.4.1. Thanks. Jimmie

4 5

default behavior of argsort
by Dr. Phillip M. Feldman 15 May '10

15 May '10

When operating on an array whose last dimension is unity, the default behavior of argsort is not very useful: |6> x=random.random((4,1)) |7> shape(x) <7> (4, 1) |8> argsort(x) <8> array([[0], [0], [0], [0]]) |9> argsort(x,axis=0) <9> array([[0], [2], [1], [3]]) -- View this message in context: http://old.nabble.com/default-behavior-of-argsort-tp28542476p28542476.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

4 7

memory leak?
by Brian Blais 15 May '10

15 May '10

Hello, I have the following code, where I noticed a memory leak with +=, but not with + alone. import numpy m=numpy.matrix(numpy.ones((23,23))) for i in range(10000000): m+=0.0 # keeps growing in memory # m=m+0.0 # is stable in memory My version of python is 2.5, numpy 1.3.0, but it also causes memory build-up in 2.6 with numpy 1.4.0, as distributed by the Enthought Python Distribution. It's easy to work around, but could cause someone some problems. Anyone else get this? bb -- Brian Blais bblais(a)bryant.edu http://web.bryant.edu/~bblais http://bblais.blogspot.com/

2 3

recarray question
by Vincent Davis 14 May '10

14 May '10

The setup: >>> Adata array([(1, 24, 'Male', '', 212, 193, 'High Pass'), (2, 26, 'Male', 'Caucasian', 234, 221, 'Honors'), (3, 31, 'Female', 'Caucasian', 182, 189, ''), (4, 27, 'Female', 'Hispanic', 214, 211, 'High Pass'), (5, 27, 'Female', 'Asian', 213, 204, 'Pass'), (6, 29, 'Female', 'Caucasian', 209, -1, 'High Pass'), (7, 26, 'Female', 'Hispanic', 212, -1, 'Honors'), (8, 25, 'Female', 'Caucasian', 230, 238, 'Honors'), (9, 27, 'Female', 'Caucasian', 239, 245, 'Honors'), (10, 26, 'Male', 'Caucasian', 226, -1, 'Honors')], dtype=[('ID', '<i8'), ('Age', '<i8'), ('Gender', '|S6'), ('Ethnicity', '|S12'), ('USMLE_1', '<i8'), ('USMLE_2', '<i8'), ('Medicine_Clerk', '|S9')]) >>> BData array([(1,), (1,), (0,), (0,), (0,), (0,), (0,), (0,), (0,), (1,)], dtype=[('Gender', '<i8')]) How do replace the "Gender" column in Adata with that in BData and update the datatype. >>> Adata['Gender'] = Bdata['Gender'] >>> Adata array([(1, 24, '1', '', 212, 193, 'High Pass'), (2, 26, '1', 'Caucasian', 234, 221, 'Honors'), (3, 31, '0', 'Caucasian', 182, 189, ''), (4, 27, '0', 'Hispanic', 214, 211, 'High Pass'), (5, 27, '0', 'Asian', 213, 204, 'Pass'), (6, 29, '0', 'Caucasian', 209, -1, 'High Pass'), (7, 26, '0', 'Hispanic', 212, -1, 'Honors'), (8, 25, '0', 'Caucasian', 230, 238, 'Honors'), (9, 27, '0', 'Caucasian', 239, 245, 'Honors'), (10, 26, '1', 'Caucasian', 226, -1, 'Honors')], dtype=[('ID', '<i8'), ('Age', '<i8'), ('Gender', '|S6'), ('Ethnicity', '|S12'), ('USMLE_1', '<i8'), ('USMLE_2', '<i8'), ('Medicine_Clerk', '|S9')]) Ok so now I need to change the dtype, not having much luck with that , do I need to copy the array? Is there a nice way to do this? That is swap Adata['Gender'] = Bdata['Gender'] and update the dtype at the same time?

1 0

missing='' not documented in genfromtxt()
by Vincent Davis 14 May '10

14 May '10

Maybe I am missing something but it does not appear that missing='' is documented although it is shown s an argument ? genfromtxt(fname, dtype=float, comments='#', delimiter=None, skiprows=0, skip_header=0, skip_footer=0, converters=None, missing='', missing_values=None, filling_values=None, usecols=None, names=None, excludelist=None, deletechars=None, autostrip=False, case_sensitive=True, defaultfmt="f%i", unpack=None, usemask=False, loose=True, invalid_raise=True):

2 3

ANN: MDP release 2.6 and MDP Sprint 2010
by Tiziano Zito 14 May '10

14 May '10

We are glad to announce release 2.6 of the Modular toolkit for Data Processing (MDP). MDP is a Python library of widely used data processing algorithms that can be combined according to a pipeline analogy to build more complex data processing software. The base of available algorithms includes, to name but the most common, Principal Component Analysis (PCA and NIPALS), several Independent Component Analysis algorithms (CuBICA, FastICA, TDSEP, JADE, and XSFA), Slow Feature Analysis, Restricted Boltzmann Machine, and Locally Linear Embedding. What's new in version 2.6? -------------------------- - Several new classifier nodes have been added. - A new node extension mechanism makes it possible to dynamically add methods or attributes for specific features to node classes, enabling aspect-oriented programming in MDP. Several MDP features (like parallelization) are now based on this mechanism, and users can add their own custom node extensions. - BiMDP is a large new package in MDP that introduces bidirectional data flows to MDP, including backpropagation and even loops. BiMDP also enables the transportation of additional data in flows via messages. - BiMDP includes a new flow inspection tool, that runs as as a graphical debugger in the webrowser to step through complex flows. It can be extended by users for the analysis and visualization of intermediate data. - As usual, tons of bug fixes The new additions in the library have been thoroughly tested but, as usual after a public release, we especially welcome user's feedback and bug reports. MDP Sprint 2010 --------------- Following our tradition of sprint-driven development, the team of the core developers decided to organize a programming sprint open to external participants. We invite in particular all users who implemented new algorithms and would like to see them integrated in MDP: you will work together with a core developer! More info: http://sourceforge.net/apps/mediawiki/mdp-toolkit/index.php?title=MDP_Sprin… Resources --------- Download: http://sourceforge.net/projects/mdp-toolkit/files Homepage: http://mdp-toolkit.sourceforge.net Mailing list: http://lists.sourceforge.net/mailman/listinfo/mdp-toolkit-users -- Pietro Berkes Volen Center for Complex Systems Brandeis University Waltham, MA, USA Rike-Benjamin Schuppner Berlin, Germany Niko Wilbert Institute for Theoretical Biology Humboldt-University Berlin, Germany Tiziano Zito Modelling of Cognitive Processes Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany

1 0

mac os x installation
by Nicola 14 May '10

14 May '10

Hi. I've to admit that I'm quite new in python and also to numpy. I'm trying to install numpy and scypy on my mac (Mac Os X 10.6.3). I've installed the last version of python (Python 2.6.5 Mac OS X Installer Disk Image<http://www.python.org/ftp/python/2.6.5/python-2.6.5-macosx10.3-2010-03-24.d…> ). After than I've downloaded the latest version of numpy ( numpy-1.4.1-py2.6-python.org.dmg<http://sourceforge.net/projects/numpy/files/NumPy/1.4.1/numpy-1.4.1-py2.6-p…>) and finally also the latest version of scypy ( scipy-0.7.2-py2.6-python.org.dmg<http://sourceforge.net/projects/scipy/files/scipy/0.7.2/scipy-0.7.2-py2.6-p…> ). When I try the test I obtain the following errors Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test('1','10') Running unit tests for numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 326, in test self._show_system_info() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 187, in _show_system_info nose = import_nose() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 69, in import_nose raise ImportError(msg) ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose >>> any idea? -- ---------------------------------------------- nicola.vianello(a)gmail.com skype:nicolavianello ----------------------------------------------

2 2

EuroScipy is finally open for registration
by Gael Varoquaux 13 May '10

13 May '10

Registration for EuroScipy <http://www.euroscipy.org//conference/euroscipy2010> is finally open To register, go to the website <http://www.euroscipy.org//conference/euroscipy2010>, create an account, and you will see a /‘register to the conference’/ button on the left. Follow it to a page which presents a /’shoping cart’/. Simply submitting this information registers you to the conference, and on the left of the website, the button will now display /‘You are registered for the conference’/. The registration fee is 50 euros for the conference, and 50 euros for the tutorial. Right now there is no payment system: you will be contacted later (in a week) with instructions for paying. We apologize for such a late set up. We do realize this has come as an inconvenience to people. *Do not wait to register: the number of people we can host is limited.* An exciting program Tutorials: from beginners to experts We have two tutorial tracks: * *Introductory tutorial* <http://www.euroscipy.org/track/871>: to get you to speed on scientific programming with Python. * *Advanced tutorial* <http://www.euroscipy.org/track/872>: experts sharing their knowledge on specific techniques and libraries. We are very fortunate to have a top notch set of presenters. Scientific track: doing new science in Python Although the abstract submission is not yet over, We can say that we are going to have a rich set of talks, looking at the current submissions. In addition to the contributed talks, we have: * *Keynote speakers* <http://www.euroscipy.org/conference/euroscipy2010>: Hans Petter Langtangen and Konrard Hinsen, two major player of scientific computing in Python. * *Lightning talks* <http://www.euroscipy.org/talk/937>: one hour will be open for people to come up and present in a flash an interesting project. Publishing papers We are talking with the editors of a major scientific computing journal, and the odds are quite high that we will be able to publish a special issue on scientific computing in Python based on the proceedings of the conference. The papers will undergo peer-review independently from the conference, to ensure high quality of the final publication. Call for papers Abstract submission is still open, though not for long. We are soliciting contributions on scientific libraries and tools developed with Python and on scientific or engineering achievements using Python. These include applications, teaching, future development directions, and current research. See the call for papers <http://www.euroscipy.org/card/euroscipy2010_call_for_papers>. *We are very much looking forward to passionate discussions about Python in science in Paris* *Nicolas Chauvat and Gaël Varoquaux*

1 0

EuroScipy is finally open for registration
by Gaël Varoquaux 13 May '10

13 May '10

The registration for EuroScipy<http://www.euroscipy.org//conference/euroscipy2010>is finally open. To register, go to the website<http://www.euroscipy.org//conference/euroscipy2010>, create an account, and you will see a *‘register to the conference’* button on the left. Follow it to a page which presents a *’shoping cart’*. Simply submitting this information registers you to the conference, and on the left of the website, the button will now display *‘You are registered for the conference’*. The registration fee is 50 euros for the conference, and 50 euros for the tutorial. Right now there is no payment system: you will be contacted later (in a week) with instructions for paying. We apologize for such a late set up. We do realize this has come as an inconvenience to people. *Do not wait to register: the number of people we can host is limited.* An exciting program Tutorials: from beginners to experts We have two tutorial tracks: - *Introductory tutorial* <http://www.euroscipy.org/track/871>: to get you to speed on scientific programming with Python. - *Advanced tutorial* <http://www.euroscipy.org/track/872>: experts sharing their knowledge on specific techniques and libraries. We are very fortunate to have a top notch set of presenters. Scientific track: doing new science in Python Although the abstract submission is not yet over, We can say that we are going to have a rich set of talks, looking at the current submissions. In addition to the contributed talks, we have: - *Keynote speakers* <http://www.euroscipy.org/conference/euroscipy2010>: Hans Petter Langtangen and Konrard Hinsen, two major player of scientific computing in Python. - *Lightning talks* <http://www.euroscipy.org/talk/937>: one hour will be open for people to come up and present in a flash an interesting project. Publishing papers We are talking with the editors of a major scientific computing journal, and the odds are quite high that we will be able to publish a special issue on scientific computing in Python based on the proceedings of the conference. The papers will undergo peer-review independently from the conference, to ensure high quality of the final publication. Call for papers Abstract submission is still open, though not for long. We are soliciting contributions on scientific libraries and tools developed with Python and on scientific or engineering achievements using Python. These include applications, teaching, future development directions, and current research. See the call for papers<http://www.euroscipy.org/card/euroscipy2010_call_for_papers> . *We are very much looking forward to passionate discussions about Python in science in Paris* *Nicolas Chauvat and Gaël Varoquaux*

1 0

efficient way to manage a set of floats?
by Dr. Phillip M. Feldman 13 May '10

13 May '10

I have an application that involves managing sets of floats. I can use Python's built-in set type, but a data structure that is optimized for fixed-size objects that can be compared without hashing should be more efficient than a more general set construct. Is something like this available? -- View this message in context: http://old.nabble.com/efficient-way-to-manage-a-set-of-floats--tp28518014p2… Sent from the Numpy-discussion mailing list archive at Nabble.com.

6 10

Downcasting an array in-place?
by Joe Kington 11 May '10

11 May '10

Is it possible to downcast an array in-place? For example: x = np.random.random(10) # Placeholder for "real" data x -= x.min() x /= x.ptp() / 255 x = x.astype(np.uint8) <-- returns a copy First off, a bit of background to the question... At the moment, I'm trying to downcast a large (>10GB) array of uint16's to uint8's. I have enough RAM to fit everything into memory, but I'd really prefer to use as few copies as possible.... In the particular case of a C-ordered uint16 array to uint8 on a little-endian system, I can do this: # "x" is the big 3D array of uint16's x -= x.min() x /= x.ptp() / 255 x = x.view(np.uint8)[:, :, ::2] That works, but a) produces a non-contiguous array, and b) is awfully case-specific. Is there a way to do something similar to astype(), but have it "cannibalize" the memory of the original array? (e.g. the "out" argument in a ufunc?) Hopefully my question makes some sense to someone other than myself... Thanks! -Joe

1 0

pareto docstring
by T J 11 May '10

11 May '10

The docstring for np.pareto says: This is a simplified version of the Generalized Pareto distribution (available in SciPy), with the scale set to one and the location set to zero. Most authors default the location to one. and also: The probability density for the Pareto distribution is .. math:: p(x) = \frac{am^a}{x^{a+1}} where :math:`a` is the shape and :math:`m` the location These two statements seem to be in contradiction. I think what was meant is that m is the scale, rather than the location. For if m were equal to zero, as the first portion of the docstring states, then the entire pdf would be zero for all shapes a>0. ---- Also, I'm not quite understanding how the stated pdf is actually the same as the pdf for the generalized pareto with the scale=1 and location=0. By the wikipedia definition of the generalized Pareto distribution, if we take \sigma=1 (scale equal to one) and \mu=0 (location equal to zero), then we get: (1 + a x)^(-1/a - 1) which is normalized over $x \in (0, \infty)$. If we compare this to the distribution stated in the docstring (with m=1) a x^{-a-1} we see that it is normalized over $x \in (1, \infty)$. And indeed, the distribution requires x > scale = 1. If we integrate the generalized Pareto (with scale=1, location=0) over $x \in (1, \infty)$ then we have to re-normalize. So should the docstring say: This is a simplified version of the Generalized Pareto distribution (available in Scipy), with the scale set to one, the location set to zero, and the distribution re-normalized over the range (1, \infty). Most authors default the location to one.

5 11

chararray stripping trailing whitespace a bug?
by Neil Crighton 11 May '10

11 May '10

I've been working with pyfits, which uses numpy chararrays. I've discovered the hard way that chararrays silently remove trailing whitespace: >>> a = np.array(['a ']) >>> b = a.view(np.chararray) >>> a[0] 'a ' >>> b[0] 'a' Note the string values stored in memory are unchanged. This behaviour caused a bug in a program I've been writing, and seems like a bad idea in general. Is it intentional? Neil

4 5

check for inequalities on a list
by gerardob 10 May '10

10 May '10

I have three lists of floats of equal lenght: upper_bound, lower_bound and x. I would like to check whether lower_bound[i]<= x[i] <= upper_bound[i] for all i in range(len(x)) Which is the best way to do this? Thanks. -- View this message in context: http://old.nabble.com/check-for-inequalities-on-a-list-tp28517353p28517353.… Sent from the Numpy-discussion mailing list archive at Nabble.com.

2 1

Question about numpy.ma masking
by Gökhan Sever 10 May '10

10 May '10

Hello, I have the following arrays read as masked array. I[10]: basic.data['Air_Temp'].mask O[10]: array([ True, False, False, ..., False, False, False], dtype=bool) [12]: basic.data['Press_Alt'].mask O[12]: False I[13]: len basic.data['Air_Temp'] -----> len(basic.data['Air_Temp']) O[13]: 1758 The first item data['Air_Temp'] has only the first element masked and this result with mask attribute being created an equal data length bool array. On the other hand data['Press_Alt'] has no elements to mask yielding a 'False' scalar. Is this a documented behavior or intentionally designed this way? This is the only case out of 20 that breaks my code as following: :) IndexError Traceback (most recent call last) 130 for k in range(len(shorter)): 131 if (serialh.data['dccnTempSF'][k] != 0) \ --> 132 and (basic.data['Air_Temp'].mask[k+diff] == False): 133 dccnConAmb[k] = serialc.data['dccnConc'][k] * \ 134 physical.data['STATIC_PR'][k+diff] * \ IndexError: invalid index to scalar variable. since mask is a scalar in this case, nothing to loop terminating with an IndexError. -- Gökhan

3 4

Another masked array question
by Gökhan Sever 10 May '10

10 May '10

Hello, Consider my masked arrays: I[28]: type basic.data['Air_Temp'] -----> type(basic.data['Air_Temp']) O[28]: numpy.ma.core.MaskedArray I[29]: basic.data['Air_Temp'] O[29]: masked_array(data = [-- -- -- ..., -- -- --], mask = [ True True True ..., True True True], fill_value = 999999.9999) I[17]: basic.data['Air_Temp'].data = np.ones(len(basic.data['Air_Temp']))*30 --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) ----> 1 2 3 4 5 AttributeError: can't set attribute Why this assignment fails? I want to set each element in the original basic.data['Air_Temp'].data to another value. (Because the main instrument was forgotten to turn on for that day, and I am using a secondary measurement data for Air Temperature for my another calculation. However it fails. Although single assignment works: I[13]: basic.data['Air_Temp'].data[0] = 30 Shouldn't this be working like the regular NumPy arrays do? Thanks. -- Gökhan

4 5

Datetime overflow error, attn Stefan.
by Charles R Harris 09 May '10

09 May '10

Hi Stefan, The windows buildbot throws the error ===================================================================== FAIL: test_creation_overflow (test_datetime.TestDateTime) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\buildbot\numpy\b11\numpy-install25\Lib\site-packages\numpy\core\tests\test_datetime.py", line 68, in test_creation_overflow err_msg='Datetime conversion error for unit %s' % unit) File "..\numpy-install25\Lib\site-packages\numpy\testing\utils.py", line 313, in assert_equal AssertionError: Items are not equal: Datetime conversion error for unit ms ACTUAL: 567052800 DESIRED: 322689600000 Because window's longs are always 32 bit, I think this is a good indication that somewhere a long is being used instead of an intp.

2 2

saving object array to ascii file
by Maria Liukis 07 May '10

07 May '10

Hello Everybody, Sorry if it's a trivial question. I'm trying to find out if there is a way to save object array to ascii file. numpy.savetxt() in NumPy V1.3.0 doesn't seem to work: >>> import numpy as np >>> obj_arr = np.zeros((2,), dtype=np.object) >>> obj_arr[0] = np.array([[1,2,3], [4,5,6], [7,8,9]]) >>> obj_arr[1] = np.array([[10,11], [12,13]]) >>> obj_arr array([[[1 2 3] [4 5 6] [7 8 9]], [[10 11] [12 13]]], dtype=object) >>> np.savetxt('obj_array.dat', obj_arr) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/io.py", line 636, in savetxt fh.write(format % tuple(row) + '\n') TypeError: float argument required >>> scipy.io.savemat() supports Matlab format of the object array, but I could not find any documentation on ASCII file format for object arrays. Thanks in advance, Masha -------------------- liukis(a)usc.edu

2 1

Decision tree-like algorithm on numpy arrays
by Martin Raspaud 07 May '10

07 May '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, I have an old c-extension I want to remove from my code to the benefit of numpy, but it looks kind of tricky to me. Here is the thing: I have a number of arrays of the same shape. On these arrays, I run a sequence of tests, leading to a kind of decision tree. In the end, based on these tests, I get a number of result arrays where, based on the tests, each element gets a value. The way to do this in an efficient way with numpy is quite unclear to me. My first thought would be: result_array1 = np.where(some_test_on(array1), np.where(some_test_on(array2), 1, 2), np.where(some_test_on(array3, array4), np.where(some_test_on(array5), 3, 4), 4)) result_array2 = np.where(some_test_on(array1), np.where(some_test_on(array2), True, True), np.where(some_test_on(array3, array4), np.where(some_test_on(array5), True, False), True)) etc... but that means running the same tests several times, which is not acceptable if the tests are lengthy. In order to avoid this problem I could also have some mask based on each test: mask1 = some_test_on(array1) mask2 = some_test_on(array2[mask1]) mask3 = some_test_on(array3[!mask1], array4[!mask1]) mask4 = some_test_on(array5[!mask1][mask3]) result_array1[mask1][mask2] = 1 result_array1[mask1][!mask2] = 2 result_array1[!mask1][mask3][mask4] = 3 result_array1[!mask1][mask3][!mask4] = 4 result_array1[!mask1][!mask3] = 4 result_array2[mask1][mask2] = True result_array2[mask1][!mask2] = True result_array2[!mask1][mask3][mask4] = True result_array2[!mask1][mask3][!mask4] = False result_array2[!mask1][!mask3] = True etc... but that looks a bit clumsy to me... The way it was done in the C-extension was to run the decision tree on each element sequentially, but I have the feeling that would not be very efficient with numpy (although I know I can't beat pure C code, I would like to have comparable times). Does any of you wise people have an opinion on this ? Thanks, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJL4ma5AAoJEBdvyODiyJI4SpgH/i0bb7PH8oTu481NRuYmbi40 VwJrOCdfSo6CauLBiIdxBZV2Hksbu2iDu5GEKJNUObf9bM7N+LK+qMwaBq1M5hF+ 47yNczSEUaxshBHzUFQMlS9XEtZewhYZGepkH1oThIQbSD2IbM6fWkVj+EJRwwJ5 2Ia4p1GIdLGMZ3loaWevvCmz8kjppX7Feei0hEP28+HIiWq/qmUlccYZm/ThZcFE 6ROEKtkepKsf3vOfpuS5Hr6U1Hb4mo7u9SmUcOvlCby6q/TbVtwAZjpRQB4qKEjm DRj9EvyWBnINgr3tKVN2Cida1El8Ki9jBjhx2GxLsy78pNKqZMI9UC/iM8cehYQ= =hUa2 -----END PGP SIGNATURE-----

2 3

Brooken Toolchain (Mac 10.6)
by Patrick Marsh 07 May '10

07 May '10

Greetings, Per my previous email, I'm trying to setup the release process for Numpy on my MacBook Pro. When trying to build Numpy 1.4.1r3 with Python 2.5.4 I get a broken toolchain error (below). I do not get this error when trying to build Numpy with Python 2.6.5 - and there is nothing fundamentally different (that I know of) between my Python 2.5.4 and Python 2.6.5 environments. To address a previous suggestion I received offline, I do have sufficient permissions (I've even tried using sudo, just to make sure) and I do have setuptools installed. Any help, would be appreciated. Patrick ======================================================================== compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' gcc: _configtest.c cc1: error: unrecognized command line option "-Wno-long-double" cc1: error: unrecognized command line option "-Wno-long-double" lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out cc1: error: unrecognized command line option "-Wno-long-double" cc1: error: unrecognized command line option "-Wno-long-double" lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 187, in <module> setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/Users/pmarsh/git/numpy.release/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 657, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program -- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com

4 7

Remove duplicate columns
by T J 07 May '10

07 May '10

Hi, Is there a way to sort the columns in an array? I need to sort it so that I can easily go through and keep only the unique columns. ndarray.sort(axis=1) doesn't do what I want as it destroys the relative ordering between the various columns. For example, I would like: [[2,1,3], [3,5,1], [0,3,1]] to go to: [[1,2,3], [5,3,1], [3,0,1]] (swap the first and second columns). So I want to treat the columns as objects and sort them. I can do this if I convert to a python list, but I was hoping to avoid doing that because I ultimately need to do element-wise bitwise operations. Thanks!

4 6

trouble with bool_
by Alan G Isaac 06 May '10

06 May '10

What information exactly is `isnan` supposed to communicate? Put another way, given that it raises NotImplemented for unknown types, and that bool(NotImplemented) is True, is there a reason by it cannot return a Python bool (which seems more useful)? Thanks, Alan Isaac > >> np.isnan(np.nan) True > >> np.isnan(np.nan) is True False > >> type(np.isnan(np.nan)) <type 'numpy.bool_'> > >> np.isnan('') NotImplemented > >> bool(_) True > >> if np.isnan(''): print "Uh oh" ... Uh oh

1 0

SciPy 2010: Bioinformatic & Parallel/cloud talks announced...& register now!
by Amenity Applewhite 06 May '10

06 May '10

Hello! Things are moving quickly in preparation for SciPy 2010: Last week we announced the General Conference schedule (http://conference.scipy.org/scipy2010/schedule.html ), Tuesday we announced our student sponsorship recipients (http://conference.scipy.org/scipy2010/student.html ) and now we're ready to tell you give you a look at the talks we have lined up for our Bioinformatics and Parallel Processing /Cloud Computing tracks. ===Parallel Processing & Cloud Computing track=== We really appreciate Brian and Ken's work organizing the papers for this specialized track. And of course, thanks to everyone who submitted a paper. There has been a great deal of interest in this set of talks — and word on the street is that Brian may even have a HPC tutorial up his sleeve... * StarCluster - NumPy/SciPy Computing in the Cloud- Justin Riley * pomsets: workflow management for your cloud- Michael J Pan * Getting Down with Big Data Jared Flatow, Anita Lillie, Ville Tuulos * StarFlow: A Cloud-Enables Python Workflow Engine for Scientific Analysis Pipelines Elaine Angelino, Dan Yamins, Margo Seltzer * A Programmatic Interface for Particle Plasma Simulation in Python, and Early Backend Results with PyCUDA Min Ragan-Kelley * Parallel Computing with IPython: an Application to Air Pollution Modeling B.E. Granger, J.G. Hemann * Astronomy App in the Cloud using Google Geo APIs and Python App Engine Shawn Shen ===Bioinformatics track=== Once again, we are indebted to Glen Otero, from Dell, for putting together the Bioinformatics track. He received some fantastic papers and we're really looking forward to these presentations: * Protein Folding with Python on Supercomputers Jan H. Meinke * Can Python Save Next-Generation Sequencing? * The Use of Galaxy for the Research and the Teaching of Genomics Roy Weckiewicz, Jim Hu, and Rodolfo Aramayo ===Early registration ends next Monday=== That's right: Only a few days left before rates increase! Think of all the BBQ and breakfast tacos you can buy with that $50-$100 you'll save by registering early. If that doesn't convince you, consider: -Cheap flights to Austin- Buy your tickets now for some very nice prices: $275 from Chicago, $330 from San Francisco, $380 from New York City, $810 from London...(prices from Kayak.com) -Convenient & affordable hotel- We got an fantastic deal for on-site accommodations at the AT&T Conference Center. Pay only $89/night for single occupancy or $105/ night for double occupancy. It will be great to have everyone staying in the same spot. Once you register, you'll get a code to book your hotel reservation. The discounted rate will be applied automatically. https://conference.scipy.org/scipy2010/accommodation.html No car necessary to get to the conference... and see Austin! An airport bus (http://capmetro.org/riding/current_schedules/maps/rt100_sb.pdf ) runs straight to and from the AT&T center, so you won't have to rent a car at all. Plus, the UT campus area is in walking distance to a number of great restaurants and activities. For any longer trips you'd like to make Austin has a great public bus system. Not to mention all of the mind-blowing things you'll learn and outstanding people you'll meet and catch up with. So what are you waiting for? Register: https://conference.scipy.org/scipy2010/registration.html Best, The SciPy 2010 Team @SciPy2010 on Twitter

1 0

PY_ARRAY_UNIQUE_SYMBOL is too far reaching?
by Austin Bingham 06 May '10

06 May '10

Hi everyone, I've recently been developing a python module and C++ library in parallel, with core functionality in python and C++ largely just layered on top of the python (with boost.python.) In some cases, however, for performance reasons, the C++ API "reaches into" the python code via the C API, and this tends to happen very often with numpy-related code. As a result, I'm using the numpy C API a lot in my C++ library. To make a long story short, I'm finding that there are many places where I need to include numpy headers in my own headers (e.g. when a template class uses part of the numpy API.) If the symbol I want in my header is in ndarrayobject.h, it seems that I'm obligated to define PY_ARRAY_UNIQUE_SYMBOL because that file includes __multiarray_api.h. However, defining that macro in a header file seems like a bad idea because of potential conflicts with headers from other libraries. As a motivating example, I have a header which implements a type-mapping template for numpy types. It maps at compile time between the NPY_TYPES enum and actual C++ types. To get the NPY_TYPES definitions, I have to include arrayobject.h. Even though my header doesn't actually use the symbols that PY_ARRAY_UNIQUE_SYMBOL influences, it seems that I need to define it; even without it being defined, __multiarray_api.h leaves a PyArray_API definition in my header. So, am I missing something obvious? Or is there no way to access symbols like NPY_TYPES without pulling in the PyArray_API functions? If there isn't, would it be possible to restructure the headers so that the symbols affected by PY_ARRAY_UNIQUE_SYMBOL are separated from the others? Thanks. Austin

5 8

embedding/extending numpy:import_array() problem
by tino 05 May '10

05 May '10

Hi guys, I've a C++ code that calls a python script. The python script, in turn, should use C modules to retrieve data (arrays). I'm having some problem with import_array. I've placed the import_array() just after the initialization of the interpreter and after PyInit_module Py_Initialize(); Py_InitModule("py_c_if", PY_C_IF_Methods); import_array(); This are inside a function called main_PYinit,called within the main C function. When I tryed to compile, I got the at the line of import_array the error "function must return a value". If I change the __multiarray_api.h and make import_array() a function with void instead of #define, the compiltion succeed. However, when then I try to use some API function I get problems.In particular I used this code: char *camera_buffer = cam_data; npy_int dim[] = {rows,cols}; /* whatever the size of your data, in C-order */ PyObject *myarray; double *array_buffer; myarray = PyArray_SimpleNew(2, &dim, NPY_INT); Py_INCREF(myarray); array_buffer = (double *)PyArray_DATA(myarray); for (int i=0; i<rows*cols; i++) *array_buffer++ = (double) *camera_buffer++; return PyArray_Return(myarray); And I goit this error: .\EyeSeeMain.cpp(242) : error C2664: 'PyObject *(PyTypeObject *,int,npy_intp *,int,npy_intp *,void *,int,int,PyObject *)' : cannot convert parameter 2 from 'npy_intp *' to 'int' for all the paramters of PyArray_SimpleNew. It seems that the complier did ot see the definition of PyAray_SimpleNew... Any help is grealy appreciated. Cheers L

1 0

Improvement of performance
by gerardob 05 May '10

05 May '10

Hello, I have written a very simple code that computes the gradient by finite differences of any general function. Keeping the same idea, I would like modify the code using numpy to make it faster. Any ideas? Thanks. def grad_finite_dif(self,x,user_data = None): assert len(x) == self.number_variables points=[] for j in range(self.number_variables): points.append(x.copy()) points[len(points)-1][j]=points[len(points)-1][j]+0.0000001 delta_f = [] counter=0 for j in range(self.number_variables): delta_f.append((self.eval(points[counter])-self.eval(x))/0.0000001) counter = counter + 1 return array(delta_f) -- View this message in context: http://old.nabble.com/Improvement-of-performance-tp28452458p28452458.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

5 8

proposing a "beware of [as]matrix()" warning
by David Warde-Farley 04 May '10

04 May '10

Trying to debug code written by an undergrad working for a colleague of mine who ported code over from MATLAB, I am seeing an ugly melange of matrix objects and ndarrays that are interacting poorly with each other and various functions in SciPy/other libraries. In particular there was a custom minimizer function that contained a line "a * b", that was receiving an Nx1 "matrix" and a N-length array and computing an outer product. Hence the unexpected 6 GB of memory usage and weird results... We've had this discussion before and it seems that the matrix class isn't going anywhere (I *really* wish it would at least be banished from the top-level namespace), but it has its adherents for pedagogical reasons. Could we at least consider putting a gigantic warning on all the functions for creating matrix objects (matrix, mat, asmatrix, etc.) that they may not behave quite so predictably in some situations and should be avoided when writing nontrivial code? There are already such warnings scattered about on SciPy.org but the situation is so bad, in my opinion (bad from a programming perspective and bad from a new user perspective, asking "why doesn't this work? why doesn't that work? why is this language/library/etc. so stupid, inconsistent, etc.?") that the situation warrants steering people still further away from the matrix object. I apologize for ranting, but it pains me when people give up on Python/NumPy because they can't figure out inconsistencies that aren't really there for a good reason. IMHO, of course. David David

16 32

Poll: Semantics for % in Cython
by Dag Sverre Seljebotn 04 May '10

04 May '10

(First off, is it OK to continue polling the NumPy list now and then on Cython language decisions? Or should I expect that any interested Cython users follow the Cython list?) In Python, if I write "-1 % 5", I get 4. However, in C if I write "-1 % 5" I get -1. The question is, what should I get in Cython if I write (a % b) where a and b are cdef ints? Should I [ ] Get 4, because it should behave just like in Python, avoiding surprises when adding types to existing algorithms (this will require extra logic and be a bit slower) [ ] Get -1, because they're C ints, and besides one isn't using Cython if one doesn't care about performance Whatever we do, this also affects the division operator, so that one in any case will have a==(a//b)*b+a%b. (Orthogonal to this, we can introduce compiler directives to change the meaning of the operator from the default in a code blocks, and/or make special functions for the semantics that are not chosen as default.) -- Dag Sverre

10 16

incremental histogram
by Neal Becker 04 May '10

04 May '10

I have coded in c++ a histogram object that can be used as: h += my_sample or h += my_vector This is very useful in simulations which are looping and developing results incrementally. It would me great to have such a feature in numpy.

2 3

datetime failure on py3k, string type issue.
by Charles R Harris 04 May '10

04 May '10

The following fails after fixing datetime_data assert_equal(datetime_data(a.dtype), ('us', 1, 1, 1)) The problem is that 'us' is unicode and the function call yields bytes. The question is: should datetime units use unicode when compiled on python >= 3k? Chuck

1 0

ndimage.label - howto force SWIG to use int32 - even on 64bit Linux ?
by Sebastian Haase 04 May '10

04 May '10

Hi, I wanted to write some C code to accept labels as they come from ndimage.label. For some reason ndimage.label produces its output as an int32 array - even on my 64bit system . BTW, could this be considered a bug ? Now, if I use the typemaps of numpy.i I can choose between NPY_LONG and NPY_INT. But those are sometimes 32 sometimes 64 bit, depending on the system. Any ideas ... ? Thanks, Sebastian Haase

6 10

&*(*&$%# Macros
by Charles R Harris 03 May '10

03 May '10

Hi Travis, Could you remove this macro? It isn't py3k compliant because of the PyCobject and we have too many public macros already. Please make it part of the interface if it needs to be exposed, otherwise make it an inline function somewhere. #define PyDataType_GetDatetimeMetaData(descr) \ ((descr->metadata == NULL) ? NULL : \ ((PyArray_DatetimeMetaData *)(PyCObject_AsVoidPtr( \ PyDict_GetItemString(descr->metadata, NPY_METADATA_DTSTR))))) Chuck

2 2

Broken numpy tests in python3k
by Charles R Harris 03 May '10

03 May '10

They seem to be string/unicode related. 1) This error remains after replacing the disappeared built in function "file" with "open" ====================================================================== ERROR: test_universal_newline (test_io.TestLoadTxt) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.1/site-packages/numpy/lib/tests/test_io.py", line 412, in test_universal_newline data = np.loadtxt(name) File "/usr/local/lib/python3.1/site-packages/numpy/lib/npyio.py", line 632, in loadtxt first_vals = split_line(first_line) File "/usr/local/lib/python3.1/site-packages/numpy/lib/npyio.py", line 610, in split_line line = line.split(comments)[0].strip() TypeError: Can't convert 'bytes' object to str implicitly 2) Not sure about this one, it doesn't show up for Python 2k ====================================================================== ERROR: test_type_check.TestDateTimeData.test_basic ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.1/site-packages/nose/case.py", line 177, in runTest self.test(*self.arg) File "/usr/local/lib/python3.1/site-packages/numpy/lib/tests/test_type_check.py", line 382, in test_basic assert_equal(datetime_data(a.dtype), ('us', 1, 1, 1)) File "/usr/local/lib/python3.1/site-packages/numpy/lib/type_check.py", line 641, in datetime_data result = func(obj) TypeError: this function takes at least 2 arguments (1 given) Chuck

1 0

Question about numpy.arange()
by Gökhan Sever 03 May '10

03 May '10

Hello, Is "b" an expected value? I am suspecting another floating point arithmetic issue. I[1]: a = np.arange(1.6, 1.8, 0.1, dtype='float32') I[2]: a O[2]: array([ 1.60000002, 1.70000005], dtype=float32) I[3]: b = np.arange(1.7, 1.8, 0.1, dtype='float32') I[4]: b O[4]: array([ 1.70000005, 1.79999995], dtype=float32) A bit conflicting with the np.arange docstring: "* Values are generated within the half-open interval ``[start, stop)`` (in other words, the interval including `start` but excluding `stop`). * " Thanks. -- Gökhan

6 6

numpy + MKL and AMD64 CPU, recognized as 32 bit
by Andrzej Giniewicz 02 May '10

02 May '10

Hi, from what I've been reading, and testing myself, 64bit CPU's from AMD and actually it seems like almost all 64bit other than IA64, use em64t in MKL - but in Numpy (at least 1.4.1), this architecture is picked only for Xeon - though older Xeons were not 64 bit iirc - I'm talking about numpy/distutils/system_info.py file, mkl_info class, __init__ function. The fragment: from cpuinfo import cpu l = 'mkl' # use shared library if cpu.is_Itanium(): plt = '64' #l = 'mkl_ipf' elif cpu.is_Xeon(): plt = 'em64t' #l = 'mkl_em64t' else: plt = '32' #l = 'mkl_ia32' with this, my AMD64 that works with em64t (that's version I have installed and that passes all tests, that is MKL 10.2.5.035) is recognized as 32 bit. Wouldn't it be better to check for is_64bit instead of is_Xeon? Thanks in advance for hints, Andrzej.

1 0

bug, issue with genfromtxt
by Vincent Davis 02 May '10

02 May '10

I ran into this issue and it was discussed on the pystatsmodels mailing list. Here is the setup Running on a Mac 10.6 Using Office 2008 Saving an spreadsheet using excel "save as" a csv file. Try to import using genfromtxt fails, report a EOL error I thought this was because the EOL was wrong, It seems the file has '\r' as the line ending (this may be wrong) anyway I changed it to '\n' and it works fine. I am told (on the pystatsmodels mailing list) that this is actually because the file is in unicode and that genfromtxt does not read the EOL correctly. To me it is a bug because one might expect a user to what to save a file from excel and read it using genfromtxt. And for useres with little experience the problem is not obvious. I guess this is not a problem with py3? ORIGINAL ATTEMPT datatype = [('date','|S9'),('gpd','i8'),('temp','i8' ('precip','f16')] data = np.genfromtxt('waterdata.csv', delimiter=',', skip_header=1, dtype=datatype) Traceback (most recent call last): File "/Applications/WingIDE.app/Contents/MacOS/src/debug/tserver/_sandbox.py", line 1, in <module> # Used internally for debug sandbox under external interpreter File "/Library/Frameworks/EPD64.framework/Versions/6.1/lib/ python2.6/site-packages/numpy/lib/io.py", line 1048, in genfromtxt raise IOError('End-of-file reached before encountering data.') IOError: End-of-file reached before encountering data. THIS DOES NOT WORK >>> s = file('data_with_CR.csv','r') >>> data = np.genfromtxt(s, delimiter=",", skip_header=1, dtype=None) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/EPD64.framework/Versions/6.1/lib/python2.6/site-packages/numpy/lib/io.py", line 1048, in genfromtxt raise IOError('End-of-file reached before encountering data.') IOError: End-of-file reached before encountering data. >>> data = np.genfromtxt(s, delimiter=",", , dtype=None) File "<stdin>", line 1 data = np.genfromtxt(s, delimiter=",", , dtype=None) THIS DOES WORK >>> s = file('data_with_CR.csv','U') >>> data = np.genfromtxt(s, delimiter=",", skip_header=1, dtype=None) >>> data array([('1/1/00', 8021472, 52, 0.02), ('1/2/00', 9496016, 46, 0.059999999999999998), ('1/3/00', 8478792, 29, 0.0), ..., ('12/29/02', 10790000, 61, 0.0), ('12/30/02', 9501000, 44, 0.0), ('12/31/02', 9288000, 53, 0.0)], dtype=[('f0', '|S8'), ('f1', '<i8'), ('f2', '<i8'), ('f3', '<f8')]) *Vincent Davis 720-301-3003 * vincent(a)vincentdavis.net my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>

1 0

NumPy-Discussion May 2010