Mailman 3 July 2008 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 17 May '11

17 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

Unpleasant behavior with poly1d and numpy scalar multiplication
by Fernando Perez 15 Feb '10

15 Feb '10

Hi all, consider this little script: from numpy import poly1d, float, float32 p=poly1d([1.,2.]) three=float(3) three32=float32(3) print 'three*p:',three*p print 'three32*p:',three32*p print 'p*three32:',p*three32 which produces when run: In [3]: run pol1d.py three*p: 3 x + 6 three32*p: [ 3. 6.] p*three32: 3 x + 6 The fact that multiplication between poly1d objects and numbers is: - non-commutative when the numbers are numpy scalars - different for the same number if it is a python float vs a numpy scalar is rather unpleasant, and I can see this causing hard to find bugs, depending on whether your code gets a parameter that came as a python float or a numpy one. This was found today by a colleague on numpy 1.0.4.dev3937. It feels like a bug to me, do others agree? Or is it consistent with a part of the zen of numpy I've missed thus far? Thanks, f

6 17

Compact way of performing array math with specified result type?
by Russell E. Owen 08 Feb '10

08 Feb '10

I often find myself doing simple math on sequences of numbers (which might or might not be numpy arrays) where I want the result (and thus the inputs) coerced to a particular data type. I'd like to be able to say: numpy.divide(seq1, seq2, dtype=float) but ufuncs don't allow on to specify a result type. So I do this instead: numpy.array(seq1, dtype=float) / numpy.array(seq2, dtype=float) Is there a more compact solution (without having to create the result array first and supply it as an argument)? -- Russell

4 3

Is anyone knowledgeable about dll deployment on windows ?
by David Cournapeau 01 Dec '09

01 Dec '09

Hi, My head hurt trying to understand dll management with windows. I wanted to find a sane way to use dll for numscons, but I can't see how to do this, so I was wondering if anyone on this ML had any deep knowledge on how to install dll, and reuse them with python extensions ? The problem is the following: - you have some dll (say MKL) in a path (say C:\program files\MKL). - how do you tell another dll (more exactly a python extension, a .pyd) to look for mkl dll into the MKL path ? It seems that there is no rpath-like capability on windows, and the only way to load dll is to put them into some windows directory, or to put the dll path into PATH. Is it really the only way ? cheers, David

8 25

Slicing/selection in multiple dimensions simultaneously
by Mike Ressler 17 Mar '09

17 Mar '09

The following seems to be a wart: is it expected? Set up a 10x10 array and some indexing arrays: a=arange(100) a.shape=(10,10) q=array([0,2,4,6,8]) r=array([0,5]) Suppose I want to extract only the "even" numbered rows from a - then print a[q,:] <works - output deleted> Every fifth column: print a[:,r] <works - output deleted> Only the even rows of every fifth column: print a[q,r] --------------------------------------------------------------------------- <type 'exceptions.ValueError'> Traceback (most recent call last) /.../.../.../<ipython console> in <module>() <type 'exceptions.ValueError'>: shape mismatch: objects cannot be broadcast to a single shape But, this works: print a[q,:][:,r] [[ 0 5] [20 25] [40 45] [60 65] [80 85]] So why does the a[q,r] form have problems? Thanks for your insights. Mike -- mike.ressler(a)alum.mit.edu

8 16

access ndarray in C++
by Thomas Hrabe 10 Dec '08

10 Dec '08

Hi all! I am currently developing a python module in C/C++ which is supposed to access nd arrays as defined by the following command in python a = numpy.array([1,1,1]) I want to access the array the following way and use the nd array data for further processing in C. mymod.doSthg(a) The example code on http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm#pgfId-36721 (!PyArg_ParseTuple(args, "O!",&PyArray_Type, &array)) does not work for nd arrays. I always get TypeError: argument 1 must be array, not numpy.ndarray I assume the error is the constant provided as the third parameter, saying that the imput is of PyArray_Type and no nd array. So here are my questions: 1. is there any good tutorial / example code for acessing nd arrays in C? 2. what is the difference between both (arrays and nd arrays? - I am new to python and heard there were different approaches to arrays and that nd arrays work better for multi dimensional applications. Is this right?) 3. which one of both will be used in the future? Thank you in advance for your help, Thomas

15 31

error importing a f2py compiled module.
by Fabrice Silva 30 Nov '08

30 Nov '08

Dear all I've tried to run f2py on a fortran file which used to be usable from python some months ago. Following command lines are applied with success (no errors raised) : f2py -m modulename -h tmpo.pyf --overwrite-signature tmpo.f f2py -m modulename -c --f90exec=/usr/bin/f95 tmpo.f The output of these commands is available here: http://paste.debian.net/7307 When importing in Python with "import modulename", I have an ImportError: Traceback (most recent call last): File "Solveur.py", line 44, in <module> import modulename as Modele ImportError: modulename.so: failed to map segment from shared object: Operation not permitted How can that be fixed ? Any suggestion ? Thanks -- Fabrice Silva LMA UPR CNRS 7051 - équipe S2M

3 6

Masked arrays and pickle/unpickle
by Anthony Floyd 07 Nov '08

07 Nov '08

We have an application that has previously used masked arrays from Numpy 1.0.3. Part of saving files from that application involved pickling data types that contained these masked arrays. In the latest round of library updates, we've decided to move to the most recent version of matplotlib, which requires Numpy 1.1. Unfortunately, when we try to unpickle the data saved with Numpy 1.0.3 in the new code using Numpy 1.1.0, it chokes because it can't import numpy.core.ma for the masked arrays. A check of Numpy 1.1.0 shows that this is now numpy.ma.core. Does anyone have any advice on how we can unpickle the old data files and update the references to the new classes? Thanks, Anthony. -- Anthony Floyd, PhD Convergent Manufacturing Technologies Inc. 6190 Agronomy Rd, Suite 403 Vancouver BC V6T 1Z3 CANADA Email: Anthony.Floyd(a)convergent.ca | Tel: 604-822-9682 x102 WWW: http://www.convergent.ca | Fax: 604-822-9659 CMT is hiring: See http://www.convergent.ca for details

5 10

huge array calculation speed
by Dan Lussier 31 Aug '08

31 Aug '08

Hello, I am relatively new to numpy and am having trouble with the speed of a specific array based calculation that I'm trying to do. What I'm trying to do is to calculate the total total potential energy and coordination number of each atom within a relatively large simulation. Each atom is at a position (x,y,z) given by a row in a large array (approximately 1e6 by 3) and presently I have no information about its nearest neighbours so each its position must be checked against all others before cutting the list down prior to calculating the energy. My current numpy code is below and in it I have tried to push as much of the work for this computation into compiled C (ideally) of numpy. However it is still taking a very long time at approx 1 second per row. At this speed even some simple speed ups like weave won't really help. Are there any other numpy ways that it would be possible to calculate this, or should I start looking at going to C/C++? I am also left wondering if there is something wrong with my installation of numpy (i.e. not making use of lapack/ATLAS). Other than that if it is relevant - I am running 32 bit x86 Centos 5.1 linux on a dual Xeon 3.0 GHz Dell tower with 4 GB of memory. Any suggestions of things to try would be great. Dan ======================================================================== ======================= def epairANDcoord(xyz,cutoff=4.0): box_length = 160.0 cut2 = cutoff**2 ljTotal = numpy.zeros(xyz.shape[0]) coord = numpy.zeros(xyz.shape[0]) i = 0 for r0 in xyz: r2 = r0-xyz # application of minimum image convention so each atom interacts with nearest periodic image w/i box_length/2 r2 -= box_length*numpy.rint(r2/box_length) r2 = numpy.power(r2,2).sum(axis=1) r2 = numpy.extract(r2<cut2, r2) coord[i] = r2.shape[0]-1 r2 = numpy.extract(r2 != 0, r2) # avoid problems of inf when considering own atom. r2 = numpy.divide(1.0,r2) ljTotal[i] = numpy.multiply(2,(numpy.power(r2,6)- numpy.power(r2,3))).sum(axis=0) # ljTotal[i] = ljPerAtom.sum(axis=0) i += 1 del r2 # if i>25: break # used to cut down the length of calculation as needed return ljTotal,coord

11 13

indexing (compared to matlab)
by Brian Blais 19 Aug '08

19 Aug '08

Hello, I wanted to do the following thing that I do in Matlab (on a bigger problem), setting the values of a part of a matrix with indexing: >> a=floor(rand(5,5)*10) % make an example matrix to work on a = 2 4 7 9 8 6 9 2 5 2 6 3 5 1 8 1 5 6 1 2 1 2 8 2 9 >> ind=[2,4] ind = 2 4 >> a(ind,ind)=a(ind,ind)+100 a = 2 4 7 9 8 6 109 2 105 2 6 3 5 1 8 1 105 6 101 2 1 2 8 2 9 =========================== In numpy, the same gives: In [11]:a=floor(random.rand(5,5)*10) In [14]:a Out[14]: array([[ 7., 7., 8., 1., 9.], [ 0., 4., 9., 0., 5.], [ 4., 3., 7., 8., 3.], [ 2., 0., 4., 2., 4.], [ 9., 5., 0., 9., 9.]]) In [15]:ind=[1,3] In [20]:a[ind,ind]+=100 In [21]:a Out[21]: array([[ 7., 7., 8., 1., 9.], [ 0., 104., 9., 0., 5.], [ 4., 3., 7., 8., 3.], [ 2., 0., 4., 102., 4.], [ 9., 5., 0., 9., 9.]]) which only replaces 2 values, not all the values in the row,col combinations of [1,1],[1,2],etc...[3,3] like matlab. Is there a preferred way to do this, which I think should be fairly common. If I know that the indices are regular (like a slice) is there a way to do this? thanks, Brian Blais -- Brian Blais bblais(a)bryant.edu http://web.bryant.edu/~bblais

4 7

NumPy-Discussion July 2008