Mailman 3 August 2006 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 17 May '11

17 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

Re: [Numpy-discussion] Floating point exception with numpy and embedded python interpreter
by Andrew Straw 20 Sep '07

20 Sep '07

Arkaitz Bitorika wrote: > > On 19 Apr 2006, at 18:37, Andrew Straw wrote: > >> >>> I've been embedding Python for ages on the same program and other >>> modules work fine, only numpy fails. >> >> >> Most other modules don't use the SSE units, so wouldn't get hit by such >> a bug. > > > Is there a way of not using those units from numpy, to check if > that's what's going on? I think that numpy only accesses the SSE units through ATLAS or other external library. So, build numpy without ATLAS. But I'm not 100% sure anymore if there aren't any optimizations that directly use SSE if it's available. > Or alternatively, how would I check if my program is messing with the > SSE bits? Hmm, I think that's a bit hairy. I'd suggest simply asking the C++ library's mailing list if they alter the error bits on the control registers of the SSE unit. (Out of curiousity, what library is it?) If you want hairy, though, I think you'd have to check from C with the appropriate calls -- I'd start with the source code in that bug report. It looks like they're inlining an assembly statement to query a SSE control register.

5 7

Numpy x Matlab: some synthetic benchmarks
by Paulo Jose da Silva e Silva 11 Feb '07

11 Feb '07

Hello, Travis asked me to benchmark numpy versus matlab in some basic linear algebra operations. Here are the resuts for matrices/vectors of dimensions 5, 50 and 500: Operation x'*y x*y' A*x A*B A'*x Half 2in2 Dimension 5 Array 0.94 0.7 0.22 0.28 1.12 0.98 1.1 Matrix 7.06 1.57 0.66 0.79 1.6 3.11 4.56 Matlab 1.88 0.44 0.41 0.35 0.37 1.2 0.98 Dimension 50 Array 9.74 3.09 0.56 18.12 13.93 4.2 4.33 Matrix 81.99 3.81 1.04 19.13 14.58 6.3 7.88 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension 500 Array 1.2 8.97 2.03 166.59 20.34 3.99 4.31 Matrix 17.95 9.09 2.07 166.62 20.67 4.11 4.45 Matlab 2.09 6.07 2.17 169.45 2.1 2.56 3.06 Obs: The operation Half is actually A*x using only the lower half of the matrix and vector. The operation 2in2 is A*x using only the even indexes. Of course there are many repetitions of the same operation: 100000 for dim 5 and 50 and 1000 for dim 500. The inner product is number of repetitions is multiplied by dimension (it is very fast). The software is numpy svn version 1926 Matlab 6.5.0.180913a Release 13 (Jun 18 2002) Both softwares are using the *same* BLAS and LAPACK (ATLAS for sse). As you can see, numpy array looks very competitive. The matrix class in numpy has too much overhead for small dimension though. This overhead is very small for medium size arrays. Looking at the results above (specially the small dimensions ones, for higher dimensions the main computations are being performed by the same BLAS) I believe we can say: 1) Numpy array is faster on usual operations but outerproduct (I believe the reason is that the dot function uses the regular matrix multiplication to compute outer-products, instead of using a special function. This can "easily" changes). In particular numpy was faster in matrix times vector operations, which is the most usual in numerical linear algebra. 2) Any operation that involves transpose suffers a very big penalty in numpy. Compare A'*x and A*x, it is 10 times slower. In contrast Matlab deals with transpose quite well. Travis is already aware of this and it can be probably solved. 3) When using subarrays, numpy is a slower. The difference seems acceptable. Travis, can this be improved? Best, Paulo Obs: Latter on (in a couple of days) I may present less synthetic benchmarks (a QR factorization and a Modified Cholesky).

13 27

some work on arpack
by David Grant 22 Oct '06

22 Oct '06

Building an arpack extension turned out to be surprisingly simple. For example for dsaupd: f2py -c dsaupd.f -m dsaupd -L/usr/lib/blas/atlas:/usr/lib/lapack/atlas -llapack -lblas -larpack It took me a long time to get the command down to something that simple. Took me a while even to figure out I could just use the arpack library on my computer rather than re-linking all of arpack! I was able to import the dsaupd.so python module just fine and I was also able to call it just fine. I'll have to tweak the pyf file in order to get some proper output. But this gives me confidence that arpack is easy to hook into which is what others have said in the past, but without any experience with f2py I had no idea myself. f2py is awesome, for anyone who doesn't know. Matlab has interfaces for the arpack functions like dsaupd, dseupd, dnaupd, znaupd, zneupd (the mex file documentation claims those are the only ones, but they have more). Matlab has a C interface to these functions in arpackc.mex* and the script eigs.m does the grunt work, providing a very high-level interface as well as doing some linear algebra (the same type of stuff that is done in arpack's examples directory I gather) and various other things. My idea is (if I have time) to write an eigs-like function in python that will only perform a subset of what Matlab's eigs does for. It will, for example, compute a certain number of eigenvalues and eigenvectors for a real, sparse, symmetric matrix (the case I'm interested in)... I hope that this subset-of-matlab's-eigs function will not be too hard to write. Then more functionality can be added on to eigs.py later... Does this make sense? Has anyone else started work on arpack integration at all? -- David Grant http://www.davidgrant.ca

8 19

Irregular arrays
by rw679aq02＠sneakemail.com 05 Sep '06

05 Sep '06

Many problems are best solved with irregular array structures. These are aggregations not having a rectangular shape. To motivate, here's one example, http://lambda-the-ultimate.org/files/HammingNumbersDeclarative.7z - from http://lambda-the-ultimate.org/node/608#comment-5746 Irregularity here changes an O(N^3) solution to O(N). (The file format is a 7zip archive with a MathReader file inside, readable in Windows or Unix with free software.) These cases also arise in simulations where physical geometry determines array shape. Here memory consumption is the minimization goal that makes irregularity desirable. The access function will return NaN or zero for out-of-bounds requests. There is no need to consume memory storing NaNs and zeros. Please advise how much support numpy/Scipy has for these structures, if any, including future plans. If support exists, could you kindly supply a Scipy declaration matching the first example. Thank you very much.

6 9

amd64 support
by Keith Goodman 01 Sep '06

01 Sep '06

I plan to build an amd64 box and run debian etch. Are there any big, 64-bit, show-stopping problems in numpy? Any minor annoyances?

3 2

dtype=object behavior change from 0.9.6 to beta 1
by Tom Denniston 31 Aug '06

31 Aug '06

In version 0.9.6 one used to be able to do this: In [4]: numpy.__version__ Out[4]: '0.9.6' In [6]: numpy.array([numpy.array([4,5,6]), numpy.array([1,2,3])], dtype=object).shape Out[6]: (2, 3) In beta 1 numpy.PyObject no longer exists. I was trying to get the same behavior with dtype=object but it doesn't work: In [7]: numpy.__version__ Out[7]: '1.0b1' In [8]: numpy.array([numpy.array([4,5,6]), numpy.array([1,2,3])], dtype=object).shape Out[8]: (2,) Is this an intentional change?

3 13

possible bug with numpy.object_
by Matt Knox 31 Aug '06

31 Aug '06

is the following behaviour expected? or is this a bug with numpy.object_ ? I'm using numpy 1.0b1 >>> print numpy.array([],numpy.float64).size0 >>> print numpy.array([],numpy.object_).size1 Should the size of an array initialized from an empty list not always be 1 ? or am I just crazy? Thanks, - Matt Knox _________________________________________________________________ Be one of the first to try Windows Live Mail. http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0e-49…

7 10

fromiter shape argument -- was Re: For loop tips
by Torgil Svensson 31 Aug '06

31 Aug '06

> return uL,asmatrix(fromiter((idx[x] for x in L),dtype=int)) Is it possible for fromiter to take an optional shape (or count) argument in addition to the dtype argument? If both is given it could preallocate memory and we only have to iterate over L once. //Torgil On 8/29/06, Keith Goodman <kwgoodman(a)gmail.com> wrote: > On 8/29/06, Torgil Svensson <torgil.svensson(a)gmail.com> wrote: > > something like this? > > > > def list2index(L): > > uL=sorted(set(L)) > > idx=dict((y,x) for x,y in enumerate(uL)) > > return uL,asmatrix(fromiter((idx[x] for x in L),dtype=int)) > > Wow. That's amazing. Thank you. > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion >

2 3

Strange exp
by Angelo Secchi 31 Aug '06

31 Aug '06

Hi, I have the following script import fileinput import string from math import * from scipy import * from rpy import * import Numeric import shelve import sys def dpolya1(n,N,b,a): a=float(a) b=float(b) L=784 probs=((special.gammaln(N+1)+special.gammaln(L*(a/b))+special.gammaln((a/b)+n)+special.gammaln((a/b)*(L-1)+N-n))-(special.gammaln(L*(a/b)+N)+special.gammaln(a/b)+special.gammaln(n+1)+special.gammaln(N-n+1)+special.gammaln(L*(a/b)-(a/b))))#) return probs and I observe the following "strange" (for me of course) behaviour >>> dpolya1(1,2,0.5,0.4) -5.9741312822170585 >>> type(dpolya1(1,2,0.5,0.4)) <type 'float64scalar'> >>> exp(dpolya1(1,2,0.5,0.4)) Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'numpy.ndarray' object has no attribute 'exp' I do not understand what's wrong. Any help? Thanks Angelo

2 1