Mailman 3 September 2006 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 17 May '11

17 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

Re: [Numpy-discussion] Floating point exception with numpy and embedded python interpreter
by Andrew Straw 20 Sep '07

20 Sep '07

Arkaitz Bitorika wrote: > > On 19 Apr 2006, at 18:37, Andrew Straw wrote: > >> >>> I've been embedding Python for ages on the same program and other >>> modules work fine, only numpy fails. >> >> >> Most other modules don't use the SSE units, so wouldn't get hit by such >> a bug. > > > Is there a way of not using those units from numpy, to check if > that's what's going on? I think that numpy only accesses the SSE units through ATLAS or other external library. So, build numpy without ATLAS. But I'm not 100% sure anymore if there aren't any optimizations that directly use SSE if it's available. > Or alternatively, how would I check if my program is messing with the > SSE bits? Hmm, I think that's a bit hairy. I'd suggest simply asking the C++ library's mailing list if they alter the error bits on the control registers of the SSE unit. (Out of curiousity, what library is it?) If you want hairy, though, I think you'd have to check from C with the appropriate calls -- I'd start with the source code in that bug report. It looks like they're inlining an assembly statement to query a SSE control register.

5 7

Numpy x Matlab: some synthetic benchmarks
by Paulo Jose da Silva e Silva 11 Feb '07

11 Feb '07

Hello, Travis asked me to benchmark numpy versus matlab in some basic linear algebra operations. Here are the resuts for matrices/vectors of dimensions 5, 50 and 500: Operation x'*y x*y' A*x A*B A'*x Half 2in2 Dimension 5 Array 0.94 0.7 0.22 0.28 1.12 0.98 1.1 Matrix 7.06 1.57 0.66 0.79 1.6 3.11 4.56 Matlab 1.88 0.44 0.41 0.35 0.37 1.2 0.98 Dimension 50 Array 9.74 3.09 0.56 18.12 13.93 4.2 4.33 Matrix 81.99 3.81 1.04 19.13 14.58 6.3 7.88 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension 500 Array 1.2 8.97 2.03 166.59 20.34 3.99 4.31 Matrix 17.95 9.09 2.07 166.62 20.67 4.11 4.45 Matlab 2.09 6.07 2.17 169.45 2.1 2.56 3.06 Obs: The operation Half is actually A*x using only the lower half of the matrix and vector. The operation 2in2 is A*x using only the even indexes. Of course there are many repetitions of the same operation: 100000 for dim 5 and 50 and 1000 for dim 500. The inner product is number of repetitions is multiplied by dimension (it is very fast). The software is numpy svn version 1926 Matlab 6.5.0.180913a Release 13 (Jun 18 2002) Both softwares are using the *same* BLAS and LAPACK (ATLAS for sse). As you can see, numpy array looks very competitive. The matrix class in numpy has too much overhead for small dimension though. This overhead is very small for medium size arrays. Looking at the results above (specially the small dimensions ones, for higher dimensions the main computations are being performed by the same BLAS) I believe we can say: 1) Numpy array is faster on usual operations but outerproduct (I believe the reason is that the dot function uses the regular matrix multiplication to compute outer-products, instead of using a special function. This can "easily" changes). In particular numpy was faster in matrix times vector operations, which is the most usual in numerical linear algebra. 2) Any operation that involves transpose suffers a very big penalty in numpy. Compare A'*x and A*x, it is 10 times slower. In contrast Matlab deals with transpose quite well. Travis is already aware of this and it can be probably solved. 3) When using subarrays, numpy is a slower. The difference seems acceptable. Travis, can this be improved? Best, Paulo Obs: Latter on (in a couple of days) I may present less synthetic benchmarks (a QR factorization and a Modified Cholesky).

13 27

some work on arpack
by David Grant 22 Oct '06

22 Oct '06

Building an arpack extension turned out to be surprisingly simple. For example for dsaupd: f2py -c dsaupd.f -m dsaupd -L/usr/lib/blas/atlas:/usr/lib/lapack/atlas -llapack -lblas -larpack It took me a long time to get the command down to something that simple. Took me a while even to figure out I could just use the arpack library on my computer rather than re-linking all of arpack! I was able to import the dsaupd.so python module just fine and I was also able to call it just fine. I'll have to tweak the pyf file in order to get some proper output. But this gives me confidence that arpack is easy to hook into which is what others have said in the past, but without any experience with f2py I had no idea myself. f2py is awesome, for anyone who doesn't know. Matlab has interfaces for the arpack functions like dsaupd, dseupd, dnaupd, znaupd, zneupd (the mex file documentation claims those are the only ones, but they have more). Matlab has a C interface to these functions in arpackc.mex* and the script eigs.m does the grunt work, providing a very high-level interface as well as doing some linear algebra (the same type of stuff that is done in arpack's examples directory I gather) and various other things. My idea is (if I have time) to write an eigs-like function in python that will only perform a subset of what Matlab's eigs does for. It will, for example, compute a certain number of eigenvalues and eigenvectors for a real, sparse, symmetric matrix (the case I'm interested in)... I hope that this subset-of-matlab's-eigs function will not be too hard to write. Then more functionality can be added on to eigs.py later... Does this make sense? Has anyone else started work on arpack integration at all? -- David Grant http://www.davidgrant.ca

8 19

Non-writeable default for numpy.ndarray
by Francesc Altet 01 Oct '06

01 Oct '06

Hello, Is the next a bug a feature? In [102]: f4=numpy.ndarray(buffer="a\x00b"*4, dtype="f4", shape=3) In [103]: f4 Out[103]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], dtype=float32) In [104]: f4[2] = 2 --------------------------------------------------------------------------- <type 'exceptions.RuntimeError'> Traceback (most recent call last) /home/faltet/python.nobackup/numpy/<ipython console> in <module>() <type 'exceptions.RuntimeError'>: array is not writeable In [105]: f4.flags.writeable = True In [106]: f4[2] = 2 In [107]: f4 Out[107]: array([ 2.60561966e+20, 8.94319890e-39, 2.00000000e+00], dtype=float32) i.e. in an array built from ndarray, the default is that it has to be read-only? -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"

4 6

return type diffences of indexed arrays with Intel C++ compiler (Python 2.5)
by Lars Bittrich 30 Sep '06

30 Sep '06

Hi all, recently I tried to compile numpy/scipy with Python 2.5 and Intel C++ compiler 9.1. At first everything was fine, but the scipy test produced a few errors. The reason was a little difference: latest svn(1.0.dev3233) with Intel C compiler (Python 2.5): ----------------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:x[i] Out[4]:1.0 latest svn(1.0.dev3233) with GCC 3.3 (Python 2.3): -------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:print x[i] Out[4]:array([1]) The Intel version gives me a scalar whereas the gcc version an array. Maybe Python 2.5 is causing this problem but my first guess was the compiler. A typical error message from scipy.test(10): ====================================================================== ERROR: check_chebyc (scipy.special.tests.test_basic.test_chebyc) ---------------------------------------------------------------------- Traceback (most recent call last): File "[...]/lib/python2.5/site-packages/scipy/special/tests/test_basic.py", line 667, in check_chebyc C1 = chebyc(1) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 461, in chebyc p = orthopoly1d(x,w,hn,kn,wfunc=lambda x: 1.0/sqrt(1-x*x/4.0),limits=(-2,2),monic=monic) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 78, in __init__ equiv_weights = [weights[k] / wfunc(roots[k]) for k in range(len(roots))] TypeError: object of type 'numpy.complex128' has no len() Does anyone know how to resolve the problem? Best regards, Lars

1 0

Problem building from svn under cygwin
by Coatimundi 30 Sep '06

30 Sep '06

I have the latest cygwin environment and am using gcc 3.4.4. to build numpy. Configure works: python setup.py configure --compiler=mingw32 Build fails: python setup.py build --compiler=mingw32 While building numpy.random.mtrand I get this: _configtest.c:7:2: #error No _WIN32 and a little bit later the build aborts completely on multiarray with this: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall creating build/temp.cygwin-1.5.21-i686-2.5 creating build/temp.cygwin-1.5.21-i686-2.5/numpy creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src compile options: '-Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src/multiarraymodule.o In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory I am puzzled, because /usr/include/sys/select.h actually does exist on my cygwin installation. But maybe that's not what pyport.h is actually looking for??? I've been banging away at this for several hours, so I must be doing something silly. Any help will be much appreciated. Thanks!

2 1

Q: NumPy (svn) with cygwin build error?
by Kyle Ferrio 29 Sep '06

29 Sep '06

Hello. Starting from a freshly installed cygwin, I have built Python 2.5, ATLAS and LAPACK all from source using gcc 3.4.4 (cygming special) for both the C and f77 compilers. Now I am trying to build NumPy from SVN. I have edited site.cfg so that python setup.py config --compiler=mingw32 finds all of the linear algebra packs and completes without error. Then I fire off python setup.py build --compiler=mingw32 and I soon get (while building extension numpy.random.mtrand) _configtest.c:7:2: #error No _WIN32 which is not immediately fatal but definitely a concern. Soon after, building numpy.core.multiarray gives me this: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall creating build/temp.cygwin-1.5.21-i686-2.5 creating build/temp.cygwin-1.5.21-i686-2.5/numpy creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src compile options: '-Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src/multiarraymodule.o In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory I am puzzzled, because /usr/include/sys/select.h does exist. But maybe that's not what pyport.h is actually looking for? I've been banging away at this for several hours, so I must be doing something silly. Any help will be much appreciated. Thanks! _________________________________________________________________ Express yourself - download free Windows Live Messenger themes! http://clk.atdmt.com/MSN/go/msnnkwme0020000001msn/direct/01/?href=http://im…

1 0

Incorrect removal of NULL char in buffers
by Francesc Altet 29 Sep '06

29 Sep '06

Hi, I'm trying to build-up numpy arrays coming from buffers, and I'm getting a somewhat unexpected result. First, for numeric values, everything seems ok (i.e. the NULL character is correctly interpretated), and works equally for both numarray and numpy: In [98]: numarray.array("a\x00b"*4, dtype='Float32',shape=3) Out[98]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], type=Float32) In [99]: numpy.ndarray(buffer="a\x00b"*4, dtype='Float32',shape=3) Out[99]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], dtype=float32) However, for string values, numpy seems to work in a strange way. The numarray have an expected behaviour, IMO: In [100]: numarray.strings.array(buffer="a\x00b"*4, itemsize=4, shape=3) Out[100]: CharArray(['a', '', 'ba']) but numpy haven't: In [101]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) Out[101]: array([aba, ba, bab], dtype='|S4') i.e. it seems like numpy is striping-off NULL chars before building the object and I don't think this is correct. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"

2 1

Negative values with unsigned data types problems
by Francesc Altet 29 Sep '06

29 Sep '06

Hi, I'm writing this here because the numpy Trac seems down: {{{ Oops... Trac detected an internal error: The Trac Environment needs to be upgraded. Run trac-admin /home/scipy/trac/numpy upgrade" }}} Well, it seems that there are inconsistencies on creating arrays coming from negative values using unsigned integers: In [41]: numpy.asarray([-1], dtype='uint8') Out[41]: array([255], dtype=uint8) In [42]: numpy.asarray([-1], dtype='uint16') Out[42]: array([65535], dtype=uint16) until here it's fine, but: In [43]: numpy.asarray([-1], dtype='uint32') --------------------------------------------------------------------------- <type 'exceptions.TypeError'> Traceback (most recent call last) /home/faltet/python.nobackup/numpy/<ipython console> in <module>() /usr/local/lib/python2.5/site-packages/numpy/core/numeric.py in asarray(a, dtype, order) 129 are converted to base class ndarray. 130 """ --> 131 return array(a, dtype, copy=False, order=order) 132 133 def asanyarray(a, dtype=None, order=None): <type 'exceptions.TypeError'>: long() argument must be a string or a number, not 'list' and the same happens with 'uint64'. My numpy version is 1.0.dev3216 Regards, -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"

2 1