Mailman 3 May 2006 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon May 16, 2011

May 16, 2011

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

Re: [Numpy-discussion] Floating point exception with numpy and embedded python interpreter
by Andrew Straw Sept. 20, 2007

Sept. 20, 2007

Arkaitz Bitorika wrote: > > On 19 Apr 2006, at 18:37, Andrew Straw wrote: > >> >>> I've been embedding Python for ages on the same program and other >>> modules work fine, only numpy fails. >> >> >> Most other modules don't use the SSE units, so wouldn't get hit by such >> a bug. > > > Is there a way of not using those units from numpy, to check if > that's what's going on? I think that numpy only accesses the SSE units through ATLAS or other external library. So, build numpy without ATLAS. But I'm not 100% sure anymore if there aren't any optimizations that directly use SSE if it's available. > Or alternatively, how would I check if my program is messing with the > SSE bits? Hmm, I think that's a bit hairy. I'd suggest simply asking the C++ library's mailing list if they alter the error bits on the control registers of the SSE unit. (Out of curiousity, what library is it?) If you want hairy, though, I think you'd have to check from C with the appropriate calls -- I'd start with the source code in that bug report. It looks like they're inlining an assembly statement to query a SSE control register.

5 7

Numpy x Matlab: some synthetic benchmarks
by Paulo Jose da Silva e Silva Feb. 11, 2007

Feb. 11, 2007

Hello, Travis asked me to benchmark numpy versus matlab in some basic linear algebra operations. Here are the resuts for matrices/vectors of dimensions 5, 50 and 500: Operation x'*y x*y' A*x A*B A'*x Half 2in2 Dimension 5 Array 0.94 0.7 0.22 0.28 1.12 0.98 1.1 Matrix 7.06 1.57 0.66 0.79 1.6 3.11 4.56 Matlab 1.88 0.44 0.41 0.35 0.37 1.2 0.98 Dimension 50 Array 9.74 3.09 0.56 18.12 13.93 4.2 4.33 Matrix 81.99 3.81 1.04 19.13 14.58 6.3 7.88 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension 500 Array 1.2 8.97 2.03 166.59 20.34 3.99 4.31 Matrix 17.95 9.09 2.07 166.62 20.67 4.11 4.45 Matlab 2.09 6.07 2.17 169.45 2.1 2.56 3.06 Obs: The operation Half is actually A*x using only the lower half of the matrix and vector. The operation 2in2 is A*x using only the even indexes. Of course there are many repetitions of the same operation: 100000 for dim 5 and 50 and 1000 for dim 500. The inner product is number of repetitions is multiplied by dimension (it is very fast). The software is numpy svn version 1926 Matlab 6.5.0.180913a Release 13 (Jun 18 2002) Both softwares are using the *same* BLAS and LAPACK (ATLAS for sse). As you can see, numpy array looks very competitive. The matrix class in numpy has too much overhead for small dimension though. This overhead is very small for medium size arrays. Looking at the results above (specially the small dimensions ones, for higher dimensions the main computations are being performed by the same BLAS) I believe we can say: 1) Numpy array is faster on usual operations but outerproduct (I believe the reason is that the dot function uses the regular matrix multiplication to compute outer-products, instead of using a special function. This can "easily" changes). In particular numpy was faster in matrix times vector operations, which is the most usual in numerical linear algebra. 2) Any operation that involves transpose suffers a very big penalty in numpy. Compare A'*x and A*x, it is 10 times slower. In contrast Matlab deals with transpose quite well. Travis is already aware of this and it can be probably solved. 3) When using subarrays, numpy is a slower. The difference seems acceptable. Travis, can this be improved? Best, Paulo Obs: Latter on (in a couple of days) I may present less synthetic benchmarks (a QR factorization and a Modified Cholesky).

13 27

recarray field names
by Erin Sheldon July 8, 2006

July 8, 2006

Hi all- I'm new to numpy and I'm slowly weening myself off of IDL. So far the experience has been very positive. I use recarrays a lot. I often read these recarrays from fits files using pyfits (modified to work with numpy). I find myself doing the following more than I would like: if 'TAG' in rec._coldefs.names: .... It seems messy to be accessing this "hidden" attribute in this way. Is there a plan to add methods to more transparently do such things? e.g. def fieldnames(): return _coldefs.names def field_exists(fieldname): return fieldname.upper() in _coldefs.names def field_index(fieldname): if field_exists(fieldname): return _coldefs.names.index(fieldname.upper()) else: return -1 # or None maybe Thanks, Erin

5 13

Any Numeric or numarray users on this list?
by Travis Oliphant June 25, 2006

June 25, 2006

Please help the developers by responding to a few questions. 1) Have you transitioned or started to transition to NumPy (i.e. import numpy)? 2) Will you transition within the next 6 months? (if you answered No to #1) 3) Please, explain your reason(s) for not making the switch. (if you answered No to #2) 4) Please provide any suggestions for improving NumPy. Thanks for your answers. NumPy Developers

24 29

test_scalarmath.py", line 63
by Nils Wagner May 31, 2006

May 31, 2006

>>> numpy.__version__ '0.9.9.2553' numpy.test(1,10) results in ====================================================================== FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_scalarmath.py", line 63, in check_types assert val.dtype.num == typeconv[k,l] and \ AssertionError: error with (0,7) ---------------------------------------------------------------------- Ran 368 tests in 0.479s FAILED (failures=1) Nils

2 1

numpy.test(1,10) results in a segfault
by Nils Wagner May 31, 2006

May 31, 2006

test_wrap (numpy.core.tests.test_umath.test_special_methods) ... ok check_types (numpy.core.tests.test_scalarmath.test_types)*** glibc detected *** free() : invalid pointer: 0xb7ab74a0 *** Program received signal SIGABRT, Aborted. [Switching to Thread 16384 (LWP 14948)] 0xb7ca51f1 in kill () from /lib/i686/libc.so.6 (gdb) bt #0 0xb7ca51f1 in kill () from /lib/i686/libc.so.6 #1 0xb7e90401 in pthread_kill () from /lib/i686/libpthread.so.0 #2 0xb7e9044b in raise () from /lib/i686/libpthread.so.0 #3 0xb7ca4f84 in raise () from /lib/i686/libc.so.6 #4 0xb7ca6498 in abort () from /lib/i686/libc.so.6 #5 0xb7cd8cf6 in __libc_message () from /lib/i686/libc.so.6 #6 0xb7cde367 in malloc_printerr () from /lib/i686/libc.so.6 #7 0xb7cdfacf in free () from /lib/i686/libc.so.6 #8 0xb7ae3fc5 in gentype_dealloc (v=0x0) at scalartypes.inc.src:281 #9 0xb7f57bed in PyEval_EvalFrame () from /usr/lib/libpython2.4.so.1.0 #10 0xb7f5ad21 in PyEval_EvalCodeEx () from /usr/lib/libpython2.4.so.1.0 #11 0xb7f58cc5 in PyEval_EvalFrame () from /usr/lib/libpython2.4.so.1.0 #12 0xb7f5ad21 in PyEval_EvalCodeEx () from /usr/lib/libpython2.4.so.1.0 #13 0xb7f1113a in function_call () from /usr/lib/libpython2.4.so.1.0 #14 0xb7ef9c37 in PyObject_Call () from /usr/lib/libpython2.4.so.1.0 #15 0xb7f02edb in instancemethod_call () from /usr/lib/libpython2.4.so.1.0 #16 0xb7ef9c37 in PyObject_Call () from /usr/lib/libpython2.4.so.1.0 ---Type <return> to continue, or q <return> to quit--- #17 0xb7f58097 in PyEval_EvalFrame () from /usr/lib/libpython2.4.so.1.0 #18 0xb7f5ad21 in PyEval_EvalCodeEx () from /usr/lib/libpython2.4.so.1.0 #19 0xb7f1113a in function_call () from /usr/lib/libpython2.4.so.1.0 #20 0xb7ef9c37 in PyObject_Call () from /usr/lib/libpython2.4.so.1.0 #21 0xb7f02edb in instancemethod_call () from /usr/lib/libpython2.4.so.1.0 #22 0xb7ef9c37 in PyObject_Call () from /usr/lib/libpython2.4.so.1.0 #23 0xb7f34c2c in slot_tp_call () from /usr/lib/libpython2.4.so.1.0 #24 0xb7ef9c37 in PyObject_Call () from /usr/lib/libpython2.4.so.1.0 #25 0xb7f58097 in PyEval_EvalFrame () from /usr/lib/libpython2.4.so.1.0 #26 0xb7f5ad21 in PyEval_EvalCodeEx () from /usr/lib/libpython2.4.so.1.0 #27 0xb7f1113a in function_call () from /usr/lib/libpython2.4.so.1.0 #28 0xb7ef9c37 in PyObject_Call () from /usr/lib/libpython2.4.so.1.0 #29 0xb7f02edb in instancemethod_call () from /usr/lib/libpython2.4.so.1.0 #30 0xb7ef9c37 in PyObject_Call () from /usr/lib/libpython2.4.so.1.0 #31 0xb7f34c2c in slot_tp_call () from /usr/lib/libpython2.4.so.1.0 #32 0xb7ef9c37 in PyObject_Call () from /usr/lib/libpython2.4.so.1.0 #33 0xb7f58097 in PyEval_EvalFrame () from /usr/lib/libpython2.4.so.1.0 ---Type <return> to continue, or q <return> to quit--- #34 0xb7f5a663 in PyEval_EvalFrame () from /usr/lib/libpython2.4.so.1.0 #35 0xb7f5ad21 in PyEval_EvalCodeEx () from /usr/lib/libpython2.4.so.1.0 #36 0xb7f58cc5 in PyEval_EvalFrame () from /usr/lib/libpython2.4.so.1.0 #37 0xb7f5ad21 in PyEval_EvalCodeEx () from /usr/lib/libpython2.4.so.1.0 #38 0xb7f58cc5 in PyEval_EvalFrame () from /usr/lib/libpython2.4.so.1.0 #39 0xb7f5ad21 in PyEval_EvalCodeEx () from /usr/lib/libpython2.4.so.1.0 #40 0xb7f5aff5 in PyEval_EvalCode () from /usr/lib/libpython2.4.so.1.0 #41 0xb7f75778 in run_node () from /usr/lib/libpython2.4.so.1.0 #42 0xb7f77228 in PyRun_InteractiveOneFlags () from /usr/lib/libpython2.4.so.1.0 #43 0xb7f77396 in PyRun_InteractiveLoopFlags () from /usr/lib/libpython2.4.so.1.0 #44 0xb7f774a7 in PyRun_AnyFileExFlags () from /usr/lib/libpython2.4.so.1.0 #45 0xb7f7d66a in Py_Main () from /usr/lib/libpython2.4.so.1.0 #46 0x0804871a in main (argc=0, argv=0x0) at ccpython.cc:10 Can someone reproduce the segfault ? Linux amanda 2.6.11.4-21.12-default #1 Wed May 10 09:38:20 UTC 2006 i686 athlon i386 GNU/Linux >>> numpy.__version__ '0.9.9.2553'

3 2

Re: Any Numeric or numarray users on this list?
by Nick Fotopoulos May 31, 2006

May 31, 2006

> 1) Have you transitioned or started to transition to NumPy (i.e. > import > numpy)? Yes, I've pretty much decided that numpy is the way to go with my young analysis codes, even though it is going to be somewhat painful to distribute to my colleagues and among compute nodes in clusters. I am saved by the fact that we all work within an NFS space, so I can host up-to-date compiles of numpy and scipy without requiring root access nor requiring everyone to compile their own. Of course, I will end up giving them a few lines of LD_LIBRARY_PATH to add. Old codes that used Numeric have all transitioned smoothly to numpy in my internal tests. I may distribute two versions (one Numeric, one numpy) for wider distribution outside my clusters, but there may be a better way to do this. > 2) Will you transition within the next 6 months? (if you answered > No to #1) > 3) Please, explain your reason(s) for not making the switch. (if you > answered No to #2) > 4) Please provide any suggestions for improving NumPy. I think NumPy is fantastic. That said, even the fantastic can improve. I am very happy with numpy as software, so my comments are mostly about adoption and accessibility. I'd really like to see the team getting packages (32 and 64 bit) into standard Redhat, Debian, and Fink repositories quickly after a release. I believe Redhat is at numpy 0.9.5 and neither Debian (testing) nor Fink have packages in the default repositories. Having to add extra lines to /etc/apt/sources.list (or Redhat equivalent) to grab packages from private repositories will dissuade a lot of people from adopting numpy. Many of them are the same people who are unable to compile it themselves. Up-to-date packages also help me with my problem in #1 -- an admin will happily yum an rpm for me an all of the cluster nodes, but might not be willing to install it from a nonstandard source. I agree that Googlability is very important and easily addressable. I'm glad someone brought this up. Btw, googling "numpy rpm" brings up http://pylab.sourceforge.net/, which is some more of Travis's old Numeric stuff (labeled as such on the page). Thanks for the hard work in coding and thanks for keeping a thriving discussion list going. Take care, Nick

1 0

numpy-0.9.5 build is not clean
by Krishna Mohan Gundu May 31, 2006

May 31, 2006

Hi, I am trying to build numpy-0.9.5, downloaded from sourceforge download page, as higher versions are not yet tested for pygsl. The build seems to be broken. I uninstall existing numpy and start build from scratch but I get the following errors when I import numpy after install. ==== >>> from numpy import * import core -> failed: module compiled against version 90504 of C-API but this version of numpy is 90500 import random -> failed: 'module' object has no attribute 'dtype' import linalg -> failed: module compiled against version 90504 of C-API but this version of numpy is 90500 ==== Any help is appreciated. Am I doing something wrong or is it known that this build is broken? thanks, Krishna.

2 3

cx_freezing numpy problem.
by Mike Rovner May 31, 2006

May 31, 2006

Hi, I'm trying to package numpy based application but get following TB: No scipy-style subpackage 'core' found in /home/mrovner/dev/psgapp/src/gui/lnx32/dvip/numpy. Ignoring: No module named _internal Traceback (most recent call last): File "/home/mrovner/src/cx_Freeze-3.0.2/initscripts/Console.py", line 26, in ? exec code in m.__dict__ File "dvip.py", line 42, in ? File "dvip.py", line 31, in dvip_gui File "mainui.py", line 1, in ? File "psgdb.pyx", line 162, in psgdb File "/lan/dfm/grp_mm_data1/dev/tools/linux-x86_32/lib/python2.4/site-packages/numpy/__init__.py", line 35, in ? verbose=NUMPY_IMPORT_VERBOSE,postpone=False) File "/lan/dfm/grp_mm_data1/dev/tools/linux-x86_32/lib/python2.4/site-packages/numpy/_import_tools.py", line 173, in __call__ self._init_info_modules(packages or None) File "/lan/dfm/grp_mm_data1/dev/tools/linux-x86_32/lib/python2.4/site-packages/numpy/_import_tools.py", line 68, in _init_info_modules exec 'import %s.info as info' % (package_name) File "<string>", line 1, in ? File "/lan/dfm/grp_mm_data1/dev/tools/linux-x86_32/lib/python2.4/site-packages/numpy/lib/__init__.py", line 5, in ? from type_check import * File "/lan/dfm/grp_mm_data1/dev/tools/linux-x86_32/lib/python2.4/site-packages/numpy/lib/type_check.py", line 8, in ? import numpy.core.numeric as _nx File "/lan/dfm/grp_mm_data1/dev/tools/linux-x86_32/lib/python2.4/site-packages/numpy/core/__init__.py", line 6, in ? import umath File "ExtensionLoader.py", line 12, in ? AttributeError: 'module' object has no attribute '_ARRAY_API' I did freezing with: FreezePython --install-dir=lnx32 --include-modules=numpy --include-modules=numpy.core dvip.py I'm using Python-2.4.2 numpy-0.9.8 cx_Freeze-3.0.2 on linux. Everything compiled from source. Any help appreciated. Thanks, Mike

1 1