Mailman 3 June 2004 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon May 17, 2011

May 17, 2011

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

Numarray header PEP
by Todd Miller July 25, 2004

July 25, 2004

Perry and I have written a draft PEP for the inclusion of some of the numarray headers within the Python distribution. The headers enable extension writers to provide better support for numarray without adding any additional dependencies (from a user's perspective) to their packages. For users who have numarray, an "improved extension" could provide better performance, and for those without numarray, the extension could work normally using the Sequence protocol. The PEP is attached. It is formatted using the docutils package which can be used to generate HTML or PDF. Comments and corrections would be appreciated. Regards, Todd

9 22

bug in numarray.maximum.reduce ?
by Sebastian Haase July 7, 2004

July 7, 2004

Hi, Is this a bug?: >>> # (import numarray as na ; 'd' is a 3 dimensional array) >>> d.type() Float32 >>> d[80, 136, 122] 80.3997039795 >>> na.maximum.reduce(d[:,136, 122]) 85.8426361084 >>> na.maximum.reduce(d) [136, 122] 37.3658103943 >>> >>> >>> na.maximum.reduce(d,0)[136, 122] 37.3658103943 >>> na.maximum.reduce(d,1)[136, 122] Traceback (most recent call last): File "<input>", line 1, in ? IndexError: Index out of range I was using na.maximum.reduce(d) to get a "pixelwise" maximum along Z (axis 0). But as seen above it does not get it right. I then tried to reproduce this with some simple arrays, but here it works just fine: >>> a = na.arange(4*4*4) >>> a.shape=(4,4,4) >>> na.maximum.reduce(a) [[48 49 50 51] [52 53 54 55] [56 57 58 59] [60 61 62 63]] >>> a = na.arange(4*4*4).astype(na.Float32) >>> a.shape=(4,4,4) >>> na.maximum.reduce(a) [[ 48. 49. 50. 51.] [ 52. 53. 54. 55.] [ 56. 57. 58. 59.] [ 60. 61. 62. 63.]] >>> Any hint ? Regards, Sebastian Haase

3 9

Speeding up wxPython/numarray
by Tim Hochberg July 1, 2004

July 1, 2004

I spend some time seeing what I could do in the way of speeding up wxPoint_LIST_helper by tweaking the numarray code. My first suspect was _universalIndexing by way of _ndarray_item. However, due to some new-style machinations, _ndarray_item was never getting called. Instead, _ndarray_subscript was being called. So, I added a special case to _ndarray_subscript. This sped things up by 50% or so (I don't recall exactly). The code for that is at the end of this message; it's not gauranteed to be 100% correct; it's all experimental. After futzing around some more I figured out a way to trick python into using _ndarray_item. I added "type->tp_as_sequence->sq_item = _ndarray_item;" to _ndarray new. I then optimized _ndarray_item (code at end). This halved the execution time of my arbitrary benchmark. This trick may have horrible, unforseen consequences so use at your own risk. Finally I commented out the __del__ method numarraycore. This resulted in an additional speedup of 64% for a total speed up of 240%. Still not close to 10x, but a large improvement. However, this is obviously not viable for real use, but it's enough of a speedup that I'll try to see if there's anyway to move the shadow stuff back to tp_dealloc. In summary: Version Time Rel Speedup Abs Speedup Stock 0.398 ---- ---- _naarray_item mod 0.192 107% 107% del __del__ 0.117 64% 240% There were a couple of other things I tried that resulted in additional small speedups, but the tactics I used were too horrible to reproduce here. The main one of interest is that all of the calls to NA_updateDataPtr seem to burn some time. However, I don't have any idea what one could do about that. That's all for now. -tim static PyObject* _ndarray_subscript(PyArrayObject* self, PyObject* key) { PyObject *result; #ifdef TAH if (PyInt_CheckExact(key)) { long ikey = PyInt_AsLong(key); long offset; if (NA_getByteOffset(self, 1, &ikey, &offset) < 0) return NULL; if (!NA_updateDataPtr(self)) return NULL; return _simpleIndexingCore(self, offset, 1, Py_None); } #endif #if _PYTHON_CALLBACKS result = PyObject_CallMethod( (PyObject *) self, "_universalIndexing", "(OO)", key, Py_None); #else result = _universalIndexing(self, key, Py_None); #endif return result; } static PyObject * _ndarray_item(PyArrayObject *self, int i) { #ifdef TAH long offset; if (NA_getByteOffset(self, 1, &i, &offset) < 0) return NULL; if (!NA_updateDataPtr(self)) return NULL; return _simpleIndexingCore(self, offset, 1, Py_None); #else PyObject *result; PyObject *key = PyInt_FromLong(i); if (!key) return NULL; result = _universalIndexing(self, key, Py_None); Py_DECREF(key); return result; #endif }

3 6

bug ? in Records arrays in numarray
by Sebastian Haase June 30, 2004

June 30, 2004

Hi, I have two record arrays. I was trying to assign one item from one recarray to the other: >>> omx.zext[0] = main.ring4ext[0,0] Traceback (most recent call last): File "<input>", line 1, in ? File "X:/PrWin\numarray\records.py", line 744, in _setitem self.field(self._names[i])[row] = value.field(self._names[i]) File "C:\Python22\Lib\site-packages\numarray\numarraycore.py", line 619, in __tonumtype__ ValueError: Can't use non-rank-0 array as a scalar. >>> `omx.zext[0]` '<numarray.records.Record instance at 0x042AFC78>' >>> `main.ring4ext[0,0]` '<numarray.records.Record instance at 0x042AFC78>' >>> q = omx.zext[0] >>> w = main.ring4ext[0,0] >>> q (2097152107, -595656700, 91141, 1.0634608642000868e+037, -1.14841804241843e +018, 1.2771574333702815e-040) >>> w (array([428]), array([75]), array([124]), array([ 1.08846451e+09], type=Float32), array([ 99.25], type=Float32), array([ 1996.82995605], type=Float32)) >>> omx.zext._formats ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] >>> main.ring4ext._formats ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] I can't track down why one (q) contains scalars and the other (w) constains arrays (is array([428]) a 'rank-0 array'? ) ! This is how I generate them: main.ring4ext = rec.RecArray(main._extHdrRingBuffer, "i4,i4,i4,f4,f4,f4", shape=(ts,nc), names=("num","min","max","time","mean","darkVal"), aligned=1) omx.zext = rec.array(formats="i4,i4,i4,f4,f4,f4", names=("num","min","max","time","mean","darkVal"),shape=tuple(shapeZ),aligned=1 ) [[BTW: (shapeZ is a list, if I say: ...,shape=shapeZ I get NameError, "Illegal shape %s" % `shape` from "records.py" in line 462 -- usually type(shape) == types.ListType should be OK, right ?) ]] Any ideas? Thanks, Sebastian Haase

4 5

Re: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem
by Todd Miller June 29, 2004

June 29, 2004

On Mon, 2004-06-28 at 19:38, Sebastian Haase wrote: > > Is 10x a measured number or a gut feel? > > I put some time.clock() statements into the wxPyPlot code > I got this: (the times are differences: T_after-T_before) > one run with numarray: > <__main__.PolyLine instance at 0x868d414> time= 1.06 > <__main__.PolyMarker instance at 0x878e9c4> time= 1.37 > a second run with numarray: > <__main__.PolyLine instance at 0x875da1c> time= 0.85 > <__main__.PolyMarker instance at 0x86da034> time= 1.04 > first run with Numeric: > <__main__.PolyLine instance at 0x858babc> time= 0.07 > <__main__.PolyMarker instance at 0x858bc4c> time= 0.14 > a second one: > <__main__.PolyLine instance at 0x858cd7c> time= 0.08 > <__main__.PolyMarker instance at 0x8585d8c> time= 0.17 > This seems to be consistent with the profiling I did before: > I get this w/ numarray: > ncalls tottime percall cumtime percall filename:lineno(function) > 1 1.140 1.140 1.320 1.320 gdi.py:554(DrawLines) > 1 1.250 1.250 1.520 1.520 gdi.py:792(_DrawRectangleList) > 50230 0.450 0.000 0.450 0.000 numarraycore.py:501(__del__) > and this with Numeric: > 1 0.080 0.080 0.080 0.080 gdi.py:554(DrawLines) > 1 0.090 0.090 0.090 0.090 gdi.py:792(_DrawRectangleList) > > so this looks to me like a factor of 10x. Me too. Can you (or somebody) post the application code which does the drawlines? I can definitely instrument the bottleneck C-code, but I don't have time to ascend the wxPython learning curve. Todd

4 7

RE: [Numpy-discussion] Can not import image module
by Nadav Horesh June 29, 2004

June 29, 2004

Same (RH9.0 + Python 2.3.4). It is OK on my home PC (Gentoo 1.4 +Python 2.3.4). I'll try your advice when I'll back at work. Thanks, Nadav. -----Original Message----- From: Todd Miller [mailto:jmiller@stsci.edu] Sent: Tue 29-Jun-04 20:36 To: Nadav Horesh Cc: numpy-discussion Subject: Re: [Numpy-discussion] Can not import image module On Tue, 2004-06-29 at 07:14, Nadav Horesh wrote: > >>> from numarray import image I tried this and it worked fine for me under RH9.0 and Python-2.3.4 and also under Windows 2000 pro. I suggest deleting your entire numarray directory from site-packages and then re-installing. Also, what's your platform and shell? Regards, Todd > Traceback (most recent call last): > File "<pyshell#4>", line 1, in -toplevel- > from numarray import image > File "/usr/local/lib/python2.3/site-packages/numarray/image/__init__.py", line 2, in -toplevel- > from combine import * > File "/usr/local/lib/python2.3/site-packages/numarray/image/combine.py", line 65, in -toplevel- > _median = _combine.median > AttributeError: 'module' object has no attribute 'median' > > > > numarray version: 1.0. I got this error also with version 0.9 > > Nadav. > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4576

1 0

Can not import image module
by Nadav Horesh June 29, 2004

June 29, 2004

>>> from numarray import image Traceback (most recent call last): File "<pyshell#4>", line 1, in -toplevel- from numarray import image File "/usr/local/lib/python2.3/site-packages/numarray/image/__init__.py", line 2, in -toplevel- from combine import * File "/usr/local/lib/python2.3/site-packages/numarray/image/combine.py", line 65, in -toplevel- _median = _combine.median AttributeError: 'module' object has no attribute 'median' numarray version: 1.0. I got this error also with version 0.9 Nadav.

2 1

numarray speed - PySequence_GetItem
by Sebastian Haase June 29, 2004

June 29, 2004

Hi, The long story is that I'm looking for a good/fast graph plotting programs; so I found WxPyPlot (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html) It uses wxPython and plots 25000 data points (with lines + square markers) in under one second - using Numeric that is. [the slow line in WxPyPlot is: dc.DrawLines(self.scaled) where self.scaled is an array of shape (25000,2) and type Float64 ] The short story is that numarray takes maybe 10 times as long as Numeric and I tracked the problem down into the wxPython SWIG typemap where he does this: <code-sniplet from wxPoint_LIST_helper() in helpers.cpp from wxPython> wxPoint* wxPoint_LIST_helper(PyObject* source, int *count) { <snip> bool isFast = PyList_Check(source) || PyTuple_Check(source); <snip> for (x=0; x<*count; x++) { // Get an item: try fast way first. if (isFast) { o = PySequence_Fast_GET_ITEM(source, x); } else { o = PySequence_GetItem(source, x); if (o == NULL) { goto error1; } } </code-sniplet> I'm not 100% sure that this is where the problem lies - is there a chance (or a known issue) that numarray does PySequence_GetItem() slower than Numeric ? I just ran this again using the python profiler and I get this w/ numarray: ncalls tottime percall cumtime percall filename:lineno(function) 1 1.140 1.140 1.320 1.320 gdi.py:554(DrawLines) 1 1.250 1.250 1.520 1.520 gdi.py:792(_DrawRectangleList) 50230 0.450 0.000 0.450 0.000 numarraycore.py:501(__del__) and this with Numeric: 1 0.080 0.080 0.080 0.080 gdi.py:554(DrawLines) 1 0.090 0.090 0.090 0.090 gdi.py:792(_DrawRectangleList) Thanks, Sebastian Haase

5 10

generalized SVD
by Curzio Basso June 29, 2004

June 29, 2004

Hi. Has anyone a suggestion on where to find a GSVD implementation which I can use with numarray? I've seen GSVD is part of LAPACK, so maybe one option could be to use a LAPACK library implementing the required functions... thanks in advance. curzio.

1 0