Mailman 3 October 2005 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 17 May '11

17 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

byteswap questions
by Russell E. Owen 03 Nov '05

03 Nov '05

I've got a module that can send array data to ds9. It screws up on byteswapped data and I'm trying to fix it. The code sends the data to ds9 via a pipe using arr.tofile(). To send byteswapped data I think I have two choices (given that I don't want to modify the input array): 1) Figure out the byte order and specify a suitable flag to ds9 (it accepts "bigendian" or "littleendian" flags). This avoids copying the data, so it sounds attractive, but can I do it without relying on "internals"? Numarrays have an "isbyteswapped" method, but all this says is "not native machine order"; it doesn't actually tell me the order. I need to know if the order is bigendian or littleendian, and I can't find a documented way to get that, just an undocumented attribute _byteorder. 2) If the array is not in native byte order, make a native byte ordered copy before sending it. The following seems to work: if arr.isbyteswapped(): arr = arr.copy() but is this the recommended way to get native-order copy? - Is "copy" guaranteed to return a copy in native byte order? The documentation doesn't say. - I was tempted to use num.array(arr) to make the copy, but the documentation doesn't say what "array" does if the input is a already a numarray array. As an aside, I tried to use the byteswapped method, but it has a totally different effect: >>> d array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], ...type=Float32) >>> d.isbyteswapped() 1 >>> dcopy = d.copy() >>> d array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], ...type=Float32) >>> dcopy.isbyteswapped() 0 >>> dswap = d.byteswapped() >>> dswap array([[ 1.91063654e-38, 1.91063654e-38, 1.91063654e-38, ..., ...type=Float32) which I guess means I'd have to byteswap the byteswapped data. Aargh. -- Russell

4 5

ANN: numarray-1.4.1
by Todd Miller 29 Oct '05

29 Oct '05

Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.4.1 II. BUGS FIXED / CLOSED 1339713 segfault when comparing string-array to self ========================================================================= Release Notes for numarray-1.4.0 I. ENHANCEMENTS 1. Speed improvement for numarray operators. The Python level hook mapping numarray operators onto universal functions has been moved down to C. 2. Speed improvement for string-array comparisons, any(), all(). String correlation is ~10x faster. 3. Better operation with py2exe to help it automatically detect the core numarray extensions to include in an installer. 4. scipy newcore compatible lower case type names (e.g. int32 not Int32) 5. scipy newcore 'dtype' keyword and .dtypechar attribute. II. BUGS FIXED / CLOSED 1323355 Apps fail with import_libnumarray 1315212 Infinite loop converting some scalar strings into a list 1298916 rank-0 tostring() broken 1297948 records.array fails to create empty fields 1286291 import sys missing from array_persist.py 1286168 Generic sequences in ``strings.array()`` 1236392 Outdated web link in announcements 1235219 LinearAlgebraError not imported in linear_algebra See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be backward binary compatible with numarray-1.3.x WHERE ----------- Numarray-1.4.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://www.stsci.edu/resources/software_hardware/numarray Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.4.0 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller(a)stsci.edu

4 10

numarray 1.4.1: Infinite loop during test
by Edward C. Jones 29 Oct '05

29 Oct '05

I have a PC with Debian unstable. I use Python 1.4.2. After installing numarray 1.4.1, I ran the tests: >>> import numarray.testall as testall >>> testall.test() Testing numarray 1.4.1 on normal Python (2, 4, 2, 'final', 0) on platform linux2 numarray.numtest: 2.89 ((0, 1205), (0, 1205)) numarray.ieeespecial: 0.07 (0, 86) numarray.records: 0.05 (0, 48) numarray.strings: 0.17 (0, 189) numarray.memmap: 0.09 (0, 82) numarray.objects: 0.14 (0, 105) numarray.memorytest: 0.01 (0, 16) numarray.examples.convolve: 0.09 ((0, 20), (0, 20), (0, 20), (0, 20)) numarray.convolve: 0.06 (0, 45) numarray.fft: 0.11 (0, 75) After the above, nothing happens. "ps ax" show a lot of time being used. Looks like an infinite loop. Process had to be killed.

2 1

numarray 1.4.0: unitary operator "+" fails
by Edward C. Jones 28 Oct '05

28 Oct '05

The unitary "+" does not work for numarray 1.4.0: > python Python 2.4.2 (#1, Oct 1 2005, 18:47:35) [GCC 4.0.2 20050821 (prerelease) (Debian 4.0.1-6)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numarray >>> arr = numarray.zeros((2,2)) >>> +arr Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: copy

1 0

ANN: numarray-1.4.0
by Todd Miller 28 Oct '05

28 Oct '05

Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modeled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.4.0 I. ENHANCEMENTS 1. Speed improvement for numarray operators. The Python level hook mapping numarray operators onto universal functions has been moved down to C. 2. Speed improvement for string-array comparisons, any(), all(). String correlation is ~10x faster. 3. Better operation with py2exe to help it automatically detect the core numarray extensions to include in an installer. 4. scipy newcore compatible lower case type names (e.g. int32 not Int32) 5. scipy newcore 'dtype' keyword and .dtypechar attribute. II. BUGS FIXED / CLOSED 1323355 Apps fail with import_libnumarray 1315212 Infinite loop converting some scalar strings into a list 1298916 rank-0 tostring() broken 1297948 records.array fails to create empty fields 1286291 import sys missing from array_persist.py 1286168 Generic sequences in ``strings.array()`` 1236392 Outdated web link in announcements 1235219 LinearAlgebraError not imported in linear_algebra See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be backward binary compatible with numarray-1.3.x WHERE ----------- Numarray-1.4.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://www.stsci.edu/resources/software_hardware/numarray Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.4.0 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller(a)stsci.edu

3 5

Re: Numeric: Change of v24.0
by Travis Oliphant 25 Oct '05

25 Oct '05

Weissmann Markus wrote: > Hi folks, > > Before beginning my rant: Thank you very much for Numeric - you're > doing a great job! > <rant> > I'm a bit pissed as you changed the - already released tarball of > version 24.0; > This is a real mess, as all build systems out there have problems > distinguishing versions, > the correct tarballs (hashes will fail) etc. I'm not sure what you are talking about. The old tarball was always a beta release. It took a long time to get the actual release out, but that is due to the lack of reports. > I don't think this is my personal, or my projects impression, as e. > g. Novell [1] & O'Reilly [2] > clearly were under the impression that the old tarball was v24.0. > </rant> Then they were mistaken. It was always clearly marked as version 24b2 (beta-release 2). > Whatever happened that lead to this confusion, it would be great if > you could at least > make a version bump to 24.1 right now so everyone can be sure that > 24.1 is 24.1 in contrast > to 24.0 not being 24.0 all the time. > Someone could make a 24.1, but I don't see the need, and I'm not really interested in doing that. I'm done with Numeric development as I've moved to scipy core. I'm sorry for your consternation. -Travis

1 0

numarray.objects
by Joris van Zwieten 24 Oct '05

24 Oct '05

Hi, Are arrays of objects (i.e. numarray.objects) still supported in Numeric3/Scipy? (I'm working on an application that should be able to work with arrays of numbers and arrays of objects. Without numarray.objects-like functionality, this would require using numarray arrays for the numbers, and nested python lists for the objects. ;) Thanks, Joris van Zwieten

1 0

Numarray - doesnt pass installation test under Windblows (was freezing wxmpl etc)
by Tim Burgess 21 Oct '05

21 Oct '05

Hi Todd, Yesterday I claimed that numarray wasnt passing the tests under python 2.4. Actually I was testing the 1.3.3 Windows exe installer version - not my cvs derrived version. *blush* The cvs version tests perfectly. I will take you up on your offer of a Windows exe from CVS please Todd. Perhaps it means it is time for a 1.3.3.1 version? :-} -- Tim Burgess IT Consultant RedHat Certified Engineer TBITC Pty Ltd Professional Computer Support for Business timb(a)tbitc.com Mobile 0422 942 972 Office 85 662 016 http://www.tbitc.com

2 1

numarray - was trying to freeze wxmpl matplotlib and wxpython
by Tim Burgess 20 Oct '05

20 Oct '05

Hi all, First thanks to all those who have been helping out - it is really appreciated. I have cross posted to the numarray, py2exe and matplotlib lists I have attempted to use numarray from cvs but I can't build it - no VC7 clang - did a penny drop anywhere? I am working with python 2.4.1 and numarray 1.3.3 for 2.4 (obviously) Since I couldnt build numarray - I looked at the new import code in the cvs init module and ripped it off and stuffed it into my numarray installation and then when that didnt work - I jammed it right up front of my application. It looks like this def main(): import numarray.numarrayall from numarray.numinclude import version as __version__ # stolen from next numarray version in cvs TjB import numarray._conv import numarray._sort import numarray._bytes import numarray._ufunc import numarray._ufuncBool import numarray._ufuncInt8 import numarray._ufuncUInt8 import numarray._ufuncInt16 import numarray._ufuncUInt16 import numarray._ufuncInt32 import numarray._ufuncUInt32 import numarray._ufuncFloat32 import numarray._ufuncFloat64 import numarray._ufuncComplex32 import numarray._ufuncComplex64 import numarray._ndarray import numarray._numarray import numarray._chararray import numarray._objectarray import numarray.memory import numarray._converter import numarray._operator import numarray._numerictype import numarray.libnumarray import numarray.libnumeric import numarray._ufuncInt64 import numarray._ufuncUInt64 print numarray.__version__ application = BoaApp(0) Still no go - but a changed error message.... grrr I get Traceback (most recent call last): File "AEMdaApp.py", line 81, in ? File "AEMdaApp.py", line 41, in main File "numarray\__init__.pyc", line 42, in ? File "numarray\numarrayall.pyc", line 1, in ? File "numarray\numerictypes.pyc", line 35, in ? File "numarray\numinclude.pyc", line 4, in ? File "numarray\_ndarray.pyc", line 9, in ? File "numarray\_ndarray.pyc", line 7, in __load ImportError: init_ndarray: can't find memory.new_memory ok I confess my bottom lip trembled.... then - probably because I was reading the numarray manual this morning - I went to my numarray install and decided to run testall.py I got F:\Python24\Lib\site-packages\numarray>\python24\python testall.py Testing numarray 1.3.3 on normal Python (2, 4, 1, 'final', 0) on platform win32 ********************************************************************** File "F:\python24\lib\site-packages\numarray\numtest.py", line 2843, in cache p ss Failed example: cPickle.loads(cPickle.dumps(arange(5)+1j)) Exception raised: Traceback (most recent call last): File "F:\Python24\lib\doctest.py", line 1243, in __run compileflags, 1) in test.globs File "<doctest cache pass[968]>", line 1, in ? PicklingError: Can't pickle <built-in function memory_from_string>: it's no the same object as memory.memory_from_string ********************************************************************** File "F:\python24\lib\site-packages\numarray\numtest.py", line 2849, in cache p ss Failed example: p = p.dump(a) Exception raised: Traceback (most recent call last): File "F:\Python24\lib\doctest.py", line 1243, in __run compileflags, 1) in test.globs File "<doctest cache pass[973]>", line 1, in ? PicklingError: Can't pickle <built-in function memory_from_string>: it's no the same object as memory.memory_from_string ********************************************************************** File "F:\python24\lib\site-packages\numarray\numtest.py", line 2850, in cache p ss Failed example: p = p.dump(b) Exception raised: and so on for quite some time. Hmm the message about memory.memory_from_string and my applications ImportError: init_ndarray: can't find memory.new_memory are these a little related? Am I clutching at straws? Yet my app runs fine under any ide I wish to use. WAIT! - my install of numarray is broken.... I deleted it and reinstalled - still broken I installed on my laptop and tested - broken there too My legacy python 2.3 and numarray 1.3.3 installation passes all the tests just fine. Is there a problem with python2.4.1 and numarray 1.3.3? Could some kind soul build me a numarray windows installer for python 2.4 from cvs - please - so that I can see if that works better with py2exe. I need a stiff drink -- Tim Burgess IT Consultant RedHat Certified Engineer TBITC Pty Ltd Professional Computer Support for Business timb(a)tbitc.com Mobile 0422 942 972 Office 85 662 016 http://www.tbitc.com

4 4