Mailman 3 February 2005 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 16 May '11

16 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

Matlab is a tool for doing numerical computations with matrices and vectors.
by Colin J. Williams 10 Mar '05

10 Mar '05

The subject line is from http://www.math.utah.edu/lab/ms/matlab/matlab.html I've seen little use of the words "matrix" or "vector" in the numarray stuff. These are one or two dimensional structures. The Numeric/numarray focus is on multidimensional arrays. Some years ago, Huaiyu Zhu made a number of postings advocating the introduction of a Python-based system permitting the usual matrix operations. It seems that the culmination of these efforts was Matpy (http://matpy.sourceforge.net/), which is not currently active. I was impressed with the case made by Huaiyu Zhu and have attempted to carry on in this direction. An outline of this work is at: http://www3.sympatico.ca/cjw/IntroToPyMatrix.pdf. This work was based on numarray, partly because it was generally considered to be the way ahead and partly because it purported to provide a class structure. numarray has proved to be a bit of a moving target. I am continuing this work in the hope that some class-based structures will come out of the current Numeric3 discussions. The hope is also that there is a need out there for matrix operations, as distinct from array operations. Colin W.

3 5

ANN: PyQwt3D-0.1 released
by Gerard Vermeulen 27 Feb '05

27 Feb '05

What is PyQwt3D? - it is a set of Python bindings for the QwtPlot3D C++ class library which extends the Qt framework with widgets for 3D data visualization. PyQwt3D inherits the snappy feel from QwtPlot3D. The examples at http://pyqwt.sourceforge.net/pyqwt3d-examples.html show how easy it is to make a 3D plot and how to save a 3D plot to an image or an (E)PS/PDF file. - it requires and extends PyQt, a set of Python bindings for Qt. - it supports the use of PyQt, Qt, Qwt, the Numerical Python extensions (either Numeric, or numarray or both) and optionally SciPy in a GUI Python application or in an interactive Python session. - it runs on POSIX, MacOS/X and Windows platforms (practically any platform supported by Qt and Python). The home page of PyQwt3D is http://pyqwt.sourceforge.net. PyQwt3D-0.1 supports: 1. Python-2.4 downto -2.3. 2. PyQt-3.14 downto -3.12. 3. SIP-4.2 downto -4.0, but SIP-4.2 is recommended. PyQwt3D does *not* support SIP-3.x (I tried, but failed). 4. Qt-3.3.4 downto -2.3.0. 5. QwtPlot3D-0.2.4-beta. Have fun -- Gerard Vermeulen

1 0

Incompatibility betwewn numarray 1.2.2 and Numeric
by Nadav Horesh 24 Feb '05

24 Feb '05

Numeric and numarray arrays became incopatible --- Is it a bug or a design change? In [10]: import Numeric as N In [11]: A = N.arange(6.0) In [12]: import numarray as n In [13]: a = n.arange(6.0) In [14]: a+A # numarray op Numeric --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /tmp/numarray-1.2.2/<console> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in __add__(self, operand) 933 return operand.__radd__(self) 934 else: --> 935 return ufunc.add(self, operand) 936 937 def __radd__(self, operand): /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in _cache_miss2(self, n1, n2, out) 918 919 def _cache_miss2(self, n1, n2, out): --> 920 (in1, in2), inform, scalar = _inputcheck(n1, n2) 921 922 mode, win1, win2, wout, cfunc, ufargs = \ /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in _inputcheck(*inargs) 374 # see if it can be made into an array 375 try: --> 376 inarg = _nc.array(inarg) 377 except TypeError: 378 raise TypeError( /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in array(sequence, typecode, copy, savespace, type, shape) 399 if shape is None: 400 shape=() --> 401 return fromlist([sequence],type,shape) 402 403 def asarray(seq, type=None, typecode=None): /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in fromlist(seq, type, shape, check_overflow, typecode) 245 tshape = _frontseqshape(seq) 246 if shape is not None and _gen.product(shape) != _gen.product(tshape): --> 247 raise ValueError("shape incompatible with sequence") 248 ndim = len(tshape) 249 if ndim <= 0: ValueError: shape incompatible with sequence In [15]: A+a # The other way around --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /tmp/numarray-1.2.2/<console> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in __radd__(self, operand) 940 return operand.__add__(self) 941 else: --> 942 r = ufunc.add(operand, self) 943 if isinstance(r, NumArray): 944 r.__class__ = self.__class__ /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in _cache_miss2(self, n1, n2, out) 918 919 def _cache_miss2(self, n1, n2, out): --> 920 (in1, in2), inform, scalar = _inputcheck(n1, n2) 921 922 mode, win1, win2, wout, cfunc, ufargs = \ /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in _inputcheck(*inargs) 374 # see if it can be made into an array 375 try: --> 376 inarg = _nc.array(inarg) 377 except TypeError: 378 raise TypeError( /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in array(sequence, typecode, copy, savespace, type, shape) 399 if shape is None: 400 shape=() --> 401 return fromlist([sequence],type,shape) 402 403 def asarray(seq, type=None, typecode=None): /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in fromlist(seq, type, shape, check_overflow, typecode) 245 tshape = _frontseqshape(seq) 246 if shape is not None and _gen.product(shape) != _gen.product(tshape): --> 247 raise ValueError("shape incompatible with sequence") 248 ndim = len(tshape) 249 if ndim <= 0: ValueError: shape incompatible with sequence -------------------------- Nadav.

2 1

ANN: numarray-1.2.2
by Todd Miller 23 Feb '05

23 Feb '05

Release Notes for numarray-1.2.2 Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on array memory mapped onto files. I. ENHANCEMENTS 1. Support for user defined universal functions with arbitrary numbers of parameters. See numarray/Examples/ufunc/setup_special.py 2. Additional modifications to support numarray operation with scipy_base. 3. Import time roughly halved thanks to Rory Yorke. 4. Improved sorting functions from Chuck Harris. 5. Simplified array() and fromfile() from Rory Yorke. 6. Port of Numeric's dotblas extension for faster matrix multiplication when linked against ATLAS, LAPACK, BLAS, etc. 7. Improvements to array indexing speed from Francesc Alted and Tim Hochberg. II. BUGS FIXED / CLOSED 1120138 Document ieeespecial 1119411 No file named "LinearAlgebra2/setup.py" 1119265 'print' fails on masked arrays 1111765 python setup.py bdist_rpm fails (1.2a) 1110655 Problem with the ufunc .sum() and a rank-0 array. 1110607 f2py file paths on windows 1108374 count method in string.array wrong 1102874 remove use of basestring in fromfile for Python 2.2 1100435 Remove NUL bytes in nd_image/test.py 1099974 .sum() gives wrong results 1091474 log(0) should be -inf 1089116 CharArray maxLen bug 1088817 Convolution with complex kernels gives wrong answer 1087378 LinAlgError is a string, not an Exception 1087304 Add toUpper() and toLower to CharArray 1087158 Float64 == records.CharType 1075802 masked_print_option produces incorrect output 1075795 float() and int() don't work on masked arrays 1075794 correlate2d should do autocorrelations 1075793 convolve2d/correlate2d should create output arrays 1075789 resized() pads with nulls not blanks: numarray 1.2.2 1069032 A little bug in setup.py 1068768 KeyboardInterrupt shows strange behavior 1067238 memory leak in determinant 1056649 Memory leak with dot and transpose 1052488 Modify sum() result type for boolean arrays 1050292 Side effects of putmask 1047272 Update license references 1045518 A disconnected numarray rant 1044885 complex number ieee oddities 1044788 wrong typecode 1038528 Making PyArray_CanCastSafely safer on 64-bit machines 1038397 numarray sum() has a possible memory leak 1038064 tofile() of zero length array 1037038 boolean cummulative operator leak 1036327 Buffer overflow in sum and add_Int64_reduce 1035701 numerical array from string list loops 1033860 Wrong array indexing when byteorder is changed 1031883 MA __setslice__ bug 1031356 Sequences with Python longs --> Int64 1030023 problem with matrixmultiply 1029547 IndexError in 4.8 Index Arrays example 1028782 matrixmultiply reference counting 1028738 atlas/blas library listings in addons.py 1025160 gcc error during install: duplicate case value 1022523 chararray concatentation problem 1018252 bad memory leak in reduce/accumulate See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be backward binary compatible with numarray-1.1.1. WHERE ----------- Numarray-1.2.2 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.2.2 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller(a)stsci.edu

1 0

ANN: PyQwt-4.2 released
by Gerard Vermeulen 23 Feb '05

23 Feb '05

What is PyQwt? - it is a set of Python bindings for the Qwt C++ class library which extends the Qt framework with widgets for scientific, engineering and financial applications. It provides a widget to plot 2-dimensional data and various widgets to display and control bounded or unbounded floating point values. - it requires and extends PyQt, a set of Python bindings for Qt. - it supports the use of PyQt, Qt, Qwt, the Numerical Python extensions (either Numeric, or numarray or both) and optionally SciPy in a GUI Python application or in an interactive Python session. - it runs on POSIX, MacOS/X and Windows platforms (practically any platform supported by Qt and Python). The home page of PyQwt is http://pyqwt.sourceforge.net. PyQwt-4.2 is a maintenance release to keep up with the recent releases of PyQt-3.14 and its code generator SIP-4.2. PyQwt-4.2 supports: 1. Python-2.4 downto -2.1. 2. PyQt-3.14 downto -3.10. 3. SIP-4.2 downto -4.0, and SIP-3.11 downto -3.10. 4. Qt-3.3.4 downto -2.3.0. 5. Qwt-4.2.0. Have fun -- Gerard Vermeulen

1 0

RE: [Numpy-discussion] PEP updated
by Sebastien.deMentendeHorne＠electrabel.com 22 Feb '05

22 Feb '05

> For sequence behaviour len, indexing and iteration should work > consistently, i.e., "a[i] for i in xrange(len(a))" and "x for x in a" > should identically generate the contents of a. Currently indexing and > iteration occur along the first dimension, which for rank 0 arrays > does not exist. I understand that there is no burning desire to > completely switch to Matlab-style one-dimensional indexing. The > alternatives are then giving rank 0 arrays exceptional semantics > inconsistent with other arrays or raising an exception if len or > iteration is applied to a rank 0 array. Without exceptional semantics > the only index expression that makes sense is a[()]. I would favor > consistency here and raise an exception but there is one matter that > raises some doubt: do programs expect that if __len__ and __iter__ > exist they will not fail? > Looking at the code I wrote that must deal with special scalar vs array cases, I noticed that an important feature that the rank-0/scalar should have is a correct indexing behaviour with: a[..., NewAxis] or a[NewAxis] or a[NewAxis, ...] By the way, isn't it easier to find a solution to this problem by looking at specific pieces of code that currently needs an ugly case for scalar ? Moreover, should this rank-0 array behaviour of scalar be extended to other "scalar" objects ? Should we be able to use: p = (3,4) a = p[NewAxis] a.shape = (1,) a.typecode() ==> 'O' ?

1 0

Numeric3 PEP
by Colin J. Williams 22 Feb '05

22 Feb '05

The PEP has: In Python there will be a hierchial set of classes defined: GenericArray (type classes) BooleanArray (Bool) NumericArray (XX is the bit-width of the type) SignedArray UnsignedArray IntegerArray SignedIntegerArray (IntXX) UnsignedIntegerArray (UIntXX) FloatArray (FloatXX) ComplexArray (FloatXX) FlexibleArray CharacterArray StringArray (String) UnicodeArray (Unicode) VoidArray (Void) ObjectArray It seems that the record type is to be handled under the VoidArray. I hope that it will permit the setting and reading of fields. For example, if recs is an instance of an Array of records, then it should be possible to write: >>> recs[22, 5].sex= 'F' >>> recs[22, 5].sex F It is not clear, from the current draft, what is intended for ObjectArray. Can these be any Python objects tuple, list etc. or instances of any user defined class? There is also a tofile method ( I would prefer toFile), does this mean that pickling would be used for the object instances? The term WRITABLE is used, is this different from "mutable", the term used by Python? "Methods such as x.r(), x.i(), and x.flatten() are proposed.". Why not use properties, x.r, x.i and x.flatten. The parentheses appear redundant. Colin W.

7 7

More PEP comments
by Tim Hochberg 20 Feb '05

20 Feb '05

From the PEP: > integers in index arrays can be negative to mean selection from the end > of the array. Indices that are too large will be taken to mean the > last element. Indicies that are too negative will mean > the first element. > Here's another little piece of the PEP that I feel should be reconsidered. In the absence of a compelling reason to do otherwise, the indices in index arrays should behave just as if one were using that index directly as in index tuple. And, index tuples should match the behaviour Python lists. In other words: alist[i] == anarray[i] == anarray(array([i]))[0] In order to make this equivalence work, both anarray[i] and anarray(array([i])) need to raise index errors when i is too large or too negative. And for consistency this rule should continue to apply when the index is a tuple rather than a simple integer index. This is more or less how numarray works and I think that's the right choice. [However, I did notice that numarray 1.1 produced nonsensical answers when index arrays with values that were too negative, when they were too positive they worked as I would expect and produced an error]. -tim

1 0

Poll on a Rank-0 arrays: please give +1, -1, or 0
by Travis Oliphant 20 Feb '05

20 Feb '05

We need to get some opinions regarding the recent discussion on rank-0 arrays. Please give your vote +1 (like), -1 (hate), or 0 (don't care) for **each** of the following possibilities. Feel free to clarify your vote if you need to. 1) Rank-0 arrays are always returned from uarray operations, Python scalars available on request only. (*If* arrayobject get's into Python core, Guido has agreed to let rank-0 integer arrays be used as index variables). Rank-0 arrays are made to resmbe scalar of 2) Rank-0 arrays are never returned from uarray operations (unless using asarray on a scalar), and when a rank-0 array naturally appears in the calculation, an appropriate Python scalar is returned (note that this would lose precision for long doubles unless a new Python object was created). 3) The current compromise is continued and for some types a Python scalar is returned, while for other types a rank-0 array is returned 4) Rank-0 arrays are made equivalent to Python scalars and a new Python scalar for each fundamental data type supported by uarray is constructed (rank-0 arrays would still probably be used internally, but users would not have to know this). The new Python-scalars would inherit from an existing Python scalar where appropriate and would have the same attributes and methods of uarrays (very likely at least initially they would be seemlessly converted to rank-0 arrays when "mixed" operations occur).

4 4