Mailman 3 November 2005 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 16 May '11

16 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

scipy cygwin compilation error
by Jordan Dawe 04 Dec '05

04 Dec '05

I just tried to install scipy under cygwin using both 0.61 and the current svn. I get this: gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.18-i686-2.4/scipy/base/umath.dll build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UBYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2215: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `USHORT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2233: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UINT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2251: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `ULONG_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2269: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `BYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2305: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o:/usr/local/core/build/src/scipy/base/src/umathmodule.c:2325: more undefined references to `_feraiseexcept' follow build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_checkfperr': /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_clearfperr': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_GenericFunction': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `construct_reduce': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UBYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2215: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `USHORT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2233: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UINT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2251: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `ULONG_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2269: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `BYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2305: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o:/usr/local/core/build/src/scipy/base/src/umathmodule.c:2325: more undefined references to `_feraiseexcept' follow build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_checkfperr': /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_clearfperr': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_GenericFunction': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `construct_reduce': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.18-i686-2.4/scipy/base/umath.dll" failed with exit status 1 Any suggestions? Jordan

2 1

Numeric3.0 - Currently available under scipy/base as version 0.4.1 (Windows)
by Colin J. Williams 04 Dec '05

04 Dec '05

I have a package based on subclassing numarray, which is working satisfactorily, and am looking at the possibility of transferring the package to work under the revised Numeric. My feeling is that the transfer is probably feasible but that it would be premature to work on it at this time. One of the problems is the cluttered namespace, through the use of "from X import *". This is a style which is deprecated, see page 401 of Alex Martelli's /Python in a Nutshell/. Another problem, at this stage, is that many doc strings are missing and that some which exist are a little cryptic. I would like to take another look when the next win32 binaries are available. Some further thoughts on the present state of Numeric3.0 are available here <http://www3.sympatico.ca/cjw/scipy1/>. Colin W,

2 3

Your PayPal Account Could Be Suspended
by PayPal 28 Nov '05

28 Nov '05

1 0

Re: help in improving data analysis code
by gf 25 Nov '05

25 Nov '05

From: Francesc Altet <faltet@ca...> Re: help in improving data analysis code 2005-11-25 07:32 >>A Divendres 25 Novembre 2005 16:27, Francesc Altet va escriure: > > print nn[argsort(abs(nn_c-nn_c.mean()),0)][:-int(sz*0.10),0] >> >> Ups. I have had a confusion. This should work better ;-) >> >> print nn[argsort(abs(nn-nn.mean()),0)][:-int(sz*0.10),0] Hi Francesc, Thank you for the suggestions. Your code is performing a different task than mine was. In particular, I believe it does not 're-mean' the data after removing each point. However, based on the great ideas from your code, I now have the function below that looks to be more efficient (although I haven't measured it). Any additional suggestions are appreciated. -g ==== from numarray import argsort, floor, absolute def eliminate_outliers(data,frac): num_to_eliminate = int(floor(data.size())*frac) for i in range(num_to_eliminate): data = data[argsort(absolute(data-data.mean()),0)][:-1,0] return data if __name__ == "__main__": from numarray.mlab import rand sz = 100 nn = rand(sz,1) nn[:10] = 20*rand(10,1) nn[sz-10:] = -20*rand(10,1) print eliminate_outliers(nn,0.10)

1 0

help in improving data analysis code
by gf 24 Nov '05

24 Nov '05

Hi, I am a newbie to Numeric/numarray programming and would appreciate your help in improving the code below (which I'm sure is quite ugly to an experienced numarray programmer). An analysis we are carrying out requires the following: 1. evaluate the mean of a set of data 2. eliminate the data point farthest from the mean 3. repeat steps 1 and 2 until a certain specified fraction of points has been eliminated. Since this analysis will have to be performed (probably repeatedly) on approximately ten thousand data sets, each of which contains 100-500 points, I would like the code to be as fast as possible. Thanks for your help. -g ==== from numarray import add, array, asarray, absolute, argsort, floor, take, size def mean(m,axis=0): m = asarray(m) return add.reduce(m,axis)/float(m.shape[axis]) def eliminate_outliers(dat,frac): num_to_eliminate = int(floor(size(dat,0)*frac)) for i in range(num_to_eliminate): ind = argsort(absolute(dat-mean(dat)),0) sdat = take(dat,ind,0)[:,0] dat = sdat[:-1] return dat #-------------------------------------------------------------------- if __name__ == "__main__": from MLab import rand sz = 100 nn = rand(sz,1) nn[:10] = 20*rand(10,1) nn[sz-10:] = -20*rand(10,1) print eliminate_outliers(nn,0.10)

2 2

Problem compiling scipy-core
by Anoop 23 Nov '05

23 Nov '05

Hi. I have problems installing scipy-core from source code on a Rocks Cluster distribution of Linux running on x86-64 hardware. The error messages tell me: /usr/bin/g77 -shared build/temp.linux-x86_64-2.3/scipy/corelib/blasdot/_dotblas.o -L/usr/lib -lblas -lg2c -o build/lib.linux-x86_64-2.3/scipy/lib/_dotblas.so /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.a when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas /usr/bin/ld: cannot find -lblas collect2: ld returned 1 exit status I really need to get this running. Help? Thanks, Anoop

2 1

ANN: PyTables 1.2
by Francesc Altet 23 Nov '05

23 Nov '05

PyTables is a library to deal with very large datasets. It leverages the excellent HDF5 and numarray libraries to allow doing that in a very efficient way using the Python language. More info in: http://pytables.sourceforge.net/ ========================= Announcing PyTables 1.2 ========================= The PyTables development team is happy to announce the availability of a new major version of PyTables package. This version sports a completely new in-memory tree implementation based around a *node cache system*. This system loads nodes only when needed and unloads them when they are rarely used. The new feature allows the opening and creation of HDF5 files with large hierarchies very quickly and with a low memory consumption (the object tree is no longer completely loaded in-memory), while retaining all the powerful browsing capabilities of the previous implementation of the object tree. You can read more about the dings and bells of the new cache system in: http://www.carabos.com/downloads/pytables/NewObjectTreeCache.pdf Also, Jeff Whitaker has kindly contributed a new module called tables.NetCDF. It is designed to be used as a drop-in replacement for Scientific.IO.NetCDF, with only minor actions to existing code. Also, if you have the Scientific.IO.NetCDF module installed, it allows to do conversions between HDF5 <--> NetCDF3 formats. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ If you want more info about this release, please check out the more comprehensive announcement message available in: http://www.carabos.com/downloads/pytables/ANNOUNCE-1.2.html Acknowledgments =============== Thanks to the users who provided feature improvements, patches, bug reports, support and suggestions. See THANKS file in distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last but not least, a big thank you to THG (http://www.hdfgroup.org/) for sponsoring many of the new features recently introduced in PyTables. --- **Enjoy data!** -- The PyTables Team -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"

3 5

Problems with linalg.cholesky?
by Andrew Jaffe 21 Nov '05

21 Nov '05

[Originally posted to g.c.p.user...] Hi all, In the newest incarnation of scipy_core, I am having trouble with the cholesky(a) routine: /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/linalg/basic_lite.py in cholesky_decomposition(a) 115 else: 116 lapack_routine = lapack_lite.dpotrf --> 117 results = lapack_routine('L', n, a, m, 0) 118 if results['info'] > 0: 119 raise LinAlgError, 'Matrix is not positive definite - Cholesky decomposition cannot be computed' LapackError: Parameter a is not contiguous in lapack_lite.dpotrf But this isn't true; I get this error even when I pass trivial and contiguous matrices such as the output of identity(). Other linalg routines (included complicated ones like singular_value_decomp) seem to work fine. Any ideas? Andrew

3 2

Memory mapped files in scipy core
by Travis Oliphant 21 Nov '05

21 Nov '05

I would appreciate understanding typically use cases for memory-mapped files. I am not sure I understand why certain choices were made for numarray's memmap and memmap slice classes. They seem like a lot of "extra" stuff and I'm not sure what all that stuff is for. Rather than just copy these over, I would like to understand what people typically want to do with memory-mapped files to see if scipy core doesn't already provide that. For example, write now I can open a file, use mmap to obtain a memory map object and then pass that object into frombuffer in scipy_core to get an ndarray whose memory maps a file on disk. Now, this ndarray can be sliced and indexed and manipulated all the while referring to the file on disk (well technically, I suppose, the memory-mapped object would need to be flushed to synchronize). Now, I could see wanting to make the process of opening the file, getting the mmap object and setting it's buffer to the array object a little easier. Thus, a simple memmap class would be a useful construct -- I could even see it inheriting from the ndarray directly and adding a few methods. I guess I just don't see why one would care about a memory-mapped slice object, when the mmaparray sub-class would be perfectly useful. On a related, but orthogonal note: My understanding is that using memory-mapped files for *very* large files will require modification to the mmap module in Python --- something I think we should push. One part of that process would be to add the C-struct array interface to the mmap module and the buffer object -- perhaps this is how we get the array interface into Python quickly. Then, if we could make a base-type mmap that did not use the buffer interface or the sequence interface (similar to the bigndarray in scipy_core) and therefore by-passed the problems with Python in those areas, then the current mmap object could inherit from the base class and provide current functionality while still exposing the array interface for access to >2GB files on 64-bit systems. Who would like to take up the ball for modifying mmap in Python in this fashion? -Travis

3 2