Mailman 3 February 2007 - NumPy-Discussion

Questions about cross-compiling extensions for mac-ppc and mac-intel
by Zachary Pincus 26 Feb '07

26 Feb '07

Hi folks, I've been doing a lot of web-reading on the subject, but have not been completely able to synthesize all of the disparate bits of advice about building python extensions as Mac-PPC and Mac-Intel fat binaries, so I'm turning to the wisdom of this list for a few questions. My general goal is to make a double-clickable Mac installer of a set of tools built around numpy, numpy's distutils, a very hacked-up version of PIL, and some fortran code too. To this end, I need to figure out how to get the numpy distutils to cross-compile, generating PPC code and Intel code in separate builds -- and/or generating a universal binary all in one go. (I'd like to distribute a universal version of numpy, but I think that my own code needs to be built/distributed separately for each architecture due to endian- ness issues.) Is there explicit support in distutils for this, or is it a matter of setting the proper environment variables to entice gcc and gfortran to generate code for a specific architecture? One problem is that PIL is a tricky beast, even in the neutered form that I'm using it. It does a compile-time check for the endian-ness of the system, and a compile-time search for the zlib to use, both of which are problematic. To address the former, I'd like to be able to (say) include something like 'config_endian --big' on the 'python setup.py' command-line, and have that information trickle down to the PIL config script (a few subpackages deep). Is this easy or possible? To address the latter, I think I need to have the PIL extensions dynamically link against '/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/ libz.dylib' which is the fat-binary version of the library, using the headers from '/Developer/SDKs/MacOSX10.4u.sdk/usr/include/zlib.h '. Right now, PIL is using system_info from numpy.distutils to find the valid library paths on which libz and its headers might live. This is nice and more or less platform-neutral, which I like. How best should I convince/configure numpy.distutils.system_info to put '/ Developer/SDKs/MacOSX10.4u.sdk/usr/{lib,include}' on the output to get_include_dirs() and get_lib_dirs()? Thanks for any advice or counsel, Zach Pincus Program in Biomedical Informatics and Department of Biochemistry Stanford University School of Medicine

3 5

NumPy array from ctypes data
by Andrew McMurry 26 Feb '07

26 Feb '07

How can I create a NumPy array which is backed by ctypes data? ie create a NumPy array from a ctypes one without any data copying? So that changing an element of the NumPy array will alter the underlying ctypes array. I have C structures that contain numerical data and would like to access that data through the NumPy array interface. I have tried the following: PyObject *make_dbl_array(double *data, int n) { npy_intp dim = n; return PyArray_SimpleNewFromData(1, &dim, NPY_DOUBLE, (void *)data); } But when called via ctypes (with restype set to ctypes.py_object) it gives a bus error. Andrew

1 0

the neighbourhood of each element of an array
by joris＠ster.kuleuven.ac.be 25 Feb '07

25 Feb '07

Hi, Given a (possibly masked) 2d array x, is there a fast(er) way in Numpy to obtain the same result as the following few lines? d = 1 # neighbourhood 'radius' Nrow = x.shape[0] Ncol = x.shape[1] y = array([[x[i-d:i+d+1,j-d:j+d+1].ravel() for j in range(d,Ncol-d)] \ for i in range(d,Nrow-d)]) What you get is an array containing all the elements in a neighbourhood for each element, disregarding the edges to avoid out-of-range problems. The code above becomes quite slow for e.g. a 2000x2000 array. Does anyone know a better approach? Ciao, Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

5 5

Fortran order arrays to and from numpy arrays
by Alexander Schmolck 25 Feb '07

25 Feb '07

Hi, I'm currently puzzling over how to best convert (column major order) matlab arrays to numpy arrays and vice versa -- I'm looking for a solution that's simple, general and reasonably fast -- being also applicable to Numeric arrays would be a plus (I'd like to retain Numeric compatibility for the current version of the code). The solution that mlabwrap (http://mlabwrap.sf.net) currently employs to transfer arrays between matlab and python is to have a low level C++ extension (mlabraw) copying the data into a new numpy array created with Pyarray_SimpleNew and using a simple or nested for-loop to copy the data around (only rank up to 2 is currently handled), e.g. //matlab matrix -> numpy for (mwIndex lCol = 0; lCol < lCols; lCol++) { double *lDst = (double *)(lRetval->data) + lCol; for (mwIndex lRow = 0; lRow < lRows; lRow++, lDst += lCols) { *lDst = *lPR++;}} //numpy 2d array -> matlab npy_intp lRowDelta = pStrides[0]/sizeof(T); npy_intp lColDelta = pStrides[1]/sizeof(T); for (npy_intp lCol=0; lCol < pCols; lCol++) { T *lSrc = pSrc + lCol*lColDelta; for (npy_intp lRow=0; lRow < pRows; lRow++, lSrc += lRowDelta) { *pDst++ = *lSrc;}} Some simple timing results suggest 2 things: 1. There is an enormous overhead associated with calling into matlab (e.g. just using mlabraw to evaluate ``sin(1);`` in matlab, without accessing the result the result is about 1000 times slower than ``math.sin(1)``. I have no idea where this rather huge overhead comes from, but I suspect there isn't much one can do about it. 2. Despite this overhead, copying around large arrays (e.g. >=1e5 elements) in above way causes notable additional overhead. Whilst I don't think there's a sane way to avoid copying by sharing data between numpy and matlab the copying could likely be done better. I guess the moral from 1 might be that ctypes will be the way to go for future version, I would assume that the overhead associated with ctypes doesn't matter that much and that it will be much less painful to maintain than having a C or C++ extension module, especially given that the Mathworks seem fond of changing the C API every minor release. I haven't used ctypes yet so I'd be interested to know what people think. The moral from 2. might be just to memcpy the data wholesale where possible and transpose and reshape afterwards or, in numpy, to set the fortran-order flag (and query fortran-contiguousness on copying to matlab) -- I suppose that specifying fortran order and tranposing/reshaping are pretty much equivalent? So matlab to numpy would look roughly so: a_fortran_order = memcpy_raw_matlab_data_into_numpy_array(ptr) a = a_fortran_order.T.reshape(a_fortran_order.shape) I would assume that this ought be on average more efficient than always transforming to C-contiguous (as some of the time the returned array will anyway be transposed afterwards or accessed in a fashion were column major is faster; the case that several operations that profit from row-major storage are carried out on the same array seems unlikely). Unfortunately I don't see an easy way to use the same approach the other way (matlab doesn't seem to offer much on the C level to manipulate arrays), so I'd presumably need something like: stuff_into_matlab_array(a.T.reshape(a.shape).copy()) the question is how to avoid doing two copies. Any comments appreciated, 'as

3 5

Managing Rolling Data
by Alexander Michael 23 Feb '07

23 Feb '07

I'm new to numpy and looking for advice on setting up and managing array data for my particular problem. I'm collecting observations of P properties for N objects over a rolling horizon of H sample times. I could conceptually store the data in three-dimensional array with shape (N,P,H) that would allow me to easily (and efficiently with strided slices) compute the statistics over both N and H that I am interested in. This is great, but the rub is that H, an interval of T, is a rolling horizon. T is to large to fit in memory, so I need to load up H, perform my calculations, pop the oldest N x P slice and push the newest N x P slice into the data cube. What's the best way to do this that will maintain fast computations along the one-dimensional slices over N and H? Is there a commonly accepted idiom? Fundamentally, I see two solutions. The first would be to essentially perform a memcpy to propagate the data. The second would be to manage the N x P slices as H discontiguous memory blocks and merely reorder the pointers with each new sample. Can I do either of these with numpy? Thanks, Alex

7 12

installating numpy 1.0.1 on AIX 5.3 using python 2.5 and xlf v10.1
by Hans-Joachim Ehlers 23 Feb '07

23 Feb '07

Given : AIX 5.3 xlf v10.1 python 2.5 numpy 1.0.1 Task: Building/installaing numpy on AIX 5.3 Problem: The install process can not determine the IBM Fortran compiler Running python ./numpy/distutils/fcompiler/ibm.py gives pretty much garbage on the screen and the resolut is 'None' . So i changed the line in ./numpy/distutils/fcompiler/ibm.py from 'version_cmd' : ["xlf"], to 'version_cmd' : ["xlf -qversion"], since xlf v10 supports the option where xlf v8 does not. Now running python ./numpy/distutils/fcompiler/ibm.py gives me: customize IbmFCompiler Couldn't match compiler version for 'IBM XL Fortran Enterprise Edition V10.1 for AIX \nVersion: 10.01.0000.0004' None Could anybody give me a solution/hint since i do not know what else has to be changed to make a match. BTW: I am not a python programmer. I just try to build numpy. Tia Hajo

1 0

dumb question about creating a complex array
by Mathew Yeates 22 Feb '07

22 Feb '07

given an array of floats, 2N columns and M rows, where the elements A[r,2*j] and A[r,2*j+1] form the real and imaginary parts of a complex number ....... What is the simplest way to create a complex array? It's a fairly large array so I want to keep copying to a minimum. (Actually, it's not a float array, its elements are byte sized, in case that matters) Mathew --

3 4

what goes wrong with cos(), sin()
by WolfgangZillig 22 Feb '07

22 Feb '07

Hi, I'm quite new to numpy/scipy so please excuse if my problem is too obvious. example code: import numpy as n print n.sin(n.pi) print n.cos(n.pi/2.0) results in: 1.22460635382e-016 6.12303176911e-017 I've expected something around 0. Can anybody explain what I am doing wrong here?

10 24

installation documentation
by Toon Knapen 22 Feb '07

22 Feb '07

Hi all, Is there detailed info on the installation process available. I'm asking because in addition to installing numpy on linux-x86, I'm also trying to install numpy on aix-power and windows-x86. So before bombarding the ml with questions, I would like to get my hands on all doc available (I already have the 'Guide to Numpy' of sept. 15 2006 but can not find all the info I'm looking for in there neither) Thanks in advance, Toon Knapen

2 2

Distributing prebuilt numpy and other extensions
by Zachary Pincus 21 Feb '07

21 Feb '07

Hello folks, I've developed some command-line tools for biologists using python/ numpy and some custom C and Fortran extensions, and I'm trying to figure out how to easily distribute them... For people using linux, I figure a source distribution is no problem at all. (Right?) On the other hand, for Mac users (whose computers by default don't have the dev tools, and even then would need to get a fortran compiler elsewhere) I'd like to figure out something a bit easier. I'd like to somehow provide an installer (double-clickable or python script) that does a version check and then installs an appropriate version of prebuilt binaries for numpy and my C and Fortran extensions. Is this possible within the bounds of the python or numpy distutils? Would setuptools be a better way to go? Preferably it would be a dead easy, one-step thing... Or is this whole idea problematic, and better to stick with source distribution in all cases? Thanks for any advice, Zach Pincus Program in Biomedical Informatics and Department of Biochemistry Stanford University School of Medicine

4 3