Mailman 3 November 2001 - NumPy-Discussion

Numeric Python EM Project has moved
by Rob 30 Nov '01

30 Nov '01

Its now at www.pythonemproject.com. I can be reached at rob(a)pythonemproject.com. All this has come about since @home is possibly suspending operation at midnite tonight :( Rob. Looks like I need to change my sig too :) -- The Numeric Python EM Project www.members.home.net/europax

1 0

equal() and complex
by Mike Romberg 30 Nov '01

30 Nov '01

I'm wondering if there is some good reason why equal(), not_equal(), nonzero() and the like do not work with numeric arrays of tyep complex. I can see why operators like less() and less_equal() do not work. But the pure equality ones seem like they should work. Or am I missing something :). Thanks, Mike Romberg (romberg(a)fsl.noaa.gov)

2 1

Using Reduce with Multi-dimensional Masked array
by sag＠hydrosphere.com 29 Nov '01

29 Nov '01

I posted the following inquiry to python-list(a)python.org earlier this week, but got no responses, so I thought I'd try a more focused group. I assume MA module falls under NumPy area. I am using 2 (and more) dimensional masked arrays with some numeric data, and using the reduce functionality on the arrays. I use the masking because some of the values in the arrays are 'missing' and should not be included in the results of the reduction. For example, assume a 5 x 2 array, with masked values for the 4th entry for both of the 2nd dimension cells. If I want to sum along the 2nd dimension, I would expect to get a 'missing' value for the 4th entry because both of the entries for the sum are 'missing'. Instead, I get 0, which might be a valid number in my data space, and the returned 1 dimensional array has no mask associated with it. Is this expected behavior for masked arrays or a bug or am I misusing the mask concept? Does anyone know how to get the reduction to produce a masked value? Example Code: >>> import MA >>> a = MA.array([[1,2,3,-99,5],[10,20,30,-99,50]]) >>> a [[ 1, 2, 3,-99, 5,] [ 10, 20, 30,-99, 50,]] >>> m = MA.masked_values(a, -99) >>> m array(data = [[ 1, 2, 3,-99, 5,] [ 10, 20, 30,-99, 50,]], mask = [[0,0,0,1,0,] [0,0,0,1,0,]], fill_value=-99) >>> r = MA.sum(m) >>> r array([11,22,33, 0,55,]) >>> t = MA.getmask(r) >>> print t None

4 6

Re: [Numpy-discussion] Meta: too many numerical libraries doing thesame
by Konrad Hinsen 28 Nov '01

28 Nov '01

"eric" <eric(a)enthought.com> writes: > The standard version that Robin Dunn distributes is compiled with MSVC. If > you build a small > extensions with gcc that make wxPython call, it'll link just fine, but > seg-faults during execution. > Does anyone know if the same sorta thing is true on the Unices? If it is, > and Numeric was written in C++ then you'd have to compile extension modules > that use Numeric arrays with the same compiler that was used to compile > Numeric. This can lead to all sorts of hassles, and it has made me lean If you rely on dynamic linking for cross-module calls, you'd have the same problem with Unix, as different compilers use different name-mangling schemes. One way around this would be to limit cross-module calls to C functions compiled with "C" linking. Better yet, don't rely on dynamic linking at all and export a module's C API via a Python CObject, as described in the extension manual, and declare all symbols as static (except for the module initialization function of course). In my experience that is the only method that works on all platforms, with all compilers. Of course this also assumes that interfaces are at the C level. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------

1 0

RE: [Numpy-discussion] Meta: too many numerical libraries doing thesame
by eric 27 Nov '01

27 Nov '01

Hey group, Blitz++ is very cool, but I'm not sure it would make a very good underpinning for reimplementing Numeric. There are 2 (well maybe 3) main points. 1. Blitz++ declares arrays in the following way: The first issue deals with how you declare arrays in Blitz++. Array<float,3> A(N,N,N); The big deal here is that the dimensionality of Array is a template parameter, not a constructor parameter. In other words, 2D arrays are effectively a different type than 3D arrays. Numeric, on the other hand represents arrays of all dimensions with a single class/type. For Python, this makes the most sense. I think you could fanagle some way of getting blitz to work, but I'm not sure it would be the desired elegant solution. I've also tinkered with building a simple C++ templated (non-blitz) implementation of Numeric for kicks, but kept coming back to using the dreaded void* to store the data arrays. I still haven't completely given up on a templated solution, but it wasn't as obvious as I thought it would be. 2. Compiling Blitz++ is slooooow. scipy.compiler spits out 200-300 line extension modules at the most. Depending on hox complicated expressions are, it can take .5-1.5 minutes to compile a single extension funtion on an 850 MHz PIII. I can't imagine how long it would take to compile Numeric arrays for 1 through 11 dimensions (the most blitz supports as I remember) for all the different data types with 100s of extension functions. The cost wouldn't be linear because you do pay a one time hit for some of the template instantiation. Also, I've heard gcc 3.0 might be better. Still, it'd be a painful development process. 3. Portability. This comes at two levels. The first is that blitz++ has heavy duty requirements of the compiler. gcc works fine which is a huge plus, but a lot of other compilers don't. MSVC is the most notable of these because it is so heavily used on windows. The second level is the portability of C++ extension modules in general. I've run into this on windows, but I think it is an issue pretty much everywhere. For example, MSVC and GCC compiled C extension libraries can call each other on Windows because they the are binary compatible. C++ classes are _not_ binary compatible. This has come up for me with wxPython. The standard version that Robin Dunn distributes is compiled with MSVC. If you build a small extensions with gcc that make wxPython call, it'll link just fine, but seg-faults during execution. Does anyone know if the same sorta thing is true on the Unices? If it is, and Numeric was written in C++ then you'd have to compile extension modules that use Numeric arrays with the same compiler that was used to compile Numeric. This can lead to all sorts of hassles, and it has made me lean back towards C as the preferred language for something as fundemental as Numeric. (Note that I do like C++ for modules that don't really define an API called by other modules). Ok, so maybe there's a 4th point. Paul D. pointed out that blitz isn't much of a win unless you have lazy evaluation (which scipy.compiler already provides). I also think improved speed _isn't_ the biggest goal of a reimplementation (although it can't be sacrificed either). I'm more excited about a code base that more people can comprehend. Perry G. et al's mixed Python/C implementation with the code generators is a very good idea and a step in this direction. I hope the speed issues for small arrays can be solved. I also hope the memory mapped aspect doesn't complicate the code base much. see ya, eric

1 0

BLAS and innerproduct
by R.M.Everson＠exeter.ac.uk 27 Nov '01

27 Nov '01

Hello, So far as I can tell Numeric.dot(), which uses innerproduct() from multiarraymodule.c doesn't call the BLAS, even if Numeric was compiled against native BLAS. This means (at least on my machine) that X = ones((150, 16384), 'd') C = dot(X, tranpose(X)) is about 15 times as slow as the comparable operations in Matlab (v6), which does, I think, use the native BLAS. I guess that multiarray.c is not particularly optimised to use the BLAS because of the difficulties of coping with all sorts of types (float32, int64 etc), and with non-contiguous arrays. The innerproduct is so basic to most of the work I use Numeric for that a speed up here would make a big difference. I'm thinking of patching multiarray.c to use the BLAS when it can, but before I start are there good reasons for doing something different? Any advice gratefully received! Cheers, Richard. -- Department of Computer Science, Exeter University Voice: +44 1392 264065 R.M.Everson(a)exeter.ac.uk Secretary: +44 1392 264061 http://www.dcs.ex.ac.uk/people/reverson Fax: +44 1392 264067

5 4

RE: [Numpy-discussion] Re: Re-implementation of Python Numerical arrays (Numeric) available for download
by Krishnaswami, Neel 27 Nov '01

27 Nov '01

Perry Greenfield [mailto:perry@stsci.edu] wrote: > > > > I know large datasets were one of your driving factors, but I really > > don't want to make performance on smaller datasets secondary. > > That's why we are asking, and it seems so far that there are enough > of those that do care about small arrays to spend the effort to > significantly improve the performance. Well, here's my application. I do data mining work, and one of the techniques I want to use Numpy for is to implement robust regression algorithms like least-trimmed-squares. Now for a k-variable regression, the best-of-breed algorithm for this involves taking hundreds of thousands of k-element samples and calculating the fitting hyperplane through them. Small matrix performance is thus something this program lives or dies by, and right now it seems like 'dies' is the right measure -- it is about 10x slower than the Gauss program that does the same thing. :( When I profiled it seems like Numpy is spending almost all of its time in _castCopyAndTranspose. Switching to the Intel MKL LAPACK had no performance effect, but changing _castCopyAndTranspose into a C function was a 20% speed increase. If Numpy2 is even slower on small matrices I'd have to give up using it, and that's a shame: it's a *much* nicer environment than Gauss is. -- Neel Krishnaswami neelk(a)cswcasa.com

1 0

Numpy2 and GSL
by Martin Wiechert 27 Nov '01

27 Nov '01

Hi! Just an uneducated question. Are there any plans to wrap GSL for Numpy2? I did not actually try it (It's not Python ;-)), but it looks clean and powerful. Regards, Martin.

5 4

Re: [Numpy-discussion] Meta: too many numerical libraries doing the same thing?
by Konrad Hinsen 26 Nov '01

26 Nov '01

"Christos Siopis <siopis(a)umich.edu>" <siopis(a)umich.edu> writes: > Don't take me wrong, i am not saying that Numpy, PDL, GSL & co. should be > somehow "merged" --obviously, one needs different wrappers to call > numerical routines from Python, Perl, C, C++ or Java. But there should be > a way so that the actual *implementation* of the numerical algorithms is > only done once and for all. I agree that sounds nice in theory. But even if it were technically feasible (which I doubt) given the language differences, it would be a development project that is simply too big for scientists to handle as a side job, even if they were willing (which again I doubt). My impression is that the organizational aspects of software development are often neglected. Some people are good programmers but can't work well in teams. Others can work in teams, but are not good coordinators. A big project requires at least one, if not several, people who are good scientist and programmers, have coordinator skills, and a job description that permits them to take up the task. Plus a larger number of people who are good scientists and programmers and can work in teams. Finally, all of these have to agree on languages, design principles, etc. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------

3 4

Re: [Numpy-discussion] Meta: too many numerical libraries doing the same thing?
by Chris Barker 26 Nov '01

26 Nov '01

"Paul F. Dubois" wrote: > We had some meetings to discuss using blitz and the truth is that as > wrapped by Python there is not much to gain. The efficiency of blitz > comes up when you do an array expression in C++. Then x = y + z + w + a > + b gets compiled into one loop with no temporary objects created. But > this trick is possible because you can bind the assignment. In python > you cannot bind the assignment so you cannot do a lazy evaluation of the > operations, unless you are willing to go with some sort of function call > like x = evaluate(y + z + w). Immediate evaluations means creating > temporaries, and performance is dead. > > The only gain then would be when you passed a Python-wrapped blitz array > back to C++ and did a bunch of operations there. Personally, I think this could be a big gain. At the moment, if you don't get the performance you need with NumPy, you have to write some of your code in C, and using the Numeric and Python C API is a whole lot of work, particularly if you want your function to work on non-contiguous arrays and/or arrays of any type. I don't know much C++, and I have no idea if Blitz++ fits this bill, but it seemed to me that using an object oriented framework that could take care of reference counting, and allow you to work with generic arrays, and index them naturally, etc, would be a great improvement, even if the performance was the same as the current C API. Perhaps NumPy2 has accomplished that, it sounds like it is a step in the right direction, at least. In a sentence: the most important reason for using a C++ object oriented multi-dimensional array package would be easy of use, not speed. It's nice to hear Blitz++ was considered, it was proably rejected for good reason, but it just looked very promising to me. -Chris -- Christopher Barker, Ph.D. ChrisHBarker(a)home.net --- --- --- http://members.home.net/barkerlohmann ---@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Oil Spill Modeling ------ @ ------ @ ------ @ Water Resources Engineering ------- --------- -------- Coastal and Fluvial Hydrodynamics -------------------------------------- ------------------------------------------------------------------------

2 2