Mailman 3 November 2001 - NumPy-Discussion

Re: Re-implementation of Python Numerical arrays (Numeric) available for download
by Perry Greenfield Nov. 26, 2001

Nov. 26, 2001

> From: Chris Barker <chrishbarker(a)home.net> > To: Perry Greenfield <perry(a)stsci.edu>, > numpy-discussion(a)lists.sourceforge.net > Subject: [Numpy-discussion] Re: Re-implementation of Python > Numerical arrays (Numeric) available > for download > > I used Poor wording. When I wrote "datatypes", I meant data types in a > much higher order sense. Perhaps structures or classes would be a better > term. What I mean is that is should be easy to use an manipulate the > same multidimensional arrays from both Python and C/C++. In the current > Numeric, most folks generate a contiguous array, and then just use the > array->data pointer to get what is essentially a C array. That's fine if > you are using it in a traditional C way, with fixed dimension, one > datatype, etc. What I'm imagining is having an object in C or C++ that > could be easily used as a multidimentional array. I'm thinking C++ would > probably neccesary, and probably templates as well, which is why blitz++ > looked promising. Of course, blitz++ only compiles with a few up-to-date > compilers, so you'd never get it into the standard library that way! > Yes, that was an important issue (C++ and the Python Standard Library). And yes, it is not terribly convenient to access multi-dimensional arrays in C (of varying sizes). We don't solve that problem in the way a C++ library could. But I suppose that some might say that C++ libraries may introduce their own, new problems. But coming up with the one solution to all scientific computing appears well beyond our grasp at the moment. If someone does see that solution, let us know! > I agree, but from the newsgroup, it is clear that a lot of folks are > very reluctant to use something that is not part of the standard > library. > We agree that getting into the standard library is important. > > > > We estimate > > > > that numarray is probably another order of magnitude worse, > > > > i.e., that 20K element arrays are at half the asymptotic > > > > speed. How much should this be improved? > > > > > > A lot. I use arrays smaller than that most of the time! > > > > > What is good enough? As fast as current Numeric? > > As fast as current Numeric would be "good enough" for me. It would be a > shame to go backwards in performance! > > > (IDL does much > > better than that for example). > > My personal benchmark is MATLAB, which I imagine is similar to IDL in > performance. > We'll see if we can match current performance (or at least present usable alternative approaches that are faster). > > 10 element arrays will never be > > close to C speed in any array based language embedded in an > > interpreted environment. > > Well, sure, I'm not expecting that > Good :-) > > 100, maybe, but will be very hard. > > 1000 should be possible with some work. > > I suppose MATLAB has it easier, as all arrays are doubles, and, (untill > recently anyway), all variable where arrays, and all arrays were 2-d. > NumPy is a lot more flexible that that. Is is the type and size checking > that takes the time? > Probably, but we haven't started serious benchmarking yet so I wouldn't put much stock in what I say now. > One of the things I do a lot with are coordinates of points and > polygons. Sets if points I can handle easily as an NX2 array, but > polygons don't work so well, as each polgon has a different number of > points, so I use a list of arrays, which I have to loop over. Each > polygon can have from about 10 to thousands of points (mostly 10-20, > however). One way I have dealt with this is to store a polygon set as a > large array of all the points, and another array with the indexes of the > start and end of each polygon. That way I can transform the coordinates > of all the polygons in one operation. It works OK, but sometimes it is > more useful to have them in a sequence. > This is a good example of an ensemble of variable sized arrays. > > As mentioned, > > we tend to deal with large data sets and so I don't think we have > > a lot of such examples ourselves. > > I know large datasets were one of your driving factors, but I really > don't want to make performance on smaller datasets secondary. > > -- > Christopher Barker, That's why we are asking, and it seems so far that there are enough of those that do care about small arrays to spend the effort to significantly improve the performance. Perry

1 0

Re: [Numpy-discussion] Meta: too many numerical libraries doing the same thing?
by Konrad Hinsen Nov. 26, 2001

Nov. 26, 2001

> We had some meetings to discuss using blitz and the truth is that as > wrapped by Python there is not much to gain. The efficiency of blitz > comes up when you do an array expression in C++. Then x = y + z + w + a > + b gets compiled into one loop with no temporary objects created. But That could still be of interest to extension module writers. And it seems conceivable to write some limited Python-C compiler for numerical expressions that generates extension modules, although this is more than a weekend project. Still, I agree that what most people care about is the speed of NumPy operations. Some lazy evaluation scheme might be more promising to eliminate the creation of intermediate objects, but that isn't exactly trivial either... Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------

1 0

Augmented matrix
by Nils Wagner Nov. 26, 2001

Nov. 26, 2001

Hi, How can I build an augmented matrix [A,b] in Numpy, where A is a m * n matrix (m>n) and b is a m*1 vector Nils

2 1

Sort , Complex array
by Nils Wagner Nov. 26, 2001

Nov. 26, 2001

Hi, How can I sort an array of complex eigenvalues with respect to the imaginary part (in ascending order) in Numpy ? All eigenvalues appear in complex cunjugate pairs. Nils

3 2

Re-implementation of Python Numerical arrays (Numeric) available for download
by Perry Greenfield Nov. 25, 2001

Nov. 25, 2001

We have been working on a reimplementation of Numeric, the numeric array manipulation extension module for Python. The reimplementation is virtually a complete rewrite and because it is not completely backwards compatible with Numeric, we have dubbed it numarray to prevent confusion. While we think this version is not yet mature enough for most to use in everyday projects, we are interested in feedback on the user interface and the open issues (see the documents on the web page shown below). We also welcome those who would like to contribute to this effort by helping with the development or adding libraries. An early beta version is available on sourceforge as the package Numarray (http://sourceforge.net/projects/numpy/) Information on the goals, changes in user interface, open issues, and design can be found at http://aten.stsci.edu/numarray

7 6

Re: Meta: too many numerical libraries doing the same thing?
by Joe Harrington Nov. 24, 2001

Nov. 24, 2001

Yes, this issue has been raised here before. It was the main conclusion of Paul Barrett's and my BOF session at ADASS a 5 years ago (see our report at http://oobleck.astro.cornell.edu/jh/ast/papers/idae96.ps). The main problems are that we scientists are too individualistic to get organized around a single library, too pushed by job pressures to commit much concentrated time to it ourselves, and too poor to pay the architects, coders, doc writers, testers, etc. to write it for us. Socially, we *want* to reinvent the wheel, because we want to be riding on our own wheels. Once we are riding reasonably well for our own needs, our interest and commitment vanishes. We're off to write the next paper. Following that conference, I took a poll on this list looking for help to implement the library. About half a dozen people responded that they could put in up to 10 hours a week, which in my experience isn't enough, once things get hard and attrition sets in. Nonetheless, Paul and I proposed to the NASA Astrophysics Data Analysis Program to hire some people to write it, but we were turned down. We proposed the idea to the head of the High Energy Astrophysics group at NASA Goddard, and he agreed -- as long as what we were really doing was writing software for his group's special needs. The frustrating thing is how many hundreds of astronomy projects hire people to do their 10% of this problem, and how unwilling they are to pool resources to do the general problem. A few of the volunteers in my query to this list have gone on to do SciPy, to their credit, but I don't see them moving in the direction we outlined. Still, they have the capacity to do it right in Python and compiled code written explicitly for Python. They won't solve the general problem, but they may solve the first problem, namely getting a data analysis environment that is OSS and as good as IDL et al. in terms of end-to-end functionality, completeness, and documentation. I like the notion that the present list is for designing and building the underlying language capabilities into Python, and for getting them standardized, tested, and included in the main Python distribution. It is also a good place for debating the merits of different implementations of particular functionality. That leaves the job of building coherent end-user data analysis packages (which necessarily have to pick one routine to be called "fft", one device-independent graphics subsystem, etc.) to application groups like SciPy. There can be more than one of these, if that's necessary, but they should all use the same underlying numerical language capability. I hope that the application groups from several array-based OSS languages will someday get together and collaborate on an ueberlibrary of numerical and graphics routines (the latter being the real sticking point) that are easily wrapped by most languages. That seems backwards, but I think the social reality is that that's the way it is going to be, if it ever happens at all. --jh--

2 1

Numeric & changes in Python division
by Edward C. Jones Nov. 22, 2001

Nov. 22, 2001

# Python 2.2b1, Numeric 20.2.0 from __future__ import division import Numeric arr = Numeric.ones((2,2), 'f') arr = arr/2.0 #Traceback (most recent call last): # File "bug.py", line 6, in ? #arr = arr/2.0 #TypeError: unsupported operand type(s) for /

2 1

Numpy for FORTRAN users
by Nils Wagner Nov. 22, 2001

Nov. 22, 2001

Hi, Currently users must be aware of the fact that multi-dimensional arrays are stored differently in Python and Fortran. Is there any progress that users do not need to worry about this rather confusing and technical detail ? Nils

1 0

Re: Re-implementation of Python Numerical arrays (Numeric) available for download
by Chris Barker Nov. 20, 2001

Nov. 20, 2001

Perry Greenfield wrote: > > One major comment that isn't directly addressed on the web page is the > > ease of writing new functions, I suppose Ufuncs, although I don't > > usually care if they work on anything other than Arrays. I hope the new > > system will make it easier to write new ones. <snip> > Absolutely. We will provide examples of how to write new ufuncs. It should > be very simple in one sense (requiring few lines of code) if our code > generator machinery is used (but context is important here so this > is why examples or a template is extremely important). But it isn't > particularly hard to do without the code generator. And such ufuncs > will handle *all* the generality of arrays including slices, non-aligned > arrays, byteswapped arrays, and type conversion. I'd like to provide > examples of writing ufuncs within a few weeks (along with examples > of other kinds of functions using the C-API as well). This sounds great! The code generting machinery sound very promising, and examples are, of course, key. I found digging through the NumPy source to figure out how to do things very treacherous. Making writing Ufuncs easy will enocourage a lot more C Ufuncs to be written which should help perfomance. > > Also, I can't help wondering if this could leverage more existing code. > > The blitz++ package being used by Eric Jones in the SciPy.compiler > > project looks very promising. It's probably too late, but I'm wondering > > what the reasons are for re-inventing such a general purpose wheel. > > > I'm not sure which "wheel" you are talking about :-) The wheel I'm talking about are multi-dimensional array objects... > We certainly > aren't trying to replilcate what Eric Jones has done with the > SciPy.compiler approach (which is very interesting in its own right). I know, I just think using an existing set of C++ classes for multiple typed multidimansional arrays would make sense, although I imagine it is too late now! > If the issue is why we are redoing Numeric: Actually, I think I had a pretty good idea why you were working on this. > 1) it has to be rewritten to be acceptable to Guido before it can be > part of the Standard Library. > 2) to add new types (e.g. unsigned) and representations (e.g., non-aligned, > byteswapped, odd strides, etc). Using memory mapped data requires some > of these. > 3) to make it more memory efficient with large arrays. > 4) to make it more generally extensible I'm particualry excited about 1) and 4) > > As a whole I have found that I would like the transition from Python to > > Compiled laguages to be smoother. The standard answer to Python > > perfomance is to profile, and then re-write the computationally intesive > > pertions in C. This would be a whole lot easier if Python used datatypes > > that are easy to use from C/C++ as well as Python. I hope NumPy2 can > > move in this direction. > > > What do you see as missing in numarray in that sense? Aside from UInt32 > I'm not aware of any missing type that is available on all platforms. > There is the issue of Float128 and such. Adding these is not hard. > The real issue is how to deal with the platforms that don't support them. I used Poor wording. When I wrote "datatypes", I meant data types in a much higher order sense. Perhaps structures or classes would be a better term. What I mean is that is should be easy to use an manipulate the same multidimensional arrays from both Python and C/C++. In the current Numeric, most folks generate a contiguous array, and then just use the array->data pointer to get what is essentially a C array. That's fine if you are using it in a traditional C way, with fixed dimension, one datatype, etc. What I'm imagining is having an object in C or C++ that could be easily used as a multidimentional array. I'm thinking C++ would probably neccesary, and probably templates as well, which is why blitz++ looked promising. Of course, blitz++ only compiles with a few up-to-date compilers, so you'd never get it into the standard library that way! This could also lead the way to being able to compile NumPy code....<end fantasy> > I think it is pretty easy to install since it use distutils. I agree, but from the newsgroup, it is clear that a lot of folks are very reluctant to use something that is not part of the standard library. > > > We estimate > > > that numarray is probably another order of magnitude worse, > > > i.e., that 20K element arrays are at half the asymptotic > > > speed. How much should this be improved? > > > > A lot. I use arrays smaller than that most of the time! > > > What is good enough? As fast as current Numeric? As fast as current Numeric would be "good enough" for me. It would be a shame to go backwards in performance! > (IDL does much > better than that for example). My personal benchmark is MATLAB, which I imagine is similar to IDL in performance. > 10 element arrays will never be > close to C speed in any array based language embedded in an > interpreted environment. Well, sure, I'm not expecting that > 100, maybe, but will be very hard. > 1000 should be possible with some work. I suppose MATLAB has it easier, as all arrays are doubles, and, (untill recently anyway), all variable where arrays, and all arrays were 2-d. NumPy is a lot more flexible that that. Is is the type and size checking that takes the time? > Another approach is to try to cast many of the functions as being > able to broadcast over repeated small arrays. After all, if one > is only doing a computation on one small array, it seems unlikely > that the overhead of Python will be objectionable. Only if you > have many such arrays to repeat calculations on, should it be > a problem (or am I wrong about that). You are probably right about that. > If these repeated calculations > can be "assembled" into a higher dimensionality array (which > I understand isn't always possible) and operated on in that sense, > the efficiency issue can be dealt with. I do that when possible, but it's not always possible. > But I guess this can only > be seen with specific existing examples and programs. I would > be interested in seeing the kinds of applications you have now > to gauge what the most effective solution would be. One of the things I do a lot with are coordinates of points and polygons. Sets if points I can handle easily as an NX2 array, but polygons don't work so well, as each polgon has a different number of points, so I use a list of arrays, which I have to loop over. Each polygon can have from about 10 to thousands of points (mostly 10-20, however). One way I have dealt with this is to store a polygon set as a large array of all the points, and another array with the indexes of the start and end of each polygon. That way I can transform the coordinates of all the polygons in one operation. It works OK, but sometimes it is more useful to have them in a sequence. > As mentioned, > we tend to deal with large data sets and so I don't think we have > a lot of such examples ourselves. I know large datasets were one of your driving factors, but I really don't want to make performance on smaller datasets secondary. I hope I'll get a chance to play with it soon.... -Chris -- Christopher Barker, Ph.D. ChrisHBarker(a)home.net --- --- --- http://members.home.net/barkerlohmann ---@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Oil Spill Modeling ------ @ ------ @ ------ @ Water Resources Engineering ------- --------- -------- Coastal and Fluvial Hydrodynamics -------------------------------------- ------------------------------------------------------------------------

1 0

Re: Re-implementation of Python Numerical arrays (Numeric) available for download
by Perry Greenfield Nov. 20, 2001

Nov. 20, 2001

> 6) Should array properties be accessible as public attributes > instead of through accessor methods? > > We don't currently allow public array attributes to make > the Python code simpler and faster (otherwise we will > be forced to use __setattr__ and such). This results in > incompatibilty with previous code that uses such attributes. > > > I prefer the use of public attributes over accessor methods. > > > -- > Paul Barrett, PhD Space Telescope Science Institute The issue of efficiency may not be a problem with Python 2.2 or later since it provides new mechanisms that avoid the need to use __setattr__ to solve this problem. (e.g. __slots__, property, __get__, and __set__). So it becomes more of an issue of which style people prefer rather than simplicity and speed of the code. Perry

1 0