Mailman 3 January 2004 - NumPy-Discussion

More on transitioning to Numarray
by Travis Oliphant Jan. 22, 2004

Jan. 22, 2004

Today, I realized that I needed to restate what my intention in raising the subject to begin with was. First of all, I would like to see everybody transition to Numarray someday. On the other hand, I'm not willing to ignore performance issues just to reach that desireable goal. I would like to recast my proposal into the framework of helping SciPy transition to Numarray. Basically, I don't think Numarray will be ready to fully support SciPy in less than a year (basically because it probably won't happen until some of us SciPy folks do a bit more work with Numarray). To help that along I am proposing making a few changes to the Numeric object that SciPy uses so that the array object SciPy expects starts looking more and more like the Numarray object. We have wanted to do this in SciPy and were simply wondering if it would make sense to change the Numeric object or to grab the Numeric code base into SciPy and make changes there. The feedback from the community has convinced me personally that we should leave Numeric alone and make any changes to something we create inside of SciPy. There is a lot of concern over having multiple implementations of nd arrays due to potential splitting of tools, etc. But, I should think that tools should be coded to an interface (API, methods, data structures) instead of a signle implementation, so that the actual underlying object should not matter greatly. I thought that was the point of modular development and object-orientedness .... Anyone doing coding with numeric arrays already has to distinguish between: Python Imaging Objects, Lists of lists, and other array-like objects. I think it is pretty clear that Numeric won't be changing. Thus, anything we do with the Numeric object will be done from the framework of SciPy. Best regards. Travis O.

1 0

How fast are small arrays currently?
by Edward C. Jones Jan. 21, 2004

Jan. 21, 2004

Has anyone recently benchmarked the speed of numarray vs. Numeric? Why are numarrays so slow to create?

5 9

re: Status of Numeric (and plotting in particular)
by Jon Peirce Jan. 21, 2004

Jan. 21, 2004

> > >We have started on this over the past month, and hope to have some >simple >functionality available within a month (though when we make it public >may >take a bit longer). It will be open source and we hope significantly >simpler >than chaco. It will not focus on speed (well, we want fairly fast >display times >for plots of a reasonable number of points, but we don't need video >refresh >rates). If your interest in plotting matches ours, then this may be for >you. >We will welcome contributions and comments once we get it off the >ground. >(We are calling it pyxis by the way). > I agree with the sentiment that chaco is a very heavy and confusing package for the average scientist (but maybe great for the full-time programmer) but I'm really concerned about the idea that we need *another* solution started from scratch. There are already so many including scipy.gplt, scipy.plt, dislin, biggles, pychart, piddle, pgplot, pyx (new)... In particular MatPlotLib looks promising - check out its examples: http://matplotlib.sourceforge.net/screenshots.html *Many* plotting types already , simple syntax, a few different backends. And already has something of a following. So is it really not possible for STScI to push its resources into aiding the development of something that's already begun? Would be great if we could develop a single package really well rather than everyone making their own. -- Jon Peirce Nottingham University +44 (0)115 8467176 (tel) +44 (0)115 9515324 (fax) http://www.psychology.nottingham.ac.uk/staff/jwp/

3 2

RE: [Numpy-discussion] Status of Numeric (and plotting in particular)
by Kuzminski, Stefan R Jan. 20, 2004

Jan. 20, 2004

I'm working on a commercial product that produces publication quality plots from data contained in Numeric arrays. I also concluded that Chaco was a bit more involved than I needed. My question is what requirements are not met by the other available plotting packages such as.. http://matplotlib.sourceforge.net/ These don't have every bell and whistle ( esp. when it comes to the interactive 'properties' dialog ) but as you point out there is a dark side to those features. There are a number of quite capable plotting packages for Python, diversity is good up to a point, but this space ( Plotting packages ) seems ripe for a shakeout. Stefan -----Original Message----- From: numpy-discussion-admin(a)lists.sourceforge.net [mailto:numpy-discussion-admin@lists.sourceforge.net] On Behalf Of Perry Greenfield Sent: Tuesday, January 20, 2004 7:02 PM To: Chris Barker Cc: numpy-discussion(a)lists.sourceforge.net Subject: Re: [Numpy-discussion] Status of Numeric (and plotting in particular) On Tuesday, January 20, 2004, at 06:08 PM, Chris Barker wrote: >> Perry Greenfield writes: > >> > It has been our intention to port scipy to use numarray soon. This >> > work has been delayed somewhat since our current focus is on >> > plotting. > > That is good news. What plotting package are you working on? Last I > heard Chaco had turned into Enthought's (and STSci) in-house Windows > only package. (Not because they want it that way, but because they > don't have funding to make it work on other platforms, and support the > broader community). > > I don't see anything new on the SciPy page after August '03. > > Frankly, weak plotting is a bigger deal to me than array performance. > Yes, I agree completely (and why we are giving plotting higher priority than scipy integration). I really was hoping to raise this issue later, but I might as well address it since the Numeric/numarray issue has raised it indirectly. Chaco had been the focus of our plotting efforts for more than a year. The effort started with our funding Enthought to start the effort. We had a number of requirements for a plotting package that weren't met by any existing package, and it didn't appear that any would be easily modified to our needs. The requirements we had (off the top of my head) included: 1) easy portability to graphics devices *and* different windowing systems. 2) it had to run on all major platforms including Solaris, Linux, Macs, and Windows. 3) the graphics had to be embedable within gui widgets. 4) it had to allow cursor interactions, at least to the point of being able to read cursor positions from python. 5) it had to be open source and preferably not gpl (though the latter was probably not a show stopper for us) 6) It also had to be customizable to the point of being able to produce very high quality hardcopy plots suitable for publication. 7) object oriented plotting framework capable of sensible composition. 8) command line interface akin to that available in matlab or IDL to make producing quick interactive plots very, very easy. Developing something that satisfies these is not at all trivial. In the process Enthought has expended much energy developing chaco, kiva and traits (and lately they are working on yet more extensions); easily much more of the effort has come from sources other than STScI. Kiva is the back end that presents a uniform api for different graphics devices. Traits handles many of the user interface issues for plot parameters, and handling the relationships of these parameters between plot components. Chaco is the higher level plotting software that provides the traditional plotting capabilities for 2-d data. Much has been invested in chaco. It is with some regret that we (STScI) have concluded that chaco is not suitable for our needs and that we need to take a different approach (or at least give it a try). I'll take some space to explain why. The short answer is that in the end we think it was too ambitious. We still aim to achieve the goals I listed above. The problem we think is that chaco was also tasked to try to achieve extra goals with regard interactive capabilities that were in the end, not really important to STScI and it's community, but were important to Enthought (and presumably its clients, and the scipy community). More specifically, a lot of thought and work went into making many aspects of the plots could be interactively modified. That is, by clicking on various aspects of plots, one could bring up editors for the attributes of that plot element, such as color, line style, font, size, etc. Many other interactive aspects have been enhanced as well. Much recent work by Enthought is going into extending the capabilities even further by adding gui kinds of features (e.g., widgets of all sorts). Unfortunately these capabilities have come at a price, namely complexity. We have found it difficult to track the ongoing changes to chaco to become proficient enough to contribute significantly by adding capabilities we have needed. Perhaps that argues that we aren't competent to do so. To a certain degree, that is probably is true. There is no doubt that Enthought has some very talented software engineers working on chaco and related products. On the other hand, our goal is to have this software be accessible by scientists in general, and particularly astronomers. Chaco is complex enough that we think that is a serious problem. Customizing it's behavior requires a very large investment of time understanding how it works, far beyond what most astronomers are willing to tackle (at least that's my impression). Much of this complexity (and many of its ongoing changes) is to support the interactive capabilities, and to make it responsive enough that plots can update themselves quickly enough not to lead to annoying lags. But frankly, we just want something to render plots on the screen and on hardcopy. Outside of being able to obtain cursor coordinates, we find many of the interactive capabilities as secondary in importance. When most astronomers want to tune a plot (either for publication quality, or for batch processing), they usually want to be able to reproduce the adjustments for new data, for which the interactive attribute editing capability is of little use. Generally they would like to script the the more customized plots so that they can be easily modified and reused. So it seems that it is too difficult to accomplish all these aims within one package. We would like to develop a different plotting package (using many of ideas from chaco, and some code) based on kiva and the traits package. We have started on this over the past month, and hope to have some simple functionality available within a month (though when we make it public may take a bit longer). It will be open source and we hope significantly simpler than chaco. It will not focus on speed (well, we want fairly fast display times for plots of a reasonable number of points, but we don't need video refresh rates). If your interest in plotting matches ours, then this may be for you. We will welcome contributions and comments once we get it off the ground. (We are calling it pyxis by the way). Enthought is continuing to work on chaco and at some point that will be mature, and will be capable of some sophisticated things. That may be more appropriate for some than what we are working on. Perry Greenfield ------------------------------------------------------- The SF.Net email is sponsored by EclipseCon 2004 Premiere Conference on Open Tools Development and Integration See the breadth of Eclipse activity. February 3-5 in Anaheim, CA. http://www.eclipsecon.org/osdn _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

1 0

RE: [Numpy-discussion] Status of Numeric
by Marcel Oliver Jan. 20, 2004

Jan. 20, 2004

Perry Greenfield writes: > Peter J. Verveer writes: > > > I was under the impression that Numarray was intended to be a > > replacement for Numeric, also as a building block for larger > > packages such as SciPy. Was Numarray not intended to be an > > "improved Numeric" in the first place? I chose to develop for > > Numarray rather than Numeric because of its improvements, under > > the assumption that eventually my code would also become > > available to the users of such packages as SciPy. (I wrote the > > nd_image extension that is now distributed with Numarray. I also > > contributed some improvements to RandomArray extension that are > > not in the Numeric version.) > > > It has been our intention to port scipy to use numarray soon. This > work has been delayed somewhat since our current focus is on > plotting. We do still intend to see that scipy works with numarray. That this discussion is happening NOW really surprises me. I have been following this list for a couple of years now, with the intention of eventually using numerical Python as the main teaching toolbox for numerical analysis, and possibly for the migration small research codes as well. The possibility of doing numerics in Phython has always intrigued me. Right now I am primarily using Matlab. It's very powerful, but not free and the language is horrible; Octave is trying to play catch up but has mostly lost steam. So a good scientific Phython environment (of any sort) would be a really cool thing to have. However, two things have always held me back (apart from coding small examples on a few occasions): 1. Numerical Phython has been in a limbo for too long (I had even assumed a few times that both Numeric and Numarray were dead for all practical purposes). If there are two incompatible version for years and no clear indication where the whole thing is going, I am very hesitant to invest any time into writing substantial code, or recommend it for class room use. 2. Plotting is a major issue. There are a couple of semi-functional packages, but neither a comprehensive solution nor a clear direction for the plotting architecture. Short, I see a lot of potential, unused mainly because the numerical Python community seems to lack clear direction and leadership. This is a real showstopper for someone who is primarily interested in building on top. I am still hopeful that something will come of all this - any progress will be very much appreciated. Best regards, Marcel --------------------------------------------------------------------- Marcel Oliver Phone: +49-421-200-3212 School of Engineering and Science Fax: +49-421-200-3103 International University Bremen m.oliver(a)iu-bremen.de Campus Ring 1 oliver(a)member.ams.org 28759 Bremen, Germany http://math.iu-bremen.de/oliver ---------------------------------------------------------------------

3 3

Status of Numeric
by Travis Oliphant Jan. 20, 2004

Jan. 20, 2004

Numarray is making great progress and is quite usable for many purposes. An idea that was championed by some is that the Numeric code base would stay static and be replaced entirely by Numarray. However, Numeric is currently used in a large installed base. In particular SciPy uses Numeric as its core array. While no doubt numarray arrays will be supported in the future, the speed of the less bulky Numeric arrays and the typical case that we encounter in SciPy of many, small arrays will make it difficult for people to abandon Numeric entirely with it's comparatively light-weight arrays. In the development of SciPy we have encountered issues in Numeric that we feel need to be fixed. As this has become an important path to success of several projects (both commercial and open) it is absolutely necessary that this issues be addressed. The purpose of this email is to assess the attitude of the community regarding how these changes to Numeric should be accomplished. These are the two options we can see: * freeze old Numeric 23.x and make all changes to Numeric 24.x still keeping Numeric separate from SciPy * freeze old Numeric 23.x and subsume Numeric into SciPy essentially creating a new SciPy arrayobject that is fast and lightweight. Anybody wanting this new array object would get it by installing scipy_base. Numeric would never change in the future but the array in scipy_base would. It is not an option to wait for numarray to get fast enough as these issues need to be addressed now. Ultimately I think it will be a wise thing to have two implementations of arrays: one that is fast and lightweight optimized for many relatively small arrays, and another that is optimized for large-scale arrays. Eventually, the use of these two underlying implementations should be automatic and invisible to the user. A few of the particular changes we need to make to the Numeric arrayobject are: 1) change the coercion model to reflect Numarray's choice and eliminate the savespace crutch. 2) Add indexing capability to Numeric arrays (similar to Numarray's) 3) Improve the interaction between Numeric arrays and scalars. 4) Optimization: Again, these changes are going to be made to some form of the Numeric arrays. What I am really interested in knowing is the attitude of the community towards keeping Numeric around. If most of the community wants to see Numeric go away then we will be forced to bring the Numeric array under the SciPy code-base and own it there. Your feedback is welcome and appreciated. Sincerely, Travis Oliphant and other SciPy developers

7 7

RE: [Numpy-discussion] Status of Numeric
by Ray Schumacher Jan. 20, 2004

Jan. 20, 2004

Hi Marcel, At 12:07 AM 1/21/2004 +0100, you wrote: ><snip> >Are you saying you have found that you have reinvented the wheel? >That's exactly what I suspect happening a lot... I'm sure a lot of people have written little plot utilities because of the size of Chaco and similar packages, or difficulty integrating with their favorite GUI or module. Ray

1 0

Re: [Numpy-discussion] Status of Numeric
by verveer＠embl.de Jan. 20, 2004

Jan. 20, 2004

Just my 2 cents on the issue of replacing Numeric by Numarray: I was under the impression that Numarray was intended to be a replacement for Numeric, also as a building block for larger packages such as SciPy. Was Numarray not intended to be an "improved Numeric" in the first place? I chose to develop for Numarray rather than Numeric because of its improvements, under the assumption that eventually my code would also become available to the users of such packages as SciPy. (I wrote the nd_image extension that is now distributed with Numarray. I also contributed some improvements to RandomArray extension that are not in the Numeric version.) I believe that it would be a bad situation if the numerical python community would be split among two different array packages. (I think Paul Dubois expressed a similar sentiment on comp.lang.python). Supporting code for two incompatible packages would be a pain (I am personally not willing to do that). Not being able to use modules designed for one package in the other would be disappointing for many people, I think... If I understood well, the only issue with Numarray seems to be that the speed for handling small arrays is too low. So would it not be more efficient to focus on that problem rather than throwing away all the excellent work that has been done already on Numarray? Best regards, Peter -- Dr. Peter J. Verveer Cell Biology and Cell Biophysics Programme European Molecular Biology Laboratory Meyerhofstrasse 1 D-69117 Heidelberg Germany

2 1

numarray 0.8 and MacOSX
by Francesc Alted Jan. 20, 2004

Jan. 20, 2004

Hi, I'm trying to compile numarray 0.8 on a MacOSX (Darwin 6.8). The compilation process seemed to go well, but an error happens when trying to import numarray: [falted@ppc-osx2:numarray-0.8]$ python Python 2.2 (#1, 10/24/02, 16:10:52) [GCC Apple cpp-precomp 6.14] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numarray Traceback (most recent call last): File "<stdin>", line 1, in ? File "/home/users/f/fa/falted/bin-macosx//lib/python/numarray/__init__.py", line 11, in ? from numarrayall import * File "/home/users/f/fa/falted/bin-macosx//lib/python/numarray/numarrayall.py", line 2, in ? from generic import * File "/home/users/f/fa/falted/bin-macosx//lib/python/numarray/generic.py", line 1030, in ? import numarraycore as _nc File "/home/users/f/fa/falted/bin-macosx//lib/python/numarray/numarraycore.py", line 29, in ? PyINT_TYPES = { NameError: name 'bool' is not defined I know that there are available ports of numarray 0.8 to Darwin, but, for a series of reasons, I prefer to compile it for myself. Anyone can provide a hint so as to compile it cleanly? Thanks, -- Francesc Alted

2 1

Re: Numpy-discussion digest, Vol 1 #808 - 2 msgs
by Paul Magwene Jan. 19, 2004

Jan. 19, 2004

> > --__--__-- > > Message: 1 > Date: Mon, 19 Jan 2004 14:32:51 -0600 > From: Travis Oliphant <oliphant(a)ee.byu.edu> > To: numpy-discussion(a)lists.sourceforge.net, python-list(a)python.org > Subject: [Numpy-discussion] Status of Numeric > > > The purpose of this email is to assess the attitude of the community > regarding how these changes to Numeric should be accomplished. > > These are the two options we can see: > * freeze old Numeric 23.x and make all changes to Numeric 24.x still > keeping Numeric separate from SciPy > * freeze old Numeric 23.x and subsume Numeric into SciPy essentially > creating a new SciPy arrayobject that is fast and lightweight. Anybody > wanting this new array object would get it by installing scipy_base. > Numeric would never change in the future but the array in scipy_base would. My preference would be for option #1 -- continue further development of Numeric as a separate package with new improvements going into the 24.x series. It's my experience that when projects get subsumed, additional requirements tend to creep in, even if they're not actually "required." --Paul Magwene

1 0