Mailman 3 January 2005 - NumPy-Discussion

[Fwd: Re: [Numpy-discussion] Speeding up Numeric]
by Travis Oliphant 22 Jan '05

22 Jan '05

Original message sent to wrong address. Forwarding to correct address.

2 1

updating Numeric
by Travis Oliphant 22 Jan '05

22 Jan '05

I would like to try to extensively update Numeric. Because of the changes, I would like to call it Numeric3.0 The goal is to create something that is an easier link between current Numeric and future numarray. It will be largely based on the current Numeric code base (same overall structure). I'm looking for feedback and criticism so don't hesitate to tell me what you think (positive or negative). I'll put on my thickest skin :-) Changes: (in priority order) ================ 1) Add support for multidimensional indexing 2) Change coercion rule to "scalars don't count rule" 3) Add support for bool, long long (__int64), unsigned long long, long double, complex long double, and unicode character arrays 4) Move to a new-style c-type (i.e. support for array objects being sub-classable which was added in Python 2.2) 5) Add support for relevant parts of numarray's C-API (to allow code written for numarray that just uses basic homogeneous data to work with Numeric) 6) Add full IEEE support 7) Add warning system much like numarray for reporting errors (eliminate check_array and friends). 8) optimize the ufuncs where-ever possible: I can see a couple of possibilities but would be interested in any help here. 9) other things I'm forgetting.... Why it is not numarray? I think there is a need for the tight code-base of Numeric to continue with incremental improvements that keeps the same concept of an array of homogeneous data types. If sub-classing in c works well, then perhaps someday, numarray could subclass Numeric for an even improved link between the two. I have not given up on the numarray and Numeric merging someday, I just think we need an update to Numeric that moves Numeric forward in directions that numarray has paved without sacrificing the things that Numeric already does well. -Travis O.

3 2

Re: Speeding up Numeric
by Steve Chaplin 22 Jan '05

22 Jan '05

On Fri, 2005-01-21 at 15:38 -0800, David M. Cooke wrote: > > First, profiling Python extensions is not easy. I've played with using Which version of Python are you using? In 2.4 the profile module has been updated so it can now profile C extension functions. Steve

2 1

Re: [Numpy-discussion] position of objects?
by Bob Klimek 21 Jan '05

21 Jan '05

Peter Verveer wrote: > The procedure you show below seems to be based on a normal watershed. > I am not completely sure how the Image-J implementation works, but one > way to do that would be to do a watershed on the distance transform of > that image (actually you would use the negative of the distance > transform, with the local minima of that as the seeds). You could do > that with watershed_ift, in this case it would give you two labeled > objects, that in contrast to your example would however touch each > other. To do the exact same as below a watershed is needed that also > gives watershed lines. Hi Peter, I thought I'd try your suggestion above but I'm falling short. Where I stall is at local minima (or local maxima if you don't invert the image). Currently there is no local minima (or maxima) function in nd_image is there (or am I missing it)? Bob

1 0

Matlab/Numeric/numarray benchmarks
by Simon Burton 20 Jan '05

20 Jan '05

Hi all, Here are some benchmarks, measured in seconds on a 1.5GHz celeron. Each test does a matrix add (1000x1000), mul (1000x1000) and eigenvalue find (500x500). Matlab: 0.0562 1.5180 3.7630 Numeric: 0.0962309813499 1.73247330189 3.72153270245 numarray: 7.17220497131 19.3960719109 5.72376401424 I have attached the code. Looks like numarray is way behind on the basic linear algebra stuff. We have (so far) chosen to go with numarray for our scientific computations, but will be needing fast add/multiply. I am surmising that these methods just have not been pluged into the native BLAS/ATLAS lib. We will also be needing other solvers from LAPACK such as dpotrf/dposv, and some of the special functions (bessel) already implemented in scipy. I understand that Todd is working on scipy/numarray compatability. How is that progressing, and what should we be doing to get the above functionality into numarray ? I have been pokeing around the code already, and am able to help out with that. bye for now, Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com

7 22

'copy' argument in records.array
by Ivan Vilata i Balaguer 20 Jan '05

20 Jan '05

Hi all! I have seen that records.array() has a boolean 'copy' argument which indicates whether copying the 'sequence' object or not when it already is an array. However, the written documentation does not mention it anywhere. Is this an officially supported argument? Thank you, Ivan PS: Would you mind cc:'ing me? I am not subscribed to the list. import disclaimer -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ Cárabos Coop. V. V V Enjoy Data ""

1 0

Re: [Numpy-discussion] Speeding up numarray -- questions on its design
by Perry Greenfield 20 Jan '05

20 Jan '05

Travis Oliphant wrote: > I have some comments based on perusing it's source. I don't want to > seem overly critical, so please take my comments with the understanding > that I appreciate the extensive work that has gone into Numarray. I do > think that Numarray has made some great strides. I would really like > to > see a unification of Numeric and Numarray. > > 1) Are there plans to move the nd array entirely into C? > -- I would like to see the nd array become purely a c-type. I would > be willing to help here. I can see that part of the work has been > done. > I don't know that I would say they are definite, but I think that at some point we thought that would be necessary. We haven't yet since doing so makes it harder to change so it would be one of the last changes to the core that we would want to do. Our current priorities are towards making all the major libraries and packages available under it first and then finishing optimization issues (another issue that has to be tackled soon is handling 64-bit addressing; apparently the work to make Python sequences use 64-bit addresses is nearing completion so we want to be able to handle that. I expect we would want to make sure we find a way of handling that before we turn it all into C but maybe it is just as easy doing them in the opposite order. > 2) Why is the ND array C-structure so large? Why are the dimensions > and strides array static? Why can't the extra stuff that the fancy > arrays need be another structure and the numarray C structure just > extended with a pointer to the extra stuff? When Todd moved NDArray into C, he tried to keep it simple. As such, it has no "moving parts." We think making dimensions and strides malloc'ed rather than static would be fairly easy. Making the "extra stuff" variable is something we can look at. The bottom line is that adding the variability adds complexity and we're not sure we understand the storage economics of why we would doing it. Numarray was designed, first and foremost, for large arrays. For that case, the array struct size is irrelevant whereas additional complexity is not. I guess we would like to see some good practical examples where the array struct size matters. Do you have code with hundreds of thousands of small arrays existing simultaneously? > 3) There seem to be too many files to define the array. The mixture of > Python and C makes trying to understand the source very difficult. I > thought one of the reasons for the re-write was to simplify the source > code. > I think this reflects the transitional nature of going from mostly Python to a hybrid. We agree that the current state is more convoluted than it ought to be. If NDarray were all C, much of this would ended (though in some respects, being all in C will make it larger, harder to understand as well). The original hope was that most of the array setup computation could be kept in Python but that is what made it slow for small arrays (but it did allow us to implement it reasonably quickly with big array performance so that we could start using for our own projects without a long development effort). Unfortunately, the simplification in the rewrite is offset by handling the more complex cases (byte-swapping, etc.) and extra array indexing capabilities. > 4) Object arrays must be supported. This was a bad oversight and an > important feature of Numeric arrays. > The current implementation does support them (though in a different way, and generally not as efficiently, though Todd is more up on the details here). What aspect of object arrays are you finding lacking? C-api? > 5) The ufunc code interface needs to continue to be improved. I do see > that some effort into understanding the old ufunc interface has taken > place which is a good sign. > You are probably referring to work underway to integrate with scipy (I'm assuming you are looking at the version in CVS). > Again, thanks to the work that has been done. I'm really interested to > see if some of these modifications can be done as in my mind it will > help the process of unifying the two camps. > I'm glad to see that you are taking a look at it and welcome the comments and any offers of help in improving speed. Perry

9 19

Re: [Numpy-discussion] position of objects?
by Bob Klimek 19 Jan '05

19 Jan '05

Peter Verveer wrote: > The watershed_ift() is a somewhat uusual implementation of watershed. > In principle it does the same as a normal watershed, except that it > does not produce watershed lines. I implemented this one, because with > the current implementation of binary morphology it is a bit cumbersome > to implement more common approaches. That will hopefully change in the > future. Well, it might turn out to still be useful. From what I'm reading, watershed from markers can do some interesting things. See the library below. > The procedure you show below seems to be based on a normal watershed. > I am not completely sure how the Image-J implementation works, but one > way to do that would be to do a watershed on the distance transform of > that image (actually you would use the negative of the distance > transform, with the local minima of that as the seeds). You could do > that with watershed_ift, in this case it would give you two labeled > objects, that in contrast to your example would however touch each > other. To do the exact same as below a watershed is needed that also > gives watershed lines. I'll give this procedure a try. Even if the labeled objects touch, some code could perhaps separate the objects by changing the touching pixels to 0. > Prompted by your earlier questions about skeletons I had a look at > what it would take to properly implement skeletons and other > morphology based algorithms, such as watersheds, and I found that I > need to rethink and reimplement the basic morphology operations first. ... Well, improving things is always good but from what I can see its not bad right now. if you are going to be changing things, one minor suggestion from me would be to make indices of label() (and sum(), mean(), ...) and find_objects() the same. For example, in an image containg two objects, label() returns a list of three: 0, 1, and 2 where 0 is the background and the two objects are labeled 1 and 2. But find_objects() returns a list of two (indices in the list being 0 and 1). Its not a big deal but in a for-loop it gets a little messy. Also forces me to do things like the following example which requires the loop to start at 1 (to skip the background) and run the range to n+1 to capture the second object. labeled, n = ND.label(binImage) objList = ND.sum(binImage, labeled, range(n+1)) for i in range(1, len(objList)): print 'object %d pixels: %d ' % (i, objList[i]) On a different note, I came across a morphology library which looks very promising. http://www.mmorph.com/pymorph/ I've contacted one of the authors of the package (R. Lotufo) and he indicated that they are thinking about updating it to run under numarray probably in about 6 months. Perhaps you and them could join forces. The only potential problem I see is that their code is designed strictly for 2D grayscale and binary images whereas you are trying to keep it general for any number of dimensions. Regards, Bob

1 0

ScientificPython with numarray support
by konrad.hinsen＠laposte.net 16 Jan '05

16 Jan '05

A development release of Scientific Python that supports numarray as an alternative to NumPy (choice made at installation time) is now available at http://starship.python.net/~hinsen/ScientificPython/ or http://dirac.cnrs-orleans.fr/ScientificPython/ (Search for "2.51".) Note that some modules do not work under numarray because they rely on a NumPy feature that is currently implemented in numarray. They are listed in the README file. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire Léon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: hinsen(a)llb.saclay.cea.fr ---------------------------------------------------------------------

3 4

Speeding up numarray -- questions on its design
by Travis Oliphant 14 Jan '05

14 Jan '05

Hello all, I don't comment much on numarray because I haven't used it that much, as Numeric fits my needs quite well. It does bother me that there are two communities coexisting and that work seems to get repeated several times, so recently I have looked at numarray to see how far it is from being acceptable as a real replacement for Numeric. I have some comments based on perusing it's source. I don't want to seem overly critical, so please take my comments with the understanding that I appreciate the extensive work that has gone into Numarray. I do think that Numarray has made some great strides. I would really like to see a unification of Numeric and Numarray. 1) Are there plans to move the nd array entirely into C? -- I would like to see the nd array become purely a c-type. I would be willing to help here. I can see that part of the work has been done. 2) Why is the ND array C-structure so large? Why are the dimensions and strides array static? Why can't the extra stuff that the fancy arrays need be another structure and the numarray C structure just extended with a pointer to the extra stuff? 3) There seem to be too many files to define the array. The mixture of Python and C makes trying to understand the source very difficult. I thought one of the reasons for the re-write was to simplify the source code. 4) Object arrays must be supported. This was a bad oversight and an important feature of Numeric arrays. 5) The ufunc code interface needs to continue to be improved. I do see that some effort into understanding the old ufunc interface has taken place which is a good sign. Again, thanks to the work that has been done. I'm really interested to see if some of these modifications can be done as in my mind it will help the process of unifying the two camps. -Travis Oliphant

1 0