Mailman 3 September 2003 - NumPy-Discussion

Migrating Char arrays
by Marc Poinot 01 Oct '03

01 Oct '03

Hi all, I'm migrating my pyCGNS package to numarray. I think that numarray is better suited for the in-memory share of array I'd like to achieve. Now, with Numeric, I use to create arrays with integers, floats, but also char type. In CGNS (http://www.cgns.org) we have types I4, I8, R4 and R8, but also C1 which is an array of chars. It looks like, in numarray, we should consider chars as unsigned ints. This is not false from a system point of view, but from an application point of view I have to make a difference between *real* char arrays, and unsigned int arrays. In other words, when I have a C1 array, I have to translate it as a String, when I have an I4 or even I8, I have to translate it as Integer array. Now I only have integers arrays, and I have to find out a way to know that my array of int is a string. The CGNS->numarray translation is OK, but now I'm loosing information when I'm going back numarray->CGNS. You can say: It's up to your application to understand that this list of ints is a string. I'll reply: So why don't you consider a list of bytes, it's up to the application to know that you're 32 or 64 bits... Is there something in numarray that could replace the Numeric interface for this string type? Is it difficult to add a 'Character' type? Is the (undocumented) CharArrayType the solution for this? Marcvs [alias Not using unicode...] ----------------------------------------------------------------------- Marc POINOT Alias: marcvs Email: poinot(a)onera.fr ONERA -MFE/DSNA/ELSA Tel: 01.46.73.42.84 Info: elsa-info(a)onera.fr 29, Div. Leclerc Fax: 01.46.73.41.66 Site: 92322 Chatillon FRANCE Project: elsA Web: http://www.onera.fr

2 3

numarray unfriendly to user defined types
by Tim Hochberg 30 Sep '03

30 Sep '03

I've run into another issue in my attempt to transition over to numarray: it's less friendly to user defined types than Numeric. I think this is mainly accidental friednliness on Numeric's part, but it's handy nonetheless. The attached file illustrates the issue. Given some object, in this case *zero*, that a numarray array does not now how to handle, it ends up raising an exception instead of giving the object a chance to try. Thus ``zero + numarray.arange(5)`` works while ``numarray.arange(5) + zero`` fails, since in the first case Zero.__add__ is called first, while in the second NumArray.__add__ is called first. This should probably be fixed, but it's not clear what the best way is. My first thought was to always give the other object a chance to use __rop__ first. This is simple and easy to explain, but fails miserably in the common case of multiplying a list and an array. Just to be concrete, __mul__ would be replaced by:: def __mul__(self, operand): try: return operand.__rmul__(self) except: return ufunc.multiply(self, operand) Next thought is to catch exceptions when they occur in numarray and then give the other operand a chance:: def __mul__(self, operand): try: return ufunc.multiply(self, operand) except: return operand.__rmul__(self) This appears like it would fix my particular problem, but still is not ideal. Since numarray is the base of libraries that it will know nothing about, it should defer to classes it doesn't know about whenever possible. Otherewise it's not possible (or maybe just hard) to create new classes that try to be clever, but still interact with numarray in a reasonable way. In my case I'm thinking of proxy objects that don't do some computations till they are actually required. So my current thinking is that __mul__ and friends would be best implemented as:: def __mul__(self, operand): if not isinstance(operand, knownTypes): try: return ufunc.multiply(self, operand) except: pass return operand.__rmul__(self) Where knownTypes is something like (int, long, float, complex, tuple, list). Anyway, that's my excessively long two cents on this. -tim import numarray as na import Numeric as np class Zero: def __add__(self, other): return other __radd__ = __add__ zero = Zero() #~ print zero + np.arange(5) #~ print np.arange(5) + zero #~ print zero + na.arange(5) #~ print na.arange(5) + zero a = na.arange(5) import copy copy.deepcopy(a)

4 18

Re: Cygwin compilation
by Jochen Küpper 30 Sep '03

30 Sep '03

On 29 Sep 2003 17:05:09 -0400 Todd Miller wrote: >> _ufunc.dll needs to be linked against /usr/lib/libmingwex.a. Doing >> that manually seems to work fine. Todd> In the file generate.py, a clause for sys.platform == "cygwin?" Todd> probably needs to be added. Then, define EXTRA_LINK_ARGS for Todd> cygwin. Ok, did that. ,----[cvs diff -u generate.py] | Warning: No xauth data; using fake authentication data for X11 forwarding. | Index: generate.py | =================================================================== | RCS file: /cvsroot/numpy/numarray/generate.py,v | retrieving revision 1.14 | diff -u -u -r1.14 generate.py | --- generate.py 16 Sep 2003 20:02:41 -0000 1.14 | +++ generate.py 30 Sep 2003 08:33:38 -0000 | @@ -51,6 +51,9 @@ | LP64, HAS_UINT64, HAS_FLOAT128 = 0, 1, 0 | elif sys.platform == "win32": | LP64, HAS_UINT64, HAS_FLOAT128 = 0, 0, 0 | +elif sys.platform == "cygwin": | + LP64, HAS_UINT64, HAS_FLOAT128 = 0, 0, 0 | + EXTRA_LINK_ARGS += ["-L/lib/mingw -lmingwex"] | elif sys.platform == "darwin": | LP64, HAS_UINT64, HAS_FLOAT128 = 0, 1, 0 | EXTRA_COMPILE_ARGS.extend(["-Ddarwin"]) `---- Should work, but I get the following strange behavior: ,---- | > python setup.py build | Using EXTRA_COMPILE_ARGS = [] | running build | running build_py | copying Lib/numinclude.py -> build/lib.cygwin-1.5.5-i686-2.3/numarray | running build_ext | building 'numarray._ufunc' extension | gcc -shared -Wl,--enable-auto-image-base -march=pentium3 -mcpu=pentium4 -O2 -Wall -pipe build/temp.cygwin-1.5.5-i686-2.3/Src/_ufuncmodule.o -L/usr/lib/python2.3/config -lpython2.3 -o build/lib.cygwin-1.5.5-i686-2.3/numarray/_ufunc.dll -L/lib/mingw -lmingwex | build/temp.cygwin-1.5.5-i686-2.3/Src/_ufuncmodule.o(.text+0x14f): In function `_CheckFPErrors': | /home/software/programming/numeric/numarray/Src/_ufuncmodule.c:136: undefined reference to `_fetestexcept' | build/temp.cygwin-1.5.5-i686-2.3/Src/_ufuncmodule.o(.text+0x17b):/home/software/programming/numeric/numarray/Src/_ufuncmodule.c:142: undefined reference to `_feclearexcept' | collect2: ld returned 1 exit status | error: command 'gcc' failed with exit status 1 | | > gcc -shared -Wl,--enable-auto-image-base -march=pentium3 -mcpu=pentium4 -O2 -Wall -pipe build/temp.cygwin-1.5.5-i686-2.3/Src/_ufuncmodule.o -L/usr/lib/python2.3/config -lpython2.3 -o build/lib.cygwin-1.5.5-i686-2.3/numarray/_ufunc.dll -L/lib/mingw -lmingwex `---- That is, it does not work when called from setup.py. When I just copy 'n' paste the link statement to the command-line, it just works.(#$%) Any ideas?? >> Is it save to use this mingw library in "normal" Cygwin programs? Todd> I don't know. I didn't get any farther than this, perhaps not as Todd> far. I got the impression that mingw was a minimal library Todd> intended for porting gcc (but it has IEEE stuff), while the Todd> cygwin libc is intended for real programs (but has no IEEE stuff Todd> because it is a lightweight knock off of glibc used to bypass Todd> the GPL). Well, as I see it Cygwin is a Unix emulation layer for Windows, whereas MINGW is a port of gcc and some other GNU tools to Windows (not using Cygwin at all). But looking around using it this way should be ok. Maybe someone who knows cygwin better can speak up here? I would carbon-copy this email to the cygwin list, but that is subscribers only nowadays:( >> Moreover I have the follwoing test error: >> ,---- >> | Failure in example: log10([1+1j]) >> | from line #2267 of numarray.numtest >> | Expected: array([ 0.150515+0.34109409j]) >> | Got: array([ 0.150515+5.21807245j]) >> `---- Todd> One of our guys found a bug in the complex log10 function; both Todd> the expected result and the implementation were changed. Seems to be resolved. Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Liberté, Égalité, Fraternité GnuPG key: CC1B0B4D (Part 3 you find in my messages before fall 2003.)

2 3

Crash with numarray testall.py
by Colin J. Williams 29 Sep '03

29 Sep '03

C:\Program Files\Python23\Lib\site-packages\numarray>python Python 2.3.1 (#47, Sep 23 2003, 23:47:32) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import testall The 21K file created by Windows to contain information about the crash is not readable. Windows XP and numarray 0.7 used. Colin W.

2 1

Blitz++, NumArray...
by Fernando Perez 26 Sep '03

26 Sep '03

Hi all, as some of you know, I'm a big fan of weave.inline() for the ease of development it offers. Essentially C++ becomes a scripting part of python, with only the occasional recompile overhead (but this is transparent to the users). inline() makes heavy use of the blitz++ library, to represent numpy arrays as blitz arrays and therefore index them as Arr(i,j,k...). This seems to me like a very sound decision, as the blitz library was specifically written to provide simultaneously fortran-like performance with high-level object orientation. With this in mind, I have a few questions: - Is there anything in Numarray's design which prevents this same trick from being used? The core of getting a Numpy array into a blitz one is simply a call to: // Convert a Numpy array to a blitz one, using the original's data (no copy) template<class T, int N> static blitz::Array<T,N> py_to_blitz(PyArrayObject *arr_obj) { blitz::TinyVector<int,N> shape(0); blitz::TinyVector<int,N> strides(0); for (int i=0;i<N;i++) { shape[i] = arr_obj->dimensions[i]; strides[i] = arr_obj->strides[i]/sizeof(T); } return blitz::Array<T,N>((T*) arr_obj->data,shape,strides, blitz::neverDeleteData); } I would not be surprised if something much like this would work for Numarrays, but I'd like to know for sure. - If the above is indeed possible for Numarray, do you guys feel that blitz should be considered an integral part of python's low-level arsenal, or not? I view its advantages especially in the field of an extremely clean, simple syntax (compared to the low-level Numpy API, I'm not familiar with Numarray yet). I've seen examples where a numpy algorithm can be recoded using blitz in a _far_ smaller amount of code, and with performance gains to boot (in some cases, not all). Please note that I am NOT second-guessing any of the design decisions made by the Numarray crowd. I just want to clarify if it's possible to have a C++ API which gives access to numarray with the simplest, highest-level syntax and behavior which is reasonable. This is important for teams where not everyone is an expert on C pointer manipulation details, but they may be perfectly ok with writing loops over things which can be indexed as A(i,j,k). - If any of this seems worthy of more detailed discussion, I told Eric Jones that I'd be happy to coordinate something along these lines at Scipy'03. If there's time in the schedule I'd be glad to do it officially, otherwise we can just try to keep it in mind for a lunch/dinner meeting. Documenting a 'Best Practices for low-level numerical work in python' would probably be a beneficial thing to do for the community. Right now there are so many options that I'm sure I'm not the only one to feel a bit confused. - If Blitz++ is indeed deemed a worthy piece of this puzzle, I wonder if we could contact some of its developers to bring them into this discussion. I worry a bit that activity on its mailing lists seems rather slow, but I noticed that an apparently central guy, Julian C. Cummings, is at CACR/Caltech. Perhaps he could stop by and let us know what the future for blitz looks like. I can fire off an email to him if there is interest in the blitz thing. Anyway, I'm just trying to 'pre-soak' a few ideas for discussion at scipy'03 on these topics. Best regards, Fernando.

3 3

numarray incompatibility: searchsorted
by Tim Hochberg 26 Sep '03

26 Sep '03

There are some compatibility and doc issues and perhaps a bug in numarray.searchsorted. The compatibility issue is that Numeric.searchsorted(a, v) accepts either a sequence or scalar value for v. Numarray.searchsorted accepts only sequence values. Second, the doc issue. The docstring for numarray.searchsorted states:: searchsorted(bins, values) searchsort(bins, values) returns the array of greatest indices 'i' such that each values[i] <= bins[i]. I assume that should really read something more like:: searchsorted(bins, values) searchsort(bins, values) returns the array A[j] of greatest indices 'i' such that each values[j] <= bins[i]. Third, the possible bug: # na = numarray, np = NumPy >>> na.searchsorted([1,2,3,4], [2.5, 3.5]) array([1, 2]) >>> np.searchsorted([1,2,3,4], [2.5, 3.5]) array([2, 3]) Hmmm. It appears that numarray result does match the numarray docs, (at least as I read them), but I like the Numeric behaviour better. The Numeric behaviour also matches the behaviour of the bisect module, which is described as:: bisect = bisect_right(a, x, lo=0, hi=None) Return the index where to insert item x in list a, assuming a is sorted. The return value i is such that all e in a[:i] have e <= x, and all e in a[i:] have e > x. So if x already appears in the list, i points just beyond the rightmost x already there. Optional args lo (default 0) and hi (default len(a)) bound the slice of a to be searched. I'd recomend matching the behaviour of the two existing modules (bisect and Numeric). -tim

2 1

Re: [Numpy-discussion] Subclassing numarray arrays
by Nadav Horesh 25 Sep '03

25 Sep '03

On Wed, 2003-09-24 at 14:31, Colin J. Williams wrote: > Nadav Horesh wrote: > > >Recently I had an idea to subclass numarray's array in order to > >implement a different scheme of array indexing via the __call__ method > >(ugly indeed, by my experience shows that it can be useful). It occurred > >to me that the array class is not in the spirit of Python 2.2+ > >classes/types unification: > > > I agree. > > > * The class name is not "trivial": numarraycore.NumArray > > * The class constructor is not built to be used by the commons. > > > I'm not sure what you have in mind here by "trivial" and "commons". commons == End users whose most scripts life-time is few days. trivial == A name one would expect, like "Array" or similar. > > >Since arrays are normally generated by functions (zeros, array, > >fromfunction etc.) and not by the constructor, subclassing > >numarraycore.NumArray is not very useful. If, for instance, "array" (or > >"Array") would be the class name, where the constructor would optionally > >have the same parameters as the array function (with the option to act > >as a copy constructor as the array function does), then it would be easy > >to subclass it: > > > >class MyArray(array): > > ... > > > > > >new_array = MyArray(zeros((10,5), type=Int64)) > > > > > > > This is desirable, with the present design, one has to go through a two > step process: > > new_array = MyArray(type=Int64, shape= (10, 5)) > load_some_data_into_array(new_array, zeros(10, 5)) > > Colin W. As I recall, one of the targets of the numarray project is to be included as a standard python library --- shouldn't it eventually comply with the new-classes standard? > > > Any comments? > > > > Nadav. > > > > > > > >------------------------------------------------------- > >This sf.net email is sponsored by:ThinkGeek > >Welcome to geek heaven. > >http://thinkgeek.com/sf > >_______________________________________________ > >Numpy-discussion mailing list > >Numpy-discussion(a)lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > >

3 4

Re: [Numpy-discussion] Subclassing numarray arrays
by Todd Miller 25 Sep '03

25 Sep '03

On Thu, 2003-09-25 at 12:28, Paul Dubois wrote: > I have not followed this thread in any detail but I continue to be > puzzled by the idea of subclassing numarray arrays. Any subclass worth > its salt has an unpredictable constructor argument list, so that > virtually every ufunc and every array method have to be overridden in > order to return the subclass object rather than a numarray. I think there is a need to effectively subclass numarray where the parameter lists don't change. Doing a Matrix subclass sounds reasonable to me and we should explore the OOP necessary to make it convenient. STSCI hasn't asked for it, but a couple different numarray contributors have. > I gather > that the defers-to-me stuff allows the subclass to get control of a > mixed operation but that was possible with the radd, etc. operators > already as in the old MA. radd does not work for numarray + MA. Instead, the expression is resolved as numarray.__add__(MA). Were MA a numarray subclass, it would work, given the facilities provided by Python alone. The defers-to-me-stuff is a simple work around to avoid having to reverse engineer this one Numeric behavior. > This reasoning led me to believe that subclassing had virtually no value > and to the design of MA that has-a numeric array rather than is-a. Since we want to port MA as-is to numarray, we're not changing that. > Is there some big breakthrough here I missed? No, but there are two issues: (1) effectively supporting subclasses (2) effectively supporting interoperability with other classes. (1) needs more discussion. (2) is already solved. -- Todd Miller jmiller(a)stsci.edu STSCI / ESS / SSB

1 0

Subclassing numarray arrays.
by Nadav Horesh 24 Sep '03

24 Sep '03

Recently I had an idea to subclass numarray's array in order to implement a different scheme of array indexing via the __call__ method (ugly indeed, by my experience shows that it can be useful). It occurred to me that the array class is not in the spirit of Python 2.2+ classes/types unification: * The class name is not "trivial": numarraycore.NumArray * The class constructor is not built to be used by the commons. Since arrays are normally generated by functions (zeros, array, fromfunction etc.) and not by the constructor, subclassing numarraycore.NumArray is not very useful. If, for instance, "array" (or "Array") would be the class name, where the constructor would optionally have the same parameters as the array function (with the option to act as a copy constructor as the array function does), then it would be easy to subclass it: class MyArray(array): ... new_array = MyArray(zeros((10,5), type=Int64)) Any comments? Nadav.

2 1

ANN: Job posting -- developer position available
by eric jones 23 Sep '03

23 Sep '03

Hey group, We have a developer position open here at Enthought, and I thought I'd see if anyone is interested. The posting is below. There is also a tech writer and HCI opening listed on the website: http://www.enthought.com/careers.html thanks! eric Position: Python Developer Type: Full-Time Enthought, Inc. is a technology startup in the Austin, TX area that is looking for a motivated Python Developer to join our development team. Responsibilities include developing and deploying scientific applications within the Fortune 500. Our size and prospects provide significant opportunity for growth and advancement for Enthought employees. Candidates should have the following qualifications: - BS or MS in Computer Science or Engineering or other scientific field - 3+ years experience in the software industry - Experience developing solutions in the following languages: - Python - C/C++ - User interface design experience - Developing and deploying custom solutions for clients - Excellent verbal and written communication skills Ideal candidates will also demonstrate the following skills: - Use of the wxPython user interface library - Experience with Geophysics applications - Experience in the fields of statistics, linear algebra and calculus Please send resumes to jobs(a)enthought.com.

3 3