Mailman 3 April 2005 - NumPy-Discussion

Quite curious behaviour in Numeric
by SIMOND Gilles 07 Apr '05

07 Apr '05

2.6.8-1-686-smp (dilinger(a)toaster.hq.voxel.net) (gcc version 3.3.4 (Debian 1:3.3.4-9)) #1 SMP Sat Aug 28 12:51:43 EDT 2004: and python2.3 >>> a=Numeric.ones((2,3),'i') >>> b=Numeric.sum(a)+1 >>> a[1]=b+1 Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: Array can not be safely cast to required type >>> a.itemsize() 4 >>> b.itemsize() 4 >>> a.typecode() 'i' and e following works >>> a=Numeric.ones((2,3)) >>> b=Numeric.sum(a)+1 >>> a[1]=b+1 >>> a.itemsize() 4 >>> b.itemsize() 4 >>> a.typecode() 'l' >>> type(1) <type 'int'> >>> Numeric.__version__ '23.6' It seems that itemsize() does not return the correct value which should be 8 for 'l' type array. This is quite annoying since this function is the only way to know actual format of the array. Gilles Simond

1 0

metadata and metabehavior for arrays (for scipy.base or Numeric3)
by S�bastien de Menten 07 Apr '05

07 Apr '05

Hi, I follow with great interest the threads around Numeric3/scipy.base. As Travis suggested (It would also help if other people who have concerns would voice them (I'm very grateful for those who have expressed their concerns) so that we can all address them and get on the same page for future development.), I voice my concert J Sometimes it is quite useful to treat data at a higher level than just an array of number of some types. Adding metadata to array (I called them augmented arrays) is a simple way to add sense to an array. I see different user cases like: 1) attaching a physical unit to array data (see for instance Unum http://home.tiscali.be/be052320/Unum.html ) 2) description of axis (see http://sourceforge.net/mailarchive/message.php?msg_id=11051806). Very useful to manipulate easily time series. 3) masked arrays as in MA module of Numeric 4) arrays for interval arithmetic where one keep another array with precision of data 5) record arrays (currently being integrated in scipy.base as a base type) The current solution for those situation is nicely summarized by quoting Konrad but rather a class written using arrays than a variety of the basic array type. Its actually pretty straightforward to implement, the most difficult choice being the form of the constructor that gives most flexibility in use. However, I disagree with the pretty straightforward to implement. In fact, if one wants to inherit most of the functionalities of Numeric, it becomes quite cumbersome. Looking at MA module, I see that it needs to: 1) redefine all methods (__add__, ) 2) redefine all ufuncs 3) redefine all array functions (like reshape, sort, argmax, ) For other purposes, the same burden may apply. A general solution to this problem is not straightforward and may be out of reach (computationally and/or conceptually). However, a quite-general-enough elegant solution could solve most practical problems. Looking at threads in this list, I think that there is enough brain power to get to something usable in the medium term. An embryo of idea would be to add hooks in the machinery to allow an object to interact with an ufunc. Currently, this is done by calling __array__ to extract a naked array (== Numeric.array vs augmented array) but the result is then always a naked array. In pseudocode, this looks like: def ufunc( augmented_array ): if not isarray(augmented_array): augmented_array = augmented_array.__array__() return ufunc.apply(augmented_array) where I would prefer something like def ufunc( augmented_array ): if not isarray(augmented_array): augmented_array, contructor = augmented_array.__array_constructor__() else: constructor = lambda x:x return constructor(ufunc.apply(augmented_array)) For array functions and methods, I have even less clues to a solution J. But calling hooks specified by some protocol would be a path: a) __array_constructor__ b) __array_binary_op__ (would be called for __add__, __sub__, ) c) __array_rbinary_op__ (would be called for __radd__, __rsub__, ) If I miss a point and there is an easy way to do this, Ill be pleased to know it. Otherwise, any feedback on this ability to easily increase array functionalities by appending metadata and related behavior. Sebastien

5 11

RE: [Numpy-discussion] metadata and metabehavior for arrays (for scipy.base or Numeric3)
by Sebastien.deMentendeHorne＠electrabel.com 07 Apr '05

07 Apr '05

> > Why not have the ability to ask the name of an ufunc to be able to > > dispatch on it ? > > That's already possible. > > > What about : > > > > object.__unary__(cos, mode = "reduce") > > object.__binary__(cos, other, mode = "reduce") > > What does "reduce" mode mean for cos? > What does a binary ufunc in reduce mode do with its second argument? raise a ValueError :-) It was an example of a way to pass argument, the focus was on cos.reduce or "cos.reduce" or cos, "reduce". > > However, for binary operations, how it the call dispatched > if one of > > the operand is of a type while the other is another type ? This > > problem is related to multimethods > > http://www.artima.com/weblogs/viewpost.jsp?thread=101605 > > No need to be innovative: Python always dispatches on the first > argument, and everybody is familiar with that approach even though it > isn't perfect. If Python 3000 has multimethods, we can still adapt. The problematic is related to multimethods, the implementation should not be specially related. In an a call like object.__binary__(add, other), if other is not of the same type of object, the latter could throw an exception as ImplementationError to give the hand to other.__binary__(add, binary) or to other.__binary__(radd, binary) or similar (i.e. those expressions may not make sense but the idea is to have a convention to give the hand to the other operand, python does this already when one overloads an operator like __add__ (__radd__)). So if we can keep this same protocol for binary ufunc, that would be great. Otherwise, I think it is not that a big deal. Sebastien ======================================================= This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake please let us know by reply and then delete it from your system; you should not copy it or disclose its contents to anyone. All messages sent to and from Electrabel may be monitored to ensure compliance with internal policies and to protect our business. Emails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. Anyone who communicates with us by email is taken to accept these risks. http://www.electrabel.be/homepage/general/disclaimer_EN.asp =======================================================

1 0

RE: [Numpy-discussion] metadata and metabehavior for arrays (for scipy.base or Numeric3)
by Sebastien.deMentendeHorne＠electrabel.com 07 Apr '05

07 Apr '05

> > On Apr 7, 2005, at 10:06, David M. Cooke wrote: > > > Hmm, I had misread your previous code. Here it is again, made more > > specific, and I'll assume this function lives in the ndarray package > > (as there is more than one package that defines ufuncs) > > At the moment, there is one in Numeric and one in numarray. > The Python > API of both is nearly or fully identical. > > > The thing is obj.__ufunc__ must understand about the *particular* > > object cos: the ndarray one. I was thinking more along the lines of > > No, it must only know the interface. In most cases, it would do > something like > > class MyArray: > def __ufunc__(self, ufunc): > return MyArray(apply(ufunc, self.data)) Exactly ! I see this as a very common use (masked arrays and all the other examples could live with that). Or more precisely (just to be explicity as the previous MyArray example is the simplest (purest) one), class MyArray: def __ufunc__(self, ufunc): metadata= process(self.metadata, ufunc) data = apply(ufunc, self.data) return MyArray(data, metadata) Or variations on this same theme. BTW, looking at Numeric3, the presence of a __mask_array__ in the array protocol looks like we want to add a specific case of "augmented array" to the core protocol. Hmmm, rather prefer to build a more generic mechanism as well as a clean interface for interacting with "augmented array". > > > obj.__ufunc__('cos'), where the name is passed instead. > > That's also an interesting option. It would require the implementing > class to choose an appropriate function from an appropriate module. > Alternatively, it would work if ufuncs were also accessible > as methods > on array objects. > Why not have the ability to ask the name of an ufunc to be able to dispatch on it ? > > For binary ufuncs, you could use (with arguments obj1 and obj2), > > obj1.__ufunc__('add', obj2) > > Except that it would perhaps be better to have a different method, as > otherwise nearly every implementation would have to start with a > condition test to distinguish unary from binary ufuncs. > > > Output argument (obj3): obj1.__ufunc__('add', obj2, obj3) > > Special methods: > > obj1.__ufunc__('add.reduce') > > obj1.__ufunc__('add.accumulate') > > obj1.__ufunc__('add.outer', obj2) > > > > Basically, special methods are just another ufunc. This > suggests that > > add.outer should optionally take an output argument... > > But they are not just another ufunc, because a standard unary ufunc > always returns an array of the same shape as its argument. > > I'd probably prefer a few explicit methods: > > object.__unary__(cos) > object.__binary__(add, other) > object.__binary_reduce__(add) > What about : object.__unary__(cos, mode = "reduce") object.__binary__(cos, other, mode = "reduce") or object.__unary__(cos.reduce) object.__binary__(cos.apply, other) or object.__binary__(cos.__call__, other) with the ability to ask to the first argument its type (with cos.mode or cos.reduce.mode ...) However, for binary operations, how it the call dispatched if one of the operand is of a type while the other is another type ? This problem is related to multimethods http://www.artima.com/weblogs/viewpost.jsp?thread=101605 ======================================================= This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake please let us know by reply and then delete it from your system; you should not copy it or disclose its contents to anyone. All messages sent to and from Electrabel may be monitored to ensure compliance with internal policies and to protect our business. Emails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. Anyone who communicates with us by email is taken to accept these risks. http://www.electrabel.be/homepage/general/disclaimer_EN.asp =======================================================

2 1

New array interface helper file
by Travis Oliphant 07 Apr '05

07 Apr '05

At http://numeric.scipy.org/array_interface.py you will find the start of a set of helper functions for the array interface that can make it more easy to deal with. It also documents the array interface with docstrings. I tried to attach these to properties, but then I don't know how to "see" them from Python. This is the kind of thing I think should go into Python If anybody would like to try their hand at converter functions to go back and forth between the struct module strings and the __array_descr__ string, make my day. -Travis

2 2

Re: [Numpy-discussion] Request for comments on a new setup.py for Numeric
by Travis Oliphant 07 Apr '05

07 Apr '05

David M. Cooke wrote: >Travis Oliphant <oliphant(a)ee.byu.edu> writes: > > > >>David M. Cooke wrote: >> >> >> >>>I've always found the Numeric setup.py to be not very user-friendly. >>>So, I rewrote it. It's available as patch #1178095 >>>http://sf.net/tracker/index.php?func=detail&aid=1178095&group_id=1369&atid=… >>> >>>Basically, all the editing you need to do is in customize.py, instead >>>of touching setup.py. No more commenting out files for lapack_lite >>>(just tell it to use the system LAPACK, and tell it where to find it). >>> >>>Also, you could now use GSL's cblas interface for dotblas. Useful if >>>you've already taken the trouble to link that with an optimized >>>Fortran BLAS. >>> >>>I didn't want to just through this into CVS without feedback first :-) >>>If it looks good, this can go in Numeric 24.0. >>> >>> >>> >>I like the new changes. I also think the setup.py file is unfriendly. >>Put them in... >> >> > >While I'm at it, I'm also thinking of writing a 'cblas_lite' for >dotblas. This would mean that dotblas would be enabled all the time. >You could use a C BLAS if you've got one (from ATLAS, say), or a >Fortran BLAS (like the cxml library on an Alpha running Tru64), or it >would use the existing blas_lite.c if you don't. > > > This is a good idea, but for more than just dotblas. It is the essential problem that must be solved to make scipy.base installable everywhere yet use fast libraries for users who have them without much fuss. -Travis

2 1

Request for comments on a new setup.py for Numeric
by cookedm＠physics.mcmaster.ca 06 Apr '05

06 Apr '05

I've always found the Numeric setup.py to be not very user-friendly. So, I rewrote it. It's available as patch #1178095 http://sf.net/tracker/index.php?func=detail&aid=1178095&group_id=1369&atid=… Basically, all the editing you need to do is in customize.py, instead of touching setup.py. No more commenting out files for lapack_lite (just tell it to use the system LAPACK, and tell it where to find it). Also, you could now use GSL's cblas interface for dotblas. Useful if you've already taken the trouble to link that with an optimized Fortran BLAS. I didn't want to just through this into CVS without feedback first :-) If it looks good, this can go in Numeric 24.0. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm(a)physics.mcmaster.ca

3 3

masked arrays and NaNs
by Eric Firing 06 Apr '05

06 Apr '05

Travis, I am whole-heartedly in favor of your efforts to end the Numeric/numarray split by combining the best of both. I am encouraged by the progress you have made, and by the depth and clarity of the accompanying technical discussions. Thank you! I am a long-time Matlab user in Physical Oceanography, and I have been trying to find a practical way to phase out Matlab. One key is matplotlib, which is coming along wonderfully. A second is the availability of a Num* (or scipy.base) module that provides the functionality and ease-of-use I presently get from Matlab. This leads to a request which I suspect and hope is consistent with your present plans: efficient handling of NaNs and/or masked arrays. In Physical Oceanography, and I suspect in many other fields, data sets are almost always full of holes. Matlab's ability to use NaN as a bad value flag provides a wonderfully simple and efficient way of dealing with missing or bad data values. A similar ease and transparency would be good in scipy.base. In addition, or as a way of implementing NaN-handling internally, it might be best to have masked arrays incorporated at the C level--with the functionality available by default--rather than bolted on as a pure-python package. I hope that inclusion of __array_mask__ in the protocol means that this is part of the plan. Eric

2 1

Re: [Numpy-discussion] Numeric3 - a Windows Problem
by Colin J. Williams 06 Apr '05

06 Apr '05

Travis Oliphant wrote: > Colin J. Williams wrote: > >> C:\Python24\Lib\site-packages\Numeric3\Download>python setup.py install >> running install >> running build >> running config >> error: The .NET Framework SDK needs to be installed before building >> extensions for Python. >> >> Is there any chance that a Windows binary could be made available for >> testing? > > > Probably not in the near term (but you could ask Michiel). > > I'm assuming you have mingw32 installed which would allow you to build > it provided you have created an exports file for python2.4 (look on > the net for how to compile extensions with mingw32 using a MSVC > compiled python). > You have to tell distutils what compiler to use: > > python setup.py config --compiler=mingw32 > python setup.py build --compiler=mingw32 > python setup.py install > > -Travis Thanks to Michiel and Travis for their suggestions. I am using Windows XP and get the following result: C:\Python24\Lib\site-packages\Numeric3\Download>python setup.py config --compiler=minw32 running config error: don't know how to compile C/C++ code on platform 'nt' with 'minw32' compiler C:\Python24\Lib\site-packages\Numeric3\Download> I would welcome any comments. Colin W.

3 3

Numeric 24.0
by Travis Oliphant 06 Apr '05

06 Apr '05

I'd like to release a Numeric 24.0 to get the array interface out there. There are also some other bug fixes in Numeric 24.0 Here is the list so far from Numeric 23.7 [Greenfield] Changed so a[0,0] and a[0][0] returns same type when a is 2-d of Int16 [unreported] Added array interface [unreported] Allow Long Integers to be used in slices [1123145] Handle mu==0.0 appropiately in ranlib/ignpoi. [unreported] Return error info in ranlib instead of printing it to stderr [1151892] dot() would quit python with zero-sized arrays when using dotblas. The BLAS routines *gemv and *gemm need LDA >= 1. [unreported] Fixed empty for Object arrays Version 23.8 March 2005 [Cooke] Fixed more 64-bit issues (patch 117603) [unreported] Changed arrayfnsmodule back to PyArray_INT where the code typecasts to (int *). Changed CanCastSafely to check if sizeof(long) == sizeof(int) I'll wait a little bit to allow last minute bug fixes to go in, but I'd realy like to see this release get out there. For users of Numeric >23.7 try Numeric.empty((10,20),'O') if you want to see an *interesting* bug that is fixed in CVS. -Travis

7 7