Mailman 3 February 2005 - NumPy-Discussion

PEP updated
by Travis Oliphant 20 Feb '05

20 Feb '05

I've updated the PEP to conform to what I think is the best hybrid solution propsed and that is to implement a tree of PythonTypes in C whose leaves are (to the Python user) new rank-0 arrays. This is more work to implement (but I don't think a great deal of work see below), and I think it will lead to the best results under our current constraints. Internally, the PyArray_Type will still deal with rank-0 arrays (in fact the new Python scalars will be converted to them internally rather than special-case a bunch of code). I don't think we should try to solve the general number-hierchy in Python at this time. We will implement the array-number hierarchy only. If this leads to others getting interested in the general problem, then that may come, but now we have to focus on our objective of getting the arrayobject accepted into Python. Note that these new rank-0 Python scalars will be simple, have the same methods and attributes as Python arrays (likely-implemented once in the GenericArrType by converting the object to a rank-0 array). Also, they will inherit (multiple-inheritance is possible in C) from the appropriate Python Type where an exact-match is available. -Travis

3 2

Response to PEP suggestions
by Travis Oliphant 19 Feb '05

19 Feb '05

I'm glad to get the feedback. 1) Types I like Francesc's suggestion that .typecode return a code and .type return a Python class. What is the attitude and opinion regarding the use of attributes or methods for this kind of thing? It always seems to me so arbitrary as to what is an attribute or what is a method. There will definitely be support for the nummary-style type specification. Something like that will be how they print (I like the 'i4', 'f4', specification a bit better though). There will also be support for specification in terms of a c-type. The typecodes will still be there, underneath. One thing has always bothered me though. Why is a double complex type Complex64? and a float complex type Complex32. This seems to break the idea that the number at the end specifies a bit width. Why don't we just call it Complex64 and Complex128? Can we change this? I'm also glad that some recognize the problems with always requiring specification of types in terms of bit-width or byte-widths as these are not the same across platforms. For some types (like Int8 or Int16) this is not a problem. But what about long double? On an intel machine long double is Float96 while on a PowerPC it is Float128. Wouldn't it just be easier to specify LDouble or 'g' then special-case your code? Problems also exist when you are interfacing with hardware or other C or Fortran code. You know you want single-precision floating point. You don't know or care what the bit-width is. I think with the Integer types the bit-width specification is more important than floating point types. In sum, I think it is important to have the ability to specify it both ways. When printing the array, it's probably better if it gives bit-width information. I like the way numarray prints arrays. 2) Multidimensional array indexing. Sometimes it is useful to select out of an array some elements based on it's linear (flattened) index in the array. MATLAB, for example, will allow you to take a three-dimensional array and index it with a single integer based on it's Fortran-order: x(1,1,1), x(2,1,1), ... What I'm proposing would have X[K] essentially equivalent to X.flat[K]. The problem with always requiring the use of X.flat[K] is that X.flat does not work for discontiguous arrays. It could be made to work if X.flat returned some kind of specially-marked array, which would then have to be checked every time indexing occurred for any array. Or, there maybe someway to have X.flat return an "indexable iterator" for X which may be a more Pythonic thing to do anyway. That could solve the problem and solve the discontiguous X.flat problem as well. If we can make X.flat[K] work for discontiguous arrays, then I would be very happy to not special-case the single index array but always treat it as a 1-tuple of integer index arrays. Capping indexes was proposed because of what numarray does. I can only think that the benefit would be that you don't have to check for and raise an error in the middle of an indexing loop or pre-scan the indexes. But, I suppose this is unavoidalbe, anyway. Currently Numeric allows specifying indexes that are too high in slices. It just chops them. Python allows this too, for slices. So, I guess I'm just specifying Python behavior. Of course indexing with an integer that is too large or too small will raise errors: In Python: a = [1,2,3,4,5] a[:20] works a[20] raises an error. 3) Always returning rank-0 arrays. This may be a bit controversial as it is a bit of a change. But, my experience is that quite a bit of extra code is written to check whether or not a calculation returns a Python-scalar (because these don't have the same methods as arrays). In particular len(a) does not work if a is a scalar, but len(b) works if b is a rank-0 array (numeric scalar). Rank-0 arrays are scalars. When Python needs a scalar it will generally ask the object if it can turn itself into an int or a float. A notable exception is indexing in a list (where Python needs an integer and won't ask the object to convert if it can). But int(b) always returns a Python integer if the array has only 1 element. I'd like to know what reasons people can think of for ever returning Python scalars unless explicitly asked for. Thanks for the suggestions. -Travis

20 64

How to subclass NumArray?
by Daehyok Shin 19 Feb '05

19 Feb '05

How can I create a simple subclass inheriting everything from NumArray? -- Daehyok Shin (Peter) Geography Department University of North Carolina-Chapel Hill USA

1 0

rank-0 arrays ideas
by Travis Oliphant 18 Feb '05

18 Feb '05

From the current PEP: Proposed Solution: The solution proposed by this PEP is to fix the places in Python that could use rank-0 arrayobjects to allow for them before raising an exception (this will be in the core after-all). A Python scalar will never be returned unless explicitly requested. I think this is the cleanest, easiest to understand and code for solution. It may require some explicity conversion to int here and there, but it seems like a case of "explicit is better than implicit" Possible compromises: - a compromise could be made for OBJECT arrays and perhaps LONG arrays if needed. - a special flag could be defined which is the default when an array of integers is constructed and which when set rank-0 array returns behave differently. It is also proposed that slicing, indexing and len() do not work (i.e. they raise an error) for rank-0 arrays. Comments: -Travis

2 1

Multiarray PEP: iterators
by konrad.hinsen＠laposte.net 18 Feb '05

18 Feb '05

The PEP says: Iterator An iterator will be defined that will walk through any array, returning a rank-0 array at each step. Rank-0 arrays act like the appropriate Python scalar and will be converted to one whenever Python asks the object explicitly to try and do so. Order of the iteration is the same for contiguous and discontiguous arrays. The last index always varies the fastest At the moment, iteration over a rank-N array yields rank-(N-1) arrays (indexing along the first dimension). I consider this vastly more useful than iterating over the rank-0 elements and thus ignoring the array structure completely. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire Léon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: hinsen(a)llb.saclay.cea.fr ---------------------------------------------------------------------

3 2

Proposed X.flat solution
by Travis Oliphant 18 Feb '05

18 Feb '05

O.K. here is my X.flat solution. I've decided to make use of the fancy-new, numarray-inspired idea of the UPDATEIFCOPY flag for array creation. Basically, X.flat will create a new array if X is discontiguous but with the UPDATEIFCOPY flag set so that upon destruction the contents will get copied back to X. So, it will seem like the X.flat array is updating the original array (provided it get's dereferenced someday (which it should because you rarely) keep references to X.flat unless you really meant ravel(X). This is the simplest solution. It's not quite as fancy as the indexable iterator (which could still be implemented at some point) but it lets me move forward. -Travis

3 4

Multiarray PEP
by Travis Oliphant 18 Feb '05

18 Feb '05

I am looking for feedback on the PEP. Of particular interest is the specification of multidimensional indexing that I've outlined. I think it is mostly the same as numarray (I'd love some feedback to be sure about that), except for the fact that X[K] where K is a a single integer index array can do 1-d indexing similar to MATLAB. The equivalent numarray indexing is available as X[K,]. Now that I've specified what is to happen, I think it won't be very difficult to code. I would also like some help on resolving the type issue. Is it important to have hierarchial classes defined (probably in Python) that can be used to check and/or specify type or are functions that check for various properties, fine. Right now x.type is specified as an attribute but the attribute could be replaced by a method. What to return is the biggest question. I've never had a problem with just the typecode characters although aliases (like Int16 or Short) which might be more meaningful are nice. Are shadow-classes important in this case? To me they look a little like extra bulk with no gain. I'd love to be educated as to why it is worth the extra effort to have a bunch of instances lying around. I don't feel very strongly about this though, so if somebody else does, I'm willing to go for it. I've designed Numeric3 so that it is not difficult to change how typecodes are specified on the Python level. We can really accommodate whatever. <Opening can ....> What is the opinion of everybody? -Travis

11 18

Some comments on the draft PEP (Rev 1.8)
by Colin J. Williams 18 Feb '05

18 Feb '05

Basic Types There is no mention of the record. This was one of the useful ideas from COBOL and carried over by PL/1. See numarray.record.py. I suggest it is worth continuing. Sequence and Mapping 2) The advanced indexing seems to offer masked extraction from an array, some examples would help. 3) I've always wondered about the rank 0 array. If such values are returned as scalars then the processing inside the array package is increased slightly. If they are returned as an array then there is more work to be done in the Python code outside the the array package. My preference is to return a scalar. Iterator. A scalar appears to be much better in this case. Methods. Why not use properties for those methods which don't require arguments? Some of the maybes are number oriented but the basic Array is intended to have a wider scope than just numbers. Implementation Plan The usual convention is that class/type names start with a capital, thus Array, rather than array. This should clash with a name on the existing array module. It is good to see that the Array is to be sub-classable. This is one of the weaknesses of numarray. 'efficiencies' meant that Python's introspection is not fully available in NumArray. C Structures Is NOTSWAPPED, based on system order, going to give trouble if data is exported from one system to another? Type characters My feeling is that the numarray.numerictypes descriptors are clearer for the user who has not memorized the typecodes. It takes a little more space overhead but simplifies the Array repr. With an editor, such as PythonWin, one can enter 'MN.numerictypes.'. The editor responds with a drop down list of available types - MN is an abbreviated code for the imported modules name (I don't use from X import *). The programmer does not have to remember the non mnemonic letter codes. One final request. In the instance constructor, please avoid the use of 'type' as a parameter name if the constructor is implemented in Python. This hides the builtin type(), which is sometimes useful in debugging or exploring code. I hope that this is of some help. Colin W.

2 1

Re: [Numpy-discussion] random and RandomArray
by Bruce Southey 17 Feb '05

17 Feb '05

Hi, The Mersenne Twister is available in Python as: /Python-2.4/Modules/_randommodule.c This file contains the outdated information: "The code in this module was based on a download from: http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html" The redirected link is titled: Mersenne Twister with improved initialization http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html A brief look at randlib() suggests that it is written so that different uniform generators could be used. Could randf() be rewritten to have the default (current status) function and a second that just to link to the random module library to use the Mersenne Twister? Okay, I do know it is not that easy (the seed code would also need to change so account for which generator is being used) but may be sufficient. Regards Bruce ---- Original message ---- >Date: Wed, 16 Feb 2005 20:05:09 -0800 >From: Robert Kern <rkern(a)ucsd.edu> >Subject: Re: [Numpy-discussion] random and RandomArray >To: numpy-discussion(a)lists.sourceforge.net > >Bruce Southey wrote: >> Hi, >> I was browsing through some of the code and realized that certain random number >> generators occur in both the Python random module and RandomArray. >> >> The random module uses C code to get a scalar uniform random number that is >> modified in Python by other generators to get random numbers from other >> distributions. In RandomArray everything is done in the C code - obviously this >> is way faster especially for arrays. >> >> In the long term, would it make sense to get use the same random number >> generators in both random and RandomArray? > >I think that it is certainly feasible and desirable that when/if a >multiarray object enters the standard library that the standard random >module be extended to produce arrays as well. > >I don't think it's terribly worthwhile to hack the random module to >expose its PRNG so that we use it's implementation without duplicating >code. I think the result will be quite fragile, and won't be useful >until Python 2.5. > >I do think it would be extremely worthwhile to implement the Mersenne >Twister for numarray/scipy. I promised some time ago to look into this, >but I have not gotten around to it, unfortunately. > >-- >Robert Kern >rkern(a)ucsd.edu > >"In the fields of hell where the grass grows high > Are the graves of dreams allowed to die." > -- Richard Harter > > >------------------------------------------------------- >SF email is sponsored by - The IT Product Guide >Read honest & candid reviews on hundreds of IT Products from real users. >Discover which products truly live up to the hype. Start reading now. >http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion(a)lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion

2 1

random and RandomArray
by Bruce Southey 16 Feb '05

16 Feb '05

Hi, I was browsing through some of the code and realized that certain random number generators occur in both the Python random module and RandomArray. The random module uses C code to get a scalar uniform random number that is modified in Python by other generators to get random numbers from other distributions. In RandomArray everything is done in the C code - obviously this is way faster especially for arrays. In the long term, would it make sense to get use the same random number generators in both random and RandomArray? By this, I am thinking along the lines that both random and RandomArray would call the same C code. This would require more changes in random than RandomArray but would minimise duplicate code and update RandomArray. So I was wondering what the thoughts of people on this list are on this before even considering this further. Regards Bruce Southey

3 2