Mailman 3 April 2005 - NumPy-Discussion

array slicing question
by Gary Ruben April 5, 2005

April 5, 2005

This may be relevant to Numeric 3, but is possibly just a general question about array slicing which will either reveal a deficiency in specifying slices or in my knowledge of slicing with numpy. A while ago I was trying to reimplement some Matlab image processing code in Numeric and revealed a deficiency in the way slices are defined. Suppose I have an n x m array and want to slice off the first and last p rows and columns where p can range from 0 to some number. Matlab provides a clean way of doing this, but in numpy it's a bit of a mess. You might think you could do >>> p=1 >>> b = a[p:-p] but if p=0, this fails. My final solution involved getting the array shape and explicitly calculating start and stop columns, but is there a better way? Gary R.

3 3

Re: [Numpy-discussion] The array interface published
by Scott Gilbert April 5, 2005

April 5, 2005

--- Michiel Jan Laurens de Hoon <mdehoon(a)ims.u-tokyo.ac.jp> wrote: > > I'm not sure what you mean by "the array interface could become > part of the Python standard as early as Python 2.5", since there > is nothing to install. Or does this mean that Python's array will > conform to the array interface? > It would be nice to have the Python array module support the protocol for the 1-Dimensional arrays that it implements. It would also be nice to add a *simple* ndarray object in the core that supports multi-dimensional arrays. I think breaking backward compatibility of the existing Python array module to support multiple dimensions would be a mistake and unlikely to get accepted. A PEP would likely be required to make the changes to the array module, and also to add an ndarray module would likely document the interface. In that regard, it could "make it into the core" for Python 2.5. But you're right that external packages could support this interface today. There is nothing to install... > > 1) The "__array_shape__" method is identical to the existing "shape" > method in Numerical Python and numarray (except that "shape" does a > little bit better checking, but it can be added easily > to "__array_shape__"). To avoid code duplication, it might be better > to keep that method. (and rename the other methods for consistency, > if desired). > The intent is that all array packages would have the required/optional protocol attributes. Of course at a higher level, this information will probably be presented to the users, but they might choose a different mechanism. So while A.__array_shape__ always returns a tuple of longs, A.shape is free to return a ShapeObject or be an assignable attribute that changes the shape of the object. With the property mechanism, there is no need to store duplicated data (__array_shape__ can be a property method that returns a dynamically generated tuple). Separating the low level description of the array data in memory from the high level interface that particular packages like scipy.base or numarray present to their users is a good thing. > > 3) Where do default values come from? Is it the responsability of the > extension module writer to find out if the array module implements e.g. > __array_strides__, and substitute the default values if it doesn't? If > so, I have a slight preference to make all methods required, since it's > not a big effort to return the defaults, and there will be more extension > modules than array packages (or so I hope). > If we can get a *simple* package into the core, in addition to implementing an ndarray object, this module could have helper functions that do this sort of thing. For instance: def get_strides(A): if hasattr(A, "__array_strides__"): return A.__array_strides__ shape = A.__array_shape__ size = get_itemsize(A) for i in range(len(shape)-1, -1, -1): strides.append(size) size *= shape[i] return tuple(strides) def get_itemsize(A): typestr = A.__array_typestr__ # skip the endian if typestr[0] in '<>': typestr = typestr[1:] # skip the char code typestr = typestr[1:] return long(typestr) def is_contiguous(A): # etc.... Those are probably buggy and need work, but you get the idea... A C implementation of the above would be easy to do and useful, and it could be done inline in a single include file (no linking headaches). Cheers, -Scott

2 1

Re: [Numpy-discussion] Array interface
by Scott Gilbert April 5, 2005

April 5, 2005

--- Magnus Lie Hetland <magnus(a)hetland.org> wrote: > > I would almost be tempted to say that if __array_descr__ is in use, > __array_typestr__ *has* to use the 'V' type. (Or, one could make some > more complicated rules, perhaps, in order to allow other types.) > Yup, having multiple ways to spell the same information will likely cause problems. Wouldn't be bad for the protocol to say "thou shalt use the specfic typestr when possible". Or to say that the __array_descr__ is only for 'V' typestrs. > > As for not supporting the 'V' type -- would that really be considered > a conforming implementation? According to the spec, "Objects wishing > to support an N-dimensional array in application code should look for > these attributes and use the information provided appropriately". The > typestr is required, so... > I think the intent is that libraries like wxPython or PIL can recognize data that they *want* to work with. They can raise an exception when passed anything that is more complicated than they're willing to deal with. I think many packages will simply punt when they see a 'V' typestr and not look at the more complicated description at all. Nothing wrong with that... The packages that produce more complicated data structures have a way to express it and pass it to the packages that are capable of consuming it. Easy things are easy, and hard things are possible. > > Perhaps the spec should be explicit about the shoulds/musts/mays of > the specific typecodes? What must be supported, what may be supported > etc.? Or perhaps that doesn't make sense? It just seems almost too bad > that one package would have to know what another package supports in > order to formulate its own typestr... It sort of throws part of the > interoperability out the window. > Being very precise in the language describing the protocol is probably a good thing, but I don't see anything that requires packages to formulate their typestr's differently. The little bit of ambiguity that is in the __array_typestr__ and __array_descr__ attributes can be easily clarified. Cheers, -Scott

2 1

__array_typestr__
by Travis Oliphant April 5, 2005

April 5, 2005

For the most part, it seems the array protocol is easy to agree on. The one difficulty is typestr. For what it's worth, here are my opinions on what has been said regarding the typestr. * Endian-ness should be included in the typestr --- it is how the data is viewed and an intrinsic part of the type as much as int, or float. * I like the fact that struct character codes are documented, but it is hard to remember. The simpler division into basic types and byte-widths that the numarray record module uses is easier to remember. * I'm mixed on whether or not support for describing complex data types should be used or if their description as a record is good enough. On the one hand we think of complex numbers as additional types, but on the other hand, in terms of machine layout they really are just two floats, so perhaps it is better to look at them that way in a protocol whose purpose is just describing how to interpret a block of memory. Especially since complex numbers could conceivably be built on top of any of the other types. In addition, it is conceivable that a rational array might be supported by some array object in the future and that would most easily be handled by a record array where the names were now something like ("numer", "denom") . The typestr argument should just help us specify what is in the memory chunk at each array element (how should it be described). * I'm wondering about including multiple types in the typestr. On the one hand we could describe complicated structures by packing all the information into the typestr. On the other hand, it may be better if we just use 'V8' to describe an 8-byte memory buffer with an additional attribute that contains both the names and the typestr: __array_recinfo__ = (('real','f4'),('imag','f4')) or for a "rational type" __array_recinfo__ = (('numer','i4'),('denom','i4')) so that the detail of the typecode for a "record" type is handled by another special method using tuples. On this level, we could add the possibility of specifying a shape for a small array inside (just like the record array of numarray does). -Travis

7 15

Re: [Numpy-discussion] Array interface
by Travis Oliphant April 5, 2005

April 5, 2005

Magnus Lie Hetland wrote: > - Does the description of __array_data__ mean that the discussed > bytes type is no longer needed? (If we can use buffers, that > sounds very good to me.) > > We can use the buffer object, now and it works as far as it goes. But, there are very important reasons for the creation of a good bytes object. Probably, THE most important reason for the bytes object is Pickle support without always making an intermediate string (and the accompanying copy that is involved). Right now, a string is the only way to Pickle array data. A bytes object would allow a way to Pickle without making a copy. -Travis

3 3

Re: [Numpy-discussion] Array interface
by Travis Oliphant April 4, 2005

April 4, 2005

Magnus Lie Hetland wrote: >Travis Oliphant <oliphant(a)ee.byu.edu>: > > >>I don't know if you have followed the array interface discussion. It >>is defined at http://numeric.scipy.org >> >> > >This very, very good! The numeric future of Python is looking very >bright, IMO :) > >Some tiny points: > > - Shouldn't the regexp for __array_typestr__ be > '[<>]?[tbiufcOSUV][0-9]+'? > > Probably. Since, I guess you can only have one of < or > . Thanks.. > - What are the semantics when __array_typestr__ isn't V[0-9]+ and > __array_descr__ is set? Is __array_typestr__ ignored? Or... What > would it be used for? > > I would say that the __array_descr__ always gives more information but not every array implementation will support looking at it. For example, current Numeric (24.0 in CVS) ignores __array_descr__ and just looks at the typestr (and doesn't support 'V'). So, I suspect that another array package that knows this may choose something else besides 'V' if it really wants Numeric to still understand it. Suppose you have a complex short int array with __array_descr__ = 'V8 > - Does the description of __array_data__ mean that the discussed > bytes type is no longer needed? (If we can use buffers, that > sounds very good to me.) > > Bytes is still needed because the buffer object is not very good and we need a good buffer object in Python for lots of other reasons. It would be very useful, for example to be able to allocate memory using the Python bytes object. But, it does mean less pressure to get it to work. > - Why the parentheses around "buffer protocol-satisfying object" in > the description of __array_mask__? And why must it be 'b1'? What > if I happen to have mask data from a non-array-protocol source, > which happens to be, say, b8 (not unreasonable, I think)? Wouldn't > it be good to allow any size of these, and just use zero/non-zero > as the criterion? Some of the point of this protocol is to avoid > copying and using the original data, after all...? (Same goes for > the requirement that it be C-contiguous. I guess I'm basically > saying that perhaps __array_mask__ should be an array itself. Or, > at least, that it could be *allowed* to be...) > > I added the mask late last night. It is probably the least thought out portion. Everything else has been through the ringer a couple more times. My whole thinking is that I just didn't want to explode the protocol with another special name for the mask type. But, saying that the mask object itself can support the array interface doesn't do that, so I think that is a good call. Last night, using the numarray exporter interface and the Numeric consumer interface I was able to share data between a Numeric array and numarray array with no copying of the data buffers. It was very nice. -Travis

2 3

Bytes Object and Metadata
by Scott Gilbert April 3, 2005

April 3, 2005

Adding metadata at the buffer object level causes problems for "view" semantics. Let's say that everyone agreed what "itemsize" and "itemtype" meant: real_view = complex_array.real The real_view will have to use a new buffer since they can't share the old one. The buffer used in complex_array would have a typecode like ComplexDouble and an itemsize of 16. The buffer in real_view would need a typecode of Double and an itemsize of 8. If metadata is stored with the buffer object, it can't be the same buffer object in both places. Another case would be treating a 512x512 image of 4 byte pixels as a 512x512x4 image of 1 byte RGBA elements. Or even coercing from Signed to Unsigned. The bytes object as proposed does allow new views to be created from other bytes objects (sharing the same memory underneath), and these views could each have separate metadata, but then you wouldn't be able to have arrays that used other types of buffers. Having arrays use mmap buffers is very useful. The bytes object shouldn't create views from arbitrary other buffer objects because it can't rely on the general semantics of the PyBufferProcs interface. The foreign buffer object might realloc and invalidate the pointer for instance... The current Python "buffer" builtin does this, and the results are bad. So creating a bytes object as a view on the mmap object doesn't work in the general case. Actually, now that I think about it, the mmap object might be safe. I don't believe the current implementation of mmap does any reallocing under the scenes and I think the pointer stays valid for the lifetime of the object. If we verified that mmap is safe enough, bytes could make a special case out of it, but then you would be locked into bytes and mmap only. Maybe that's acceptable... Still, I think keeping the metadata at a different level, and having the bytes object just be the Python way to spell a call to C's malloc will avoid a lot of problems. Read below for how I think the metadata stuff could be handled. --- Chris Barker <Chris.Barker(a)noaa.gov> wrote: > > There are any number of Third party extensions that could benefit from > being able to directly read the data in Numeric* arrays: PIL, wxPython, > etc. Etc. My personal example is wxPython: > > At the moment, you can pass a Numeric or numarray array into wxPython, > and it will be converted to a wxList of wxPoints (for instance), but > that is done by using the generic sequence protocol, and a lot of type > checking. As you can imagine, that is pretty darn slow, compared to just > typecasting the data pointer and looping through it. Robin Dunn, quite > reasonably, doesn't want wxPython to depend on Numeric, so that's what > we've got. > > My understanding of this memory object is that an extension like > wxPython wouldn't not need to know about Numeric, but could simply get > the memory Object, and there would be enough meta-data with it to > typecast and loop through the data. I'm a bit skeptical about how this > would work. It seems that the metadata required would be the full set of > stuff in an array Object already: > > type > dimensions > strides > > This could be made a bit simpler by allowing only contiguous arrays, but > then there would need to be a contiguous flag. > > To make use of this, wxPython would have to know a fair bit about > Numeric Arrays anyway, so that it can check to see if the data is > appropriate. I guess the advantage is that while the wxPython code would > have to know about Numeric arrays, it wouldn't have to include Numeric > headers or code. > I think being able to traffic in N-Dimensional arrays without requiring linking against the libraries is a good thing. Several years ago, I proposed a solution to this problem. Actually I did a really poor job of proposing it and irritated a lot of people in the process. I'm embarrassed to post a link to the following thread, but here it is anyway: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/1166013 Accept my appologies if you read the whole thing just now. :-) Accept my sincere appologies if you read it at the time. I think the proposal is still relevant today, but I might revise it a bit as follows. A bear minimum N-Dimensional array for interchanging data across libraries could get by with following attributes: # Create a simple record type for storing attributes class BearMin: pass bm = BearMin() # Set the attributes sufficient to describe a simple ndarray bm.buffer = <a buffer or sequence object> bm.shape = <a tuple of ints describing it's shape> bm.itemtype = <a string describing the elements> The bm.buffer and bm.shape attributes are pretty obvious. I would suggest that the bm.itemtype borrow it's typecodes from the Python struct module, but anything that everyone agreed on would work. (The struct module is nice because it is already documented and supports native and portable types of many sizes in both endians. It also supports composite struct types.) Those attributes are sufficient for someone to *produce* an N-Dimensional array that could be understood by many libraries. Someone who *consumes* the data would need to know a few more: bm.offset = <an integer offset into the buffer> bm.strides = <a tuple of ints for non-contiguous or Fortran arrays> The value of bm.offset would default to zero if it wasn't present, and the the tuple bm.strides could be generated from the shape assuming it was a C style array. Subscripting operations that returned non-contiguous views of shared data could change bm.offset to non-zero. Subscripting would also affect the bm.strides, and creating a Fortran style array would require bm.strides to be present. You might also choose to add bm.itemsize in addition to bm.itemtype when you can describe how big elements are, but you can't sufficiently describe what the data is using the agreed upon typecodes. This would be uncommon. The default for bm.itemsize would come from struct.calcsize(bm.itemtype). You might also choose to add bm.complicated for when the array layout can't be described by the shape/offset/stride combination. For instance bm.complicated might get used when creating views from more sophisticated subscripting operations like index arrays or mask arrays. Although it looks like Numeric3 plans on making new contiguous copies in those cases. The C implementations of arrays would only have to add getattr like methods, and the data could be stored very compactly. >From those minimum 5-7 attributes (metadata), an N-Dimensional array consumer could determine most everything it needed to know about the data. Simple routines could determine things like iscontiguous(bm), iscarray(bm) or isfortran(bm). I expect libraries like wxPython or PIL could punt (raise an exception) when the water gets too deep. It also doesn't prohibit other attributes from being added. Just because an N-Dimensional array described it's itemtype using the struct module typecodes doesn't mean that it couldn't implement more sophisticated typing hierarchies with a different attribute. There are a few commonly used types like "long double" which are not supported by the struct module, but this could be addressed with a little discussion. Also you might want a "bit" or "Object" typecode for tightly packed mask arrays and Object arrays. The names could be argued about, and something like: bm.__array_buffer__ bm.__array_shape__ bm.__array_itemtype__ bm.__array_offset__ bm.__array_strides__ bm.__array_itemsize__ bm.__array_complicated__ would really bring home the notion that the attributes are a description of what it means to participate in an N-Dimensional array protocol. Plus names this long and ugly are unlikely to step on the existing attributes already in use by Numeric3 and Numarray. :-) Anyway, I proposed this a long time ago, but the belief was that one of the standard array packages would make it into the core very soon. With a standard array library in the core, there wouldn't be as much need for general interoperability like this. Everyone could just use the standard. Maybe that position would change now that Numeric3 and Numarray both look to have long futures. Even if one package made it in, the other is likely to live on. I personally think the competition is a good thing. We don't need to have only one array package to get interoperability. I would definitely like to see the Python core acquire a full fledged array package like Numeric3 or Numarray. When I log onto a new Linux or MacOS machine, the array package would just be there. No installs, no hassle. But I still think a simple community agreed upon set of attributes like this would be a good idea. --- Peter Verveer <verveer(a)embl.de> wrote: > > It think it would be a real shame not to support non-contiguous data. > It would be great if such a byte object could be used instead of > Numeric/numarray arrays when writing extensions. Then I could write C > extensions that could be made available very easily/efficiently to any > package supporting it without having to worry about the specific C api > of those packages. If only contiguous byte objects are supported that > byte object is not a good option anymore for implementing extensions > for Numeric unless I am prepared to live with a lot of copying of > non-contiguous arrays. > I'm hoping I made a good case for a slightly different strategy above. But even if the metadata did go into the bytes object itself, the metadata could describe a non-contiguous layout on top of the contiguous chunk of memory. There is another really valid argument for using the strategy above to describe metadata instead of wedging it into the bytes object: The Numeric community could agree on the metadata attributes and start using it *today*. If you wait until someone commits the bytes object into the core, it won't be generally available until Python version 2.5 at the earliest, and any libraries that depended on using bytes stored metadata would not work with older versions of Python. Cheers, -Scott

7 24

Numeric3 - a Windows Problem
by Colin J. Williams April 3, 2005

April 3, 2005

C:\Python24\Lib\site-packages\Numeric3\Download>python setup.py install running install running build running config error: The .NET Framework SDK needs to be installed before building extensions for Python. Is there any chance that a Windows binary could be made available for testing? Colin W.

2 2

scipy.base (Numeric3) now has math
by Travis Oliphant April 2, 2005

April 2, 2005

I've updated scipy.base (Numeric3) so math is now supported (uses the old ufunc apparatus with new added types support). There is still some work to be done so this is still very alpha (but at least math operations work): - update the ufunc apparatus to use buffers to avoid copying an entire array just for type casting (and to support unaligned and non byteswapped arrays) - update the way error handling is done. - update the coercion strategy like numarray does - fix all the bugs. I've also fixed things so Numeric extension modules should compile --- Please report warnings and bugs with this as well. Thanks for all your help, -Travis

1 0

Re: [Numpy-discussion] Thoughts on getting "something" in the Python core
by Travis Oliphant April 2, 2005

April 2, 2005

>>I'm still very committed to Numeric3 as I want to bring the numarray and >>Numeric people together behind a single array object for scientific >>computing. >> >> Notice that regardless of what I said about what goes into standard Python, something like Numeric3 will always exist for use by scientific users. It may just be a useful add on package like Numeric has always been. There is no way I'm going to abandon use of a more capable Numeric. >Right. I believe that, among all libraries related with numeric array, >eventually only one library in the Python core will survive no matter >how much advanced functions are available, because of the strong >compatibility with other packages. > > I don't think this is true. Things will survive based on utility. What we are trying to do with the Python core is define a standard protocol that is flexible enough to handle anybody's concept of an advanced array (in particular the advanced array that will be in scipy.base). >Totally agree. I doubt that Guido will accept a large and complex >library into the standard Python core. I think Numeric is already too >complex, and numarray is far more complex to be a standard lib in the >Python core. Numeric3 must shift its focus from better Numeric to >scale-downed Numeric. > > I disagree about "shifting focus." Personally, I'm not going to work on something like that until we have a single array package that fulfills the needs of all Numeric and most numarray users. I'm just pointing out that what goes in to the Python core should probably be a scaled down object with a souped-up "protocol" so that the array object in scipy.base can be used through the array protocol by any other package without worrying about having scipy_core at compile time. >For example, how many Python users care about masked arrays? How many >Python users want the advanced type from the Python core? I think the >advanced array type should in some extension lib, not in core array >lib. > Perhaps you do see my point of view. Not all Python users care about an advanced array object but nearly all technical (scientific and engineering users) will. We just need interoperability. >If we make clear our target – becoming a standard library in the >Python core, we may have no problem in determining what functions >should be in the core array lib and what functions should be in >extension libraries using the core array type. > > >Today, the array type in the Python core is almost useless. >If Numeric3 offers just much faster performance on numeric types, many >Python users will start to use new array type in their applications. >Once it happens, we can create a bunch of extension libraries for more >advanced operations on the new array type. > > The "bunch of extension libraries" is already happening and is already in progress. I think we've overshot the mark for the Python core, however. No need to wait "til something happens" >With all my heart I hope that Numeric3 gears to this direction before > > >we get the tragedy to have Numeric4, Numeric5, and so on. > > I'm coming to see that what is most important for the Python core is "protocols". Then, there can be a "million" different array types that can all share each other's memory without hassle and much overhead. I'm still personally interested in a better Numeric, however, and so won't be abandoning the concept of Numeric3 (notice I now call it scipy.base --- not a change of focus just a change of name). I just wanted to encourage some discussion on the array protocol. -Travis

3 3