Mailman 3 April 2005 - NumPy-Discussion

Re: [Numpy-discussion] Questions about the array interface.
by Scott Gilbert April 8, 2005

April 8, 2005

--- Andrew Straw <strawman(a)astraw.com> wrote: > > > > This behavior is explained by Tim Peters: > > > > http://groups-beta.google.com/group/comp.lang.python/msg/16dbf848c050405a > > > I feared it was something like that. (No platform independent way to > represent special values like nan, inf, and so on.) So I think if we're > going to require an encoding character such as '<' or '>' we should also > include one that means native which CAN handle these special values... > And document why it's needed and why it may get one into trouble. > The data is either big endian or little endian (or possibly a single byte in which case it doesn't matter). Whether or not the (hardware, operating system, C runtime library, C compiler, or Python implementation) can handle NaNs or Infs is not a property of the data. What does an additional code or two get you? Let's say we used ']' for big endian native, and '[' for little endian native? Does that just indicate the possible presence of NaNs for Infs in the data? Adding those codes doesn't have any affect on whether or not libraries can deal with them. I guess I'm not understanding something. Cheers, -Scott

1 0

Re: [Numpy-discussion] Questions about the array interface.
by Travis Oliphant April 8, 2005

April 8, 2005

Scott Gilbert wrote: >I agree, we need a road map of some sort. It could be multiple PEPs >depending, but it should include most of the following: > > - Get the bytes object submitted. There are only a few small > things in PEP 296 that should be changed. > > > #4 > - I'm not particularly interested in implementing the new bytes > literal and other features discussed in PEP 332, but it is > related to this topic. (The proposal is for b"xxxxxx" to be a > bytes literal.) We should make note that while this is not > part of the numpy roadmap, nothing prohibits that from being > implemented by another user. > > > - Add an ndarray module. This module will contain the ndarray > object as well as a superset of your helper functions. I > think implementing it in pure Python on top of the bytes > object is the right course. It's partly for documentation. > > - Add an include file to make this protocol easily accessible > from C. It's not much code, and the entire thing could be > done with inline/static functions in the .h file. It would > be nice if this went into Python too, but not strictly > required. > > I put these together at #1 > - Add the array protocol attributes to the existing array > object. > > #2 > - Flesh out the "locked buffer" stuff in PEP 298. Add support > for locking the buffer to the existing array object, the > bytes object, the mmap object, and anything else (string?) > that doesn't meet too much resistance. > > #3 > - Fix the existing buffer object to regrab it's pointer > every time it's needed. Could also add support to use > the "locked buffer" interface where possible. I gather > that you are using this particular object in scipy.base > (is that true??). Several shortcomings of it could be > easily fixed at the Python level, but I don't feel > strongly that this would have to be done... Then again > it isn't much work. > > #5 I can't think of anything you've missed. I'm very supportive of this, but I have to finish scipy.base first. I think Perry is supportive as well. I know he's been playing catch-up in the reading. I'm not sure of Todd's opinion. I suspect he would welcome these changes to Python. My preference order is 1) the ndarray module and ndarray.h header with these interface definitions and methods. 2) Add array interface attributes to array module 3) Flesh out locked buffer API 4) Bytes object (with Pickling support) 5) Fix current buffer object. -Travis

2 1

Re: [Numpy-discussion] Alternate C-only array protocol for speed?
by Scott Gilbert April 8, 2005

April 8, 2005

--- "David M. Cooke" <cookedm(a)physics.mcmaster.ca> wrote: > > It seems that people are worried about speed of the attribute-based > array interface when using small arrays in C. > I'm really not worried about it... I just don't want "performance" to be used as an argument for a given design decisions when the proposed change won't actually make things faster. > > Here's an alternative: Define some attribute (for now, call it > [snip] > This would definitely be faster. Faster yet would be doing a PyNumeric_Check (or PyNumarray_Check, or whatever they're called) and just cast the pointer to the underlying representation. If you must go fast, go as fast as possible... I'd rather we didn't add a lot complexity to the array protocol to just go at a medium speed. Cheers, -Scott

1 0

Re: [Numpy-discussion] Questions about the array interface.
by Scott Gilbert April 8, 2005

April 8, 2005

--- Tim Hochberg <tim.hochberg(a)cox.net> wrote: > > I think there is a trade off, but not the one that Chris is worried > about. It should be easy to hide complexity of dealing with missing > attributes through the various helper functions. The cost will be in > speed and will probably be most noticable in C extensions using small > arrays where the extra code to check if an attribute is present will be > signifigant. > > How signifigant this will be, I'm not sure. And frankly I don't care all > that much since I generally only use large arrays. However, since one of > the big faultlines between Numarray and Numeric involves the former's > relatively poor small array performance, I suspect someone might care. > You must check the return value of the PyObject_GetAttr (or PyObject_GetAttrString) calls regardless. Otherwise the extension will die with an ugly segfault the first time one passes an float where an array was expected. If we're talking about small light-weight arrays and a C/C++ function that wants to work with them very efficiently, I'm not convinced that requiring the attributes be present will make things faster. As we're talking about small light weight arrays, it's unlikely the individual arrays will have __array_shape__ or __array_strides__ already stored as tuples. They'll probably store them as a C array as part of their PyObject structure. In the world where some of these attributes are optional: If an attribute like __array_offset__ or __array_shape__ isn't present, the C code will know to use zero or the default C-contiguous layout. So the check failed, but the failure case is probably very fast (since a temporary tuple object doesn't have to be built by the array on the fly). In the world where all of the attributes are required: The array object will have to generate the __array_offset__ int/long or __array_shape___ tuple from it's own internal representation. Then the C/C++ consumer code will bust apart the tuple to get the values. So the check succeeded, but the success code needs to grab the parts of the tuple. The C helper code could look like: struct PyNDArrayInfo { int ndims; int endian; char itemcode; size_t itemsize; Py_LONG_LONG shape[40]; /* assume 40 is the max for now... */ Py_LONG_LONG offset; Py_LONG_LONG strides[40]; /* More Array Info goes here */ }; int PyNDArray_GetInfo(PyObject* obj, PyNDArrayInfo* info) { PyObject* shape; PyObject* offset; PyObject* strides; int ii, len; info->itemsize = too_long_for_this_example(obj); shape = PyObject_GetAttrString(obj, "__array_shape__"); if (!shape) return 0; len = PySequence_Size(shape); if (len < 0) return 0; if (len > 40) return 0; /* This needs work */ info->ndims = len; for (ii = 0; ii<len; ii++) { PyObject* val = PySequence_GetItem(shape, ii); info->shape[ii] = PyLong_AsLongLong(val); Py_DECREF(val); } Py_DECREF(shape); offset = PyObject_GetAttrString(obj, "__array_offset__"); if (offset) { /*** THIS PART MIGHT BE SLOWER WHEN IT SUCCEEDS ***/ info->offset = PyLong_AsLongLong(offset); Py_DECREF(offset); } else { PyErr_Clear(); info->offset = 0; } strides = PyObject_GetAttrString(obj, "__array_strides__"); if (strides) { /*** THIS PART IS ALMOST CERTAINLY SLOWER ***/ for (ii = 0; ii<ndims; ii++) { PyObject* val = PySequence_GetItem(strides, ii); info->strides[ii] = PyLong_AsLongLong(val); Py_DECREF(val); } Py_DECREF(strides); } else { /*** THIS FAILURE PATH IS PROBABLY FASTER ***/ size_t size = info->size; PyErr_Clear(); for (ii = ndims-1; ii>=0; ii--) { info->strides[ii] = size; size *= info->shape[ii]; } } /* More code goes here */ } I have no idea how expensive PyErr_Clear() is. We'd have to profile it to see for certain. If PyErr_Clear() is not expensive, then we could make a strong argument that *not* requiring the attributes will be more efficient. It could also be so close that it doesn't matter - in which case it's back to being a matter of taste... Cheers, -Scott

3 4

Re: [Numpy-discussion] Questions about the array interface.
by Scott Gilbert April 8, 2005

April 8, 2005

--- Andrew Straw <strawman(a)astraw.com> wrote: > > Here's a bit of weirdness which has prevented me from using '<' or '>' > in the past with the struct module. I'm not guru enough to know what's > going on, but it has prevented me from being explicit rather than > implicit. > > In [1]:import struct > > In [2]:from numarray.ieeespecial import nan > > In [3]:nan > Out[3]:nan > > In [4]:struct.pack('<d',nan) > --------------------------------------------------------------------------- > exceptions.SystemError Traceback (most > recent call last) > > /home/astraw/<console> > > SystemError: frexp() result out of range > > In [5]:struct.pack('d',nan) > Out[5]:'\x00\x00\x00\x00\x00\x00\xf8\xff' > No clue why that is, but it certainly looks like a bug in the struct module. It shouldn't make any difference about whether or not the array protocol reports the endian though. It's using a different notation for typecodes. Cheers, -Scott

3 2

profile reveals calls to astype()
by Curzio Basso April 8, 2005

April 8, 2005

Hi all, I have a problem trying to profile a program using numarray, maybe someone with more experience can give me a hint... basically, the program I am profiling has a function like this: foo(): # some code # a call to astype() for i in xrange(N): # some other code and NO explicit call to astype() the problem is that when I print the 'callees' of foo(), astype() gets listed with an occurrence of N+1, as if it was called inside the loop. So now the first doubt I have is that astype() gets listed because called from some function called by foo(), even if this should not happen. Here is the list of numarray functions called in foo() Function called... generic.py:651(getshape)(14) 0.070 generic.py:918(reshape)(2) 0.000 generic.py:1013(where)(2) 0.050 generic.py:1069(concatenate)(2) 4.270 morphology.py:150(binary_erosion)(2) 0.070 numarraycore.py:698(__del__)(120032) 3.240 numarraycore.py:817(astype)(12002) 37.290 numarraycore.py:857(is_c_array)(36000) 10.450 numarraycore.py:878(type)(4) 0.000 numarraycore.py:964(__mul__)(12) 0.340 numarraycore.py:981(__div__)(8) 0.010 numarraycore.py:1068(__pow__)(8) 0.000 numarraycore.py:1180(__imul__)(12000) 0.930 numarraycore.py:1250(__eq__)(2) 0.080 numarraycore.py:1400(zeros)(54) 0.060 numarraycore.py:1409(ones)(8) 0.020 The second thing I can think of is that astype() is implicitly called by some conversion. Can this be? curzio

2 3

Re: [Numpy-discussion] Questions about the array interface.
by Scott Gilbert April 7, 2005

April 7, 2005

Oops, sent too fast. Quick correction... > > In the world where some of these attributes are optional: If an > attribute like __array_offset__ or __array_shape__ isn't present, > the C code will know to use zero or the default C-contiguous layout. > So the check failed, but the failure case is probably very fast > (since a temporary tuple object doesn't have to be built by the array > on the fly). > I meant to say "__array_offset__ or __array_stides___". The __array_shape__ attribute would always be required for arrays... Cheers, -Scott

1 0

Re: [Numpy-discussion] Questions about the array interface.
by Scott Gilbert April 7, 2005

April 7, 2005

--- Chris Barker <Chris.Barker(a)noaa.gov> wrote: > > Again, I'm uncomfortable with something that I have to check being > optional. If it is, we're encouraging people to not check it, and that' > a recipe for bugs later on down the road. > [snip] > > > I guess all I'm saying is that I wouldn't assume the offset is zero... > > Good point. All the more reason to have the offset be mandatory. > Lot's of protocols have optional parts. The helper functions would hide this level of detail. > > Yes, if there is a C/C++ version of all these helper functions, I'll be > a lot happier. And you're right, the same information should not be > encoded in two places, so my "iscontiguous" attribute should be a helper > function or maybe a method. > > > In a short while, you shouldn't have to check any __array_metadata__ > > attributes directly. There should even be a helper function for > > getting the array elements. > > Cool. How would that work? A C++ iterator? I"m thinking not, as this is > all C, no? > I think this will take shape as an include file with static/inline functions. No linking required, just #include <ndarray.h> and call the functions. It would be nice but not necessary that this was distributed with Python. I would be in favor of having some C++ iterator interfaces (possibly a template class) inside of a #ifdef __cplusplus block. Python doesn't seem to have a a lot C++ in the core so I wonder if this would meet resistance (even when it's inside of a #ifdef block). > > > It wouldn't be a horrible mistake to have all the attributes be > > mandatory, but it doesn't get array consumes any benefit that they > > can't get from a well written helper library, and it does add some > > burden to array producers. > > Hardly any. I'm assuming that there will be a base_array class that can > be used as a base class or mixin, so it wouldn't be any work at all to > have a full set of attributes with defaults. It would take up a little > bit of memory. I'm assuming that the whole point of this is to support > large datasets, but maybe that isn't a valid assumption, After all, > small array support has turned out to be very important for Numeric. > If the protocol can make things easy without the use of a mixin or base class, all the better to my way of thinking. I don't think the memory use is very relevant as the attributes would only require storage in the class object, not the instances. There is something elegant about making array creation as easy as: class easy_array: def __init__(self, filename): data = open(filename, 'r').read() self.__array_data__ = data self.__array_shape__ = (len(data)/4,) self.__array_typestr__ = '>i4' Like I said, I don't think it would be *horrible* to require all the attributes, but I don't see how it will benefit you at all. And even if all the attributes are mandatory, there are still a number of details to get right in reading the memory. You'll likely want to use the helper libraries/modules regardless. (Once they're completed of course...) > > As a rule of thumb, I think there will be [more] consumers of arrays > than producers, so I'd rather make it easy on the consumers that the > producers, if we need to make such a trade off. Maybe I'm biased, > because I'm a consumer. > I don't see the trade off. It will be easy for you either way, but harder for array producers (admittedly only a little). This has to be easier than the situation you have today right? Imagine the code you'd have to write to special case Numeric, scipy.base, Numarray, and Python's array module. Cheers, -Scott

3 2

array interface nitpicks
by cookedm＠physics.mcmaster.ca April 7, 2005

April 7, 2005

Just some small nitpicks in the array interface document (http://numeric.scipy.org/array_interface.html): As written: """ __array_shape__ (required) Tuple showing size in each dimension. Each entry in the tuple must be a Python (long) integer. Note that these integers could be larger than the platform "int" or "long" could hold. Use Py_LONG_LONG if accessing the entries of this tuple in C. """ Since this is supposed to be an interface, not an implementation (duck-typing and all that), I think this is too strict: __array_shape__ should just be a sequence of integers, not necessarily a tuple. I'd suggest something like this: ''' __array_shape__ (required) Sequence whose elements are the size in each dimension. Each entry is an integer (a Python int or long). Note that these integers could be larger than the platform "int" or "long" could hold (a Python int is a C long). It is up to the calling code to handle this appropiately; either by raising an error when overflow is possible, or by using Py_LONG_LONG as the C type for the shapes. ''' This is clearer about the users responsibility -- note that Numeric is taking the first approach (error), as the dimensions in PyArrayObject are ints. Similiar comments about __array_strides. I'd reword it along the lines of ''' __array_strides__ (optional) Sequence of strides which provides the number of bytes needed to jump to the next array element in the corresponding dimension. Each entry must be integer (a Python int or long). As with __array_shape__, the values may be larger than can be represented by a C "int" or "long"; the calling code should handle this appropiately, either by raising an error, or by using Py_LONG_LONG in C. Default is a strides tuple which implies a C-style contiguous memory buffer. In this model, the last dimension of the array varies the fastest. For example, the default __array_strides__ tuple for an object whose array entries are 8 bytes long and whose __array_shape__ is (10,20,30) would be (4800, 240, 8) Default: C-style contiguous ''' I'm mostly worried about the use of Python longs; it shouldn't be necessary in almost all cases, and adds extra complications (in normal usage, you don't see Python longs all that much). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm(a)physics.mcmaster.ca

6 12

Re: [Numpy-discussion] Possible example application of the array interface
by Michael Sorich April 7, 2005

April 7, 2005

I think that this is a great idea! While I have a strong preference for python, I generally use R for statistical analyses due to the large number of mature libraries available. There are also some aspects of the R data types (eg data-frames and column/row names for 2D arrays) that are really nice for spreadsheet like data. I hope that scipy.base record arrays will be as easily manipulated as data-frames are. While RPy works well for small simple problems, there are data conversion limitations between R and Python. If one could efficiently convert between the major R data types and python scipy.base data types without loss of data, it would become possible to do most of the data manipulation in python and freely mix in R functions when required. This may encourage the use of python for the development of statistical routines. >From my meager understanding of RPy: R vectors are converted to python lists. It may make more sense to convert them to an array (either stdlib or scipy.base version) - without copying data if possible. R arrays and matrices are converted to Numeric arrays. Eg In [8]: r.array([1,2,3,4,5,6],dim=[2,3]) Out[8]: array([[1, 3, 5], [2, 4, 6]]) However, column and row names (or dimnames for arrays with >2 dimensions) are lost in R->Py conversion. I do not know whether these conversions require copying of the data. R data-frames are currently converted to python dictionaries and I dont think that there is any simple way to convert a python object to an R data frame. This is the biggest limitation of rpy in my opinion. In [16]: r.data_frame(col1=[1,2,3,4],col2=['one','two','three','four']) Out[16]: {'col2': ['one', 'two', 'three', 'four'], 'col1': [1, 2, 3, 4]} If it were possible to convert between an R data-frame and a scipy.base record array without copying or losing data, RPy would become more useful. I wish I understood C, scipy.base and R well enough to give this a go. However, this is Way over my head! Mike --- Magnus Lie Hetland <magnus(a)hetland.org> wrote: > I was just thinking about some experimental designs, > and whether I > could, perhaps, do the statistics in Python. I > remembered having used > RPy [1] briefly at some time (there may be other > similar bindings out > there -- I don't remember) and started thinking > about whether I could, > perhaps, combine it with numpy in some way. My first > thought was to > reimplement the relevant statistical functions; then > I thought about > how to convert data back and forth -- but then it > occurred to me that > R also uses arrays extensively, and that it could, > perhaps, be > possible to expose those (through something like > RPy) through the > array interface/protocol! > > This would be (IMO) a good example of the benefits > of the array > protocol; it's not a matter of "getting yet another > array module". RPy > is an external library/language with *lots* of > features that might be > useful to numpy users, many of which aren't likely > to be implemented > in Python for quite a while, I'd guess (unless, > perhaps, someone > writes a translator from R, which I'm sure is > doable). > > I don't know enough (at least yet ;) about the > implementation of RPy > and the R library to say for sure whether this would > even be possible, > but it does seem like it could be really useful... > > [1] rpy.sf.net > > -- > Magnus Lie Hetland Fall seven > times, stand up eight > http://hetland.org > [Japanese proverb] > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT > Products from real users. > Discover which products truly live up to the hype. > Start reading now. > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > Find local movie times and trailers on Yahoo! Movies. http://au.movies.yahoo.com

5 5