Mailman 3 January 2007 - NumPy-Discussion

subclassing float64 (and friends)
by eric jones Jan. 5, 2007

Jan. 5, 2007

Hey all, I am playing around with sub-classing the new-fangled float64 objects and friends. I really like the new ndarray subclassing features (__array_finalize__, etc.), and was exploring whether or not the scalars worked the same way. I've stubbed my toe right out of the blocks though. I can sub-class from standard python floats just fine, but when I try to do the same from float64, I get a traceback. (examples below) Anyone have ideas on how to do this correctly? thanks, eric class MyFloat(float): def __new__(cls, data, my_attr=None): obj = float.__new__(cls, data) obj.my_attr = my_attr return obj a = MyFloat(1.2,my_attr="hello") print a, a.my_attr output: 1.2 hello from numpy import float64 class MyFloat2(float64): def __new__(cls, data, my_attr=None): obj = float64.__new__(cls, data) obj.my_attr = my_attr return obj a = MyFloat2(1.2,my_attr="hello") print a, a.my_attr output: Traceback (most recent call last): File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", line 33, in ? a = MyFloat2(1.2,my_attr="hello") File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", line 30, in __new__ obj.my_attr = my_attr AttributeError: 'numpy.float64' object has no attribute 'my_attr'

3 5

1.0.1 release notes missing?
by Sven Schreiber Jan. 5, 2007

Jan. 5, 2007

I found the release notes on the way to 1.0 very useful. However, they stop at 1.0, and I also didn't find any in the 1.0.1 source distribution. Do they exist somewhere? Or are there really only bugfixes in 1.0.1? That information would be useful, too. Thank you, Sven

1 0

ndarray newbie question
by belinda thom Jan. 5, 2007

Jan. 5, 2007

Pierre, Thanks for the reply. > Some context: the type of introspection I'm often wishing I could do > in a single, easy command usually has to do w/getting to know the > software better. dir? obj.__doc__ ? When I try the latter recommendation above w/numpy arrays, I get: In [1]: import numpy as N In [2]: a = N.array([1,2,3]) In [3]: a.__doc__? Type: NoneType Base Class: <type 'NoneType'> String Form: None Namespace: Interactive Docstring: <no docstring> Did you have something else in mind? Thx, --b

4 4

returning recarray records as plain arrays
by Matthew Koichi Grimes Jan. 5, 2007

Jan. 5, 2007

As per Stefan's help, I've made a subclass of recarray called nnvalue. It just fixes the dtype to [('x', 'f8'), ('dx', 'f8'), ('delta', 'f8)], and adds a few member functions. I basically treat nnvalue as a struct with three equal-shaped array fields: x, dx, and delta. I'd like it if, when I reference individual fields of a nnvalue, they were returned as plain arrays. Instead, they are returned as nnarrays with empty .x, .dx, and .deltas fields. These fields' dType is also still set to [('x', 'f8'), ('dx', 'f8'), ('delta', 'f8)], when I wish they'd just be 'f8'. The following illustrates the problem, followed by the awkward workaround I'm currently using: >>> nnv = nnvalue(3) # a 3-element nnvalue >>> nnv nnvalue([(0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0)], dtype=[('x', '<f8'), ('dx', '<f8'), ('delta', '<f8')]) >>> nnv.x nnvalue([0.0, 0.0, 0.0], dtype=[('x', '<f8'), ('dx', '<f8'), ('delta', '<f8')]) >>> nnv.x.view(dtype('f8')) array([ 0., 0., 0.]) How can I get my recarray subclass to return individual records as plain array views? -- Matt numpy-discussion-request(a)scipy.org wrote: > Send Numpy-discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://projects.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Numpy-discussion digest..." > > ------------------------------------------------------------------------ > > Today's Topics: > > 1. The risks of empty() (Bock, Oliver BGI SYD) > 2. Re: The risks of empty() (Robert Kern) > 3. Re: The risks of empty() (Tim Hochberg) > 4. Re: The risks of empty() (A. M. Archibald) > 5. Re: newbie: attempt at data frame (Sven Schreiber) > 6. subclassing float64 (and friends) (eric jones) > 7. Re: newbie: attempt at data frame (Sven Schreiber) > 8. Re: subclassing float64 (and friends) (Stefan van der Walt) > > > ------------------------------------------------------------------------ > > Subject: > [Numpy-discussion] The risks of empty() > From: > "Bock, Oliver BGI SYD" <Oliver.Bock(a)barclaysglobal.com> > Date: > Wed, 3 Jan 2007 10:48:19 +1100 > To: > <numpy-discussion(a)scipy.org> > > To: > <numpy-discussion(a)scipy.org> > > > Some of my arrays are not fully populated. (I separately record which > entries are valid.) I want to use numpy.empty() to speed up the > creation of these arrays, but I'm worried about what will happen if I > apply operations to the entire contents of these arrays. E.g. > > a + b > > I care about the results where valid entries align, but not otherwise. > Given that numpy.empty() creates an ndarray using whatever junk it finds > on the heap, it seems to me that there is the possibility that this > could include bit patterns that are not valid floating point > representations, which might raise floating point exceptions if used in > operations like the one above (if they are "signalling" NaNs). Will > this be a problem, or will the results of operations on invalid floating > point numbers yield NaN? > > Or to put it another way: do I need to ensure that array data is > initialised before using it? > > > Oliver > > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] The risks of empty() > From: > Robert Kern <robert.kern(a)gmail.com> > Date: > Tue, 02 Jan 2007 18:04:13 -0600 > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > > Bock, Oliver BGI SYD wrote: > >> Some of my arrays are not fully populated. (I separately record which >> entries are valid.) I want to use numpy.empty() to speed up the >> creation of these arrays, but I'm worried about what will happen if I >> apply operations to the entire contents of these arrays. E.g. >> >> a + b >> >> I care about the results where valid entries align, but not otherwise. >> Given that numpy.empty() creates an ndarray using whatever junk it finds >> on the heap, it seems to me that there is the possibility that this >> could include bit patterns that are not valid floating point >> representations, which might raise floating point exceptions if used in >> operations like the one above (if they are "signalling" NaNs). Will >> this be a problem, or will the results of operations on invalid floating >> point numbers yield NaN? >> >> Or to put it another way: do I need to ensure that array data is >> initialised before using it? >> > > You have essentially full control over floating point exceptions using seterr(), > so you can silence even the signalling NaNs if you want. > > olderrstate = seterr(all='ignore') > # Do stuff that might generate spurious warnings. > seterr(**olderrstate) > > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] The risks of empty() > From: > Tim Hochberg <tim.hochberg(a)ieee.org> > Date: > Tue, 02 Jan 2007 17:08:05 -0700 > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > > Bock, Oliver BGI SYD wrote: >> Some of my arrays are not fully populated. (I separately record which >> entries are valid.) I want to use numpy.empty() to speed up the >> creation of these arrays, but I'm worried about what will happen if I >> apply operations to the entire contents of these arrays. E.g. >> >> a + b >> >> I care about the results where valid entries align, but not otherwise. >> Given that numpy.empty() creates an ndarray using whatever junk it finds >> on the heap, it seems to me that there is the possibility that this >> could include bit patterns that are not valid floating point >> representations, which might raise floating point exceptions if used in >> operations like the one above (if they are "signalling" NaNs). Will >> this be a problem, or will the results of operations on invalid floating >> point numbers yield NaN? >> > This depends on what the error state is set to. You can set it to > ignore floating point errors, in which case this will almost certainly > work. > > However, why take the chance. Why not just build your arrays on top of > zeros instead of empty? Most of the ways that I can think of filling > in a sparse array are slow enough to overwhelm the extra overhead of > zeros versus empty. >> Or to put it another way: do I need to ensure that array data is >> initialised before using it? >> > I think that this should work if you set the err state correctly (for > example (seterr(all="ignore"). However, I don't like shutting down the > error checking unless absolutely necessary, and overall it just seems > better to initialize the arrays. > > -tim > > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] The risks of empty() > From: > "A. M. Archibald" <peridot.faceted(a)gmail.com> > Date: > Tue, 2 Jan 2007 20:12:24 -0400 > To: > "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > > To: > "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > > > On 02/01/07, Bock, Oliver BGI SYD <Oliver.Bock(a)barclaysglobal.com> wrote: >> Some of my arrays are not fully populated. (I separately record which >> entries are valid.) I want to use numpy.empty() to speed up the >> creation of these arrays, but I'm worried about what will happen if I >> apply operations to the entire contents of these arrays. E.g. >> >> a + b > > Have you looked at masked arrays? They are designed to do what you want. > >> I care about the results where valid entries align, but not otherwise. >> Given that numpy.empty() creates an ndarray using whatever junk it finds >> on the heap, it seems to me that there is the possibility that this >> could include bit patterns that are not valid floating point >> representations, which might raise floating point exceptions if used in >> operations like the one above (if they are "signalling" NaNs). Will >> this be a problem, or will the results of operations on invalid floating >> point numbers yield NaN? > > There is indeed the possibility. Even with floating-point exceptions > turned off, on some machines (e.g., Pentium Ms) NaNs are extremely > slow to calculate with (because they are handled in software). I'm not > sure that there *are* bit patterns that are not valid floating-point > numbers, but in any case while using empty does not in practice seem > to lead to trouble, you could have some surprising slowdowns if the > array happens to be filled with NaNs. > > I recommend using masked arrays, which have the further advantage that > values in invalid ("masked") entries are not computed at all. (If your > invalid entries were few or arose naturally or you use (say) Opterons, > I might recommend using NaNs to mark invalid entries.) > >> Or to put it another way: do I need to ensure that array data is >> initialised before using it? > > It does not seem to be a problem in practice, but there are tools to > help with what you want to do. > > A. M. Archibald > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] newbie: attempt at data frame > From: > Sven Schreiber <svetosch(a)gmx.net> > Date: > Wed, 03 Jan 2007 10:00:10 +0100 > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > > Vincent Nijs schrieb: > > >> If there is an easy way to read array data + variable names using the csv >> module it would be great if that could be added to cookbook/InputOutput. I >> couldn't figure out how to do it. >> >> >> > > Hi Vincent, of course it depends a little on how exactly your csv file > looks like, but if you just have column headers and the actual data, you > might try something like the following: > > import csv > from numpy import mat # or array if you like > file_to_read = file(filename, 'r') > read_from = csv.reader(file_to_read, skipinitialspace = True) > obslist = [] > datalist = [] > for line in read_from: > obslist.append(line[0]) > datalist.append(line[1:]) > file_to_read.close() > # (datalist should now be a nested list, first index rows, second > # columns) > # (still contains the headers) > varnames = datalist.pop(0) > # now the real data > data = mat(datalist, dtype = float) > > -sven > > > > ------------------------------------------------------------------------ > > Subject: > [Numpy-discussion] subclassing float64 (and friends) > From: > eric jones <eric(a)enthought.com> > Date: > Wed, 03 Jan 2007 04:29:10 -0600 > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > > Hey all, > > I am playing around with sub-classing the new-fangled float64 objects > and friends. I really like the new ndarray subclassing features > (__array_finalize__, etc.), and was exploring whether or not the > scalars worked the same way. I've stubbed my toe right out of the > blocks though. I can sub-class from standard python floats just fine, > but when I try to do the same from float64, I get a traceback. > (examples below) Anyone have ideas on how to do this correctly? > > thanks, > eric > > class MyFloat(float): > > def __new__(cls, data, my_attr=None): > obj = float.__new__(cls, data) > obj.my_attr = my_attr > return obj > > a = MyFloat(1.2,my_attr="hello") > print a, a.my_attr > > output: > 1.2 hello > > > from numpy import float64 > > class MyFloat2(float64): > > def __new__(cls, data, my_attr=None): > obj = float64.__new__(cls, data) > obj.my_attr = my_attr > return obj > > a = MyFloat2(1.2,my_attr="hello") > print a, a.my_attr > > > output: > Traceback (most recent call last): > File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", > line 33, in ? > a = MyFloat2(1.2,my_attr="hello") > File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", > line 30, in __new__ > obj.my_attr = my_attr > AttributeError: 'numpy.float64' object has no attribute 'my_attr' > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] newbie: attempt at data frame > From: > Sven Schreiber <svetosch(a)gmx.net> > Date: > Wed, 03 Jan 2007 12:07:15 +0100 > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > > Sven Schreiber schrieb: > > >> Hi Vincent, of course it depends a little on how exactly your csv file >> looks like, but if you just have column headers and the actual data, you >> might try something like the following: >> >> > > Ok sorry the previous thing doesn't work, I also stumbled over the > strings. Here's the next attempt, also shorter. (this time even tested ;-) > > import csv > from numpy import mat > > read_from = csv.reader(file(filename, 'r'), skipinitialspace = True) > stringlist = [ line for line in read_from ] > varnames = stringlist.pop(0)[1:] > datalist = [ map(float, line[1:]) for line in stringlist ] > > # now the real data > data = mat(datalist, dtype = float) > > > I actually quite like it... python lists are very nice. This discards > the observation labels, but it's not difficult to add that, of course. > > -sven > > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] subclassing float64 (and friends) > From: > Stefan van der Walt <stefan(a)sun.ac.za> > Date: > Wed, 3 Jan 2007 14:44:45 +0200 > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > To: > Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > > On Wed, Jan 03, 2007 at 04:29:10AM -0600, eric jones wrote: > >> I am playing around with sub-classing the new-fangled float64 objects >> and friends. I really like the new ndarray subclassing features >> (__array_finalize__, etc.), and was exploring whether or not the scalars >> worked the same way. I've stubbed my toe right out of the blocks >> though. I can sub-class from standard python floats just fine, but when >> I try to do the same from float64, I get a traceback. (examples below) >> Anyone have ideas on how to do this correctly? >> >> from numpy import float64 >> >> class MyFloat2(float64): >> >> def __new__(cls, data, my_attr=None): >> obj = float64.__new__(cls, data) >> obj.my_attr = my_attr >> return obj >> >> a = MyFloat2(1.2,my_attr="hello") >> print a, a.my_attr >> >> >> output: >> Traceback (most recent call last): >> File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", >> line 33, in ? >> a = MyFloat2(1.2,my_attr="hello") >> File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", >> line 30, in __new__ >> obj.my_attr = my_attr >> AttributeError: 'numpy.float64' object has no attribute 'my_attr' >> > > With classes defined in C I've often noticed that you can't add > attributes, i.e. > > f = N.float64(1.2) > f.x = 1 > > breaks with > > AttributeError: 'numpy.float64' object has no attribute 'x' > > The way to fix this for arrays is to first view the array as the new > subclass, i.e. > > x = N.array([1]) > class Ary(N.ndarray): > pass > x = x.view(Ary) > x.y = 1 > > However, with floats I noticed that the following fails: > > import numpy as N > f = N.float64(1.2) > class Floaty(N.float64): > pass > f.view(Floaty) > > with > > TypeError: data type cannot be determined from type object > > Maybe this is part of the problem? > > Regards > Stéfan > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion >

3 4

compress numpy vs .numarray
by Sebastian Haase Jan. 5, 2007

Jan. 5, 2007

Hi! when calling compress I get this error message after moving to numpy: ValueError: 'condition must be 1-d array' Is the reason for this the change of the default axis from axis=0 to axis=None What does axis=None mean in this case !? Thanks, -Sebastian

2 3

Update to Array-Interface PEP
by Travis E. Oliphant Jan. 4, 2007

Jan. 4, 2007

Here is a new update. I added the PY_POINTER, and PY_ARRAYOF specifications to simplify the extended specification and allow memory that is a pointer to another memory. This should be the last update today. -Travis PEP: <unassigned> Title: Extending the buffer protocol to include the array interface Version: $Revision: $ Last-Modified: $Date: $ Author: Travis Oliphant <oliphant(a)ee.byu.edu> Status: Draft Type: Standards Track Created: 28-Aug-2006 Python-Version: 2.6 Abstract This PEP proposes extending the tp_as_buffer structure to include function pointers that incorporate information about the intended shape and data-format of the provided buffer. In essence this will place an array interface directly into Python. Rationale Several extensions to Python utilize the buffer protocol to share the location of a data-buffer that is really an N-dimensional array of a specific data-format. However, there is no standard way to exchange the additional N-dimensional array information and data-format information so that the data-buffer can be interpreted correctly. The NumPy project introduced an array interface (http://numpy.scipy.org/array_interface.shtml) through a set of attributes on the object itself. While this approach works, it requires attribute lookups which can be expensive when sharing many small arrays. One of the key reasons that users often request to place something like NumPy into the standard library is so that it can be used as standard for other packages that deal with arrays or extension modules that wish to do some kind of computation on a block of memory (that is viewed as an array). This PEP provides a mechanism for extending the buffer protocol (which already allows data sharing) to add the additional information needed to understand the data. This should be of benefit to all third-party modules that want to share memory through the buffer protocol such as GUI toolkits, PIL, PyGame, CVXOPT, PyVoxel, PyMedia, audio libraries, video libraries etc. It will also be of benefit to extension writers who are writing mathematical algorithms that expect a chunk of memory of a certain data-format to be passed in. A large number of NumPy C-API users simply use the standard mechanism to request a contiguous chunk of memory upon which the algorithms can be written. This proposal will in essence provide a way for extension writers to write to a generic array API that is in Python itself intead of an array-package specific API. Proposal Add bf_getarrview and bf_relarrview function pointers to the buffer protocol to allow objects to share a view on a memory pointer including information about accessing it as an N-dimensional array. Add the TP_HAS_ARRAY_BUFFER flag to types that define this extended buffer protocol. Also some additional C-API calls should be added to Python to facilitate creating new PyArrViewObjects and requesting views of a specific kind. Specification: static PyObject* bf_getarrayview (PyObject *obj) This function must return a new reference to a PyArrViewObject which contains the details of the array information exposed by the object. If failure occurs, then NULL is returned and an exception set. The requirement of an array is that each item be a contiguous segment of memory of a specific size. static int bf_relarrayview(PyObject *obj) If not NULL then this will be called when the object returned by bf_getarrview is destroyed so that the underlying object can be aware when acquired "views" are released. The object that defines bf_getarrview should not re-allocate memory (re-size itself) while views are extant. A 0 is returned on success and a -1 and an error condition set on failure. The PyArrayViewObject has the structure typedef struct { PyObject_HEAD void *data; /* pointer to the beginning of data */ int nd; /* the number of dimensions */ Py_ssize_t *shape; /* c-array of size nd giving shape */ Py_ssize_t *strides; /* SEE BELOW */ PyObject *base; /* the object this is a "view" of */ PyObject *format; /* SEE BELOW */ int flags; /* SEE BELOW */ } PyArrayViewObject; strides -- a c-array of size nd providing the striding information which is the number of bytes (or bits if format->primitive is PY_BIT) to skip to get to the next element in that dimension. format -- a Python data-format object (PyDataFormatObject) which contains information about how each item in the array should be interpreted. flags -- an integer of flags. PYARR_WRITEABLE is the only flag that must be set appropriately by types. Other flags: PYARR_ALIGNED, PYARR_C_CONTIGUOUS, PYARR_F_CONTIGUOUS, and PYARR_NOTSWAPPED can all be determined from the rest of the PyArrayViewObject using the UpdateFlags C-API. PYARR_UPDATEIFCOPY allows automatic backward copying of data into the source if the view created a copy. The memory model that an array represents is that each element of the array takes up a specific, fixed amount of memory and strides are used to walk memory to get to each element of the array. The PyDataFormatObject represents how each fixed-sized item in the array should be interpreted. It has the structure. typedef struct { PyObject_HEAD PySimpleformat primitive; /* basic primitive type */ int flags; /* byte-order, hasobject */ int totsize; /* SEE BELOW */ int alignment; /* SEE BELOW */ PyObject *extended; /* SEE BELOW */ } PyDataFormatObject; enum Pysimpleformat {PY_BIT='1', PY_BOOL='?', PY_BYTE='b', PY_SHORT='h', PY_INT='i', PY_LONG='l', PY_LONGLONG='q', PY_UBYTE='B', PY_USHORT='H', PY_UINT='I', PY_ULONG='L', PY_ULONGLONG='Q', PY_FLOAT='f', PY_DOUBLE='d', PY_LONGDOUBLE='g', PY_CFLOAT='F', PY_CDOUBLE='D', PY_CLONGDOUBLE='G', PY_OBJECT='O', PY_CHAR='c', PY_UCS2='u', PY_UCS4='w', PY_POINTER='P', PY_ARRAYOF='A', PY_FUNCPTR='X', PY_STRUCTURE='T', PY_NOFORMAT=0}; flags -- flags for the data-format object. Specified masks are PY_NATIVEORDER 3 PY_BIGENDIAN 2 PY_LITTLEENDIAN 1 PY_IGNORE 0 PY_NEEDSREFCOUNT 4 copying this element requires reference counting. totsize -- the total size represented by this data-format in bytes unless the primitive is PY_BIT in which case it is the size in bits. For data-formats that are simply 1-d arrays of the underlying primitive, this total size can represent more than one primitive). This is a short-cut representation to having primitive be 'A' with extended fixed appropriately. alignment -- For the primitive types this is offsetof(struct {char c; type v;},v) extended -- NULL if this is a primitive data-type or no additional information is available. If primitive is PY_FUNCPTR, then this can be a list (of at least length 2) of data-format objects specifying the input argument formats with the last argument specifying the output argument data-format (use None for void inputs and/or outputs). If primitive is PY_POINTER then this must be a data-format object showing what the current data-format object is a pointer to. If primitive is PY_ARRAYOF then this must be a tuple (base-format, dim0, dim1, dim2, ...) so that this data-format represents a (contiguous) array of size (dim0, dim1, dim2) of base-format memory. Last index varies the fastest when walking through the contiguous memory sequentially. For other primitives (especially PY_STRUCTURE), this can be a tuple with 2 elements: (names, fields) names -- A list of string or unicode objects giving the names of the fields for a structured data-format. fields -- A Python dictionary with ordered-keys given by the list in names. Each entry in the dictionary is a 3-tuple containing (data-format-object, offset, meta-information) where meta-information is Py_None if there is no meta-information. Offset is given in bytes from the start of the record or in bits if PY_BIT is the primitive. The constructor of a PyArrViewObject allocates the memory for shape and strides and the destructor frees that memory. The constructor of a PyDataFormatObject allocates the objects it needs for fields, names, and shape. New Python Objects Equality-testing on DataFormat objects should return equal if they are equivalent (i.e. on 32-bit platforms PY_INT and PY_LONG are identical so the DataFormat Objects using those should also be identical. C-API void PyArrayView_UpdateFlags(PyObject *view, int flags) /* update the flags on the array view object provided */ PyObject * PyDataFormat_FromPrimitive(Pysimpleformat primitive) /* return a new primitive data-format object */ PyObject * PyDataFormat_FromCType(PyObject *ctype) /* return a new data-format object from a ctype */ int PyGet_PrimitiveSize(Pysimpleformat primitive) /* return the size (in bytes) of the provided primitive */ PyObject * PyDataFormat_Align(PyObject *format) /* take a data-format object and construct an aligned data-format object where all fields are aligned on appropriate boundaries for the compiler */ PyObject * PyArrayView_GetAsFormat(PyObject *obj, Pysimpleformat format, int requirements) /* Get a new view of obj ensuring it is of the requested format and satisfies the given requirements. Raise an error and return NULL on failure. If format is PY_NOFORMAT then no checking for format is done. Otherwise, a check for one of the simple primitive formats can be done. The requirements argument is an OR'd list of flags that should be tested for on the view. If PYARR_UPDATEIFCOPY is given, then (as long as the data-format is equivalent, a copy of the data is made and a view to that new memory is given. When the view is destroyed, then the data will be copied back into the memory it was taken from. */ Discussion The information provided in the array view object is patterned after the way a multi-dimensional array is defined in NumPy -- including the data-format object which allows a variety of descriptions of memory depending on the need. Reference Implementation Supplied when the PEP is accepted. Copyright This document is placed in the public domain.

1 0

Latest Array-Interface PEP
by Travis Oliphant Jan. 4, 2007

Jan. 4, 2007

I'm attaching my latest extended buffer-protocol PEP that is trying to get the array interface into Python. Basically, it is a translation of the numpy header files into something as simple as possible that can still be used to describe a complicated block of memory to another user. My purpose is to get feedback and criticisms from this community before display before the larger Python community. -Travis PEP: <unassigned> Title: Extending the buffer protocol to include the array interface Version: $Revision: $ Last-Modified: $Date: $ Author: Travis Oliphant <oliphant(a)ee.byu.edu> Status: Draft Type: Standards Track Created: 28-Aug-2006 Python-Version: 2.6 Abstract This PEP proposes extending the tp_as_buffer structure to include function pointers that incorporate information about the intended shape and data-format of the provided buffer. In essence this will place an array interface directly into Python. Rationale Several extensions to Python utilize the buffer protocol to share the location of a data-buffer that is really an N-dimensional array. However, there is no standard way to exchange the additional N-dimensional array information so that the data-buffer is interpreted correctly. The NumPy project introduced an array interface (http://numpy.scipy.org/array_interface.shtml) through a set of attributes on the object itself. While this approach works, it requires attribute lookups which can be expensive when sharing many small arrays. One of the key reasons that users often request to place something like NumPy into the standard library is so that it can be used as standard for other packages that deal with arrays. This PEP provides a mechanism for extending the buffer protocol (which already allows data sharing) to add the additional information needed to understand the data. This should be of benefit to all third-party modules that want to share memory through the buffer protocol such as GUI toolkits, PIL, PyGame, CVXOPT, PyVoxel, PyMedia, audio libraries, video libraries etc. Proposal Add bf_getarrview and bf_relarrview function pointers to the buffer protocol to allow objects to share a view on a memory pointer including information about accessing it as an N-dimensional array. Add the TP_HAS_ARRAY_BUFFER flag to types that define this extended buffer protocol. Also a few additionsl C-API calls should perhaps be added to Python to facilitate creating new PyArrViewObjects. Specification: static PyObject* bf_getarrayview (PyObject *obj) This function must return a new reference to a PyArrViewObject which contains the details of the array information exposed by the object. If failure occurs, then NULL is returned and an exception set. static int bf_relarrayview(PyObject *obj) If not NULL then this will be called when the object returned by bf_getarrview is destroyed so that the underlying object can be aware when acquired "views" are released. The object that defines bf_getarrview should not re-allocate memory (re-size itself) while views are extant. A 0 is returned on success and a -1 and an error condition set on failure. The PyArrayViewObject has the structure typedef struct { PyObject_HEAD void *data; /* pointer to the beginning of data */ int nd; /* the number of dimensions */ Py_ssize_t *shape; /* c-array of size nd giving shape */ Py_ssize_t *strides; /* SEE BELOW */ PyObject *base; /* the object this is a "view" of */ PyObject *format; /* SEE BELOW */ int flags; /* SEE BELOW */ } PyArrayViewObject; strides -- a c-array of size nd providing the striding information which is the number of bytes to skip to get to the next element in that dimension. format -- a Python data-format object (PyDataFormatObject) which contains information about how each item in the array should be interpreted. flags -- an integer of flags. PYARR_WRITEABLE is the only flag that must be set appropriately by types. Other flags: PYARR_ALIGNED, PYARR_C_CONTIGUOUS, PYARR_F_CONTIGUOUS, and PYARR_NOTSWAPPED can all be determined from the rest of the PyArrayViewObject using the UpdateFlags C-API. The PyDataFormatObject has the structure typedef struct { PyObject_HEAD PySimpleformat primitive; /* basic primitive type */ int flags; /* byte-order, isaligned */ int itemsize; /* SEE BELOW */ int alignment; /* SEE BELOW */ PyObject *extended; /* SEE BELOW */ } PyDataFormatObject; enum Pysimpleformat {PY_BIT='1', PY_BOOL='?', PY_BYTE='b', PY_SHORT='h', PY_INT='i', PY_LONG='l', PY_LONGLONG='q', PY_UBYTE='B', PY_USHORT='H', PY_UINT='I', PY_ULONG='L', PY_ULONGLONG='Q', PY_FLOAT='f', PY_DOUBLE='d', PY_LONGDOUBLE='g', PY_CFLOAT='F', PY_CDOUBLE='D', PY_CLONGDOUBLE='G', PY_OBJECT='O', PY_CHAR='c', PY_UCS2='u', PY_UCS4='w', PY_FUNCPTR='X', PY_VOIDPTR='V'}; Each of these simple formats has a special character code which can be used to identify this primitive in a nested python list. flags -- flags for the data-format object. Specified masks are PY_NATIVEORDER PY_BIGENDIAN PY_LITTLEENDIAN PY_IGNORE itemsize -- the total size represented by this data-format in bytes unless the primitive is PY_BIT in which case it is the size in bits. For data-formats that are simple 1-d arrays of the underlying primitive, this total size can represent more than one primitive (with extended still NULL). alignment -- For the primitive types this is offsetof(struct {char c; type v;},v) extended -- NULL if this is a primitive data-type or no additional information is available. If primitive is PY_FUNCPTR, then this can be a tuple with >=1 element: (args, {dim0, dim1, dim2, ...}). args -- A list (of at least length 2) of data-format objects specifying the input argument formats with the last argument specifying the output argument data-format (use None for void inputs and/or outputs). For other primitives, this can be a tuple with >=2 elements: (names, fields, {dim0, dim1, dim2, ...}) Use None for both names and fields if they should be ignored. names -- An ordered list of string or unicode objects giving the names of the fields for a structure data-format. fields -- a Python dictionary with ordered-keys given by the list in names. Each entry in the dictionary is a 3-tuple containing (data-format-object, offset, meta-information) where meta-information is Py_None if there is no meta-information. Offset is given in bytes from the start of the record or in bits if PY_BIT is the primitive. Any additional entries in the extended tuple (dim0, dim1, etc.) are interpreted as integers which specify that this data-format is an array of the given shape of the fundamental data-format specified by the remainder of the DataFormat Object. The dimensions are specified so that the last-index is always assumed to vary the fastest (C-order). The constructor of a PyArrViewObject allocates the memory for shape and strides and the destructor frees that memory. The constructor of a PyDataFormatObject allocates the objects it needs for fields, names, and shape. C-API void PyArrayView_UpdateFlags(PyObject *view, int flags) /* update the flags on the array view object provided */ PyDataFormatObject *Py_NewSimpleFormat(Pysimpleformat primitive) /* return a new primitive data-format object */ PyDataFormatObject *Py_DataFormatFromCType(PyObject *ctype) /* return a new data-format object from a ctype */ int Py_GetPrimitiveSize(Pysimpleformat primitive) /* return the size (in bytes) of the provided primitive */ PyDataFormatObject *Py_AlignDataFormat(PyObject *format) /* take a data-format object and construct an aligned data-format object where all fields are aligned on appropriate boundaries for the compiler */ Discussion The information provided in the array view object is patterned after the way a multi-dimensional array is defined in NumPy -- including the data-format object which allows a variety of descriptions of memory depending on the need. Reference Implementation Supplied when the PEP is accepted. Copyright This document is placed in the public domain.

1 0

ndarray newbie question
by belinda thom Jan. 4, 2007

Jan. 4, 2007

Hello, I wrote a "display obj" method for viewing instance data: def dobj(obj) : """extended object viewer that displays arg""" print 'Class: ', obj.__class__.__name__, ' id:', id(obj) for k in obj.__dict__.keys() : print ' ', k, ': ', obj.__dict__[k], ' id:', id(obj.__dict__ [k]) and I'm wondering why it doesn't work on ndarrays: >>> m = numpy.zeros(3) >>> dobj(m) complains because: AttributeError: 'numpy.ndarray' object has no attribute '__dict__' I am also fairly new to python, but would have thought all objects had dictionaries, especially when they are comprised of other things, e.g. dir(m) produces a slew of components, e.g. tofile, ..., transpose, ..., view, etc. Thanks for any insight. --b

4 5

OpenSuse 10.2 and numpy
by Nils Wagner Jan. 4, 2007

Jan. 4, 2007

Hi all, I have tried to build the latest numpy on a 64-bit machine using OpenSuSE10.2. I have build BLAS, LAPACK and ATLAS from scratch (g77). python setup.py build yields ... creating build/temp.linux-x86_64-2.5/numpy/linalg compile options: '-DATLAS_INFO="\"3.7.11\"" -Inumpy/core/include -Ibuild/src.linux-x86_64-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: numpy/linalg/lapack_litemodule.c /usr/bin/g77 -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o -L/usr/local/lib/atlas -L/usr/lib/python2.5/config -llapack -lf77blas -lcblas -latlas -lpython2.5 -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: cannot find -lgcc_s /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: cannot find -lgcc_s error: Command "/usr/bin/g77 -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o -L/usr/local/lib/atlas -L/usr/lib/python2.5/config -llapack -lf77blas -lcblas -latlas -lpython2.5 -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so" failed with exit status 1 Any pointer how to fix this problem is appreciated. Nils g77 -v Reading specs from /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/specs Configured with: ../configure --enable-threads=posix --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --enable-languages=c,f77 --disable-checking --libdir=/usr/lib64 --disable-libgcj --with-slibdir=/lib64 --with-system-zlib --enable-shared --enable-__cxa_atexit x86_64-suse-linux Thread model: posix gcc version 3.3.5 20050117 (prerelease) (SUSE Linux) nwagner@noname:~/svn/numpy> gcc -v Using built-in specs. Target: x86_64-suse-linux Configured with: ../configure --enable-threads=posix --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.1.2 --enable-ssp --disable-libssp --disable-libgcj --with-slibdir=/lib64 --with-system-zlib --enable-shared --enable-__cxa_atexit --enable-libstdcxx-allocator=new --program-suffix=-4.1 --enable-version-specific-runtime-libs --without-system-libunwind --with-cpu=generic --host=x86_64-suse-linux Thread model: posix gcc version 4.1.2 20061115 (prerelease) (SUSE Linux)

3 6

Scipy - Numpy incompatiblility when calling upon Old Numeric
by frbeaxs Jan. 3, 2007

Jan. 3, 2007

I am using Python 2.4 with Numpy 0.9.8. Matpylib graphs function under these two version except when old numeric must be called to utilized the spline function in contouring the graphs leading to the error: Traceback (most recent call last): File "C:\File\XXX", line 3, in ? from scipy.sandbox.delaunay import * File "C:\Python24\lib\site-packages\scipy\__init__.py", line 32, in ? from numpy import oldnumeric File "C:\Python24\lib\site-packages\numpy\oldnumeric\__init__.py", line 3, in ? from compat import * File "C:\Python24\lib\site-packages\numpy\oldnumeric\compat.py", line 133, in ? from numpy import deprecate ImportError: cannot import name deprecate Scipy 0.5.0 seems to be compatible only with Numpy 101b, not version 0.9.8 and I have to switch version of Numpy to should I need to utilize Scipy, however certain matpylib graphs will not function. Upgrading Scipy to 0.5.2 and Numpy 101 has no negative effects on the programs I wrote or the matplotlib sample programs, however the following error message replaces the Import error above Matrix = matrix NameError:name matrix is not defined. Downgrading NUMPY to ver 9.6 produces even more problems. The best match seems to be Mumpy 0.9.8 used with SciPy 0.50 seems to be the best combination however it prohibits calling upon Oldnemeric which seems to be necessary to use the spline function. Does anyone have any idea on how to get around this? frbeaxs

2 1