Mailman 3 March 2005 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon 17 May '11

17 May '11

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

ANN: numarray-1.2.3
by Todd Miller 10 May '05

10 May '05

numarray-1.2.3 is a bugfix release for numarray-1.2.2 which fixes a problem with universal function setup caching which noticeably impaired 1.2.2 small array performance. Get it if you are new to numarray, haven't upgraded to 1.2.2 yet, or use a lot of small arrays. numarray-1.2.3 is here: http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=32367 Thanks to Ralf Juengling for quietly reporting this and working with me to identify and fix the problem.

2 12

Array Metadata
by Scott Gilbert 06 Apr '05

06 Apr '05

I got back late last night, and there were lots of things I wanted to comment on. I've put parts of several threads into this one message since they're all dealing with the same general topic: Perry Greenfield wrote: > > I'm not sure how the support for large data sets should be handled. > I generally think that it will be very awkward to handle these > until Python does as well. Speaking of which... > I agree that it's going to be difficult to have general support for large PyBufferProcs objects until the Python core is made 64 bit clean. But specific support can be added for buffer types that are known in advance. For instance, the bytes object PEP proposes an alternate way to get a 64 bit length, and similar support could easily be added to Numarray.memory, mmap.mmap, and whatever else on a case by case basis. So you could get a 64 bit pointer from some types of buffers before the rest of Python becomes 64 bit clean. If the ndarray consumer (wxWindows for instance) doesn't recognize the particular implementation, it has to stick with the limitations of the standard PyBufferProcs and assume a 32 bit length would suffice. Travis Oliphant wrote: > > I prefer __array_data__ (it's a common name for Numeric and > numarray, It can be interpreted as a sequence object if desired). > So long as everyone agrees it doesn't matter what name it is. Sounds like __array_data__ works for everyone. > > I also like __array_typestr__ or __array_typechar__ better as a name. > A name is a name as far as I'm concerned. The name __array_typestr__ works for me. The name __array_typechar__ implies a single character, and that won't be true. > > Don't like 'D' for long double. Complex floats is already > using it. I'm not sure I like the idea of moving to two > character typecodes at this point because it indicates more > internal changes to Numeric3 (otherwise we have two typecharacter > standards which is not a good thing). What is wrong with 'g' > and 'G' for long double and complex long double respectively. > Nothing in this array protocol should *require* internal changes to either Numeric3 or Numarray. I suspect Numarray is going to keep it's type hierarchy, and Numeric3 can use single character codes for it's representation if it wants. However, both Numeric3 and Numarray might (probably would) have to translate their internal array type specifiers into the agreed upon "type code string" when reporting out this attribute. The important qualities __array_typestr__ should have are: 1) Everyone should agree on the interpretation. It needs to be documented somewhere. Third party libraries should get the same __array_typestr__ from Numarray as they do from Numeric3. 2) It should be sufficiently general in it's capabilities to describe a wide category of array types. Simple things should be simple, and harder things should be possible. An ndarray of double should have a simple common well recognized value for __array_typestr__. An ndarray of a multi-field structs should be representable too. > > > > > __array_complicated__ > > > > I don't see the utility here I guess, If it can't be described by a > shape/strides combination then how can it participate in the protocol? > I'm not married to this one. I don't know if Numarray or Numeric3 will ever do such a thing, but I can imagine more complicated schemes of arranging the data than offset/shape/strides are capable of representing. So this is forward compatibility with "Numarric4" :-). Pretty hypothetical, but imagine that typically Numarric4 can represent it's data with offset/shape/strides, but for more advanced operations that falls apart. I could bore you with a detailed example... The idea is that if array consumers like wxPython were aware that more complicated implementations can occur in the future, they can gracefully bow out and raise an exception instead of incorrectly interpreting the data. If you need it later, you can't easily add it after the fact. Take it or leave it I guess - it's possibly a YAGNI. > > After more thought, I think here we need to also allow the > "c-type" independent way of describing an array (i.e. numarray > introduced 'c4' for a complex-valued 4 byte itemsize array). > So, perhaps __array_ctypestr_ and __array_typestr__ should be > two ways to get the information (or overload the __array_typestr__ > interface and reequire consumers to accept either style). > I don't understand what you are proposing here. Why would you want to represent the same information two different ways? Perry Greenfield wrote: > > I think we need to think about what the typecharacter is supposed > to represent. Is it the value as the user will see it or to indicate > what the internal representation is? These are two different things. > I think __array_typestr__ should accurately represent the internal representation. It is not intended for typical end users. The whole of the __array_*metadata*__ stuff is intended for third party libraries like wxPython or PIL to be able to grab a pointer to the data, calculate offsets, and cast it to the appropriate type without writing lots of special case code to handle the differences between Numeric, Numarray, Numeric3, and whatever else. > > Then again, I'm not sure how this info is exposed to the user; if it > is appropriately handled by intermediate code it may not matter. For > example, if this corresponds to what the user will see for the type, > I think it is bad. Most of the time they don't care what the internal > representation is, they just want to know if it is Int16 or whatever; > with the two combined, they have to test for both variants. > Typical users would call whatever attribute or method you prefer (.type() or .typecode() for instance), and the type representation could be classes or typecodes or whatever you think is best. The __array_typestr__ attribute is not for typical users (unless they start to care about the details under the hood). It's for libraries that need to know what's going on in a generic fashion. You don't have to store this attribute as separate data, it can be a property style attribute that calculates it's value dynamically from your own internal representation. Francesc Altet wrote: > > Considering that heterogenous data is to be suported as well, and > there is some tradition of assigning names to the different fields, > I wonder if it would not be good to add something like: > > __array_names__ (optional comma-separated names for record fields) > I really like this idea. Although I agree with David M. Cooke that it should be a tuple of names. Unless there is a use case I'm not considering, it would be preferrable if the names were restricted to valid Python identifiers. Travis Oliphant wrote: > > After more thought, I think using the struct-like typecharacters > is not a good idea for the array protocol. I think that the > character codes used by the numarray record array: kind_character > + byte_width is better. Commas can separate heterogeneous data. > The problem is that if the data buffer originally came from a > different machine or saved with a different compiler (e.g. a mmap'ed > file), then the struct-like typecodes only tell you the c-type that > machine thought the data was. It does not tell you how to interpret > the data on this machine. > The struct module has a portable set of typecodes. They call it "standard", but it's the same thing. The struct module let's you specify either standard or native. For instance, the typecode for "standard long" ("=l") is always 4 bytes while a "native long" ("@l") is likely to be 4 or 8 bytes depending on the platform. The __array_typestr__ codes should require the "standard" sizes. There is a table at the bottom of the documentation that goes into detail: http://docs.python.org/lib/module-struct.html The only problem with the struct module is that it's missing a few types... (long double, PyObject, unicode, bit). > > I also think that rather than attach < or > to the start of the > string it would be easier to have another protocol for endianness. > Perhaps something like: > > __array_endian__ (optional Python integer with the value 1 in it). > If it is not 1, then a byteswap must be necessary. > This has the problem you were just describing. Specifying "byteswapped" like this only tells you if the data was reversed on the machine it came from. It doesn't tell you what is correct for the current machine. Assuming you represented little endian as 0 and big endian as 1, you could always figure out whether to byteswap like this: byteswap = data_endian ^ host_endian Do you want to have an __array_endian__ where 0 indicates "little endian", 1 indicates "big endian", and the default is whatever the current host machine uses? I think this would work for a lot of cases. A limitation of this approach is that it can't adequately represent struct/record arrays where some fields are big endian and others are little endian. > > Bool -- "b%d" % sizeof(bool) > Signed Integer -- "i%d" % sizeof(<some int>) > Unsigned Integer -- "u%d" % sizeof(<some uint>) > Float -- "f%d" % sizeof(<some float>) > Complex -- "c%d" % sizeof(<some complex>) > Object -- "O%d" % sizeof(PyObject *) > --- this would only be useful on shared memory > String -- "S%d" % itemsize > Unicode -- "U%d" % itemsize > Void -- "V%d" % itemsize > The above is a nice start at reinventing the struct module typecodes. If you and Perry agree to it, that would be great. A few additions though: I think you're proposing that "struct" or "record" arrays would be a concatenation of the above strings. If so, you'll need an indicator for padding bytes. (You probably know this, but structs in C frequently have wasted bytes inserted by the compiler to make sure data is aligned on the machine addressable boundaries.) I also assume that you intend the ("c%d" % itemsize) to always represent complex floating point numbers. That leaves my favorite example of complex short integer data with no way to be represented... I guess I could get by with "i2i2". How about not having a complex type explicitly, but representing complex data as something like: __array_typestr__ = "f4f4 __array_names__ = ("real", "imag") Just a thought... I do like it though. I think that both Numarray and Numeric3 are planning on storing booleans in a full byte. A typecode for tightly packed bits wouldn't go unused however... > > 1) How do you support > 2Gb memory mapped arrays on 32 bit systems > and other large-object arrays only a part of which are in memory at > any given time > Doing this well is a lot like implementing mmap in user space. I think this is a modification to the buffer protocol, not the array protocol. It would add a bit of complexity if you want to deal with it, but it is doable. Instead of just grabbing a pointer to the whole thing, you need to ask the object to "page in" ranges of the data and give you a pointer that is only valid in that range. Then when you're done with the pointer, you need to explicitly tell the object so that it can write back if necessary and release the memory for other requests. Do you think Numeric3 or Numarray would support this? I think it would be very cool functionality to have. > > (there is an equivalent problem for > 8 Eb (exabytes) on 64 bit > systems, an Exabyte is 2^60 bytes or a giga-giga-byte). > I think it will be at least 10-20 years before we could realisticly exceed a 64 bit address space. Probably a lot longer. That's a billion times more RAM than any machine I've ever worked on, and it's a million times more bytes than any RAID set I've worked with. Are there any super computers approaching this level? Even at Moore's law rates, I'm not worried about that one just yet. > > But, I've been thinking about the array protocol and thinking that > it would be a good thing if this became universal. One of the ways > to make it universal is by having something that follows it in the > Python core. > > So, what if we proposed for the Python core not something like > Numeric3 (which would still exist in scipy.base and be everybody's > favorite array :-) ), but a very minimal array object (scaled back > even from Numeric) that followed the array protocol and had some > C-API associated with it. > > This minimal array object would support 5 basic types ('bool', > 'integer', 'float', 'complex', 'Object'). (Maybe a void type > could be defined and a void "scalar" introduced (which would be > the bytes object)). These types correspond to scalars already > available in Python and so the whole 0-dim array Python scalar > arguments could be ignored. > I really like this idea. It could easily be implemented in C or Python script. Since half it's purpose is for documentation, the Python script implementation might make more sense. Additionally, a module that understood the defaults and did the right thing with the metadata attributes would be useful: def get_ndims(a): return len(a.__array_shape__) def get_offset(a): if hasattr(a, "__array_offset__"): return a.__array_offset__ return 0 def get_strides(a): if hasattr(a, "__array_strides__"): return a.array_strides # build the default strides from the shape def is_c_contiguous(a): shape = a.__array_shape__ strides = get_strides(a) # determine if the strides indicate it is contiguous def is_fortran_contiguous(a): # similar to is_c_contiguous etc... Thes functions could be useful for third party libraries to work with *any* of the array packages. > > An alternative would be to "add" multidimensionality to the array > object already part of Python, fix it's reallocating with an exposed > buffer problem, and add the array protocol. > I'd recommend not breaking backward compatibility on the array.array object, but adding the __array_*metadata*__ attributes wouldn't hurt anything. (The __array_shape__ would always be a tuple of length one, but that's allowed...). Magnus Lie Hetland wrote: > > Wohoo! Niiice :) > > (Okay, a bit "me too"-ish, but I just wanted to contribute some > enthusiasm ;) > I completely agree! :-) Cheers, -Scott

3 2

Bytes Object and Metadata
by Scott Gilbert 03 Apr '05

03 Apr '05

Adding metadata at the buffer object level causes problems for "view" semantics. Let's say that everyone agreed what "itemsize" and "itemtype" meant: real_view = complex_array.real The real_view will have to use a new buffer since they can't share the old one. The buffer used in complex_array would have a typecode like ComplexDouble and an itemsize of 16. The buffer in real_view would need a typecode of Double and an itemsize of 8. If metadata is stored with the buffer object, it can't be the same buffer object in both places. Another case would be treating a 512x512 image of 4 byte pixels as a 512x512x4 image of 1 byte RGBA elements. Or even coercing from Signed to Unsigned. The bytes object as proposed does allow new views to be created from other bytes objects (sharing the same memory underneath), and these views could each have separate metadata, but then you wouldn't be able to have arrays that used other types of buffers. Having arrays use mmap buffers is very useful. The bytes object shouldn't create views from arbitrary other buffer objects because it can't rely on the general semantics of the PyBufferProcs interface. The foreign buffer object might realloc and invalidate the pointer for instance... The current Python "buffer" builtin does this, and the results are bad. So creating a bytes object as a view on the mmap object doesn't work in the general case. Actually, now that I think about it, the mmap object might be safe. I don't believe the current implementation of mmap does any reallocing under the scenes and I think the pointer stays valid for the lifetime of the object. If we verified that mmap is safe enough, bytes could make a special case out of it, but then you would be locked into bytes and mmap only. Maybe that's acceptable... Still, I think keeping the metadata at a different level, and having the bytes object just be the Python way to spell a call to C's malloc will avoid a lot of problems. Read below for how I think the metadata stuff could be handled. --- Chris Barker <Chris.Barker(a)noaa.gov> wrote: > > There are any number of Third party extensions that could benefit from > being able to directly read the data in Numeric* arrays: PIL, wxPython, > etc. Etc. My personal example is wxPython: > > At the moment, you can pass a Numeric or numarray array into wxPython, > and it will be converted to a wxList of wxPoints (for instance), but > that is done by using the generic sequence protocol, and a lot of type > checking. As you can imagine, that is pretty darn slow, compared to just > typecasting the data pointer and looping through it. Robin Dunn, quite > reasonably, doesn't want wxPython to depend on Numeric, so that's what > we've got. > > My understanding of this memory object is that an extension like > wxPython wouldn't not need to know about Numeric, but could simply get > the memory Object, and there would be enough meta-data with it to > typecast and loop through the data. I'm a bit skeptical about how this > would work. It seems that the metadata required would be the full set of > stuff in an array Object already: > > type > dimensions > strides > > This could be made a bit simpler by allowing only contiguous arrays, but > then there would need to be a contiguous flag. > > To make use of this, wxPython would have to know a fair bit about > Numeric Arrays anyway, so that it can check to see if the data is > appropriate. I guess the advantage is that while the wxPython code would > have to know about Numeric arrays, it wouldn't have to include Numeric > headers or code. > I think being able to traffic in N-Dimensional arrays without requiring linking against the libraries is a good thing. Several years ago, I proposed a solution to this problem. Actually I did a really poor job of proposing it and irritated a lot of people in the process. I'm embarrassed to post a link to the following thread, but here it is anyway: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/1166013 Accept my appologies if you read the whole thing just now. :-) Accept my sincere appologies if you read it at the time. I think the proposal is still relevant today, but I might revise it a bit as follows. A bear minimum N-Dimensional array for interchanging data across libraries could get by with following attributes: # Create a simple record type for storing attributes class BearMin: pass bm = BearMin() # Set the attributes sufficient to describe a simple ndarray bm.buffer = <a buffer or sequence object> bm.shape = <a tuple of ints describing it's shape> bm.itemtype = <a string describing the elements> The bm.buffer and bm.shape attributes are pretty obvious. I would suggest that the bm.itemtype borrow it's typecodes from the Python struct module, but anything that everyone agreed on would work. (The struct module is nice because it is already documented and supports native and portable types of many sizes in both endians. It also supports composite struct types.) Those attributes are sufficient for someone to *produce* an N-Dimensional array that could be understood by many libraries. Someone who *consumes* the data would need to know a few more: bm.offset = <an integer offset into the buffer> bm.strides = <a tuple of ints for non-contiguous or Fortran arrays> The value of bm.offset would default to zero if it wasn't present, and the the tuple bm.strides could be generated from the shape assuming it was a C style array. Subscripting operations that returned non-contiguous views of shared data could change bm.offset to non-zero. Subscripting would also affect the bm.strides, and creating a Fortran style array would require bm.strides to be present. You might also choose to add bm.itemsize in addition to bm.itemtype when you can describe how big elements are, but you can't sufficiently describe what the data is using the agreed upon typecodes. This would be uncommon. The default for bm.itemsize would come from struct.calcsize(bm.itemtype). You might also choose to add bm.complicated for when the array layout can't be described by the shape/offset/stride combination. For instance bm.complicated might get used when creating views from more sophisticated subscripting operations like index arrays or mask arrays. Although it looks like Numeric3 plans on making new contiguous copies in those cases. The C implementations of arrays would only have to add getattr like methods, and the data could be stored very compactly. >From those minimum 5-7 attributes (metadata), an N-Dimensional array consumer could determine most everything it needed to know about the data. Simple routines could determine things like iscontiguous(bm), iscarray(bm) or isfortran(bm). I expect libraries like wxPython or PIL could punt (raise an exception) when the water gets too deep. It also doesn't prohibit other attributes from being added. Just because an N-Dimensional array described it's itemtype using the struct module typecodes doesn't mean that it couldn't implement more sophisticated typing hierarchies with a different attribute. There are a few commonly used types like "long double" which are not supported by the struct module, but this could be addressed with a little discussion. Also you might want a "bit" or "Object" typecode for tightly packed mask arrays and Object arrays. The names could be argued about, and something like: bm.__array_buffer__ bm.__array_shape__ bm.__array_itemtype__ bm.__array_offset__ bm.__array_strides__ bm.__array_itemsize__ bm.__array_complicated__ would really bring home the notion that the attributes are a description of what it means to participate in an N-Dimensional array protocol. Plus names this long and ugly are unlikely to step on the existing attributes already in use by Numeric3 and Numarray. :-) Anyway, I proposed this a long time ago, but the belief was that one of the standard array packages would make it into the core very soon. With a standard array library in the core, there wouldn't be as much need for general interoperability like this. Everyone could just use the standard. Maybe that position would change now that Numeric3 and Numarray both look to have long futures. Even if one package made it in, the other is likely to live on. I personally think the competition is a good thing. We don't need to have only one array package to get interoperability. I would definitely like to see the Python core acquire a full fledged array package like Numeric3 or Numarray. When I log onto a new Linux or MacOS machine, the array package would just be there. No installs, no hassle. But I still think a simple community agreed upon set of attributes like this would be a good idea. --- Peter Verveer <verveer(a)embl.de> wrote: > > It think it would be a real shame not to support non-contiguous data. > It would be great if such a byte object could be used instead of > Numeric/numarray arrays when writing extensions. Then I could write C > extensions that could be made available very easily/efficiently to any > package supporting it without having to worry about the specific C api > of those packages. If only contiguous byte objects are supported that > byte object is not a good option anymore for implementing extensions > for Numeric unless I am prepared to live with a lot of copying of > non-contiguous arrays. > I'm hoping I made a good case for a slightly different strategy above. But even if the metadata did go into the bytes object itself, the metadata could describe a non-contiguous layout on top of the contiguous chunk of memory. There is another really valid argument for using the strategy above to describe metadata instead of wedging it into the bytes object: The Numeric community could agree on the metadata attributes and start using it *today*. If you wait until someone commits the bytes object into the core, it won't be generally available until Python version 2.5 at the earliest, and any libraries that depended on using bytes stored metadata would not work with older versions of Python. Cheers, -Scott

7 24

Thoughts on getting "something" in the Python core
by Travis Oliphant 01 Apr '05

01 Apr '05

To all interested in the future of arrays... I'm still very committed to Numeric3 as I want to bring the numarray and Numeric people together behind a single array object for scientific computing. But, I've been thinking about the array protocol and thinking that it would be a good thing if this became universal. One of the ways to make it universal is by having something that follows it in the Python core. So, what if we proposed for the Python core not something like Numeric3 (which would still exist in scipy.base and be everybody's favorite array :-) ), but a very minimal array object (scaled back even from Numeric) that followed the array protocol and had some C-API associated with it. This minimal array object would support 5 basic types ('bool', 'integer', 'float', 'complex', 'Object'). (Maybe a void type could be defined and a void "scalar" introduced (which would be the bytes object)). These types correspond to scalars already available in Python and so the whole 0-dim array Python scalar arguments could be ignored. Math could be done without ufuncs initially (people really needing speed would use scipy.base anyway). But, more people in the Python community would be able to use arrays and get used to them. And we would have a reference array_protocol object so that extension writers could write to it. I would not try a project like this until after scipy_core is out, but it's an interesting thing to think about. I mainly wanted feedback on the basic concept. An alternative would be to "add" multidimensionality to the array object already part of Python, fix it's reallocating with an exposed buffer problem, and add the array protocol. -Travis

4 4

Pickle complete (new ideas for Python arrays)
by Travis Oliphant 30 Mar '05

30 Mar '05

Hi all, Pickling is now implemented for scipy.base (was calling it Numeric3) Anybody wanting to tackle a function to read old Numeric and/or numarray pickles is welcome. I think this could be all in Python. Ideally, we should be able to read these pickles without having those packages installed. I think the PEP for Python should be converted to a bare-bones protocol (e.g. the one that is emerging) Optionally we could create a very simple default arrayobject for Python that just has a default pickle implementation and knows how to get data through the buffer interface from other objects). That way any array implementation just has to talk the Python array protocol to be interoperable with any other array implementation. -Travis

1 0

Linear programming
by Magnus Lie Hetland 29 Mar '05

29 Mar '05

Is there some standard Python (i.e., numarray/Numeric) mapping for some linear programming package out there? Might be rather useful... -- Magnus Lie Hetland Fall seven times, stand up eight http://hetland.org [Japanese proverb]

4 4

searching a list of arrays
by Darren Dale 29 Mar '05

29 Mar '05

Hi, I have a list of numeric-23.8 arrays: a = [array([0,1]), array([0,1]), array([1,0]), array([1,0])] b = [array([0,1,0]), array([0,1,0]), array([1,0,0]), array([1,0,0])] and I want to make a new list out of b: c = [array([0,1,2]), array([1,0,2])] where the last index in each array is the result of b.count([0,1,0]) # or [1,0,0] The problem is that the result of b.count(array([1,0,0])) is 4, not 2, and b.remove(array([1,0,0])) indescriminantly removes arrays from the list. a.count and a.remove work the way I expected. Does anyone know why 1x2 arrays work, but 1x3 or larger arrays do not? Thanks, Darren

3 2

zeros() default type in Numeric3
by Stephen Walton 29 Mar '05

29 Mar '05

zeros() in Numeric3 defaults to typecode='d' while in numarray it defaults to typecode=None, which in practice means 'i' by default. Is this deliberate? Is this desirable? I'd vote for zeros(), ones() and the like to default to 'i' or 'f' rather than 'd' in the interest of space and speed.

6 10

Trying out Numeric3
by Michiel Jan Laurens de Hoon 28 Mar '05

28 Mar '05

First of all, thanks to the Numerical Python developers for releasing version 23.8 of Numerical Python. It compiles out of the box and avoids the blas/lapack compilation problems in earlier versions, which makes my life as a developer a lot easier. Thanks! Travis Oliphant wrote: > I wanted to let people who may be waiting, that now is a good time to > help with numeric3. The CVS version builds (although I"m sure there are > still bugs), but more eyes could help me track them down. > > Currently, all that remains for the arrayobject is to implement the > newly defined methods (really it"s just a re-organization and > re-inspection of the code in multiarraymodule.c to call it using methods). > I downloaded Numeric3 today and installed it. The compilation and installation run fine. There are still some warnings from the compiler here and there, but I guess they will be fixed some other time. During compilation, I noticed that some test program is run, presumably for configuration. The test program is compiled by a different compiler as the one used in the build process. Note that "python setup.py config" is available in the standard distutils, so it may be better to use that instead of a self-defined configuration tool. For one thing, it'll make sure that the compiler used for configuration is the same as the one used for compilation. To use Numeric3, I did "from ndarray import *". I guess for the final version, this will be "from Numeric import *"? When using ndarray, I got a core dump using "zeros": $ python Python 2.5a0 (#1, Mar 2 2005, 12:15:06) [GCC 3.3.3] on cygwin Type "help", "copyright", "credits" or "license" for more information. >>> from ndarray import * >>> zeros(5) creating data 0xa0c03d0 associated with 0xa0d52c0 array([0.0, 0.0, 0.0, 0.0, 0.0], 'd') Segmentation fault (core dumped) With Python 2.4, the segmentation fault occurs slightly later: $ python2.4 Python 2.4 (#1, Dec 5 2004, 20:47:03) [GCC 3.3.3] on cygwin Type "help", "copyright", "credits" or "license" for more information. >>> from ndarray import * >>> zeros(5) creating data 0xa0a07f8 associated with 0xa0d6230 array([0.0, 0.0, 0.0, 0.0, 0.0], 'd') >>> >>> ^D freeing 0xa0a07f8 associated with array 0xa0d6230 freeing 0xa123b88 associated with array 0xa0d6230 Segmentation fault (core dumped) Finally, I tried to compile a C extension module that uses Numerical Python (by replacing #include <Numeric/arrayobject.h> by #include <ndarray/arrayobject.h>): $ python setup.py build running build running build_py running build_ext building 'Pycluster.cluster' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -Isrc -Iranli b/src -I/usr/local/include/python2.5 -c python/clustermodule.c -o build/temp.cyg win-1.5.12-i686-2.5/python/clustermodule.o In file included from python/clustermodule.c:2: /usr/local/include/python2.5/ndarray/arrayobject.h:76: warning: redefinition of `ushort' /usr/include/sys/types.h:85: warning: `ushort' previously declared here /usr/local/include/python2.5/ndarray/arrayobject.h:77: warning: redefinition of `uint' /usr/include/sys/types.h:86: warning: `uint' previously declared here These two warnings are probably not so serious, but it would be better to get rid of them anyway. python/clustermodule.c: In function `parse_data': python/clustermodule.c:38: warning: passing arg 1 of pointer to function from in compatible pointer type The offending line 38 is: { PyArrayObject* av = (PyArrayObject*) PyArray_Cast(*array, PyArray_DOUBLE); where array is a PyArrayObject**. Another warning was that PyArrayObject's "dimensions" doesn't seem to be an int array any more. Finally, when linking I get an undefined reference to _PyArray_API. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

7 25