Mailman 3 January 2003 - NumPy-Discussion

[Fwd: Re: [Numpy-discussion] typecodes in numarray]
by Todd Miller 24 Jan '03

24 Jan '03

1 0

typecodes in numarray
by Francesc Alted 24 Jan '03

24 Jan '03

Maybe I'm becoming a bit tedious with this, but if you look at: >>> import numerictypes >>> numerictypes.typecode {Complex64: 'D', Int32: 'l', UInt16: 's', Complex32: 'F', Float64: 'd', UInt8: 'b', Int16: 's', Float32: 'f', Int8: '1'} you can find some incongruencies that lead to weird things like: >>> array([1,2], Int16).typecode() 's' >>> array([1,2], UInt16).typecode() 's' # --> same as Int16! >>> array([1,2], Int64).typecode() Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.2/site-packages/numarray/numarray.py", line 730, in typecode return numerictypes.typecode[self._type] KeyError: numarray type: Int64 >>> array([1,2], UInt64).typecode() Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.2/site-packages/numarray/numarray.py", line 730, in typecode return numerictypes.typecode[self._type] KeyError: numarray type: UInt64 Also, 'l' is used here to map Int32, while in recarray is used to map Boolean. Moreover, Numeric 22.0 introduced the equivalent of UInt16 and UInt32 types as 'w' and 'u' respectively. But, again, 'u' is used in recarray as synonym of Uint8. I think it's important to agree with a definitive set of charcodes and use them uniformly throughout numarray. Suggestion: if recarray charcodes are not necessary to match the Numeric ones, I propose that using the Python convention maybe a good idea. Look at the table in: http://www.python.org/doc/current/lib/module-struct.html. -- Francesc Alted

2 1

Two bugs in numerictypes.py
by Edward C. Jones 23 Jan '03

23 Jan '03

In "numerictypes.py' it says "This module is designed so 'from numerictypes import *' is safe." When I "import numerictypes", and run dir(), I get: ['Any', 'AnyType', 'Bool', 'BooleanType', 'Byte', 'Complex', 'Complex32', 'Complex64', 'ComplexType', 'Double', 'Float', 'Float32', 'Float64', 'FloatingType', 'Int', 'Int16', 'Int32', 'Int64', 'Int8', 'IntegralType', 'Long', 'MAX_ALIGN', 'NumericType', 'Short', 'SignedIntegralType', 'SignedType', 'UInt16', 'UInt32', 'UInt64', 'UInt8', 'UnsignedIntegralType', 'UnsignedType', '__builtins__', '__doc__', '__name__', 'genericCoercions', 'genericPromotionExclusions', 'genericTypeRank', 'inttype1', 'inttype2', 'kind', 'mapto', 'maptype1', 'maptype2', 'nt1', 'nt2', 'ntypesize1', 'ntypesize2', 'numinclude', 'outtype', 'pythonTypeMap', 'pythonTypeRank', 'rank1', 'rank2', 'scalarTypeMap', 'signedtype1', 'signedtype2', 'typeDict', 'typecode', 'typecodes'] A bunch of leading "_" are needed. ---------------- My code is: #! /usr/bin/env python from numerictypes import UInt8 print UInt8 == "ABC" The message is: Traceback (most recent call last): File "./silly02.py", line 5, in ? print UInt8 == "ABC" File "/usr/lib/python2.2/site-packages/numarray/numerictypes.py", line 101, in __cmp__ other = typeDict[other] KeyError: ABC I would expect that the only object == to UInt8 be itself. Maybe add a function comparetypecodes(x, y) which returns True iff x and y are either NumericType's or strings which represent the same type.

3 4

Incomplete support in certain Numeric emulation functions in numarray
by Francesc Alted 22 Jan '03

22 Jan '03

Hi, I have discovered that the Numeric emulation functions in numarray doesn't accept a character typecode as type parameter. This is not immediately apparent because type parameter is of type 'int', and passing it a 'char' maybe not a good practice. But the fact is that Numeric *do* accept the charcodes in the type parameter. For example, this is the normal way to call the PyArray_FromDims function: arr = PyArray_FromDims(self.rank, self.dimensions, tFloat64) but, in Numeric, this other manner also works: arr = PyArray_FromDims(self.rank, self.dimensions, 'd') Now, in numarray, if you pass a character to the type parameter, a "segmentation fault" is issued. Look at the end of Numeric-22.0/Src/arraytypes.c, to see how characters are handled as types in Numeric. I think something like this should be added to the deferred_libnumarray_init in numarray-0.4/Src/newarray.ch. Another thing. It seems to me that NA_New and NA_Empty functions are not well documented in the numarray documentation as they differ from the definitions in numarray-0.4/Src/newarray.ch. I hope that the latter will stay, because I prefer them a lot more than the documented ones :-) Bye, -- Francesc Alted

2 2

Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?
by Francesc Alted 21 Jan '03

21 Jan '03

A Dimarts 21 Gener 2003 17:26, vàreu escriure: > Francesc Alted wrote: > >Anybody is aware of any function (either in C or Python or a mixture of > >both) to easily convert Numerical Python arrays from/to numarray arrays? > > I think you should look at numarray.fromlist() and NumArray.tolist(). I > think fromlist() will work on a nested sequence object, and hence a > Numeric array. Yeah, I knew that, but I was looking for something more optimal. > > >I mean, I would like to use such a funtion that, without having to copy > >element by element all the data, be able to copy the data buffer (or even > >use the same if possible at all) from one object to the other. > > I have not looked at this yet; it's a very good question. Note that > going from numarray to Numeric there are issues with making the buffer > well-behaved. I think this should be not too difficult to achieve and I'll try to explain why. When going from numarray to Numeric, numarray already have NA_InputArray C-API function that returns a well-behaved array. But strictly speaking, we don't even need a well-behaved array (this is a too restrictive condition) as both Numeric and numarray support discontiguous data. Even the byteorder should be not a problem, because, as Numeric itself has no such a property, we can create a Numeric array that is in native order as the result and byteswap the numarray object (if needed) before doing the conversion. So, non-alignment remains as the only issue that may cause a buffer copy during numarray ==> Numeric conversion. Is that correct?. If yes, it is possible to do a workaround about that, i.e. we can still get a Numeric from a numarray without copying the data in case of numarray misaligned objects?. Regarding to going in the other sense (ie. Numeric ==> numarray), as numarray supports discontiguity, misalignment and byteswapped data, this conversion should not imply a data buffer copy at all. Once we have a pointer to the data buffer, it is only a matter of wrapping a Numeric or numarray object around it getting this info from the original object, and returning the new object as a result. All in all, this conversion *seems* to be not a too difficult task. Making such a conversion functions (in C, but also having Python counterparts) available might represent to open the door to a co-existence of Numeric and numarray objects in the same program, and that would easy the numarray deployment in existing Numeric software. Comments? -- Francesc Alted

2 1

array operation
by Dave Bazell 21 Jan '03

21 Jan '03

I am trying to see if I can use where() or choose() to do this. I can't really figure it out. I have a 2-d array data where each row is an observation and each column is an attribute of the observation: data = [[.3, .2, 2.3,...] <- observation 1 [.7, 1.2, .4...] <- observation 2 ...]] I have another 1-d array that contains a code for the class of object: class = [0,1,0,1,1,3,2,0,...] where class[i] = the class of the ith object in the data array. Thus, observation 1 above is class 0, observation 2 is class 1, and so on. I want to select all objects of a given class from data array. I can do this with a loop for i in range(ndat): if class == 0: do something .... Is there a way to use where() or choose() to do this? Would it be more efficient? Thanks, Dave

3 2

RE: numarray/Numeric upkeep?
by paul＠pfdubois.com 21 Jan '03

21 Jan '03

Here are some of the factors leading to the slow rate of change of Numeric lately. a. I changed to a new project and have had a lot of startup learning to do. My new project uses Numeric but not in as central a way as my old one. b. I mistakenly thought numarray would be ready sooner so that I was trying to let it slide. c. I announced last year, in view of (a), that I was needing to be replaced as HeadNummie. It would be logical to turn this over to the Numarray people, but they aren't ready to do it until Numarray is ready, so nothing happened. d. Except for Travis, most of the other listed Numeric developers aren't in fact doing patches, releases, etc. e. Not all patches that are submitted are correct or desirable, historically. I'm not saying anything about any patches you may have submitted, just pointing out that applying them requires real work, not just mechanical patching. In fact the rate of error in patches is quite high and I've learned to be cautious. f. Some patches interfere with each other; for example, a patch for making 64 bit machines work right and a patch for some specific bug collided. I've started to work on the MA for Numarray but I'm not able to do much work on Numeric right now. This is a place where someone else has to help. >-- Original Message -- >To: dubois(a)users.sourceforge.net >Subject: numarray/Numeric upkeep? >From: Michael Stone <mbrierst(a)users.sourceforge.net> >Cc: <mbrierst(a)users.sourceforge.net> >Date: Tue, 21 Jan 2003 11:32:03 -0800 > > > >No one seems to be doing bugfixes for Numeric or numarray. >Nothing seems to have happened for several months. Lots of bugs have been >posted for Numeric, some easily fixable (I submitted one with a patch). > >Any idea if either project will become active again anytime soon?

2 1

Conversion functions between Numeric and numarray arrays?
by Francesc Alted 21 Jan '03

21 Jan '03

Hi, Anybody is aware of any function (either in C or Python or a mixture of both) to easily convert Numerical Python arrays from/to numarray arrays? I mean, I would like to use such a funtion that, without having to copy element by element all the data, be able to copy the data buffer (or even use the same if possible at all) from one object to the other. Thanks, -- Francesc Alted

2 2

recarray2 re-visited
by Francesc Alted 20 Jan '03

20 Jan '03

Hi, As I needed a byteswap() method for recarray, after a bit of hacking I've made one myself. This is based on my own version of recarray to take advantage of the _fields cache so as to both speed-up and simplify the new code. Basically, the new method takes a recarray, checking which columns are numarray arrays and invoking their byteswap() method if needed. Easy, but effective. Moreover, a _byteswap() and togglebyteorder() are provided to be compatible with existing methods in NumArray objects. As a plus, the recarray __str__ has been modified in order to allow a printing having in mind the byteorder of the recarray, and improving the speed of printing by a factor of 30, that can be handy in some situations. Do with it whatever you want, -- Francesc Alted

1 0

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]
by Todd Miller 20 Jan '03

20 Jan '03

Francesc Alted wrote: >Hi, > >I'm trying to make a C array from a Numeric "c" (Character) typecode array >using the high level call: > >NA_InputArray(PyObject *numarray, NumarrayType t, int requires) > Unified handling of character arrays and numeric arrays doesn't exist yet in numarray. There is no C-API for the chararray module because we haven't needed one. But CharArrays are NDArrays and have attributes stored in PyArrayObjects just like numarrays. >with no success. > >As I have been able to access all the other types (i.e. >'1','b','s','i','l','f','d') successfully, perhaps character type is not >supported? > >In the NumarrayType enum, there is no tChar, but I've tried tUInt8 and tAny >as the value for NumarrayType parameter, but both choices issues the same >error: > >Traceback (most recent call last): > File "table-tree2.py", line 77, in ? > h5file.createArray('/columns', 'name', array(names), "Name column") > File "/home/falted/PyTables/pytables-0.3/tables/File.py", line 400, in >createArray > setattr(group, name, object) > File "/home/falted/PyTables/pytables-0.3/tables/Group.py", line 355, in >__setattr__ > value._f_putObjectInTree(name, self) > File "/home/falted/PyTables/pytables-0.3/tables/Leaf.py", line 71, in >_f_putObjectInTree > self.create() > File "/home/falted/PyTables/pytables-0.3/tables/Array.py", line 83, in >create > self.createArray(self.object, self.title) > File "/home/falted/PyTables/pytables-0.3/src/hdf5Extension.pyx", line 913, >in createArray > array = NA_InputArray(arr, numfmt2[arr.typecode()], C_ARRAY) >libnumarray.error: getShape: sequence object nested more than MAXDIM deep. > NA_InputArray was intended to accept non-numeric sequences. It could report this better... >although I was passing only a Numeric 'c' with a rather small shape (10,16). > >I just want to access the buffer data, and the shape of this object from C >(well, I'm actually using Pyrex, but I think this is not important). Is that >possible by only using numarray C calls? > Look at Lib/chararray.py and Src/_chararraymodule.c. If you can handle using a CharArray or RawCharArray, try: 1. call NA_updateDataPtr( array ) to refresh the data buffer pointer in the PyArrayObject. Even _chararraymodule.c doesn't do this right yet. 2. call NA_OFFSETDATA(array) to add the byteoffset to the pointer. 3. shape, strides, and itemsize should be directly accessible from the PyArrayObject. CharArray has some extra stripping and padding semantics; these are lazy and hence absent without extra care in C. RawCharArray has none. CharArrays are really arrays of fixed length strings of bytes. The string length is defined by the array itemsize. >Thanks, > > > Todd

2 3