Mailman 3 February 2005 - NumPy-Discussion

Numeric3 and speed of new-style classes
by Travis Oliphant 04 Feb '05

04 Feb '05

I fixed a major oversight in Numeric3 and now it works pretty well (it is basically old Numeric with a bunch of new types, no math (yet)), using new-style classes, and some checks for alignment when dereferencing data. I did some simple array creation timing tests. An example of the results is given below. The bottom line is that new-style classes do nothing but speed up (a little bit) basic object creation. FYI: First test (Numeric3): .196081876755 import sys sys.path.insert(0,"../build/lib.linux-i686-2.3/") import time import multiarray start = time.time() for k in range(100000): g = multiarray.arraytype((4,3),'d') print time.time() - start Second test (Numeric): .203589916229 import Numeric import time start = time.time() for k in range(100000): g = Numeric.empty((4,3),'d') print time.time() - start

2 3

Problems compiling against ATLAS on x86_64
by Matt Hyclak 04 Feb '05

04 Feb '05

I'm about ready to pull out my hair trying to get this stuff to compile for a faculty member. I'm running RHEL 3 AS on a dual opteron machine. ATLAS compiles fine as I go through xconfig, but I can't for the life of me get Numeric to compile against it. I've tried ATLAS 3.6.0 and 3.7.8. I'm trying to build Numeric 23.7. I keep getting the following error: gcc -shared build/temp.linux-x86_64-2.2/lapack_litemodule.o -L/usr/lib64/atlas -llapack -lcblas -lf77blas -latlas -lg2c -o build/lib.linux-x86_64-2.2/lapack_lite.so /usr/bin/ld: /usr/lib64/atlas/liblapack.a(dgesv.o): relocation R_X86_64_32 can not be used when making a shared object; recompile with -fPIC /usr/lib64/atlas/liblapack.a: could not read symbols: Bad value collect2: ld returned 1 exit status I built ATLAS with -fPIC, but that didn't seem to help. Could someone *please* point me in the right direction before I start hitting things? Thanks! Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263

2 2

Load numarray with specific shape cause error
by Vincent Picavet 03 Feb '05

03 Feb '05

Hi, Here is what I launch in the Python interpreter : ---- import numarray simulation_file = "/path/to/my/file.bin" a = numarray.fromfile(simulation_file, 'Float32',(3144, 33, 65)) a.sum() ---- And here I get the following error : ---- Traceback (most recent call last): File "<stdin>", line 1, in ? File "/nfs/data2/picavet/workspace/softs/python/lib/python2.4/site-packages/ numarray/numarraycore.py", line 1133, in sum return ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type) libnumarray.error: add_Float64_reduce: access beyond buffer. offset=25151 buffersize=17160 ---- Just after that, I execute : ---- a.sum() 588108255.50844395 a.mean() 87.206215933326803 ---- ... and as you can see I got no error the second time. The binary file I try to load is a raw file of 3144*33*64 4-Bytes float elements. In fact I just saw that if I use the shape (65,33,3144) when calling fromfile(), everything works just fine. And after having called a.sum() the first time, the shape of the array has been changed. ---- >>> a.getshape() (3144, 65, 33) >>> a.sum() Traceback (most recent call last): [snip, same as above] >>> a.getshape() (33, 65, 3144) ---- Any idea why I can't load an array with such a shape ? I use Python 2.4 and the latest version of numarray. Python 2.4 (#1, Jan 27 2005, 15:21:15) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Thanks, Vincent -- Vincent Picavet, Tel : +33 (0) 1 39 63 52 14 INRIA Rocquencourt, E-mail : vincent.picavet(a)inria.fr Projet CLIME http://www-rocq.inria.fr/clime

2 1

pyrex numarray
by Simon Burton 03 Feb '05

03 Feb '05

Has anyone considered using pyrex to implement numarray ? I don't know a lot of the details but it seems to me that pyrex could unify numarray's python/c source code mixture and smooth the transition from python (==untyped pyrex) code to c (==typed pyrex) code. It would also help clueless users like me understand, and perhaps contribute to, the codebase. Also, the pyrex people have talked about applying this technique to the standard python libraries, both for readability and speed. ciao, Simon.

5 4

Re: [ctypes-users] copying/slicing ctypes arrays, (c_ulong *n)() - to Numeric
by Ray S 03 Feb '05

03 Feb '05

At 07:32 PM 2/3/2005 +0100, you wrote: >Don't be confused that the buffer() object says <read-only buffer ...>! >The buffer call only asks for readable memory..., but ctypes doesn't >care about the readonly attribute - it will happily write into this >memory. Hi Thomas, Yes, I was thinking of what the shell error said upon assignment... Upon adding to some working code all is well: >>> import Numeric, ctypes, string >>> N = Numeric.zeros((10,), Numeric.Float) >>> buf = buffer(N) >>> buf <read-only buffer for 0x008F9C28, ptr 0x008D7780, size 80 at 0x008FE220> >>> int(string.split(repr(buf))[5][:-1], 16) 9271168 ## numarray version # nAddress = int(string.split(repr(N._data))[2], 16) ## Numeric version NAddress = int(string.split(repr(buffer(N)))[5][:-1], 16) ## Load DLL here... ## do this to get data from the USB A/D's DLL usb.GetData(usb.Sn, (bufferInsertPos * N.itemsize()) + NAddress, ctypes.byref( (types.c_long * buffersize)() ) ) Which is faster than getting data into a ctypes array (c_ulong *n)() and then doing memmove() to Numeric - one less step. Maybe this snip would be of help to some others, although more so to numpy people. Of course, the Python array works the same: >>> a = array.array('l',[1,2,3]) >>> int(string.split(repr(buffer(a)))[5][:-1], 16) 8380408 >If this is too confusing, and this may well be, ctypes could expose a >memory() function which would insist on read-write memory, but apart >from that do the same that buffer does: No, not confusing, just not clear to a non-expert C person that ctypes ignores where Python is read-only. A simple note in the tutorial would be fine. Some over at numpy were also unaware of memmove()s' existence in the new releases, and seemed interested. Thanks again, Ray

1 0

Numeric3 status
by Travis Oliphant 03 Feb '05

03 Feb '05

I've gotten Numeric3 (equivalent to Numeric 30.0) into a state where multiarray compiles. It is far from production ready. Several sections of the code still need to be cleaned up to avoid segfaults. But, the general outline is there. As this is very alpha code (though based on a very mature code-base) it is named Numeric3 so it does not clobber your current Numeric installation just in case you accidentally do an install. You will have to make sure the correct multiarray is being loaded through your sys.path list (or just point your sys.path to the build location of Numeric3) if you want to play with it. No array_number methods have been loaded yet (there is an interface to load your own if you like). I expect to have ufuncs working with the new arrayobject by the end of the month. Until then, arrayobjects are purely structural beasts. If anybody would like to comment on the design or direction of Numeric 30.0, your comments are welcome. If you would like to checkout the code through CVS you may do that using cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/numpy login press <Enter> when prompted for a password. cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/numpy co -P /Numeric3/ A rough design document is available by following the link at http://www.scipy.org/wikis/numdesign/num3news The major goal is: Numeric 30.0 will have all the important new features of numarray as well as maintain or improve on the speed of Numeric for all array types. Besides this, Numeric 30.0 will not include any FFT's, LinearAlgebra, or Random Numbers (these are all provided much more proficiently by SciPy). Masked Arrays, and Matrices will be provided as standard subclasses in Python. Ideally, RecordArrays will be provided in the same way. I will not re-invent code that I can borrow from past versions of Numeric and or numarray. Best wishes, -Travis Oliphant

4 3

Numeric array's actual data address?
by Ray S 01 Feb '05

01 Feb '05

If I have an array N: >>> N = Numeric.zeros((1000,), Numeric.Float) >>> repr(N.__copy__) '<built-in method __copy__ of array object at 0x00809F00>' What is the actual address of the first element? Or, as an offset from the object? numarray gives us that: >>> N = numarray.zeros((1000,), numarray.Float) >>> N.info() class: <class 'numarray.numarraycore.NumArray'> shape: (1000,) strides: (8,) byteoffset: 0 bytestride: 8 itemsize: 8 aligned: 1 contiguous: 1 data: <memory at 009d67b8 with size:8000 held by object 009d6798 aliasing object 00000000> byteorder: little byteswap: 0 type: Float64 In numarray, the offset appears to be 20. If I try to use memmove() to fill a Numeric array it faults when using an offset of 20... Ray

3 2