Mailman 3 November 2005 - NumPy-Discussion

More on array protocol
by Francesc Altet 03 Nov '05

03 Nov '05

Hi, I'm having problems in using the array protocol in numarray-->Numeric in some situations. I'm using the PyBuffer_FromReadWriteMemory C call in order to create buffers that will be later used to create numarray objects (both NumArray and CharArray) in Python space. The problem is that the array protocol implementation is doing badly this conversion. I was able to create a small example in pure python that reproduces the problem: >>> import numarray >>> from numarray import memory >>> import Numeric >>> na=numarray.array([1,2]) >>> Numeric.array(na) array([1, 2]) but ... >>> wbuf = memory.writeable_buffer(na) >>> nawb = numarray.array(wbuf,type="Int32",shape=(2,)) >>> nawb array([1, 2]) >>> Numeric.array(nawb) array([ 2, 135510112]) I'm afraid that the problem should be related with the kind of buffer that is attached to the numarray object: >>> na._data <memory at 0x082577f8 with size:0x00000008 held by object 0x401f10a0 aliasing object 0x00000000> >>> nawb._data <read-write buffer for 0x401f10a0, size -1, offset 0 at 0x40572020> Travis, Todd, are you going to support read-write buffers or should I try to move to using the memory module in order to cope with this? For a series of reasons I'd like to keep using regular read-write buffers, but anyway... I think I'll be able to do the change if absolutely necessary. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"

2 1

byteswap questions
by Russell E. Owen 03 Nov '05

03 Nov '05

I've got a module that can send array data to ds9. It screws up on byteswapped data and I'm trying to fix it. The code sends the data to ds9 via a pipe using arr.tofile(). To send byteswapped data I think I have two choices (given that I don't want to modify the input array): 1) Figure out the byte order and specify a suitable flag to ds9 (it accepts "bigendian" or "littleendian" flags). This avoids copying the data, so it sounds attractive, but can I do it without relying on "internals"? Numarrays have an "isbyteswapped" method, but all this says is "not native machine order"; it doesn't actually tell me the order. I need to know if the order is bigendian or littleendian, and I can't find a documented way to get that, just an undocumented attribute _byteorder. 2) If the array is not in native byte order, make a native byte ordered copy before sending it. The following seems to work: if arr.isbyteswapped(): arr = arr.copy() but is this the recommended way to get native-order copy? - Is "copy" guaranteed to return a copy in native byte order? The documentation doesn't say. - I was tempted to use num.array(arr) to make the copy, but the documentation doesn't say what "array" does if the input is a already a numarray array. As an aside, I tried to use the byteswapped method, but it has a totally different effect: >>> d array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], ...type=Float32) >>> d.isbyteswapped() 1 >>> dcopy = d.copy() >>> d array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], ...type=Float32) >>> dcopy.isbyteswapped() 0 >>> dswap = d.byteswapped() >>> dswap array([[ 1.91063654e-38, 1.91063654e-38, 1.91063654e-38, ..., ...type=Float32) which I guess means I'd have to byteswap the byteswapped data. Aargh. -- Russell

4 5

Problem with array assignment
by Francesc Altet 03 Nov '05

03 Nov '05

Hi, I've recently upgraded to Numeric 24.0 and numarray 1.4.1 and I'm having a problem that did not exist before: >>> import Numeric >>> import numarray >>> Numeric.__version__ '24.0' >>> numarray.__version__ '1.4.1' >>> na=numarray.zeros((3,),'f') >>> num=Numeric.zeros((3,),'f') >>> na[...] = num Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: array sizes must be consistent. but... >>> na=numarray.zeros((3,),'i') >>> num=Numeric.zeros((3,),'i') >>> na[...] = num >>> na array([0, 0, 0]) ????? However, using Numeric 23.8 and numarray 1.3.3: >>> import Numeric >>> import numarray >>> Numeric.__version__ '23.8' >>> numarray.__version__ '1.3.3' >>> na=numarray.zeros((3,),'f') >>> num=Numeric.zeros((3,),'f') >>> na[...] = num >>> na array([ 0., 0., 0.], type=Float32) I don't know if the problem is coming from Numeric or numarray, but it only happens when one mix both: >>> Numeric.__version__ '24.0' >>> numarray.__version__ '1.4.1' >>> na=numarray.zeros((3,),'f') >>> na2=numarray.zeros((3,),'f') >>> na2[...] = na >>> na2 array([ 0., 0., 0.], type=Float32) >>> num=Numeric.zeros((3,),'f') >>> num2=Numeric.zeros((3,),'f') >>> num2[...] = num >>> num array([ 0., 0., 0.],'f') Perhaps the new array protocol is involved here? -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"

2 3

1.4.1 build failure under OSX
by Chris Fonnesbeck 03 Nov '05

03 Nov '05

Hi, Compiling 1.4.1 under OSX with either gcc 3.3 or 4.0 fails (all previous versions of numarray compiled without error). Here is the full output: Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] Using external BLAS and LAPACK running config Wrote config.h In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_convmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_sortmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_bytesmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, ... several dozen of the above, then ... Packages/LinearAlgebra2/Src/lapack_litemodule.c:12: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c:17: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgeev': Packages/LinearAlgebra2/Src/lapack_litemodule.c:107: warning: implicit declaration of function 'dgeev_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dsyevd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:199: warning: implicit declaration of function 'dsyevd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zheevd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:296: warning: implicit declaration of function 'zheevd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgelsd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:340: warning: implicit declaration of function 'dgelsd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgesv': Packages/LinearAlgebra2/Src/lapack_litemodule.c:378: warning: implicit declaration of function 'dgesv_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgesdd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:418: warning: implicit declaration of function 'dgesdd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgetrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:449: warning: implicit declaration of function 'dgetrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zungqr': Packages/LinearAlgebra2/Src/lapack_litemodule.c:477: warning: implicit declaration of function 'zungqr_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dorgqr': Packages/LinearAlgebra2/Src/lapack_litemodule.c:508: warning: implicit declaration of function 'dorgqr_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgeqrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:539: warning: implicit declaration of function 'zgeqrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgeqrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:570: warning: implicit declaration of function 'dgeqrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dpotrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:594: warning: implicit declaration of function 'dpotrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgeev': Packages/LinearAlgebra2/Src/lapack_litemodule.c:632: warning: implicit declaration of function 'zgeev_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgelsd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:679: warning: implicit declaration of function 'zgelsd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgesv': Packages/LinearAlgebra2/Src/lapack_litemodule.c:714: warning: implicit declaration of function 'zgesv_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgesdd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:759: warning: implicit declaration of function 'zgesdd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgetrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:794: warning: implicit declaration of function 'zgetrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zpotrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:818: warning: implicit declaration of function 'zpotrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: At top level: Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: 'lapack_liteError' defined but not used powerpc-apple-darwin8-gcc-4.0.0: -framework: linker input file unused because linking not done powerpc-apple-darwin8-gcc-4.0.0: vecLib: linker input file unused because linking not done In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/ranlib.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/ranlibmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/com.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_dotblas.c:12: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from Src/_dotblas.c:15: /System/Library/Frameworks/vecLib.framework/Headers/cblas.h:665: warning: function declaration isn't a prototype In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/high_levelmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/element_wisemodule.c:3: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/one_dimensionalmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/numpy_compatmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/numpy_compat2.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/testlite.c:26: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list Any ideas for a solution to this problem are welcome. -- Chris Fonnesbeck Atlanta, GA

2 1

enumerate indices and values of a multidimensionnal numarray.array
by Philou 02 Nov '05

02 Nov '05

Hi list, I'm a new subsciber to this numpy list. I have a multidimensionnal array (>2). For example: x = numarray.array([[[1,2,3],[4,5,6]],[[1,2,3],[4,5,6]]]) print x [[[1 2 3] [4 5 6]] [[1 2 3] [4 5 6]]] I need to enumerate both indices and final values (1D) of the x array. So i need: (0,0,0) 1 (0,0,1) 2 (0,0,2) 2 (0,1,0) 4 (0,1,1) 5 (0,1,2) 6 (1,0,0) 1 (1,0,1) 2 How can i do without using the x.flat() method? Thanks a lot for the answer. regards, Philippe Collet

2 1

Performance of the array protocol
by Francesc Altet 01 Nov '05

01 Nov '05

Hi, I'm trying to start using the array protocol for conversion between Numeric <--> numarray (and newcore in the future), but I'm a bit disappointed because of its performance. For numarray --> Numeric we have: >>> t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import Numeric; na=numarray.arange(10)") >>> t1.repeat(3,10000) [0.59375977516174316, 0.57908082008361816, 0.56574010848999023] >>>t2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(10)") >>> t2.repeat(3,10000) [0.11653494834899902, 0.1140749454498291, 0.1141819953918457] i.e. the array protocol seems 5x slower than the fromstring() method. Conversely, for Numeric --> numarray: >>> t3=timeit.Timer("na=numarray.array(num)", "import numarray; import Numeric;num=Numeric.arange(10)") >>> t3.repeat(3,10000) [1.3475611209869385, 1.3277668952941895, 1.3417830467224121] >>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(10)") >>> t4.repeat(3,10000) [0.42027187347412109, 0.41690587997436523, 0.41626906394958496] in this case, the array protocol is 3x slower than using the buffer interface. I'm wondering whether this relatively poor performance in present implementation of the array protocol is surmountable or is an intrinsic limitation of it. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"

5 15