Mailman 3 June 2006 - NumPy-Discussion

memory leak in array
by saagesen＠sfu.ca June 15, 2006

June 15, 2006

Hi I'm new to programming in python and I hope that this is the problem. I've created a cellular automata program in python with the numpy array extensions. After each cycle/iteration the memory used to examine and change the array as determined by the transition rules is never freed. I've tried using "del" on every variable possible, but that hasn't worked. I've read all the forums for helpful hints on what to do, but nothing has worked so far. I've even tried the "python memory verification" (beta) program, which did point to numpy.dtype and numpy.ndarray as increasing objects, before the whole computer crashed. I can supply the code if needed. I'm desperate because this is part of my thesis, and if I can't get this fixed, I'll try another programming language. Update: I posted this message on the comp.lang.python forum and their response was to get the numbers of references with sys.getrefcount(obj). After doing this I see that iterative counters used to count occurrences and nested loop counters (ii & jj) as seen in the code example below are the culprits with the worst ones over 1M: for ii in xrange(0,40): for jj in xrange(0,20): try: nc = y[a+ii,b+jj] except IndexError: nc = 0 if nc == "1" or nc == "5": news = news +1 if news == 100: break else: pass y[a+ii,b+jj] = 4 else: pass The version of python I'm using is 2.4.3 and the version of NumPy is 0.9.8 thanks in advance Sonja

2 1

acml and numpy install problems
by JJ June 15, 2006

June 15, 2006

Hello. I wrote to the list about a week ago regarding slow speed of numpy relative to matlab. Im fairly sure that my installation of numpy had problems. So I am trying this time with the acml libraries for my AMD Athelon 64 bit machine. New machine with FC_5. I was able to install the acml libraries without much trouble, and install UMFPACK and AMD without apparent errors. But I did have many errors when I tried to install numpy. My install messages are copied below. Apparently, numpy does see the acml libraries but finds them faulty, or something. I could use some clues if anyone has any. Also, I did set: setenv LD_LIBRARY_PATH /opt/acml3.1.0/gnu64/lib # setenv LD_RUN_PATH /opt/acml3.1.0/gnu64/lib Here is my config file: ----------------------------------- [atlas] library_dirs = /opt/acml3.1.0/gnu64/lib include_dirs = /opt/acml3.1.0/gnu64/include atlas_libs = acml language = f77 [blas] library_dirs = /opt/acml3.1.0/gnu64/lib include_dirs = /opt/acml3.1.0/gnu64/include atlas_libs = acml language = f77 [laplack] library_dirs = /opt/acml3.1.0/gnu64/lib include_dirs = /opt/acml3.1.0/gnu64/include atlas_libs = acml language = f77 [amd] library_dirs = /usr/local/scipy/AMD/Lib include_dirs = /usr/local/scipy/AMD/Include amd_libs = amd language =c [umfpack] library_dirs = /usr/local/scipy/UMFPACK/Lib include_dirs = /usr/local/scipy/UMFPACK/Include umfpack_libs = umfpack language = c ------------------------------------ I have set symbolic links between lacml and libacml. Here is the first half of the output, where most of the errors are: -------------------------------- [root@fedora-newamd numpy]# python setup.py install Running from numpy source directory. No module named __svn_version__ F2PY Version 2_2624 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not find in /usr/local/lib libraries mkl,vml,guide not find in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['acml'] library_dirs = ['/opt/acml3.1.0/gnu64/lib'] language = c customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config compiling '_configtest.c': /* This file is generated from numpy_distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/opt/acml3.1.0/gnu64/lib -lacml -o _configtest _configtest.o: In function `main': /usr/local/numpy/_configtest.c:5: undefined reference to `ATL_buildinfo' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `do_lio' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `e_wsle' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `e_wsfe' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `z_abs' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined ... <snip> ... reference to `s_wsle' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `s_wsfe' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `s_copy' collect2: ld returned 1 exit status _configtest.o: In function `main': /usr/local/numpy/_configtest.c:5: undefined reference to `ATL_buildinfo' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `do_lio' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `e_wsle' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `e_wsfe' ... <snip> ... reference to `s_wsle' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `s_wsfe' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `s_copy' collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Status: 255 Output: FOUND: libraries = ['acml'] library_dirs = ['/opt/acml3.1.0/gnu64/lib'] language = c define_macros = [('NO_ATLAS_INFO', 2)] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not find in /usr/local/lib libraries mkl,vml,guide not find in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries lapack_atlas not find in /opt/acml3.1.0/gnu64/lib libraries lapack not find in /opt/acml3.1.0/gnu64/lib libraries acml not find in /usr/local/lib libraries lapack_atlas not find in /usr/local/lib libraries acml not find in /usr/lib libraries lapack_atlas not find in /usr/lib numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS /usr/local/numpy/numpy/distutils/system_info.py:881: UserWarning: ********************************************************************* Could not find lapack library within the ATLAS installation. ********************************************************************* warnings.warn(message) Setting PTATLAS=ATLAS FOUND: libraries = ['acml'] library_dirs = ['/opt/acml3.1.0/gnu64/lib'] language = c define_macros = [('ATLAS_WITHOUT_LAPACK', None)] customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config compiling '_configtest.c': /* This file is generated from numpy_distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/opt/acml3.1.0/gnu64/lib -lacml -o _configtest _configtest.o: In function `main': /usr/local/numpy/_configtest.c:5: undefined reference to `ATL_buildinfo' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `do_lio' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `e_wsle' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `e_wsfe' ...<snip> ... /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `acos' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `s_wsle' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `s_wsfe' /opt/acml3.1.0/gnu64/lib/libacml.so: undefined reference to `s_copy' collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Status: 255 Output: lapack_info: libraries lapack not find in /usr/local/lib libraries lapack not find in /usr/lib NOT AVAILABLE /usr/local/numpy/numpy/distutils/system_info.py:1163: UserWarning: Lapack (http://www.netlib.org/lapack/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [lapack]) or by setting the LAPACK environment variable. warnings.warn(LapackNotFoundError.__doc__) lapack_src_info: NOT AVAILABLE /usr/local/numpy/numpy/distutils/system_info.py:1166: UserWarning: Lapack (http://www.netlib.org/lapack/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [lapack_src]) or by setting the LAPACK_SRC environment variable. warnings.warn(LapackSrcNotFoundError.__doc__) NOT AVAILABLE running install running build running config_fc running build_src building py_modules sources creating build creating build/src.linux-x86_64-2.4 creating build/src.linux-x86_64-2.4/numpy creating build/src.linux-x86_64-2.4/numpy/distutils building extension "numpy.core.multiarray" sources creating build/src.linux-x86_64-2.4/numpy/core Generating build/src.linux-x86_64-2.4/numpy/core/config.h customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-I/usr/include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:50: warning: format %d expects type int, but argument 4 has ty pe long unsigned int _configtest.c:57: warning: format %d expects type int, but argument 4 has ty pe long unsigned int _configtest.c:72: warning: format %d expects type int, but argument 4 has ty pe long unsigned int gcc -pthread _configtest.o -L/usr/local/lib -L/usr/lib -o _configtest /usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lp thread /usr/bin/ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpt hread /usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc /usr/bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c gcc -pthread _configtest.o -o _configtest _configtest.o: In function `main': /usr/local/numpy/_configtest.c:5: undefined reference to `exp' collect2: ld returned 1 exit status _configtest.o: In function `main': /usr/local/numpy/_configtest.c:5: undefined reference to `exp' collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c gcc -pthread _configtest.o -lm -o _configtest _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D _FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c: In function main: _configtest.c:4: warning: statement with no effect gcc -pthread _configtest.o -lm -o _configtest success! removing: _configtest.c _configtest.o _configtest adding 'build/src.linux-x86_64-2.4/numpy/core/config.h' to sources. executing numpy/core/code_generators/generate_array_api.py adding 'build/src.linux-x86_64-2.4/numpy/core/__multiarray_api.h' to sources. creating build/src.linux-x86_64-2.4/numpy/core/src conv_template:> build/src.linux-x86_64-2.4/numpy/core/src/scalartypes.inc adding 'build/src.linux-x86_64-2.4/numpy/core/src' to include_dirs. conv_template:> build/src.linux-x86_64-2.4/numpy/core/src/arraytypes.inc numpy.core - nothing done with h_files= ['build/src.linux-x86_64-2.4/numpy/core/ src/scalartypes.inc', 'build/src.linux-x86_64-2.4/numpy/core/src/arraytypes.inc' , 'build/src.linux-x86_64-2.4/numpy/core/config.h', 'build/src.linux-x86_64-2.4/ numpy/core/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src.linux-x86_64-2.4/numpy/core/config.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.linux-x86_64-2.4/numpy/core/__ufunc_api.h' to sources. conv_template:> build/src.linux-x86_64-2.4/numpy/core/src/umathmodule.c adding 'build/src.linux-x86_64-2.4/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src.linux-x86_64-2.4/numpy/core/ src/scalartypes.inc', 'build/src.linux-x86_64-2.4/numpy/core/src/arraytypes.inc' , 'build/src.linux-x86_64-2.4/numpy/core/config.h', 'build/src.linux-x86_64-2.4/ numpy/core/__ufunc_api.h'] building extension "numpy.core._sort" sources adding 'build/src.linux-x86_64-2.4/numpy/core/config.h' to sources. adding 'build/src.linux-x86_64-2.4/numpy/core/__multiarray_api.h' to sources. conv_template:> build/src.linux-x86_64-2.4/numpy/core/src/_sortmodule.c numpy.core - nothing done with h_files= ['build/src.linux-x86_64-2.4/numpy/core/ config.h', 'build/src.linux-x86_64-2.4/numpy/core/__multiarray_api.h'] building extension "numpy.core.scalarmath" sources adding 'build/src.linux-x86_64-2.4/numpy/core/config.h' to sources. adding 'build/src.linux-x86_64-2.4/numpy/core/__multiarray_api.h' to sources. adding 'build/src.linux-x86_64-2.4/numpy/core/__ufunc_api.h' to sources. conv_template:> build/src.linux-x86_64-2.4/numpy/core/src/scalarmathmodule.c numpy.core - nothing done with h_files= ['build/src.linux-x86_64-2.4/numpy/core/ config.h', 'build/src.linux-x86_64-2.4/numpy/core/__multiarray_api.h', 'build/sr c.linux-x86_64-2.4/numpy/core/__ufunc_api.h'] building extension "numpy.core._dotblas" sources adding 'numpy/core/blasdot/_dotblas.c' to sources. building extension "numpy.lib._compiled_base" sources building extension "numpy.dft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources creating build/src.linux-x86_64-2.4/numpy/linalg ### Warning: Using unoptimized lapack ### --------------------------------------------- Any ideas? I am still a novice and could use some suggestions. Thanks much. JJ __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

"车间管理人员八项修炼(AD)"
by "6月24-25,周六日,上海" June 15, 2006

June 15, 2006

1 0

old-Numeric: OverflowError on exp(-760)
by Sebastian Haase June 14, 2006

June 14, 2006

Hi, I'm using Konrad Hinsen's LeastSquares.leastSquaresFit for a convenient way to do a non linear minimization. It uses the "old" Numeric module. But since I upgraded to Numeric 24.2 I get OverflowErrors that I tracked down to >>> Numeric.exp(-760.) Traceback (most recent call last): File "<input>", line 1, in ? OverflowError: math range error From numarray I'm used to getting this: >>> na.exp(-760) 0.0 Mostly I'm confused because my code worked before I upgraded to version 24.2. Thanks for any hints on how I could revive my code... -Sebastian Haase

2 3

installation problems: stupid question
by Eric Emsellem June 14, 2006

June 14, 2006

Hi, I just switched to Suse 10.1 (from Suse 10.0) and for some reason now the new installed modules do not go under /usr/lib/python2.4/site-packages/ as usual but under /usr/local/lib/python2.4/site-packages/ (the "local" is the difference). How can I go back to the normal setting ? thanks a lot for any input there. Eric P.S.: I seem to then have problem with lapack_lite.so (undefined symbol: s_cat) and it may be linked

2 1

maximmum.reduce and nans
by Martin Wiechert June 14, 2006

June 14, 2006

Hi list, does anybody know, why maximum.reduce (()) does not return -inf? Looks very natural to me and as a byproduct maximum.reduce would ignore nans, thereby removing the need of nanmax etc. The current convention gives >>> from numpy import * >>> maximum.reduce ((1,nan)) 1.0 >>> maximum.reduce ((nan, 1)) nan >>> maximum.reduce (()) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: zero-size array to ufunc.reduce without identity >>> Cheers, Martin

2 1

numpy.test() fails on Redhat Enterprise and Solaris
by Christopher Hanley June 14, 2006

June 14, 2006

The daily numpy build and tests I run have failed for revision 2617. Below is the error message I receive on my RHE 3 box: ====================================================================== FAIL: Check reading the nested fields of a nested array (1st level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/sparty1/dev/site-packages/lib/python/numpy/core/tests/test_numerictypes.py", line 283, in check_nested1_acessors dtype='U2')) File "/data/sparty1/dev/site-packages/lib/python/numpy/testing/utils.py", line 139, in assert_equal return assert_array_equal(actual, desired, err_msg) File "/data/sparty1/dev/site-packages/lib/python/numpy/testing/utils.py", line 215, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/data/sparty1/dev/site-packages/lib/python/numpy/testing/utils.py", line 207, in assert_array_compare assert cond, msg AssertionError: Arrays are not equal (mismatch 100.0%) x: array([u'NN', u'OO'], dtype='<U8') y: array([u'NN', u'OO'], dtype='<U2') On my Solaris 8 box this same test causes a bus error: Check creation of single-dimensional objects ... ok Check creation of 0-dimensional objects ... ok Check creation of multi-dimensional objects ... ok Check creation of single-dimensional objects ... ok Check reading the top fields of a nested array ... ok Check reading the nested fields of a nested array (1st level)Bus Error (core dumped) Chris

1 0

Re: [Numpy-discussion] addressing a submatrix
by Martin Wiechert June 14, 2006

June 14, 2006

Thanks Pau, that's exactly what I was looking for. Martin On Wednesday 14 June 2006 12:01, you wrote: > On 6/14/06, Martin Wiechert <Martin.Wiechert(a)mpimf-heidelberg.mpg.de> wrote: > > Hi Simon, > > > > thanks for your reply. > > > > A [I, J] > > > > seems to only work if the indices are *strides* as in your example. I > > need fancy indices (like I = (1,3,4), J = (0,3,5)), and for them A [I, J] > > won't do what I want. As you can see from the example session I posted it > > does not address the whole rectangle IxJ but only the elements (I_1, > > J_1), (I_2, J_2). E.g., if I==J this is the diagonal of the submatrix, > > not the full submatrix. > > you can use A[ ix_(I,J) ] to do what you want. > > But, if you just want subrectangular regions then A[1:4,1:4] is enough. > Please note that A[1:4,1:4] is not the same as A[ arange(1,4), arange(1,4) > ], but is the same as A[ ix_(arange(1,4), arange(1,4)) ]. > > hope this heps > pau

1 0

Back to numexpr
by Tim Hochberg June 14, 2006

June 14, 2006

I've finally got around to looking at numexpr again. Specifically, I'm looking at Francesc Altet's numexpr-0.2, with the idea of harmonizing the two versions. Let me go through his list of enhancements and comment (my comments are dedented): - Addition of a boolean type. This allows better array copying times for large arrays (lightweight computations ara typically bounded by memory bandwidth). Adding this to numexpr looks like a no brainer. Behaviour of booleans are different than integers, so in addition to being more memory efficient, this enables boolean &, |, ~, etc to work properly. - Enhanced performance for strided and unaligned data, specially for lightweigth computations (e.g. 'a>10'). With this and the addition of the boolean type, we can get up to 2x better times than previous versions. Also, most of the supported computations goes faster than with numpy or numarray, even the simplest one. Francesc, if you're out there, can you briefly describe what this support consists of? It's been long enough since I was messing with this that it's going to take me a while to untangle NumExpr_run, where I expect it's lurking, so any hints would be appreciated. - Addition of ~, & and | operators (a la numarray.where) Sounds good. - Support for both numpy and numarray (use the flag --force-numarray in setup.py). At first glance this looks like it doesn't make things to messy, so I'm in favor of incorporating this. - Added a new benchmark for testing boolean expressions and strided/unaligned arrays: boolean_timing.py Benchmarks are always good. Things that I want to address in the future: - Add tests on strided and unaligned data (currently only tested manually) Yep! Tests are good. - Add types for int16, int64 (in 32-bit platforms), float32, complex64 (simple prec.) I have some specific ideas about how this should be accomplished. Basically, I don't think we want to support every type in the same way, since this is going to make the case statement blow up to an enormous size. This may slow things down and at a minimum it will make things less comprehensible. My thinking is that we only add casts for the extra types and do the computations at high precision. Thus adding two int16 numbers compiles to two OP_CAST_Ffs followed by an OP_ADD_FFF, and then a OP_CAST_fF. The details are left as an excercise to the reader ;-). So, adding int16, float32, complex64 should only require the addition of 6 casting opcodes plus appropriate modifications to the compiler. For large arrays, this should have most of the benfits of giving each type it's own opcode, since the memory bandwidth is still small, while keeping the interpreter relatively simple. Unfortunately, int64 doesn't fit under this scheme; is it used enough to matter? I hate pile a whole pile of new opcodes on for something that's rarely used. Regards, -tim

4 11

Re: [Numpy-discussion] addressing a submatrix
by Pau Gargallo June 14, 2006

June 14, 2006

On 6/14/06, Martin Wiechert <Martin.Wiechert(a)mpimf-heidelberg.mpg.de> wrote: > Hi Simon, > > thanks for your reply. > > A [I, J] > > seems to only work if the indices are *strides* as in your example. I need > fancy indices (like I = (1,3,4), J = (0,3,5)), and for them A [I, J] won't do > what I want. As you can see from the example session I posted it does not > address the whole rectangle IxJ but only the elements (I_1, J_1), (I_2, J_2). > E.g., if I==J this is the diagonal of the submatrix, not the full submatrix. you can use A[ ix_(I,J) ] to do what you want. But, if you just want subrectangular regions then A[1:4,1:4] is enough. Please note that A[1:4,1:4] is not the same as A[ arange(1,4), arange(1,4) ], but is the same as A[ ix_(arange(1,4), arange(1,4)) ]. hope this heps pau

1 0