Mailman 3 November 2007 - NumPy-Discussion

segfault
by Eric Firing Nov. 4, 2007

Nov. 4, 2007

A quick scan of the tickets did not show me anything like the following, but I might have missed it. The attached script generates a segfault on my ubuntu feisty system with svn numpy. Running inside of ipython, the segfault occurs upon exiting ipython, not upon running the script. Running the script from the command line gives the attached trace. Eric efiring@manini:~/test$ python numpysegfault.py *** glibc detected *** python: free(): invalid next size (fast): 0x08388d98 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6[0xb7dbc7cd] /lib/tls/i686/cmov/libc.so.6(cfree+0x90)[0xb7dbfe30] /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so[0xb7bd414e] python[0x8085269] python(PyDict_SetItem+0x6e)[0x8086bae] python(_PyModule_Clear+0x158)[0x8088808] python(PyImport_Cleanup+0x153)[0x80dcba3] python(Py_Finalize+0xbf)[0x80e8d9f] python(Py_Main+0x4d1)[0x8058e41] python(main+0x22)[0x8058862] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xdc)[0xb7d6aebc] python[0x80587b1] ======= Memory map: ======== 08048000-0813f000 r-xp 00000000 08:05 212471 /usr/bin/python2.5 0813f000-08164000 rw-p 000f6000 08:05 212471 /usr/bin/python2.5 08164000-08430000 rw-p 08164000 00:00 0 [heap] b6c00000-b6c21000 rw-p b6c00000 00:00 0 b6c21000-b6d00000 ---p b6c21000 00:00 0 b6d02000-b6d0d000 r-xp 00000000 08:05 586432 /lib/libgcc_s.so.1 b6d0d000-b6d0e000 rw-p 0000a000 08:05 586432 /lib/libgcc_s.so.1 b6d20000-b6d25000 r-xp 00000000 08:05 342572 /usr/lib/python2.5/lib-dynload/_struct.so b6d25000-b6d26000 rw-p 00005000 08:05 342572 /usr/lib/python2.5/lib-dynload/_struct.so b6d26000-b6d39000 r-xp 00000000 08:05 853437 /usr/local/lib/python2.5/site-packages/ctypes-1.0.2-py2.5-linux-i686.egg/_ctypes.so b6d39000-b6d3b000 rw-p 00013000 08:05 853437 /usr/local/lib/python2.5/site-packages/ctypes-1.0.2-py2.5-linux-i686.egg/_ctypes.so b6d3b000-b6d6d000 r-xp 00000000 08:05 750591 /usr/local/lib/python2.5/site-packages/numpy/random/mtrand.so b6d6d000-b6d71000 rw-p 00032000 08:05 750591 /usr/local/lib/python2.5/site-packages/numpy/random/mtrand.so b6d71000-b6d7a000 r-xp 00000000 08:05 752453 /usr/local/lib/python2.5/site-packages/numpy/fft/fftpack_lite.so b6d7a000-b6d7b000 rw-p 00008000 08:05 752453 /usr/local/lib/python2.5/site-packages/numpy/fft/fftpack_lite.so b6d7b000-b6e1f000 r-xp 00000000 08:05 752470 /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so b6e1f000-b6e20000 rw-p 000a3000 08:05 752470 /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so b6e20000-b6e87000 rw-p b6e20000 00:00 0 b6e87000-b6e8a000 r-xp 00000000 08:05 752451 /usr/local/lib/python2.5/site-packages/numpy/lib/_compiled_base.so b6e8a000-b6e8b000 rw-p 00002000 08:05 752451 /usr/local/lib/python2.5/site-packages/numpy/lib/_compiled_base.so b6e8b000-b6ea5000 r-xp 00000000 08:05 752464 /usr/local/lib/python2.5/site-packages/numpy/core/scalarmath.so b6ea5000-b6ea6000 rw-p 0001a000 08:05 752464 /usr/local/lib/python2.5/site-packages/numpy/core/scalarmath.so b6ea6000-b6ea9000 r-xp 00000000 08:05 344739 /usr/lib/python2.5/lib-dynload/mmap.so b6ea9000-b6eaa000 rw-p 00003000 08:05 344739 /usr/lib/python2.5/lib-dynload/mmap.so b6eaa000-b6ead000 r-xp 00000000 08:05 342574 /usr/lib/python2.5/lib-dynload/cStringIO.so b6ead000-b6eae000 rw-p 00003000 08:05 342574 /usr/lib/python2.5/lib-dynload/cStringIO.so b6eae000-b6ec7000 r-xp 00000000 08:05 214313 /usr/lib/libg2c.so.0.0.0 b6ec7000-b6ec8000 rw-p 00019000 08:05 214313 /usr/lib/libg2c.so.0.0.0 b6ec8000-b6ed5000 rw-p b6ec8000 00:00 0 b6ed5000-b74ac000 r-xp 00000000 08:05 343193 /usr/lib/atlas/sse2/libblas.so.3.0 b74ac000-b74b1000 rw-p 005d6000 08:05 343193 /usr/lib/atlas/sse2/libblas.so.3.0 b74b1000-b79fa000 r-xp 00000000 08:05 343194 /usr/lib/atlas/sse2/liblapack.so.3.0 b79fa000-b79fd000 rw-p 00549000 08:05 343194 /usr/lib/atlas/sse2/liblapack.so.3.0 b79fd000-b7b01000 rw-p b79fd000 00:00 0 b7b01000-b7b12000 r-xp 00000000 08:05 344738 /usr/lib/python2.5/lib-dynload/cPickle.so b7b12000-b7b13000 rw-p 00010000 08:05 344738 /usr/lib/python2.5/lib-dynload/cPickle.so b7b13000-b7b18000 r-xp 00000000 08:05 749369 /usr/local/lib/python2.5/site-packages/numpy/core/_dotblas.so b7b18000-b7b19000 rw-p 00004000 08:05 749369 /usr/local/lib/python2.5/site-packages/numpy/core/_dotblas.so b7b19000-b7b7a000 rw-p b7b19000 00:00 0 b7b7a000-b7ba0000 r-xp 00000000 08:05 749367 /usr/local/lib/python2.5/site-packages/numpy/core/umath.so b7ba0000-b7ba3000 rw-p 00025000 08:05 749367 /usr/local/lib/python2.5/site-packages/numpy/core/umath.so b7ba3000-b7ba4000 rw-p b7ba3000 00:00 0 b7ba4000-b7bfe000 r-xp 00000000 08:05 749349 /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so b7bfe000-b7c04000 rw-p 00059000 08:05 749349 /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so b7c04000-b7c05000 rw-p b7c04000 00:00 0 b7c05000-b7c4a000 r-xp 00000000 08:05 586444 /lib/libncursesw.so.5.5 b7c4a000-b7c52000 rw-p 00045000 08:05 586444 /lib/libncursesw.so.5.5 b7c52000-b7c53000 rw-p b7c52000 00:00 0 b7c55000-b7c64000 r-xp 00000000 08:05 752457 /usr/local/lib/python2.5/site-packages/numpy/core/_sort.so b7c64000-b7c65000 rw-p 0000e000 08:05 752457 /usr/local/lib/python2.5/site-packages/numpy/core/_sort.so b7c65000-b7c74000 r-xp 00000000 08:05 344754 /usr/lib/python2.5/lib-dynload/_curses.so b7c74000-b7c75000 rw-p 0000f000 08:05 344754 /usr/lib/python2.5/lib-dynload/_curses.so b7c75000-b7c77000 r-xp 00000000 08:05 342580 /usr/lib/python2.5/lib-dynload/fcntl.so b7c77000-b7c78000 rw-p 00002000 08:05 342580 /usr/lib/python2.5/lib-dynload/fcntl.so b7c78000-b7c7c000 r-xp 00000000 08:05 342578 /usr/lib/python2.5/lib-dynload/binascii.so b7c7c000-b7c7d000 rw-p 00003000 08:05 342578 /usr/lib/python2.5/lib-dynload/binascii.so b7c7d000-b7c80000 r-xp 00000000 08:05 342586 /usr/lib/python2.5/lib-dynload/math.so b7c80000-b7c81000 rw-p 00002000 08:05 342586 /usr/lib/python2.5/lib-dynload/math.so b7c81000-b7c85000 r-xp 00000000 08:05 342592 /usr/lib/python2.5/lib-dynload/strop.so b7c85000-b7c87000 rw-p 00003000 08:05 342592 /usr/lib/python2.5/lib-dynload/strop.so b7c87000-b7c89000 r-xp 00000000 08:05 344730 /usr/lib/python2.5/lib-dynload/_heapq.so b7c89000-b7c8b000 rw-p 00002000 08:05 344730 /usr/lib/python2.5/lib-dynload/_heapq.so b7c8b000-b7c8e000 r-xp 00000000 08:05 342596 /usr/lib/python2.5/lib-dynload/time.so b7c8e000-b7c90000 rw-p 00002000 08:05 342596 /usr/lib/python2.5/lib-dynload/time.so b7c90000-b7c97000 r--s 00000000 08:05 782059 /usr/lib/gconv/gconv-modules.cache b7c97000-b7cd2000 r--p 00000000 08:05 261980 /usr/lib/locale/en_US.utf8/LC_CTYPE b7cd2000-b7d55000 rw-p b7cd2000 00:00 0 b7d55000-b7e90000 r-xp 00000000 08:05 620099 /lib/tls/i686/cmov/libc-2.5.so b7e90000-b7e91000 r--p 0013b000 08:05 620099 /lib/tls/i686/cmov/libc-2.5.so b7e91000-b7e93000 rw-p 0013c000 08:05 620099 /lib/tls/i686/cmov/libc-2.5.so b7e93000-b7e96000 rw-p b7e93000 00:00 0 b7e96000-b7ebb000 r-xp 00000000 08:05 620107 /lib/tls/i686/cmov/libm-2.5.so b7ebb000-b7ebd000 rw-p 00024000 08:05 620107 /lib/tls/i686/cmov/libm-2.5.so b7ebd000-b7ebf000 r-xp 00000000 08:05 620133 /lib/tls/i686/cmov/libutil-2.5.so b7ebf000-b7ec1000 rw-p 00001000 08:05 620133 /lib/tls/i686/cmov/libutil-2.5.so b7ec1000-b7ec2000 rw-p b7ec1000 00:00 0 b7ec2000-b7ec4000 r-xp 00000000 08:05 620105 /lib/tls/i686/cmov/libdl-2.5.so b7ec4000-b7ec6000 rw-p 00001000 08:05 620105 /lib/tls/i686/cmov/libdl-2.5.so b7ec6000-b7ed9000 r-xp 00000000 08:05 620125 /lib/tls/i686/cmov/libpthread-2.5.so b7ed9000-b7edb000 rw-p 00013000 08:05 620125 /lib/tls/i686/cmov/libpthread-2.5.so b7edb000-b7edd000 rw-p b7edb000 00:00 0 b7edd000-b7edf000 r-xp 00000000 08:05 342568 /usr/lib/python2.5/lib-dynload/_random.so b7edf000-b7ee0000 rw-p 00002000 08:05 342568 /usr/lib/python2.5/lib-dynload/_random.so b7ee0000-b7ee1000 r-xp 00000000 08:05 344729 /usr/lib/python2.5/lib-dynload/_bisect.so b7ee1000-b7ee2000 rw-p 00001000 08:05 344729 /usr/lib/python2.5/lib-dynload/_bisect.so b7ee2000-b7ee7000 r-xp 00000000 08:05 342588 /usr/lib/python2.5/lib-dynload/operator.so b7ee7000-b7ee8000 rw-p 00005000 08:05 342588 /usr/lib/python2.5/lib-dynload/operator.so b7ee8000-b7eed000 r-xp 00000000 08:05 342584 /usr/lib/python2.5/lib-dynload/itertools.so b7eed000-b7eef000 rw-p 00005000 08:05 342584 /usr/lib/python2.5/lib-dynload/itertools.so b7eef000-b7ef1000 rw-p b7eef000 00:00 0 b7ef1000-b7f0a000 r-xp 00000000 08:05 586389 /lib/ld-2.5.so b7f0a000-b7f0c000 rw-p 00019000 08:05 586389 /lib/ld-2.5.so bf997000-bf9be000 rwxp bf997000 00:00 0 [stack] bf9be000-bf9bf000 rw-p bf9be000 00:00 0 ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso] Aborted (core dumped)

2 1

NumPy trunk is frozen for upcoming 1.0.4 release
by Jarrod Millman Nov. 3, 2007

Nov. 3, 2007

Hello, NumPy 1.0.4 is almost out! I will be tagging the release this Sunday (11/4), so please be extra careful about anything you check-in over the next 24 hours. Once the tag is made, the trunk will be open for changes. I plan to announce the 1.0.4 release by Wednesday (11/7). I am hoping that everyone will be able to take the time to test the NumPy trunk this weekend and alert me of any problems. Please check-out the trunk, build it, and run the full test suite: svn co http://svn.scipy.org/svn/numpy/trunk ./numpy-trunk cd numpy-trunk python setup.py build sudo python setup.py install python import numpy numpy.test(10,10) Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

1 0

C or C++ package like NumPy?
by Bill Baxter Nov. 2, 2007

Nov. 2, 2007

Does anyone know of a C or C++ library that's similar to NumPy? Seems like all the big C++ efforts are focused on linear algebra rather than general purpose multidimensional arrays. I've written a multidimensional array class in the D programming language with an API modeled loosely after NumPy's. It works ok but there are a lot of differences between a statically typed language like C++ or D and a dynamic one like Python. So I was looking around for API inspiration from other C/C++ libraries. But the projects I know about are all focused on linear algebra (like ublas and MTL) and don't support general N-dimensional arrays. Thanks for your comments. --bb

6 9

weird numpy/pickle problem
by Brian Blais Nov. 2, 2007

Nov. 2, 2007

Hello, I encountered a peculiar numpy and pickle problem. My version: Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) Mac OS X Tiger In [2]:numpy.__version__ Out[2]:'1.0.4.dev3869' I pickle a matrix, and reload it. Some operations work ok, but others give a hardware error and a crash! I boiled it down to the code below. Can anyone reproduce (or not) this error? thanks, bb -- Brian Blais bblais(a)bryant.edu http://web.bryant.edu/~bblais ======================================================================= from numpy import matrix,random import pickle w_test=matrix(random.rand(3,21)) fid=open('blah0_test.pickle','wb') pickle.dump(w_test,fid,0) fid.close() fid=open('blah0_test.pickle','rb') w_xh=pickle.load(fid) fid.close() # this works print w_xh*w_xh.T #this works x=matrix(random.rand(2,21)) print x*w_xh.T # this gives a hardware error! x=matrix([0]*21) print x*w_xh.T

2 1

Re: [Numpy-discussion] weird numpy/pickle problem
by Neil Martinsen-Burrell Nov. 2, 2007

Nov. 2, 2007

On 11/2/07 12:00, "numpy-discussion-request(a)scipy.org" <numpy-discussion-request(a)scipy.org> wrote: > Date: Fri, 2 Nov 2007 12:58:33 -0400 > From: Brian Blais <bblais(a)bryant.edu> > Subject: [Numpy-discussion] weird numpy/pickle problem > To: numpy-discussion(a)scipy.org > > I boiled it down to the code below. Can anyone reproduce (or not) > this error? Works for me on OS X 10.4.9, Python 2.5.0, numpy 1.0.4dev3977. -Neil -- "The theory of probabilities is at bottom nothing but common sense reduced to calculus." -- Pierre Simon de Laplace

1 0

Re: [Numpy-discussion] [Pythonmac-SIG] Problem with numpy on Leopard
by Brian Granger Nov. 2, 2007

Nov. 2, 2007

On 11/1/07, Bill Janssen <janssen(a)parc.com> wrote: > > > It's not entirely silly. This has been the advice given to app > > > developers on this list and the PyObjC list for years now. It's nice > > > to have a better system Python for quick scripts, but it's still the > > > System Python. It's Apples, for their stuff that uses Python. And it > > > is specific to OS 10.5. If you use it to build your app, your app is > > > pretty much guaranteed not to work when someone tries to run it on > > > 10.6, etc. > > > > At some level I agree - I haven't used the system python on Tiger for > > a long time now. But, that is not how Apple "sells" the python on > > their system. > > This is really interesting. For my apps, I use the system Python on > Tiger, and expect to do it again with Leopard. If I have to have a > specific version of an extension that's needed, I make sure the > directory where I keep it is on the sys.path of my application before > the standard directories. That seems to be about all that's required. > I'm not sure what all the FUD about the system Python is... There is little fear, uncertainty or doubt about this. The problem is that this represents a huge change of behavior for users making the transition to Leopard. On Tiger, a user could download any basically any python pacakge and do "python setup.py install" and things would work. Now under Leopard, this becomes "python setup.py install" + muck with PYTHONPATH or .pth files. I personally have no problem doing this, but many of the casual python users with whom I work will be tripped up by this....and they will come and ask me what to do....over and over again.... Brian

1 0

Re: [Numpy-discussion] numpy FFT memory accumulation
by Ray Schumacher Nov. 2, 2007

Nov. 2, 2007

At 10:57 PM 11/1/2007, Charles R Harris wrote: > > An additional complication is that I pass the numpy (or Numeric) > > array address to the ctypes library call so that the data is placed > > directly into the array from the call. I use the if/else end wrap > > logic to determine whether I need to do a split and copy if the new > > data wraps. > >OK. Hmm, I wonder if you would lose much by taking a straight forward >radix-2 fft and teaching it to use modular indices? Probably not worth the >trouble, but an fft tailored to a ring buffer might be useful for other >things. The problem is, I once compiled my own FFT dll to call from Python and it was 2x slower than FFTPACK - I'm not math-smart enough to make all of the caching and numerical shortcuts. That, and Intel's optimized FFT is 3x faster than FFTPACK... I may still try to do a "zoom/range FFT" which does not compute all bins, and maybe only with a sine transform, which (I think) should be sufficient to determine peak real frequency and maybe use fewer cycles. >Probably the easiest thing is to just copy the ring buffer out into >a linear array. I do that for the buffer-wrap condition, while simply assigning a slice (no copy) to the temp array otherwise. > > I used Numeric functions for the ~40% speed increase, but I don't > >I know that numarray was slow in creating small arrays, but is Numpy >really that bad compared to Numeric? I just saw the # of FFTs/sec go from 390 to 550 just by switching numpy to Numeric (Intel Core Duo). Add a timer to my previous code posts and see how your results look. For the mega-arrays a lot of the numpy developers work with it is much faster, and I now find Numeric is lacking many other niceties, like frombuffer(). Ray -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.15.18/1104 - Release Date: 11/1/2007 6:47 PM

1 0

vectorizing loops
by Mathew Yeates Nov. 2, 2007

Nov. 2, 2007

Anybody know of any tricks for handling something like z[0]=1.0 for i in range(100): out[i]=func1(z[i]) z[i+1]=func2(out[i]) ??

10 19

Re: [Numpy-discussion] Problem with numpy on Leopard
by Ned Deily Nov. 2, 2007

Nov. 2, 2007

In article <07Nov1.205537pst."57996"@synergy1.parc.xerox.com>, Bill Janssen <janssen(a)parc.com> wrote: > This is really interesting. For my apps, I use the system Python on > Tiger, and expect to do it again with Leopard. If I have to have a > specific version of an extension that's needed, I make sure the > directory where I keep it is on the sys.path of my application before > the standard directories. That seems to be about all that's required. > I'm not sure what all the FUD about the system Python is... For your own apps on your own machines, sure - you can manage the whole environment and the system Python should be fine (that is, if and until you need a newer build of Python itself). I think the "FUD" you refer to concerns a different problem: developing and distributing multiple-component Python apps to multiple users on multiple machines. Controlling all those environments isn't so easy. Deploying apps and components as eggs using setuptools is one way to simplify that problem; building standalone OSX apps with embedded stripped-down Python runtimes and components using py2app is another. -- Ned Deily, nad(a)acm.org

1 0

Re: [Numpy-discussion] [Pythonmac-SIG] Problem with numpy on Leopard
by Brian Granger Nov. 2, 2007

Nov. 2, 2007

> It's not entirely silly. This has been the advice given to app > developers on this list and the PyObjC list for years now. It's nice > to have a better system Python for quick scripts, but it's still the > System Python. It's Apples, for their stuff that uses Python. And it > is specific to OS 10.5. If you use it to build your app, your app is > pretty much guaranteed not to work when someone tries to run it on > 10.6, etc. At some level I agree - I haven't used the system python on Tiger for a long time now. But, that is not how Apple "sells" the python on their system. > Maybe they have changed how py2app works in 10.5, but the old behavior > was to embed Python into your app, UNLESS it was the system python. > Unless there is evidence to the contrary (I don't have Leopard yet), I > assume that is still true. > > The old advice still holds: For building apps, install your own > framework build of Python, put it first in your path, and add any > libraries you require to it. Then you have control over what goes > into your app, not Apple. > > --Dethe > _______________________________________________ > Pythonmac-SIG maillist - Pythonmac-SIG(a)python.org > http://mail.python.org/mailman/listinfo/pythonmac-sig >

1 0