Mailman 3 November 2008 - NumPy-Discussion

array indexing question
by Catherine Moroney 14 Nov '08

14 Nov '08

Hello, I know that there must be a fast way of solving this problem, but I don't know what it is. I have three arrays, with dimensions: A[np] L[np] S[np] where L and S indicate the line, smp co-ordinates for each of the "np" rows. I want to reconstruct the contents of [A] as a 2-dimensional matrix. The brain-dead version of what I want is: results = numpy.zeros(some_size, some_other_size) for idata in xrange(0,np): results[L[idata], S[idata]] = A[idata] but I'm dealing with large arrays and this is very slow. How do I speed it up? The array A is the result of performing kmeans clustering on a large 2-d image, presented as a 1-d vector and I want the result as a 2-d image. In the real application, I won't have data for each point in the 2-d image, so np will be less than the full image size. Catherine

2 1

numpy, Py_ssize_t, cython and 64 bits python 2.4
by David Cournapeau 14 Nov '08

14 Nov '08

Hi, I took a quick look at two bugs in scipy.spatial and scipy.special, linked to cython and 64 bits on python 2.4: http://scipy.org/scipy/scipy/ticket/785 At first, I was confused by the (runtime) error message given by cython; Py_ssize_t is a feature added in 2.5. The definition when built against python 2.4 is coming from numpy, which defines Py_ssize_t in numpy/ndarrayobject, and defines it to an int in that case (as recommended in PEP 353). Is this a cython limitation (in which case I will move this discussion to cython-dev), or is there a solution to support 64 bits python 2.4 in numpy/scipy ? cheers, David

5 17

Numpy and MKL, update
by Matthieu Brucher 14 Nov '08

14 Nov '08

Hi, I got an answer from the Intel Support about the issues between MKL and Numpy (and Matlab and ...). To use MKL with Numpy, we must know explicitely use the static MKL libraries (libmkl_intel_lp64.a, libmkl_intel_thread.a and libmkl_core.a). The same applies for Scipy and every other Python module you build with the MKL. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher

6 9

Segfault with dotblas on OS X 10.5.5/PPC (but not on Intel?)
by David Warde-Farley 13 Nov '08

13 Nov '08

Hello folks, I'm doing some rather big matrix products on a G5, and ran into this. Strangely on the same OS version on my Intel laptop, this isn't an issue. Available memory isn't the problem either, I don't think, this machine is pretty beefy. I'm running the python.org 2.5.2 build of Python, and the latest SVN build of numpy (though the same thing happened with 1.1.0). Here's the debug info that OS X gives me: Process: Python [84664] Path: /Library/Frameworks/Python.framework/Versions/2.5/ Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: ??? (???) Code Type: PPC (Native) Parent Process: bash [79733] Date/Time: 2008-11-12 16:21:39.983 -0500 OS Version: Mac OS X 10.5.5 (9F33) Report Version: 6 Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x00000000129f0ca0 Crashed Thread: 2 Thread 0: 0 libSystem.B.dylib 0x9267ae4c __semwait_signal + 12 1 libBLAS.dylib 0x936fd58c ATL_join_tree + 28 2 libBLAS.dylib 0x936f21e8 ATL_dptgemm + 296 3 _dotblas.so 0x00fd67f0 dotblas_matrixproduct + 5040 (_dotblas.c:751) ... ... <snip, snip, don't think most of this is relevant> Thread 1: 0 libSystem.B.dylib 0x926d8ae4 select$DARWIN_EXTSN + 12 1 com.tcltk.tcllibrary 0x020e8de8 NotifierThreadProc + 448 Thread 2 Crashed: 0 libBLAS.dylib 0x93192d40 ATL_dgezero + 96 1 libBLAS.dylib 0x93301984 ATL_dNCmmIJK + 1684 2 libBLAS.dylib 0x932ffcb4 ATL_dgemm + 708 3 libBLAS.dylib 0x936f1dfc ATL_dptgemm0 + 108 4 libSystem.B.dylib 0x926b6658 _pthread_start + 316 Thread 3: 0 libBLAS.dylib 0x931b9fa8 ATL_dJIK0x0x0NN0x0x0_aX_bX + 2328 1 libBLAS.dylib 0x93302314 ATL_dNCmmJIK + 1652 2 libBLAS.dylib 0x932ffcb4 ATL_dgemm + 708 3 libBLAS.dylib 0x936f1dfc ATL_dptgemm0 + 108 4 libSystem.B.dylib 0x926b6658 _pthread_start + 316 Thread 4: 0 libBLAS.dylib 0x93192d70 ATL_dgezero + 144 1 libBLAS.dylib 0x93302314 ATL_dNCmmJIK + 1652 2 libBLAS.dylib 0x932ffcb4 ATL_dgemm + 708 3 libBLAS.dylib 0x936f1dfc ATL_dptgemm0 + 108 4 libSystem.B.dylib 0x926b6658 _pthread_start + 316 Thread 5: 0 libBLAS.dylib 0x93307440 ATL_drow2blkT_KB_a1 + 80 1 libBLAS.dylib 0x93307754 ATL_drow2blkT2_a1 + 324 2 libBLAS.dylib 0x93305928 ATL_dmmJIK + 2200 3 libBLAS.dylib 0x932ff2a8 ATLU_dusergemm + 104 4 libBLAS.dylib 0x932ffcb4 ATL_dgemm + 708 5 libBLAS.dylib 0x936f1dfc ATL_dptgemm0 + 108 6 libSystem.B.dylib 0x926b6658 _pthread_start + 316

3 4

f2py: Can anyone tell me why this simple call-back example fails?
by Dave Lang 13 Nov '08

13 Nov '08

My system...... Ubuntu v8.0.4; Gnu gcc v4.0.3; g95 v0.92 Python v2.5.2; Numpy v1.2.1; f2py v2.5972 Hope someone can see what's wrong here? thanks Dave Lang ..............Here is the F90 code........... ! PROGRAM TO DEMONSTRATE PYTHON-->F90 AND F90-->PYTHON INTERFACE SUBROUTINE CBSET(N, call_back) !f2py intent(in) N !f2py intent(callback) call_back EXTERNAL call_back INTEGER N, m ! IDENTIFY THAT FORTRAN IS EXECUTING & WHO CALLED CBSET PRINT *,'Enter CBSET.F, N=',N IF(N .EQ. 1) THEN PRINT *,' in CBSET: This Call originated to test call-back' PRINT *,' in CBSET: Make callback to PYTHON...EXPECT reply' m=1 CALL call_back(m) PRINT *,' in CBSET: Returned from call-back to PYTHON' END IF IF(N .EQ. 2) THEN PRINT *,' in CBSET: Call came from Python to test Py->For' END IF RETURN END ..............Here is the f2py created Signature code........... ! -*- f90 -*- ! Note: the context of this file is case sensitive. python module cbset__user__routines interface cbset_user_interface subroutine call_back(m) ! in :foo:CBSET.F90:cbset:unknown_interface intent(callback) call_back integer :: m end subroutine call_back end interface cbset_user_interface end python module cbset__user__routines python module foo ! in interface ! in :foo subroutine cbset(n) ! in :foo:CBSET.F90 use cbset__user__routines integer intent(in) :: n intent(callback) call_back external call_back end subroutine cbset end interface end python module foo ! This file was auto-generated with f2py (version:2_5972). ! See http://cens.ioc.ee/projects/f2py2e/ ..............Here is the Python Main Program code........... # Main Python program to test nominal "wrap" and "call back" import foo def main(): print 'HELLO, pymain is about to call CBSET to test f2py operation' print 'BUT, 1st test call_back subr from within python itself' call_back(979) #test "Python --> FORTRAN" (nominal wrap) from Python print 'NOW, call CBSET to test Python --> FORTRAN wrap' n = 2 foo.cbset(n, call_back) #test "FORTRAN --> Python" (call-back) scheme as seen from Python print 'NOW, call CBSET to test FORTRAN --> Python call-back' n = 1 foo.cbset(n, call_back) # define call-back routine def call_back(m): m +=20 print '..Confirming call_back, Python is entered from Fortran' print '..in call_back: m = ', m return # end of call-back defn if __name__ =='__main__': main() ..............Here is the execution result........... david@david-desktop:~/Pyth/ACB$ python pymain.py HELLO, pymain is about to call CBSET to test f2py operation BUT, 1st test call_back subr from within python itself ..Confirming call_back, Python is entered from Fortran ..in call_back: m = 999 NOW, call CBSET to test Python --> FORTRAN wrap Enter CBSET.F, N= 2 in CBSET: Call came from Python to test Py->For NOW, call CBSET to test FORTRAN --> Python call-back Enter CBSET.F, N= 1 in CBSET: this Call originated to test call-back in CBSET: Make callback to PYTHON...EXPECT reply Segmentation fault

1 0

f2py questions
by Catherine Moroney 13 Nov '08

13 Nov '08

Hello, I'm not sure if this is the right list for this question, but attempts to subscribe myself to f2py-users fail, as I never get an email back asking me to confirm my subscription request. Is that list no longer active? I'm trying to compile a medium-short Fortran90 code using f2py, and get the error below. The code compiles fine under the gfortran compiler, which I'm specifying on the f2py command-line. I'm using numpy 1.2.1, and python 2.5 on a 64-bit Linux AMD box, with the command: >>f2py --verbose --f90exec='/usr/bin/gfortran' --f90flags='--fixed- line-length-none' \ -c -m M23mod M23match.f File "/usr/lib64/python2.5/site-packages/numpy/f2py/crackfortran.py", line 419, in readfortrancode dowithline(finalline) File "/usr/lib64/python2.5/site-packages/numpy/f2py/ crackfortran.py", line 601, in crackline % (groupcounter) crackline: groupcounter(=0) is nonpositive. Check the blocks. Can somebody point me to the right place to ask this question, and/or give me some hints about tracking down this error. Is there someway to get f2py to give me the line number of the problematic statement in the Fortran code? Catherine

1 0

Changes to histogram semantics: follow-up
by David Huard 13 Nov '08

13 Nov '08

NumPy users, Revision 6020 proceeds with the planned changes to histogram semantics for the 1.3 release. This modification brings no change in functionality, only changes in the warnings being raised: No warning is printed for the default behaviour (new=None). new=False now raises a DeprecationWarning. Users relying on the old behaviour are encouraged to switch to the new semantics. new=True warns users that the `new` keyword will disappear in 1.4 Regards, David Huard

2 1

Solved Re: Bugs in histogram and matplotlib-hist
by Mike Ressler 13 Nov '08

13 Nov '08

On Wed, Nov 12, 2008 at 2:09 PM, David Huard <david.huard(a)gmail.com> wrote: > > Mike, before patching, please take a look at the tickets related to > histogram on the numpy trac. Previously, histogram returned only used the > left bin edges and it caused a lot of problems with outliers and > normalization. We are not going back there. Our immediate problem has been solved by compiling matplotlib-0.98.3 by hand on Fedora 9 - in spite of its bleeding edge reputation, Fedora can get behind sometimes. The problem with histogram I reported also seems to have disappeared after completely uninstalling and reinstalling numpy and all its brethren. Something seems to have been weird about the recent Fedora updates. I did not mean to suggest any changes to the behavior of histogram, only to get it working on our system. Anyway, problem solved, we're happy. Sorry for the noise. (And thanks again Scott for the Fedora problem link.) Mike -- mike.ressler(a)alum.mit.edu

1 0

Re: [Numpy-discussion] Bugs in histogram and matplotlib-hist
by Scott Sinclair 13 Nov '08

13 Nov '08

> "Mike Ressler" <mike.ressler(a)alum.mit.edu> 11/12/08 1:19 AM > I did an update to a Fedora 9 workstation yesterday that included > updating numpy to 1.2.0 and matplotlib 0.98.3 (python version is > 2.5.1). This seems to have played havoc with some of the histogram > plotting we do. I was aware of the histogram changes in 1.2.0, but > something doesn't seem to have worked out right. Hi Mike, Someone else just posted regarding this problem on the matplotlib list http://www.nabble.com/histogram-examples--to20443846.html They reported that the Fedora 9 matplotlib package is 0.91.4, which doesn't work with numpy-1.2.0. Perhaps the matplotlib on your system isn't what you expect? It looks as if there's a problem with the Fedora packaging, the current release of matplotlib is 0.98.3. Regards, Scott Please find our Email Disclaimer here: http://www.ukzn.ac.za/disclaimer/

3 2

setting element
by Charles سمير Doutriaux 12 Nov '08

12 Nov '08

Hello, I'm wondering if there's aquick way to do the following: s[:,5]=value in a "general" function def setval(array,index,value,axis=0): ## code here The issue is to put enough ":" before the index value inside the square bracket of the assignement. Thanks, C.

4 6