Mailman 3 December 2007 - NumPy-Discussion

ising model: f2py vs cython comparison
by Ondrej Certik 23 Dec '07

23 Dec '07

Hi, I need to write 2D Ising model simulation into my school, so I wrote it in Python, then rewrote it in Fortran + f2py, and also Cython: http://hg.sharesource.org/isingmodel/ And Cython solution is 2x faster than f2py. I understand, that I am comparing many things - wrappers, my fortran coding skills vs Cython C code generation and gcc vs gfortran. But still any ideas how to speed up fortran+f2py solution is very welcomed. How to play with that - just do this (after installing Mercurial): $ hg clone http://hg.sharesource.org/isingmodel/ [...] $ hg up db7dd01cdc26 # just to be sure that we are talking about the same revision / code $ cd isingmodel $ make [...] $ time python simulate.py [...] real 0m2.026s user 0m1.988s sys 0m0.020s This runs Cython code. Then apply this patch to run fortran code instead: $ hg di diff -r db7dd01cdc26 simulate.py --- a/simulate.py Sun Dec 23 02:23:30 2007 +0100 +++ b/simulate.py Sun Dec 23 02:24:33 2007 +0100 @@ -31,8 +31,8 @@ def MC(mu = 1, temp = 2, dim = 20, steps J=1 #coupling constant k=1 #Boltzman constant - #from mcising import mc - from pyising import mc + from mcising import mc + #from pyising import mc B = D1(A) mc(B, dim, steps, temp, H, mu, J, k) return D2(B) And then again: $ time python simulate.py [...] real 0m3.600s user 0m3.528s sys 0m0.052s So it's a lot slower. Maybe I did some stupid mistake, like doing different amount of steps in Cython and Fortran, but I tried to check that and if I did everything all right, it's interesting to find why is fortran+f2py slower. Ondrej

3 4

Re: [Numpy-discussion] [SciPy-dev] SciPy Sprint results
by Travis E. Oliphant 22 Dec '07

22 Dec '07

Stefan van der Walt wrote: > Hi Travis, > > During the sprint I also merged Pierre's MaskedArray code into the > maskedarray branch. That is nearly done, with only a few unit tests > still failing -- ones brought over from the old numpy.ma. > > This is mainly due to some changes in the API, for example put and > putmask now behave like their ndarray counterparts. Pierre, would you > remind us of any other such changes, and why they were made? > > What is the road forward with this code? We will probably only merge > API changes into 1.4. Do we use svnmerge to keep the branch up to > date until then? > I'd like to move forward with it sooner (for 1.0.5) if the API changes are not drastic. Although ideally 0 API changes would be desireable, I'm not sure if that is feasible. Are put and putmask the only changes in the API. What are the rest of them? -Travis

4 8

new P3 binaries for NumPy 1.0.4 and SciPy 0.6.0
by Jarrod Millman 21 Dec '07

21 Dec '07

Hey, If you are having problems with NumPy and SciPy on Pentium III machines running Windows, please try the newly released binaries: numpy-1.0.4.win32-p3-py2.3.exe numpy-1.0.4.win32-p3-py2.4.exe numpy-1.0.4.win32-p3-py2.5.exe numpy-1.0.4.win32-p3-py2.5.msi scipy-0.6.0.win32-p3-py2.3.exe scipy-0.6.0.win32-p3-py2.4.exe scipy-0.6.0.win32-p3-py2.5.exe scipy-0.6.0.win32-p3-py2.5.msi Enjoy, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

3 2

setting items in a matrix
by devnew＠gmail.com 21 Dec '07

21 Dec '07

hi i am a beginner with numpy and python,so pardon me if this doubt seems silly i want to create a matrix with say 3 rows and 5 columns..and then set the values of each item in it .for this i did something like below myarray=zeros((3,5)) #then set the items for row in range(3): for col in range(5): myarray[row][col]=999999.9999 mymatrix=matrix(myarray) is this the way to do the matrix creation and value setting? is the use of zeros() unnecessary? i am in the early learning stage so your reply wd help me much dn

3 2

Loading a > GB file into array
by Martin Spacek 21 Dec '07

21 Dec '07

I need to load a 1.3GB binary file entirely into a single numpy.uint8 array. I've been using numpy.fromfile(), but for files > 1.2GB on my win32 machine, I get a memory error. Actually, since I have several other python modules imported at the same time, including pygame, I get a "pygame parachute" and a segfault that dumps me out of python: data = numpy.fromfile(f, numpy.uint8) # where f is the open file 1382400000 items requested but only 0 read Fatal Python error: (pygame parachute) Segmentation Fault If I stick to just doing it at the interpreter with only numpy imported, I can open up files that are roughly 100MB bigger, but any more than that and I get a clean MemoryError. This machine has 2GB of RAM. I've tried setting the /3GB switch on winxp bootup, as well as all the registry suggestions at http://www.msfn.org/board/storage-process-command-t62001.html. No luck. I get the same error in (32bit) ubuntu for a sufficiently big file. I find that if I load the file in two pieces into two arrays, say 1GB and 0.3GB respectively, I can avoid the memory error. So it seems that it's not that windows can't allocate the memory, just that it can't allocate enough contiguous memory. I'm OK with this, but for indexing convenience, I'd like to be able to treat the two arrays as if they were one. Specifically, this file is movie data, and the array I'd like to get out of this is of shape (nframes, height, width). Right now I'm getting two arrays that are something like (0.8*nframes, height, width) and (0.2*nframes, height, width). Later in the code, I only need to index over the 0th dimension, i.e. the frame index. I'd like to access all the data using a single range of frame indices. Is there any way to combine these two arrays into what looks like a single array, without having to do any copying within memory? I've tried using numpy.concatenate(), but that gives me a MemoryError because, I presume, it's doing a copy. Would it be better to load the file one frame at a time, generating nframes arrays of shape (height, width), and sticking them consecutively in a python list? I'm using numpy 1.0.4 (compiled from source tarball with Intel's MKL library) on python 2.5.1 in winxp. Thanks for any advice, Martin

13 29

segfault with atlas3-sse2, works with atlas3-sse
by Ondrej Certik 21 Dec '07

21 Dec '07

Hi, when compiled on Debian, numpy segfaults when used with ATLAS sse2, but works when used against ATLAS sse. More information here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=448530 I have no idea, where the problem is. Do you have any ideas, what we should try? Thanks a lot, Ondrej

4 5

Fwd: SSE2 segfault
by Ondrej Certik 21 Dec '07

21 Dec '07

Hi, forwarding more information about the problem + backtrace. Ondrej ---------- Forwarded message ---------- From: Jan Medlock <medlock(a)turboshower.net> Date: Dec 21, 2007 4:17 AM Subject: Re: Bug#448530: bug reproduced To: Ondrej Certik <ondrej(a)certik.cz> Cc: 448530(a)bugs.debian.org, Kumar Appaiah <akumar(a)ee.iitm.ac.in> On Thu, Dec 20, 2007 at 08:01:01PM +0100, Ondrej Certik wrote: > Paul Metcalfe has sent me this explanation: > > I think it's a packaging problem with linear algebra. > > ... Hi Ondrej, I'm still here. I saw the bug in the NumPy bug tracker that suggests the problem is specific to Debian & Ubuntu and Paul Metcalfe's details about linking. Linking against ATLAS instead of BLAS & LAPACK may indeed ultimately be the problem and I'm afraid I don't understand the details. That said, I believe the problem is SSE2 specific. The machine that I'm having the problem with is an Intel Pentium4 with SSE2, but I also have an AMD K7 machine that has 3DNow but not SSE2. The AMD machine does not exhibit the bug with the 3DNow version of ATLAS. (Of course neither do with the SSE or base versions of ATLAS.) I haven't participated in the thread on the NumPy mailing list because I don't see a way to reply to the thread since I wasn't subscribed at the time... Attached is a gdb backtrace. Please let me know how I can make it more informative, if necessary. Thanks again for tracking down this bug, Jan

1 0

Propose modification to binary_repr
by David Huard 20 Dec '07

20 Dec '07

Hi, The current behavior of numpy's binary_repr is the following: >>> binary_repr(1,width=2) '01' >>> binary_repr(0,width=2) '0' This seems inconsistent and I'd suggest always padding with zeros to make sure that the return string always has length=width. Objections ? David

8 9

Indexing into a matrix
by Keith Goodman 20 Dec '07

20 Dec '07

>> x = M.rand(3,1) <-------- 3x1 >> x[M.isfinite(x)] matrix([[ 0.36541551, 0.6305087 , 0.66054899]]) <---------- Should this be 3x1? My workaround is awkward: >> x[M.where(M.isfinite(x).A)[0]] matrix([[ 0.36541551], [ 0.6305087 ], [ 0.66054899]])

1 0

Compiler change for Windows version between 1.0.3.1 and 1.0.4?
by Stuart Brorson 20 Dec '07

20 Dec '07

Hi -- I have found a bug in numpy-1.0.4. The bug is a crash when doing linalg.inv() on a Windows machine. The machine is an old Pentium 3 machine. The error report indicates an invalid op code. Details about this bug are in the numpy tracker under ticket 635. Numpy-1.0.3.1 doesn't crash when running the same code. I installed the pre-built binaries available from numpy's SF site. Since the binaries were built by you, I can only suspect that the problem involves using a different compiler to build 1.0.4. I suspect the new version was built with a compiler which emitted op codes the old P3 chip couldn't handle. Can the developers confirm or deny my hypothesis? Thanks, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/

3 2