Re: [Edu-sig] Introducing Python to Engineering Students
At 03:12 PM 3/11/2008 -0700, Rob Malouf wrote:
On Mar 11, 2008, at 2:57 PM, David MacQuigg wrote:
I guess what I should conclude is that when performance is important, don't bother trying to optimize Python. Go straight to C, and get 10 or 100X improvement.
That hasn't always been my experience. I found that using psyco and numpy (along with careful profiling and optimization), I can often get essentially the same performance from pure Python that I can get from mixed Python/C solutions, and with much less trouble.
I agree that the Python/C interface is more trouble than it should be, especially for freshman engineering students and technical professionals who don't have time for messy programming details. What I would like to see is something like a simple "directive" I could put in my Python code to say "The following function is in C", and have Python set up the linkage for me. It would make a nice improvement in this Mandelbrot demo if you could show me a way to significantly improve the speed of the Python I already have, perhaps avoiding the need for C. I've posted the complete Python module at http://ece.arizona.edu/~edatools/ece175/projects/mandelbrots/mandel.py Just run the command $ python mandel.py and you should see the output from two examples in the doctests: 1a) Python speed = 787 points/sec 1b) C speed = 125500 points/sec 2a) Python speed = 823 points/sec 2b) C speed = 134300 points/sec The C source, mandelbrotC.c, is also in that same directory. The tests above were done on an old Windows machine running Cygwin. -- Dave
On Mar 11, 2008, at 5:11 PM, David MacQuigg wrote:
What I would like to see is something like a simple "directive" I could put in my Python code to say "The following function is in C", and have Python set up the linkage for me.
Actually, there is! It's part of scipy and it's *very* slick: http://www.scipy.org/Weave It works great, though as I said, for my problems it's not always faster than python+psyco+numpy (which I have to say I find disappointing, considering how cool weave is).
It would make a nice improvement in this Mandelbrot demo if you could show me a way to significantly improve the speed of the Python I already have, perhaps avoiding the need for C.
I'll take a look at it. --- Rob Malouf <rmalouf@mail.sdsu.edu> Department of Linguistics and Asian/Middle Eastern Languages San Diego State University
On Mar 11, 2008, at Mar 11:8:11 PM, David MacQuigg wrote:
At 03:12 PM 3/11/2008 -0700, Rob Malouf wrote:
On Mar 11, 2008, at 2:57 PM, David MacQuigg wrote:
I guess what I should conclude is that when performance is important, don't bother trying to optimize Python. Go straight to C, and get 10 or 100X improvement.
That hasn't always been my experience. I found that using psyco and numpy (along with careful profiling and optimization), I can often get essentially the same performance from pure Python that I can get from mixed Python/C solutions, and with much less trouble.
I agree that the Python/C interface is more trouble than it should be, especially for freshman engineering students and technical professionals who don't have time for messy programming details. What I would like to see is something like a simple "directive" I could put in my Python code to say "The following function is in C", and have Python set up the linkage for me.
One of the things that drew me to Python for Numerical work was Pyrex (and now the Cython project). Here is your code in Pyrex (sans-doc string) def getpts(double cx0,double cy0,double dx): vals = [] cdef int p,i cdef double cx,zx,zy,zx2,zy2 for p from 0<=p<100: # 100 points cx = cx0 + p * dx # moving right x . . . . . . zx = zy = 0.0 for i from 0<=i<1000: # 100 points zx2 = zx*zx zy2 = zy*zy if (zx2 + zy2) > 4.0 : break # escape zy = 2*zx*zy + cy0 zx = zx2 - zy2 + cx vals.append(i) return vals This code gets compiled to C, and returns this unit test: Expected: 1a) Python speed = 787 points/sec 1b) C speed = 125500 points/sec 2a) Python speed = 823 points/sec 2b) C speed = 134300 points/sec Got: 1a) Python speed = 1214 points/sec 1b) C speed = 125700 points/sec 2a) Python speed = 1287 points/sec 2b) C speed = 134500 points/sec not too shabby! And for only putting the types of the variable into the existing python code, and a slight modification of the for-loop syntax, the conversion of a slow python function into a fast extension module is trivial. One of the best things here is that for debugging, you can put in arbitrary python code. You take the performance hit, but you don't care during debugging, and you take it out again for production. bb -- Brian Blais bblais@bryant.edu http://web.bryant.edu/~bblais
On Mar 11, 2008, at 5:11 PM, David MacQuigg wrote:
It would make a nice improvement in this Mandelbrot demo if you could show me a way to significantly improve the speed of the Python I already have, perhaps avoiding the need for C.
Actually, I don't see a clean way to vectorize that inner loop, so maybe numpy isn't a good fit here. Which means weave is. First, running the program as-is, I get: 1a) Python speed = 776 points/sec 1b) C speed = 103200 points/sec 2a) Python speed = 833 points/sec 2b) C speed = 108600 points/sec With this added to the beginning of the program to load and invoke psyco: import psyco psyco.full() I get: 1a) Python speed = 2700 points/sec 1b) C speed = 101600 points/sec 2a) Python speed = 2800 points/sec 2b) C speed = 110100 points/sec Or, instead, replacing your getpts with the weave-d version: from numpy import zeros from scipy import weave ... vals = zeros(100,int) code = r""" double cx = cx0 - dx; for (int p=0; p<100; p++) { double zx = 0.0; double zy = 0.0; int i; cx += dx; for (i=0; i<999; i++) { double zx2 = zx*zx; double zy2 = zy*zy; if ((zx2 + zy2) > 4.0) break; zy = 2*zx*zy + cy0; zx = zx2 - zy2 + cx; } vals[p] = i; } """ weave.inline(code,['cx0','cy0','dx','vals']) I get: 1a) Python speed = 102200 points/sec 1b) C speed = 103300 points/sec 2a) Python speed = 108400 points/sec 2b) C speed = 110700 points/sec Not bad! There's probably a more pythonic way to write that, but this'll do. And after all, this is a realistic situation: you've got the inner loop of a program written in C, but you'd rather write all the supporting code around it in something like Python. And note that weave automatically compiles and links (if necessary) and then loads the C part when you run mandel.py. You (or your students) don't need to remember where the python headers are or how to build a shared library on whatever platform they're using. --- Rob Malouf <rmalouf@mail.sdsu.edu> Department of Linguistics and Asian/Middle Eastern Languages San Diego State University
At 07:24 PM 3/11/2008 -0700, Rob Malouf wrote:
On Mar 11, 2008, at 5:11 PM, David MacQuigg wrote:
It would make a nice improvement in this Mandelbrot demo if you could show me a way to significantly improve the speed of the Python I already have, perhaps avoiding the need for C. ... Or, instead, replacing your getpts with the weave-d version:
from numpy import zeros from scipy import weave ... vals = zeros(100,int)
code = r""" double cx = cx0 - dx; for (int p=0; p<100; p++) { double zx = 0.0; double zy = 0.0; int i; cx += dx; for (i=0; i<999; i++) { double zx2 = zx*zx; double zy2 = zy*zy; if ((zx2 + zy2) > 4.0) break; zy = 2*zx*zy + cy0; zx = zx2 - zy2 + cx; } vals[p] = i; } """ weave.inline(code,['cx0','cy0','dx','vals'])
I get:
1a) Python speed = 102200 points/sec 1b) C speed = 103300 points/sec 2a) Python speed = 108400 points/sec 2b) C speed = 110700 points/sec
Excellent!!! This is exactly what we need. Our students will have no problem writing the C code string, and the Python wrappings are simple enough they can just follow this example verbatim.
There's probably a more pythonic way to write that, but this'll do. And after all, this is a realistic situation: you've got the inner loop of a program written in C, but you'd rather write all the supporting code around it in something like Python. And note that weave automatically compiles and links (if necessary) and then loads the C part when you run mandel.py. You (or your students) don't need to remember where the python headers are or how to build a shared library on whatever platform they're using.
That was my next problem. Our students use mostly Windows, some on MacOS, and a few on Linux. If all we need beyond the latest Python is numpy and scipy, that is great!! I'll have to experiment with this later. Many thanks. -- Dave
I got scipy.weave working in my Mandelbrot demo in Python2.5 under Cygwin, and the speed improvement is comparable to my hand-coded <Pyton.h> extension module. 1a) Python speed = 678 points/sec 1b) C speed = 115200 points/sec 169X 2a) Python speed = 721 points/sec 2b) C speed = 126400 points/sec 175X See http://ece.arizona.edu/~edatools/ece175/projects/mandelbrots/mandel_weave.py for the complete program, including both the Python and C-coded versions of the getpts function. It should run now in any Python setup with the required packages (numpy and weave). As Rob pointed out, this eliminates all the <Python.h> interface functions, the need to build a shared library for every little test, and the clutter in my site-packages directory from all these tests. Now, I can just tell the students "write your own C code, and put it in the stub function in this Python module." The stub function runs 10X faster than the C speed above, so we can be confident the Python overhead is negligible. At 09:30 PM 3/13/2008 -0700, Rob Malouf wrote:
Ugh! I'm afraid I don't use windows, so I can't offer any advice. I do know that compiling scipy wasn't that difficult under linux or mac os x, so it shouldn't be too bad under cygwin either.
Weave and Numpy compiled without any snags, and that is all I need for now.
There are few extra libraries you need, but you can probably find binaries for them out there somewhere. In fact, I'm surprised there isn't a cygwin binary for scipy... if you get it working maybe you should post it somewhere!
Or at least some instructions that are more clear on building such a binary. Many thanks for all the help. -- Dave At 07:24 PM 3/11/2008 -0700, Rob Malouf wrote:
On Mar 11, 2008, at 5:11 PM, David MacQuigg wrote:
It would make a nice improvement in this Mandelbrot demo if you could show me a way to significantly improve the speed of the Python I already have, perhaps avoiding the need for C.
Actually, I don't see a clean way to vectorize that inner loop, so maybe numpy isn't a good fit here. Which means weave is.
First, running the program as-is, I get:
1a) Python speed = 776 points/sec 1b) C speed = 103200 points/sec 2a) Python speed = 833 points/sec 2b) C speed = 108600 points/sec
With this added to the beginning of the program to load and invoke psyco:
import psyco psyco.full()
I get:
1a) Python speed = 2700 points/sec 1b) C speed = 101600 points/sec 2a) Python speed = 2800 points/sec 2b) C speed = 110100 points/sec
Or, instead, replacing your getpts with the weave-d version:
from numpy import zeros from scipy import weave ... vals = zeros(100,int)
code = r""" double cx = cx0 - dx; for (int p=0; p<100; p++) { double zx = 0.0; double zy = 0.0; int i; cx += dx; for (i=0; i<999; i++) { double zx2 = zx*zx; double zy2 = zy*zy; if ((zx2 + zy2) > 4.0) break; zy = 2*zx*zy + cy0; zx = zx2 - zy2 + cx; } vals[p] = i; } """ weave.inline(code,['cx0','cy0','dx','vals'])
I get:
1a) Python speed = 102200 points/sec 1b) C speed = 103300 points/sec 2a) Python speed = 108400 points/sec 2b) C speed = 110700 points/sec
Not bad! There's probably a more pythonic way to write that, but this'll do. And after all, this is a realistic situation: you've got the inner loop of a program written in C, but you'd rather write all the supporting code around it in something like Python. And note that weave automatically compiles and links (if necessary) and then loads the C part when you run mandel.py. You (or your students) don't need to remember where the python headers are or how to build a shared library on whatever platform they're using.
--- Rob Malouf <rmalouf@mail.sdsu.edu> Department of Linguistics and Asian/Middle Eastern Languages San Diego State University
participants (4)
-
Brian Blais -
David MacQuigg -
Rob Malouf -
Rob Malouf