Python extension performance

[David Jones]

Jack Diederich wrote:

> On Fri, Apr 08, 2005 at 10:14:52PM -0400, David Jones wrote:
> 
>>I am trying to hunt down the difference in performance between some raw 
>>C++ code and calling the C++ code from Python.  My goal is to use Python 
>>to control a bunch of number crunching code, and I need to show that 
>>this will not incur a (big) performance hit.
>>
>>My C++ function (testfunction) runs in 2.9 seconds when called from a 
>>C++ program, but runs in 4.3 seconds when called from Python. 
>>testfunction calculates its own running time with calls to clock(), and 
>>this is for only one iteration, so none of the time is in the SWIG code 
>>or Python.
>>
>>Some potential causes of my problems:
>>
>>- linking to a shared library instead building a static exe.
>>- intel libraries are not being used when I think they are
>>- libpython.so was built with gcc, so I am getting some link issues
>>- can linking to python affect my memory allocation and deallocation in 
>>c++??
> 
> The main overhead of calling C/C++ from python is the function call overhead
> (python creating the stack frame for the call, and then changing the python
> objects into regular ints, char *, etc).  You don't mention how many times
> you are calling the function.  If it is only once and the difference is 1.4
> seconds then something is really, really, messed up.  So I'll guess it is
> hundreds of thousands of times?  Let us know.

Sorry I was not clearer above;  the function is only called one time.  I 
have run out of obvious things I may have screwed up.  The part that 
bugs me most is that these are built from the same .o files except for 
the .o file that has the wrapper function for python.

> 
>>Some things I can try:
>>- recompile python with the intel compiler and try again
>>- compile my extension into a python interpreter, statically
>>- segregate the memory allocations from the numerical work and compare 
>>how the C++ and Python versions compare
> 
> Recompiling with the Intel compiler might help, I hear it is faster than 
> GCC for all modern x86 platforms.  I think CPython is only tested on GCC
> and windows Visual-C-thingy so you might be SOL.  The other two ideas
> seem much harder to do and less likely to show an improvement.
 >
 > -jackdied
 >

By the second option, I meant to compile my extension statically instead 
of using a shared library by unpacking the source rpm and putting my 
code in the Modules/ directory.  That is a pretty standard thing to do, 
isn't it?


Thanks for the comments.

Dave