[Tutor] how to optimize this code?

[Emile van Sebille]

On 3/28/2011 12:37 PM Albert-Jan Roskam said...
> Hi Stefan,
>
> Thanks for your advice. I seriously thought ctypes was the module to
> use. That was before I found out the evaluating all 10**9 values of my
> test data set is glacially slow (several hours). You're right, the dll
> implies the program is running on windows. I've also been trying to make
> it work under Linux but I wanted to get the basic algorithm right first.
> Also, it was quite a PIA to get all the dependencies of the (old) .so files.
>
> Your speed tip reminded me of:
> http://wiki.python.org/moin/PythonSpeed/Performan ceTips#Avoiding_dots
> <http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Avoiding_dots>...
> Does this mean that "from ctypes import *"

That's not necessary.  You can also make them local by performing the 
name lookup only once:

   import ctypes
   myXxx = ctypes.Xxx

Then use myXxx going forward.


> gives slightly faster code
> than "import ctypes"? If so: wow! I've always avoided the first notation
> like the plague.
>
> What do you mean with '... using a constant pointer for numValue' ?

It looks like your function can reuse a once-set value where you set 
numValue outside your function and refer to it.

Emile



> Is
> this the byref/pointer object distinction? I replaced a the pointer
> object with a byref object, which reduced processing time by about 10 %.
>
> Cython might be interesting as a hobby project, but I'm affraid I'll
> never get the ICT droids in my office to install that.
>
> Cheers!!
> Albert-Jan
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> All right, but apart from the sanitation, the medicine, education, wine,
> public order, irrigation, roads, a fresh water system, and public
> health, what have the Romans ever done for us?
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
> *From:* Stefan Behnel <stefan_ml at behnel.de>
> *To:* tutor at python.org
> *Sent:* Mon, March 28, 2011 7:43:16 AM
> *Subject:* Re: [Tutor] how to optimize this code?
>
> Albert-Jan Roskam, 27.03.2011 21:57:
>  > I made a program that reads spss data files. I ran cProfile to see if
> I can
>  > optimize things (see #1 below).
>
> First thing to note here: sort the output by "time", which refers to the
> "tottime" column. That will make it more obvious where most time is
> really spent.
>
>
>  > It seems that the function getValueNumeric is a pain spot (see #2
>  > below). This function calls a C function in a dll for each numerical
>  > cell value. On the basis of this limited amount of info, what could I do
>  > to further optimize the code? I heard about psyco, but I didn't think
>  > such tricks would be necessary as the function spssGetValueNumeric is is
>  > implemented in C already (which should be fast).
>
> The problem is that you are using ctypes to call it. It's useful for
> simple things, but it's not usable for performance critical things, such
> as calling a C function ten million times in your example. Since you're
> saying "dll", is this under Windows? It's a bit more tricky to set up
> Cython on that platform than on pretty much all others, since you
> additionally need to install a C compiler, but if you want to go that
> route, it will reward you with a much faster way to call your C code,
> and will allow you to also speed up the code that does the calls.
>
> That being said, see below.
>
>
>  > ## most time consuming function
>  >
>  > def getValueNumeric(fh, spssio, varHandle):
>  > numValue = ctypes.c_double()
>  > numValuePtr = ctypes.byref(numValue)
>  > retcode = spssio.spssGetValueNumeric(fh,
>  > &n bsp; ctypes.c_double(varHandle),
>  > numValuePtr)
>
> You may still be able to make this code a tad faster, by avoiding the
> function name lookups on both the ctypes module and "spssio", and by
> using a constant pointer for numValue (is you're not using threads).
> That may not make enough of a difference, but it should at least be a
> little faster.
>
> Stefan
>
> _______________________________________________
> Tutor maillist - Tutor at python.org <mailto:Tutor at python.org>
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor