[Tutor] improving speed using and recalling C functions

[Martin A. Brown]

Gabriele,

>         21071736 function calls in 199.883 seconds

The 21 million function calls isn't really a surprise to me, given 
18 million calls to file.write().  Given that the majority of the 
time is still spent in skymaps5.py, I think you'll need to 
instrument that a bit more to figure out where the hotspot is.

>   Ordered by: internal time
>   List reduced from 188 to 10 due to restriction <10>
>
>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>        1  149.479  149.479  199.851  199.851 skymaps5.py:16(mymain)
> 18101000   28.682    0.000   28.682    0.000 {method 'write' of 'file'
> objects}
>
> the major time is required by mymain that is the whole program. 
> the write on file is an operation that I do in the end but if I 
> increase the number of data it doesn't increase (I tried doubing 
> the sample of values and I know why it's behaving in this way) the 
> third are interpolate and kappa: the two functions (one called 
> inside the other one)

This is a good finding, in fact.  Now, you know which module 
contains the bottleneck.  Is your CPU pegged when you run that 
skymap5.py code?  Can you run the profiler on the skymaps5.py code, 
too, to locate the specific hotspot?  Great to see that you are 
using the profiler, effectively, too!

> So they are the ones that are taking time.
> Now what can I do?

I think you now need to profile skymaps5.py code?  Specifically the 
stuff in main().

OK, so I have not myself used scipy.interpolate.interp1d before, but 
I went to have a look at it.  So, you feed interp1d() an x and a y 
(e.g. a plot line on a diagram), and it essentially produces its 
best guess of a function which will fit that curve, correct?

Well, this is how it performs with random input and an identity 
function:

   element count 100, duration 0.000
   element count 1000, duration 0.001
   element count 10000, duration 0.005
   element count 100000, duration 0.055
   element count 1000000, duration 0.858
   element count 10000000, duration 30.404

So, with 10 million inputs on an admittedly brain-dead function, 
there's not a performance bottleneck.  If you can find the parts of 
your skymaps5.py code that are the bottleneck, then you could post 
it here.

I hadn't thought of using interpolate, myself, as I didn't even know 
it existed.

Thanks and good luck,

-Martin

-- 
Martin A. Brown
http://linux-ip.net/