[Cython] Generators & closure optimization

[Stefan Behnel]

Vitja Makarov, 21.12.2011 19:48:
> Some time ago we were talking about generators optimization by copying
> local variables from closure into local scope.

Yes, I think that will make it easier for the C compiler to make optimistic 
assumptions about external values.


> Now I think that should be a good idea to implement this for both
> generators and regular closure functions. So local var will be used
> for reference and assignment should be made to both copies.  Of course
> there are some variables that shouldn't be copied: non-local vars,
> arrays, C++ classes and structures.

Basically, anything that external code can modify. That makes it a bit 
tricky to do it also for 'normal' closure functions - the whole idea is 
that there is more than one function that can refer to a variable.


> Also it may be a good idea to move outer scope pointer into local variable.
>
> So I'm wondering what is a good test to measure actual speedup?

http://blog.behnel.de/index.php?p=163

Just take the plain Python versions of the iterparse functions and compare 
them before and after the change. The raw C implementation in CPython gives 
a good baseline.

Actually, it would be generally interesting to run the Cython versions 
through callgrind to see where the time is actually being spent.

Stefan