[Cython] AddTraceback() slows down generators

[mark florisson]

On 28 January 2012 18:38, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 27.01.2012 09:02:
>> any exception *propagation* is
>> still substantially slower than necessary, and that's a general issue.
>
> Here's a general take on a code object cache for exception propagation.
>
> https://github.com/scoder/cython/commit/ad18e0208
>
> When I raise an exception in test code that propagates through a Python
> call hierarchy of four functions before being caught, the cache gives me
> something like a 2x speedup in total. Not bad. When I do the same for cdef
> functions, it's more like 4-5x.
>
> The main idea is to cache the objects in a reallocable C array and bisect
> into it based on the C code "__LINE__" of the exception, which should be
> unique enough for a given module.
>
> It's a global cache that doesn't limit the lifetime of code objects  (well,
> up to the lifetime of the module, obviously). I don't know if that's a
> problem because the number of code objects is only bounded by the number of
> exception origination points in the C source code, which is usually quite
> large. However, only a tiny fraction of those will ever raise or propagate
> an exception in practice, so the real number of cached code objects will be
> substantially smaller.
>
> Maybe thorough test suites with lots of failure testing would notice a
> difference in memory consumption, even though a single code objects isn't
> all that large either...
>
> What do you think?
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Nice. I have a question, couldn't you save the frame object instead of
the code object?
I do think PyCodeObject is rather large, on my 64 bit machine it is
120 bytes, not accounting for any of the objects it holds (not saying
that's a problem, just pointing it out).

Would it help if we would pass in the position information string
object constant, to avoid the PyString_Format? That optimization would
only save 14% though. But additionally, the function name could be a
string object constant, which could be shared by all exceptions in one
function, avoiding another PyString_FromString.