[Python-Dev] Bug in generator if the generator in created in a C thread

[Tim Lesher]

On Thu, Mar 29, 2012 at 17:26, Victor Stinner <victor.stinner at gmail.com> wrote:
> The problem is not the frame, but the Python thread state referenced by the
> frame. It's a private attribute. My patch just updates this reference before
> running the generator (and it clears the reference when the generator
> excution is stopped or finished).

Right--my thought was the the situation we saw might be similarly
triggered because we were storing an exception with traceback (and
associated frames) generated by thread A, and then re-throwing it from
thread B some time after thread A has exited.  The frame attached to
the exception's traceback would still, then, be referencing a
nonexistent thread state, correct?

My concern was that this might one instance of a broader problem for
folks who embed Python and see the attractive PyGILState_Ensure() API.
It already prevents us from using subinterpreters (which for us might
have been a better solution than repeated initialize/finalize, with
its known issues).

We recently made a change to dispose of the traceback before storing
the exception, and that appears to eliminate the corruption we were
seeing, so that's making me more suspicious.

> You may leak memory if your threads have a short lifetime and you create
> many threads. For example if one thread is only used to process one request
> and then is destroyed.

Absolutely--this particular hack was only used for a thread created
outside Python that had to call into the VM. Their behavior is
well-defined in our case--two particular OS threads, with lifetimes
longer than those of the interpreters we create and finalize.

-- 
Tim Lesher <tlesher at gmail.com>