[C++-sig] Managing the GIL across competing threads

[Niall Douglas]

The key to avoiding deadlocks is, in every case, the appropriate 
design.

In what you posted you appear to be doing too much at once, so you're 
holding onto too many locks at once. Either replace those with a 
single, master lock or try to never hold more than one lock at any 
one time. That includes the GIL. In well designed code, one almost 
NEVER holds more than two locks at any time.

Consider breaking object instantiation up into well defined stages. 
Consider throwing away your current implementation entirely and 
starting from scratch. Debugging a bad design will take longer and 
cost more than throwing it away and starting again with the benefit 
of hindsight.

Niall


On 16 Mar 2012 at 0:04, Adam Preble wrote:

> I edited the post so many times I forgot the subject line!  I am a bad
> person.
> 
> Maybe the most condensed question here is: How do you normally manage
> multiple resources competing with the GIL in a way that could cause
> deadlocks with it?  For the details, see the long post.
> 
> On Thu, Mar 15, 2012 at 11:59 PM, Adam Preble <adam.preble at gmail.com> wrote:
> 
> > I discovered recently that during callbacks to Python objects, sometimes
> > the interpreter would try to do stuff at the same time, despite the fact I
> > made a call to ensure the GIL.  I read the solution for that kind of thing
> > I'm doing is calling PyEval_InitThreads().  This worked at first, but like
> > more race conditions and stuff, all it takes is walking away from the
> > computer for it all to fall apart.
> >
> > What I'm seeing is a rather elaborate deadlock situation revolved around
> > securing the GIL.  I think I need to basically put my interpreter in its
> > own subsystem, design-wise, and ferret calls to invoke things in the
> > interpreter to it, in order to ultimately get around this.  However, what
> > I'm asking of the distribution is how they've gotten around this.
> >
> > To give something a little more concrete--this is pretty gross:
> > 1. Main thread starts interpreter and is running a script
> > 2. The script defines an implementation of callback interface A
> > 3. The script starts some objects that represent threads in the C++
> > runtime.  These are threads 1 and 2.
> > 4. The script starts to create an object that is wrapped from C++
> > 5. The object requires a resource from thread 1, where I use promises and
> > futures enqueue and fulfill the request when thread #1 isn't busy.
> > 6. Meanwhile, the interpreter thread is waiting for the resource since it
> > cannot put it off any further
> > 7. At this point, thread 2 tries to invoke the callback to interface A,
> > and it needs the interpreter thread.
> > 8. thread #1 needs thread #2 to complete this critical step before the
> > get() call will complete for the master interpreter thread
> > 9. Thread #2 needs thread #1 to finish so it can get the GIL.  It's
> > basically locked up in PyGILState_Ensure().
> >
> > Heck of a deadlock.  I am pondering having something else control the
> > interpreter in its own thread and have everybody enqueue stuff up to run in
> > it, like with the promises and futures I'm using elsewhere already.  The
> > reason is that thread #2 doesn't really need to steal the interpreter at
> > that very exact moment.  And furthermore, I'm trying to use Stackless, and
> > it's my understanding there I can link it up so that the interpreter gets
> > ahold of the Python runtime at controlled intervals--if desired--to invoke
> > other stuff.
> >
> >
> > _______________________________________________
> > Cplusplus-sig mailing list
> > Cplusplus-sig at python.org
> > http://mail.python.org/mailman/listinfo/cplusplus-sig
> >
> 


-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/