Re: [C++-sig] Managing the GIL across competing threads
I edited the post so many times I forgot the subject line! I am a bad person. Maybe the most condensed question here is: How do you normally manage multiple resources competing with the GIL in a way that could cause deadlocks with it? For the details, see the long post. On Thu, Mar 15, 2012 at 11:59 PM, Adam Preble <adam.preble@gmail.com> wrote:
I discovered recently that during callbacks to Python objects, sometimes the interpreter would try to do stuff at the same time, despite the fact I made a call to ensure the GIL. I read the solution for that kind of thing I'm doing is calling PyEval_InitThreads(). This worked at first, but like more race conditions and stuff, all it takes is walking away from the computer for it all to fall apart.
What I'm seeing is a rather elaborate deadlock situation revolved around securing the GIL. I think I need to basically put my interpreter in its own subsystem, design-wise, and ferret calls to invoke things in the interpreter to it, in order to ultimately get around this. However, what I'm asking of the distribution is how they've gotten around this.
To give something a little more concrete--this is pretty gross: 1. Main thread starts interpreter and is running a script 2. The script defines an implementation of callback interface A 3. The script starts some objects that represent threads in the C++ runtime. These are threads 1 and 2. 4. The script starts to create an object that is wrapped from C++ 5. The object requires a resource from thread 1, where I use promises and futures enqueue and fulfill the request when thread #1 isn't busy. 6. Meanwhile, the interpreter thread is waiting for the resource since it cannot put it off any further 7. At this point, thread 2 tries to invoke the callback to interface A, and it needs the interpreter thread. 8. thread #1 needs thread #2 to complete this critical step before the get() call will complete for the master interpreter thread 9. Thread #2 needs thread #1 to finish so it can get the GIL. It's basically locked up in PyGILState_Ensure().
Heck of a deadlock. I am pondering having something else control the interpreter in its own thread and have everybody enqueue stuff up to run in it, like with the promises and futures I'm using elsewhere already. The reason is that thread #2 doesn't really need to steal the interpreter at that very exact moment. And furthermore, I'm trying to use Stackless, and it's my understanding there I can link it up so that the interpreter gets ahold of the Python runtime at controlled intervals--if desired--to invoke other stuff.
_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org http://mail.python.org/mailman/listinfo/cplusplus-sig
The key to avoiding deadlocks is, in every case, the appropriate design. In what you posted you appear to be doing too much at once, so you're holding onto too many locks at once. Either replace those with a single, master lock or try to never hold more than one lock at any one time. That includes the GIL. In well designed code, one almost NEVER holds more than two locks at any time. Consider breaking object instantiation up into well defined stages. Consider throwing away your current implementation entirely and starting from scratch. Debugging a bad design will take longer and cost more than throwing it away and starting again with the benefit of hindsight. Niall On 16 Mar 2012 at 0:04, Adam Preble wrote:
I edited the post so many times I forgot the subject line! I am a bad person.
Maybe the most condensed question here is: How do you normally manage multiple resources competing with the GIL in a way that could cause deadlocks with it? For the details, see the long post.
On Thu, Mar 15, 2012 at 11:59 PM, Adam Preble <adam.preble@gmail.com> wrote:
I discovered recently that during callbacks to Python objects, sometimes the interpreter would try to do stuff at the same time, despite the fact I made a call to ensure the GIL. I read the solution for that kind of thing I'm doing is calling PyEval_InitThreads(). This worked at first, but like more race conditions and stuff, all it takes is walking away from the computer for it all to fall apart.
What I'm seeing is a rather elaborate deadlock situation revolved around securing the GIL. I think I need to basically put my interpreter in its own subsystem, design-wise, and ferret calls to invoke things in the interpreter to it, in order to ultimately get around this. However, what I'm asking of the distribution is how they've gotten around this.
To give something a little more concrete--this is pretty gross: 1. Main thread starts interpreter and is running a script 2. The script defines an implementation of callback interface A 3. The script starts some objects that represent threads in the C++ runtime. These are threads 1 and 2. 4. The script starts to create an object that is wrapped from C++ 5. The object requires a resource from thread 1, where I use promises and futures enqueue and fulfill the request when thread #1 isn't busy. 6. Meanwhile, the interpreter thread is waiting for the resource since it cannot put it off any further 7. At this point, thread 2 tries to invoke the callback to interface A, and it needs the interpreter thread. 8. thread #1 needs thread #2 to complete this critical step before the get() call will complete for the master interpreter thread 9. Thread #2 needs thread #1 to finish so it can get the GIL. It's basically locked up in PyGILState_Ensure().
Heck of a deadlock. I am pondering having something else control the interpreter in its own thread and have everybody enqueue stuff up to run in it, like with the promises and futures I'm using elsewhere already. The reason is that thread #2 doesn't really need to steal the interpreter at that very exact moment. And furthermore, I'm trying to use Stackless, and it's my understanding there I can link it up so that the interpreter gets ahold of the Python runtime at controlled intervals--if desired--to invoke other stuff.
_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org http://mail.python.org/mailman/listinfo/cplusplus-sig
-- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
Well the current design does have a problem, but I think it's more to do with the Python side than the sum of the whole thing. Most of my threads are basically subsystems in their own rights, and whatever that subsystem manage is encapsulated inside it. I get into this little spats when I need to request a resource from one or more of them--not necessarily from one subsystem to another, just in my main script I have to create an object that involves a piece from one and a piece from another. The subsystems will normally get to them when they are not in their own critical sections. I just got bit when they both went through what is essentially an unprotected interpreter. So I assume I would satisfy your suggestion of a master lock if I basically put the interpreter in its own subsystem. Everybody then gets their turn at it. Something I'm curious about has to do with the busy what in my promises and future implementation. When I end up waiting on something, is there a way right now with Python to give up the GIL until the wait is done? If I were to do a Release before the wait and an Ensure right after it, will there be inconsistent issues? I ask this despite probably trying it tonight since these are the kind of issues that tend to bite me after the fact and not up front. On Fri, Mar 16, 2012 at 10:38 AM, Niall Douglas <s_sourceforge@nedprod.com>wrote:
The key to avoiding deadlocks is, in every case, the appropriate design.
In what you posted you appear to be doing too much at once, so you're holding onto too many locks at once. Either replace those with a single, master lock or try to never hold more than one lock at any one time. That includes the GIL. In well designed code, one almost NEVER holds more than two locks at any time.
Consider breaking object instantiation up into well defined stages. Consider throwing away your current implementation entirely and starting from scratch. Debugging a bad design will take longer and cost more than throwing it away and starting again with the benefit of hindsight.
Niall
On 16 Mar 2012 at 12:57, Adam Preble wrote:
Well the current design does have a problem, but I think it's more to do with the Python side than the sum of the whole thing.
If by "Python side" you mean Boost.Python, then I agree: BPL has no support for GIL management at all, and it really ought to. This was one of the things that was discussed in the BPL v3 discussions on this list a few months ago. If by "Python side" you just mean Python, well it's basically one big fat lock. There's nothing wrong with that design, indeed it's extremely common.
So I assume I would satisfy your suggestion of a master lock if I basically put the interpreter in its own subsystem. Everybody then gets their turn at it.
You need to clarify what you mean by "own subsystem". Do you mean "own process"?
Something I'm curious about has to do with the busy what in my promises and future implementation. When I end up waiting on something, is there a way right now with Python to give up the GIL until the wait is done? If I were to do a Release before the wait and an Ensure right after it, will there be inconsistent issues? I ask this despite probably trying it tonight since these are the kind of issues that tend to bite me after the fact and not up front.
You're going to have to be a lot clearer here. Do you mean you want BPL to give up the GIL until the wait is done, or Python? Regarding inconsistency over releases and locks, that's 101 threading theory. Any basic threading programming textbook will tell you how to handle such issues. Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
On Sat, Mar 17, 2012 at 1:44 PM, Niall Douglas <s_sourceforge@nedprod.com>wrote:
If by "Python side" you mean Boost.Python, then I agree: BPL has no support for GIL management at all, and it really ought to. This was one of the things that was discussed in the BPL v3 discussions on this list a few months ago.
Hey do you know any terms or thread names where I could go digging through some of those discussions? I'm just trying stuff superficially and finding some things that are at least interesting, but I'm not sure I got that stuff yet.
You need to clarify what you mean by "own subsystem". Do you mean "own process"?
In the model I'm using, a subsystem would be a thread taking care of a particular resource. In this case, I figured I would make the Python interpreter that resource, and install better guards around interacting with it. For one thing, rather than anything else grabbing the GIL, they would enqueue stuff for it to execute. That's about as far as I got with it in my head. I managed to unjam the deadlock I was experiencing by eliminating the contention the two conflicting threads were having with each other.
You're going to have to be a lot clearer here. Do you mean you want BPL to give up the GIL until the wait is done, or Python?
Something, somewhere. I wasn't being to picky. I wondered if there was a way to do it that I hadn't found with the regular Python headers.
Regarding inconsistency over releases and locks, that's 101 threading theory. Any basic threading programming textbook will tell you how to handle such issues.
I'm hoping I'm asking about, say, 102 threading stuff instead. ;) Specifically, I'm trying my darndest to keep explict lock controls outside of the main code because it really is hard to get right all the time. Rather, I am trying to apply some kind of structure. The subsystem model I did a terrible job of mentioning is an example, as well as using promises and futures. If there are tricks, features, or something similar to keep in mind, that would steer how I approach this.
On 03/17/2012 11:20 PM, Adam Preble wrote:
On Sat, Mar 17, 2012 at 1:44 PM, Niall Douglas <s_sourceforge@nedprod.com <mailto:s_sourceforge@nedprod.com>> wrote:
If by "Python side" you mean Boost.Python, then I agree: BPL has no support for GIL management at all, and it really ought to. This was one of the things that was discussed in the BPL v3 discussions on this list a few months ago.
Hey do you know any terms or thread names where I could go digging through some of those discussions?
Look for "[Boost.Python v3]" and "New Major-Release Boost.Python Development" in the subject line. We didn't go into a lot of depth on the threading, I'm afraid, as one of the problems is that the guy starting the effort - me - doesn't actually know much about threaded programming. But I am hoping that I can design things in such a way that someone like Niall could easily take it from there. Jim
On 2012-03-18 11:54, Jim Bosch wrote:
On 03/17/2012 11:20 PM, Adam Preble wrote:
On Sat, Mar 17, 2012 at 1:44 PM, Niall Douglas <s_sourceforge@nedprod.com <mailto:s_sourceforge@nedprod.com>> wrote:
If by "Python side" you mean Boost.Python, then I agree: BPL has no support for GIL management at all, and it really ought to. This was one of the things that was discussed in the BPL v3 discussions on this list a few months ago.
Hey do you know any terms or thread names where I could go digging through some of those discussions?
Look for "[Boost.Python v3]" and "New Major-Release Boost.Python Development" in the subject line.
We didn't go into a lot of depth on the threading, I'm afraid, as one of the problems is that the guy starting the effort - me - doesn't actually know much about threaded programming. But I am hoping that I can design things in such a way that someone like Niall could easily take it from there.
I recall seeing a discussion of locking policy being attached to individual functions by means of the return-value and argument-passing policy traits; in other words, something that's associated with from-python and to-python conversion. I found that rather elegant. I'm not sure whether anyone has any practical experience with that technique, or whether it was just an idea worth exploring. FWIW, Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 18 Mar 2012 at 12:05, Stefan Seefeld wrote:
We didn't go into a lot of depth on the threading, I'm afraid, as one of the problems is that the guy starting the effort - me - doesn't actually know much about threaded programming. But I am hoping that I can design things in such a way that someone like Niall could easily take it from there.
I recall seeing a discussion of locking policy being attached to individual functions by means of the return-value and argument-passing policy traits; in other words, something that's associated with from-python and to-python conversion. I found that rather elegant. I'm not sure whether anyone has any practical experience with that technique, or whether it was just an idea worth exploring.
Indeed, I had argued for a DLL and SO aware type registry to hold default BPL<=>Python enter/exit policy (of which locking is an example) which can be optionally overridden at the point of call. It was from this and other reasons I had argued for a fused compile time/runtime registry, and it was at that point that Dave came in to argue against such a design for a series of good reasons. I actually don't think myself and Dave disagreed, rather as usual he sees the world one way and I see the world another, and we use different vocabulary so we both think we're arguing when often we agree. That said, I would be at pains to think twice or thrice about an idea if Dave queries it, not least because it encourages me to articulate myself much more clearly. He's also generally correct, or has had an insight I have missed. Actually, on the basis of that previous discussion I have raised within ISO standards the point that we really ought to do something about standardising shared libraries in ISO WG14. It's long overdue and every attempt has failed so far. Yet as things stand, BPL is simply broken in a multi-extension use context and there is no standards-compliant way of fixing it. As the C language is the primary interop for all other programming languages, and POSIX's approach to shared libraries is broken, I fear we might just have to go stomp on some eggshells to get this fixed. That said, if done right we could seriously improve interop for all languages. Imagine a standardised way of talking to Haskell from Python via a modern C interop specification? Now that would be cool! Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
On 18 Mar 2012 at 11:54, Jim Bosch wrote:
We didn't go into a lot of depth on the threading, I'm afraid, as one of the problems is that the guy starting the effort - me - doesn't actually know much about threaded programming. But I am hoping that I can design things in such a way that someone like Niall could easily take it from there.
I'm currently in early negotiations with a Silicon Valley crowd, part of which involve doing some substantial improvements to Boost.Python in this area. I generally don't believe anyone until I either see money or a signed contract, so take this news with a pinch of salt, but I hope that the list knows that I would love to work on this topic if it were financially viable for me to do so. Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
On 17 Mar 2012 at 22:20, Adam Preble wrote:
If by "Python side" you mean Boost.Python, then I agree: BPL has no support for GIL management at all, and it really ought to. This was one of the things that was discussed in the BPL v3 discussions on this list a few months ago.
Hey do you know any terms or thread names where I could go digging through some of those discussions? I'm just trying stuff superficially and finding some things that are at least interesting, but I'm not sure I got that stuff yet.
Try "[Boost.Python v3] Conversions and Registries", about October of last year. If you search right back to many years ago, I used to maintain a patch which implemented GIL management for Boost.Python. It's long bitrotted though.
You need to clarify what you mean by "own subsystem". Do you mean "own process"?
In the model I'm using, a subsystem would be a thread taking care of a particular resource. In this case, I figured I would make the Python interpreter that resource, and install better guards around interacting with it. For one thing, rather than anything else grabbing the GIL, they would enqueue stuff for it to execute. That's about as far as I got with it in my head. I managed to unjam the deadlock I was experiencing by eliminating the contention the two conflicting threads were having with each other.
The only way to get two Python interpreters to run at once is to put each into its own process. As it happens, IPC is usually fast enough relative to the slowness of Python that often this actually works very well.
You're going to have to be a lot clearer here. Do you mean you want BPL to give up the GIL until the wait is done, or Python?
Something, somewhere. I wasn't being to picky. I wondered if there was a way to do it that I hadn't found with the regular Python headers.
Generally Python releases the GIL around anything it thinks might wait. So long as you write your extension code to do the same, all should be well.
Regarding inconsistency over releases and locks, that's 101 threading theory. Any basic threading programming textbook will tell you how to handle such issues.
I'm hoping I'm asking about, say, 102 threading stuff instead. ;) Specifically, I'm trying my darndest to keep explict lock controls outside of the main code because it really is hard to get right all the time. Rather, I am trying to apply some kind of structure. The subsystem model I did a terrible job of mentioning is an example, as well as using promises and futures. If there are tricks, features, or something similar to keep in mind, that would steer how I approach this.
Y'see, I'd look at promises and futures primarily as a latency hiding mechanism rather than for lock handling. I agree it would have been super-sweet if Python had been much more thread aware in its design, but we live with the hand we're dealt. Trying to bolt on futures and promises to a system which wasn't designed for it is likely to be self-defeating. There is a school of thought that multithreading is only really worthwhile doing in statically compiled languages. Anything interpreted tends, generally speaking, to be too chunky to fine grain holding locks to make threading useful for anything except i/o. Certainly Python is extremely chunky, and I'd generally avoid using threads in Python at all except as a way of portably doing asynchronous i/o. Otherwise it's not worth it. If you're thinking it is, it's time for a redesign. Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
participants (4)
-
Adam Preble -
Jim Bosch -
Niall Douglas -
Stefan Seefeld