Am I crazy or is it my computer?
This is diving me nuts. I need to triple check an assumption. I am *extending* python using Boost::Python, by wrapping a complex c++ library that was written years ago with no thought of future use in python. The library works by a system of callouts and callbacks with extensive multithreading. There are a number of minor worker threads in the background, but [ASSUMPTION ONE:] this should not matter because they never need to use data or call code that is known to the interpretor. There is also a callback thread on which the callbacks are called. This is NOT a python thread, an interpretor thread, or ANY other special kind of thread. It was not started through the C/Python API. It is a system thread started through the WIN32 API in the byzantine depths of the library. It has not been introduced to the interpretor in any way. ASSUMPTION TWO: I can have that thread call into python code by using the PyGILState_Ensure & PyGILState_Release pair of functions as described in http://www.python.org/dev/peps/pep-0311/ and http://docs.python.org/api/threads.html to acquire the Global Interpretor Lock. (Actually: I am using Resource Acquisition Is Initialization idiom with the following simple class: class callback { public: inline callback() { state = PyGILState_Ensure(); } inline ~callback() { PyGILState_Release(state); } private: PyGILState_STATE state; }; This may be ASSUMPTION THREE) Are my assumptions valid? This may seem like a simple question that I should be googling for, but like I said I am triple checking some assumptions. I have been trying to debug the same extremely bizarre problem for a couple of days now and it only shows up in the asynchronous callbacks.
Ok, I just found this: http://www.boost.org/libs/python/doc/v2/faq.html#threadsupport I seems to be saying that I can not do what I want with out patching Boost::Python (or does this only apply to multiple interpretors? ) Is this still true (please tell me it's been fixed)? if so, where can I get this patch and how do I apply it? I asked about multi-threading and boost python in on this list a while ago, and got no answer. I took this a a "go ahead, no problems". If I am being an idiot, could you tell me so? Just flame away at me so I know I've been stupid. A lot of people here actually know what they are doing. I do not have that luxury. I am finding boost python extremely complex and non-intuitive. There is also a lack of good documentation (example code is nice to have, but does not constitute proper documentation all by itself). On 10/30/07, Matthew Scouten <matthew.scouten@gmail.com> wrote:
This is diving me nuts. I need to triple check an assumption.
I am *extending* python using Boost::Python, by wrapping a complex c++ library that was written years ago with no thought of future use in python.
The library works by a system of callouts and callbacks with extensive multithreading.
There are a number of minor worker threads in the background, but [ASSUMPTION ONE:] this should not matter because they never need to use data or call code that is known to the interpretor.
There is also a callback thread on which the callbacks are called. This is NOT a python thread, an interpretor thread, or ANY other special kind of thread. It was not started through the C/Python API. It is a system thread started through the WIN32 API in the byzantine depths of the library. It has not been introduced to the interpretor in any way.
ASSUMPTION TWO: I can have that thread call into python code by using the PyGILState_Ensure & PyGILState_Release pair of functions as described in http://www.python.org/dev/peps/pep-0311/ and http://docs.python.org/api/threads.html to acquire the Global Interpretor Lock.
(Actually: I am using Resource Acquisition Is Initialization idiom with the following simple class:
class callback { public: inline callback() { state = PyGILState_Ensure(); }
inline ~callback() { PyGILState_Release(state); }
private: PyGILState_STATE state; };
This may be ASSUMPTION THREE)
Are my assumptions valid?
This may seem like a simple question that I should be googling for, but like I said I am triple checking some assumptions. I have been trying to debug the same extremely bizarre problem for a couple of days now and it only shows up in the asynchronous callbacks.
On Oct 31, 2007 1:11 PM, Matthew Scouten <matthew.scouten@gmail.com> wrote:
Ok, I just found this: http://www.boost.org/libs/python/doc/v2/faq.html#threadsupport
I seems to be saying that I can not do what I want with out patching Boost::Python (or does this only apply to multiple interpretors? ) Is this still true (please tell me it's been fixed)? if so, where can I get this patch and how do I apply it?
I think this is simply saying that you need to treat the Python interpreter as a protected resource/critical section. As I understand your problem, you have only one C++ thread using the interpreter so the fact that BP isn't thread-aware shouldn't matter to you. My thought on reading your original post was it's possible that your "minor worker threads" are interacting with variables used by your Python-using thead, and thus that your assumption #1 doesn't hold. However, presumably you've already checked out this possibility or you wouldn't be triple-checking your assumptions. -Max
On 31 Oct 2007 at 15:11, Matthew Scouten wrote:
Ok, I just found this: http://www.boost.org/libs/python/doc/v2/faq.html#threadsupport
I seems to be saying that I can not do what I want with out patching Boost::Python (or does this only apply to multiple interpretors? ) Is this still true (please tell me it's been fixed)? if so, where can I get this patch and how do I apply it?
Follow the link to TnFOX given in the FAQ. Then study FXPython - heavily - and with a very fine toothed comb.
I asked about multi-threading and boost python in on this list a while ago, and got no answer. I took this a a "go ahead, no problems". If I am being an idiot, could you tell me so? Just flame away at me so I know I've been stupid.
I remember your question, and I didn't answer because the FAQ answers and I don't have the time to answer FAQ answered questions. I think the Boost.Python homepage could do with a large flashing neon sign saying "CHECK THE FAQ BEFORE ASKING A QUESTION" right before "SEARCH THE C++ SIG MAILING LIST ARCHIVES BEFORE ASKING A QUESTION". Had you done either, you would have known the answer. Now with the berating being done, I must express my sympathies for your task. I wrapped TnFOX for Python using Boost.Python and it was extremely painful indeed. It can be done, but expect at least 200 hours of programmer time to wrap your head around it. Once you know what you're doing, it becomes quite easy.
A lot of people here actually know what they are doing. I do not have that luxury. I am finding boost python extremely complex and non-intuitive. There is also a lack of good documentation (example code is nice to have, but does not constitute proper documentation all by itself).
Actually, the docs are pretty good. I would have cut out my own tongue to admit that a few years ago, but the problem is not the documentation - it's the programmer thinking in old style C++ ways. Once you wrap your head around Boost.Python, it becomes quite intuitive. I can now happilly plug around its internals and do anything I like fairly confidently. It's actually quite a simple & intuitive design though I do wish there were more usage examples out there - and also I wish more interfaces were publicly defined & solid. Now I have designed the machinery needed to get threaded C++ and Python working. It took me months, and it's not pretty at all and rather easy to break if you modify it. But copying me would be the easiest by far. BTW I reckon your problems stem from the GIL not only being a lock but also being a state variable for Python. This state varies, unpredictably, during and between locks whenever you touch Python at all (including via any other thread). Touching BPL in any way usually means touching Python, so basically dragons abound here. This will explain the often erratic & weird ordering of actions in FXPython in TnFOX - I worked it out after many hours in the debugger watching Python, C++, and Boost.Python interact. I suggest you copy my form closely in your endeavours. Good luck! Cheers, Niall
On Nov 1, 2007 1:58 PM, Niall Douglas <s_sourceforge@nedprod.com> wrote:
I remember your question, and I didn't answer because the FAQ answers and I don't have the time to answer FAQ answered questions. I think the Boost.Python homepage could do with a large flashing neon sign saying "CHECK THE FAQ BEFORE ASKING A QUESTION" right before "SEARCH THE C++ SIG MAILING LIST ARCHIVES BEFORE ASKING A QUESTION".
Matthew actually did mention the FAQ in his second post. Since I'm not clear on the issue yet either, is the FAQ saying that it's not safe to use Boost::Python in multi-threaded C++ code even if only one C++ thread uses the interpreter? Because that seems very odd to me. -Max
On 1 Nov 2007 at 16:50, Maximilian Wilson wrote:
I remember your question, and I didn't answer because the FAQ answers and I don't have the time to answer FAQ answered questions. I think the Boost.Python homepage could do with a large flashing neon sign saying "CHECK THE FAQ BEFORE ASKING A QUESTION" right before "SEARCH THE C++ SIG MAILING LIST ARCHIVES BEFORE ASKING A QUESTION".
Matthew actually did mention the FAQ in his second post. Since I'm not clear on the issue yet either, is the FAQ saying that it's not safe to use Boost::Python in multi-threaded C++ code even if only one C++ thread uses the interpreter? Because that seems very odd to me.
You're 100% safe if: 1. Both BPL and python uses and is used in one and only one thread. 2. No other thread should touch anything related to BPL/python or anything associated with BPL/python. Now all that said, you can get away with a lot more than that. But python really wasn't designed to handle threads at all, you can't even have more than one interpreter in a single process yet :( I personally find this latter restriction to be incredibly daft and the clear result of vast overuse of static global variables :( Cheers, Niall
On 11/1/07, Niall Douglas <s_sourceforge@nedprod.com> wrote:
On 31 Oct 2007 at 15:11, Matthew Scouten wrote:
Ok, I just found this: http://www.boost.org/libs/python/doc/v2/faq.html#threadsupport
I seems to be saying that I can not do what I want with out patching Boost::Python (or does this only apply to multiple interpretors? ) Is this still true (please tell me it's been fixed)? if so, where can I get this patch and how do I apply it?
Follow the link to TnFOX given in the FAQ. Then study FXPython - heavily - and with a very fine toothed comb.
I asked about multi-threading and boost python in on this list a while ago, and got no answer. I took this a a "go ahead, no problems". If I am being an idiot, could you tell me so? Just flame away at me so I know I've been stupid.
I remember your question, and I didn't answer because the FAQ answers and I don't have the time to answer FAQ answered questions. I think the Boost.Python homepage could do with a large flashing neon sign saying "CHECK THE FAQ BEFORE ASKING A QUESTION" right before "SEARCH THE C++ SIG MAILING LIST ARCHIVES BEFORE ASKING A QUESTION".
Had you done either, you would have known the answer.
Now with the berating being done, I must express my sympathies for your task. I wrapped TnFOX for Python using Boost.Python and it was extremely painful indeed. It can be done, but expect at least 200 hours of programmer time to wrap your head around it. Once you know what you're doing, it becomes quite easy.
A lot of people here actually know what they are doing. I do not have that luxury. I am finding boost python extremely complex and non-intuitive. There is also a lack of good documentation (example code is nice to have, but does not constitute proper documentation all by itself).
Actually, the docs are pretty good. I would have cut out my own tongue to admit that a few years ago, but the problem is not the documentation - it's the programmer thinking in old style C++ ways.
Once you wrap your head around Boost.Python, it becomes quite intuitive. I can now happilly plug around its internals and do anything I like fairly confidently. It's actually quite a simple & intuitive design though I do wish there were more usage examples out there - and also I wish more interfaces were publicly defined & solid.
Now I have designed the machinery needed to get threaded C++ and Python working. It took me months, and it's not pretty at all and rather easy to break if you modify it. But copying me would be the easiest by far.
BTW I reckon your problems stem from the GIL not only being a lock but also being a state variable for Python. This state varies, unpredictably, during and between locks whenever you touch Python at all (including via any other thread). Touching BPL in any way usually means touching Python, so basically dragons abound here.
This will explain the often erratic & weird ordering of actions in FXPython in TnFOX - I worked it out after many hours in the debugger watching Python, C++, and Boost.Python interact. I suggest you copy my form closely in your endeavours.
Good luck!
Cheers, Niall
I have downloaded your code and had a look at it. I have a few questions. First of all I do not have and do not need multiple interpretors(thank goodness). I am not sure what parts of what you did are for multi interpretors and what parts are for simple multi threading with one interpretor. I noticed that you did not use the PyGILState_Ensure & PyGILState_Release pair. This makes sense, because they is meant for a single interpretor environment. What I do not understand is: why they would not be sufficient in my environment. from the information at: http://svn.python.org/projects/peps/trunk/pep-0311.txt and http://docs.python.org/api/threads.html I can safely say that I am certain that they would be sufficient in a pure C/Python environment. I am calling _Ensure before touching any boost python, and not calling _Release until I am done with all boost python calls in a given callback. I have posted my example before: void DataConsumer_wrapper::OnSampleCallback( int a1, int a2) { {callback call; if (override func = this->get_override("OnSampleCallback")) { logfunc log("DataConsumer_wrapper::OnSampleCallback", true); func(a1,a2 ); return; } }// end callback call; return DataConsumer::OnSampleCallback(a1,a2); } This is a virtual function callback. The "callback call" line is calling PyGILState_Ensure: class callback { public: inline callback() { state = PyGILState_Ensure(); } inline ~callback() { PyGILState_Release(state); } private: PyGILState_STATE state; }; So If this was pure C/Python, I have obeyed the rules. My question is: Is there something in boost::python itself (as opposed to C/Python) that may be having a problem with this model?
Hello, On Fri, 02 Nov 2007, Matthew Scouten wrote:
What I do not understand is: why they would not be sufficient in my environment. from the information at: http://svn.python.org/projects/peps/trunk/pep-0311.txt and http://docs.python.org/api/threads.html
I just skimmed through this thread, but I think you might be missing a call to PyEval_InitThreads() inside your BOOST_PYTHON_MODULE definition. I found that without this call, the calls to PyGILState_Ensure and PyGILState_Release are essentially no-ops. I don't know too much about threading, so I could be wrong, but my module seems to work, so maybe not. The code for my module is here: http://pyactivemq.googlecode.com/svn/trunk/ Mainly you want to look at: http://pyactivemq.googlecode.com/svn/trunk/src/main/pyactivemq.cpp http://pyactivemq.googlecode.com/svn/trunk/src/main/MessageListener.cpp Cheers, Albert
participants (4)
-
Albert Strasheim -
Matthew Scouten -
Maximilian Wilson -
Niall Douglas