Re: [C++-sig] Am I crazy or is it my computer?
I don't know too much about threading, but I think - you still need Niall Douglas' patches if your threads want to call back into Python - it is fair to say that the real problem is "Python & threading", mainly because of the dreaded global interpreter lock (GIL) Recently there has been some discussion on python-dev about getting rid of the GIL, but I'm not holding my breath... IIUC, if your threads never call back into Python you don't have to deal with the GIL. If you have to deal with the GIL I'm not surprised to see you frustrated. This is something that was largely ignored in the Boost.Python development (there is only so much you can hope to achieve in a given amount of time...). In my work I've consciously avoided threading. Under this simplification I find Boost.Python well documented and intuitive :-) Ralf ----- Original Message ---- From: Matthew Scouten <matthew.scouten@gmail.com> To: Development of Python/C++ integration <C++-sig@python.org> Sent: Wednesday, October 31, 2007 1:11:27 PM Subject: Re: [C++-sig] Am I crazy or is it my computer? Ok, I just found this: http://www.boost.org/libs/python/doc/v2/faq.html#threadsupport I seems to be saying that I can not do what I want with out patching Boost::Python (or does this only apply to multiple interpretors? ) Is this still true (please tell me it's been fixed)? if so, where can I get this patch and how do I apply it? I asked about multi-threading and boost python in on this list a while ago, and got no answer. I took this a a "go ahead, no problems". If I am being an idiot, could you tell me so? Just flame away at me so I know I've been stupid. A lot of people here actually know what they are doing. I do not have that luxury. I am finding boost python extremely complex and non-intuitive. There is also a lack of good documentation (example code is nice to have, but does not constitute proper documentation all by itself).
The you seem to be saying that the problem is the GIL rather than the problem is with the GIL rather than Boost::Python itself. Does the I am already making sure to acquire the GIL before I call into python. Here is an example of a callback wrapper: void DataConsumer_wrapper::OnSampleCallback( int a1, int a2) { {callback call; if (override func = this->get_override("OnSampleCallback")) { func(a1,a2 ); return; } }// end callback call; return DataConsumer::OnSampleCallback(a1,a2); } As I mentioned before: 'callback call' calls PyGILState_Ensure in its ctor and PyGILState_Release in its dtor. So I believe I have acquired the GIL correctly. My thought was that it was some internal data structure in boost python that needed locking as well. But I have the GIL before I touch any thing in boost::python, so nothing is getting called from python and the callback thread should be the only one in B::P code. Do Niall Douglas' patches do anything except manage the GIL? If not than maybe my problem is not thread related after all, or maybe the GIL issue is subtler than I thought. Is there any one on this list who knows about the guts of Boost Python? I would appreciate their comments. On 10/31/07, Ralf W. Grosse-Kunstleve <rwgk@yahoo.com> wrote:
I don't know too much about threading, but I think
- you still need Niall Douglas' patches if your threads want to call back into Python - it is fair to say that the real problem is "Python & threading",
mainly because of the dreaded global interpreter lock (GIL)
Recently there has been some discussion on python-dev about getting
rid of the GIL, but I'm not holding my breath...
IIUC, if your threads never call back into Python you don't have to
deal with the GIL. If you have to deal with the GIL I'm not surprised to see you frustrated. This is something that was largely ignored in the Boost.Python development (there is only so much you can hope to achieve in a given amount of time...).
In my work I've consciously avoided threading. Under this simplification I find Boost.Python well documented and intuitive :-)
Ralf
----- Original Message ----
From: Matthew Scouten <matthew.scouten@gmail.com>
To: Development of Python/C++ integration <C++-sig@python.org>
Sent: Wednesday, October 31, 2007 1:11:27 PM
Subject: Re: [C++-sig] Am I crazy or is it my computer?
Ok, I just found this: http://www.boost.org/libs/python/doc/v2/faq.html#threadsupport
I seems to be saying that I can not do what I want with out patching Boost::Python (or does this only apply to multiple interpretors? )
Is this still true (please tell me it's been fixed)?
if so, where can I get this patch and how do I apply it?
I asked about multi-threading and boost python in on this list a while ago, and got no answer. I took this a a "go ahead, no problems". If I am being an idiot, could you tell me so? Just flame away at me so I know I've been stupid. A lot of people here actually know what they are doing. I do not have that luxury. I am finding boost python extremely complex and non-intuitive. There is also a lack of good documentation (example code is nice to have, but does not constitute proper documentation all by itself).
_______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
Sorry apparently I can not write right today make that first paragraph: The you seem to be saying that the problem is the GIL rather than Boost::Python itself. I am already making sure to acquire the GIL before I call into python. Here is an example of a callback wrapper: On 11/1/07, Matthew Scouten <matthew.scouten@gmail.com> wrote:
The you seem to be saying that the problem is the GIL rather than the problem is with the GIL rather than Boost::Python itself. Does the I am already making sure to acquire the GIL before I call into python. Here is an example of a callback wrapper:
void DataConsumer_wrapper::OnSampleCallback( int a1, int a2) { {callback call; if (override func = this->get_override("OnSampleCallback")) { func(a1,a2 ); return; } }// end callback call; return DataConsumer::OnSampleCallback(a1,a2); }
As I mentioned before: 'callback call' calls PyGILState_Ensure in its ctor and PyGILState_Release in its dtor. So I believe I have acquired the GIL correctly.
My thought was that it was some internal data structure in boost python that needed locking as well. But I have the GIL before I touch any thing in boost::python, so nothing is getting called from python and the callback thread should be the only one in B::P code.
Do Niall Douglas' patches do anything except manage the GIL? If not than maybe my problem is not thread related after all, or maybe the GIL issue is subtler than I thought.
Is there any one on this list who knows about the guts of Boost Python? I would appreciate their comments.
On 10/31/07, Ralf W. Grosse-Kunstleve <rwgk@yahoo.com> wrote:
I don't know too much about threading, but I think
- you still need Niall Douglas' patches if your threads want to call back into Python - it is fair to say that the real problem is "Python & threading",
mainly because of the dreaded global interpreter lock (GIL)
Recently there has been some discussion on python-dev about getting
rid of the GIL, but I'm not holding my breath...
IIUC, if your threads never call back into Python you don't have to
deal with the GIL. If you have to deal with the GIL I'm not surprised to see you frustrated. This is something that was largely ignored in the Boost.Python development (there is only so much you can hope to achieve in a given amount of time...).
In my work I've consciously avoided threading. Under this simplification I find Boost.Python well documented and intuitive :-)
Ralf
----- Original Message ----
From: Matthew Scouten <matthew.scouten@gmail.com>
To: Development of Python/C++ integration <C++-sig@python.org >
Sent: Wednesday, October 31, 2007 1:11:27 PM
Subject: Re: [C++-sig] Am I crazy or is it my computer?
Ok, I just found this: http://www.boost.org/libs/python/doc/v2/faq.html#threadsupport
I seems to be saying that I can not do what I want with out patching Boost::Python (or does this only apply to multiple interpretors? )
Is this still true (please tell me it's been fixed)?
if so, where can I get this patch and how do I apply it?
I asked about multi-threading and boost python in on this list a while ago, and got no answer. I took this a a "go ahead, no problems". If I am being an idiot, could you tell me so? Just flame away at me so I know I've been stupid. A lot of people here actually know what they are doing. I do not have that luxury. I am finding boost python extremely complex and non-intuitive. There is also a lack of good documentation (example code is nice to have, but does not constitute proper documentation all by itself).
_______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
participants (2)
-
Matthew Scouten -
Ralf W. Grosse-Kunstleve