[C++-sig] boost::python and threads
Paul Scruby
paul at gingernut.tv
Mon Jul 6 12:39:39 CEST 2009
Hello again,
Sorry, I spoke too soon. The good news is that wrapping my virtual method
calls into Python between PyGILState_Ensure() and PyGILState_Release() fixed
the crash under Python2.6.2 on Windows, but not when I tested it again under
Python 2.4.4 on Solaris. Under Python 2.4.4 on SolarisSolaris it now calls
and exits the virtual method in Python sucessfully, but then crashes when
C++ trys to release the global interpretor lock.
The call-stack with the Sun C++ 5.9 compiler under Solaris is
t at 2 (l at 2) signal SEGV (no mapping at the fault address) in sem_invalid
at 0xfec453ed
0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
Current function is Ticker::operator()
28 PyGILState_Release(state_);
(dbx) where
current thread: t at 2
[1] sem_invalid(0x0), at 0xfec453ed
[2] _sem_post(0x0), at 0xfec454c4
[3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
0xfe77ef38, 0xfef441b5), at 0xfef492dc
[4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
[5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
0x0, 0x80a3140), at 0xfef43eba
[6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
=>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
[8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56
in "thread.hpp"
[9] thread_proxy(0x810a288), at 0xfea78ce4
[10] _thr_setup(0xfe670200), at 0xfee159b9
[11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
0xfe77ef14), at 0xfee15ca0
Do you think it's worth repeating this test using gcc/linux, or do you think
that this is just a limitation of using Python with threads?
Thanks again,
Paul
t at 2 (l at 2) signal SEGV (no mapping at the fault address) in sem_invalid at
0xfec453ed
0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
Current function is Ticker::operator()
28 PyGILState_Release(state_);
(dbx) where
current thread: t at 2
[1] sem_invalid(0x0), at 0xfec453ed
[2] _sem_post(0x0), at 0xfec454c4
[3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
0xfe77ef38, 0xfef441b5), at 0xfef492dc
[4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
[5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
0x0, 0x80a3140), at 0xfef43eba
[6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
=>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
[8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56 in
"thread.hpp"
[9] thread_proxy(0x810a288), at 0xfea78ce4
[10] _thr_setup(0xfe670200), at 0xfee159b9
[11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
0xfe77ef14), at 0xfee15ca0
"Paul Scruby" <paul at gingernut.tv> wrote in message
news:h2sgic$ad4$1 at ger.gmane.org...
> Hiya,
>
> That's fantastic, all I needed to do was to put PyGILState_Ensure();
> before my virtual function calls into python from another thread and my
> program no longer crashes. Problem solved, isn't boost::python great!
>
> Many thanks,
>
> Paul
>
>
> "Renato Araujo" <renatox at gmail.com> wrote in message
> news:95291a80907041315k41b7ad88o32d2111ae8fe1e91 at mail.gmail.com...
> Hi Paul
>
> In my bindings I had a problem like this, to solve I created a simple
> class like that:
>
> class thread_locker
> {
> thread_locker()
> {
> if (thread_support::enabled())
> m_gstate = PyGILState_Ensure();
> }
>
> ~thread_locker()
> {
> if (thread_support::enabled())
> PyGILState_Release(m_gstate);
> }
> };
>
> then in my wrapper virtual implementation I did this:
>
> ...
> void wrapper::virtual_func(..)
> {
> thread_locker lock;
> .. my code ..
> }
> ....
>
> this solve my problems with call of virtual functions in thread
> enviroment.
>
> BR
>
>
>
>
> On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig at wdtinc.com> wrote:
>> Whoops, I think this problem is a little uglier than I thought, since you
>> overrode the onTick() function in python with a call to print, which
>> needs access to the interpreter in your new thread. See the link Thomas
>> posted for dealing with the GIL (along with worrying about any possible
>> garbage collection issues). Now I remember why I kept my C++ threads
>> isolated from Python stuff....sorry, it's been a while....
>>
>> Bill
>> ________________________________________
>> From: cplusplus-sig-bounces+wladwig=wdtinc.com at python.org
>> [cplusplus-sig-bounces+wladwig=wdtinc.com at python.org] On Behalf Of
>> William Ladwig [wladwig at wdtinc.com]
>> Sent: Saturday, July 04, 2009 1:34 PM
>> To: Development of Python/C++ integration
>> Subject: Re: [C++-sig] boost::python and threads
>>
>> It looks to me like you have a garbage collection problem going on. If
>> you create a wrapped c++ object in python, then python is going to own
>> the object and will destroy it when its reference count goes to 0. In
>> your python example script at the bottom, you call the Ticker's run()
>> function, which from the python point of view, returns quickly and the
>> script ends. Python has no idea that you spawned off a new thread from
>> the C++ side, so when the script ends, python destroys the object and now
>> you have a problem. One way that you can check to see if this is what is
>> going on is to add this to the bottom of the test script and see if the
>> crashes go away:
>>
>> # Warning, you'll need to kill this script manually
>> import time
>> while True:
>> time.sleep(1)
>>
>> Generally when I want to fire off a new C++ thread, I hold any objects
>> that the thread needs in an auto_ptr (see held type for class wrappers),
>> take ownership of them in C++ (see the FAQ in the documentation) and
>> start the thread from the C++ side. Or, you can also create C++ wrappers
>> which accept shared_ptr arguments, while holding your classes in
>> shared_ptrs, and this should handle the reference counting as well.
>> Unfortunately, this may require some interface changes to what you have
>> already written (or possibly some clever wrapping).
>>
>> Hope this helps,
>> Bill
>>
>>
>> ________________________________________
>> From: cplusplus-sig-bounces+wladwig=wdtinc.com at python.org
>> [cplusplus-sig-bounces+wladwig=wdtinc.com at python.org] On Behalf Of Paul
>> Scruby [paul at gingernut.tv]
>> Sent: Friday, July 03, 2009 6:15 AM
>> To: cplusplus-sig at python.org
>> Subject: [C++-sig] boost::python and threads
>>
>> I am having some problems using boost::python with boost::thread. I'm
>> using
>> threads because I want to run some tasks in the background when I'm using
>> the Python's interactive shell. However, when I use get_override() to
>> call
>> a Python method from another boost::thread it crashes internally. For
>> example:
>>
>> #include <boost/python.hpp>
>> #include <boost/thread/thread.hpp>
>> #include <boost/thread/xtime.hpp>
>>
>> using namespace boost::python;
>>
>> class Ticker
>> : public wrapper<Ticker>
>> {
>> private:
>> bool run_;
>> volatile bool * running_;
>> boost::thread * thread_;
>> boost::xtime xt_;
>> public:
>> Ticker() : running_(&run_) { *running_ = false; }
>>
>> void operator()()
>> {
>> while (*running_)
>> {
>> boost::xtime_get(&xt_, boost::TIME_UTC);
>> ++xt_.sec;
>> boost::thread::sleep(xt_);
>> onTick();
>> }
>> }
>>
>> void run()
>> {
>> if (*running_ == false)
>> {
>> *running_ = true;
>> thread_ = new boost::thread(*this);
>> }
>> }
>>
>> void stop()
>> {
>> if (*running_ == true)
>> {
>> *running_ = false;
>> thread_->join();
>> delete thread_;
>> }
>> }
>>
>> virtual void onTick() { get_override("onTick")(); }
>> void default_onTick() {}
>> };
>>
>> BOOST_PYTHON_MODULE(tick)
>> {
>> class_<Ticker, boost::noncopyable> ("Ticker")
>> .def("run", &Ticker::run)
>> .def("stop", &Ticker::stop)
>> .def("onTick", &Ticker::default_onTick);
>> }
>>
>> Here is a test script that which will crash when you import it into
>> Python's
>> interactive shell.
>>
>> from tick import Ticker
>>
>> class MyTicker(Ticker):
>> def onTick(self):
>> print "Each second"
>>
>> myticker = MyTicker()
>> myticker.run()
>>
>> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler
>> on
>> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008
>> on
>> Windows XP.
>>
>> The call-stack in dbx on Solaris:
>>
>> >>> t at 2 (l at 2) signal SEGV (no mapping at the fault address) in
>> PyErr_Restore at 0xfef38fa1
>> 0xfef38fa1: PyErr_Restore+0x0031: movl 0x00000028(%edi),%ecx
>> Current function is boost::python::override::operator()
>> 99 detail::method_result x(
>>
>> (dbx) where
>> current thread: t at 2
>> [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
>> 0x80652fc), at 0xfef38fa1
>> [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
>> [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
>> [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
>> [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
>> 0xfef2bf02
>> [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
>> =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
>> "override.hpp"
>> [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
>> [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
>> [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
>> 56 in "thread.hpp"
>> [11] thread_proxy(0x810a288), at 0xfea78ce4
>> [12] _thr_setup(0xfe670200), at 0xfee159b9
>> [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
>> 0x80652fc, 0x80f1220), at 0xfee15ca0
>>
>> The call-stack in Visual Studio 2008:
>>
>> python26.dll!1e013595()
>> [Frames below may be incorrect and/or missing, no symbols loaded for
>> python26.dll]
>> python26.dll!1e09ee7d()
>> > tick.pyd!boost::python::override::operator()() Line 103 + 0x16 bytes
>> C++
>> 00f3fd64()
>> tick.pyd!Ticker::operator()() Line 27 + 0xe bytes C++
>> tick.pyd!boost::detail::thread_data<Ticker>::run() Line 57 C++
>> tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
>> param=0x00245f30) Line 168 C++
>> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
>> msvcr90d.dll!_threadstartex(void * ptd=0x00d46938) Line 331 C
>> kernel32.dll!7c80b729()
>>
>>
>> Have a missed a trick using the wrapper, or does boost::python not
>> support
>> threading?
>>
>> Many thanks,
>>
>> Paul
>>
>>
>>
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig at python.org
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig at python.org
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig at python.org
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>
>
>
>
> --
> Renato Araujo Oliveira Filho
More information about the Cplusplus-sig
mailing list