Need some help with Python/C api and threading
djc at object-craft.com.au
Mon Jul 5 08:29:01 CEST 2004
Steve Menard wrote:
> Les Smithson wrote:
>>>>>>> "Steve" == Steve Menard <steve.menard at videotron.ca> writes:
>> I haven't done this for a while and I'm a little hazy on it, so this
>> may be incorrect:
>> I used 'PyThreadState *ts = Py_NewInterpreter();' to set a new
>> sub-interpreter state if called in a new thread.
>> If the embedded script calls back into the extension, it restores that
>> thread state and acquires the GIL before making any other Py* calls by
>> calling 'PyEval_RestoreThread(ts);'. Before returning, it calls
> Thanks, however I dont think thid will work. The doc for
> Py_NewInterpreter says that it created a "an (almost) totally separate
> environment for the execution of Python code. In particular, the new
> interpreter has separate, independent versions of all imported modules".
> This is not good for me, as the callbacks must come in the "main"
> interpreter context.
> Is there a tutorial somewhere? Or a particularly well written extension
> module whose source code I could take a look at?
> Let me summarize my situation :
> I am writing a python extension, not embedding python. As such, I have
> no control over the interpreter, or the threads.
> The library I am embedding is not of my own writing. It can create any
> number of threads. It can make callbacks into the Python interpreter on
> any such thread.
> A given thread can original either in python or the library, but control
> can go back and forth : A python method can call a library method, which
> in turn calls back into python, which calls a linrary method, etc ...
> This is a potential problem, because trying to grab in GIL twice from
> the same thread will cause a deadlock.
> So far, here is what I am doing (without success).
> 1) In the init_XXX method, I call PyEval_InitThreads().
> 2) Every time I pass control to the library, I wrap the call into a
> Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS pair. Note that I adde this
> recently, and get an error the second time Py_BEGIN_ALLOW_THREADS is
> called, with the following error "Fatal Python error: PyEval_SaveThread:
> NULL tstate"
> 3) Finally, whenever I receive a callback from the library, I added
> these lines to the start and end of the method :
> PyInterpreterState* interp = PyInterpreterState_New();
> PyThreadState *tstate = PyThreadState_New(interp);
> Thats about it. I am sure someone, somewhere has done what I need :(
In the code for the Sybase extension module I ended up with some code to
manage the GIL and a lock per database context and database connection.
The code is a little complicated in that you can disable all locking
support at compile time if it is not important to you.
More information about the Python-list