Need some help with Python/C api and threading

[Simon Dahlbacka]

Steve Menard wrote:
> Les Smithson wrote:
> 
>>>>>>> "Steve" == Steve Menard <steve.menard at videotron.ca> writes:
> 
> 
> <SNIP>
> 
>>
>> I haven't done this for a while and I'm a little hazy on it, so this
>> may be incorrect:
>>
>> I used 'PyThreadState *ts = Py_NewInterpreter();' to set a new
>> sub-interpreter state if called in a new thread.
>>
>> If the embedded script calls back into the extension, it restores that
>> thread state and acquires the GIL before making any other Py* calls by
>> calling 'PyEval_RestoreThread(ts);'. Before returning, it calls
>> 'PyEval_SaveThread()'.
>>
>>
> 
> Thanks, however I dont think thid will work. The doc for 
> Py_NewInterpreter says that it created a "an (almost) totally separate 
> environment for the execution of Python code. In particular, the new 
> interpreter has separate, independent versions of all imported modules". 
> This is not good for me, as the callbacks must come in the "main" 
> interpreter context.
> 
> Is there a tutorial somewhere? Or a particularly well written extension 
> module whose source code I could take a look at?
> 
> Let me summarize my situation :
> 
> I am writing a python extension, not embedding python. As such, I have 
> no control over the interpreter, or the threads.
> 
> The library I am embedding is not of my own writing. It can create any 
> number of threads. It can make callbacks into the Python interpreter on 
> any such thread.
> 
> A given thread can original either in python or the library, but control 
> can go back and forth : A python method can call a library method, which 
> in turn calls back into python, which calls a linrary method, etc ... 
> This is a potential problem, because trying to grab in GIL twice from 
> the same thread will cause a deadlock.
> 
> 
> So far, here is what I am doing (without success).
> 
> 1) In the init_XXX method, I call PyEval_InitThreads().
> 
> 2) Every time I pass control to the library, I wrap the call into a 
> Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS pair. Note that I adde this 
> recently, and get an error the second time Py_BEGIN_ALLOW_THREADS is 
> called, with the following error "Fatal Python error: PyEval_SaveThread: 
> NULL tstate"
> 
> 3) Finally, whenever I receive a callback from the library, I added 
> these lines to the start and end of the method :
> 
>         PyInterpreterState* interp = PyInterpreterState_New();
>         PyThreadState *tstate = PyThreadState_New(interp);
>         PyEval_AcquireThread(tstate);
> 
> and
> 
>         PyEval_ReleaseThread(tstate);
>         PyThreadState_Delete(tstate);
>         PyInterpreterState_Delete(interp);
> 
> 
> Thats about it. I am sure someone, somewhere has done what I need :(
> 
> 
> Thanks for any help you can provide,
would this:
http://www.python.org/peps/pep-0311.html

..be of any help?

/Simon