"Mark Hammond" <mhammond@skippinet.com.au> writes:
My goal:
For a multi-threaded application (generally this will be a larger app embedding Python, but that is irrelevant), make it reasonably easy to accomplish 2 things:
1) Allow "arbitrary" threads (that is, threads never before seen by Python) to acquire the resources necessary to call the Python C API.
2) Allow Python extensions to be written which support (1) above.
3) Allow "arbitrary" threads to acquire the resources necessary to call the Python C API, even if they already have those resources, and to later release them if they did not have those resources.
Currently (2) is covered by Py_BEGIN_ALLOW_THREADS, except that it is kinda like only having a hammer in your toolbox <wink>. I assert that 2) could actually be split into discrete goals:
I'm going to ask some questions just to make sure your terminology is clear to me:
2.1) Extension functions that expect to take a lot of time, but generally have no thread-state considerations. This includes sleep(), all IO functions, and many others. This is exactly what Py_BEGIN_ALLOW_THREADS was designed for.
In other words, functions which will not call back into the Python API?
2.2) Extensions that *may* take a little time, but more to the point, may directly and synchronously trigger callbacks.
By "callbacks", do you mean "functions which (may) use the Python C API?"
That is, it is not expected that much time will be spent outside of Python, but rather that Python will be re-entered. I can concede that functions that may trigger asynch callbacks need no special handling here, as the normal Python thread switch mechanism will ensure correct their dispatch.
By "trigger asynch callbacks" do you mean, "cause a callback to occur on a different thread?"
Currently 2.1 and 2.2 are handled the same way, but this need not be the case. Currently 2.2 is only supported by *always* giving up the lock, and at each entry point *always* re-acquiring it. This is obviously wasteful if indeed the same thread immediately re-enters - hence we are here with a request for "how do I tell if I have the lock?".
Yep, that pinpoints my problem.
Combine this with the easily stated but tricky to implement (1) and no one understands it at all <frown>
I also propose that we restrict this to applications that intend to use a single "PyInterpreterState" - if you truly want multiple threads running in multiple interpreters (and good luck to you - I'm not aware anyone has ever actually done it <wink>) then you are on your own.
Fine with me ;-). I think eventually we'll need to come up with a more precise definition of exactly when "you're on your own", but for now that'll do.
Are these goals a reasonable starting point? This describes all my venturing into this area.
Sounds about right to me. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution