--- Mark Hammond <mhammond@skippinet.com.au> wrote:
[...] I believe it is clear and would prefer to see a single other person with a problem [...]
I've been reading this thread with interest since we've recently fought (and lost) this battle at my company. Here is our use case: We use Python in primarily two ways, the first and obvious use is as a scripting language with a small group of us creating extensions to talk to existing libraries. There are no relevant problems here. The second use is as a data structures library used from C++. We created a very easy to use C++ class that has a bazillion operator overloads and handles all the reference counting and what not for the user. It used to handle threading too, but that proved to be very difficult. Think of this C++ class as something similar to what boost::python::{object, dict, list, tuple, long, numeric} provides, but intended for users who don't really like or want to know C++. Most of our users write small C++ processes that communicate amongst themselves via an assortment of IPC mechanisms. Occasionally these C++ processes are threaded, and we wanted to handle that. Our model was that C++ code would never hold the GIL, and that before we entered the Python API we would use pthread_getspecific (thread local storage) to see if there was a valid PyThreadState to use. If there wasn't a thread state, we would create one. Since C++ code never held the GIL, we'd always acquire it. This strategy allows all Python threads to take turns running, and allows any C++ threads to enter into Python when needed. Performance lagged a little this way, but not so much that we cared. The problem came when our users started to write generic libraries to be used from C++ and also wanted these libraries as Python extensions. In one case, their library would be used up in a standalone C++ process (where the GIL was not held), and in another they would use boost to try and export their library as an extension to Python (where the GIL was held). The same C++ library couldn't know in advance if the GIL was held. The way boost templatizes on your functions and classes, it is not at all clear when you can safely release the GIL for the benefit of the C++ library being wrapped up that expects the GIL is not held. Since being able to support writing generic libraries easily is more important to us than supporting multithreaded C++ processes (using Python as a data structure library), we changed our strategy and made it so that in C++ the GIL was held by default. Since for these types of processes "most" of our time is spent in C++, no Python threads ever get a chance to run without additional work from the C++ author. It also requires additional work to have multiple C++ threads use Python. This was pretty unsatisfying to those of us who like to work with threads. It's too late to make this long story short, but what would have made our situation much easier would be something like: void *what_happened = Py_AcquireTheGilIfIDontAlreadyHaveIt(); // Can safely call Python API functions here, no matter what the // context is... Py_ReleaseTheGilIfImSupposedTo(what_happened); I hope seeing another side of this is of some use. Cheers, -Scott __________________________________________________ Do you Yahoo!? Yahoo! Mail Plus - Powerful. Affordable. Sign up now. http://mailplus.yahoo.com