I am working on an application which embeds python in the application
threads and each thread can further create threads using the python thread
module
few things i don't understand:
thread state messed up any time my thread is running. Which implies
- why does the interpreter ie the main thread keeps releasing and
reacquiring the GIL every few instruction. why isn't GIL simply allowed to
be acquired by any thread which requires it then. I mean why is the
acquisition and release of a lock kept periodic and not on need basis?(
which is obviously the way it should be) - The purpose of GIL was not very clear to me from python-docs. is it just
a single lock to guard every global variable? or is it something more? - suppose I am not making any blocking call then I would never release GIL
since that is the only thing guarding my thread state and i don't want my
unacceptable starvation of other threads.
2008/9/2 <Swapnil.ST@gmail.com>
I am working on an application which embeds python in the application threads and each thread can further create threads using the python thread module few things i don't understand:
- why does the interpreter ie the main thread keeps releasing and reacquiring the GIL every few instruction. why isn't GIL simply allowed to be acquired by any thread which requires it then. I mean why is the acquisition and release of a lock kept periodic and not on need basis?( which is obviously the way it should be)
Because Python _always_ needs the lock as long as there is Python code left to execute. If it wasn't periodically released and re-acquired, other threads that want to execute Python code would forever block.
- The purpose of GIL was not very clear to me from python-docs. is it just a single lock to guard every global variable? or is it something more?
Anything that needs to call a Python API needs to acquire this lock first. GIL == Global Interpreter Lock. So it really guards access to the whole Python subsystem.
- suppose I am not making any blocking call then I would never release GIL since that is the only thing guarding my thread state and i don't want my thread state messed up any time my thread is running. Which implies unacceptable starvation of other threads.
You don't release the GIL but Python will release it periodically, as you mentioned above. If you need more protection, you need to use your own locks (threading.Lock).
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Swapnil.ST@gmail.com writes:
- why does the interpreter ie the main thread keeps releasing and reacquiring the GIL every few instruction. why isn't GIL simply allowed to be acquired by any thread which requires it then.
Because a thread requires it to even be able to run. Release the GIL == give other threads a chance to run.
I mean why is the acquisition and release of a lock kept periodic and not on need basis?( which is obviously the way it should be)
Because holding the GIL allows a thread to run, and all threads need to run at (almost) all times. The GIL is somewhat peculiar in its mode of operation. A typical lock guards a resource and is by default released. The thread that needs a resource acquires the lock, uses the resource, and releases the lock.
GIL, on the other hand, is almost always acquired by some thread, and that thread is allowed to access Python data and run Python code. Py_BEGIN_ALLOW_THREADS actually *releases* the lock (in contrast to "begin foo" synchronization blocks which tend to acquire locks), giving other threads a chance to run during a blocking syscall or a long-winded calculation that doesn't touch Python objects. Py_END_ALLOW_THREADS reacquires the lock.
- suppose I am not making any blocking call then I would never release GIL since that is the only thing guarding my thread state and i don't want my thread state messed up any time my thread is running. Which implies unacceptable starvation of other threads.
If you're referring to Python code, the interpreter will occasionally release the GIL, as Gustavo said. But if you're referring to C code, it is your responsibility to release the lock at the places where you want other threads to run. If you have a long calculation, I suppose you could add pairs of Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to avoid starvation of other threads.
participants (3)
-
Gustavo Carneiro -
Hrvoje Niksic -
Swapnil.ST＠gmail.com