I am working on an application which embeds python in the application
threads and each thread can further create threads using the python thread
module few things i don't understand:
I am working on an application which embeds python in the application threads and each thread can further create threads using the python thread module few things i don't understand:
Because Python _always_ needs the lock as long as there is Python code left to execute. If it wasn't periodically released and re-acquired, other threads that want to execute Python code would forever block.
Anything that needs to call a Python API needs to acquire this lock first. GIL == Global Interpreter Lock. So it really guards access to the whole Python subsystem.
You don't release the GIL but Python will release it periodically, as you mentioned above. If you need more protection, you need to use your own locks (threading.Lock).
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Because a thread requires it to even be able to run. Release the GIL == give other threads a chance to run.
I mean why is the acquisition and release of a lock kept periodic and not on need basis?( which is obviously the way it should be)
Because holding the GIL allows a thread to run, and all threads need to run at (almost) all times. The GIL is somewhat peculiar in its mode of operation. A typical lock guards a resource and is by default released. The thread that needs a resource acquires the lock, uses the resource, and releases the lock.
GIL, on the other hand, is almost always acquired by some thread, and that thread is allowed to access Python data and run Python code. Py_BEGIN_ALLOW_THREADS actually releases the lock (in contrast to "begin foo" synchronization blocks which tend to acquire locks), giving other threads a chance to run during a blocking syscall or a long-winded calculation that doesn't touch Python objects. Py_END_ALLOW_THREADS reacquires the lock.
If you're referring to Python code, the interpreter will occasionally release the GIL, as Gustavo said. But if you're referring to C code, it is your responsibility to release the lock at the places where you want other threads to run. If you have a long calculation, I suppose you could add pairs of Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to avoid starvation of other threads.