[capi-sig] Threading in Python

[Hrvoje Niksic]

Swapnil.ST at gmail.com writes:

> 1. why does the interpreter ie the main thread keeps releasing and
> reacquiring the GIL every few instruction. why isn't GIL simply
> allowed to  be acquired by any thread which requires it then.

Because a thread requires it to even be able to run.  Release the GIL
== give other threads a chance to run.

> I mean why is the acquisition and release of a lock kept periodic
> and not on need basis?( which is obviously the way it should be)

Because holding the GIL allows a thread to run, and all threads need
to run at (almost) all times.  The GIL is somewhat peculiar in its
mode of operation.  A typical lock guards a resource and is by default
released.  The thread that needs a resource acquires the lock, uses
the resource, and releases the lock.

GIL, on the other hand, is almost always acquired by some thread, and
that thread is allowed to access Python data and run Python code.
Py_BEGIN_ALLOW_THREADS actually *releases* the lock (in contrast to
"begin foo" synchronization blocks which tend to acquire locks),
giving other threads a chance to run during a blocking syscall or a
long-winded calculation that doesn't touch Python objects.
Py_END_ALLOW_THREADS reacquires the lock.

> 3. suppose I am not making any blocking call then I would never
> release GIL  since that is the only thing guarding my thread state and
> i don't want my  thread state messed up any time my thread is
> running. Which implies  unacceptable starvation of other threads.

If you're referring to Python code, the interpreter will occasionally
release the GIL, as Gustavo said.  But if you're referring to C code,
it is your responsibility to release the lock at the places where you
want other threads to run.  If you have a long calculation, I suppose
you could add pairs of Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to
avoid starvation of other threads.