PyGilState_Ensure interrupts python critical sections

[billy.omahony at gmail.com]

Many thanks for the clarification. Also good idea to focus on lock
ownership rather that thread activity in your diagram.

To be honest I was not actually experiencing deadlock issues. I had
just deduced (incorrectly) that I might do so if I started using locks
in my py code called from c-land. As I was suffering some horrible
race-condtions that had suddenly appeared in code that had been
perfectly stable for the past 6 months I didn't want to go down that
route given my misconceptions; Even if introducing the locks worked
I'd always feel it was a deadlock waiting to happen.

Misunderstanding arose as for some reason I had decided that python
code called from C did not allow any other python threads to run until
control had returned to C and the gil had been released from C-land.
I.e. once the gil was given to the C thread then python had to wait
until it was given back.

I have to say the docs could be a little more explicit on the
mechanisms involved. They tend to be better at stating what individual
functions do rather that giving overall explanations. Maybe I'll get
around to submitting a chapter on it ;)


On Nov 12, 4:54 pm, "Chris Mellon" <arka... at gmail.com> wrote:
> On Nov 12, 2007 6:56 AM,  <billy.omah... at gmail.com> wrote:
>
>
>
> > Hi,
>
> > I have a native windows thread in a c python module which calls into
> > python code and adds an item to a data structure (a home-grown
> > circular buffer). At the same time my main python application is
> > removing items from this data structure.
>
> > Unlike native python containers adding and removing items from the
> > circular buffer is not atomic. So I need to put access to it in a
> > critical section. Using threading.Lock will work for native python
> > threads competing for access. But when access is from a windows thread
> > this will not work. That is because my call to PyGilState_ensure will
> > preempt my native python thread ***even when it is inside the critical
> > section***.
>
> > What is going on looks something like this (I think).
>
> >    Py      Python       Windows       Py       threading.Lock
> > resource
> >   Sched    Thread       Thread       Code            |              |
> >     |                     |            |             |              |
> >     |Go (GIL)#            |            |             |              |
> >     |        #            |            |             |              |
> >     |        #            |            |             |              |
> >     |        #...Doit.....|...........>|             |              |
> >     |        #            |            |. acquire...>|              |
> >     |<-PyGILState_Ensure--|            |             |              |
> >     |       ...          ...          ...           ...
> >     |Stop    #
> >     |-------`|
> >     |
> >     |----Ensure rtns-----># PyObject_  |                            |
> >              |   :        |CallMethod  |             |              |
> >              |   :        |.(Doit)...> |. acquire...>| DEADLOCK     |
> >                  :
> >                  :
> >                  :.how does python thread tell
> >                     PyScheduler not to give away
> >                       Gil until we are done with
> >                           critical section??
>
> > So my question is how in python do I tell the scheduler not to prempt
> > the current python thread. Especially how to tell it not to give the
> > GIL to any calls to PyGILState_ensure until further notice (i.e. until
> > I am outside my critical section?? It sounds like a reasonable request
> > - surely this exists already but I can't find it.
>
> It took me some time to understand the problem you were describing
> because you've got some terms backward - there's no Python scheduler,
> and calling PyGILState_ensure doesn't preempt anything. The Python
> interpreter *releases* the GIL every so often to allow other threads
> looking for it to run, but calling the python GIL functions has no
> effect on preemption.
>
> The problem is that the GIL is being released while your object lock
> is held, a second thread (started from C) acquires the GIL and then
> blocks on the object lock. What you seem to be seeing is that it
> blocking on the object lock is preventing it from releasing the GIL,
> which prevents the python thread from running and releasing the lock.
>
> This shouldn't happen - blocking on a lock releases the GIL, so the
> python thread should run, release the GIL, and eventually your C
> thread should be able to acquire both locks at the same time. Are you
> sure that you're correctly acquiring the GIL in your C code?
>
> The data flow you *should* be seeing should look something like this:
>
> GIL     object lock (P=Python, C=C, *=released)
> -------------------
> P       P           Python holds both locks
> *       P           Python releases the GIL, inside critical section
> C       P           C thread acquires the GIL and starts executing Python code
> *       P           C thread tries to acquire object lock and blocks,
> releasing the GIL
> P       P           Python thread re-acquires the GIL
> P       *           Python thread exits critical section and releases
> the object lock
> *       *           Python thread releases the GIL (can be in any
> order with next state)
> *       C           The C thread acquires the object lock and blocks on the GIL
> C       C           C thread acquires the GIL and continues execution.
>
> > One thing that may work (though the documentation does not
> > specifically say so) is using setcheckinterval() to set the check
> > interval to a very large value, acessing my shared structure and then
> > setting the check interval back to the previous value. Provided my
> > access to the shared structure takes less byte codes than what I set
> > checkinterval to I should be okay. However that would be very
> > dependant on the exact fine detail of how the check interval works and
> > may not be compatible with other Python releases
>
> > Maybe someone familiar with the python source code would know for
> > sure?
>
> > I am using Python 2.4.3 on windows XP.
>
> > Thanks for any help/suggestions offered!
> > BR,
> > Billy.
>
> > --
> >http://mail.python.org/mailman/listinfo/python-list