(Win32 API) callback to Python, threading hiccups

[Francois De Serres]

Hiho,

could somebody please enlighten me about the mechanics of C callbacks to 
Python? My domain is more specifically callbacks from the win32 API, but 
I'm not sure that's where the problem lies. Here's a description...

I want a callback-based MIDI input/processing, so PortMidi was not an 
alternative. I have written a C extension module that links to the mmsys 
MIDI API. I separated the win32-dependant code from the Python extension 
code, so a) the module interface is system-neutral, b) the 
implementation can be tested (re-used) outside of Python. So, that code 
is tested OK, but it might be useful to sketch it's design:
- as you may know, the MIDI input on win32 is already managed thru a 
callback mechanism; the driver calls back your program with data buffers
- yet, I don't call directly into Python from the callback, since some 
restrictions apply on what system calls you can make from there, and I 
don't know what Python's interpreter / scripts might call.
- so, on callback, I create a new thread, after checking that the 
previous one has returned already (WaitOnSingleObject(mythread)) so we 
only have one thread involved.
- this is this thread that calls the user callback, yet this callback 
isn't yet a Python callable, we're still in native C code.
- on the side of the extension module now, I merely copied the callback 
example from the Python/C API doc, and added GIL management around the call:

static PyObject * my_callback = NULL; //this gets fixed by a 
setCallback() func
static void external_callback(const MidiData * const data) {   
    if (my_callback && (my_callback != Py_None)) {
        if (! data) {
            PyErr_SetString(PyExc_IndexError, getLastErrorMessage());
        } else {  
            PyObject * arglist = NULL;
            PyObject * result = NULL;
            arglist = Py_BuildValue("(i,i,s#)", data->deviceIndex, 
data->timestamp, data->buffer, data->size);// 0, 0, "test", 4);//
   
            PyGILState_STATE gil = PyGILState_Ensure();
            result = PyEval_CallObject(my_callback, arglist);   
            PyGILState_Release(gil);
           
            Py_DECREF(arglist);
            Py_DECREF(result);  
        }
    }
}

- this one above is what is actually passed as callback to the 'native' 
C part. So, here's what (I presume) happens:
1. the driver calls into my C code
2. my C code spawns a thread that calls into the extension
3. the extension calls into Python, after acquiring the GIL

Now, here's the hiccup:
inside a Python script, anytime a Python object is accessed by both the 
(Python) callback and the main program, I get a GPF :/

You bet I tried to use locks, Queues and whatnot, but nothing will do, 
it seems I get protection faults on accessing... thread exclusion objects.

Yet, there is a way where it works seamlessly: it's when the __main__ 
actually spawns a threading.Thread to access the shared object. Hence I 
am lost.

This works:

def input_callback(msg):
    midiio.send(msg) ##MIDI thru, a call into my extension that wraps 
the implementation call with BEGIN_ALLOW_THREADS and END_...
__main__:
    raw_input()

This results in GPF as soon as the callback is fired:

tape = Queue.Queue()
def input_callback(msg):
    tape.put(msg)
__main__:
    while True:
       print tape.get()

This works like a charm:

tape = Queue.Queue()
def input_callback(msg):
    tape.put(msg)
def job():
    while True:
       print tape.get()
__main__:
    t = threading.Thread(target = job)
    t.start()
    raw_input()

Note that I also get a GPF in the later case if I stop the program with 
KeyInterrupt, but I guess it's something I'll look into afterwards...

While I can think of hiding away the added threading within a wrapper 
module, I won't sleep well untill I figure out what's really happening. 
Could someone save my precious and already sparse sleeping time, by 
pointing me to what I'm missing here?

WinXP SP1
Python 2.4.1
MinGW GCC 3.4.2

TIA,
Francois De Serres