Threads, Callbacks, Classes [Long]

[Gerson Kurz]

I want to implement a wrapper for an asynchronous API in win32, that
uses window messages for completions. Wrapping synchronous functions
from a DLL seems to work easily enough, but I have serious trouble
with the asynchronous ones. 

Question 1a: I would like to make sure I understand how to properly
issue a python callback. Here is my C code:

--- cut here ---
static PyObject* func_TestCallback(PyObject* self, PyObject* args)
{
    PyObject* callback;
    if( !PyArg_ParseTuple(args,"O",&callback) )
    {
        PTrace( "ERROR, PyArg_ParseTuple() failed" );
        return NULL;
    }   
    PyObject_CallFunction(callback,"s","It worked !");
    return Py_None;
}
--- cut here ---

and this is how I can issue the callback from Python:

--- cut here ---
import pxfs
def myfunc(x):
	print x
pxfs.TestCallback(myfunc)
--- cut here ---

This seems to work. Its not asynchronous yet, but at least the
function gets called. However, I have a feeling I should somehow deal
with the returncode of PyObject_CallFunction ? 

Question 1b: As a followup on this, the code doesn't seem to work if I
give a member function:

--- cut here ---
import pxfs, Queue
msgport = Queue.Queue()
pxfs.TestCallback(msgport.put)
--- cut here ---

I would expect that the callbacks puts a message in the queue, but it
doesn't seem to work: no message is received when I later ask
msgport.get().

Question 2A: How to issue a callback from another thread.
 
The same code shown above doesn't work if issued from another thread
(a thread that has not been created by python, but by the asynchronous
API). I have found a way to get it "partially" to work - my callback
gets called - but eventually the python interpreter bails out with
thread state errors. I didn't quite get the (meager) documentation on
this. This is what I do:

- When requesting the function, I save PyThreadState_Get()->interp
- In the thread, I do

--- cut here ---
// m_ips is the interpreterstate saved from another thread
PyThreadState* pts = PyThreadState_New(m_ips)
PyEval_AcquireThread(m_pts);
PyObject_CallFunction(m_callback,"s","It Worked");
PyEval_ReleaseThread(m_pts);
--- cut here ---

- In the python program, I do

--- cut here ---
import pxfs, Queue

msgport = Queue.Queue()

def sendmessage(x):
    global msgport
    print "SENDMESSAGE CALLED USING", x
    msgport.put(x)

# This will start a NT native thread and do the 
# code shown above
pxfs.TestDeferredCallback(sendmessage)
while 1:
    msg = msgport.get()
    print "GOT MESSAGE", msg
--- cut here ---

I see the "SENDMESSAGE CALLED USING" bit, and then an invalid thread
state occurs.
- Question 2B: Do I have to call PyThreadState_Delete() at some point?
- Question 2C: Is there a more elaborate discussion on this topic
(Threads & Callbacks in Python) available somewhere on the net ?

Question 2D: Is my approach maybe completely wrong ? 

My problem is: the existing API drives hardware. The hardware
functions can take a long time (several minutes), during which normal
user input should be possible. On Windows, you have a message queue
and you have to manually keep this alive to wait for async
completions.

I could use win32api style window handles for the message processing,
but the code would quickly become as messy as it is in C. (A windows
message has two longs as parameters - thats it). I would love to use
the Queue.Queue mechanism, because it allows for much more "elaborate"
messages, but Queue.Queue seems to block the windows message queue
(its probably implemented using semaphores and not window handles ?).

Also, I later would like to add some wxPython GUI to it. AFAIK
callbacks would suit that framework best. 

Question 3: How to export classes

Exporting functions from an extension DLL is easy enough. How do I
export classes ? Exporting types seems to work (I tried the example
from the docs) but I don't see where I would add my class members to a
custom type.