[Twisted-Python] Creating a COM object in a thread
I need to create a COM object in a Windows application, and call it. Since the call will take some time to execute, I wrap it in a deferToThread.
I've found that, when I create the object inline, it works. However, when I defer it, it hangs up in the win32com.client.Dispatch call. I've tried several things, including digging into the guts of Dispatch -- the hangup occurs during the creation of the object.
I've discussed this with Mark Hammond, who suggests one lead:
"The Windows message loop is used by the COM marshalling process. IIRC, the first thread to initialize COM in a process is the thread in which single-threaded objects will always end up being called in. If a different thread creates the object, COM uses Windows messages to marshall all calls back to that main thread. IOW, your second thread makes a call - even to create the object - which results in that thread sending a windows message to the main thread to act on the request.
What this probably means in practice is that twisted needs to use a reactor which calls MsgWaitForMultipleObjects() and runs a message loop when the function detects a new message is in the queue. I'm not sure if there is an existing reactor which does this."
I'm posting this on the off chance that someone else has wandered into this corner of Windows arcana, and has come up with something useful.
I'm about to give up on doing this "inside" the twisted app, and running it in a separate app that I'll call from the thread.
TIA,
On Mon, 2010-06-28 at 17:43 -0700, Don Dwiggins wrote:
What this probably means in practice is that twisted needs to use a reactor which calls MsgWaitForMultipleObjects() and runs a message loop when the function detects a new message is in the queue. I'm not sure if there is an existing reactor which does this."
win32eventreactor uses WaitForMultipleObjects, so could probably be modified quite simply to support this. It's also vaguely possible you could use the more scalable IOCP reactor, but someone with more win32 knowledge than I should comment on that.
Itamar, Thanks for the suggestions. The documentation of win32eventreactor sounds kind of scary, but still probably the best base to work from.
I did some searching, and found Thomas Heller's win32guireactor at http://osdir.com/ml/python.twisted/2003-02/msg00190.html. I'm still naive enough about reactors not to be able to compare it to win32eventreactor, except to note that it does run PumpWaitingMessages, which is what Mark thinks is needed. (This was posted in 2003; as far as I can tell, nothing further has been done with it.)
Any advice on this? I'm willing to spend some time to give it a try, and to post my experiences, if it'll help advance the "state of the art" of win32 reactors.
On 6/28/2010 6:17 PM, Itamar Turner-Trauring wrote:
On Mon, 2010-06-28 at 17:43 -0700, Don Dwiggins wrote:
What this probably means in practice is that twisted needs to use a reactor which calls MsgWaitForMultipleObjects() and runs a message loop when the function detects a new message is in the queue. I'm not sure if there is an existing reactor which does this."
win32eventreactor uses WaitForMultipleObjects, so could probably be modified quite simply to support this. It's also vaguely possible you could use the more scalable IOCP reactor, but someone with more win32 knowledge than I should comment on that.
I'm sure you've already checked the simple things like this, but have you called CoInitialize() in your thread? If you're using the win32 extensions, it'll be called for you in the main thread, but any new thread needs to call it before attempting to do any COM work.
On Mon, Jun 28, 2010 at 7:43 PM, Don Dwiggins ddwiggins@advpubtech.com wrote:
I need to create a COM object in a Windows application, and call it. Since the call will take some time to execute, I wrap it in a deferToThread.
I've found that, when I create the object inline, it works. However, when I defer it, it hangs up in the win32com.client.Dispatch call. I've tried several things, including digging into the guts of Dispatch -- the hangup occurs during the creation of the object.
I've discussed this with Mark Hammond, who suggests one lead:
"The Windows message loop is used by the COM marshalling process. IIRC, the first thread to initialize COM in a process is the thread in which single-threaded objects will always end up being called in. If a different thread creates the object, COM uses Windows messages to marshall all calls back to that main thread. IOW, your second thread makes a call - even to create the object - which results in that thread sending a windows message to the main thread to act on the request.
What this probably means in practice is that twisted needs to use a reactor which calls MsgWaitForMultipleObjects() and runs a message loop when the function detects a new message is in the queue. I'm not sure if there is an existing reactor which does this."
I'm posting this on the off chance that someone else has wandered into this corner of Windows arcana, and has come up with something useful.
I'm about to give up on doing this "inside" the twisted app, and running it in a separate app that I'll call from the thread.
TIA,
Don Dwiggins Advanced Publishing Technology
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
The way I do this is I isolate all the COM stuff in a separate thread from thread that runs my reactor.
This thread starts by calling:
pythoncom.CoInitializeEx(pythoncom.COINIT_APARTMENTTHREADED)
it then creates the COM object and runs an event loop something like this:
while keepRunning: n = MsgWaitForMultipleObjects(handles, False, timeout, QS_ALLINPUT) if n == WAIT_OBJECT_0 + nHandles: # There is a win32 message waiting. if pythoncom.PumpWaitingMessages(): keepRunning = False
I send messages to the thread by putting them in a queue and setting an event. The messages cause methods to be called on the COM object. I send the results of the COM methods back to the reactor thread using reactor.callFromThread().
It's a bit of effort to do this, but it works really well. It's been in production for several years and I've never had a problem with it.
Hope this helps, Hugh
On Tue, Jun 29, 2010 at 10:43 AM, Don Dwiggins ddwiggins@advpubtech.com wrote:
I need to create a COM object in a Windows application, and call it. Since the call will take some time to execute, I wrap it in a deferToThread.
I've found that, when I create the object inline, it works. However, when I defer it, it hangs up in the win32com.client.Dispatch call. I've tried several things, including digging into the guts of Dispatch -- the hangup occurs during the creation of the object.
I've discussed this with Mark Hammond, who suggests one lead:
"The Windows message loop is used by the COM marshalling process. IIRC, the first thread to initialize COM in a process is the thread in which single-threaded objects will always end up being called in. If a different thread creates the object, COM uses Windows messages to marshall all calls back to that main thread. IOW, your second thread makes a call - even to create the object - which results in that thread sending a windows message to the main thread to act on the request.
What this probably means in practice is that twisted needs to use a reactor which calls MsgWaitForMultipleObjects() and runs a message loop when the function detects a new message is in the queue. I'm not sure if there is an existing reactor which does this."
I'm posting this on the off chance that someone else has wandered into this corner of Windows arcana, and has come up with something useful.
I'm about to give up on doing this "inside" the twisted app, and running it in a separate app that I'll call from the thread.
TIA,
Don Dwiggins Advanced Publishing Technology
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hugh,
The way I do this is I isolate all the COM stuff in a separate thread from thread that runs my reactor.
This all looks good, and I think I understand it. One question, though: from my investigation, it seems that the hangup is in the COM object creation itself, where Mark and I think it's waiting on a Windows message. Wouldn't I have to have the event loop running before calling Dispatch? (FWIW, I do call CoInitializeEx the same way you do, and have the object built with apartment threading.)
This thread starts by calling:
pythoncom.CoInitializeEx(pythoncom.COINIT_APARTMENTTHREADED)it then creates the COM object and runs an event loop something like this:
while keepRunning: n = MsgWaitForMultipleObjects(handles, False, timeout, QS_ALLINPUT) if n == WAIT_OBJECT_0 + nHandles: # There is a win32 message waiting. if pythoncom.PumpWaitingMessages(): keepRunning = FalseI send messages to the thread by putting them in a queue and setting an event. The messages cause methods to be called on the COM object. I send the results of the COM methods back to the reactor thread using reactor.callFromThread().
Thanks much for the help,
On Tue, Jul 6, 2010 at 11:26 AM, Don Dwiggins ddwiggins@advpubtech.com wrote:
Hugh,
The way I do this is I isolate all the COM stuff in a separate thread from thread that runs my reactor.
This all looks good, and I think I understand it. One question, though: from my investigation, it seems that the hangup is in the COM object creation itself, where Mark and I think it's waiting on a Windows message. Wouldn't I have to have the event loop running before calling Dispatch? (FWIW, I do call CoInitializeEx the same way you do, and have the object built with apartment threading.)
I got the order wrong. On further reading of the code, the COM object is created in response to a message from the Twisted thread, so at that point the event loop has iterated at least once.
In my case the COM object is wrapped in a C++ wrapper that I don't have the source for, so I don't know exactly what happens in there. That C++ wrapper is then wrapped in a very thin python extension module which doesn't do anything interesting.
Thanks much for the help,
My pleasure.
Hugh
On 7/5/2010 10:53 PM, Hugh Emberson wrote:
I got the order wrong. On further reading of the code, the COM object is created in response to a message from the Twisted thread, so at that point the event loop has iterated at least once.
In my case the COM object is wrapped in a C++ wrapper that I don't have the source for, so I don't know exactly what happens in there. That C++ wrapper is then wrapped in a very thin python extension module which doesn't do anything interesting.
OK. In my case, the "wrapper" and Python extension is Hammond's win32com Dispatch.
Question: to your knowledge, would it work to run the message pumping loop in a different thread from the one that creates and calls the COM object? I think this would simplify things a bit.
-- Don Dwiggins Advanced Publishing Technology
On Thu, Jul 8, 2010 at 9:44 AM, Don Dwiggins ddwiggins@advpubtech.com wrote:
On 7/5/2010 10:53 PM, Hugh Emberson wrote:
I got the order wrong. On further reading of the code, the COM object is created in response to a message from the Twisted thread, so at that point the event loop has iterated at least once.
In my case the COM object is wrapped in a C++ wrapper that I don't have the source for, so I don't know exactly what happens in there. That C++ wrapper is then wrapped in a very thin python extension module which doesn't do anything interesting.
OK. In my case, the "wrapper" and Python extension is Hammond's win32com Dispatch.
Question: to your knowledge, would it work to run the message pumping loop in a different thread from the one that creates and calls the COM object? I think this would simplify things a bit.
You'd have to use a multi-threaded apartment model. See the 'Processes, Threads, and Apartments' article in MSDN ( http://msdn.microsoft.com/en-us/library/ms693344%28v=VS.85%29.aspx ). I guess it should work, but I've never tried it. Please report back if you try it.
Hugh
participants (4)
-
Don Dwiggins -
Hugh Emberson -
Itamar Turner-Trauring -
Mark Wright