Re: [Twisted-Python] Wrapping a Perspective
Hi Rasjid:
It might be possible to create a new twisted reactor that integrates into the python com server event loop. That would presumably be the correct way to deal with this. But that would seem like far more work that I have time to devote to this problem at this time.
Hopefully, I understood your problem correctly... If you are willing to use Stackless Python, I would recommend you use Christopher Armstrong's threadlesss.py blockOn function. This would allow Twisted calls to block. A few months ago, I posted a solution that would work, but it would be overkill. I am also curious why you can't use inline callbacks? If I can find the time, I will post an example.
So my question really is: What are the problems / brawbacks with running the twisted reactor in a non-main thread, provided all calls to the reactor are done through twisted.internet.blockingCallFromThread?
I have used threads with Twisted and Stackless and found I took a performance hit. Also I found it prudent to heed the advice and try not to mix threads with Twisted (or Stackless) Cheers, Andrew ____________________________________________________________________________________ Fussy? Opinionated? Impossible to please? Perfect. Join Yahoo!'s user panel and lay it on us. http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7
On 9/6/07, Andrew Francis <andrewfr_ice@yahoo.com> wrote:
If you are willing to use Stackless Python, I would recommend you use Christopher Armstrong's threadlesss.py blockOn function. This would allow Twisted calls to block. A few months ago, I posted a solution that would work, but it would be overkill.
I have read about Stackless, and it looks like it would be really fun to play with. But I don't think my employer would be keen to go as 'fringe' as Stackless.
I am also curious why you can't use inline callbacks?
I've only been playing with Twisted seriously for a couple of weeks, but as far as I understand inline callbacks, they just make the code 'look' synchronous, but the end result is still a deferred. In my playing with them, they make twisted code much more succinct and generally easier to read, but it is all still really just deferreds underneath and there is no actual blocking going on. I want to have my python code being called from an external process via COM, and the external process will expect to be calling a class method that will do something (blocking as long as required) and then return the result. Rewriting the external app to deal with everything being async is not an option. I'd love to be shown that I'm wrong on this, but can't see how inline callbacks solve my problem.
So my question really is: What are the problems / brawbacks with running the twisted reactor in a non-main thread, provided all calls to the reactor are done through twisted.internet.blockingCallFromThread?
I have used threads with Twisted and Stackless and found I took a performance hit. Also I found it prudent to heed the advice and try not to mix threads with Twisted (or Stackless)
The other option I have thought about is using the Python processing module (http://pypi.python.org/pypi/processing), which would have the com server and the twisted client each in their own python process (and thus Twisted would then be single threaded), although it is unclear to me if this is making the situation better or worse in reality. Cheers, Rasjid.
I've only been playing with Twisted seriously for a couple of weeks, but as far as I understand inline callbacks, they just make the code 'look' synchronous, but the end result is still a deferred. In my playing with them, they make twisted code much more succinct and generally easier to read, but it is all still really just deferreds underneath and there is no actual blocking going on.
Correct. Effectively, it uses python generators to make the code "nice"; state variables can be kept as function-locals, and flow control can be somewhat easier to visualise.
I want to have my python code being called from an external process via COM, and the external process will expect to be calling a class method that will do something (blocking as long as required) and then return the result. Rewriting the external app to deal with everything being async is not an option.
I suggest you abandon Perspective Broker; it's a fully async protocol, and by trying to call it in a sync/blocking manner you're going to get nowhere fast. Use something easy like XMLRPC. Twisted can implement that as a server with full deferred-capable semantics, it'll run over HTTPS, it's got authentication, it's got .net bindings etc.
On 9/6/07, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
I want to have my python code being called from an external process via COM, and the external process will expect to be calling a class method that will do something (blocking as long as required) and then return the result. Rewriting the external app to deal with everything being async is not an option.
I suggest you abandon Perspective Broker; it's a fully async protocol, and by trying to call it in a sync/blocking manner you're going to get nowhere fast.
Use something easy like XMLRPC. Twisted can implement that as a server with full deferred-capable semantics, it'll run over HTTPS, it's got authentication, it's got .net bindings etc.
That doesn't make any sense: XMLRPC can be used asynchronously, as PB can. Switching to XMLRPC won't change anything about his situation, the biggest problem being that his COM interface doesn't support callbacks (?). Rasjid: Being asynchronous on the server does not require being asynchronous on the client. Communication boundaries let you do that. You should check to see if the "being a COM server" interface allows you to accept a call, and then later asynchronously trigger the response. Ideally, it would allow this, and the client would be none the wiser about whether the implementation of the server is blocking or not, while it's waiting for the response to its request. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/
That doesn't make any sense: XMLRPC can be used asynchronously, as PB can. Switching to XMLRPC won't change anything about his situation, the biggest problem being that his COM interface doesn't support callbacks (?).
I read the OPs problem is precisely the opposite; they are wanting to use PB sync (which involves horrible reactor tricks).
On 9/7/07, Christopher Armstrong <radix@twistedmatrix.com> wrote:
Switching to XMLRPC won't change anything about his situation, the biggest problem being that his COM interface doesn't support callbacks (?).
Actually, on further research it looks like the Python COM implementation does support callbacks, although it looks somewhat complex.
Rasjid: Being asynchronous on the server does not require being asynchronous on the client. Communication boundaries let you do that.
Got it. :-) I think I'm beginning to understand where the risks with threads are. With threads, you can accidentally bypass the communication boundaries, and code may mostly appear to work but randomly fail at times. Thanks to all for the responses. Cheers, Rasjid.
On 9/6/07, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
I suggest you abandon Perspective Broker; it's a fully async protocol, and by trying to call it in a sync/blocking manner you're going to get nowhere fast.
Use something easy like XMLRPC. Twisted can implement that as a server with full deferred-capable semantics, it'll run over HTTPS, it's got authentication, it's got .net bindings etc.
I have used XMLRPC in the past (until recently it was my main use of Twisted), but it does not support complex objects, nor callbacks. PB does both, but just in an async manner. I could use CORBA instead of PB, but then I'm likely to be in the position where I want to learn Twisted and CORBA - I'd rather just learn Twisted, as long as I can get it to play with Windows COM in the way that I want.
On Fri, 2007-09-07 at 17:21 +1000, Rasjid Wilcox wrote:
On 9/6/07, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
I suggest you abandon Perspective Broker; it's a fully async protocol, and by trying to call it in a sync/blocking manner you're going to get nowhere fast.
Use something easy like XMLRPC. Twisted can implement that as a server with full deferred-capable semantics, it'll run over HTTPS, it's got authentication, it's got .net bindings etc.
I have used XMLRPC in the past (until recently it was my main use of Twisted), but it does not support complex objects,
Well, it an pass arrays and dictionaries; how much more complex can objects actually get, unless you want callbacks?
nor callbacks. PB does both, but just in an async manner.
Ah. You want callbacks. ;o) So, let me be be sure I've understood what you want: windows client | \- VB process | <COM> | \- python COM component | <maybe Twisted in a thread> | <some kind of complex object/callback capable RMI> | /- Python object ----> another COM object | /- Twisted Process | windows server I have a hard time seeing why you wouldn't just use DCOM for this, but anyway... Is the python COM component on the client in-process or out of process? I assume you have no issues creating and talking to the COM object on the server? Christopher Armstrong has already suggested looking at COM to see if it can support "async" components; it's been a while, but I seem to recall it can. You'll still need to run the reactor in a thread of course.
I could use CORBA instead of PB, but then I'm likely to be in the
I don't see how that would help you.
position where I want to learn Twisted and CORBA - I'd rather just learn Twisted, as long as I can get it to play with Windows COM in the way that I want.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 9/7/07, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
So, let me be be sure I've understood what you want:
windows client | \- VB process | <COM> | \- python COM component | <maybe Twisted in a thread> | <some kind of complex object/callback capable RMI> | /- Python object ----> another COM object | /- Twisted Process | windows server
That is pretty much it exactly! :-)
I have a hard time seeing why you wouldn't just use DCOM for this, but anyway...
Primarily because DCOM seems pretty fixed client -> server, whereas I want to be able to send out a UDP multicast packet, discover which PCs I wish to talk to, query a number of them, and then run some remote methods. And as indicated in your diagram above, the remote python object may or may not be a COM object. The above arrangement also allows the remote object to be running on a Linux box for example, which is trickier (or at least more expensive) with DCOM. I would also consider running PB over SSL across the public internet, whereas DCOM would have to be via a VPN. The very long term goal is the gradual migration from VB to Python. That is, gradually more all core code into Python, wraping Python code up as COM objects so it can be used within the current program. At some point (might take a year or two) the VB program is nothing but a front-end to Python code, at which point we could migrate the front-end to wxPython or similar. And at that point a migration from Windows to Linux at least becomes a feasible option to contemplate. Thus I have a long term motivation to avoid DCOM even if it was easier in the first instance.
Is the python COM component on the client in-process or out of process?
I have not decided that yet. I have a feeling that out-of-process is probably better in this case, but I'm still doing a fair bit of research and learning on all this.
I assume you have no issues creating and talking to the COM object on the server?
At the moment the 'server side' object in question is a Python object that wraps up several COM objects and presents a much simpler interface.
Christopher Armstrong has already suggested looking at COM to see if it can support "async" components; it's been a while, but I seem to recall it can.
I believe it can, but the examples in win32com are a bit terse.
You'll still need to run the reactor in a thread of course.
Yes! And this is the part where I want some advice. What are the do's and don't of doing this? And what are the most important things I need to read / research? Cheers, Rasjid.
participants (4)
-
Andrew Francis -
Christopher Armstrong -
Phil Mayers -
Rasjid Wilcox