[Twisted-Python] Writing a wrapper from asynchronous to synchronous
Hi everyone, I know someone is going to say it's a flaw in the design - and I partially agree with that. However I have an application that allows a fairly unexperienced programmer to add modules. What I need to do is to write a wrapper that hides the asynchronous nature of pb (well, Twisted that is, but I'm only using pb), simply because most programmers at the potential customers don't have any knowledge about asynchonous programming. The API for the modules should hide all communications to the server side. So my question would be, is there a way to write some kind of wrapper that will make the caller wait for the deferred callback to be invoked? I have one specific method that just has a flag which is set to true in the deferred callaback which then in turn triggers the action required. So basically it's a recursive method that get's called via a reactor.callLater and if the flag is true calls whatever is needed. The point is to suspend execution of a certain method until some prerequisite initializations are finished. I'd like to do that in an as convenient way as possible. So it's like: 1) module calls some method #1 2) #1 depends on certain initializations to finish 3) the call to #1 doesn't return unless step 2) is done any smart idea's how to wrap that up? Any hints are highly appreciated UC -- Open Source Solutions 4U, LLC 1618 Kelly St Phone: +1 707 568 3056 Santa Rosa, CA 95401 Cell: +1 650 302 2405 United States Fax: +1 707 568 6416
"Uwe C. Schroeder"
1) module calls some method #1 2) #1 depends on certain initializations to finish 3) the call to #1 doesn't return unless step 2) is done
I've not yet learned PB, so maybe this isn't a good fit but what about having modules run in threads and use Queues for them to send and receive messages (method calls / return values) to/from the framework? This way they won't block the framework but will appear to run synchronously. You could hide the Queue manipulation in a thin API that shadows the API of the framework. The timeline would be: Client calls wrapper function Function builds message, sends it to nonblocking framework queue and then listens on blocking reply queue. Framework checks input queue in nonblocking way. When it recieves a message it calls the corresponding asynchronous method and attaches code to the callback that will send the reply down the reply queue. Wrapper function gets reply, unpacks return and returns it to synchronous caller. I've used a similar pattern for doing the opposite of sending commands from async framework to syncronous client code. -Brett.
On Thu, 23 Feb 2006 22:46:03 -0800, "Uwe C. Schroeder"
Hi everyone,
I know someone is going to say it's a flaw in the design
I would hate to disappoint you. "It's a flaw in the design."
- and I partially agree with that. However I have an application that allows a fairly unexperienced programmer to add modules. What I need to do is to write a wrapper that hides the asynchronous nature of pb
No, you don't. What you need to do is to edcucate your target developer market about asynchronous programming. It's not that hard. The TurboGears "20 minute wiki" tutorial even covers Deferreds, without really bothering to explain how they work. If average JavaScript monkeys can figure this out, I think you're not giving your target developers enough credit. You cannot "hide the asynchronous nature of PB". The protocol is fully asynchronous even if you forget all of Twisted and just look at the byte-level spec. Your choice is not "synchronous or asynchronous interface". It is "educate novice developers about Deferreds" or "educate novice developers about multithreaded concurrency, reactor implementation details, livelock, deadlock, and spurious reentrancy". That's assuming that you fix any outstanding reactor bugs related to this issue, fully test the related code paths, AND there are no bugs in your wrapper layer. Trust me: there will be bugs in your wrapper layer. There are maybe 20 people in the world who could make Twisted do sync/async mapping *really* correctly, and all of them understand the problem too well to bother to try.
On Fri, 24 Feb 2006 10:53:17 -0600,
On Thu, 23 Feb 2006 22:46:03 -0800, "Uwe C. Schroeder"
wrote: Hi everyone,
I know someone is going to say it's a flaw in the design
I would hate to disappoint you.
"It's a flaw in the design."
- and I partially agree with that. However I have an application that allows a fairly unexperienced programmer to add modules. What I need to do is to write a wrapper that hides the asynchronous nature of pb
[SNIP Glyph explains why this is not a good thing, or even possible] I'm always fascinated with these kinds of issues, because they come up fairly often, and are so illustrative of human nature. The original poster feels that he needs to do X, and is most likely not interested in anyone's opinion of the actual desirability/validity of so doing. And yet, what response does his question receive? A detailed explanation of why he should not consider doing X, even though it is clear that X is what he feels he must do. Of course, I agree that X is definitely not something that the original poster should contemplate doing; it is just a bad idea, period, and will likely result in frustration and disappointment. However, I think it is important to point out that the original poster is most likely pursuing this line of inquiry because the possibility of educating his target audience is slim, or none. People are much harder to deal with than machines, especially when it comes to learning new things. Forcing twisted to behave synchronously doubtless sounds easier than teaching an inexperienced coder how to write twisted-friendly code. My pointless musings aside, I do have a concrete suggestion for the original poster: Create an HTTP server that proxies the pb calls on behalf of your programmers. Let them just post messages to that server, using whatever synchronous HTTP client (urllib, etc) they desire. Not technically elegant, no, but far easier than mangling twisted, and far easier than trying to teach your programmers how to write async code. Note that the server doesn't have to be HTTP, but most programmers have some passing acquaintance with it, and client libraries are ubiquitous, so it is an easy option. Hope this helps, L. Daniel Burr
On Thursday 23 February 2006 11:46 pm, Uwe C. Schroeder wrote:
Hi everyone,
I know someone is going to say it's a flaw in the design - and I partially agree with that. However I have an application that allows a fairly unexperienced programmer to add modules. What I need to do is to write a wrapper that hides the asynchronous nature of pb
Lol, when I read that I wondered how long it will it take before someone _did_ respond with "It's a flaw in the design" I actually was trying to do this _exact_ thing (making pb appear synchronous) and met with similar remarks to my post for help. But even though I did manage to write a hack using a interleaved threadedselectreactor, threading.Event and a forced event processing while loop, the result was mostly functional but ugly and a pain to debug. It was a flaw in my design. And after some serious reworking and acceptance of a more asynchronous methodology, my program is the better for it. Way better. In short, I would recommend _not_ forcing/hacking twisted to be something its not. Use L. Daniel's suggestion, use a synchronous network system like xmlrpclib, or rethink the meta problem your are trying to solve. --gabe
On Friday 24 February 2006 11:33, Gabe Rudy wrote:
On Thursday 23 February 2006 11:46 pm, Uwe C. Schroeder wrote:
Hi everyone,
I know someone is going to say it's a flaw in the design - and I partially agree with that. However I have an application that allows a fairly unexperienced programmer to add modules. What I need to do is to write a wrapper that hides the asynchronous nature of pb
Lol, when I read that I wondered how long it will it take before someone _did_ respond with "It's a flaw in the design"
Me too actually :-)
I actually was trying to do this _exact_ thing (making pb appear synchronous) and met with similar remarks to my post for help. But even though I did manage to write a hack using a interleaved threadedselectreactor, threading.Event and a forced event processing while loop, the result was mostly functional but ugly and a pain to debug.
It was a flaw in my design.
And after some serious reworking and acceptance of a more asynchronous methodology, my program is the better for it. Way better.
In short, I would recommend _not_ forcing/hacking twisted to be something its not. Use L. Daniel's suggestion, use a synchronous network system like xmlrpclib, or rethink the meta problem your are trying to solve.
In the end it's not a flaw in the design. The software package with close to 1.5 million lines of code runs just fine and did so for many years. The problem is that customers hire some homegrown MS Access programmers for whom anything remotely like network programming is a black box and expect them to be able to write a module for my system. It's pretty hard to explain deferred's to someone who thinks a socket is the power outlet where he plugs his vacuum in and a port is the thing where the ships go to. Guess who gets the blame when their employees can't handle it. Thus the idea to make that API a bit less "deferred". Yeah - I should tell them "hire decent programmers who have a clue"... Well, was worth a try - someone might have come up with something... UC -- Open Source Solutions 4U, LLC 1618 Kelly St Phone: +1 707 568 3056 Santa Rosa, CA 95401 Cell: +1 650 302 2405 United States Fax: +1 707 568 6416
On 2/24/06, Uwe C. Schroeder
In the end it's not a flaw in the design. The software package with close to 1.5 million lines of code runs just fine and did so for many years. The problem is that customers hire some homegrown MS Access programmers for whom anything remotely like network programming is a black box and expect them to be able to write a module for my system. It's pretty hard to explain deferred's to someone who thinks a socket is the power outlet where he plugs his vacuum in and a port is the thing where the ships go to. Guess who gets the blame when their employees can't handle it. Thus the idea to make that API a bit less "deferred". Yeah - I should tell them "hire decent programmers who have a clue"... Well, was worth a try - someone might have come up with something...
I came across this problem recently. I ended up using XML-RPC.NET for the VB side, and twsited's own XML-RPC server for my side. It took me about a week to realise that this was how to resolve my problem, rather than working out (in my case) how to make PB be COM-Compliant. It had the advantage that I just passed an interface file alogn to my VB programmer who could plug it into hsi system, and he got all his fancy syntax checkign and pop up values and all the rest. I did consider SOAP, but WSDL gave me a headache, and I will apparently have to cope with a VB6 project hooking up to my code in the near future, for which there is a COM-based XML-RPC library available. If it is something they're goign to be "hanging" on for a significant amount of time, you can, at the very least, offer them a UUID and a method that they can poll to see if the deferred has fired. Moof
participants (6)
-
Brett Viren
-
Gabe Rudy
-
glyph@divmod.com
-
L. Daniel Burr
-
Moof
-
Uwe C. Schroeder