[Twisted-Python] waitForDeferred Question
Hi, I have a function that returns a Deferred. I need to have the result of this Deferred returned in a (apparently) blocking/synchronous manner: def myfuncBlocking(): d = myfuncReturnsDeferred() ... result = return result I need to be able to call this function like: result = myfuncBlocking() The question is how to get the result out of the Deferred() and make it *look* like myfuncBlocking() has blocked. I have looked over the waitForDeferred() docstrings and code, but it is still not obvious how to use waitForDeferred() to accomplish this. The blockOn() capability of the gthreadless.py module (in the sandbox) seems to accomplish what I want, but I would like to use things that are in the main twisted trunk. What is the best way to go about this? Thanks Brian -- Brian Granger Santa Clara University ellisonbg@gmail.com
On Fri, 10 Mar 2006 17:12:43 -0800, Brian Granger <ellisonbg@gmail.com> wrote:
What is the best way to go about this?
"The only way to win is not to play the game." This issue has been discussed repeatedly - long story short, it's just a bad idea. Since you didn't explain your exact problem, I have to assume that there's a way to do the higher-level thing that you want without presenting a blocking interface. If you don't mind some vitriol, there is more information here, on my blog: http://glyf.livejournal.com/40037.html
I probably should have given more details about what I am trying to do. Before I get going, I should mention that I have been using Twisted heavily for 1.5 years and I have, for the most part, learned to play the "Twisted Game." So here is what I am working on: I am developing objects (for scientific computing) that 1) use Twisted to talk over the network and 2) need to be used interactively from a python prompt. This second point is really where the difficult and interesting things are. To allow Twisted-things to be used interactively I have built a version of PyCrust/PyShell that is "Twisted enabled." We have used the standard threadedselectreactor to interleave the Twisted event loop with that of PyShell. We also inject the reactor into the users interactive namespace. Thus, you can use all of Twisted's capabilities from an interactive python prompt. As a side note, this situation is extremely nice for playing around with Twisted and doing interactive debugging of Twisted-using applications. Here is a typical (madeup) usage case:
a = TwistedEnabledObject() a.connect(addr) # This uses reactor.connectTCP to connect to a server myresult = a.computeSomethingUsingTwisted(args) # This should block!!! ...and maybe... y = f(myresult) plot(y)
The key point is that the user will want to direclty use myresult interactively. Also they won't know ahead of time what actions they will want to do. Anyone who has used Matlab or Mathematica (scientists) will find this way of working extremely familiar. There is simply no way that users will want to get a Deferred() and add callbacks to trigger actions. Thus I would argue that I DO want to make my properly written asynchronous code blocking. The implementation of computeSomethingUsingTwisted() is the difficult part: class TwistedEnabledObject(): def connect(addr): self.factory = MyClientFactory() self.connector = reactor.connectTCP(addr[0],addr[1],self.factory) def computeSomethingUsingTwisted(args): d = self.connector.transport.protocol.computeAndReturnDeferred(args) # Now I have a Deferred that will have the result, but I want to wait until # the result or an error is ready and then decide what the result should be result = # what to put here? return result I don't think what I am doing goes against the "Twisted-way." And my needs are not coming from any inability on my part to write proper asynchronous code. The high level stuff here is really bbeing driven by the need to use these object interactively. On 3/10/06, glyph@divmod.com <glyph@divmod.com> wrote:
If you don't mind some vitriol, there is more information here, on my blog: http://glyf.livejournal.com/40037.html
I am familiar with this and other similar "vitriols." I fully agree with most of what you say about threads. But what does my usage case have to do with threads? cheers, Brian Brian Granger Santa Clara University ellisonbg@gmail.com
On 11 Mar 2006, at 15:44, Brian Granger wrote:
Here is a typical (madeup) usage case:
a = TwistedEnabledObject() a.connect(addr) # This uses reactor.connectTCP to connect to a server myresult = a.computeSomethingUsingTwisted(args) # This should block!!! ...and maybe... y = f(myresult) plot(y)
The key point is that the user will want to direclty use myresult interactively. Also they won't know ahead of time what actions they will want to do. Anyone who has used Matlab or Mathematica (scientists) will find this way of working extremely familiar. There is simply no way that users will want to get a Deferred() and add callbacks to trigger actions. Thus I would argue that I DO want to make my properly written asynchronous code blocking.
It sounds like this is a user-interface problem rather than a software-design problem. A common tool for solving such problems is to ask, "What does other, similar software do?". It occurs to me that the problem you're facing is not unlike, oh, job- control in bash or zsh. I'd imagine a usage case that looks something like this:
a = TwistedEnabledObject() a.connect(addr) [1] TwistedEnabledObject.connect(self, addr) [1] done TwistedEnabledObject.connect(self, addr) result1 = a.computeSomethingUsingTwisted(args) [1] TwistedEnabledObject.computeSomethingUsingTwisted(args) result2 = a.computeSomeOtherThingUsingTwisted(args) [2] TwistedEnabledObject.computeSomeOtherThingUsingTwisted(args) print 1+3 4 jobs() [1] running TwistedEnabledObject.computeSomethingUsingTwisted(args) [2] running TwistedEnabledObject.computeSomeOtherThingUsingTwisted(args) print repr(result1) <deferredResult (waiting for result)> [1] done TwistedEnabledObject.computeSomethingUsingTwisted(args) print repr(result1) <deferredResult with value: 42> print result.value 42
I think I'd appreciate an environment like that more than an environment that made me wait for things all the time. I don't know if it's possible to achieve with your PyCrust/Twisted hybrid, of course.
On 3/10/06, Tim Allen <screwtape@froup.com> wrote:
On 11 Mar 2006, at 15:44, Brian Granger wrote:
Here is a typical (madeup) usage case:
a = TwistedEnabledObject() a.connect(addr) # This uses reactor.connectTCP to connect to a server myresult = a.computeSomethingUsingTwisted(args) # This should block!!! ...and maybe... y = f(myresult) plot(y)
The key point is that the user will want to direclty use myresult interactively. Also they won't know ahead of time what actions they will want to do. Anyone who has used Matlab or Mathematica (scientists) will find this way of working extremely familiar. There is simply no way that users will want to get a Deferred() and add callbacks to trigger actions. Thus I would argue that I DO want to make my properly written asynchronous code blocking.
It sounds like this is a user-interface problem rather than a software-design problem. A common tool for solving such problems is to ask, "What does other, similar software do?".
It occurs to me that the problem you're facing is not unlike, oh, job- control in bash or zsh. I'd imagine a usage case that looks something like this:
I agree that it is a user interface question. Other software along these lines, like Mathematica and Matlab are fully blocking. Personally, I think this is too restrictive. For the cases you bring up, where the computeSomethingUsingTwisted() method could take a long time it completely makes sense to have something like a job control interface. In fact, we are building this design pattern into our UI where appropriate. But, in many cases, the computeSomethingUsingTwisted() doesn't take long at all. An example of this would be probing the status of the remote server or a trivial computation that a server has relevant data for. In these cases, the result is available nearly instantly (faster then the user could detect, like 1 ms) and it is silly to have a job interface. You simply want the result directly. In our system, we can easily imagine a user making dozens of interactive calls like this over short periods of time. When all of these results are available essentially immediately, it iseems like overkill to have a more complicated UI. But, it is very appropriate to use Twisted underneath, because it really is an asynchronous system and all the error detection and handling is best dealt with that asynchronously. But as long as there are no errors, I woult like the Twisted machinery to just pass along the result directly.
a = TwistedEnabledObject() a.connect(addr) [1] TwistedEnabledObject.connect(self, addr) [1] done TwistedEnabledObject.connect(self, addr) result1 = a.computeSomethingUsingTwisted(args) [1] TwistedEnabledObject.computeSomethingUsingTwisted(args) result2 = a.computeSomeOtherThingUsingTwisted(args) [2] TwistedEnabledObject.computeSomeOtherThingUsingTwisted(args) print 1+3 4 jobs() [1] running TwistedEnabledObject.computeSomethingUsingTwisted(args) [2] running TwistedEnabledObject.computeSomeOtherThingUsingTwisted(args) print repr(result1) <deferredResult (waiting for result)> [1] done TwistedEnabledObject.computeSomethingUsingTwisted(args) print repr(result1) <deferredResult with value: 42> print result.value 42
I think I'd appreciate an environment like that more than an environment that made me wait for things all the time. I don't know if it's possible to achieve with your PyCrust/Twisted hybrid, of course.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Brian Granger Santa Clara University ellisonbg@gmail.com
At Fri, 10 Mar 2006 22:13:05 -0800, Brian Granger wrote:
In our system, we can easily imagine a user making dozens of interactive calls like this over short periods of time. When all of these results are available essentially immediately, it iseems like overkill to have a more complicated UI. But, it is very appropriate to use Twisted underneath, because it really is an asynchronous system and all the error detection and handling is best dealt with that asynchronously. But as long as there are no errors, I woult like the Twisted machinery to just pass along the result directly.
It sounds like you need to teach your interpreter to deal with deferreds. I'd do something very vaugely like this: def eval_loop(result): print result expression = raw_input(">>> ") d = maybeDeferred(eval(expression)) d.addCallback(lambda r:reactor.callLater(eval_loop, r)) reactor.callWhenRunning(eval_loop, "Welcome to StupidPython") reactor.run() Obviously this is not the correct way to implement an interpreter and I don't know whether I've remembered all the names and signatures correctly, but hopefully it conveys the idea. Mike.
Brian Granger wrote:
But, in many cases, the computeSomethingUsingTwisted() doesn't take long at all. An example of this would be probing the status of the remote server or a trivial computation that a server has relevant data for. In these cases, the result is available nearly instantly (faster then the user could detect, like 1 ms) and it is silly to have a job interface. You simply want the result directly.
Another approach: wait 10ms every time you push out a prompt. The problem is you're trying to generate the prompt in a synchronous manner in an asynchronous environment. 10ms is well below the threshhold that users will notice. You'll still want job control for long calculations, but you can present it in a more natural way.. if it's not available at the first prompt, it's a background job. C
On 3/10/06, Cory Dodt <corydodt@twistedmatrix.com> wrote:
Brian Granger wrote:
But, in many cases, the computeSomethingUsingTwisted() doesn't take long at all. An example of this would be probing the status of the remote server or a trivial computation that a server has relevant data for. In these cases, the result is available nearly instantly (faster then the user could detect, like 1 ms) and it is silly to have a job interface. You simply want the result directly.
Another approach: wait 10ms every time you push out a prompt. The problem is you're trying to generate the prompt in a synchronous manner in an asynchronous environment. 10ms is well below the threshhold that users will notice. You'll still want job control for long calculations, but you can present it in a more natural way.. if it's not available at the first prompt, it's a background job.
I am not super familiar with PyShell/PyCrust, but I think everything (including the prompt generation) is done asynchronously. The only thing that could really block would be long running python code executed in the users namespace in PyShell. But, I do like the idea of having immediate results return directly, but have everything else become a background job. But, the question still remains - how do you convert a Deferred() to a blocking, directly returned result?
C
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Brian Granger Santa Clara University ellisonbg@gmail.com
> I am not super familiar with PyShell/PyCrust, but I think everything > (including the prompt generation) is done asynchronously. The only > thing that could really block would be long running python code > executed in the users namespace in PyShell. But, I do like the idea > of having immediate results return directly, but have everything else > become a background job. But, the question still remains - how do you > convert a Deferred() to a blocking, directly returned result? You could use something like http://svn.twistedmatrix.com/cvs/sandbox/exarkun/threadwrapper.py?view=markup&rev=12804 but I really dont' reccommend it. I think redefining the way the interpreter works is the way ahead. One way to do it would be to think of the space between the last >>> prompt and the current one as "flexible". How about this for an example session: >>> somethingThatReturnsADeferred() this will return a deferred, so natually the first thing you do is attach callbacks and errbacks to it to know when the deferred has finished. You display some indication of this: <Deferred 1 at 0x12345678. Currently: Running> >>> You even make a nice coloured "running" icon to put on one side, if you want. When it stops, you change "Running" to "Completed", in the same line. Assuming nothing has been done yet, you either raise the exception (or, I'd prefer printing out the traceback and putting the Failure in a special variable, called, say _f, it's more flexible) or print out repr(d.result) before putting d.result in _ and the deferred in, say, d. The important thing here is to print out the result in the space between the two >>>, expanding it as necessary. Thus the screen ends up looking like this: >>> somethingThatReturnsADeferred() <Deferred 1 at 0x12345678. Currently: Completed> Result from Deferred 1: ['this', 'is', 'a', 'result'] >>> If someone has already executed a new command, you display *which* deferred the result comes form and skip the _ and d assignemnt step, maybe assiging it to another special variable. Other enhancement would be to add currently pending deferreds to a variable or command you can quickly use to check the status, eg: >>> jobs 1. <Deferred 1 at 0x12345678: Running, Next callback: somefunc(result, arg1, arg2, kw1=val1> 2. <Deferred 2 at 0x12345679: Running, Next callback: somefunc(result, arg1, arg2, kw1=val4> 3. <Deferred 3 at 0x12345680: Running, Next callback: somefunc(result, arg1, arg2, kw1=val3> >>> important_deferred = jobs[1] #so that deferred will not be garbage collected once it falls out of scope You might also want to add a convenience function result() which takes a deferred and does the whole putting of the result in _ and the deferred in d. There are plenty of ways to do it. Trying to make it look blocking is the wrong way to go about it, IMO. Also, I'd be quite interested to see a twisted-compatible PyCrust. Maybe even adding the twisted log to a tab in the bottom pane of the window. I'd be happy to lend a hand in coding such a beast, if you'd like one. It'd be especilly nice if it could do something manhole-compatible, so pycrust could attach to remote twisted processes... Moof
On 3/11/06, Moof <moof@metamoof.net> wrote:
> > I am not super familiar with PyShell/PyCrust, but I think everything
> > (including the prompt generation) is done asynchronously. The only
> > thing that could really block would be long running python code
> > executed in the users namespace in PyShell. But, I do like the idea
> > of having immediate results return directly, but have everything else
> > become a background job. But, the question still remains - how do you
> > convert a Deferred() to a blocking, directly returned result?
>
> You could use something like
> http://svn.twistedmatrix.com/cvs/sandbox/exarkun/threadwrapper.py?view=markup&rev=12804
> but I really dont' reccommend it. I think redefining the way the
> interpreter works is the way ahead.
>
> One way to do it would be to think of the space between the last >>>
> prompt and the current one as "flexible". How about this for an
> example session:
>
> >>> somethingThatReturnsADeferred()
>
> this will return a deferred, so natually the first thing you do is
> attach callbacks and errbacks to it to know when the deferred has
> finished. You display some indication of this:
>
> <Deferred 1 at 0x12345678. Currently: Running>
> >>>
>
> You even make a nice coloured "running" icon to put on one side, if
> you want. When it stops, you change "Running" to "Completed", in the
> same line. Assuming nothing has been done yet, you either raise the
> exception (or, I'd prefer printing out the traceback and putting the
> Failure in a special variable, called, say _f, it's more flexible) or
> print out repr(d.result) before putting d.result in _ and the deferred
> in, say, d. The important thing here is to print out the result in the
> space between the two >>>, expanding it as necessary. Thus the screen
> ends up looking like this:
>
> >>> somethingThatReturnsADeferred()
> <Deferred 1 at 0x12345678. Currently: Completed>
> Result from Deferred 1:
> ['this', 'is', 'a', 'result']
> >>>
>
> If someone has already executed a new command, you display *which*
> deferred the result comes form and skip the _ and d assignemnt step,
> maybe assiging it to another special variable.
>
The problem is that the user will want to do:
>>> result = computeSomething()
OR
>>> result = 2.0*computeOneThing()*math.sin(computeAnother())
What should happen in these cases? Ideally, the _entire_ command
should become a "job" that evaluates in a "smart" manner. That is,
the final result shouldn't be computed until all other results are
available. Meanwhile, the prompt should retrn to the user so they can
continue working. But, I don't think this is possible w/o making
changes to CPython itself. The alternative is to simply make each
command appear to block.
> Other enhancement would be to add currently pending deferreds to a
> variable or command you can quickly use to check the status, eg:
>
> >>> jobs
> 1. <Deferred 1 at 0x12345678: Running, Next callback: somefunc(result,
> arg1, arg2, kw1=val1>
> 2. <Deferred 2 at 0x12345679: Running, Next callback: somefunc(result,
> arg1, arg2, kw1=val4>
> 3. <Deferred 3 at 0x12345680: Running, Next callback: somefunc(result,
> arg1, arg2, kw1=val3>
> >>> important_deferred = jobs[1] #so that deferred will not be garbage
> collected once it falls out of scope
>
> You might also want to add a convenience function result() which takes
> a deferred and does the whole putting of the result in _ and the
> deferred in d.
>
> There are plenty of ways to do it. Trying to make it look blocking is
> the wrong way to go about it, IMO.
>
For long running commands I completely agree. But for commands that
complete quickly (unless they fail in which case an exception should
be raised) I disagree. For example:
>>> a = getPreComputedVariable('a')
Ib this case the onject 'a' on some server needs to be serialized and
brought back to the local machine. There is no reason to mess with
Deferrreds in this case.
> Also, I'd be quite interested to see a twisted-compatible PyCrust.
> Maybe even adding the twisted log to a tab in the bottom pane of the
> window. I'd be happy to lend a hand in coding such a beast, if you'd
> like one.
>
> It'd be especilly nice if it could do something manhole-compatible, so
> pycrust could attach to remote twisted processes...
>
> Moof
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
--
Brian Granger
Santa Clara University
ellisonbg@gmail.com
On Sat, 11 Mar 2006 10:10:38 -0800, Brian Granger <bgranger@scu.edu> wrote:
[snip]
The problem is that the user will want to do:
result = computeSomething()
OR
result = 2.0*computeOneThing()*math.sin(computeAnother())
What should happen in these cases? Ideally, the _entire_ command should become a "job" that evaluates in a "smart" manner. That is, the final result shouldn't be computed until all other results are available. Meanwhile, the prompt should retrn to the user so they can continue working. But, I don't think this is possible w/o making changes to CPython itself. The alternative is to simply make each command appear to block.
The difficulty you are encountering is that you want to use a language which is very much like Python as the input for your command shell, but which is not quite Python. If you look at it this way, it should be clear that simply evaluating the input as Python will never give you a satisfactory solution. One approach you might take, instead, is to parse the input using the compiler module, rewrite the AST so as to account for a Deferred at every step of the evaluation of the expression, and then compile the result into bytecode and execute that. Taking your example: result = 2.0 * computeOneThing() * math.sin(computeAnother()) You would want to rewrite this to: _d1 = defer.maybeDeferred(computeAnother) _d1.addCallback(lambda result: math.sin(result)) _d2 = defer.maybeDeferred(computeOneThing) _d2.addCallback(lambda result: result * 2.0) _d3 = defer.gatherResults([_d1, _d2]) _d3.addCallback(lambda (left, right): left * right) _d3.addCallback(lambda result: assign("result", result)) _d3.addCallback(lambda ign: continueREPL()) As you can see, this is a fairly simple mechanical transformation. In addition to the steps I've taken above, you may also want to serialize the operations, in case allowing computeOneThing() and computeAnother() to run in parallel would be surprising to your users. Jean-Paul
Brian Granger wrote:
I probably should have given more details about what I am trying to do. Before I get going, I should mention that I have been using Twisted heavily for 1.5 years and I have, for the most part, learned to play the "Twisted Game."
If that is the case, you really ought to have known the answer was "you don't" ;o)
So here is what I am working on:
I am developing objects (for scientific computing) that 1) use Twisted to talk over the network and 2) need to be used interactively from a python prompt. This second point is really where the difficult and interesting things are.
That is not an unreasonable use-case in fact (IMHO of course). The sad thing is that PyCrust is wxPython so by definition AWFUL. The many suggestions already are correct - stop the interpreter GUI for a certain time (say 10ms) then if the deferred calls back, restart with the result else return a "backgrounded" job That said, possible someone can comment if this code is current: http://svn.twistedmatrix.com/cvs/trunk/doc/core/examples/threadedselect/bloc... The Manager class getDeferred method might be a start, if you can integrate it with the wx-style tsr.
On 3/11/06, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Brian Granger wrote:
I probably should have given more details about what I am trying to do. Before I get going, I should mention that I have been using Twisted heavily for 1.5 years and I have, for the most part, learned to play the "Twisted Game."
If that is the case, you really ought to have known the answer was "you don't" ;o)
I respectfully disagree that what I am trying to do somehow violates the unwritten Twisted ethic. I am not using threads and I don't have blocking code anywhere. When I write things like:
myresult = a.computeSomethingThatUsesTwisted()
It does not really block - it only appears to block. What I mean by this is that is does not block execution, it only blocks the user from starting new commands. The full machinery of Twisted is free to proceed while the result is pending. I guess I think of it as a shorthand for making the callback of a Deferred an assignment of the result to a global variable myresult. Everything is asynchronous.
So here is what I am working on:
I am developing objects (for scientific computing) that 1) use Twisted to talk over the network and 2) need to be used interactively from a python prompt. This second point is really where the difficult and interesting things are.
That is not an unreasonable use-case in fact (IMHO of course).
The sad thing is that PyCrust is wxPython so by definition AWFUL. The many suggestions already are correct - stop the interpreter GUI for a certain time (say 10ms) then if the deferred calls back, restart with the result else return a "backgrounded" job
That said, possible someone can comment if this code is current:
http://svn.twistedmatrix.com/cvs/trunk/doc/core/examples/threadedselect/bloc...
The Manager class getDeferred method might be a start, if you can integrate it with the wx-style tsr.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Brian Granger Santa Clara University ellisonbg@gmail.com
On Fri, 2006-03-10 at 20:44 -0800, Brian Granger wrote:
To allow Twisted-things to be used interactively I have built a version of PyCrust/PyShell that is "Twisted enabled." We have used the standard threadedselectreactor to interleave the Twisted event loop with that of PyShell. We also inject the reactor into the users interactive namespace. Thus, you can use all of Twisted's capabilities from an interactive python prompt.
I think the new manhole (python interpreter over ssh/telnet) in twisted.conch has some Deferred support you might want to look at.
Regardless of whether or not I should *want* to do this (get the result of a Deferred() in a manner the *appears* synchronous) there is a question that remains: Can it be accomplish using waitForDeferred? The docstring seems to imply that it can, but it is not clear. If so, could someone provide an working example. Just a side note, I seem to recall Guido has even citing this type of usage scenario as one of the motivations for improving generators. Cheers, Brian
On 3/11/06, Brian Granger <bgranger@scu.edu> wrote:
Regardless of whether or not I should *want* to do this (get the result of a Deferred() in a manner the *appears* synchronous) there is a question that remains:
Can it be accomplish using waitForDeferred? The docstring seems to imply that it can, but it is not clear.
If so, could someone provide an working example.
Just a side note, I seem to recall Guido has even citing this type of usage scenario as one of the motivations for improving generators.
waitForDeferred really can't do this for you. I hope that the docstring is clear enough to show that it really requires special syntax every time you want to wait for a Deferred. As far as the improved generators in the upcoming release of Python, they *still* would require you to have special syntax when you want to wait for a Deferred, but it's fewer lines of code. I think what other people have suggested, about just having your UI handle Deferreds specially, is the way to go. Of course, >>> 3 * fetchDeferred() will be a quite difficult thing to do. I'm not sure if it'd be possible at all with Python itself, so you may just end up exposing some minilanguage other than Python to your end users. You *might* be able to do some weird hacks with trace hooks or something to do it in Python, but I'm not certain. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/
participants (11)
-
Brian Granger -
Brian Granger -
Christopher Armstrong -
Cory Dodt -
glyph＠divmod.com -
Itamar Shtull-Trauring -
Jean-Paul Calderone -
Mike Pelletier -
Moof -
Phil Mayers -
Tim Allen