Mailman 3 November 2009 - Twisted

[Twisted-Python] The case of the missing Traceback
by Dan 03 Nov '09

03 Nov '09

I was playing around with Twisted today and ran into a problem where Tracebacks aren't emitted on stderr after a deferred callback from a process called launched by reactor.spawnProcess. The simplified test case is below. I put "x = y" in the callback function to generate an exception, y doesn't exist. Uncomment "x = z" right before the callback and it produces the traceback on stderr just fine. System is Linux, Python 2.6 and Twisted 8.2. from twisted.internet import reactor, protocol, defer import sys def err(s): sys.stderr.write(s+'\n') sys.stderr.flush() def test(): err('test') cat = MyPP() reactor.spawnProcess(cat, "/bin/cat", ["cat", "-"], {}) d = cat.send("Hello world") d.addCallback(response) def response(resp): err('response called') x = y err(resp) class MyPP(protocol.ProcessProtocol): def __init__(self): self.defer = defer.Deferred() def outReceived(self, data): err("outReceived! with %d bytes" % len(data)) err(" %s" % data) self.defer.callback(data) def send(self, c): self.transport.write("%s\n" % c) #x = z return self.defer if(__name__=='__main__'): reactor.callWhenRunning(test) reactor.run()

1 0

[Twisted-Python] reverse web proxy & threads
by Artem Bokhan 03 Nov '09

03 Nov '09

Hello! I'm playing with reverse proxy from examples (Twisted-8.2.0/doc/web/examples/proxy.py). May somebody explain what part of code uses threads there? pstree -p | grep python -python(12786)-+-{python}(12795) |-{python}(13050) |-{python}(15547) `-{python}(20767)

2 1

[Twisted-Python] inlineCallbacks
by Pet 03 Nov '09

03 Nov '09

Hi, calling in client code self.cred.login() I can't print login result. What I'm doing wrong? login method returns deffered, but should yield result from callRemote method. Thanks for any help! Pet class Cred: @inlineCallbacks def login(self): proxy = Proxy(LOGIN_PROXY) l = {"user":LOGIN_USERNAME, "pass":LOGIN_PASSWD} loginResult = yield proxy.callRemote('login', l) print "RESULT", loginResult.result returnValue(loginResult)

7 23

Re: [Twisted-Python] Using a custom reactor in twisted trial for test cases?
by Andrew Francis 02 Nov '09

02 Nov '09

Hello Glyph: >These examples aren't the greatest, because they tend to assume that >you _always_ have Stackless code that's ready to run, and you want to >run at some number of "frames per second". In 2007, folks (especially myself) are still trying to figure out if Stackless can play with Twisted correctly. I know my stuff in the repository is old. > For simple examples this makes sense, but for a system architecture >this is a limiting approach. If your stackless application wants to >sit idle and wait for input, you're still going to wake up once every >1/30 second and checking to see if there's anything to do, burning CPU >cycles and battery life. An approach would be for the reactor tasklet to look at the runnable list to see if there are any entries. If there are no entries (i.e., tasklets are blocked and dependent on the reactor tasklet), then the reactor tasklet can stop the loopingCall. This is easy enough to do. However I suspect one starts to get into custom reactor land. >Also, if you want to run *faster* than the arbitrary timeout you've >selected (1/30 second can be a very long time, especially if you're >doing something pseudo-realtime, like audio playback) you're out of luck. I believe there are two scenarios here: 1) no blocked tasklets and only runnable ones. If the runnable tasklets make no IO requests, it should be possible to put the reactor tasklet to sleep. Let the system run as fast as it can. 2) There is a mix of runnable (scheduled) and blocked tasklets. This scenario requires a more sophisticated scheduler that is trying to compute an optimal loopingCall value. This scenario is tricky. >Better would be to have tasklets schedule *themselves* for when they >want to run. In the Stackless world, tasklets start running from the moment they are created/inserted into the runnable list. Stackless in non-preemptive mode is more about knowing when to make a tasklet yield to the scheduler. Cheers, Andrew

1 0

Re: [Twisted-Python] Using a custom reactor in twisted trial for test cases?
by Andrew Francis 02 Nov '09

02 Nov '09

Crispin: >OK. Having a closer look, its not that looping call doesn't work, its >that there is some unknown number of "reactor pumps" between starting >the test, and finishing it. What I need is a way for the reactor to be >pumping away while a particular test function of a testcase continues >working. As an example, here is a non-working test case. Notice the >comment "#what to do here to pump the reactor?". ... > while task.alive: > pass #what to do here to pump the reactor? This won't work because as long as the tasklet is alive (scheduled, paused, or blocked), you will be in a tight loop. And if that function is not running in the reactor, then the reactor will starve. If you are waiting to see if the tasklet finished, write a synchronizer. Cheers, Andrew

1 0

[Twisted-Python] Using a custom reactor in twisted trial for test cases?
by Crispin Wellington 02 Nov '09

02 Nov '09

Hi there, I am using twisted trial to run test cases for an application. The application however uses stackless python and has a custom stackless reactor. I implemented this reactor like this... -------------------- stacklessreactor.py ----------------------- # Use epoll() as our base reactor from twisted.internet.epollreactor import EPollReactor as StacklessBaseReactor import stackless # seconds between running the greenthreads. 0.0 for flat out 100% CPU STACKLESS_MAX_PUMP_RATE = 0.1 class StacklessReactor(StacklessBaseReactor): """This reactor does the stackless greenthread pumping in the main thread, interwoven with the reactor pump""" def doIteration(self, timeout): """Calls the base reactors doIteration, and then fires off all the stackless threads""" if timeout > STACKLESS_MAX_PUMP_RATE: timeout = STACKLESS_MAX_PUMP_RATE stackless.schedule() return StacklessBaseReactor.doIteration(self,timeout) def install(): """ Install the stackless() reactor. """ p = StacklessReactor() from twisted.internet.main import installReactor installReactor(p) ------------------------------------------------------------------- And I install this as my reactor in my application with... import stacklessreactor stacklessreactor.install() ...placed right at the top of my .tac python file. And this all works. Running the app with twistd, the custom reactor is installed and is used as the reactor for the app. Now however, I come to write tests and run them with trial. I *need* the tests to be run under the stackless reactor or things simply wont work (a lot of the code I need to test are stackless tasklets). When I go "/usr/local/stackless/bin/trial --help-reactors" I get the following list: kqueue kqueue(2)-based reactor. win32 Win32 WaitForMultipleObjects-based reactor. epoll epoll(4)-based reactor. iocp Win32 IO Completion Ports-based reactor. gtk Gtk1 integration reactor. cf CoreFoundation integration reactor. gtk2 Gtk2 integration reactor. default The best reactor for the current platform. debug-gui Semi-functional debugging/introspection reactor. poll poll(2)-based reactor. glib2 GLib2 event-loop integration reactor. select select(2)-based reactor. wx wxPython integration reactor. qt QT integration reactor One of these I can use by passing in --reactor=name. So the question is, is there a way of getting the trial framework to use my custom reactor? Is there a way to get my reactor into that list somehow? Is this not a supported feature of trial? And... if this isn't a supported feature, what is the best way to get a TestCase that will run under that reactor? Look forward to any help people can offer me. With kind regards Crispin Wellington

4 6

Re: [Twisted-Python] Retrying function calls
by Terry Jones 02 Nov '09

02 Nov '09

Hi Tim > Since this is a generator function, it will automatically raise > StopIteration once control-flow falls off the end of the function, so your > while-loop could just be written: Ah yes, thanks a lot. Terry

1 0

[Twisted-Python] Retrying function calls
by Terry Jones 02 Nov '09

02 Nov '09

These days I often find myself writing code to talk to services that are periodically briefly unavailable. An error of some kind occurs and the correct (and documented) action to take is just to retry the original call. Examples include using Amazon's S3 service and the Twitter API. In both of these services, transient failures happen fairly frequently. So I wrote the class below to retry calls, and tried to make it fairly general. I'd be happy to hear comments on it, because it's pretty simple and if it can be made bullet proof I imagine others will use it too. First off, here's the class that handles the calling: from twisted.internet import reactor, defer, task from twisted.python import log, failure class RetryingCall(object): """Calls a function repeatedly, passing it args and kw args. Failures are passed to a user-supplied failure testing function. If the failure is ignored, the function is called again after a delay whose duration is obtained from a user-supplied iterator. The start method (below) returns a deferred that fires with the eventual non-error result of calling the supplied function, or fires its errback if no successful result can be obtained before the delay backoff iterator raises StopIteration. """ def __init__(self, f, *args, **kw): self._f = f self._args = args self._kw = kw def _err(self, fail): try: fail = self._failureTester(fail) except: self._deferred.errback() else: if isinstance(fail, failure.Failure): self._deferred.errback(fail) else: log.msg('RetryingCall: Ignoring %r' % (fail,)) self._call() def _call(self): try: delay = self._backoffIterator.next() except StopIteration: log.msg('StopIteration in RetryingCall: ran out of attempts.') self._deferred.errback() else: d = task.deferLater(reactor, delay, self._f, *self._args, **self._kw) d.addCallbacks(self._deferred.callback, self._err) def start(self, backoffIterator=None, failureTester=None): self._backoffIterator = backoffIterator or simpleBackoffIterator() self._failureTester = failureTester or (lambda f: f) self._deferred = defer.Deferred() self._call() return self._deferred You call the constructor with function and its args. When you call start() you get back a deferred that eventually fires with the result of the call, or an error. BTW, I called it "start" to mirror task.LoopingCall. There's a helper function for producing successive inter-call delays: from operator import mul from functools import partial def simpleBackoffIterator(maxResults=10, maxDelay=5.0, now=True, initDelay=0.01, incFunc=None): assert maxResults > 0 remaining = maxResults delay = initDelay incFunc = incFunc or partial(mul, 2.0) if now: yield 0.0 remaining -= 1 while True: if remaining == 0: raise StopIteration yield (delay if delay < maxDelay else maxDelay) delay = incFunc(delay) remaining -= 1 By default this will generate the sequence of inter-call delays 0.0, 0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56 and it should be easy to see how you could write your own. Or you can just supply a list, etc. When the backoff iterator finishes, the RetryingCall class gives up on trying to get a non-error result from the function. In that case you just get a StopIteration exception in the failure that start() deferred returns (I was originally returning the original failure, but decided to simplify. If you want that, you can keep it yourself in an error tracking class, see below). You get to specify a function for testing failures. If it ever raises or returns a failure, the start() deferred's errback is called. The failure tester can just ignore whatever failures should be considered transient. So, for example, if you were calling S3 and wanted to ignore 504 errors, you could supply a failureTester arg like this: from twisted.web import error, http def test(self, failure): failure.trap(error.Error) if int(failure.value.status) != http.GATEWAY_TIMEOUT: return failure As another example, while using the Twitter API you might want to allow a range of HTTP errors and also exactly one 404 error, seeing as a 404 *might* be an error on the part of Twitter (I don't mean to suggest that actually happens). It's probably definitive - but, why not try it once again just to be more sure? So, pass RetryingCall a failureTester that's an instance of a class like this: class TwitterFailureTester(object): okErrs = (http.INTERNAL_SERVER_ERROR, http.BAD_GATEWAY, http.SERVICE_UNAVAILABLE) def __init__(self): self.seen404 = False def __call__(self, failure): failure.trap(error.Error) status = int(failure.value.status) if status == http.NOT_FOUND: if self.seen404: return failure else: self.seen404 = True elif status not in self.okErrs: return failure Changing existing code to use RetryingCall is pretty trivial. Take something like this from twisted.web import client def getUserByScreenname(screenname): d = client.getPage( 'http://twitter.com/users/show.json?screen_name=glyf') return d and change it to look like this: def getUserByScreenname(screenname): r = RetryingCall(client.getPage, 'http://twitter.com/users/show.json?screen_name=glyf') d = r.start(failureTester=TwitterFailureTester()) return d I wrote the above last night, so I don't know if it's fully robust. But I dropped it into some of my own stuff last night and it seems to work. I also have a small test suite in case anyone wants it. My questions are: Is this robust? Can/should it be improved? Any criticism of the code (especially wrt Twisted best practices) would be very welcome. Thanks! Terry

3 2

Re: [Twisted-Python] Retrying function calls
by Terry Jones 02 Nov '09

02 Nov '09

Hi Blair > Looks pretty good. To streamline the usage and make the intent more > apparent from a brief glance at the header you might consider turning it > into a decorator, like Thanks for the suggestion & the pointer. I'd not seen that page. I don't think this is well suited to decorators, at least not with the kinds of usages I am imagining. If you decorate a function, it's done once and for all. So anyone who calls the function gets the single one-size-fits-all decorated behavior. I'd rather the behavior was left in the hands of the caller. That's kind of the point: give the caller flexible control over what happens if something goes wrong, including passing in your custom failure handler, etc. Maybe a hybrid approach would be more useful: write a function which, passed a function, returns a retrying version of that function that returns a deferred that fires when the original function has succeeded (or ultimately failed). The result could then be passed around, called by multiple pieces of code, etc. Hmmm... Thanks again. Terry

1 0

Re: [Twisted-Python] Retrying function calls
by Terry Jones 01 Nov '09

01 Nov '09

Following up on my own post: > When the backoff iterator finishes, the RetryingCall class gives up on > trying to get a non-error result from the function. In that case you just > get a StopIteration exception in the failure that start() deferred > returns (I was originally returning the original failure, but decided to > simplify. If you want that, you can keep it yourself in an error tracking > class, see below). Thinking about it a bit more it seems more sensible that the start method's default failureTester function ignores all failures, and that in the case of an eventual failure, you get the first ever (or last ever) failure back. Those changes would make the RetryingCall class more useful (as it stands with the defaults you get a failure back the first time something goes wrong, so in the default case the class doesn't even do what it says). I wont post the changes here, as I don't even know if anyone else is interested. Contact me if you want an updated version. Terry

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 01 Nov '09

01 Nov '09

1 0