Mailman 3 November 2008 - Twisted

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones 22 Nov '08

22 Nov '08

Hi Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: >> inlineCallbacks appears to have a bug: 'raise' before 'yield' in a >> generator results in a synchronous exception rather than an errback, >> although its documentation does not explain this. glyph> Ugh, scratch that. No, it doesn't have this bug. Yes, you're right. glyph> I did some quick testing and saw some tracebacks, but apparently glyph> wasn't paying very close attention to them Nor was I, sorry! glyph> If func "raises before it gets to its first yield", we get the right glyph> behavior. If it just raises and doesn't yield *anywhere*, then it's glyph> not a generator and that's the same as your other case of glyph> accidentally-not- returning-a-generator. Still worth debugging, but glyph> not as serious. Agreed. I also agree with your earlier remarks about dropping the isinstance(result, GeneratorType). That leaves me with an alternative: def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): try: result = f(*args, **kwargs) except Exception, e: # f was not a generator. return failure.Failure() return _inlineCallbacks(None, result, Deferred()) return mergeFunctionMetadata(f, unwindGenerator) and still the problem that _inlineCallbacks raises if result doesn't have a send() method, etc. I'm also not sure of the best way to check for that, but don't think it should be in the _inlineCallbacks loop. --- BTW, there is a coding lesson here (at least for me): The reason I started to think there was a problem with Exceptions, tracebacks and _inlineCallbacks was from running code like this: @inlineCallbacks def f(): try: # something except Exception: # clean up raise When I ran it, the traceback of the Exception that propagated back to the attached errback would (sometimes) show the exception as having being triggered in _inlineCallbacks, like this: 2008/11/22 17:29 +0200 [-] File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 726, in _inlineCallbacks 2008/11/22 17:29 +0200 [-] result = g.send(result) The problem, as I guess will be obvious, is that my raise was delivering whatever sys.exc_info returned after the clean-up was done. In my case this was confusing as although the clean-up code had succeeded, it had called things that also made use of inlineCallbacks and the exception I was finally raising, and whose traceback I was printing, was a StopIteration raised inside *another* call of _inlineCallbacks! So it looked like _inlineCallbacks was somehow raising instead of sending a failure back to the errback... Re-raising the original exception made everything make sense. Thanks a lot for taking a look. Terry

2 1

[Twisted-Python] [JOB] Sr. Backend Developer, Eugene or Portland | 60-90k
by Beau Gould 22 Nov '08

22 Nov '08

Responsibilities: * Work in a team building scalable web applications * Use test-driven development practices to ensure code correctness * Document code, track and fix bugs, respond rapidly to emergencies in production environment * Work with product group throughout development cycle to ensure quality of end product * Networking protocols such as TCP, UDP, HTTP, DNS, NAT, etc * Clustering and load balancing solutions for redundancy, high availability, scalability * Network programming (event-driven, asynchronous) * Experience analyzing large and complex systems for performance bottlenecks * Python, AJAX, PHP, C/C++ * LAMP * Scalable MySQL schema designs * Jabber/XMPP * Twisted Python * Spread Toolkit Requirements for this position: * Bachelors in Computer Science or related work experience. Masters degree is a plus * Ability to work on-site in Eugene, Oregon (preferred) or Portland, Oregon * 5+ years experience developing large scale software systems (10K+ concurrent users) To be considered, please submit your resume along with your salary requirements to bg @ capitalmarketsp.com Beau Gould Executive Advisor Capital Markets Placement www.cmp.jobs bg @ capitalmarketsp.com

1 0

[Twisted-Python] Libevent bindings and reactor
by Thomas Hervé 22 Nov '08

22 Nov '08

Hi all! I've been working on a Libevent reactor for quite a long time now: http://twistedmatrix.com/trac/ticket/1930 (sheesh, 2 years). For different reasons I didn't manage (yet) to include something within Twisted itself, so I (re)started an external project to do it. For those who don't know libevent, it's an abstraction layer around select/poll/epoll/... to write portable applications. See http://www.monkey.org/~provos/libevent/. I thought it'll be easier for people who just want to use the libevent bindings to have an external project, and it's also a way to make people test it and get some feedback, without having to checkout a SVN branch. The project is now accessible on launchpad: https://launchpad.net/python-libevent (original project is recorded there: http://pypi.python.org/pypi/libevent-python). The latest version, 0.3, includes a libevent reactor, and a twisted plugins so that it's easily usable with trial and twistd (just pass -r libevent to both once installed). Note that these bindings only work with a recent enough version of Libevent itself (1.4). My PPA (https://launchpad.net/~therve/+archive) should help users of Hardy (although for some reasons the twisted plugin doesn't install correctly). Thanks for any comment/feedback. -- Thomas

1 0

[Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones 21 Nov '08

21 Nov '08

Here's a suggestion for making inlineCallbacks more consistent and less confusing. Let's suppose you're writing something like this: @inlineCallbacks def func(): .... result = func() There are 2 things that could be better, IMO: 1. func may not yield. In that case, you either get an AttributeError when inlineCallbacks tries to send(). Or worse, the call to send might actually work, and do who knows what. I.e., func() could return an object with a send method but which is not a generator. For some fun, run some code that calls the following decorated function (see if you can figure out what will happen before you do): @defer.inlineCallbacks def f(): class yes(): def send(x, y): print 'yes' # accidentally_destroy_the_universe_too() return yes() 2. func might raise before it get to its first yield. In that case you'll get an exception thrown when the inlineCallbacks decorator tries to create the wrapper function: File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 813, in unwindGenerator return _inlineCallbacks(None, f(*args, **kwargs), Deferred()) There's a simple and consistent way to handle both of these. Just have inlineCallbacks do some initial work based on what it has been passed: def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): deferred = defer.Deferred() try: result = f(*args, **kwargs) except Exception, e: deferred.errback(e) return deferred if isinstance(result, types.GeneratorType): return defer._inlineCallbacks(None, result, deferred) deferred.callback(result) return deferred return mergeFunctionMetadata(f, unwindGenerator) This has the advantage that (barring e.g., a KeyboardInterrupt in the middle of things) you'll *always* get a deferred back when you call an inlineCallbacks decorated function. That deferred might have already called or erred back (corresponding to cases 1 and 2 above). I'm going to use this version of inlineCallbacks in my code. There's a case for it making it into Twisted itself: inlinecallbacks is already cryptic enough in its operation that anything we can do to make its operation more uniform and less surprising, the better. You might think that case 1 rarely comes up. But I've hit it a few times, usually when commenting out sections of code for testing. If you accidentally comment out the last yield in func, it no longer returns a generator and that causes a different error. And case 2 happens to me too. Having inlinecallbacks try/except the call to func is nicer because it means I don't have to be quite so defensive in coding. So instead of me having to write try: d = func() except Exception: # ??? and try to figure out what happened if an exception fired, I can just write d = func() and add errbacks as I please (they then have to figure out what happened). The (slight?) disadvantage to my suggestion is that with the above try/except fragment you can tell if the call to func() raised before ever yielding. You can detect that, if you need to, with my approach if you're not offended by looking at d.called immediately after calling func. The alternate approach also helps if you're a novice, or simply being lazy/careless/forgetful, and writing: d = func() d.addCallback(ok) d.addErrback(not_ok) thinking you have your ass covered, but you actually don't (due to case 2). There's some test code attached that illustrates all this. Terry import types from twisted.internet import defer, reactor from twisted.python.util import mergeFunctionMetadata def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): deferred = defer.Deferred() try: result = f(*args, **kwargs) except Exception, e: deferred.errback(e) return deferred if isinstance(result, types.GeneratorType): return defer._inlineCallbacks(None, result, deferred) deferred.callback(result) return deferred return mergeFunctionMetadata(f, unwindGenerator) @altInlineCallbacks def f0(): # This causes mayhem with the normal inlineCallbacks class yes(): def send(x, y): print 'yes' return yes() @altInlineCallbacks def f1(): return 'hey' @altInlineCallbacks def f2(): raise Exception('whoops!') @altInlineCallbacks def f3(): if True: raise Exception('ouch!') d = defer.Deferred() reactor.callLater(0, d.callback, None) yield d @altInlineCallbacks def f4(): d = defer.Deferred() reactor.callLater(0, d.callback, None) yield d def ok(result, name): print '%s ok: %s' % (name, result) def fail(failure, name): print '%s fail: %s' % (name, failure) def stop(_): reactor.stop() def main(): L = [] for func in f0, f1, f2, f3, f4: name = func.__name__ L.append(func().addCallbacks( ok, fail, callbackArgs=[name], errbackArgs=[name])) d = defer.DeferredList(L) d.addBoth(stop) if __name__ == '__main__': reactor.callLater(0, main) reactor.run()

3 3

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones 21 Nov '08

21 Nov '08

Here's another way of looking at what I proposed. When you call a function, you can divide what happens into 3 possibilities: 1) An exception is raised 2) You get back a generator 3) You get back something else (Yes, I'm ignoring aborts, sys.exit, etc). As it stands, all three of these are handled inside _inlineCallbacks (not in inlineCallbacks). With the code I suggested, the call to f is made before _inlineCallbacks is called. Only in case 2, where we actually have a generator, do we ask _inlineCallbacks to deal with unwinding it. In case 1 we pass the exception back via a deferred errback with a failure and in case 2 we pass the result back in a deferred whose callback is already called. There's a good case (perhaps a stronger one) for just letting the Exception go on its merry way and raise back in the calling code. But it's hard to argue that knowing that we don't even have a generator result that passing it on to _inlineCallbacks in the knowledge that it's going to cryptically raise is better than just returning the result of the user's function, especially seeing as it has just successfully run to completion! Terry

1 0

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones 21 Nov '08

21 Nov '08

>>>>> "Drew" == Drew Smathers <drew.smathers(a)gmail.com> writes: Drew> On Fri, Nov 21, 2008 at 1:04 PM, Terry Jones <terry(a)jon.es> wrote: Drew> Why not just return a Deferred from the function and not decorate it. Drew> Or document the function as returning a value if it doesn't block. In that case (1) I'm considering what happens if you take the last yield out of a function decorated with inlineCallbacks. You might do it via commenting something out for testing, or you might do it just as a matter of course in changing the behavior of the function. If you do either of those, inlineCallbacks will raise an exception. With the suggestion I sent, it will just run as expected: the original function will be called, and you'll get the value it returns back as the result of the returned deferred. Drew> Essentially this is equivalent to _not_ decorating your function and Drew> returning a Deferred via defer.succeed(value) or defer.fail(error). Drew> I'm still not sure why it's necessary to stitch this kind of behavior Drew> into inlineCallbacks(). It's not necessary, it just makes inlineCallbacks gracefully cover two cases that it currently throws exceptions on, and it makes inlineCallbacks always return a deferred, which is arguably also an advantage. Drew> From my perspective inlineCallbacks() simply equates: "I want to do Drew> more than one asynchronous operation inline and here's some syntactic Drew> sugar using yield as an expression." Right. I'm just making what's happening underneath a little more forgiving and consistent in these two edge cases where things go awry. The main point, to me, is that code changes over time. So you write a function and use inlineCallbacks decorator for syntactic sugar. Then you, or someone else, comes along a while later and change a few things. If you take out the last yield without noticing, or your change throws an exception before the first yield, then you've broken things and it may not be clear why. In the first case, you get an exception thrown from deep inside inlineCallbacks, and many/most people would have some trouble figuring out what's going on. There's no need for that though. >> This has the advantage that (barring e.g., a KeyboardInterrupt in the >> middle of things) you'll *always* get a deferred back when you call an >> inlineCallbacks decorated function. That deferred might have already >> called or erred back (corresponding to cases 1 and 2 above). Drew> You will always get a Deferred back if the function is a generator. But if you take out the last deferred, it's no longer a generator. And if an exception happens before the first yield you don't get a deferred back because the exception is raised before the inlineCallbacks created function gets its first value from your generator. They're just two simple scenarios that can be dealt with more cleanly. If it wasn't clear, the inlineCallbacks created function is identical. Your code is still run identically. There's just a little pre-processing to see if it even needs to be run. Drew> That's what unit tests are for :) But I'm not sure how you could Drew> easily wind up in this scenario considering generators don't allow Drew> return with an argument. Right. But a function can just fall off the end, implicitly returning None. And I gave two examples, plus simple code, that illustrates how both of the problems I'm addressing can arise. It's not about having return in a function. One part is about turning a generator into a non-generator by removing the final yield. The other's just about introducing an exception into your code. I've run into both situations, more than once, and in both had the reaction that inlineCallbacks could have been more accomodating or helpful in how it behaves. Fortunately I can just use my own version! :-) Sorry if I wasn't being clear enough first time round. Regards, Terry

1 0

[Twisted-Python] Connectionless AMP?
by Peter Sabaini 21 Nov '08

21 Nov '08

Hello, I want to implement a messageing protocol running on top of TIPC / SOCK_RDM to be able to do multicasting; ie. send messages in a connectionless manner. I implemented some TIPC support, but I now wonder how to go about the protocol. Originally, I thought about using AMP, but this seems to assume a connection-oriented transport. How would you go about this? Should it be possible to use a datagram-like, connection-less transport with AMP? Or should I rather take a more low-level approach, eg. by using IntNStringReceiver and roll my own? Thanks for any advice. peter.

4 8

[Twisted-Python] Connection Manager
by Reza Lotun 20 Nov '08

20 Nov '08

Hi All, I've been thinking of implementing a connection manager in twisted. The app I'm working on has various parts or "modules" that individually make separate connections to various services and what not. They basically act independently of each other - as in they're all scheduled connection calls and what not. I'm interested in mediating these connections through a "Dispatcher" of sorts - what I'm thinking would be a thin wrapper over the reactor, with extra state. One of the things the Dispatcher would do is gauge network connection health - something like periodically attempting to ping/connect to well known internet sites. If the connection ever degrades or fails, the Dispatcher would change state. Fairly straightforward, conceptually at least. My questions are twofold - is the general idea sound - how is such a thing implemented elsewhere? Also, what would be the best strategy to implement the above, if the idea is sane? I was thinking something like this: my Dispatcher would have something like a connectTCP call which would in turn call the reactor.connectTCP with a special DispatcherFactory. As the returned IConnector calls startedConnecting on the DispatcherFactory, it would check internal state, figure out a policy and decide whether to continue with the connection and call the passed in factory's buildProctol, or call the IConnector stopConnecting to pause the connection attempt. If the situation improves later on, the IConnector.connect can be called to "resume" the connection attempt. Any feedback would be greatly appreciated. Thanks, Reza -- Reza Lotun Senior Software Engineer GetPeer Limited reza(a)getpeer.com

1 0

[Twisted-Python] Exception handling in t.e.adbapi
by Don Dwiggins 20 Nov '08

20 Nov '08

I've been using adbapi (with pyodbc talking to MS SQL Server) successfully in a Windows service. I've discovered that occasionally, the network between the service and the database machine goes down and comes back up (either that or the database server itself cycles down and back up). When this happens, any pyodbc connections hanging around are corrupted, so that the next query gets an exception (class pyodbc.Error). Currently, the best thing I can do is to restart the service. I'm trying to rewrite the service to be able to catch, analyze, and respond to exceptions from pyodbc. In particular, in the case above, it's possible to reconnect and retry the query. If the network connection is still down, I'd like to log it, send a message to alert someone, or whatever. I've rewritten a ConnectionPool.runQuery call to use runInteraction to call a function (in a thread) that works with the cursor created in runInteraction to execute the query. The function wraps the cursor.execute(...) in a try-except. I can successfully catch the exception, and I try to recover by doing cursor.reconnect(), then cursor.reopen(), and finally cursor.execute(...) again. I'm finding that this fails with a "wrong connection for thread" exception. I haven't been able to figure out why that should happen; more importantly, I'm not sure that I'm going about this in the right way. Any insights or pointers to code that does this kind of thing would be appreciated. Thanks, -- Don Dwiggins Advanced Publishing Technology

2 2

[Twisted-Python] trial testing messages
by Gabriel Rossetti 19 Nov '08

19 Nov '08

Hello, I am writing unit tests using trial. I am trying to make sure that the received messages are processed correctly and the output messages are correct. I tried using toXml(), but the XML attributes aren't necessaraly in the correct order, so a string comparison doesn't work. I had originally submitted a patch to allow domish.Element to be compared (ticket #3344), but my patch isn't liked. So my question is, how may I test this? Thank you, Gabriel

2 2