Mailman 3 November 2008 - Twisted

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 23, 2008

Nov. 23, 2008

Hi Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: >> inlineCallbacks appears to have a bug: 'raise' before 'yield' in a >> generator results in a synchronous exception rather than an errback, >> although its documentation does not explain this. glyph> Ugh, scratch that. No, it doesn't have this bug. Yes, you're right. glyph> I did some quick testing and saw some tracebacks, but apparently glyph> wasn't paying very close attention … [View More]to them Nor was I, sorry! glyph> If func "raises before it gets to its first yield", we get the right glyph> behavior. If it just raises and doesn't yield *anywhere*, then it's glyph> not a generator and that's the same as your other case of glyph> accidentally-not- returning-a-generator. Still worth debugging, but glyph> not as serious. Agreed. I also agree with your earlier remarks about dropping the isinstance(result, GeneratorType). That leaves me with an alternative: def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): try: result = f(*args, **kwargs) except Exception, e: # f was not a generator. return failure.Failure() return _inlineCallbacks(None, result, Deferred()) return mergeFunctionMetadata(f, unwindGenerator) and still the problem that _inlineCallbacks raises if result doesn't have a send() method, etc. I'm also not sure of the best way to check for that, but don't think it should be in the _inlineCallbacks loop. --- BTW, there is a coding lesson here (at least for me): The reason I started to think there was a problem with Exceptions, tracebacks and _inlineCallbacks was from running code like this: @inlineCallbacks def f(): try: # something except Exception: # clean up raise When I ran it, the traceback of the Exception that propagated back to the attached errback would (sometimes) show the exception as having being triggered in _inlineCallbacks, like this: 2008/11/22 17:29 +0200 [-] File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 726, in _inlineCallbacks 2008/11/22 17:29 +0200 [-] result = g.send(result) The problem, as I guess will be obvious, is that my raise was delivering whatever sys.exc_info returned after the clean-up was done. In my case this was confusing as although the clean-up code had succeeded, it had called things that also made use of inlineCallbacks and the exception I was finally raising, and whose traceback I was printing, was a StopIteration raised inside *another* call of _inlineCallbacks! So it looked like _inlineCallbacks was somehow raising instead of sending a failure back to the errback... Re-raising the original exception made everything make sense. Thanks a lot for taking a look. Terry [View Less]

2 1

[Twisted-Python] [JOB] Sr. Backend Developer, Eugene or Portland | 60-90k
by Beau Gould Nov. 22, 2008

Nov. 22, 2008

Responsibilities: * Work in a team building scalable web applications * Use test-driven development practices to ensure code correctness * Document code, track and fix bugs, respond rapidly to emergencies in production environment * Work with product group throughout development cycle to ensure quality of end product * Networking protocols such as TCP, UDP, HTTP, DNS, NAT, etc * Clustering and load balancing solutions for redundancy, high availability, scalability * Network programming (… [View More]

1 0

[Twisted-Python] Libevent bindings and reactor
by Thomas Hervé Nov. 22, 2008

Nov. 22, 2008

Hi all! I've been working on a Libevent reactor for quite a long time now: http://twistedmatrix.com/trac/ticket/1930 (sheesh, 2 years). For different reasons I didn't manage (yet) to include something within Twisted itself, so I (re)started an external project to do it. For those who don't know libevent, it's an abstraction layer around select/poll/epoll/... to write portable applications. See http://www.monkey.org/~provos/libevent/. I thought it'll be easier for people who just want to use … [View More]

1 0

[Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 22, 2008

Nov. 22, 2008

Here's a suggestion for making inlineCallbacks more consistent and less confusing. Let's suppose you're writing something like this: @inlineCallbacks def func(): .... result = func() There are 2 things that could be better, IMO: 1. func may not yield. In that case, you either get an AttributeError when inlineCallbacks tries to send(). Or worse, the call to send might actually work, and do who knows what. I.e., func() could return an object with a send method but which … [View More]is not a generator. For some fun, run some code that calls the following decorated function (see if you can figure out what will happen before you do): @defer.inlineCallbacks def f(): class yes(): def send(x, y): print 'yes' # accidentally_destroy_the_universe_too() return yes() 2. func might raise before it get to its first yield. In that case you'll get an exception thrown when the inlineCallbacks decorator tries to create the wrapper function: File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 813, in unwindGenerator return _inlineCallbacks(None, f(*args, **kwargs), Deferred()) There's a simple and consistent way to handle both of these. Just have inlineCallbacks do some initial work based on what it has been passed: def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): deferred = defer.Deferred() try: result = f(*args, **kwargs) except Exception, e: deferred.errback(e) return deferred if isinstance(result, types.GeneratorType): return defer._inlineCallbacks(None, result, deferred) deferred.callback(result) return deferred return mergeFunctionMetadata(f, unwindGenerator) This has the advantage that (barring e.g., a KeyboardInterrupt in the middle of things) you'll *always* get a deferred back when you call an inlineCallbacks decorated function. That deferred might have already called or erred back (corresponding to cases 1 and 2 above). I'm going to use this version of inlineCallbacks in my code. There's a case for it making it into Twisted itself: inlinecallbacks is already cryptic enough in its operation that anything we can do to make its operation more uniform and less surprising, the better. You might think that case 1 rarely comes up. But I've hit it a few times, usually when commenting out sections of code for testing. If you accidentally comment out the last yield in func, it no longer returns a generator and that causes a different error. And case 2 happens to me too. Having inlinecallbacks try/except the call to func is nicer because it means I don't have to be quite so defensive in coding. So instead of me having to write try: d = func() except Exception: # ??? and try to figure out what happened if an exception fired, I can just write d = func() and add errbacks as I please (they then have to figure out what happened). The (slight?) disadvantage to my suggestion is that with the above try/except fragment you can tell if the call to func() raised before ever yielding. You can detect that, if you need to, with my approach if you're not offended by looking at d.called immediately after calling func. The alternate approach also helps if you're a novice, or simply being lazy/careless/forgetful, and writing: d = func() d.addCallback(ok) d.addErrback(not_ok) thinking you have your ass covered, but you actually don't (due to case 2). There's some test code attached that illustrates all this. Terry import types from twisted.internet import defer, reactor from twisted.python.util import mergeFunctionMetadata def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): deferred = defer.Deferred() try: result = f(*args, **kwargs) except Exception, e: deferred.errback(e) return deferred if isinstance(result, types.GeneratorType): return defer._inlineCallbacks(None, result, deferred) deferred.callback(result) return deferred return mergeFunctionMetadata(f, unwindGenerator) @altInlineCallbacks def f0(): # This causes mayhem with the normal inlineCallbacks class yes(): def send(x, y): print 'yes' return yes() @altInlineCallbacks def f1(): return 'hey' @altInlineCallbacks def f2(): raise Exception('whoops!') @altInlineCallbacks def f3(): if True: raise Exception('ouch!') d = defer.Deferred() reactor.callLater(0, d.callback, None) yield d @altInlineCallbacks def f4(): d = defer.Deferred() reactor.callLater(0, d.callback, None) yield d def ok(result, name): print '%s ok: %s' % (name, result) def fail(failure, name): print '%s fail: %s' % (name, failure) def stop(_): reactor.stop() def main(): L = [] for func in f0, f1, f2, f3, f4: name = func.__name__ L.append(func().addCallbacks( ok, fail, callbackArgs=[name], errbackArgs=[name])) d = defer.DeferredList(L) d.addBoth(stop) if __name__ == '__main__': reactor.callLater(0, main) reactor.run() [View Less]

3 3

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 21, 2008

Nov. 21, 2008

Here's another way of looking at what I proposed. When you call a function, you can divide what happens into 3 possibilities: 1) An exception is raised 2) You get back a generator 3) You get back something else (Yes, I'm ignoring aborts, sys.exit, etc). As it stands, all three of these are handled inside _inlineCallbacks (not in inlineCallbacks). With the code I suggested, the call to f is made before _inlineCallbacks is called. Only in case 2, where we actually have a generator, do we … [View More]

1 0

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 21, 2008

Nov. 21, 2008

>>>>> "Drew" == Drew Smathers <drew.smathers(a)gmail.com> writes: Drew> On Fri, Nov 21, 2008 at 1:04 PM, Terry Jones <terry(a)jon.es> wrote: Drew> Why not just return a Deferred from the function and not decorate it. Drew> Or document the function as returning a value if it doesn't block. In that case (1) I'm considering what happens if you take the last yield out of a function decorated with inlineCallbacks. You might do it via commenting something out for … [View More]testing, or you might do it just as a matter of course in changing the behavior of the function. If you do either of those, inlineCallbacks will raise an exception. With the suggestion I sent, it will just run as expected: the original function will be called, and you'll get the value it returns back as the result of the returned deferred. Drew> Essentially this is equivalent to _not_ decorating your function and Drew> returning a Deferred via defer.succeed(value) or defer.fail(error). Drew> I'm still not sure why it's necessary to stitch this kind of behavior Drew> into inlineCallbacks(). It's not necessary, it just makes inlineCallbacks gracefully cover two cases that it currently throws exceptions on, and it makes inlineCallbacks always return a deferred, which is arguably also an advantage. Drew> From my perspective inlineCallbacks() simply equates: "I want to do Drew> more than one asynchronous operation inline and here's some syntactic Drew> sugar using yield as an expression." Right. I'm just making what's happening underneath a little more forgiving and consistent in these two edge cases where things go awry. The main point, to me, is that code changes over time. So you write a function and use inlineCallbacks decorator for syntactic sugar. Then you, or someone else, comes along a while later and change a few things. If you take out the last yield without noticing, or your change throws an exception before the first yield, then you've broken things and it may not be clear why. In the first case, you get an exception thrown from deep inside inlineCallbacks, and many/most people would have some trouble figuring out what's going on. There's no need for that though. >> This has the advantage that (barring e.g., a KeyboardInterrupt in the >> middle of things) you'll *always* get a deferred back when you call an >> inlineCallbacks decorated function. That deferred might have already >> called or erred back (corresponding to cases 1 and 2 above). Drew> You will always get a Deferred back if the function is a generator. But if you take out the last deferred, it's no longer a generator. And if an exception happens before the first yield you don't get a deferred back because the exception is raised before the inlineCallbacks created function gets its first value from your generator. They're just two simple scenarios that can be dealt with more cleanly. If it wasn't clear, the inlineCallbacks created function is identical. Your code is still run identically. There's just a little pre-processing to see if it even needs to be run. Drew> That's what unit tests are for :) But I'm not sure how you could Drew> easily wind up in this scenario considering generators don't allow Drew> return with an argument. Right. But a function can just fall off the end, implicitly returning None. And I gave two examples, plus simple code, that illustrates how both of the problems I'm addressing can arise. It's not about having return in a function. One part is about turning a generator into a non-generator by removing the final yield. The other's just about introducing an exception into your code. I've run into both situations, more than once, and in both had the reaction that inlineCallbacks could have been more accomodating or helpful in how it behaves. Fortunately I can just use my own version! :-) Sorry if I wasn't being clear enough first time round. Regards, Terry [View Less]

1 0

[Twisted-Python] Connectionless AMP?
by Peter Sabaini Nov. 21, 2008

Nov. 21, 2008

Hello, I want to implement a messageing protocol running on top of TIPC / SOCK_RDM to be able to do multicasting; ie. send messages in a connectionless manner. I implemented some TIPC support, but I now wonder how to go about the protocol. Originally, I thought about using AMP, but this seems to assume a connection-oriented transport. How would you go about this? Should it be possible to use a datagram-like, connection-less transport with AMP? Or should I rather take a more low-level … [View More]

4 8

[Twisted-Python] Connection Manager
by Reza Lotun Nov. 20, 2008

Nov. 20, 2008

Hi All, I've been thinking of implementing a connection manager in twisted. The app I'm working on has various parts or "modules" that individually make separate connections to various services and what not. They basically act independently of each other - as in they're all scheduled connection calls and what not. I'm interested in mediating these connections through a "Dispatcher" of sorts - what I'm thinking would be a thin wrapper over the reactor, with extra state. One of the things the … [View More]

1 0

[Twisted-Python] Exception handling in t.e.adbapi
by Don Dwiggins Nov. 20, 2008

Nov. 20, 2008

I've been using adbapi (with pyodbc talking to MS SQL Server) successfully in a Windows service. I've discovered that occasionally, the network between the service and the database machine goes down and comes back up (either that or the database server itself cycles down and back up). When this happens, any pyodbc connections hanging around are corrupted, so that the next query gets an exception (class pyodbc.Error). Currently, the best thing I can do is to restart the service. I'm … [View More]

2 2

[Twisted-Python] trial testing messages
by Gabriel Rossetti Nov. 20, 2008

Nov. 20, 2008

Hello, I am writing unit tests using trial. I am trying to make sure that the received messages are processed correctly and the output messages are correct. I tried using toXml(), but the XML attributes aren't necessaraly in the correct order, so a string comparison doesn't work. I had originally submitted a patch to allow domish.Element to be compared (ticket #3344), but my patch isn't liked. So my question is, how may I test this? Thank you, Gabriel

2 2