Mailman 3 April 2009 - Twisted

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com April 26, 2009

April 26, 2009

1 0

Re: [Twisted-Python] Transparent pooling of deferreds to be fired upon another deferred firing
by Terry Jones April 23, 2009

April 23, 2009

BTW, below is a version of my code from yesterday that works on class methods. I didn't write the __get__ code, but took it from a decorator tutorial on the web. This version works with standalone functions, and with class methods that also use @inlineCallbacks. So you can do class MyClass(object): @DeferredPooler @defer.inlineCallbacks def xxx(self, ...): d = funcReturningADeferred() d.addCallbacks(...) yield d and it works. I also wrote a small test suite in case anyone wants it. There are still some issues (e.g., with timeouts), but I've decorated some methods with DeferredPooler and am happy with the result so far. Terry from twisted.internet import defer from twisted.python import failure class DeferredPooler(object): def __init__(self, func): self.pool = {} self.func = func def _callOthers(self, result, key): if isinstance(result, failure.Failure): for d in self.pool[key]: d.errback(result) else: for d in self.pool[key]: d.callback(result) del self.pool[key] return result def __call__(self, *args, **kwargs): key = (args, hash(tuple(sorted(kwargs.items())))) if key in self.pool: d = defer.Deferred() self.pool[key].append(d) return d else: self.pool[key] = [] d = self.func(*args, **kwargs) assert isinstance(d, defer.Deferred) d.addBoth(self._callOthers, key) return d def __get__(self, obj, type=None): if obj is None: return self new_func = self.func.__get__(obj, type) return self.__class__(new_func)

1 0

Re: [Twisted-Python] Re: Transparent pooling of deferreds to be fired upon another
by Terry Jones April 23, 2009

April 23, 2009

Hi Andrew >>>>> "Andrew" == Andrew Francis <andrewfr_ice(a)yahoo.com> writes: Andrew> Why can't you have the callbacks attached to the deferred, share a flag? Andrew> global flag Andrew> def callback(....): Andrew> global flag Andrew> if flag == False: Andrew> flag = True Andrew> # do body of function Andrew> else: Andrew> # don't do anything - flag have been previously fired Andrew> If I understand what is happening, this should work because of the Andrew> non-preemptive nature of the reactor, that is you ought not get a Andrew> race condition. I'm not sure if I fully understand you. 1. If your callback above is the thing that does the work (i.e., my alertSupervisor function), then yes, you could use a flag to make sure the work is only done one. 2. If your callback really is a callback :-) then this doesn't seem to address what I was after. It would stop the same piece of code in the same file from doing something twice. In the case of 1, I don't get what I want because I want various pieces of code to be able to call the deferred-returning method and each of them to get a valid deferred back, which fires as usual. I also don't get to call the function and then call it again later, because the flag is one-shot. I'm trying to deal with identical calls that happen more-or-less at the same time. In the case of 2, this is also not what I was after. I want, for example, code running in a different process or on a different box to be able to make the call (via RPC), get a deferred, etc. So a client-side solution isn't going to work. Or maybe you meant something else? Sorry - I probably wasn't clear enough in my original post, and thanks for the suggestion. Terry

1 0

[Twisted-Python] Re: Transparent pooling of deferreds to be fired upon another
by Andrew Francis April 23, 2009

April 23, 2009

Hi Terry: --- On Thu, 4/23/09, twisted-python-request(a)twistedmatrix.com > Suppose I have a function called alertSupervisor that takes > a string message for the supervisor and which returns a deferred. I > want to be able to call that function from anywhere in my code, and I > always want to get a deferred back, but if there's already a call in >progress to tell the supervisor the same thing then I don't want to bug >him/her by sending another message, but I do want to know when the >original >call fires. Why can't you have the callbacks attached to the deferred, share a flag? global flag def callback(....): global flag if flag == False: flag = True # do body of function else: # don't do anything - flag have been previously fired If I understand what is happening, this should work because of the non-preemptive nature of the reactor, that is you ought not get a race condition. Cheers, Andrew

1 0

[Twisted-Python] Transparent pooling of deferreds to be fired upon another deferred firing
by Terry Jones April 23, 2009

April 23, 2009

I have a need to make sure that a deferred-returning function is not called multiple times, but I want to be able to write code without having to worry about that. To be more concrete: Suppose I have a function called alertSupervisor that takes a string message for the supervisor and which returns a deferred. I want to be able to call that function from anywhere in my code, and I always want to get a deferred back, but if there's already a call in progress to tell the supervisor the same thing then I don't want to bug him/her by sending another message, but I do want to know when the original call fires. So code fragment A calls d = alertSupervisor("We have a problem in the boiler room!") and before the deferred has fired, code fragment B notices the same underlying problem, and also calls d = alertSupervisor("We have a problem in the boiler room!") I want A and B's deferreds to both be called back with the result/failure. If I can write it like the above it has the advantage of looking like regular code and no-one has to think about what's going on. Underneath there's a pool of deferreds whose firing is triggered by the firing of the single original deferred. Note that I specifically only want to pool deferreds like this if there is a call already underway (i.e., the original deferred has not fired). That implies you could not pool calls to functions that return via defer.succeed or defer.fail since those deferreds have already fired. If someone calls alertSupervisor("We have a problem in the boiler room!") and their deferred fires, then a subsequent call to alertSupervisor("We have a problem in the boiler room!") should get through to the supervisor, as per normal. This may all seem a bit esoteric, but I have several uses for it. In particular when the service providing alertSupervisor lives on one machine and is being called via RPC by A and B, both on separate machines. As for how to nicely make RPC calls like this, see Thrift [1] which now has built-in support for Twisted. Here's a first cut at a solution: from twisted.internet import defer from twisted.python import failure class DeferredPooler(object): def __init__(self, func): self.pool = {} self.func = func def _callOthers(self, result, key): if isinstance(result, failure.Failure): for d in self.pool[key]: d.errback(result) else: for d in self.pool[key]: d.callback(result) del self.pool[key] return result def __call__(self, *args, **kwargs): key = (args, hash(tuple(sorted(kwargs.items())))) if key in self.pool: d = defer.Deferred() self.pool[key].append(d) return d else: self.pool[key] = [] d = self.func(*args, **kwargs) assert isinstance(d, defer.Deferred) d.addBoth(self._callOthers, key) return d You use the class transparently, via a decorator: from twisted.internet import reactor @DeferredPooler def x(*args, **kwargs): print 'x called: args = %r, kwargs = %r' % (args, kwargs) d = defer.Deferred() reactor.callLater(2, d.callback, args) return d def printer(what): print 'printer received:', what def stop(_): reactor.stop() if __name__ == '__main__': d1 = x('fred') d1.addCallback(printer) d2 = x('fred') d2.addCallback(printer) d = defer.DeferredList([d1, d2]) d.addBoth(stop) reactor.run() If you run this you'll see x is called only once, but d1 and d2 are each fired properly. For more Twisted pleasure/pain, you can also try out funky things like: d1 = x('fred') d1.addCallback(printer) d1.addCallback(x) d1.addCallback(printer) d2 = x('fred') d2.addCallback(printer) d2.addCallback(x) d2.addCallback(printer) Here there are 4 calls to x, but only 2 get through. I could say more, as usual. But I'll wait to see if there's any reaction. One issue to consider further is the simplistic use of hash. Terry [1] Thrift: http://incubator.apache.org/thrift/

1 0

[Twisted-Python] One of a kind Gucci reps
by Hope Rios April 22, 2009

April 22, 2009

So you thought owning a brand name watch was out of your reach? Think again! http://www.wofovaxuc.cn Diam0nd Reps has it all: Rolex, Cartier, Tag Heuer, Breitling, and many more, for a fraction of the price of an original watch. And don't forget: when you order two watches, you get an extra 15 percent discount over our already low prices! http://www.wofovaxuc.cn With so many watches that look and work like the real thing, I guarantee you'll have a delicious time finding yours at our store!

1 0

[Twisted-Python] _Win32Waker
by Aron Bierbaum April 21, 2009

April 21, 2009

I have been using a custom Qt4 reactor that derives from PosixReactorBase. As a result it creates a _Win32Waker to allow threads and signals to wake up the IO thread. It seems though that the current implementation only works about half of the time. The other half it exists with : File "...\Lib\python2.6\site-packages\twisted\internet\posixbase.py", line 170, in __init__ ReactorBase.__init__(self) File "...\Lib\python2.6\site-packages\twisted\internet\base.py", line 424, in __init__ self._initThreads() File "...\Lib\python2.6\site-packages\twisted\internet\base.py", line 813, in _initThreads self.installWaker() File "...\Lib\python2.6\site-packages\twisted\internet\posixbase.py", line 206, in installWaker self.waker = _Waker(self) File "...\Lib\python2.6\site-packages\twisted\internet\posixbase.py", line 77, in __init__ client.connect(server.getsockname()) File "<string>", line 1, in connect socket.error: [Errno 10049] The requested address is not valid in its context I have attached a simple test that shows that the following code does not always return "127.0.0.1", but sometimes returns "0.0.0.0" as the IP address. # Following select_trigger (from asyncore)'s example; server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) client.setsockopt(socket.IPPROTO_TCP, 1, 1) server.bind(('127.0.0.1', 0)) server.listen(1) client.connect(server.getsockname()) My current workaround just calls the following instead: client.connect(('127.0.0.1', server.getsockname()[1])) Any ideas on what is really causing the error? If there is not a better solution can this be added to trunk for future releases? Thanks, Aron

2 7

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com April 19, 2009

April 19, 2009

1 0

[Twisted-Python] Writing a shutdown command that sends an ACK back
by Rutt, Benjamin April 15, 2009

April 15, 2009

Is there a good way, in, say, a LineReceiver based server, to implement a bit of code in lineReceived() that, if you receive a line like, let's say, 'shutdown' from a client, you can call: 1) self.transport.write('goodbye\n') # send a goodbye message to the client And then call 2) self.transport.close() # close the client's socket And then 3) reactor.stop() # terminate the server The problem I'm finding is that since I believe self.transport.write() enqueues the writing for later, and self.transport.close() enqueues the closing for later, both to be picked up by the reactor soon after my lineReceived() returns, so then if you stop the reactor in #3, it won't be available anymore to carry out any enqueued tasks for you. At least I think that's what's happening in my code. Is there any good workarounds? E.g., is reactor.callLater(5, reactor.stop) too much of a hack?

2 1

[Twisted-Python] Can't quit a Tkinter application on MacOS X when using Twisted; fix? workaround?
by Russell E. Owen April 15, 2009

April 15, 2009

I am trying to convert a cross-platform Python/Tkinter application to use Twisted and have run into a problem: at least on MacOS X I can no longer quit the application. I have appended a minimal script showing the problem. This is probably not an issue on unix and WIndows because I find I have to add my own Quit menu item anyway, so I can make it do what I like. But on MacOS X with Aqua Tcl/Tk that item is added by the system and its meaning appears to be hard-wired -- I suspect to it calls "quit" on Tk's mainloop, but I don't know. Any suggestion for a workaround would be most appreciated. Also if this is not a known issue I will file a bug report. -- Russell """Script showing failure-to-quit bug At least on MacOS XO 10.5.6 with twisted 8.2.0, python 2.5.2 (from python.org) and Aqua Tck/Tk 8.4.19 the File>Quit menu has no effect and typing ctrl-Q causes the menu to briefly highlight (as usual) but has no other effect. "" import Tkinter import twisted.internet.tksupport root = Tkinter.Tk() twisted.internet.tksupport.install(root) reactor = twisted.internet.reactor reactor.run()

2 5