Mailman 3 May 2009 - Twisted

Re: [Twisted-Python] Consistent interfaces to asynchronous partially-available services using Deferreds and state machines (was Re: Another approach to allowing __init__ to work with Deferreds)
by Terry Jones 14 May '09

14 May '09

>>>>> "Phil" == Phil Christensen <phil(a)bubblehouse.org> writes: Phil> I don't know if I agree with the need for such a feature (that is, Phil> deferred __init__ usage), but it was a very interesting coding Phil> challenge I wanted to take a whack at. I *think* I might have found a Phil> solution, but I don't know if it falls under the heading of Phil> "decorator abuse" ;-) Hi Phil I finally had time to look at your solution a bit (though I've not run it). It does a couple of things I wouldn't have thought of, like putting the dictionary onto the deferredInit function. A couple of comments, supposing I understand your code properly: - One thing I had hoped to avoid was to slow the class methods down by having them always check the original deferred (or a flag) before taking action. My approach does this by moving them aside and then putting them back in place once the deferred fires. Your solution requires that every decorated method does several extra things before it gets going. That could be greatly reduced if you were to check self.initDeferred.called and simply call the original function if the deferred has fired. - If multiple calls are made to instance methods before the init deferred has fired, they will, as I read it, all try to del deferredInit.waiting[self] in _finish. So I guess that del needs to be conditional or in a try/except. - Using self as a key into the dict on initDeferred seems like it addresses Glyph's observation/criticism that my approach raises questions wrt inheritance. - You could use chainDeferred where you're currently using .addCallbacks(resultDeferred.callback, resultDeferred.errback) That's all for now. I'll see if I have more time to think about all this. When I tried to use a decorator the first time, I was also using a super class (whereas you're putting state into a dict on the deferredInit function) but I got into a mess accessing self properly (partly because, I think, I wanted to have a mixin class and I was looking at self.__class__.__mro__). In any case, thanks for replying, for playing with it, and for posting your code. I got to learn new things as a result, which is really great :-) Terry

2 1

[Twisted-Python] XMPP/Jabber server example
by Garret Heaton 14 May '09

14 May '09

I'm trying to create an XMPP server using twisted.words and am having difficulty finding the correct way to make use of the classes provided. I was hoping someone might have an example or short explanation of which classes would be used to create a very basic server (using the latest classes<http://twistedmatrix.com/documents/8.2.0/api/twisted.words.protocols.jabber…>from 8.2.0). So far I have a t.w.p.j.xmlstream.XmlStreamServerFactory object creating t.w.x.xmlstream.XmlStream protocol instances. The t.w.p.j.xmlstream.XmlStream objects are being given a t.w.p.j.xmlstream.ListenAuthenticator object. The factory is being created and given to a t.a.internet.TCPServer service which has a t.a.service.Application as its parent service. Everything runs fine and I can see data coming in, but I wasn't sure what the correct functions to override and deal with the data were, or where to attach event listeners using addBootstrap(). Also, how do xmlstream.StreamManager, xmlstream.XMPPHandler, and xmlstream.XMPPHandlerCollection fit into this. Are they appropriate to use for a server? Is there still work being done on the goals listed at http://twistedmatrix.com/trac/wiki/XMPPServerArchitecture? Thanks! Really enjoying Twisted so far.

3 3

[Twisted-Python] How do I turn off certain log messages?
by Reza Lotun 14 May '09

14 May '09

Hi, I was just wondering about the best way to customize twisted logging - specifically, I'd like to remove the: "Starting factory.." and "Stopping factory.." log messages that pop up whenever I use twisted.web.client HTTPClientFactory. Is it simply a matter of writing a custom Logging Observer and specifically filtering those kinds of messages? I run a twisted.web server with twistd (with DailyLogFile) that for every connection to it makes a number of web-service calls using web.client. I want to have Apache style access logs and be able to spit out error conditions *without* anything extraneous creeping in. What are the idioms and best practices? Thanks, Reza -- Reza Lotun +44 (0)7521 310 763 rlotun(a)gmail.com

3 4

[Twisted-Python] Recompiling _epoll.pyx
by Reza Lotun 14 May '09

14 May '09

Hi, I noticed in the documentation for _epoll.pyx that the corresponding .c file was generated using pyrex 0.9.4.1. On a whim I downloaded 0.9.8.5 and regenerated the file and rebuilt python. I'm currently testing my server but noticing nothing out of the ordinary. Before I march on my merry way I was wondering if there could be any potential pitfall of using the non-standard _epoll.c? I only though to try it because - I'm using Python 2.6 which I compiled from source - I'm using Twisted trunk - A quick glance between pyrex 0.9.4.1 to 0.9.8.5 showed a lot of activity, some of which seemed marked with "bug fix" and "more efficient code" - I'm currently stress testing my server and wanted to see what sort of changes could be induced by modifying different parts of the system. Anyway, just wondering. Cheers, Reza -- Reza Lotun +44 (0)7521 310 763 rlotun(a)gmail.com

1 1

Re: [Twisted-Python] can't get a simple client/server example to work!
by bbxx789_05ss＠yahoo.com 13 May '09

13 May '09

Hi, --- On Wed, 5/13/09, Adam <adam.bcn(a)gmail.com> wrote: > From: Adam <adam.bcn(a)gmail.com> > Subject: Re: [Twisted-Python] can't get a simple client/server example to work! > To: "Twisted general discussion" <twisted-python(a)twistedmatrix.com> > Date: Wednesday, May 13, 2009, 1:08 AM > 2009/5/13 bbxx789_05ss(a)yahoo.com > <bbxx789_05ss(a)yahoo.com> > > > Here is the server > code:-------------from > twisted.internet.protocol import Protocol, > Factoryfrom twisted.internet import > reactorimport sys > class MyServer(Protocol): > def connectionMade(self): > > self.transport.write("Hello") > factory = Factory()factory.protocol = > MyServer > reactor.listenTCP(2000, factory) > > reactor.run()----------- > > Here is the client > code:--------------from > twisted.internet.protocol import Protocol, > ClientCreatorfrom twisted.internet import > reactorimport sys > class MyClient(Protocol): def > dataReceived(self, data): > sys.stdout.write(data) > > self.transport.loseConnection()------------ > What do I need to add to the client code to make > it connect to the server? I think the "How To" > documentation is woefully inadequate. For instance, it > says this: > > ----Here is a simple example: > from > twisted.internet.protocol > import Protocol > from sys import stdout > > class Echo(Protocol): > def dataReceived(self, > data): > > stdout.write(data)------ > Anyone who knows even a little bit of python > should be able to recognize that that code does absolutely > nothing. > > > Greetings nameless-and-frustrated Twisted beginner! Have > a look at the echoclient.py example, which you can download > here: > > http://twistedmatrix.com/projects/core/documentation/examples/index.html > > > Essentially you also need to create an EchoFactory and set > it to run within a reactor (notice that your server example > already does this). > > Hope that helps, > > ~Adam > I can't get anything to work. I asked a simple question, but I can't seem to pry an answer out of this group, and the simple examples in the documentation are incomplete. I've decided not to pursue Twisted any longer. Don't bother responding. I am unsubscribing.

2 1

Re: [Twisted-Python] can't get a simple client/server example to work!
by bbxx789_05ss＠yahoo.com 13 May '09

13 May '09

Hi, --- On Wed, 5/13/09, Itamar Shtull-Trauring <itamar(a)itamarst.org> wrote: > > Have you read the client-side howto? > > http://twistedmatrix.com/projects/core/documentation/howto/clients.html > > Have you read my email?

1 0

[Twisted-Python] can't get a simple client/server example to work!
by bbxx789_05ss＠yahoo.com 13 May '09

13 May '09

Here is the server code:-------------from twisted.internet.protocol import Protocol, Factoryfrom twisted.internet import reactorimport sys class MyServer(Protocol): def connectionMade(self): self.transport.write("Hello") factory = Factory()factory.protocol = MyServer reactor.listenTCP(2000, factory) reactor.run()----------- Here is the client code:--------------from twisted.internet.protocol import Protocol, ClientCreatorfrom twisted.internet import reactorimport sys class MyClient(Protocol): def dataReceived(self, data): sys.stdout.write(data) self.transport.loseConnection()------------ What do I need to add to the client code to make it connect to the server? I think the "How To" documentation is woefully inadequate. For instance, it says this: ----Here is a simple example:from twisted.internet.protocol import Protocol from sys import stdout class Echo(Protocol): def dataReceived(self, data): stdout.write(data)------ Anyone who knows even a little bit of python should be able to recognize that that code does absolutely nothing.

3 2

[Twisted-Python] installation error
by Alec Matusis 12 May '09

12 May '09

Installing 8.2.0: creating /web20/encap/python-2.4.4/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-li nux-x86_64.egg Extracting Twisted-8.2.0-py2.4-linux-x86_64.egg to /web20/encap/python-2.4.4/lib/python2.4/site-packages File "/web20/encap/python-2.4.4/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-l inux-x86_64.egg/twisted/test/generator_failure_tests.py", line 66 yield ^ SyntaxError: invalid syntax Adding Twisted 8.2.0 to easy-install.pth file I think generator_failure_tests.py expects python 2.5+ Perhaps it should skip this test for python 2.4?

2 2

[Twisted-Python] Another approach to allowing __init__ to work with Deferreds
by Terry Jones 12 May '09

12 May '09

I posted to this list back in Nov 2008 with subject: A Python metaclass for Twisted allowing __init__ to return a Deferred Briefly, I was trying to find a nice way to allow the __init__ method of a class to work with deferreds in such a way that methods of the class could use work done by __init__ safe in the knowledge that the deferreds had completed. E.g., if you have class X(object): def __init__(self, host, port): def final(connection): self.db = connection d = makeDBConnection(host, port) d.addCallback(final) def query(self, q): return self.db.runQuery(q) Then when you make an X and call query on it, there's a chance the deferred wont have fired, and you'll get an error. This is just a very simple illustrative example. There are many more, and this is a general problem of the synchronous world (in which __init__ is supposed to prepare a fully-fledged class instance and cannot return a deferred) meeting the asynchronous world in which we would like to (and must) use deferreds. The earlier thread: http://twistedmatrix.com/pipermail/twisted-python/2008-November/018600.html Although I learned a lot in that thread, I wasn't completely happy with any of the solutions. Some of the things that still bugged me are in posts towards the end of the thread: http://twistedmatrix.com/pipermail/twisted-python/2008-November/018624.html http://twistedmatrix.com/pipermail/twisted-python/2008-November/018634.html The various approaches we took back then all boiled down to waiting for a deferred to fire before the class instance was fully ready to use. When that happened, you had your instance and could call its methods. I had also thought about an alternate approach: having __init__ add a callback to the deferreds it dealt with to set a flag in self and then have all dependent methods check that flag to see if the class instance was ready for use. But that 1) is ugly (too much extra code); 2) means the caller has to be prepared to deal with errors due to the class instance not being ready, and 3) adds a check to every method call. It would look something like this: class X(object): def __init__(self, host, port): self.ready = False def final(connection): self.db = connection self.ready = True d = makeDBConnection(host, port) d.addCallback(final) def query(self, q): if not self.ready: raise IAmNotReadyException() return self.db.runQuery(q) That was too ugly for my taste, for all of the above reasons, most especially for forcing the unfortunate caller of my code to handle IAmNotReadyException. Anyway.... fast forward 6 months and I've hit the same problem again. It's with existing code, in which I would like an __init__ to call something that (now, due to changes elsewhere) returns a deferred. So I started thinking again, and came up with a much cleaner way to do the alternate approach via a class mixin: from twisted.internet import defer class deferredInitMixin(object): def wrap(self, d, *wrappedMethods): self.waiting = [] self.stored = {} def restore(_): for method in self.stored: setattr(self, method, self.stored[method]) for d in self.waiting: d.callback(None) def makeWrapper(method): def wrapper(*args, **kw): d = defer.Deferred() d.addCallback(lambda _: self.stored[method](*args, **kw)) self.waiting.append(d) return d return wrapper for method in wrappedMethods: self.stored[method] = getattr(self, method) setattr(self, method, makeWrapper(method)) d.addCallback(restore) You use it as in the class Test below: from twisted.internet import defer, reactor def fire(d, value): print "I finally fired, with value", value d.callback(value) def late(value): d = defer.Deferred() reactor.callLater(1, fire, d, value) return d def called(result, what): print 'final callback of %s, result = %s' % (what, result) def stop(_): reactor.stop() class Test(deferredInitMixin): def __init__(self): d = late('Test') deferredInitMixin.wrap(self, d, 'f1', 'f2') def f1(self, arg): print "f1 called with", arg return late(arg) def f2(self, arg): print "f2 called with", arg return late(arg) if __name__ == '__main__': t = Test() d1 = t.f1(44) d1.addCallback(called, 'f1') d2 = t.f1(33) d2.addCallback(called, 'f1') d3 = t.f2(11) d3.addCallback(called, 'f2') d = defer.DeferredList([d1, d2, d3]) d.addBoth(stop) reactor.run() Effectively, the __init__ of my Test class asks deferredInitMixin to wrap some of its methods. deferredInitMixin stores the original methods away and replaces each of them with a function that immediately returns a deferred. So after __init__ finishes, code that calls the now-wrapped methods of the class instance before the deferred has fired will get a deferred back as usual (but see * below). As far as they know, everything is normal. Behind the scenes, deferredInitMixin has arranged for these deferreds to fire only after the deferred passed from __init__ has fired. Once that happens, deferredInitMixin also restores the original functions to the instance. As a result there is no overhead later to check a flag to see if the instance is ready to use. If the deferred from __init__ happens to fire before any of the instance's methods are called, it will simply restore the original methods. Finally (obviously?) you only pass the method names to deferredInitMixin that depend on the deferred in __init__ being done. BTW, calling the methods passed to deferredInitMixin "wrapped" isn't really accurate. They're just temporarily replaced. I quite like this approach. It's a second example of something I did in http://twistedmatrix.com/pipermail/twisted-python/2009-April/019522.html in which a pool of deferreds is accumulated and they're all fired when another deferred fires. It's nice because you don't reply with an error and there's no need for locking or other form of coordination - the work you need done is already in progress, so you get back a fresh deferred and everything goes swimmingly. * Minor note: the methods you wrap should probably be ones that already return deferreds. That way you always get a deferred back from them, whether they're temporarily wrapped or not. The above mixin works just fine if you ask it to wrap non-deferred-returning functions, but you have to deal with the possibility that they will return a deferred (i.e., if you call them while they're wrapped). Comments welcome / wanted. Terry

3 2

Re: [Twisted-Python] Another approach to allowing __init__ to work with Deferreds
by Terry Jones 12 May '09

12 May '09

Hi again Drew I realized I could have added another reason why I like my approach (for my current situation): I'm changing an existing (i.e., deployed) class so that its __init__ calls something returning a deferred which some instance methods of my class rely on having fired. I already have other code that uses my class, and that code is all written to use my class in the regular way. I could go change all those calls, but that gets me into a further mess as some of those calls are in the __init__ methods of still other classes... So I'd have to go change all the uses of those classes too. This part of my preference for my solution is due to having the class already deployed. It's not really part of the core problem, just my particular situation. Terry

1 0