Mailman 3 November 2008 - Twisted

[Twisted-Python] AMP port to C?
by David Ripton 06 Nov '08

06 Nov '08

Has anyone ported Twisted AMP to C? If so, is the code publicly available? Thanks. -- David Ripton dripton(a)ripton.net

2 1

Re: [Twisted-Python] RPython and Twisted
by Andrew Francis 05 Nov '08

05 Nov '08

Hi J.P: AF>I wish to start experimenting by re-writing an example (the programme I AF>posted in RPtyhon and converting it to C. Oops. I meant Stackless Python not RPython. >Since the RPython/PyPy toolchain currently has better support for >generating standalone executables than for doing much else, I'd suggest >taking that approach. Compile your RPython to a standalone executable, >run it from a Twisted-based process and do some simple RPC over stdio. > It's a bit heavy handed, but I think it's the most straightforward >approach given the state of RPython. It is a good approach. Bear with me, I am still reading up but let me run this one by you: compile a RPython programme to C. Embed a Python interpreter running Twisted. Still heavy handed but I think that brings us closer to understanding integration issues (and getting something to work). >There's probably nothing special about using RPython with Twisted - Again, I am not sure - if the Twisted parts could not be converted, I believe the programme would run as a regular Python programme. Then is the question of how difficult would it be to write Twisted in RPython? Would it be worth it? I suspect that the I/O would still be bottleneck. >from the RPython perspective, Twisted is just another pile of Python >code that you need to figure out an integration strategy for. I don't know about this. I am reading the FAQ: [blockquote] Can I use PyPy and RPython to compile smaller parts of my Python program? No. That would be possible, and we played with early attempts in that direction, but there are many delicate issues: for example, how the compiled and the non-compiled parts exchange data. Supporting this in a nice way would be a lot of work. [/blocquote] J.P thanks for the suggestion. I guess it is time to write some tests :-) Cheers, Andrew

1 0

Re: [Twisted-Python] FirstError handling from defer.gatherResults()?
by Terry Jones 05 Nov '08

05 Nov '08

Hi Justin >>>>> "Justin" == Justin Warren <daedalus(a)eigenmagic.com> writes: Justin> If an error occurs, gatherResults() behaves like Justin> DeferredList(fireOnOneErrback=True), but the errback will receive a Justin> FirstError object, not a Failure. Justin> Normally, I'd do something like this in an Errback: Justin> def cb_handle_failure(failure): Justin> e = failure.trap(MyNonFatalException) Justin> But what is the right way to do this when I receive a FirstError? The FirstError holds the original failure in an attribute called subFailure. So you should be able to do e = failure.value.subFailure.trap(MyNonFatalException) Terry

3 3

[Twisted-Python] FirstError handling from defer.gatherResults()?
by Justin Warren 04 Nov '08

04 Nov '08

I'm missing something obvious, so I must consult the gurus of the list: I have some code that gets info from remote devices, a little like this: def get_thing_info(): dlist = [] for thing in thinglist: d = thing.fetch_info() dlist.append(d) return defer.gatherResults(dlist) If an error occurs, gatherResults() behaves like DeferredList(fireOnOneErrback=True), but the errback will receive a FirstError object, not a Failure. Normally, I'd do something like this in an Errback: def cb_handle_failure(failure): e = failure.trap(MyNonFatalException) if e: log.debug("Not a fatal failure. Getting info again.") return get_thing_info() But what is the right way to do this when I receive a FirstError? -- Justin Warren <daedalus(a)eigenmagic.com>

1 0

Re: [Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones 04 Nov '08

04 Nov '08

[I'm replying the original because I want to comment on comments by all of JP, Esteve and Glyph] Glyph - I appreciate your comments on testing, and agree that's problematic. I also like your pattern, have run into it myself, and will use it - thanks. Glyph also said I didn't really provide a use-case, so I'll do that a bit more clearly now. That leads me back in the direction of preferring my metaclass solution. from my.project.code import database class CoordinatorHandler(object): def __init__(self, tableName, tableSpec, dbURI): def setDb(db): self.db = db d = database.getDB(dbURI) d.addCallback(setDb) d.addCallback(self.createDBTable, tableName, tableSpec) return d def _createDBTable(self, txn, tableName, tableSpec): txn.execute('CREATE TABLE %s(%s)' % (tableName, tableSpec)) def createDBTable(self, result, tableName, tableSpec): assert result is None return self.db.runInteraction(self._createDBTable, tableName, tableSpec) Here database.getDB returns a Deferred, as does self.createDBTable. The problem with approaches that don't actually create the class instance, is that __init__ is calling self.createDBTable, but self doesn't exist yet. So putting code to deal with Deferreds into __new__ wont help unless that code has nothing to do with the instance of the class. But using a metaclass makes it really easy. You just write your code completely as usual, using whatever Twisted/Deferred calls you want (including digging around in self), and return the Deferred result. You can wrap your __init__ in inlineCallbacks if you like. The difference is that my metaclass approach does create an instanced of the class. It just moves __init__ out of the way (into the __hidden__ dict, along with the args and kwargs) and calls it when you call __instantiate__. So I find the metaclass approach more general, though of course I don't like the clutter of __hidden__ or __instantiate__. With JP's suggestion of cleaning up the return result of __instantiate__ to just return the instance, I think it's a pretty clean solution - from the POV of the caller. You just act as normal in writing your class, but stick in a __metaclass__. Then calling gets you a Deferred, which is perfectly normal in the Twisted world. I guess what you should do depends on how complex your situation is, and also on whether you've already written your class code when you realize you need __init__ to somehow deal with Deferreds (in which case it may be easier to just add __metaclass__). If you do have the simpler situation, in which the Deferred is coming from the outside world (i.e., not from calling methods on self), then Glyph's approach is probably the nicest (the following is untested): class X: def __init__(self, *args, dResult=None, **kw): # do something @classmethod def fromDeferred(cls, d, *args, **kw): def cb(result): return cls(*args, dResult=result, **kw) return d.addCallback(cb) Which lets you create an object directly (no deferred involved), lets you create it after a deferred fires, and also lets you directly send in a fake deferred result. Terry

3 2

Re: [Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones 04 Nov '08

04 Nov '08

Hi again Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> You may want to create some _other_ configuration object, loaded glyph> from the remote database, in the middle of startService. You may glyph> even want to have a multi-phase internal startup procedure to glyph> generate that object. But you need to keep track of what state glyph> those things are in so that the infrastructure can come around and glyph> say stopService to you in the middle of that procedure and still get glyph> a clean, ordered shutdown. If the infrastructure were to get back a glyph> Deferred, there's nothing to call stopService on. Sorry to post so much on this - I do hope it's useful to other people. Here's what I think you were saying, but in my own words: I could do more complex work in startService, including dealing with Deferreds. So to give a simple (non-working) example based on your earlier classmethod suggestion, here's some sample code: from myservice import myhandler class MyService(internet.TCPServer): def __init__(self, port, dbURI): self.port = port self.dbURI = dbURI @defer.inlineCallbacks def startService(self): self.handler = yield myhandler.Handler.fromDbURI(self.dbURI) internet.TCPServer.__init__(self, self.port, self.handler.getFactory()) yield self.handler.startDB() yield self.handler.createTables() # Perhaps do other init things... internet.TCPServer.startService(self) def stopService(self): self.handler.exit() return internet.TCPServer.stopService(self) As you suggested, myhandler.Handler.fromDbURI is a classmethod that gets a Deferred and adds a callback to it that will return an instance of myhandler.Handler. Then I do a couple of other Deferred-producing operations and finally start a TCP server. The problem with this is that the caller of startService gets a Deferred back immediately after myhandler.Handler.fromDbURI returns its Deferred. As far as I can see, no-one looks at the return result of startService. So the calling code may proceed willy nilly on the assumption that the service has started, when in fact it hasn't. I.e., when startService returns its Deferred, the myhandler.Handler.fromDbURI likely wont have completed, and the subsequent calls (startDB, createTables) wont have even started. You went on to say: glyph> You may even want to have a multi-phase internal startup procedure glyph> to generate that object. But you need to keep track of what state glyph> those things are in so that the infrastructure can come around and glyph> say stopService to you in the middle of that procedure and still get glyph> a clean, ordered shutdown. Which I guess you'd just repeat again now? IOW, if my startService is going to do various operations with Deferreds, then as those things happen I should be (for example) updating handler.state to indicate whether the handler is actually ready. And if a stopService call comes along, I have the handler object and can examine its state to shut things down properly, depending on where I am in setting things up. To close a further hole, I'd put an attribute onto the MyService instance to indicate when the yield that will result in self.handler being assigned is still awaiting the generator send in inlineCallbacks. Unfortunately this seems to imply that I also then need to put checks into handler methods to make sure it's actually ready before trying anything. I suppose I'd just log and ignore those early calls, maybe checking that they actually were early. And/or add a timer. What an involved mess! Is all this a reasonable summary of what you were imagining / describing? Thanks again, and apologies for so many mails. Terry

1 0

[Twisted-Python] DeferredDict, defaultdict, map/reduce, silliness
by Terry Jones 04 Nov '08

04 Nov '08

This isn't meant to be taken too seriously - perhaps just food for someone else to chew on while trying to relax... I was thinking the other night about what a DeferredDict might look like, what it might be useful for, if anything, etc. We have DeferredList of course, so why not a dict? Then I also remembered collections.defaultdict and couldn't resist the appeal of trying defaultdict(defer.Deferred). That thinking led to the map/reduce silliness below. I don't know that it leads anywhere - I was just playing around. Terry import sys from collections import defaultdict from twisted.internet import defer, reactor class MapReduce(object): def __init__(self, init, cb): self.init = init self.cb = cb self.d = defaultdict(defer.Deferred) def map(self, *keys): for key in keys: self.d[key].addCallback(self.cb) def reduce(self): def finalize(result, key): return (key, result) deferreds = [] for key, d in self.d.iteritems(): d.addCallback(finalize, key) deferreds.append(d) d.callback(self.init) return defer.gatherResults(deferreds) def main(): def printResult(result): print result reactor.stop() mr = MapReduce(0, lambda x: x + 1) for line in sys.stdin.readlines(): mr.map(*line.split()) d = mr.reduce().addCallback(printResult) if __name__ == '__main__': reactor.callLater(0, main) reactor.run()

3 3

[Twisted-Python] Trying to proxy through multiple IPs
by Erik Wickstrom 04 Nov '08

04 Nov '08

Hi all, I'm trying to write a proxy server that accepts connections on multiple ports, and depending on the port, using a different IP for the outgoing connection. My code works except that adding additional reactor.listenTCP(...)s overwrite the IP of the previous listenTCPs. So the proxy accepts connections on all the desired ports, but only uses the last IP address for outgoing connections. The ip (bindIP) is somehow being overwritten. Based on advice from IRC (exarkun --thanks!), I've tried a couple attempts at using closures to solve the problem, but non of my implementations have done the trick. Can anyone see any other changes that might get this working? # Also pasted at http://dpaste.com/88623/ incase of email garbling... #from twisted.web import proxy, http import urlparse from urllib import quote as urlquote from twisted.internet import reactor from twisted.internet.protocol import ClientFactory from twisted.web.resource import Resource from twisted.web.server import NOT_DONE_YET from twisted.web.http import HTTPClient, Request, HTTPChannel from twisted.web.proxy import ProxyClient, ProxyClientFactory from twisted.web import http from twisted.internet import reactor from twisted.python import log import sys log.startLogging(sys.stdout) class ProxyRequest(Request): """ Used by Proxy to implement a simple web proxy. @ivar reactor: the reactor used to create connections. @type reactor: object providing L{twisted.internet.interfaces.IReactorTCP} """ protocols = {'http': ProxyClientFactory} ports = {'http': 80} def __init__(self, channel, queued, reactor=reactor): Request.__init__(self, channel, queued) self.reactor = reactor def process(self): parsed = urlparse.urlparse(self.uri) protocol = parsed[0] host = parsed[1] port = self.ports[protocol] if ':' in host: host, port = host.split(':') port = int(port) rest = urlparse.urlunparse(('', '') + parsed[2:]) if not rest: rest = rest + '/' class_ = self.protocols[protocol] headers = self.getAllHeaders().copy() if 'host' not in headers: headers['host'] = host self.content.seek(0, 0) s = self.content.read() clientFactory = class_(self.method, rest, self.clientproto, headers, s, self) self.reactor.connectTCP(host, port, clientFactory, bindAddress=(self.bindIP,0)) class Proxy(HTTPChannel): requestFactory = ProxyRequest class ProxyFactory(http.HTTPFactory): def class_factory(self, bindIP): def closure(ip): klass2 = ProxyRequest setattr(klass2, 'bindIP', ip) return klass2 klass = Proxy setattr(klass, 'requestFactory', closure(bindIP)) return klass def __init__(self, ip): http.HTTPFactory.__init__(self) self.ip = ip #self.protocol = proxy.Proxy self.protocol = self.class_factory(ip) reactor.listenTCP(8080, ProxyFactory('192.168.168.101'), interface="192.168.168.1") reactor.listenTCP(8081, ProxyFactory('192.168.168.102'), interface="192.168.168.1") reactor.run() ## Thanks! Erik

3 2

Re: [Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones 04 Nov '08

04 Nov '08

Hi again. Despite all the useful suggestions, I've just realized that I still can't complete what I was originally trying to do... I'm trying to make a Twisted service. In my twisted/plugins/myService.py file I have something like this: from zope.interface import implements from twisted.plugin import IPlugin from twisted.application.service import IServiceMaker from twisted.python import usage from my.application import myservice class Options(usage.Options): optParameters = [ ['db-uri', None, None, 'A db URI'], ] class MyServiceMaker(object): implements(IServiceMaker, IPlugin) tapname = 'My service' description = 'Desc.' options = Options def makeService(self, options): return myservice.MyService(dbURI=options['db-uri']) myServiceMaker = MyServiceMaker() The problem here is that the service class I want to instantiate and return from makeService calls something that returns a Deferred. In Glyph's classmethod, Esteve's __new__, and my metaclass code, the thing you get back when you go to use class is a Deferred. You have to wait for that Deferred to fire before you get your instance. In JP's approach you also wait for a Deferred before instantiating your class. That would be fine if makeService could in turn return a Deferred that would result in an object implementing IServiceMaker, but as far as I can tell, makeService does not allow this. And in the context of a Twisted plugin file like the above I'm even less clear on how to go about it. Am I missing something here? If not, wouldn't it be nice if makeService allowed you to return a Deferred? AKA teaching your grandmother to suck eggs :-) Terry

2 1

Re: [Twisted-Python] DeferredDict, defaultdict, map/reduce, silliness
by Terry Jones 04 Nov '08

04 Nov '08

Hi Valentino >>>>> "Valentino" == Valentino Volonghi <dialtone(a)gmail.com> writes: Valentino> This is actually a very very nice pattern. I thought it was too cute not to post, though I didn't really know what to do with it. Something along the lines of map/reduce occurred to me quite late. I bet there are other nice things you could do, mainly along the lines of processing/bucketing a bunch of unpredictable input. One thing I'd (probably) change would be to not use Deferred as the argument to default dict, but to use a class that looked like it but with the Deferred already called. That way you don't have the potentially very long chain of callbacks building up. It's also partly a question of when you want to do the work - each time you call addCallback, or at the end when you've accumulated all the input and sent it off to its respective callback chain. So you could do something like this: def getDefault(init): def x(): return defer.succeed(init) return x and then the MapReduce class would look like class MapReduce(object): def __init__(self, init, cb): self.cb = cb self.d = defaultdict(getDefault(init)) Or, more flexibly, pass the defaultdict into the MapReduce class, giving you the option of not having the callback chain already called, etc. Anyway, I'm glad you like it, and hope you can do something nice with it. Terry

1 0