Mailman 3 November 2008 - Twisted

[Twisted-Python] AMP port to C?
by David Ripton Nov. 6, 2008

Nov. 6, 2008

Has anyone ported Twisted AMP to C? If so, is the code publicly available? Thanks. -- David Ripton dripton(a)ripton.net

2 1

Re: [Twisted-Python] RPython and Twisted
by Andrew Francis Nov. 5, 2008

Nov. 5, 2008

Hi J.P: AF>I wish to start experimenting by re-writing an example (the programme I AF>posted in RPtyhon and converting it to C. Oops. I meant Stackless Python not RPython. >Since the RPython/PyPy toolchain currently has better support for >generating standalone executables than for doing much else, I'd suggest >taking that approach. Compile your RPython to a standalone executable, >run it from a Twisted-based process and do some simple RPC over stdio. > It's a bit heavy … [View More]

1 0

Re: [Twisted-Python] FirstError handling from defer.gatherResults()?
by Terry Jones Nov. 5, 2008

Nov. 5, 2008

Hi Justin >>>>> "Justin" == Justin Warren <daedalus(a)eigenmagic.com> writes: Justin> If an error occurs, gatherResults() behaves like Justin> DeferredList(fireOnOneErrback=True), but the errback will receive a Justin> FirstError object, not a Failure. Justin> Normally, I'd do something like this in an Errback: Justin> def cb_handle_failure(failure): Justin> e = failure.trap(MyNonFatalException) Justin> But what is the right way to do this when I … [View More]

3 3

[Twisted-Python] FirstError handling from defer.gatherResults()?
by Justin Warren Nov. 5, 2008

Nov. 5, 2008

I'm missing something obvious, so I must consult the gurus of the list: I have some code that gets info from remote devices, a little like this: def get_thing_info(): dlist = [] for thing in thinglist: d = thing.fetch_info() dlist.append(d) return defer.gatherResults(dlist) If an error occurs, gatherResults() behaves like DeferredList(fireOnOneErrback=True), but the errback will receive a FirstError object, not a Failure. Normally, I'd do something like this in an Errback: … [View More]

1 0

Re: [Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones Nov. 4, 2008

Nov. 4, 2008

[I'm replying the original because I want to comment on comments by all of JP, Esteve and Glyph] Glyph - I appreciate your comments on testing, and agree that's problematic. I also like your pattern, have run into it myself, and will use it - thanks. Glyph also said I didn't really provide a use-case, so I'll do that a bit more clearly now. That leads me back in the direction of preferring my metaclass solution. from my.project.code import database class CoordinatorHandler(object): … [View More] def __init__(self, tableName, tableSpec, dbURI): def setDb(db): self.db = db d = database.getDB(dbURI) d.addCallback(setDb) d.addCallback(self.createDBTable, tableName, tableSpec) return d def _createDBTable(self, txn, tableName, tableSpec): txn.execute('CREATE TABLE %s(%s)' % (tableName, tableSpec)) def createDBTable(self, result, tableName, tableSpec): assert result is None return self.db.runInteraction(self._createDBTable, tableName, tableSpec) Here database.getDB returns a Deferred, as does self.createDBTable. The problem with approaches that don't actually create the class instance, is that __init__ is calling self.createDBTable, but self doesn't exist yet. So putting code to deal with Deferreds into __new__ wont help unless that code has nothing to do with the instance of the class. But using a metaclass makes it really easy. You just write your code completely as usual, using whatever Twisted/Deferred calls you want (including digging around in self), and return the Deferred result. You can wrap your __init__ in inlineCallbacks if you like. The difference is that my metaclass approach does create an instanced of the class. It just moves __init__ out of the way (into the __hidden__ dict, along with the args and kwargs) and calls it when you call __instantiate__. So I find the metaclass approach more general, though of course I don't like the clutter of __hidden__ or __instantiate__. With JP's suggestion of cleaning up the return result of __instantiate__ to just return the instance, I think it's a pretty clean solution - from the POV of the caller. You just act as normal in writing your class, but stick in a __metaclass__. Then calling gets you a Deferred, which is perfectly normal in the Twisted world. I guess what you should do depends on how complex your situation is, and also on whether you've already written your class code when you realize you need __init__ to somehow deal with Deferreds (in which case it may be easier to just add __metaclass__). If you do have the simpler situation, in which the Deferred is coming from the outside world (i.e., not from calling methods on self), then Glyph's approach is probably the nicest (the following is untested): class X: def __init__(self, *args, dResult=None, **kw): # do something @classmethod def fromDeferred(cls, d, *args, **kw): def cb(result): return cls(*args, dResult=result, **kw) return d.addCallback(cb) Which lets you create an object directly (no deferred involved), lets you create it after a deferred fires, and also lets you directly send in a fake deferred result. Terry [View Less]

3 2

Re: [Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones Nov. 4, 2008

Nov. 4, 2008

Hi again Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> You may want to create some _other_ configuration object, loaded glyph> from the remote database, in the middle of startService. You may glyph> even want to have a multi-phase internal startup procedure to glyph> generate that object. But you need to keep track of what state glyph> those things are in so that the infrastructure can come around and glyph> say stopService to you … [View More]in the middle of that procedure and still get glyph> a clean, ordered shutdown. If the infrastructure were to get back a glyph> Deferred, there's nothing to call stopService on. Sorry to post so much on this - I do hope it's useful to other people. Here's what I think you were saying, but in my own words: I could do more complex work in startService, including dealing with Deferreds. So to give a simple (non-working) example based on your earlier classmethod suggestion, here's some sample code: from myservice import myhandler class MyService(internet.TCPServer): def __init__(self, port, dbURI): self.port = port self.dbURI = dbURI @defer.inlineCallbacks def startService(self): self.handler = yield myhandler.Handler.fromDbURI(self.dbURI) internet.TCPServer.__init__(self, self.port, self.handler.getFactory()) yield self.handler.startDB() yield self.handler.createTables() # Perhaps do other init things... internet.TCPServer.startService(self) def stopService(self): self.handler.exit() return internet.TCPServer.stopService(self) As you suggested, myhandler.Handler.fromDbURI is a classmethod that gets a Deferred and adds a callback to it that will return an instance of myhandler.Handler. Then I do a couple of other Deferred-producing operations and finally start a TCP server. The problem with this is that the caller of startService gets a Deferred back immediately after myhandler.Handler.fromDbURI returns its Deferred. As far as I can see, no-one looks at the return result of startService. So the calling code may proceed willy nilly on the assumption that the service has started, when in fact it hasn't. I.e., when startService returns its Deferred, the myhandler.Handler.fromDbURI likely wont have completed, and the subsequent calls (startDB, createTables) wont have even started. You went on to say: glyph> You may even want to have a multi-phase internal startup procedure glyph> to generate that object. But you need to keep track of what state glyph> those things are in so that the infrastructure can come around and glyph> say stopService to you in the middle of that procedure and still get glyph> a clean, ordered shutdown. Which I guess you'd just repeat again now? IOW, if my startService is going to do various operations with Deferreds, then as those things happen I should be (for example) updating handler.state to indicate whether the handler is actually ready. And if a stopService call comes along, I have the handler object and can examine its state to shut things down properly, depending on where I am in setting things up. To close a further hole, I'd put an attribute onto the MyService instance to indicate when the yield that will result in self.handler being assigned is still awaiting the generator send in inlineCallbacks. Unfortunately this seems to imply that I also then need to put checks into handler methods to make sure it's actually ready before trying anything. I suppose I'd just log and ignore those early calls, maybe checking that they actually were early. And/or add a timer. What an involved mess! Is all this a reasonable summary of what you were imagining / describing? Thanks again, and apologies for so many mails. Terry [View Less]

1 0

[Twisted-Python] DeferredDict, defaultdict, map/reduce, silliness
by Terry Jones Nov. 4, 2008

Nov. 4, 2008

This isn't meant to be taken too seriously - perhaps just food for someone else to chew on while trying to relax... I was thinking the other night about what a DeferredDict might look like, what it might be useful for, if anything, etc. We have DeferredList of course, so why not a dict? Then I also remembered collections.defaultdict and couldn't resist the appeal of trying defaultdict(defer.Deferred). That thinking led to the map/reduce silliness below. I don't know that it leads anywhere - … [View More]

3 3

[Twisted-Python] Trying to proxy through multiple IPs
by Erik Wickstrom Nov. 4, 2008

Nov. 4, 2008

Hi all, I'm trying to write a proxy server that accepts connections on multiple ports, and depending on the port, using a different IP for the outgoing connection. My code works except that adding additional reactor.listenTCP(...)s overwrite the IP of the previous listenTCPs. So the proxy accepts connections on all the desired ports, but only uses the last IP address for outgoing connections. The ip (bindIP) is somehow being overwritten. Based on advice from IRC (exarkun --thanks!), I've … [View More]tried a couple attempts at using closures to solve the problem, but non of my implementations have done the trick. Can anyone see any other changes that might get this working? # Also pasted at http://dpaste.com/88623/ incase of email garbling... #from twisted.web import proxy, http import urlparse from urllib import quote as urlquote from twisted.internet import reactor from twisted.internet.protocol import ClientFactory from twisted.web.resource import Resource from twisted.web.server import NOT_DONE_YET from twisted.web.http import HTTPClient, Request, HTTPChannel from twisted.web.proxy import ProxyClient, ProxyClientFactory from twisted.web import http from twisted.internet import reactor from twisted.python import log import sys log.startLogging(sys.stdout) class ProxyRequest(Request): """ Used by Proxy to implement a simple web proxy. @ivar reactor: the reactor used to create connections. @type reactor: object providing L{twisted.internet.interfaces.IReactorTCP} """ protocols = {'http': ProxyClientFactory} ports = {'http': 80} def __init__(self, channel, queued, reactor=reactor): Request.__init__(self, channel, queued) self.reactor = reactor def process(self): parsed = urlparse.urlparse(self.uri) protocol = parsed[0] host = parsed[1] port = self.ports[protocol] if ':' in host: host, port = host.split(':') port = int(port) rest = urlparse.urlunparse(('', '') + parsed[2:]) if not rest: rest = rest + '/' class_ = self.protocols[protocol] headers = self.getAllHeaders().copy() if 'host' not in headers: headers['host'] = host self.content.seek(0, 0) s = self.content.read() clientFactory = class_(self.method, rest, self.clientproto, headers, s, self) self.reactor.connectTCP(host, port, clientFactory, bindAddress=(self.bindIP,0)) class Proxy(HTTPChannel): requestFactory = ProxyRequest class ProxyFactory(http.HTTPFactory): def class_factory(self, bindIP): def closure(ip): klass2 = ProxyRequest setattr(klass2, 'bindIP', ip) return klass2 klass = Proxy setattr(klass, 'requestFactory', closure(bindIP)) return klass def __init__(self, ip): http.HTTPFactory.__init__(self) self.ip = ip #self.protocol = proxy.Proxy self.protocol = self.class_factory(ip) reactor.listenTCP(8080, ProxyFactory('192.168.168.101'), interface="192.168.168.1") reactor.listenTCP(8081, ProxyFactory('192.168.168.102'), interface="192.168.168.1") reactor.run() ## Thanks! Erik [View Less]

3 2

Re: [Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones Nov. 4, 2008

Nov. 4, 2008

Hi again. Despite all the useful suggestions, I've just realized that I still can't complete what I was originally trying to do... I'm trying to make a Twisted service. In my twisted/plugins/myService.py file I have something like this: from zope.interface import implements from twisted.plugin import IPlugin from twisted.application.service import IServiceMaker from twisted.python import usage from my.application import myservice class Options(usage.Options): … [View More]

2 1

Re: [Twisted-Python] DeferredDict, defaultdict, map/reduce, silliness
by Terry Jones Nov. 4, 2008

Nov. 4, 2008

Hi Valentino >>>>> "Valentino" == Valentino Volonghi <dialtone(a)gmail.com> writes: Valentino> This is actually a very very nice pattern. I thought it was too cute not to post, though I didn't really know what to do with it. Something along the lines of map/reduce occurred to me quite late. I bet there are other nice things you could do, mainly along the lines of processing/bucketing a bunch of unpredictable input. One thing I'd (probably) change would be to not use … [View More]

1 0