Mailman 3 December 2009 - Twisted

[Twisted-Python] Syntactic sugar for deferreds
by Konrads Smelkovs Dec. 7, 2009

Dec. 7, 2009

Hi, One project I had a look at had this nice syntactic sugar for async function chaining: >>> event_one() | event_two() | event_three() I think this could be an interesting alternative to addCallback or yield. I propose the following syntax # get deferred with one () and two () chained as callbacks, equivalent to d=Deferred().addCallback(one).addCallback(two) >>> d = Deferred() | one | two # Equivalent to d=Deferred().addCallback(one).addErrback(two) >>> d = … [View More]

4 3

[Twisted-Python] A simple DeferredPool class
by Terry Jones Dec. 6, 2009

Dec. 6, 2009

I submitted a talk on Twisted Deferreds to the US PyCon about, but it was unfortunately rejected. The main point I'd planned to make was how nice the deferred mechanism is, as evidenced by the number of times I've wanted to do things that might be complex in other scenarios but which just fall out using deferreds. More importantly, you rarely need to use anything more than the basic building blocks found in t.i.defer. I like posting small self-contained examples of this sort of thing, so here … [View More]are another couple. If you know deferreds well I guess these will seem trivial. If not, you might find them valuable. Here's the situation I found myself in today. I'll use an example based on twisted.web. Suppose you have a twisted.web service running and you want to shut the service down. But for various reasons you don't want to interrupt any of the work that it has in progress. The currently outstanding work the service is doing is bound up in a set of deferreds that have not yet fired. E.g., a request has come in and the server called something that returns a deferred, attached a callback to it that will finish the request, and returned NOT_DONE_YET. Something like this: def render_GET(self, request): d = defer.maybeDeferred(someFunc) d.addCallback(_finish, request) return server.NOT_DONE_YET def _finish(self, result, request): request.finish() A the moment that you send the command to shut the service down, zero of more instances of d (above) may exist, and these will be in various stages of completion. Given this setup, how can you arrange to wait for all outstanding requests (if any) before shutting the service down? A simple solution is just to define a DeferredPool class that maintains a pool of deferreds which can provide you with a method to (at any time) obtain a deferred that will fire when/if the pool size next goes to zero. Here's what I wrote (untested): from twisted.internet import defer class DeferredPool(object): def __init__(self, initialContents=None): self._pool = set() self._waiting = [] if initialContents: for d in initialContents: self.add(d) def _fired(self, result, d): self._pool.remove(d) if not self._pool: waiting, self._waiting = self._waiting, [] for waiter in waiting: waiter.callback(None) return result def add(self, d): d.addBoth(self._fired, d) self._pool.add(d) return d def deferUntilEmpty(self, testImmediately=True): if testImmediately and not self._pool: return defer.succeed(None) else: d = defer.Deferred() self._waiting.append(d) return d In your server's startService method, you can do this: def startService(self): self.pool = DeferredPool() and the above render_GET code gets changed to look like this: def render_GET(self, request): d = defer.maybeDeferred(someFunc) d.addCallback(_finish, request) self.pool.add(d) return server.NOT_DONE_YET and in your stopService method: def stopService(self): d = self.pool.deferUntilEmpty() d.addCallback( whatever else you need to do ) return d And that's it. As usual with these simple deferred solutions, it's simple, it's general, and it's more widely useful than you might have initially planned. The DeferredPool class is a little reminiscent of DeferredList, in that the deferreds that are submitted each get a call/errback added to themselves that monitors the progress of the collection and which triggers the waiting deferreds (if any) once some condition is satisfied. But it's more dynamic, as the pool can grown and shrink while you're waiting, it can have zero or more waiters, it's long-lived as the pool can go to down zero size and come back up and have more waiters added to it, etc. And, as with all these nice tricks, the existence of the pool and its operation is totally transparent to the deferreds in use by the original code. Yes, you have to insert one call to add a deferred to the pool, but that's it. That's part of what I like the most about this sort of this: adding transparent call/errbacks into the chain of existing deferreds to do something no-one thought of initially, and knowing that unless you do something really dumb you'll have no noticeable effect on the operation of the original code. That's part of why I find Twisted's deferreds so elegant. I also wrote a somewhat more general (and slower) version of questionable utility. It allows you to pass a function when you ask for a deferred - and the deferred you get fires when your function returns True. Your function is called each time a deferred in the pool fires. If you don't pass a function, you get the above behavior (fire when the pool is empty). I wont post the code it here. If you want it, just ask. Comments welcome on all this, of course. I'm interested to hear how people would write tests for the above. Terry [View Less]

1 0

[Twisted-Python] AbstractDatagramProtocol implementation question
by Olivier Parisy Dec. 6, 2009

Dec. 6, 2009

I am studying the implementation of twisted UDP-related classes, and I do not understand the point of the "numPorts" attribute of AbstractDatagramProtocol. It is clearly designed to avoid multiple calls to startProtocol() from doStart(), but I fail in imagining a scenario where this may happen since, as the comment states, doStart() should only be called by makeConnection(), which has an explicit "assert self.transport == None" guard which, indeed, prevents multiple calls. So why bother … [View More]

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Dec. 6, 2009

Dec. 6, 2009

1 0

[Twisted-Python] Help installing twisted rev 27662, OSX10.6, Python 2.6.4
by Erik van Blokland Dec. 3, 2009

Dec. 3, 2009

Hullo, I'm trying to get the current svn version of twisted to run on a current python 2.6.4 (not the stock python, a newer one) on Mac OSX 10.6. > In file included from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/unicodeobject.h:4, > from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:85, > from twisted/runner/portmap.c:10: > /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: … [View More]stdarg.h: No such file or directory > lipo: can't figure out the architecture type of: /var/tmp//ccpMpUAs.out > error: command 'gcc-4.0' failed with exit status 1 The reference to 10.4 makes me go meh. All the versions and reports: # checking which gcc I have installed: [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% gcc i686-apple-darwin10-gcc-4.2.1: no input files # the python version [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% which python /Library/Frameworks/Python.framework/Versions/2.6/bin/python [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% python Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin # the twisted svn revision [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% svn up At revision 27662. # the full report [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% sudo python setup.py install running install running bdist_egg running egg_info writing requirements to Twisted.egg-info/requires.txt writing Twisted.egg-info/PKG-INFO writing top-level names to Twisted.egg-info/top_level.txt writing dependency_links to Twisted.egg-info/dependency_links.txt writing manifest file 'Twisted.egg-info/SOURCES.txt' installing library code to build/bdist.macosx-10.3-fat/egg running install_lib running build_py running build_ext gcc-4.0 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c conftest.c -o conftest.o gcc-4.0 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c conftest.c -o conftest.o conftest.c:1:23: error: sys/epoll.h: No such file or directory conftest.c:1:23: error: sys/epoll.h: No such file or directory lipo: can't figure out the architecture type of: /var/tmp//ccz08AYz.out building 'twisted.runner.portmap' extension gcc-4.0 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c twisted/runner/portmap.c -o build/temp.macosx-10.3-fat-2.6/twisted/runner/portmap.o In file included from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/unicodeobject.h:4, from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:85, from twisted/runner/portmap.c:10: /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory In file included from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/unicodeobject.h:4, from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:85, from twisted/runner/portmap.c:10: /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory lipo: can't figure out the architecture type of: /var/tmp//ccpMpUAs.out error: command 'gcc-4.0' failed with exit status 1 All pointers welcomed. Many thanks, Erik [View Less]

1 0

[Twisted-Python] deferToThread and thread ID
by masetto scarface Dec. 3, 2009

Dec. 3, 2009

Hi all, Last week i've started to study (and love :P) Twisted . Well, just for didactical purpose i've written a little script which download a web page (getPage) and calculate a long Fibonacci number within a deferToThread and it works perfectly. I would like to see the different threads id of the two operations (download a web page and the calculation of the number), always for didactical purposes. Is there any way to do this? Here is a piece of my script: ... d1 = getPage('http://www.… [View More]

2 1

[Twisted-Python] Lore to Sphinx Conversion Progress Report
by Kevin Horn Dec. 3, 2009

Dec. 3, 2009

The previous thread was getting a little long, so I thought I'd start a new one... I haven't made as much headway on this as I would have liked, but I'm just going to blame the US Thanksgiving holiday and hope no one looks too closely at the dates of the previous emails. ;) Here's what I've done lately: - The proposal [1] has been updated, with a number of minor spelling formatting fixes, and a detailed timeline for each phase of the project. Please give feedback if you see anything missing … [View More]

4 10

Re: [Twisted-Python] ANN: Twisted 9.0.0
by exarkun＠twistedmatrix.com Dec. 2, 2009

Dec. 2, 2009

On 12:18 am, tjreedy(a)udel.edu wrote: >Christopher Armstrong wrote: >>= Twisted 9.0.0 = >> >>I'm happy to announce Twisted 9, the first (and last) release of >>Twisted in 2009. The previous release was Twisted 8.2 in December of >>2008. Given that, a lot has changed! >> >>This release supports Python 2.3 through Python 2.6, though it is the >>last one that will support Python 2.3. The next release will support >>only Python 2.4 and above.… [View More]

2 1

[Twisted-Python] ANN: Twisted 9.0.0
by Christopher Armstrong Dec. 2, 2009

Dec. 2, 2009

= Twisted 9.0.0 = I'm happy to announce Twisted 9, the first (and last) release of Twisted in 2009. The previous release was Twisted 8.2 in December of 2008. Given that, a lot has changed! This release supports Python 2.3 through Python 2.6, though it is the last one that will support Python 2.3. The next release will support only Python 2.4 and above. Twisted: the framework of the future! You can download the new release at our web site, http://twistedmatrix.com/ There were around … [View More]

1 0

[Twisted-Python] is there way to baypass value limit of 64kb in twisted.protocols.amp
by MārisR Dec. 1, 2009

Dec. 1, 2009

Hello! I'm currently trying to figure out, how to bypass amp value limit of 64kb. First tought was to split string into chunks <64kb, then join before processing. Problem is, how to dinamicaly generate amp.Command argument list. Maybe there is other well known method how to achieve this? Thank You!

3 2