Mailman 3 December 2009 - Twisted

[Twisted-Python] Syntactic sugar for deferreds
by Konrads Smelkovs 07 Dec '09

07 Dec '09

Hi, One project I had a look at had this nice syntactic sugar for async function chaining: >>> event_one() | event_two() | event_three() I think this could be an interesting alternative to addCallback or yield. I propose the following syntax # get deferred with one () and two () chained as callbacks, equivalent to d=Deferred().addCallback(one).addCallback(two) >>> d = Deferred() | one | two # Equivalent to d=Deferred().addCallback(one).addErrback(two) >>> d = Deferred() | one ^ two and finally: # d=Deferred().addBoth(one,two) >>> d = Deferred() & (one,two) This could make use of currying/ -- Konrads Smelkovs Applied IT sorcery.

4 3

[Twisted-Python] A simple DeferredPool class
by Terry Jones 06 Dec '09

06 Dec '09

I submitted a talk on Twisted Deferreds to the US PyCon about, but it was unfortunately rejected. The main point I'd planned to make was how nice the deferred mechanism is, as evidenced by the number of times I've wanted to do things that might be complex in other scenarios but which just fall out using deferreds. More importantly, you rarely need to use anything more than the basic building blocks found in t.i.defer. I like posting small self-contained examples of this sort of thing, so here are another couple. If you know deferreds well I guess these will seem trivial. If not, you might find them valuable. Here's the situation I found myself in today. I'll use an example based on twisted.web. Suppose you have a twisted.web service running and you want to shut the service down. But for various reasons you don't want to interrupt any of the work that it has in progress. The currently outstanding work the service is doing is bound up in a set of deferreds that have not yet fired. E.g., a request has come in and the server called something that returns a deferred, attached a callback to it that will finish the request, and returned NOT_DONE_YET. Something like this: def render_GET(self, request): d = defer.maybeDeferred(someFunc) d.addCallback(_finish, request) return server.NOT_DONE_YET def _finish(self, result, request): request.finish() A the moment that you send the command to shut the service down, zero of more instances of d (above) may exist, and these will be in various stages of completion. Given this setup, how can you arrange to wait for all outstanding requests (if any) before shutting the service down? A simple solution is just to define a DeferredPool class that maintains a pool of deferreds which can provide you with a method to (at any time) obtain a deferred that will fire when/if the pool size next goes to zero. Here's what I wrote (untested): from twisted.internet import defer class DeferredPool(object): def __init__(self, initialContents=None): self._pool = set() self._waiting = [] if initialContents: for d in initialContents: self.add(d) def _fired(self, result, d): self._pool.remove(d) if not self._pool: waiting, self._waiting = self._waiting, [] for waiter in waiting: waiter.callback(None) return result def add(self, d): d.addBoth(self._fired, d) self._pool.add(d) return d def deferUntilEmpty(self, testImmediately=True): if testImmediately and not self._pool: return defer.succeed(None) else: d = defer.Deferred() self._waiting.append(d) return d In your server's startService method, you can do this: def startService(self): self.pool = DeferredPool() and the above render_GET code gets changed to look like this: def render_GET(self, request): d = defer.maybeDeferred(someFunc) d.addCallback(_finish, request) self.pool.add(d) return server.NOT_DONE_YET and in your stopService method: def stopService(self): d = self.pool.deferUntilEmpty() d.addCallback( whatever else you need to do ) return d And that's it. As usual with these simple deferred solutions, it's simple, it's general, and it's more widely useful than you might have initially planned. The DeferredPool class is a little reminiscent of DeferredList, in that the deferreds that are submitted each get a call/errback added to themselves that monitors the progress of the collection and which triggers the waiting deferreds (if any) once some condition is satisfied. But it's more dynamic, as the pool can grown and shrink while you're waiting, it can have zero or more waiters, it's long-lived as the pool can go to down zero size and come back up and have more waiters added to it, etc. And, as with all these nice tricks, the existence of the pool and its operation is totally transparent to the deferreds in use by the original code. Yes, you have to insert one call to add a deferred to the pool, but that's it. That's part of what I like the most about this sort of this: adding transparent call/errbacks into the chain of existing deferreds to do something no-one thought of initially, and knowing that unless you do something really dumb you'll have no noticeable effect on the operation of the original code. That's part of why I find Twisted's deferreds so elegant. I also wrote a somewhat more general (and slower) version of questionable utility. It allows you to pass a function when you ask for a deferred - and the deferred you get fires when your function returns True. Your function is called each time a deferred in the pool fires. If you don't pass a function, you get the above behavior (fire when the pool is empty). I wont post the code it here. If you want it, just ask. Comments welcome on all this, of course. I'm interested to hear how people would write tests for the above. Terry

1 0

[Twisted-Python] AbstractDatagramProtocol implementation question
by Olivier Parisy 06 Dec '09

06 Dec '09

I am studying the implementation of twisted UDP-related classes, and I do not understand the point of the "numPorts" attribute of AbstractDatagramProtocol. It is clearly designed to avoid multiple calls to startProtocol() from doStart(), but I fail in imagining a scenario where this may happen since, as the comment states, doStart() should only be called by makeConnection(), which has an explicit "assert self.transport == None" guard which, indeed, prevents multiple calls. So why bother to count calls? Moreover, makeConnection() could be called multiple times on the same DatagramProtocol instance if it was "bound" to multiple ports (multiple calls to listenUDP), which can be an interesting scenario but is prevented by : - the above assertion, - the fact that AbstractDatagramProtocol.transport is not a collection, so only the last Port would be available anyway. Is my reasoning wrong or incomplete? If not, couldn't we simplify AbstractDatagramProtocol implementation by removing numPorts? Two other small remarks regarding documentation / understandability : - The note "This does not actually implement IListeningPort" on udp.Port lead me to think that IReactorUDP.listenUDP should not be declared as returning an object implementing IListeningPort in the first place (confused me!). - I had to read the code to understand that a "transport" and a "port" are exactly the same thing in the UDP implementation. The tutorial and the documentation were not clear about this point, so I did not make the link between the "self.transport" available in DatagramProtocol and the value returned by listenUDP (confused me!). Regards, Olivier.

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 06 Dec '09

06 Dec '09

1 0

[Twisted-Python] Help installing twisted rev 27662, OSX10.6, Python 2.6.4
by Erik van Blokland 03 Dec '09

03 Dec '09

Hullo, I'm trying to get the current svn version of twisted to run on a current python 2.6.4 (not the stock python, a newer one) on Mac OSX 10.6. > In file included from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/unicodeobject.h:4, > from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:85, > from twisted/runner/portmap.c:10: > /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory > lipo: can't figure out the architecture type of: /var/tmp//ccpMpUAs.out > error: command 'gcc-4.0' failed with exit status 1 The reference to 10.4 makes me go meh. All the versions and reports: # checking which gcc I have installed: [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% gcc i686-apple-darwin10-gcc-4.2.1: no input files # the python version [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% which python /Library/Frameworks/Python.framework/Versions/2.6/bin/python [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% python Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin # the twisted svn revision [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% svn up At revision 27662. # the full report [ebook:~/Develop/svn.twistedmatrix.com/trunk] erik% sudo python setup.py install running install running bdist_egg running egg_info writing requirements to Twisted.egg-info/requires.txt writing Twisted.egg-info/PKG-INFO writing top-level names to Twisted.egg-info/top_level.txt writing dependency_links to Twisted.egg-info/dependency_links.txt writing manifest file 'Twisted.egg-info/SOURCES.txt' installing library code to build/bdist.macosx-10.3-fat/egg running install_lib running build_py running build_ext gcc-4.0 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c conftest.c -o conftest.o gcc-4.0 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c conftest.c -o conftest.o conftest.c:1:23: error: sys/epoll.h: No such file or directory conftest.c:1:23: error: sys/epoll.h: No such file or directory lipo: can't figure out the architecture type of: /var/tmp//ccz08AYz.out building 'twisted.runner.portmap' extension gcc-4.0 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c twisted/runner/portmap.c -o build/temp.macosx-10.3-fat-2.6/twisted/runner/portmap.o In file included from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/unicodeobject.h:4, from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:85, from twisted/runner/portmap.c:10: /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory In file included from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/unicodeobject.h:4, from /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:85, from twisted/runner/portmap.c:10: /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory lipo: can't figure out the architecture type of: /var/tmp//ccpMpUAs.out error: command 'gcc-4.0' failed with exit status 1 All pointers welcomed. Many thanks, Erik

1 0

[Twisted-Python] deferToThread and thread ID
by masetto scarface 03 Dec '09

03 Dec '09

Hi all, Last week i've started to study (and love :P) Twisted . Well, just for didactical purpose i've written a little script which download a web page (getPage) and calculate a long Fibonacci number within a deferToThread and it works perfectly. I would like to see the different threads id of the two operations (download a web page and the calculation of the number), always for didactical purposes. Is there any way to do this? Here is a piece of my script: ... d1 = getPage('http://www.google.com/') d1.addCallback(printContents) d = threads.deferToThread(largeFibonacciNumber) d.addCallback(fibonacciCallback) if __name__ == '__main__': run() reactor.run() Is it possibile to print within these functions their thread id (PID is always the same :P)? Something like: ... Executing printContents func PID: 1234 Thread ID: x ... Executing Fibonacci func PID: 1234 Thread ID: y I suppose it works in this way, isn't it? Thanks --- Masetto

2 1

[Twisted-Python] Lore to Sphinx Conversion Progress Report
by Kevin Horn 03 Dec '09

03 Dec '09

The previous thread was getting a little long, so I thought I'd start a new one... I haven't made as much headway on this as I would have liked, but I'm just going to blame the US Thanksgiving holiday and hope no one looks too closely at the dates of the previous emails. ;) Here's what I've done lately: - The proposal [1] has been updated, with a number of minor spelling formatting fixes, and a detailed timeline for each phase of the project. Please give feedback if you see anything missing from the timelines, anything you think should be done in a different order, or any other unaddressed issues. - I've done a small amount of work on the lore2sphinx tool, and the new output of the conversion process is now online [2]. Not a lot changes, the only obviosu thing is that <img> tags are now handled. The rest has been mostly refactoring and generally making the code easier to work with. - I also did a bit of digging around in the Divmod SVN repository, and it looks like Nevow, Mantissa, and Epsilon are the only projects that have Lore source docs. Can anyone out there verify that this is correct? Did I miss anything? I'll try to send out a progress report like this every week or so in order to avoid people thinking that I've lost interest or gotten hit by a bus. Thanks for "listening"! Kevin Horn [1] http://twistedsphinx.funsize.net/proposal.html [2] http://twistedsphinx.funsize.net/index.html

4 10

Re: [Twisted-Python] ANN: Twisted 9.0.0
by exarkun＠twistedmatrix.com 03 Dec '09

03 Dec '09

On 12:18 am, tjreedy(a)udel.edu wrote: >Christopher Armstrong wrote: >>= Twisted 9.0.0 = >> >>I'm happy to announce Twisted 9, the first (and last) release of >>Twisted in 2009. The previous release was Twisted 8.2 in December of >>2008. Given that, a lot has changed! >> >>This release supports Python 2.3 through Python 2.6, though it is the >>last one that will support Python 2.3. The next release will support >>only Python 2.4 and above. Twisted: the framework of the future! > >Not unless it supports 3.1+. Is that in the cards (tickets)? Somewhat. A description of the plan on stackoverflow: http://bit.ly/6hWqYU A message with some ticket links from a thread on the twisted-python mailing list: http://bit.ly/8csFSa Jean-Paul

2 1

[Twisted-Python] ANN: Twisted 9.0.0
by Christopher Armstrong 02 Dec '09

02 Dec '09

= Twisted 9.0.0 = I'm happy to announce Twisted 9, the first (and last) release of Twisted in 2009. The previous release was Twisted 8.2 in December of 2008. Given that, a lot has changed! This release supports Python 2.3 through Python 2.6, though it is the last one that will support Python 2.3. The next release will support only Python 2.4 and above. Twisted: the framework of the future! You can download the new release at our web site, http://twistedmatrix.com/ There were around 285 tickets resolved in this release. The full list of changes is available here: http://twistedmatrix.com/trac/browser/tags/releases/twisted-9.0.0/NEWS?form… It's quite a huge list of changes spanning almost all of the Twisted projects, so here are some of the more exciting changes: In the core: - The Windows IOCP reactor now supports SSL. - The memcache protocol implementation got some nice new features. In Twisted Web: - There's a new HTTP client API and protocol implementation, starting at twisted.web.client.Agent. It's still pretty low-level, but much more flexible than the old API. - There were many improvements to the WSGI support. In Twisted Conch: - PyASN1 is now used to parse SSH keys (which means you now need to install it to use Conch). - SFTP servers (especially on Windows) now behave a lot better. In Twisted Mail: - The IMAP server and client protocol implementations had many fixes. For example, SASL PLAIN credentials now work. In Twisted Words: - XMPP clients now support the ANONYMOUS SASL authentication type. - The IRC protocol implementations had many fixes. And a lot more. = What is Twisted? = >From the web site: Twisted is an event-driven networking engine written in Python and licensed under the MIT license. See the FAQ for commonly asked questions about Twisted. http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions If you want to get started with Twisted, the first thing you should do is read the Twisted Core Documentation. http://twistedmatrix.com/projects/core/documentation/howto/index.html Twisted projects variously support TCP, UDP, SSL/TLS, multicast, Unix sockets, a large number of protocols (including HTTP, NNTP, IMAP, SSH, IRC, FTP, and others), and much more. Enjoy! -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/

1 0

[Twisted-Python] is there way to baypass value limit of 64kb in twisted.protocols.amp
by MārisR 01 Dec '09

01 Dec '09

Hello! I'm currently trying to figure out, how to bypass amp value limit of 64kb. First tought was to split string into chunks <64kb, then join before processing. Problem is, how to dinamicaly generate amp.Command argument list. Maybe there is other well known method how to achieve this? Thank You!

3 2