Mailman 3 June 2012 - Twisted

[Twisted-Python] need to make imap server replies
by Ali Ghanavatian 26 Jun '12

26 Jun '12

hello there, here's the code i got: http://pastebin.ubuntu.com/1058070/ it's implementing an imap server, problem is it does not retrieve emails. mail client can capture new emails only once and after that, it should disconnect and reconnect again to receive new emails (again only once). any idea what's wrong here? thanks in advance -- Sincerely A. Ghanavatian <http://www.google.com/profiles/ghanavatian.ali>

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 24 Jun '12

24 Jun '12

1 0

[Twisted-Python] Speed of rendering?
by Peter Westlake 22 Jun '12

22 Jun '12

How fast is rendering in nevow? I have a page which is mostly a large table with a couple of hundred rows, each containing a form. The generated HTML is about 500 KB. Leaving aside the question of whether this is a good design or not, how long would you expect it to take? I'm interested specifically in what happens between the end of beforeRender and the request being completed. It takes about a minute and a quarter. Is that normal, or is there a delay in my code that I haven't found yet? Thanks, Peter.

2 3

[Twisted-Python] AMP WebSockets bridging
by Laurens Van Houtven 20 Jun '12

20 Jun '12

Hi, I'm trying to get browsers to communicate with an AMP API. Right now I'm trying to do that using WebSockets with Corbin SImpson's txWS. Eventually I'll try to write a Twisted implementation of SockJS, which is basically portable websockets all the way down to IE6. The issue is that while AMP is inherently a binary protocol, WebSockets transfer text (well, that's a lie: there's a binary version, but it's even less widely supported than WebSockets themselves -- and SockJS doesn't support it at all, although it's on the roadmap). So, I need an alternative, preferably 7-bit, serialization for AMP boxes. My first attempt at that serialization is JSON -- widely supported in browsers, and should map pretty closely to what AMP can do: - AMPLIsts are objects - ListOfs are arrays - numeric types and strings are more or less natively supported (except JSON is always text, never bytes) - booleans are booleans Right now I have something that almost works, but is really a crude, cheap, untested hack. I'm wondering if there's a better integration point. """ AMP over WebSockets support. """ import json import txws from twisted.internet import defer, protocol from twisted.python import log class Bridge(protocol.Protocol): """ Two-way AMP over WebSockets bridge. """ def __init__(self, amp): self._amp = amp amp.startReceivingBoxes(self) amp.boxSender = self def sendBox(self, box): """ Sends a box over the WebSocket as JSON. """ log.msg("Sending box: {}".format(box)) self.transport.write(json.dumps(box)) def jsonObjectReceived(self, obj): """ Hands the JSON object (dict) over to ampBoxReceived. """ log.msg("JSON object received: {}".format(obj)) self._amp.boxReceiver.ampBoxReceived(obj) def dataReceived(self, data): """ Calls jsonObjectReceived. This assumes that JSON objects will always arrive as 1 chunk. """ self.jsonObjectReceived(json.loads(data)) class BridgeFactory(protocol.Factory): """ A factory for AMP over WebSockets bridges. """ def __init__(self, ampFactory): self._ampFactory = ampFactory def buildProtocol(self, addr): """ Builds a bridge and associates it with an AMP protocol instance. """ return Bridge(self._ampFactory.buildProtocol(addr)) def makeFactory(ampFactory): """ Makes a WebSocket factory that bridges AMP messages. """ return txws.WebSocketFactory(BridgeFactory(ampFactory)) An issue I'm running into is an AMP ListOf. With the above code, ListOf still gets translated to a string, and obviously I want it to be a list. When that ListOf has actual data in it, I get the binary representation. The other issues that I predict will happen but haven't ran into yet (because I don't use that functionality) are: - booleans -- expects "True" or "False" but gets True or False (true and false in JSON) -- can be fixed both on the client and the server side, not sure where the optimal place would be - AMPList -- same reason as ListOf ints, floats… will magically work, but it's a bit ugly: the only reason it works is that the factory for them is int and float, and, well, int(5) == 5 == int("5"), so it doesn't actually matter if it's already an int or a str. In my Command implementation, I'm returning {"k": anAxiomQuery} -- so it's iterable but not a list. I can write a custom JSON encoder that can take arbitrary iterables, of course. cheers lvh

6 13

[Twisted-Python] Python 3 support in Twisted: Request for relaxation of coverage requirements
by Vladimir Perić 19 Jun '12

19 Jun '12

Hello everyone, as some of you may be aware, I was accepted as a Google Summer of Code student, with the project of porting Twisted to Python 3 (or getting as close as possible)[1]. Now, there are basically two strategies: maintaining py2 code and using 2to3 when installing under py3, or maintaining a codebase which is compatible with both 2 and 3 (a third strategy is to have two codebases, but I feel this would introduce a too high maintenance burden).[2] What both these approaches have in common, is that it is advantageous (and mandatory if not using 2to3) to update the syntax to more modern idioms (eg. exception syntax, uses of apply and has_key, the print statement...). And therein lies my problem: Twisted policy is that all code should be tested, and tests should be added to modified, untested code... and I have to make a lot of trivial syntax changes to a lot of code, some of which is untested. Now, I agree that tests are very important, and in general I'm for the 100% coverage policy Twisted is trying to approach, but I feel that this is beyond the scope of my project and, more importantly, would slow my progress considerably. As such, I would like to request permission for the coverage requirements to be relaxed for specific Python 3 changes. As mentioned above, this would allow me to progress much faster and concentrate on the bigger incompatibilities. Of course, I would still write tests for more complex code, if and where required. Furthermore, I propose to use the 2to3 tool to make these conversions, which relies on analysis of the underlying code - as such, it is much better than any grep-related solution and much less prone to errors. It is also very widely used, so I consider it a very robust solution. Still, I realize this would be quite a big breach of policy, so I would like to make a case for each change individually. This list is not exhaustive, merely what I have encountered so far. 1) print statement Probably the most noticeable difference, the print statement is a function in Python 3. Now, if we opt for a single, compatible codebase, all print statements should be converted to functions. Luckily, 2.6 can import it from __future__ so this is not a problem. There's more than 1000 print statements in Twisted code and even just checking if each is covered would be a daunting job. Even if no other exception is permitted, I think this one makes a lot of sense as it's very hard to do anything wrong here. 2) has_key method The issue that started this (see ticket 4053); the has_key method on dicts has been changed to the builtin keyword "in" ("key in dict" instead of dict.has_key(key)). Now, exarkun disliked my automatic change, so I split the work into tested and untested uses (the tested are fixed) but there's still about a hundred remaining. Now, a concern here is that some Twisted classes could also be defining has_key methods, but all three such classes also provide an equivalent __contains__ method so this cannot be a problem (plus, I believe the has_key methods should be deprecated but that's a separate issue). In the end, this is a purely syntatic change and I do not feel like writing tests everywhere around Twisted. 3) apply To quote Lennart Regebro, "The Python 2 builtin apply() has been removed in Python 3. It’s used to call a function, but since you can call the function directly it serves no purpose and has been deprecated since Python 2.3." I've replaced those covered by tests, but there's still some 23 untested uses left, most of them in spread/ui/tkutil, and I would *really* prefer not to write tests for that. 4) Exceptions The exceptions syntax has changed to be less ambiguous, and the new "except Exception as e" syntax is available in 2.6 and up. This is a trivial syntax change (change a comma to "as") and can be performed automatically on the approx. 230-240 uses. I'm not sure how many of these are already tested, though. Some other things also changed about Exceptions, but these I will address separately. That's as far as I got so far. As I said above, most of these are simple syntax changes which are hard to get wrong. If we opt for the 2to3 route, all of those would be converted automatically but it is still preferable to get the code in a more "modern" state. Of course, if not using 2to3, all of the above changes will have to be made. Waiving the tests requirement would allow me to progress much faster, and, at the end of the day, my task is to add Python 3 support, not improve test coverage to 100%. Thank you all in advance for any input (or help!). [1] If you are interested, the best way of following my progress is probably to subscribe to my blog (which is also syndicated on Planted Twisted): vperic.blogspot.com [2] The merits of each approach can be discussed elsewhere, this is not the goal of this e-mail. As of right now, there is sufficient work remaining that will be equally beneficial no matter which one is chosen, and this is the stuff I'm focusing on now. -- Vladimir Perić

6 9

[Twisted-Python] ANN: Foolscap-0.6.4 released
by Brian Warner 19 Jun '12

19 Jun '12

Just a brief note, I released Foolscap-0.6.4 tonight. This contains a handful of small bugfixes and cleanups, gathered over the last 5 months. The most significant change was that the "git-foolscap" tools were cleaned up. These live in doc/examples/ , and let you configure remote access to your Git repositories that uses a FURL instead of e.g. a locked-down ssh shell. Tarballs are available from the usual places: PyPI: http://pypi.python.org/pypi/foolscap/0.6.4 github: https://github.com/warner/foolscap/ trac: http://foolscap.lothar.com/trac/ Tarballs are signed with the Foolscap Release Signing Key (0xAF1B4A2A), fingerprint = 7341 45FE BE70 E5D0 59C7 EDCD 3E86 92AF AF1B 4A2A. Full release notes attached below. have a groovy day, -Brian * Release 0.6.4 (18-Jun-2012) ** Minor Changes The unreliable 'extras_require' property in setup.py, which allowed other python programs to declare a dependency on foolscap's "secure_connections" feature, was removed. See README.packagers for alternate instructions. (#174) 'flogtool' log-dumping commands (dump, tail, web-viewer) now accept a consistent --timestamps= argument to control how event times are displayed (UTC, local, seconds-since-epoch, etc). (#192, #193) Certain invalid "location" strings (accepted by Tub.setLocation and put into FURLs) are rejected earlier, and with better error messages. The error message produced when 'flogtool dump' is given a FURL-file (instead of an event log file) has been improved. The Incident Gatherer will tolerate incident-file errors better, fetching remaining incidents instead of halting. (#190) The git-over-foolscap tools were cleaned up, and the documentation was brought into line with the implementation. (#197) Other minor bugs were fixed: #179, #191, #194, #195, #196

1 0

[Twisted-Python] Solaris 11 Build Slave
by Justin Venus 17 Jun '12

17 Jun '12

I am in the process of setting up a build Solaris 11 build slave for twisted (currently at the bzr checkout phase). I am interested in having it join the other build slaves. My long term goal for adding this build slave is to help resolve a low priority ticket[1] that I am interested in seeing completed. Thank you, -- Justin Venus justin.venus(a)gmail.com 1) http://twistedmatrix.com/trac/ticket/5538

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 17 Jun '12

17 Jun '12

1 0

Re: [Twisted-Python] AMP WebSockets bridging
by Tom Sheffler 16 Jun '12

16 Jun '12

> > On Jun 15, 2012, at 10:13 AM, Laurens Van Houtven wrote: > > > Hi, > > > > I'm trying to get browsers to communicate with an AMP API. > > This is also what <https://launchpad.net/frack> is doing. I hope that it > becomes a common enough practice that there's a library for it :). > > > The issue is that while AMP is inherently a binary protocol, WebSockets > transfer text (well, that's a lie: there's a binary version, but it's even > less widely supported than WebSockets themselves -- and SockJS doesn't > support it at all, although it's on the roadmap). So, I need an > alternative, preferably 7-bit, serialization for AMP boxes. > > Did you immediately discard the obvious solution of "base64 everything"? :) I had success simply base64'ing binary data. And I believe encoding that is safe for binary data, but I may be wrong. Use this: data = binascii.b2a_base64(bindata).strip()

1 0

[Twisted-Python] Quan Nguyen: Twisted Python questions
by Quan Nguyen 13 Jun '12

13 Jun '12

Hello, I am new to Twisted. I am extending the twisted Echo server example by sending the data that the server receives from telnet into a child process that processes the data. The function outReceived in censoringProtocol is called when the child writes the result and the result is received. The process of passing data from parent to child and back to parent is done by having a while loop of stdin.readline() and stdout.write() in the child process. Specifically, I generate a deferred object each time I write to the child using the function sendData (which then calls writeToChild) and use a queue to manage the deferred objects. I add a callBack (printData) that will broadcast the result to all instances of Echo when outReceived is called. I have two questions: 1/ Is this the right approach to communicate with the child process in order to achieve what I want? Are there any gotcha's? 2/ This server will be exposed to high loads. There will be many instances of Echo sending data to the child at a high rate. Are there any limitations that I have to watch for, such as the limit on child process's pipe sizes or anything else? My code is down below, as well as in this link: https://gist.github.com/2920032 Thank You for your time. Quan Nguyen from sys import executable from os import environ from twisted.internet import reactor, defer from twisted.internet.protocol import Protocol from twisted.internet.protocol import Factory from twisted.internet import protocol from twisted.protocols.basic import LineReceiver import sys import time import Queue import os implementation = """ import os import time import sys import string def censor(): BW = ["sfs", "fsdfwe", "tryt", "sfsvx"] while (True): a = sys.stdin.readline() if a: for bannedWord in BW: a = string.replace(a, bannedWord , '<banned>') sys.stdout.write(a) sys.stdout.flush() """ class Echo(LineReceiver): def lineReceived(self, data): d = self.factory.process.sendData(data+'\r\n') #reactor.callLater(3,d.addCallback, printData) d.addCallback(printData) prot.transport.write(data) def createProcess(): pp = censoringProtocol() Process = reactor.spawnProcess(pp, executable, [executable, "-c", implementation], env=environ, childFDs = {0:"w", 1:"r", 2:"r"}) pp.process = Process return pp class EchoFactory(Factory): protocol = Echo telnetConnectionList = [] process = None deferQ = None def __init__(self): EchoFactory.process = createProcess() EchoFactory.deferQ = Queue.Queue() def buildProtocol (self, addr): newProtocol = Factory.buildProtocol(self, addr) EchoFactory.telnetConnectionList.append(newProtocol) return newProtocol class censoringProtocol(protocol.ProcessProtocol): def __init__(self): self.process = None def connectionMade(self): print "connectionMade!" def outReceived(self, data): EchoFactory.deferQ.get().callback(data) def sendData(self,data): d = defer.Deferred() EchoFactory.deferQ.put(d) self.process.writeToChild(0, data) return d reactor.listenTCP(11111, EchoFactory()) print 'in parent', os.getpid() reactor.run()

2 1