Mailman 3 November 2015 - Twisted

[Twisted-Python] Multiple reactors, connecting to self, or other solution?
by Oon-Ee Ng 20 Nov '15

20 Nov '15

Based on my reading/searching, multiple reactors in the same process (even in multiple threads) is pretty much a no-go because twisted.internet.reactor is a global singleton. I'm also unable to find any information about connecting to self (for example, to send messages from one reactor to itself). The situation I find myself in is that of message-passing between a data-access component of a process (separate thread) and a GUI component. In this case, something like blinker works fine. However I want to replace blinker with twisted message-passing, because I want the data-access component to be accessible by different GUI components, either on the same machine or from a different machine. Or from the same process (default behaviour). If I'm able to somehow either run multiple reactors or connect-to-self, then I only need to maintain one API (a twisted API) for this communication, which would be identical between the one-process, multi-process, and multi-machine case. As an aside - I want to avoid restricting myself to a multi-process implementation as that would mean I can't write for iOs (no multi-processing allowed).

4 8

[Twisted-Python] Implementing Postfix Inet Policy Check Client
by Tom Boland 20 Nov '15

20 Nov '15

Greetings all. This may be a very basic question. I'm hoping to implement a postfix policy check client in twisted. It's a simple protocol. You send newline separated key value pairs like: recipient=email(a)ddr.ess sender=another(a)ddr.ess size=1024 helo_name=mail.server etc.. you terminate the request with an additional newline. The response comes back like action=OK You can send mutliple requests in the same connection. What I'm envisaging is a module that can be used to provide a deferred request/response pairing to my calling application. The module class will manage the single connection to the postfix policy daemon (I'm actually going to have persistent connections to a few daemons), and reconnect when necessary etc. Any requests will return a deferred that I can add callbacks to. How would you design this with twisted? I can easily envisage a way of using a clientfactory to instantiate separate connections for each request/response, but actually being able to simply send a request and receive the single response for that request is something I'm struggling to do within a LineReceiver instance (for instance). Would the twisted.protocols.amp module help given that I can't change the server-side protocol? Any advice much appreciated! Thanks. Tom.

3 5

[Twisted-Python] Migration to Git / GitHub
by Adi Roiban 18 Nov '15

18 Nov '15

Hi, So I am bringing this to the list for a greater audience and to reach all borders and timezones :) I think that most people agree that Twisted should get rid of SVN and move to Git. General rules by Glyph as they were sent to me :) Development can't stop, the website can't go down, and we can't lose any data. If you have a plan that migrates absolutely everything to github, including all of our issues to github issues, and all of our review queue stuff to github PRs, that is fine. You cannot, however, just push everything to github and delete the ticket database and all the outstanding branches and just say "okay everybody file github issues now". There has to be clear communication about what a developer who shows up on any given day to work on a Twisted ticket should do. Current requirements from Glyph: - be able to accept PRs on github.com - host code primarily on github - make sure all the same committers still have access (at least active ones) - make sure the website doesn't go down - break as little functionality as possible (kenaan, highscores, etc) - communicate clearly to contributors what they have to do in order to work on Twisted in every step of the process ------ Also from Glyph there are lots of "nice to have" things like it would be nice to have people authenticate to twistedmatrix.com via github so we can get rid of our terrible auth database and so they have one set of credentials for working It would be nice if we could automatically sync any relevant information between PRs and issues ----------- I would prefer to do baby steps and as a start just have the main repo in git hosted by github.com. Using GitHub it will force us (for the better or for the worse) to rethink the infrastructure using web hooks... and for "modern" hosting platforms. Once we have webhooks we should be able to migrate to any other provider... so it should be for the better Also, we need to migrate to GitHub as this was already agreed (one year ago) ... and if we re-start the conversation regarding the hosting platform, we are back on point 0 and still on SVN. We don't plan to migrate to GitHub Issues / GitHub Wiki / GitHub Pages ------------- So... if you have anything to comment regarding the git / github.com migration please send your feedback. Later we will announce the plan , break it into small task and start working on them. Cheers -- Adi Roiban

7 11

[Twisted-Python] Twisted 15.5 Prerelease 1 Announcement
by Amber "Hawkie" Brown 18 Nov '15

18 Nov '15

Hello everyone! Hot off the presses comes Twisted 15.5.0pre1, the prerelease of Twisted 15.5, which has been described by some as "totally radical" and "off the wall". In this release: - Python 3.5 support on POSIX was added, and Python 2.6 support was dropped. - More than nine additional modules have been ported to Python 3, ranging from Twisted Web's Agent and downloadPage, twisted.python.logfile, and many others, as well as... - twistd is ported to Python 3, and its first plugin, web, is ported. - twisted.python.url, a new URL/IRI abstraction, has been introduced to answer the question "just what IS a URL" in Twisted, once and for all. - NPN and ALPN support has been added to Twisted's TLS implementation, paving the way for HTTP/2. - Conch now supports the DH group14-sha1 and group-exchange-sha256 key exchange algorithms, as well as hmac-sha2-256 and hmac-sha2-512 MAC algorithms. Conch also works nicer with newer OpenSSH implementations. - Twisted's IRC support now has a sendCommand() method, which enables the use of sending messages with tags. - 55+ closed tickets overall. As usual, it's available for download -- go here (https://twistedmatrix.com/Releases/pre/15.5.0pre1/) to get the prerelease tarballs and the full NEWS file. Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :) Thanks! <3 Amber Brown Twisted Release Manager

2 1

[Twisted-Python] Twisted Trac filters my messages as Spam when trying to update ticket #8102
by Paweł Miech 17 Nov '15

17 Nov '15

Hello, I submitted some patches yesterday, got review from Adi and today I tried to respond. However I'm not able to respond because all my submissions are marked as spam and blocked by SpamBayes with super high probability. I wonder how many other users experience that problem? Can you please fix SpamBayes? I dont think there is anything in my messages that warrants calling it spam. Since I'm not able to update ticket in any way I'm going to post my message here. It's about ticket https://twistedmatrix.com/trac/ticket/8102 my response is following: Thanks for review adiroiban! > I assume that people will try to keep US-ASCII for their method names, so we might want to reject even valind latin1 methods. I agree with that part. All standard HTTP method names are ascii, so I dont see much reason to support non-ascii HTTP methods. They were not supported before this patch (they caused failures), if we fix that and allow latin1 we will allow characters that are outside ascii in method names, e.g. users will be able to have non-standard latin1 method name: 'GET£'. In my opinion it's better to only allow ascii and nothing else. > For non US-ASCII or non latin1 encoding, I think that we should reject the request much earlier. That is when we first parse the request line, and not when we try to process the request Yes I agree, so patch should go somewhere to twisted.web.http.HTTPChannel.linereceived when we first get method (line 1709) right? Also is it ok we raise 501? With current patch we raise 501 Not Suppoted but maybe 400 Bad Request would be better here?

2 1

[Twisted-Python] Twisted Web at 630k reqs/sec
by Tobias Oberstein 16 Nov '15

16 Nov '15

Hi, maybe it's of interest how far it's possible to drive Twisted Web on modern multi-core CPUs. Here are 2 data points from measuring: * 627990 HTTP requests/s at 360 us avg latency * 12.6 GB/s HTTP reply traffic This is using 40 Xeon cores and serves HTTP over loopback TCP to 8 threads of wrk. The Twisted Web resource served was https://github.com/crossbario/crossbarexamples/blob/master/benchmark/web/my… Details, figures and more results are here: https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web Cheers, /Tobias

3 7

[Twisted-Python] HTTP/2 and Twisted
by Cory Benfield 16 Nov '15

16 Nov '15

All, With the work on #7860 nearly done, Twisted should be in a good place to have a HTTP/2 implementation. There’s currently a Trac ticket (#7460) for adding HTTP/2 support to twisted.web, which is obviously a good idea. I’m happy to take on that work. What I want to get a picture for is how much of the Twisted support should live in Twisted itself. Currently I’m planning to base the implementation on Hyper-h2[0]. This is because it seems totally needless to write a new HTTP/2 state machine when a perfectly good one already exists (full disclosure: I’m the maintainer of Hyper-h2). However, we’ll still need a HTTP/2 Protocol, and the twisted.web integration. The twisted.web integration will definitely need to be done in Twisted, but Adi has pointed out to me that it may be better for the HTTP/2 Protocol itself to live outside of core Twisted (probably as a sub-project of Hyper, with a working title of txh2). This would give us a lot more flexibility and speed to iterate. I want to get a sense of what the team believes is the best approach. Can I have your opinions? How much of this should be in Twisted itself? Cory [0]: http://python-hyper.org/h2/

6 6

[Twisted-Python] How to design REST API with Twisted?
by Wang Yan 13 Nov '15

13 Nov '15

Hi, I'm confused about how to design REST APIs with Twisted, especially when my app have to interact with MySQL. Is there any basic design patterns for this kind of situation? Thanks!

4 8

[Twisted-Python] Conch SSH client dropping support for non/pre RFC 4419 clients
by Adi Roiban 11 Nov '15

11 Nov '15

Hi, I just want to let you know about a future change in the SSH client implementation. Full details https://twistedmatrix.com/trac/ticket/8100 Basically in order to work with OpenSSH version 6.9 and newer we will break support for very old (2004-2006) SSH servers which do not support RFC 4419 In case you care about Twisted SSH client interacting with very of SSH servers, please take a look and send your feedback over the Trac issue Otherwise, feel free to ignore and enjoy the rest of the day /night :) -- Adi Roiban

1 0

[Twisted-Python] Dumb Logger API question
by Kevin Mcintyre 11 Nov '15

11 Nov '15

http://twistedmatrix.com/documents/current/core/howto/logger.html I'm looking at the new-ish logger api and wonder how to log to console as a first step to transitioning. Any pointers? Thanks.

2 2