[Twisted-Python] which async framework?
![](https://secure.gravatar.com/avatar/276928028a2075ceeb0b7aface8e2e2a.jpg?s=120&d=mm&r=g)
Hi All, Apologies for the cross-post, but this is a "which framework" question so seemed the most constructive way. Not interested in religious debates, just trying to pick the best tool for the job and didn't get much of a useful response from python-list... So, I see python now has a plethora of async frameworks and I need to try and pick one to use from: - asyncio/tulip - tornado - twisted From my side, I'm looking to experimentally build a network testing tool that will need to speak a fair few protocols, both classic tcp and multicast-based, and have a web api living on top of it that most likely will have a websocket for pumping data to the browser. It'll also need to write out JUnit-compatible xml results, but that's like the easiest bit ;-) I'd like to be able to serve the rest of the web api using a pyramid wsgi app if possible, and I'd like to be able to write the things that process requests in and validation out in a synchronous fashion, most likely spinning off a thread for each one. The protocols are all financial (do we really not have a pure-python FIX library?!) but none are likely to have existing python implementations. How should I pick between the options? What would people recommend and why? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
![](https://secure.gravatar.com/avatar/1846c8040fcf70e9b55bb7bfcdb78bc4.jpg?s=120&d=mm&r=g)
Hi Chris, It will probably not surprise you that I like Twisted :-) On Wed, Mar 12, 2014 at 8:28 AM, Chris Withers <chris@simplistix.co.uk>wrote:
Twisted has a thing called Cyclone, which I hear (but that's only hearsay) gives you Tornado's API on top of twisted, so you get all of the stuff below for free.
I don't know which protocols you are interested in specifically; later on in the e-mail you mention that it's *probably* not already extant. Twisted comes with a huge number of protocols already implemented. It has TCP support (duh) as well as an interface for UDP multicast. Twisted has a third party project called txsockjs which works excellently, both by itself and in the context of other (HTTP) resources. sockjs is a protocol which is basically "websockets, damnit", even when the consumer is a bad browser like IE6.
Twisted comes with a threadpool-backed WSGI server. (When I say threadpool-backed I mean that the WSGI requests are handled in threads; the IO itself of course comes from the Twisted reactor).
The protocols are all financial (do we really not have a pure-python FIX library?!) but none are likely to have existing python implementations.
I find twisted to be a great tool for writing protocol implementations. I have written tools for querying all sorts of gnarly proprietary protocols over all sorts of gnarly transports (packet radio; it's totally a thing), and more recently for doing crazy things like multiplexing stream transports over existing RPC protocols. (Like, you see a local port come up, and that actually creates a virtual stream connection over an existing RPC thing to some virtual server on the other end of the wire: https://www.youtube.com/watch?v=W_jEIvugwes). Twisted has had so many people write so many protocols in it that also the testing tools (MemoryReactor, StringTransport) are great. Especially if you are writing something very close to a wire protocol you will undoubtedly enjoy those amenities. There's also tons of composable things for receiving delimited lines, nestrings, etc. It's hard to tell what you will be looking for since I don't know details about your protocol, but having written more than a few protocol implementations I'm going to wager a guess and say Twisted has it or a third party thing for twisted has it. Even though I mostly write Clojure now, I still write my networking stuff in Twisted. Macros are cool. Eleven years worth of battle-tested code is better. That said, tulip is nice in that you can also write protocol implementations that will look similar :-)
How should I pick between the options? What would people recommend and why?
Twisted, emphatically and without reservation, for all the above reasons. It's stable. All the stuff you need has been tried and tested extensively in production. It runs on PyPy, and usually a damn sight faster than on CPython, too. A common criticism of Twisted is that it "takes over" your codebase. I am speaking at PyCon in about a month to demonstrate that that isn't true. (It just looks that way, because once people use it, they don't want to go back... ;-)) hth lvh
![](https://secure.gravatar.com/avatar/276928028a2075ceeb0b7aface8e2e2a.jpg?s=120&d=mm&r=g)
On 12/03/2014 08:53, Laurens Van Houtven wrote:
Aiming low (insert sarcastic look here), here's the first few: http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_FIX_S... http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_Binar... http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_PITCH... http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_MC_PI...
Thankfully, I only have to support sane browsers...
Yeah, I remember that, and I remember liking it :-)
Cool!
Yep, certainly sounds good. I guess I have concerns about discoverability and making sure I'm picking the right things to use rather than re-implementing things when I don't need to. What's the best way for me to find things I should be using? I guess my fallback position is to ask lots of clueless questions on this list and hope I don't annoy to many people, is that viable?
Didn't know about the PyPy thing, that will be interesting if I ever hit performance problems...
Gutted I can't make it to PyCon this year, look forward to watching the video! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
![](https://secure.gravatar.com/avatar/1816aaddac0110ea78185e7759156907.jpg?s=120&d=mm&r=g)
On 12 March 2014 07:28, Chris Withers <chris@simplistix.co.uk> wrote:
Tornado is a web-framework, if you are going to use pyramid for handling wsgi requests asyncio or twisted are probably a better solution. They can handle your synchronous wsgi responses via the event loop executor (a thread pool).
The protocols are all financial (do we really not have a pure-python FIX library?!) but none are likely to have existing python implementations.
Both asyncio and twisted offer an extensive, somehow similar, API for writing protocols, TCP and UDP.
How should I pick between the options? What would people recommend and why?
If you are using python 3.3 I would strongly recommend asyncio, it has a clean and unintrusive API and, importantly, it is part of the standard library (in python 3.4). If you are using python 2 I would still recommend it, the trollius project has backported the library (although there are some important differences). Pulsar, is a server side library I maintain which is now fully based on asyncio, works on python 3 & 2, and does more or less what you are looking for. Of course, twisted is more tested and has a longer history, but if you are forward looking asyncio might be a better bet.
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
From my side, I'm looking to experimentally build a network testing tool that will need to speak a fair few protocols, both classic tcp and multicast-based,
Tornado is a _Web_ framework. A requirement for UDP, and multicast UDP, narrows suitable choices to Twisted and asyncio. Twisted is around much longer, and also has a broader focus .. lots of protocols implemented on top of TCP/IP. Asyncio is opiniated towards a co-routine based style. Twisted probably more towards "classical" Deferred based style, though you can go co-routine style as well.
and have a web api living on top of it that most likely will have a websocket for pumping data to the browser. It'll also need to write out JUnit-compatible
If you need WebSocket anyway, you can directly compare Twisted and asyncio programming with these examples: https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... https://github.com/tavendo/AutobahnPython/tree/master/examples/asyncio/webso... Disclaimer: I am affiliated with that stuff.
I'd like to be able to serve the rest of the web api using a pyramid wsgi app if
Twisted can act as a WSGI host, and that can be combined with WebSocket also (in one server, and both services running on one port). /Tobias
![](https://secure.gravatar.com/avatar/20316199a8197e9c349a945df8afcd90.jpg?s=120&d=mm&r=g)
The protocols are all financial (do we really not have a pure-python FIX library?!) but none are likely to have existing python implementations.
How should I pick between the options? What would people recommend and why?
This might be a starting point for the FIX stuff https://pypi.python.org/pypi/fixlib/0.5 Tim Hughes mailto:thughes@thegoldfish.org
![](https://secure.gravatar.com/avatar/1846c8040fcf70e9b55bb7bfcdb78bc4.jpg?s=120&d=mm&r=g)
Hi Chris, It will probably not surprise you that I like Twisted :-) On Wed, Mar 12, 2014 at 8:28 AM, Chris Withers <chris@simplistix.co.uk>wrote:
Twisted has a thing called Cyclone, which I hear (but that's only hearsay) gives you Tornado's API on top of twisted, so you get all of the stuff below for free.
I don't know which protocols you are interested in specifically; later on in the e-mail you mention that it's *probably* not already extant. Twisted comes with a huge number of protocols already implemented. It has TCP support (duh) as well as an interface for UDP multicast. Twisted has a third party project called txsockjs which works excellently, both by itself and in the context of other (HTTP) resources. sockjs is a protocol which is basically "websockets, damnit", even when the consumer is a bad browser like IE6.
Twisted comes with a threadpool-backed WSGI server. (When I say threadpool-backed I mean that the WSGI requests are handled in threads; the IO itself of course comes from the Twisted reactor).
The protocols are all financial (do we really not have a pure-python FIX library?!) but none are likely to have existing python implementations.
I find twisted to be a great tool for writing protocol implementations. I have written tools for querying all sorts of gnarly proprietary protocols over all sorts of gnarly transports (packet radio; it's totally a thing), and more recently for doing crazy things like multiplexing stream transports over existing RPC protocols. (Like, you see a local port come up, and that actually creates a virtual stream connection over an existing RPC thing to some virtual server on the other end of the wire: https://www.youtube.com/watch?v=W_jEIvugwes). Twisted has had so many people write so many protocols in it that also the testing tools (MemoryReactor, StringTransport) are great. Especially if you are writing something very close to a wire protocol you will undoubtedly enjoy those amenities. There's also tons of composable things for receiving delimited lines, nestrings, etc. It's hard to tell what you will be looking for since I don't know details about your protocol, but having written more than a few protocol implementations I'm going to wager a guess and say Twisted has it or a third party thing for twisted has it. Even though I mostly write Clojure now, I still write my networking stuff in Twisted. Macros are cool. Eleven years worth of battle-tested code is better. That said, tulip is nice in that you can also write protocol implementations that will look similar :-)
How should I pick between the options? What would people recommend and why?
Twisted, emphatically and without reservation, for all the above reasons. It's stable. All the stuff you need has been tried and tested extensively in production. It runs on PyPy, and usually a damn sight faster than on CPython, too. A common criticism of Twisted is that it "takes over" your codebase. I am speaking at PyCon in about a month to demonstrate that that isn't true. (It just looks that way, because once people use it, they don't want to go back... ;-)) hth lvh
![](https://secure.gravatar.com/avatar/276928028a2075ceeb0b7aface8e2e2a.jpg?s=120&d=mm&r=g)
On 12/03/2014 08:53, Laurens Van Houtven wrote:
Aiming low (insert sarcastic look here), here's the first few: http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_FIX_S... http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_Binar... http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_PITCH... http://cdn.batstrading.com/resources/participant_resources/BATS_Europe_MC_PI...
Thankfully, I only have to support sane browsers...
Yeah, I remember that, and I remember liking it :-)
Cool!
Yep, certainly sounds good. I guess I have concerns about discoverability and making sure I'm picking the right things to use rather than re-implementing things when I don't need to. What's the best way for me to find things I should be using? I guess my fallback position is to ask lots of clueless questions on this list and hope I don't annoy to many people, is that viable?
Didn't know about the PyPy thing, that will be interesting if I ever hit performance problems...
Gutted I can't make it to PyCon this year, look forward to watching the video! Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
![](https://secure.gravatar.com/avatar/1816aaddac0110ea78185e7759156907.jpg?s=120&d=mm&r=g)
On 12 March 2014 07:28, Chris Withers <chris@simplistix.co.uk> wrote:
Tornado is a web-framework, if you are going to use pyramid for handling wsgi requests asyncio or twisted are probably a better solution. They can handle your synchronous wsgi responses via the event loop executor (a thread pool).
The protocols are all financial (do we really not have a pure-python FIX library?!) but none are likely to have existing python implementations.
Both asyncio and twisted offer an extensive, somehow similar, API for writing protocols, TCP and UDP.
How should I pick between the options? What would people recommend and why?
If you are using python 3.3 I would strongly recommend asyncio, it has a clean and unintrusive API and, importantly, it is part of the standard library (in python 3.4). If you are using python 2 I would still recommend it, the trollius project has backported the library (although there are some important differences). Pulsar, is a server side library I maintain which is now fully based on asyncio, works on python 3 & 2, and does more or less what you are looking for. Of course, twisted is more tested and has a longer history, but if you are forward looking asyncio might be a better bet.
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
From my side, I'm looking to experimentally build a network testing tool that will need to speak a fair few protocols, both classic tcp and multicast-based,
Tornado is a _Web_ framework. A requirement for UDP, and multicast UDP, narrows suitable choices to Twisted and asyncio. Twisted is around much longer, and also has a broader focus .. lots of protocols implemented on top of TCP/IP. Asyncio is opiniated towards a co-routine based style. Twisted probably more towards "classical" Deferred based style, though you can go co-routine style as well.
and have a web api living on top of it that most likely will have a websocket for pumping data to the browser. It'll also need to write out JUnit-compatible
If you need WebSocket anyway, you can directly compare Twisted and asyncio programming with these examples: https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... https://github.com/tavendo/AutobahnPython/tree/master/examples/asyncio/webso... Disclaimer: I am affiliated with that stuff.
I'd like to be able to serve the rest of the web api using a pyramid wsgi app if
Twisted can act as a WSGI host, and that can be combined with WebSocket also (in one server, and both services running on one port). /Tobias
![](https://secure.gravatar.com/avatar/20316199a8197e9c349a945df8afcd90.jpg?s=120&d=mm&r=g)
The protocols are all financial (do we really not have a pure-python FIX library?!) but none are likely to have existing python implementations.
How should I pick between the options? What would people recommend and why?
This might be a starting point for the FIX stuff https://pypi.python.org/pypi/fixlib/0.5 Tim Hughes mailto:thughes@thegoldfish.org
participants (6)
-
Antoine Pitrou
-
Chris Withers
-
Laurens Van Houtven
-
Luca Sbardella
-
Tim Hughes
-
Tobias Oberstein