[Twisted-Python] Twisted Endpoints and WebSocket / Autobahn
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Hi, I am working on Twisted Endpoint support in Autobahn https://github.com/tavendo/AutobahnPython. So far, I made Autobahn able to talk WebSocket _over_ arbitrary Twisted Endpoints, e.g. WebSocket over Unix domain sockets works. This is already quite nifty. Is there anything like Twisted Endpoints for Processes and/or Serial? Like: endpoint = serverFromString(reactor,"process:program=/usr/local/bin/myprogram") or endpoint = serverFromString(reactor,"serial:port=/dev/tty1:baudrate=115000") That would allow to talk WebSocket over stdin/stdout to a program or a serial conneced device. Both can be useful in certain situations. I'd also like to add the other: creating an Twisted Endpoint _from_ an Autobahn server/client to be able to talk any (stream) protocol _over_ WebSocket (like SSH or VNC over WebSocket). endpoint = serverFromString(reactor,"websocket:tcp:80:url=ws://myhostname.com/somepath") I guess I need to provide an implementation of http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I... What else? Is there some docs or recipe? Thanks! /Tobias
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Dec 12, 2013, at 12:29 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
Hi,
I am working on Twisted Endpoint support in Autobahn https://github.com/tavendo/AutobahnPython.
Yay!
Yes and no. For serial port support, see <https://twistedmatrix.com/trac/ticket/4847>. For subprocess support, <https://twistedmatrix.com/documents/current/api/twisted.internet.endpoints.P...>. However, process support is unfinished, as you can see in <https://twistedmatrix.com/trac/ticket/5813>; there still needs to be a string parser plugin, so you can't do what you suggest with serverFromString. I hope you'll poke those tickets along through the process.
That would allow to talk WebSocket over stdin/stdout to a program or a serial conneced device. Both can be useful in certain situations.
And that would be awesome.
I'd also like to add the other: creating an Twisted Endpoint _from_ an Autobahn server/client to be able to talk any (stream) protocol _over_ WebSocket (like SSH or VNC over WebSocket).
endpoint = serverFromString(reactor,"websocket:tcp:80:url=ws://myhostname.com/somepath")
There's totally a plugin architecture for this, and I'm excited for you to use it :-). One quick comment though: would you be OK with using 'autobahn' as your identifier rather than 'websocket'? There are still plans to include basic websocket support within Twisted itself and it would be a shame to start conflicting on that identifier.
Sadly it seems that although <http://twistedmatrix.com/documents/current/core/howto/endpoints.html> alludes to the plugin API, it doesn't actually link to it. The way that you add a new type of endpoint to the string parser is by registering a Twisted plugin (assuming that you know how to do that) providing <https://twistedmatrix.com/documents/current/api/twisted.internet.interfaces....>. Hopefully the API documentation there is helpful; from there you'll need to write implementations of IStreamServerEndpoint, IListeningPort, and ITransport; the breadcrumbs should be laid out well enough that it's pretty clear where each of those needs to be though. Hope that helps, and thanks for actually wanting to use these plugin APIs we worked so hard on, -glyph
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Endpoints are cool. FWIW, here is a working example of WebSocket I tested over both Unix domain sockets and the usual TCP/TLS endpoints: https://github.com/tavendo/AutobahnPython/tree/master/examples/websocket/ech... This works with all features Autobahn provides like support for different WebSocket versions (Hixie-76, RFC6455) and WebSocket extensions (like permessage-compression with deflate, snappy, bzip2). [snip: talk on Endpoints for Serial and Process]
I hope you'll poke those tickets along through the process.
Thanks for directions! I will have a look into those.
Sure. Can I use: "autobahn.websocket" ? Since Autobahn also provides other protocols (WAMP), and .. == Thanks for your pointers on the Endpoint plugin architecture. I guess thats enough to get me started - modulo below .. if you could give me 1 more reply with feedback, that would be awesome! In the meantime I did a little more thinking about WebSocket and endpoints. I can see two approaches - which one is the way forward? Variant 1) endpoint = serverFromString(reactor,"autobahn.websocket: tcp:80:url=ws://myhostname.com/somepath") Here, a single endpoint descriptor is specifying parameters for both WebSocket, and the underlying transport. Underlying transport: tcp:80 WebSocket overlay: autobahn.websocket:url=ws://myhostname.com/somepath:extensions:permessage-deflate;permessage-snappy So the plugin needs to parse not only the "overlay transport" spec, but also the underlying spec ("tcp:80") and all variants thereof? Or can I let the existing Twisted machinery do the parsing of the underlying, and only parse off the remaining stuff ("url=ws://myhostname.com/somepath:extensions:permessage-deflate;permessage-snappy" in this case) And if not, does that mean not only the endpoint main identifier ("tcp" vs "unix" vs "autobahn.websocket") must be non-overlapping, but also all the individual parameters? Like in the case for WebSocket, there is a need for a "url" parameter. These Qs have brought me to .. Variant 2) endpoint = serverFromString(reactor, "tcp:80") wrappedEndpoint = serverFromString(reactor, endpoint, "autobahn.websocket:url=ws://myhost.com/mypath") where I would start a program like python myprog.py --transport "tcp:80" --wrappingTransport "autobahn.websocket:url=ws://myhost.com/mypath" Is there such thing as a WrappingEndpoint? == This is yet another topic/question, but it fits here. Twisted has defined interfaces for both stream (and datagram transports). WebSocket is essentially a bidirectional, reliable, ordered datagram transport (with 2 payload types: binary and utf8). Is there a Twisted Interface for such transports? I am asking, since I am right now also refactoring the code within Autobahn that lays above raw WebSocket .. a PubSub+RPC protocol, that essentially only assumes a transport like above. I want to make it independent of WebSocket and only assume a bidirectional, reliable, ordered, datagram transport. Thanks a bunch for your thoughts and help! /Tobias
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Dec 13, 2013, at 7:09 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
Cool!
Sure.
There's some discussion of this here: <http://twistedmatrix.com/trac/ticket/5642>. It's probably possible, the rules are reasonably simple.
And if not, does that mean not only the endpoint main identifier ("tcp" vs "unix" vs "autobahn.websocket") must be non-overlapping, but also all the individual parameters?
Nope; the first one has to be non-overlapping, then you just have to decide on a definite place where the "wrapped" endpoint begins.
This variant is worse because you can't change the command lines of other twistd plugins, which might be able to use websockets unchanged if you could just get it returned by serverFromString.
Is there such thing as a WrappingEndpoint?
Not yet, no. Comment on that ticket :).
No. Websockets are incredibly bizarre; most protocols support either arbitrarily typed data or just bytes/just text. The hybrid weirdness of "binary" (with no additional type information) and "text" is unique.
I am asking, since I am right now also refactoring the code within Autobahn that lays above raw WebSocket .. a PubSub+RPC protocol, that essentially only assumes a transport like above.
I want to make it independent of WebSocket and only assume a bidirectional, reliable, ordered, datagram transport.
You can do this with length prefixing, but arguably Twisted should have such an interface that can express this combination of concepts, because I think these are, technically speaking, the semantics of a UNIX datagram endpoint, but the required interface (actually a superclass, bleh) is the same as for UDP, implying unreliability. -glyph
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Hi, I've made Autobahn fully Endpoint aware. You can run WebSocket over any stream-based endpoint https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... and you can run any stream-based endpoint over WebSocket (which in turn runs over any stream-based underlying endpoint): https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... Endpoints are cool! Cheers, /Tobias PS: Above is in the repo, but not yet released. The upcoming Autobahn 0.7.0 release will have it - as well as more, big advances.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Dec 30, 2013, at 3:34 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
AWESOME!!!! I'm just thrilled at more third-party plugins starting to take advantage of this powerful extensibility API. This looks great. (Sorry about the somewhat poorly-thought-out syntax there, with the colons that need to be escaped all over the place; hopefully we can fix that one day somehow.) Is there an example of doing this with 'twistd' anywhere? -glyph
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Thanks for endpoints! I think it's a concept in very good Unix tradition: make everything look the same (stream/endpoint). Make it transparent for users.
Yes;) The syntax works, but it could be nicer. Well, nothing is perfect ..
Is there an example of doing this with 'twistd' anywhere?
I was hoping you or someone could tell me how to do one;) Thing is: I was looking for --endpoint "..." argument with twistd, but there isn't. How is the full endpoint "vision" supposed to work out? I mean, being able to: twistd conch --endpoint "autobahn:tcp\:80:url=ws\://myhost.com" to run a conch server that can be accessed running SSH over WebSocket over TCP. /Tobias
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Thanks! That's what I was missing. It works: I can run twistd -n telnet --port "autobahn:tcp\:9000:url=ws\://localhost\:9000" and access the Python shell from a browser: http://picpaste.com/Clipboard23-fJfJCg7E.png I'll write a small Telnet client in JS .. and will post a complete example. This is cool=) /Tobias
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Matt,
That's neat! Endpoints rock. I'd like to put this up on twistedftw.org -- I look forward to your complete example.
Great! All examples and all code is here https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... and here https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... Cheers, /Tobias
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Is there an example of doing this with 'twistd' anywhere?
I've added a generic endpoint forwarder plugin for twistd to Autobahn: The example also includes a Terminal running in a browser, which talks over WebSocket to the endpoint forwarder in twistd, which forwards the traffic to a Telnet server: http://picpaste.com/Clipboard24-LLwRCPKG.png https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... In other words: this shows that endpoint plugins can be combined with twistd plugins. The forwarder should be able to forward between any pair of Twisted stream server endpoint to any stream client endpoint. As soon as the Process endpoints are ready (the parsing), this would allows to expose programs stdin/stdout to WebSocket eg. Or access any serial device over WebSocket (when serial endpoints are there). This functionality is the equivalent (actually a superset) of the following projects: https://github.com/kanaka/websockify https://github.com/joewalnes/websocketd Cheers, /Tobias PS 1: I couldn't get Manhole fully working with the JS Terminal: https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... PS 2: the forwarder is _generic_ https://github.com/tavendo/AutobahnPython/blob/master/autobahn/autobahn/twis... and probably should not be part of Autobahn but Twisted. And it should be robustified;)
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 06:33 pm, tobias.oberstein@tavendo.de wrote:
Indeed. This should ultimately be what `twistd portforward` offers. :) https://twistedmatrix.com/trac/ticket/4927 is related. Jean-Paul
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Dec 12, 2013, at 12:29 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
Hi,
I am working on Twisted Endpoint support in Autobahn https://github.com/tavendo/AutobahnPython.
Yay!
Yes and no. For serial port support, see <https://twistedmatrix.com/trac/ticket/4847>. For subprocess support, <https://twistedmatrix.com/documents/current/api/twisted.internet.endpoints.P...>. However, process support is unfinished, as you can see in <https://twistedmatrix.com/trac/ticket/5813>; there still needs to be a string parser plugin, so you can't do what you suggest with serverFromString. I hope you'll poke those tickets along through the process.
That would allow to talk WebSocket over stdin/stdout to a program or a serial conneced device. Both can be useful in certain situations.
And that would be awesome.
I'd also like to add the other: creating an Twisted Endpoint _from_ an Autobahn server/client to be able to talk any (stream) protocol _over_ WebSocket (like SSH or VNC over WebSocket).
endpoint = serverFromString(reactor,"websocket:tcp:80:url=ws://myhostname.com/somepath")
There's totally a plugin architecture for this, and I'm excited for you to use it :-). One quick comment though: would you be OK with using 'autobahn' as your identifier rather than 'websocket'? There are still plans to include basic websocket support within Twisted itself and it would be a shame to start conflicting on that identifier.
Sadly it seems that although <http://twistedmatrix.com/documents/current/core/howto/endpoints.html> alludes to the plugin API, it doesn't actually link to it. The way that you add a new type of endpoint to the string parser is by registering a Twisted plugin (assuming that you know how to do that) providing <https://twistedmatrix.com/documents/current/api/twisted.internet.interfaces....>. Hopefully the API documentation there is helpful; from there you'll need to write implementations of IStreamServerEndpoint, IListeningPort, and ITransport; the breadcrumbs should be laid out well enough that it's pretty clear where each of those needs to be though. Hope that helps, and thanks for actually wanting to use these plugin APIs we worked so hard on, -glyph
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Endpoints are cool. FWIW, here is a working example of WebSocket I tested over both Unix domain sockets and the usual TCP/TLS endpoints: https://github.com/tavendo/AutobahnPython/tree/master/examples/websocket/ech... This works with all features Autobahn provides like support for different WebSocket versions (Hixie-76, RFC6455) and WebSocket extensions (like permessage-compression with deflate, snappy, bzip2). [snip: talk on Endpoints for Serial and Process]
I hope you'll poke those tickets along through the process.
Thanks for directions! I will have a look into those.
Sure. Can I use: "autobahn.websocket" ? Since Autobahn also provides other protocols (WAMP), and .. == Thanks for your pointers on the Endpoint plugin architecture. I guess thats enough to get me started - modulo below .. if you could give me 1 more reply with feedback, that would be awesome! In the meantime I did a little more thinking about WebSocket and endpoints. I can see two approaches - which one is the way forward? Variant 1) endpoint = serverFromString(reactor,"autobahn.websocket: tcp:80:url=ws://myhostname.com/somepath") Here, a single endpoint descriptor is specifying parameters for both WebSocket, and the underlying transport. Underlying transport: tcp:80 WebSocket overlay: autobahn.websocket:url=ws://myhostname.com/somepath:extensions:permessage-deflate;permessage-snappy So the plugin needs to parse not only the "overlay transport" spec, but also the underlying spec ("tcp:80") and all variants thereof? Or can I let the existing Twisted machinery do the parsing of the underlying, and only parse off the remaining stuff ("url=ws://myhostname.com/somepath:extensions:permessage-deflate;permessage-snappy" in this case) And if not, does that mean not only the endpoint main identifier ("tcp" vs "unix" vs "autobahn.websocket") must be non-overlapping, but also all the individual parameters? Like in the case for WebSocket, there is a need for a "url" parameter. These Qs have brought me to .. Variant 2) endpoint = serverFromString(reactor, "tcp:80") wrappedEndpoint = serverFromString(reactor, endpoint, "autobahn.websocket:url=ws://myhost.com/mypath") where I would start a program like python myprog.py --transport "tcp:80" --wrappingTransport "autobahn.websocket:url=ws://myhost.com/mypath" Is there such thing as a WrappingEndpoint? == This is yet another topic/question, but it fits here. Twisted has defined interfaces for both stream (and datagram transports). WebSocket is essentially a bidirectional, reliable, ordered datagram transport (with 2 payload types: binary and utf8). Is there a Twisted Interface for such transports? I am asking, since I am right now also refactoring the code within Autobahn that lays above raw WebSocket .. a PubSub+RPC protocol, that essentially only assumes a transport like above. I want to make it independent of WebSocket and only assume a bidirectional, reliable, ordered, datagram transport. Thanks a bunch for your thoughts and help! /Tobias
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Dec 13, 2013, at 7:09 AM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
Cool!
Sure.
There's some discussion of this here: <http://twistedmatrix.com/trac/ticket/5642>. It's probably possible, the rules are reasonably simple.
And if not, does that mean not only the endpoint main identifier ("tcp" vs "unix" vs "autobahn.websocket") must be non-overlapping, but also all the individual parameters?
Nope; the first one has to be non-overlapping, then you just have to decide on a definite place where the "wrapped" endpoint begins.
This variant is worse because you can't change the command lines of other twistd plugins, which might be able to use websockets unchanged if you could just get it returned by serverFromString.
Is there such thing as a WrappingEndpoint?
Not yet, no. Comment on that ticket :).
No. Websockets are incredibly bizarre; most protocols support either arbitrarily typed data or just bytes/just text. The hybrid weirdness of "binary" (with no additional type information) and "text" is unique.
I am asking, since I am right now also refactoring the code within Autobahn that lays above raw WebSocket .. a PubSub+RPC protocol, that essentially only assumes a transport like above.
I want to make it independent of WebSocket and only assume a bidirectional, reliable, ordered, datagram transport.
You can do this with length prefixing, but arguably Twisted should have such an interface that can express this combination of concepts, because I think these are, technically speaking, the semantics of a UNIX datagram endpoint, but the required interface (actually a superclass, bleh) is the same as for UDP, implying unreliability. -glyph
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Hi, I've made Autobahn fully Endpoint aware. You can run WebSocket over any stream-based endpoint https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... and you can run any stream-based endpoint over WebSocket (which in turn runs over any stream-based underlying endpoint): https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... Endpoints are cool! Cheers, /Tobias PS: Above is in the repo, but not yet released. The upcoming Autobahn 0.7.0 release will have it - as well as more, big advances.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Dec 30, 2013, at 3:34 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
AWESOME!!!! I'm just thrilled at more third-party plugins starting to take advantage of this powerful extensibility API. This looks great. (Sorry about the somewhat poorly-thought-out syntax there, with the colons that need to be escaped all over the place; hopefully we can fix that one day somehow.) Is there an example of doing this with 'twistd' anywhere? -glyph
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Thanks for endpoints! I think it's a concept in very good Unix tradition: make everything look the same (stream/endpoint). Make it transparent for users.
Yes;) The syntax works, but it could be nicer. Well, nothing is perfect ..
Is there an example of doing this with 'twistd' anywhere?
I was hoping you or someone could tell me how to do one;) Thing is: I was looking for --endpoint "..." argument with twistd, but there isn't. How is the full endpoint "vision" supposed to work out? I mean, being able to: twistd conch --endpoint "autobahn:tcp\:80:url=ws\://myhost.com" to run a conch server that can be accessed running SSH over WebSocket over TCP. /Tobias
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Thanks! That's what I was missing. It works: I can run twistd -n telnet --port "autobahn:tcp\:9000:url=ws\://localhost\:9000" and access the Python shell from a browser: http://picpaste.com/Clipboard23-fJfJCg7E.png I'll write a small Telnet client in JS .. and will post a complete example. This is cool=) /Tobias
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Matt,
That's neat! Endpoints rock. I'd like to put this up on twistedftw.org -- I look forward to your complete example.
Great! All examples and all code is here https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... and here https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... Cheers, /Tobias
![](https://secure.gravatar.com/avatar/83111e3ce141b383d5169be6edd768d1.jpg?s=120&d=mm&r=g)
Is there an example of doing this with 'twistd' anywhere?
I've added a generic endpoint forwarder plugin for twistd to Autobahn: The example also includes a Terminal running in a browser, which talks over WebSocket to the endpoint forwarder in twistd, which forwards the traffic to a Telnet server: http://picpaste.com/Clipboard24-LLwRCPKG.png https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... In other words: this shows that endpoint plugins can be combined with twistd plugins. The forwarder should be able to forward between any pair of Twisted stream server endpoint to any stream client endpoint. As soon as the Process endpoints are ready (the parsing), this would allows to expose programs stdin/stdout to WebSocket eg. Or access any serial device over WebSocket (when serial endpoints are there). This functionality is the equivalent (actually a superset) of the following projects: https://github.com/kanaka/websockify https://github.com/joewalnes/websocketd Cheers, /Tobias PS 1: I couldn't get Manhole fully working with the JS Terminal: https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/webso... PS 2: the forwarder is _generic_ https://github.com/tavendo/AutobahnPython/blob/master/autobahn/autobahn/twis... and probably should not be part of Autobahn but Twisted. And it should be robustified;)
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 06:33 pm, tobias.oberstein@tavendo.de wrote:
Indeed. This should ultimately be what `twistd portforward` offers. :) https://twistedmatrix.com/trac/ticket/4927 is related. Jean-Paul
participants (5)
-
exarkun@twistedmatrix.com
-
Glyph
-
Hynek Schlawack
-
Matt Haggard
-
Tobias Oberstein