Tristan Seligmann writes:

Currently there is no way to explicitly compose Twisted endpoints, but several endpoint implementations have arisen that explicitly wrap another endpoint, and so have needed a way to do this.

A couple other examples: Autobahn provides 'Web Application Messaging Protocol' (WAMP) endpoints that can use either a Unix, TCP or WebSockets protocol under the hood -- having a proper endpoint syntax for this would be nice. There is *some* support for endpoint-strings in Autobahn using the backslash trick, but this also results in some ugliness like: r"autobahn:tcp\:9000\:interface\=0.0.0.0:url=ws\://localhost\:9000:compress=false" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ~~~~~~~~~~~~~~~~~~~~~ Ideally txtorcon would have a way to specify "how" to connect for client-side connections meaning "I want a stream over the Tor whose control-port is 'unix:/foo' that connects via SOCKS at unix:/bar to https://meejah.ca". (This can only be done in code, currently, not endpoint-strings) Now think about specifying that you want the above Autobahn connection to go over Tor ;) that is, replacing "https://meejah.ca" with the monster above... * * * On a possible tangent: I wonder if this also ties in with trying to wrap protocols to "hand off" their transport to another one? Examples of this are: SOCKS (e.g. speaking one protocol "after" another), or in Autobahn where it's nice to listen for both "normal" Web requests and also WebSockets requests on the same port (so there's a protocol that "peeks" and hands off to HTTP or WebSockets handlers). I can also imagine doing the same thing with http://magic-wormhole.io where you would establish a connection via the wormhole mechanism, and then pass over the established transport to the "real" protocol (e.g. could be HTTP, WAMP, SSH, WebSockets, something custom, etc) and this at least has an obvious need for a corresponding string-parser syntax like the one suggested.

tls:awesome.site.example.com:443->tcp:7.6.5.4:443 A less whimsical syntax than "->" might be better; for example, semicolons, or something like that.

I wonder if a simple space would work? Downsides would be: requiring quoting on shells; maybe it would present problems in config-file use-cases; ... So, we want "something" the highest-level parser can split on before handing off bits (or all) of it to the actual plugins: - can't have ":" since that's already a separatator - can't have "=" since it's already used - shouldn't already be in popular protocols/options (e.g. anything valid in a URI?) In some ways having a two-character separator could be really nice, as it's far less likely to collide with things? I like that the intent of "->" is also reasonably obvious, I think. Or at least hopefully looks strange enough that you'll look it up ;) Thanks for kicking off discussion :) -- meejah

I'm not opposed to a fresh syntax, but I do believe the current implementation can be used for composition. The parser for endpoint strings is simplistic, like Glyph points out, but there is nothing preventing it from having nested endpoint definitions. We used the existing syntax when writing the HAProxy endpoint wrapper: https://github.com/twisted/twisted/blob/trunk/src/twisted/protocols/haproxy/... . Granted, this case doesn't come with any configuration options but it shows a potential path for adding wrapping functionality in the current implementation. I think are some downsides when it comes to args, kwargs management. To support them across multiple, arbitrary nested endpoints the kwargs would need to have non-colliding names and the wrapper would need to do some amount of introspection on args to determine if the leading values are for it or another endpoint. I don't think the developer experience would be a good one, but there _is_ a way to compose endpoints if you're set on doing so. On Tue, Sep 6, 2016 at 6:08 PM Glyph Lefkowitz wrote:

On Sep 6, 2016, at 11:55 AM, Tristan Seligmann wrote:

Currently there is no way to explicitly compose Twisted endpoints, but several endpoint implementations have arisen that explicitly wrap another endpoint, and so have needed a way to do this. So far, this has been implementing by passing in an endpoint description, and then calling serverFromString/clientFromString internally in the endpoint to construct the wrapped endpoint. I've seen two different ways of encoding the "inner" endpoint:

Some other points to keep in mind for this discussion:

1. We may want a syntax that supports composing multiple endpoints, not just 2. 2. The existing syntax is kind of crummy; ":" as a separator has serious problems, considering its presence in both URLs and IPv6 literals. I wouldn't say we should *necessarily* re-design the whole syntax to accommodate this, but just having a whole new syntax might not be a bad thing either.

-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

However, it’s worth highlighting that endpoints are inches away from being a really powerful composable tool for saying “tunnel this protocol over this other protocol”.

I'm not sure if this is the same concern as the OP. What you've described is mixing the ideas of composing protocols and composing transports (which may be inherent to endpoints). TCP, UDP, and UNIX socks are transport layer choices and adapting them to each other requires opening new file descriptors. Protocols are transport agnostic and can already be infinitely composed if implemented as well behaved protocol wrappers. On Wed, Sep 7, 2016, 07:26 Cory Benfield wrote:

On 7 Sep 2016, at 00:05, Glyph Lefkowitz wrote:

On Sep 6, 2016, at 11:55 AM, Tristan Seligmann wrote:

Currently there is no way to explicitly compose Twisted endpoints, but several endpoint implementations have arisen that explicitly wrap another endpoint, and so have needed a way to do this. So far, this has been implementing by passing in an endpoint description, and then calling serverFromString/clientFromString internally in the endpoint to construct the wrapped endpoint. I've seen two different ways of encoding the "inner" endpoint:

Some other points to keep in mind for this discussion:

1. We may want a syntax that supports composing multiple endpoints, not just 2. 2. The existing syntax is kind of crummy; ":" as a separator has serious problems, considering its presence in both URLs and IPv6 literals. I wouldn't say we should *necessarily* re-design the whole syntax to accommodate this, but just having a whole new syntax might not be a bad thing either.

I mentioned this casually to Tristan in IRC, but the current syntax and use of endpoints in Twisted gets close to a quite profound idea about protocol nesting that is lurking in the space of convention. Extending the endpoint syntax to have a blessed way of essentially composing endpoints together gives the potential of using the endpoint syntax to design and implement various “tunneling” features that are very useful.

If we take the arrow syntax, for a moment, you could conceive a truly insane client wanting to write an endpoint to run FTP over that’s a bit like:

tcp:host=someftp.server:port=21->http:verb=connect->tcp:->socks5:targetname=mypersonalhttpproxy.server->tcp:host=mycorporatesocksproxy.server:port=2121

This would represent tunneling FTP over TCP over HTTP over TCP over SOCKS over TCP. For extra fun you can throw in some Tor.

Alternatively, and quite a bit more realistically, you could have a userspace SCTP implementation that supports being tunnelled over UDP. In this instance, rather than write a single “sctp-over-udp” endpoint, you could write a generic sctp endpoint that, if it is composed with another endpoint, expects that endpoint to provide a datagram-style transport to it.

All of this is rather pie in the sky, and potentially the purest example of YAGNI that it’s possible to imagine. However, it’s worth highlighting that endpoints are inches away from being a really powerful composable tool for saying “tunnel this protocol over this other protocol”. We may not *want* that, but it’s an interesting thought.

Cory

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Oct 12, 2016, at 2:11 PM, Tom Prince wrote:

I think if we are reconsidering the endpoint syntax, we should explicitly have nested delimiters for quoting (so something like () or {}), to easily allow multiple levels of nested endpoint strings (or any other kind of string).

Do you have a suggestion or an example of how this might be used? -g

On Oct 12, 2016, at 7:35 PM, Tom Prince wrote:

Thinking about it some more, there isn't currently any endpoint descriptions that have an empty name, so we could have a entirely new syntax that starts with `:`. If we went in that direction, we'd definitely want to think about future extensibility when designing it.

This applies more generally; no need for any weird hacks. Any 'new' plugin could just opt in to a different syntax; we can just look up until the first ':'; we just need to define a new interface for a new syntax. -glyph

This applies more generally; no need for any weird hacks. Any 'new' plugin could just opt in to a different syntax; we can just look up until the first ':'; we just need to define a new interface for a new syntax.

I don't think that this provides a good user experience. 1) There are existing endpoints that want nestable endpoints, so either a) They don't change, somewhat defeating the purpose of having a new syntax (or cluttering the endpoint namespace with less than useful endpoints). b) They change incompatibility, defeating the purpose of trying to maintain backwards compatability. 2) As user, I need to learn which endpoints support the new syntax, thus potentially needing to know both methods of quoting and switch between them as appropriate. There are a couple of possible ways around this, without requiring a weird hack. - I wonder how many endpoints strings have ever been written whose value starts with any of `[` `(` or `{`? I suspect that the number might in fact be 0. In which case, although the change is technically incompatible, in practice it wouldn't be. - Alternatively, we could deprecate an unquoted [, (, { at the beginning of a value, and then after a suitable deprecation period (perhaps additionally a release where it is just an error), we could repurpose one of them to act as quoting (leaving the other two for future extensiblity).