[Twisted-Python] Setting socket options before bind/listen
Hi, for a multiprocess server, I'd like to set SO_REUSEPORT on a TCP listening socket. The option needs to be set _before_ calling bind() and listen() on the socket. Is there a recommended way of doing this? One option might be subclassing `twisted.internet.tcp.Port` and overriding `createInternetSocket` - is that recommended? And if so: how do I make the CustomPort be used with reactor.listenTCP/listenSSL and endpoints? Any hints appreciated, thanks! /Tobias
On 08:41 pm, tobias.oberstein@tavendo.de wrote:
Hi,
for a multiprocess server, I'd like to set SO_REUSEPORT on a TCP listening socket.
You might find that https://stackoverflow.com/questions/10077745 /twistedweb-on-multicore-multiprocessor obviates the need for this option (and thus your follow-up features as well). Not that Twisted shouldn't offer some way to gain more control over this kind of platform-specific option. But, until it does... Jean-Paul
The option needs to be set _before_ calling bind() and listen() on the socket.
Is there a recommended way of doing this?
One option might be subclassing `twisted.internet.tcp.Port` and overriding `createInternetSocket` - is that recommended?
And if so: how do I make the CustomPort be used with reactor.listenTCP/listenSSL and endpoints?
Any hints appreciated, thanks!
/Tobias
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
for a multiprocess server, I'd like to set SO_REUSEPORT on a TCP listening socket.
You might find that https://stackoverflow.com/questions/10077745 /twistedweb-on-multicore-multiprocessor obviates the need for this option (and thus your follow-up features as well).
Jean-Paul, yes, I am aware of this (awesome) answer;) This works, but has 2 slightly undesirable aspects: 1) it requires to have a master process that creates the socket and starts workers which "adopt" the port. I'd like to be able to fire up workers independently .. with no master at all. 2) the distribution of incoming connections to workers isn't completely uniformly distributed across workers. This problem is explained here (they report up to 3:1 ratios of connections per worker): https://lwn.net/Articles/542629/ https://lwn.net/Articles/542718/ and I could verify this during some experiments (though not such extreme non-uniformity)
Not that Twisted shouldn't offer some way to gain more control over this kind of platform-specific option. But, until it does...
I guess that means there is no "recommended" way currently;) Would you mind giving a tip on how to make use of a CustomPort deriving of Port? How to "plug" that into Twisted? Thanks! /Tobias
Jean-Paul
The option needs to be set _before_ calling bind() and listen() on the socket.
Is there a recommended way of doing this?
One option might be subclassing `twisted.internet.tcp.Port` and overriding `createInternetSocket` - is that recommended?
And if so: how do I make the CustomPort be used with reactor.listenTCP/listenSSL and endpoints?
Any hints appreciated, thanks!
/Tobias
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 09:49 pm, tobias.oberstein@tavendo.de wrote:
for a multiprocess server, I'd like to set SO_REUSEPORT on a TCP listening socket.
You might find that https://stackoverflow.com/questions/10077745 /twistedweb-on-multicore-multiprocessor obviates the need for this option (and thus your follow-up features as well).
Jean-Paul,
yes, I am aware of this (awesome) answer;) This works, but has 2 slightly undesirable aspects:
1) it requires to have a master process that creates the socket and starts workers which "adopt" the port. I'd like to be able to fire up workers independently .. with no master at all.
There are ways around this but they're a bit more work (eg, workers listen on a unix socket and hand out the port to new workers that start up and ask for it).
2) the distribution of incoming connections to workers isn't completely uniformly distributed across workers.
This problem is explained here (they report up to 3:1 ratios of connections per worker): https://lwn.net/Articles/542629/ https://lwn.net/Articles/542718/
and I could verify this during some experiments (though not such extreme non-uniformity)
Huh. I didn't know about that. That's too bad.
Not that Twisted shouldn't offer some way to gain more control over this kind of platform-specific option. But, until it does...
I guess that means there is no "recommended" way currently;)
Correct.
Would you mind giving a tip on how to make use of a CustomPort deriving of Port? How to "plug" that into Twisted?
I have two suggestions, one of which I hope you'll like and the other of which you might not. :) First, endpoints are the intended extension point for this sort of thing now. You can write a plugin for the parser so that `serverFromString` will give out server endpoints for your port type (giving an appropriate string description). Applications won't be able to tell what's going on because the server endpoint interface is just `.listen(factory)´. Second, please don't subclass `Port`. It's true there are no underscores anywhere in its name (`twisted.internet.tcp.Port`) making it part of Twisted's public API. Nevertheless, it's very much a reactor implementation detail. It's a mistake that it's public. To compounded this, the exactly interface between a class and its subclasses is hazy and gross at best. I'd discourage you (and everyone else) from subclassing *most* things in Twisted these days (lots of our APIs are still subclassing-oriented so it's not always possible to avoid, of course). In this case, I think you might actually be able to re-use all of the important parts of `Port` without subclassing it. You can create a bare-bones implementation of `IListeningPort` that creates a socket and sets the flags you want on it. Then, use `reactor.adoptStreamPort` to get the reactor to create and initialize a new `Port` with your socket. This leaves you with a little code to duplicate (basically `createInternetSocket`) but a pretty small amount - and the upside is that you're totally isolated from the internals of `Port`, from the accidentally-public implementation details of `Port`, and even from the implementation detail of whether the reactor even *uses* `Port` or not. All you rely on is `reactor.adoptStreamPort` which is a nice, documented, tested, intentionally-public interface. :) (For SSL, then you can wrap your own twisted.protocols.tls wrapper around the factory - which is all the reactor's listenSSL does these days, anyway). Hope this helps, Jean-Paul
Would you mind giving a tip on how to make use of a CustomPort deriving of Port? How to "plug" that into Twisted?
I have two suggestions, one of which I hope you'll like and the other of which you might not. :)
Awesome! This is enough hints to get me started. I already have Twisted endpoint plugins, and I will follow that road, something like tcpwithopts:tcp:9000:interface=0.0.0.0:reuse-port:true:disable-nagle:true Subclassing Port / adoptStream: I see;) Not sure if this SO_REUSEPORT works on Windows - if it does, another aspect then is that adoptStream is Unix-only .. I will see. Thanks again! /Tobias
First, endpoints are the intended extension point for this sort of thing now. You can write a plugin for the parser so that `serverFromString` will give out server endpoints for your port type (giving an appropriate string description). Applications won't be able to tell what's going on because the server endpoint interface is just `.listen(factory)´.
Second, please don't subclass `Port`. It's true there are no underscores anywhere in its name (`twisted.internet.tcp.Port`) making it part of Twisted's public API. Nevertheless, it's very much a reactor implementation detail. It's a mistake that it's public. To compounded this, the exactly interface between a class and its subclasses is hazy and gross at best. I'd discourage you (and everyone else) from subclassing *most* things in Twisted these days (lots of our APIs are still subclassing-oriented so it's not always possible to avoid, of course).
In this case, I think you might actually be able to re-use all of the important parts of `Port` without subclassing it. You can create a bare-bones implementation of `IListeningPort` that creates a socket and sets the flags you want on it. Then, use `reactor.adoptStreamPort` to get the reactor to create and initialize a new `Port` with your socket. This leaves you with a little code to duplicate (basically `createInternetSocket`) but a pretty small amount - and the upside is that you're totally isolated from the internals of `Port`, from the accidentally-public implementation details of `Port`, and even from the implementation detail of whether the reactor even *uses* `Port` or not. All you rely on is `reactor.adoptStreamPort` which is a nice, documented, tested, intentionally-public interface. :)
(For SSL, then you can wrap your own twisted.protocols.tls wrapper around the factory - which is all the reactor's listenSSL does these days, anyway).
Hope this helps,
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
You can also accomplish this using systemd socket activation with the ReusePort= option [1]. Twisted supports inheriting such sockets from systemd. [1] http://www.freedesktop.org/software/systemd/man/systemd.socket.html
participants (3)
-
David Timothy Strauss -
exarkun＠twistedmatrix.com -
Tobias Oberstein