On 09:49 pm, tobias.oberstein@tavendo.de wrote:
for a multiprocess server, I'd like to set SO_REUSEPORT on a TCP listening socket.
You might find that https://stackoverflow.com/questions/10077745 /twistedweb-on-multicore-multiprocessor obviates the need for this option (and thus your follow-up features as well).
Jean-Paul,
yes, I am aware of this (awesome) answer;) This works, but has 2 slightly undesirable aspects:
1) it requires to have a master process that creates the socket and starts workers which "adopt" the port. I'd like to be able to fire up workers independently .. with no master at all.
There are ways around this but they're a bit more work (eg, workers listen on a unix socket and hand out the port to new workers that start up and ask for it).
2) the distribution of incoming connections to workers isn't completely uniformly distributed across workers.
This problem is explained here (they report up to 3:1 ratios of connections per worker): https://lwn.net/Articles/542629/ https://lwn.net/Articles/542718/
and I could verify this during some experiments (though not such extreme non-uniformity)
Huh. I didn't know about that. That's too bad.
Not that Twisted shouldn't offer some way to gain more control over this kind of platform-specific option. But, until it does...
I guess that means there is no "recommended" way currently;)
Correct.
Would you mind giving a tip on how to make use of a CustomPort deriving of Port? How to "plug" that into Twisted?
I have two suggestions, one of which I hope you'll like and the other of which you might not. :) First, endpoints are the intended extension point for this sort of thing now. You can write a plugin for the parser so that `serverFromString` will give out server endpoints for your port type (giving an appropriate string description). Applications won't be able to tell what's going on because the server endpoint interface is just `.listen(factory)´. Second, please don't subclass `Port`. It's true there are no underscores anywhere in its name (`twisted.internet.tcp.Port`) making it part of Twisted's public API. Nevertheless, it's very much a reactor implementation detail. It's a mistake that it's public. To compounded this, the exactly interface between a class and its subclasses is hazy and gross at best. I'd discourage you (and everyone else) from subclassing *most* things in Twisted these days (lots of our APIs are still subclassing-oriented so it's not always possible to avoid, of course). In this case, I think you might actually be able to re-use all of the important parts of `Port` without subclassing it. You can create a bare-bones implementation of `IListeningPort` that creates a socket and sets the flags you want on it. Then, use `reactor.adoptStreamPort` to get the reactor to create and initialize a new `Port` with your socket. This leaves you with a little code to duplicate (basically `createInternetSocket`) but a pretty small amount - and the upside is that you're totally isolated from the internals of `Port`, from the accidentally-public implementation details of `Port`, and even from the implementation detail of whether the reactor even *uses* `Port` or not. All you rely on is `reactor.adoptStreamPort` which is a nice, documented, tested, intentionally-public interface. :) (For SSL, then you can wrap your own twisted.protocols.tls wrapper around the factory - which is all the reactor's listenSSL does these days, anyway). Hope this helps, Jean-Paul