Here's some ideas I've been thinking about with aid of my housemate (who is CC'd).
A feature of some servers is the ability to run chroot'd, so say you want to be a WebDAV server for a site, when a user logs in, it forks, chroots to the users home directory, and sets its uid/gid to that of the user, thereby restricting the potential for security holes to damage the system. (The chroot is an optional paranoid extra if you are dropping uid/gid)
Consider twisted.web.distrib (I might have bits of this slightly wrong; please correct me if so). It currently does something like this, although manually: each user has to start their own twistd, which only runs twisted.web, and talks to the superserver with PB via a pipe. This is good, but not as general as it could be. It's limited to twisted.web, and users have to manually ensure that twistd is running.
I think it'd be nice to generalise this to *any* service that can run within the context of different users, e.g. for FTP, WebDAV, whatever. So there'd be some automated way to tell a Twisted application "Ok, now serve this as a particular user by passing it to a user-owned twistd".
What I'm envisaging is something like this: * The superserver receives a WidgetService request from user Bob * The superserver connects to the user's WidgetService twistd, starting it if necessary (more on this in a moment) * The user's WidgetService then does all the hard work :)
If the user doesn't have a currently running twistd, then you could do, say: * The superserver creates a pipe * The superserver (which is running as root (or something effectively root)) forks a child * The child does a reactor.crash() to stop it interfering with the parent's normal handling of events * The child closes all file descriptors in the reactor, except for the pipe. It should probably start logging to a different log file as well. Also, things like DB-backed twisted.cred authenticators probably should be shutdown... hmm, this part is messy :( * The child then starts a fresh Application, which runs the user's WidgetService and starts processing for the superserver
Now that I think about it, it is probably cleaner to simply spawn a fresh process rather than futzing around with fork, so just: * The superserver spawns a user twistd as a particular uid/gid, and talks to it over a PB pipe.
The nice thing about this model is that there is only 1 twistd running per user (and you could get the user twistds to automatically shutdown if inactive for, say, 10 minutes to free resources), which means your process load is still mostly independent of the number of connections. It'd be *really* cool if a superserver listening on multiple protocols, e.g. "WidgetService" and HTTP, could have only a *single* twistd per user, regardless of number of protocols the superserver is dealing with.
It'd be really nice to be able to chroot as well, but I suspect python doesn't work cleanly with chroot, due to file descriptors being left open to modules...
So yeah, this is a long-winded way of saying "twisted.web.distrib is cool, but should be general and more automatic", because I think this would probably be useful.
In vaguely related news, I've nearly got a simple Twisted inetd replacement written... expect a checkin sometime this weekend.