On Mar 28, 2008, at 9:18 AM, Simon Pickles wrote:
Midway thru my server dev, I am seeing the light... well, a light anyway.
Its a stackless python server and not too bad, running on Ubuntu Linux. Stackless Python is great, but single threaded (I DO do my DB lookups and network comms in other threads).
I figure its GOT to be more concurrent, if I'm looking to use future architecture.
So I planned to rewrite the server into modules, each a separate app. I guess this is a virtualisation process. Labour could be subdivided spatially, or by task (db access/playermanager/AI), or both
I'm going to go ahead assuming you know where your load problems are (or will be), and which things are most in need of being separated into components. Of course, this is pretty much the hardest part.
The modules plug in to each other, communicating thru the network (twisted).
This means the prototype server would run all the module apps on a single machine, then spreading to other machines to allow scalability. Being async networked (hinged on twisted.deferred), these modules run concurrently, using all the cores available.
SO - Am I reinventing the wheel? Does Twisted do this, or other Frameworks? In my limited experience, I need to create a server hub then have all the component modules connect to that as clients.
Is there a better way?
I don't believe there's nothing in Twisted that implicitly provides the functionality you're talking about, but you have a wide array of options on how to implement this yourself. It all depends on what kind of information you'll need to pass between modules, and how much refactoring you're willing to do to the existing application. The biggest issue will probably be making a uniform and maintainable way for modules to discover each other and make connections without introducing security holes. If you are sure to have complete control over the network they're running on (say, in a private IP space) this may be a non-issue, so it's really just about logistics. My initial reaction was to suggest Perspective Broker, but someone else might recommend AMP, which I'm just starting to learn the ins and outs of.... -phil