[Twisted-Python] Using Twisted for a clustering library.
I've been working on a cluster library for MOSIX and I've noticed that parts could possibly be replaced with some Twisted stuff. Briefly what I need to be able to do is the following: o Have the server spawn off client processes and communicates with them using IPC. o The client process waits for a task to arrive from the server and replies back to the server with the results of the complete task. o The server sends tasks to available clients then waits for a client to repond, sends the now idle client the next task and finally handles the response. After browsing through the Twisted source, I'm guessing that I'll need to write my own Reactor, Protocol, Factory, Application classes and possibly a new main module. Does this sound about right or am I over complicating things? Thanks for your help. ============================ Patrick Day uf069@freenet.victoria.bc.ca
From: "Patrick B. Day" <uf069@victoria.tc.ca> Subject: [Twisted-Python] Using Twisted for a clustering library. Date: Wed, 12 Jun 2002 14:27:08 -0700 (PDT)
I've been working on a cluster library for MOSIX and I've noticed that parts could possibly be replaced with some Twisted stuff.
The twisted.sister package is the beginnings of a clustering library. The requirements being used to drive it are rather MMPORPG-specific at this point, so the only code in it so far is a mechanism for "locking resources" so that a game server can atomically determine who "owns" a particular game object. The requirements that you described sound like the as-yet-unwritten "spawn server" component of twisted.sister. So, if you'd like to contribute work in this area, we'd love to have it! Alternatively you can wait for months to see if somebody will do it for you ;-) Also, as Itamar said, depending on your requirements, you shouldn't have to write anything aside from either a Protocol and Factory, or a PB client or server object. If you have a specific wire-protocol you need to speak, you'll need to write a Protocol and probably a Factory. Have you read Moshe Zadka's excellent HOWTO on the subject? If you just need to send some messages to a remote process, then Perspective Broker is probably your best bet. The documentation here is a little weaker (since there is a lot more high-level functionality available), but it's described at an overview level in the IPC10 paper, and there are several examples in the doc/examples package, as well as twisted.web.distrib and the "Reality" CVS module. -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
Patrick B. Day wrote:
After browsing through the Twisted source, I'm guessing that I'll need to write my own Reactor, Protocol, Factory, Application classes and possibly a new main module. Does this sound about right or am I over complicating things?
1) No need for new reactor class - let processes communicate via unix sockets (reactor.listenTCP). Or TCP (reactor.listenTCP). Or use reactor.spawnProcess and communicate via stdin/stdout. 2) No need for new Application class - use the existing one. Or just use the reactor object directly. 3) You can write your own protocol, which means a new Protocol and Factory, yes. Or you can use twisted.spread.pb, a remote object protocol, which almost certainly has all functionality you need. I recommend reading the writing servers tutorial: http://twistedmatrix.com/documents/howto/servers And look at simple example in tarball - doc/examples/simpleserv.py. Then read the API docs for reactors in twisted.internet.interfaces.
participants (3)
-
Glyph Lefkowitz -
Itamar Shtull-Trauring -
Patrick B. Day