[Twisted-Python] New to twisted, need a bit of high-level design help.

Hello all, I'm new to using twisted and am attempting to implement a client and server for use a distributed computing environment with a centralized task server. Clients query the task server for a work list, apply a bunch of transforms to this work list and then return the transformed work list back to the server. The server, for its part, has to send out these work lists, keep track of which client is processing what (for timeout handling) and collate the transformed work lists so that it can writing the data out to disk in a sorted manner. I've spent the better part of a week banging out prototypes, reading the docs and asking questions on IRC (sorry, dear IRC folks, for all the timeouts), but the things I've built feel clunky. I have primarily focused my efforts on using t.s.pb and t.p.amp as I control both ends of the wire. t.p.amp is probably too limited, though, as the values in packet hashes must be <= 65535 bytes, too small for my purposes unless I break my loads into peices, which is not an obvious or easy thing to do given the structure of my computed data. t.s.pb's documentation seems to concern itself with examples involiving client requests, server responses and clients working on the response with no communication back to the server and, though it'd be nicer to work with probably, I can't seem to figure out how to cleanly implement communication back to the server. My prototype server is relatively clean, supporting a remote_checkin which clients call with a valid, unique ID to receive data and a remote_checkout, which clients call with their computed data to register it with the server. The clients, however, are a mess, mostly because the client transforms are not simple and might fail in a multitude of ways. (I really rather need to keep a reference around to the server so I can arbitrarily push data toward it). As my attempts to use twisted for this purpose have been frustrated, due to my inexperience, I'm sure, would it be possible for someone to explain how I could use twisted in the construction of a distributed system with a central data collection and collation server? Any comments are greatly appreciated, from critiques on the bits of twisted I've attempted to use to suggestions for new areas to explore to code samples. (I must apologize that I cannot give any of my prototype code as I do not own the copyright on it.) I've worked myself into rather a rut and need help climbing out. Thanks, Brian

Brian L. Troutwine wrote:
http://foss.eepatents.com/Twisted-Goodies/browser/trunk/twisted_goodies/asyn... Best regards, Ed Suominen

Brian L. Troutwine wrote:
Thanks! That's exactly what I was looking for.
You're welcome. Twisted is a great piece of code, and I'm happy to help people put it to work as I have. I'm actually going to be working on the AsynCluster package some more in the next week or two. Feel free to contact me offline with any questions or problems. Best regards, Ed

Brian L. Troutwine wrote:
http://foss.eepatents.com/Twisted-Goodies/browser/trunk/twisted_goodies/asyn... Best regards, Ed Suominen

Brian L. Troutwine wrote:
Thanks! That's exactly what I was looking for.
You're welcome. Twisted is a great piece of code, and I'm happy to help people put it to work as I have. I'm actually going to be working on the AsynCluster package some more in the next week or two. Feel free to contact me offline with any questions or problems. Best regards, Ed
participants (2)
-
Brian L. Troutwine
-
Ed Suominen