On Fri, Feb 19, 2010 at 5:45 AM, Johann Borck <johann.borck@densedata.com> wrote:
I see. So if I get these requirements right you have 5-10 services with 1-30 instances of each, with following properties relevant to the task at hand:
1. those pollable by your existing program. 2. those incompatible with your existing program. 3. those that do not stream additional data. 4. those that do stream additional data.
since sets (1 and 2) and (3 and 4) are distinct, the combinations (1,3),(1,4),(2,3),(2,4) are possible.
(1,3) the easy one, obviously you just need an object that polls the service using your program in intervals specific to the service. (1,4) Question: will the data you're interested in be collected by your existing program or by twisted? In the former case, it's basically the same as (1,3), in the latter you'll have to implement a protocol. (2,3) for these you'll need to implement a protocol class. (2,4) here you'll have to implement one or two protocols, depending on how the service is implemented.
Is the above about correct? I think it would be a good idea to have an object OB that keeps references to all objects that gather data from the services, grouped by the type of service they're responsible for (defaultdict(list or dict) comes to mind). And then you'll probably either want factories that take care of handing newly created protocol instances over to OB or some instances (one or two per service in sets (2,3,4) ) of a multipurpose factory that can be initialized with the respective protocol and the information how to pass the created protocol over to OB, maybe just a simple method that is able to distinguish the protocols by the interfaces they implement.
One pitfall might be your polling program in case you're using t.i.u.getProcessOutput. It (t.i.u...) provides an asynchronous interface, so the worst that can happen is a stray process that doesn't return, but you still might want to consider implementing a ProcessProtocol (http://twistedmatrix.com/documents/current/core/howto/process.html) with a reasonable timeout, to be able to kill the spawned process in case it doesn't terminate. Thinking about it, it's the better solution anyway, because process protocols are just another type of protocol in twisted, and can be integrated consistently with the rest of your app.
Another utility you might want is t.i.task.LoopingCall, for obvious reasons. Given your requirements something along these lines would be my approach, although I'd probably reimplement the polling thingy in twisted if it's not too complex :)
hope that makes sense, Johann
Johann, Thank you very much for your detailed response. It is greatly appreciated. I think what you describe makes sense, and I hope I understand what you mean and how I can implement it: I need to create a "suberobject" OB which references all the poller-objects. I need to create one object per detected server instance, which takes care of creating a timed spawning of the external poller-process, and passing it along to OB. For those servers that require streamlisteners, I really only need one factory per server type, with the ablity to match streamed data with polled data via the poller-object. Is this somewhat on the right track? I absolutely hate creating lots of code just to find that it was done in the "wrong" way, structurally speaking, and that it requires a lot of work to rewrite in the "correct way" to allow for further expansion of the application. Your answers are very helpful in letting me avoid spaghetti-code. Cheers, Einar