[Twisted-Python] Need help for structured application development
Hi, I am planning to develop an application that will query a number of services at given intervals, and Twisted seems like a very nice platform to do so. However, because of little previous experience with Twisted, especially with building proper applications, using interfaces and using plugins, I could really need some pointers to get me started. In the following I will try to explain in broad terms what the application will do, and humbly ask for input on how I may structure it using factories, protocols etc.: 1. A number of more-or-less similar server applications (5-10) have a number of concurrent instances/servers (1-30) each. 2. I need to acquire the status of each instance at given intervals. These will typically range from every 20 seconds to every 3 minutes, and depend on the which service the instance is based on. 3. A list of the server instances shall be acquired via an XML-query every few minutes to pick up on new or removed instances. 4. An executable already exists which can be used to status-poll almost all services using different parameters. I will be calling it from my application to do the polling instead of recreating it in Python. 5. I need to be able to create my own polling-protocol for at least one of the service types which is incompatible with the executable. 6. Some services can not only be polled at intervals, but additionally stream information that I am interested in listening to. 7. At a later point in time, I wish to be able to communicate with IRC, MSN and other services/protocols using the queried and streamed data. My initial general thought is the following: 1. Design a class for each server application, where all classes implement the same interface. The class defines how to deal with server instances of its corresponding server application type. Classes are created as Twisted-plugins. 2. For each new server instance that is discovered, create an object which contains a timer and polls the server at specific intervals, writing the result to a globally accessible data structure. 3. For server instances that stream data continuously in addition to being polled at regular intervals, create a separate object which listens to the server. I believe I should somehow have a global data structure which contains references to all the server instance objects, regardless of type, and allows them to store data. This is sort of like the factory does for protocols, except in my case I don't think I can use a regular factory since I need both clients (to poll at intervals) and servers (to listen to streaming data). I suppose I could use a ClientFactory for the pollers and a Factory for the stream-listeners, and let them have a common data-store? But is it possible to use a single ClientFactory to spawn all poller-objects regardless of server type? Or do I need one ClientFactory for each server type? Do I perhaps need one ClientFactory per server instance? Or should I just ignore factories all together since it is primarily the timer and asyncronicity of Twisted I need? I am quite certain there is a correct twistified way of handling this sort of problem. I am just not sure what it is, and would sincerely welcome all feedback on how this should be done. Feel free to point to examples of applications that are similar to what I sketched above, and tell me to study the code :) Cheers, Einar
Hello Einar, This is a quite long email. What I can tell you, is that querying many command line applications with t.i.utils.getProcessOutput and gathering them in a DeferredList worked for me. Example: the list_cameras in http://svn.sat.qc.ca/trac/miville/browser/trunk/py/scenic/devices/cameras.py Good luck ! a 2010/2/15 Einar S. Idsø <einar.twisted@norsk-esport.no>:
Hi,
I am planning to develop an application that will query a number of services at given intervals, and Twisted seems like a very nice platform to do so. However, because of little previous experience with Twisted, especially with building proper applications, using interfaces and using plugins, I could really need some pointers to get me started.
In the following I will try to explain in broad terms what the application will do, and humbly ask for input on how I may structure it using factories, protocols etc.:
1. A number of more-or-less similar server applications (5-10) have a number of concurrent instances/servers (1-30) each. 2. I need to acquire the status of each instance at given intervals. These will typically range from every 20 seconds to every 3 minutes, and depend on the which service the instance is based on. 3. A list of the server instances shall be acquired via an XML-query every few minutes to pick up on new or removed instances. 4. An executable already exists which can be used to status-poll almost all services using different parameters. I will be calling it from my application to do the polling instead of recreating it in Python. 5. I need to be able to create my own polling-protocol for at least one of the service types which is incompatible with the executable. 6. Some services can not only be polled at intervals, but additionally stream information that I am interested in listening to. 7. At a later point in time, I wish to be able to communicate with IRC, MSN and other services/protocols using the queried and streamed data.
My initial general thought is the following: 1. Design a class for each server application, where all classes implement the same interface. The class defines how to deal with server instances of its corresponding server application type. Classes are created as Twisted-plugins. 2. For each new server instance that is discovered, create an object which contains a timer and polls the server at specific intervals, writing the result to a globally accessible data structure. 3. For server instances that stream data continuously in addition to being polled at regular intervals, create a separate object which listens to the server.
I believe I should somehow have a global data structure which contains references to all the server instance objects, regardless of type, and allows them to store data. This is sort of like the factory does for protocols, except in my case I don't think I can use a regular factory since I need both clients (to poll at intervals) and servers (to listen to streaming data). I suppose I could use a ClientFactory for the pollers and a Factory for the stream-listeners, and let them have a common data-store? But is it possible to use a single ClientFactory to spawn all poller-objects regardless of server type? Or do I need one ClientFactory for each server type? Do I perhaps need one ClientFactory per server instance? Or should I just ignore factories all together since it is primarily the timer and asyncronicity of Twisted I need?
I am quite certain there is a correct twistified way of handling this sort of problem. I am just not sure what it is, and would sincerely welcome all feedback on how this should be done. Feel free to point to examples of applications that are similar to what I sketched above, and tell me to study the code :)
Cheers, Einar
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Alexandre Quessy http://alexandre.quessy.net/
Yeah, sorry about the long mail. I thought it would be a good idea to explain my app and its requirements, and thereby get better advice on how to structure it, than having to iterate to and from this list a number of times. Perhaps that was the wrong strategy. Thanks for your assistance. I've looked at t.i.utils.getProcessOutput, and it is what I need to run my query software. Unfortunately that is only 5% of the puzzle - my real problem is working out what kind of top-level classes I should be using to spawn my service-checkers as they are needed. I guess I need a factory that launches server checker objects not as a result of a TCP/UDP connection, but as a result of new servers being detected. This is not how I've seen factories being used regularly, and I would love to get some input on this. If I am not making sense at all, please let me know :) Cheers, Einar On Tue, Feb 16, 2010 at 11:57 AM, Alexandre Quessy <alexandre@quessy.net> wrote:
Hello Einar, This is a quite long email. What I can tell you, is that querying many command line applications with t.i.utils.getProcessOutput and gathering them in a DeferredList worked for me. Example: the list_cameras in http://svn.sat.qc.ca/trac/miville/browser/trunk/py/scenic/devices/cameras.py
Good luck !
a
2010/2/15 Einar S. Idsø <einar.twisted@norsk-esport.no>:
Hi,
I am planning to develop an application that will query a number of services at given intervals, and Twisted seems like a very nice platform to do so. However, because of little previous experience with Twisted, especially with building proper applications, using interfaces and using plugins, I could really need some pointers to get me started.
In the following I will try to explain in broad terms what the application will do, and humbly ask for input on how I may structure it using factories, protocols etc.:
1. A number of more-or-less similar server applications (5-10) have a number of concurrent instances/servers (1-30) each. 2. I need to acquire the status of each instance at given intervals. These will typically range from every 20 seconds to every 3 minutes, and depend on the which service the instance is based on. 3. A list of the server instances shall be acquired via an XML-query every few minutes to pick up on new or removed instances. 4. An executable already exists which can be used to status-poll almost all services using different parameters. I will be calling it from my application to do the polling instead of recreating it in Python. 5. I need to be able to create my own polling-protocol for at least one of the service types which is incompatible with the executable. 6. Some services can not only be polled at intervals, but additionally stream information that I am interested in listening to. 7. At a later point in time, I wish to be able to communicate with IRC, MSN and other services/protocols using the queried and streamed data.
My initial general thought is the following: 1. Design a class for each server application, where all classes implement the same interface. The class defines how to deal with server instances of its corresponding server application type. Classes are created as Twisted-plugins. 2. For each new server instance that is discovered, create an object which contains a timer and polls the server at specific intervals, writing the result to a globally accessible data structure. 3. For server instances that stream data continuously in addition to being polled at regular intervals, create a separate object which listens to the server.
I believe I should somehow have a global data structure which contains references to all the server instance objects, regardless of type, and allows them to store data. This is sort of like the factory does for protocols, except in my case I don't think I can use a regular factory since I need both clients (to poll at intervals) and servers (to listen to streaming data). I suppose I could use a ClientFactory for the pollers and a Factory for the stream-listeners, and let them have a common data-store? But is it possible to use a single ClientFactory to spawn all poller-objects regardless of server type? Or do I need one ClientFactory for each server type? Do I perhaps need one ClientFactory per server instance? Or should I just ignore factories all together since it is primarily the timer and asyncronicity of Twisted I need?
I am quite certain there is a correct twistified way of handling this sort of problem. I am just not sure what it is, and would sincerely welcome all feedback on how this should be done. Feel free to point to examples of applications that are similar to what I sketched above, and tell me to study the code :)
Cheers, Einar
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Alexandre Quessy http://alexandre.quessy.net/
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Feb 17, 2010, at 1:14 PM, Einar S. Idsø wrote:
Yeah, sorry about the long mail. I thought it would be a good idea to explain my app and its requirements, and thereby get better advice on how to structure it, than having to iterate to and from this list a number of times. Perhaps that was the wrong strategy.
It is definitely the right strategy :). If you provide these requirements, then it's much easier to just provide an answer, rather than having to ask you for more about what you're really trying to do. Please feel free to include lots of detail in your next question too!
[...] I guess I need a factory that launches server checker objects not as a result of a TCP/UDP connection, but as a result of new servers being detected. How does this detection work? Are those servers based on twisted? Also, do you control them (start/stop/configuration), do you have access to
Einar S. Idsø wrote: the machines they're on, or are you detecting them over network only? Johann
On Thu, Feb 18, 2010 at 11:36 AM, Johann Borck <johann.borck@densedata.com> wrote:
[...] I guess I need a factory that launches server checker objects not as a result of a TCP/UDP connection, but as a result of new servers being detected. How does this detection work? Are those servers based on twisted? Also, do you control them (start/stop/configuration), do you have access to
Einar S. Idsø wrote: the machines they're on, or are you detecting them over network only?
Johann: Thanks for your response. I do control these services myself, and they run on our own hosts, but they are created and spawned via an external web interface which also has a Soap API. I do not plan on starting/stopping them via this application I'm developing. I just need to get the list of servers and start polling them. I get the list by calling the Soap webservice. It simply outputs a list of servers, and it is my job to query each and every one at specific intervals for status. I was thinking I would have the Soap-query run as a timed function every few minutes, and compare the new list with the old to work out any new/removed servers. Whenever a new server is identified, I will spawn a new object to deal with it, including, but not limited to, polling at given intervals. What kind of object should this be? A ClientFactory? Another kind of object that is already defined somewhere in Twisted? A completely custom object, built from scratch? I am leaning towards ClientFactory, but I am definitely not certain. Glyph: Thanks for letting me know I'm at least asking correctly :) Cheers, Einar
Einar S. Idsø wrote:
On Thu, Feb 18, 2010 at 11:36 AM, Johann Borck <johann.borck@densedata.com> wrote:
Einar S. Idsø wrote:
[...] I guess I need a factory that launches server checker objects not as a result of a TCP/UDP connection, but as a result of new servers being detected.
How does this detection work? Are those servers based on twisted? Also, do you control them (start/stop/configuration), do you have access to the machines they're on, or are you detecting them over network only?
Johann: Thanks for your response. I do control these services myself, and they run on our own hosts, but they are created and spawned via an external web interface which also has a Soap API. I do not plan on starting/stopping them via this application I'm developing. I just need to get the list of servers and start polling them. I get the list by calling the Soap webservice. It simply outputs a list of servers, and it is my job to query each and every one at specific intervals for status.
I was thinking I would have the Soap-query run as a timed function every few minutes, and compare the new list with the old to work out any new/removed servers. Whenever a new server is identified, I will spawn a new object to deal with it, including, but not limited to, polling at given intervals. What kind of object should this be? A ClientFactory? Another kind of object that is already defined somewhere in Twisted? A completely custom object, built from scratch? I am leaning towards ClientFactory, but I am definitely not certain.
I see. So if I get these requirements right you have 5-10 services with 1-30 instances of each, with following properties relevant to the task at hand: 1. those pollable by your existing program. 2. those incompatible with your existing program. 3. those that do not stream additional data. 4. those that do stream additional data. since sets (1 and 2) and (3 and 4) are distinct, the combinations (1,3),(1,4),(2,3),(2,4) are possible. (1,3) the easy one, obviously you just need an object that polls the service using your program in intervals specific to the service. (1,4) Question: will the data you're interested in be collected by your existing program or by twisted? In the former case, it's basically the same as (1,3), in the latter you'll have to implement a protocol. (2,3) for these you'll need to implement a protocol class. (2,4) here you'll have to implement one or two protocols, depending on how the service is implemented. Is the above about correct? I think it would be a good idea to have an object OB that keeps references to all objects that gather data from the services, grouped by the type of service they're responsible for (defaultdict(list or dict) comes to mind). And then you'll probably either want factories that take care of handing newly created protocol instances over to OB or some instances (one or two per service in sets (2,3,4) ) of a multipurpose factory that can be initialized with the respective protocol and the information how to pass the created protocol over to OB, maybe just a simple method that is able to distinguish the protocols by the interfaces they implement. One pitfall might be your polling program in case you're using t.i.u.getProcessOutput. It (t.i.u...) provides an asynchronous interface, so the worst that can happen is a stray process that doesn't return, but you still might want to consider implementing a ProcessProtocol (http://twistedmatrix.com/documents/current/core/howto/process.html) with a reasonable timeout, to be able to kill the spawned process in case it doesn't terminate. Thinking about it, it's the better solution anyway, because process protocols are just another type of protocol in twisted, and can be integrated consistently with the rest of your app. Another utility you might want is t.i.task.LoopingCall, for obvious reasons. Given your requirements something along these lines would be my approach, although I'd probably reimplement the polling thingy in twisted if it's not too complex :) hope that makes sense, Johann
On Fri, Feb 19, 2010 at 5:45 AM, Johann Borck <johann.borck@densedata.com> wrote:
I see. So if I get these requirements right you have 5-10 services with 1-30 instances of each, with following properties relevant to the task at hand:
1. those pollable by your existing program. 2. those incompatible with your existing program. 3. those that do not stream additional data. 4. those that do stream additional data.
since sets (1 and 2) and (3 and 4) are distinct, the combinations (1,3),(1,4),(2,3),(2,4) are possible.
(1,3) the easy one, obviously you just need an object that polls the service using your program in intervals specific to the service. (1,4) Question: will the data you're interested in be collected by your existing program or by twisted? In the former case, it's basically the same as (1,3), in the latter you'll have to implement a protocol. (2,3) for these you'll need to implement a protocol class. (2,4) here you'll have to implement one or two protocols, depending on how the service is implemented.
Is the above about correct? I think it would be a good idea to have an object OB that keeps references to all objects that gather data from the services, grouped by the type of service they're responsible for (defaultdict(list or dict) comes to mind). And then you'll probably either want factories that take care of handing newly created protocol instances over to OB or some instances (one or two per service in sets (2,3,4) ) of a multipurpose factory that can be initialized with the respective protocol and the information how to pass the created protocol over to OB, maybe just a simple method that is able to distinguish the protocols by the interfaces they implement.
One pitfall might be your polling program in case you're using t.i.u.getProcessOutput. It (t.i.u...) provides an asynchronous interface, so the worst that can happen is a stray process that doesn't return, but you still might want to consider implementing a ProcessProtocol (http://twistedmatrix.com/documents/current/core/howto/process.html) with a reasonable timeout, to be able to kill the spawned process in case it doesn't terminate. Thinking about it, it's the better solution anyway, because process protocols are just another type of protocol in twisted, and can be integrated consistently with the rest of your app.
Another utility you might want is t.i.task.LoopingCall, for obvious reasons. Given your requirements something along these lines would be my approach, although I'd probably reimplement the polling thingy in twisted if it's not too complex :)
hope that makes sense, Johann
Johann, Thank you very much for your detailed response. It is greatly appreciated. I think what you describe makes sense, and I hope I understand what you mean and how I can implement it: I need to create a "suberobject" OB which references all the poller-objects. I need to create one object per detected server instance, which takes care of creating a timed spawning of the external poller-process, and passing it along to OB. For those servers that require streamlisteners, I really only need one factory per server type, with the ablity to match streamed data with polled data via the poller-object. Is this somewhat on the right track? I absolutely hate creating lots of code just to find that it was done in the "wrong" way, structurally speaking, and that it requires a lot of work to rewrite in the "correct way" to allow for further expansion of the application. Your answers are very helpful in letting me avoid spaghetti-code. Cheers, Einar
participants (4)
-
Alexandre Quessy -
Einar S. Idsø -
Glyph Lefkowitz -
Johann Borck