[Twisted-Python] Implementing an event manager in twisted
Hi folks, I have had an idea to implement an event manager to glue a few different system components together, and I am wondering which pieces of twisted would be best suited to the task, or even if I should attempt it at all. I have programs that poll for data. I have a database that stores polled information. I have programs that listen for data to be provided to them from external entities. I have a desire to provide scheduler functionality (at time x, take action y). It seems to me that I could join all of these things via an event manager: A program polls for data, succeeds and notifies the event manager of the data that was polled. The event manager notifies a 'poll storage' object that stores the data in the database. It could also notify other things. Data is received from an external entity by a listener. The listener notifies the event manager of the data. The event manager notifies an object that stores it in the database, or runs a script, or uploads something to flickr, or whatever. Time passes. The event manager notifies interested objects that this is so. They take whatever action they deem necessary. None of this is new, so I wonder what others have done in this area? Am I heading down a dangerous path? What are the traps for new players? Has someone already written something that does all of this that I can use, saving myself time and headaches? -- Justin Warren <daedalus@eigenmagic.com>
Hi Justin, The scheduling is easily doable using twisted as a scheduler eg) callLater or task.LoopingCall. Writing an event dispatcher isn't too tough either. A very simple one allows you to register using a hash-able "name" object eg) a string. You call the dispatcher with the same object when you want to notify the registered handlers. Twisted has one see: http://twistedmatrix.com/docum<http://twistedmatrix.com/documents/current/api/twisted.python.dispatch.EventDispatcher.html> ents/current/api/twisted.python<http://twistedmatrix.com/documents/current/api/twisted.python.dispatch.Event...> .dispatch.EventDispatcher.html<http://twistedmatrix.com/documents/current/api/twisted.python.dispatch.Event...>. That could easily be tied in to generating events from a listener having received data. It seems that you have various programs that you want to coordinate using twisted. One way to accomplish this is like you said to write an event manager with twisted. Your listeners would connect (how? over sockets using existing protocols or write your own...or use Perspective Broker) to your twisted event manager to generate "received data" events. Your "received data" event handlers could spawn processes and pipe the data to them, or you can use twisted's adbapi to store the data in the database. On 4/11/07, Justin Warren <daedalus@eigenmagic.com> wrote:
Hi folks,
I have had an idea to implement an event manager to glue a few different system components together, and I am wondering which pieces of twisted would be best suited to the task, or even if I should attempt it at all.
I have programs that poll for data. I have a database that stores polled information. I have programs that listen for data to be provided to them from external entities.
I have a desire to provide scheduler functionality (at time x, take action y).
It seems to me that I could join all of these things via an event manager:
A program polls for data, succeeds and notifies the event manager of the data that was polled. The event manager notifies a 'poll storage' object that stores the data in the database. It could also notify other things.
Data is received from an external entity by a listener. The listener notifies the event manager of the data. The event manager notifies an object that stores it in the database, or runs a script, or uploads something to flickr, or whatever.
Time passes. The event manager notifies interested objects that this is so. They take whatever action they deem necessary.
None of this is new, so I wonder what others have done in this area? Am I heading down a dangerous path? What are the traps for new players? Has someone already written something that does all of this that I can use, saving myself time and headaches?
-- Justin Warren <daedalus@eigenmagic.com>
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, 12 Apr 2007 09:40:24 -0700, Manuel Arias <manuel.arias@gmail.com> wrote:
Hi Justin,
The scheduling is easily doable using twisted as a scheduler eg) callLater or task.LoopingCall.
Writing an event dispatcher isn't too tough either. A very simple one allows you to register using a hash-able "name" object eg) a string. You call the dispatcher with the same object when you want to notify the registered handlers. Twisted has one see: http://twistedmatrix.com/docum<http://twistedmatrix.com/documents/current/api/twisted.python.dispatch.EventDispatcher.html> ents/current/api/twisted.python<http://twistedmatrix.com/documents/current/api/twisted.python.dispatch.Event...> .dispatch.EventDispatcher.html<http://twistedmatrix.com/documents/current/api/twisted.python.dispatch.Event...>.
Note that this is deprecated and new software should not use it. Feel free to copy it into your own project if you think it's useful, but it hardly does anything at all. Jean-Paul
On 12:26 am, daedalus@eigenmagic.com wrote:
I have had an idea to implement an event manager to glue a few different system components together, and I am wondering which pieces of twisted would be best suited to the task, or even if I should attempt it at all.
Nothing about what you're suggesting sounds dangerous or weird. In fact, it really sounds like what Twisted is already :). If you want to write a layer over it to provide higher-level events, that's fine; most Twisted applications do something of the sort. The one thing I'd caution you against is attempting to make something at the same level of abstraction as Twisted by building on top of Twisted. That would just add complexity and overhead without any benefit. So here are some things you've described which sound, to me, exactly like what Twisted already does:
I have a desire to provide scheduler functionality (at time x, take action y).
This sounds like reactor.callLater(). (Or if you're talking about persistent scheduling, axiom's IScheduler.schedule).
Data is received from an external entity by a listener.
This sounds like IProtocol's dataReceived method.
A program polls for data, succeeds and notifies the event manager of the data that was polled. The event manager notifies a 'poll storage' object that stores the data in the database. It could also notify other things.
In python, "notifying things" is just calling functions or methods; in the most advanced case, it's still just calling functions or methods which have been placed into a list. Twisted has lots of such lists, and it doesn't seem worthwhile to me to unify them into a central structure. If it does make sense to you to unify them, you might want to look at Louie: http://pylouie.org/ . I've never used it and I don't really get what it's for, but several folks that I know seem to like it a lot for gluing events together.
On Thu, 2007-04-12 at 18:12 +0000, glyph@divmod.com wrote:
On 12:26 am, daedalus@eigenmagic.com wrote:
I have had an idea to implement an event manager to glue a few different system components together, and I am wondering which pieces of twisted would be best suited to the task, or even if I should attempt it at all.
Nothing about what you're suggesting sounds dangerous or weird. In fact, it really sounds like what Twisted is already :). If you want to write a layer over it to provide higher-level events, that's fine; most Twisted applications do something of the sort. The one thing I'd caution you against is attempting to make something at the same level of abstraction as Twisted by building on top of Twisted. That would just add complexity and overhead without any benefit. So here are some things you've described which sound, to me, exactly like what Twisted already does:
Whew. :) I certainly don't plan to re-implement Twisted, since it already exists and it's ace; I want something at a higher level. My aim here is to make my existing application(s) even more modular so that functionality can be added and extended in a simple, common way. This principle has already provided excellent benefits in the past.
I have a desire to provide scheduler functionality (at time x, take action y).
This sounds like reactor.callLater(). (Or if you're talking about persistent scheduling, axiom's IScheduler.schedule).
Yes, it will be persistent scheduling, so I'll check out IScheduler. "30 seconds have passed" would most likely be a reactor.callLater(30, timePassed) thing though, yes, inside an EventProducer type object.
Data is received from an external entity by a listener.
This sounds like IProtocol's dataReceived method.
Yes, and I use it to great effect within the existing code. By listener, I really mean a daemon process that listens, and there may be many of them. Rather than have a single, monolithic twistd with many Services, I think I would prefer a multi-process model so that each individual process can be started/stopped independently, new ones can be written that just talk to the event manager API, etc. In this way, the listener process would purely listen for incoming data (eg: SNMP Traps), and then talk to the event manager when it does. In fact, now that I think about it, it would be neat if the event manager API could allow third party programs to talk to the event manager via a standard protocol, such as XML-RPC. Such programs could then be written in the author's favourite language, which may not be Python+twisted, and potentially operate on remote systems.
A program polls for data, succeeds and notifies the event manager of the data that was polled. The event manager notifies a 'poll storage' object that stores the data in the database. It could also notify other things.
In python, "notifying things" is just calling functions or methods; in the most advanced case, it's still just calling functions or methods which have been placed into a list. Twisted has lots of such lists, and it doesn't seem worthwhile to me to unify them into a central structure.
True, though the more I think about it, I think I want an RPC mechanism of some kind, and I believe Perspective Broker is such a beast? I haven't really looked at it yet, but I've long been intrigued by what people have been saying about it. So it seems I want to add a veneer of RPC to my programs to allow them to communicate at a distance. Imagine a poller process on one host, an SNMP trap listener on another, a database storage process on still another. This sort of modularity would provide excellent scalability. Any suggestions for people's favourite ways of doing this with twisted?
If it does make sense to you to unify them, you might want to look at Louie: http://pylouie.org/ . I've never used it and I don't really get what it's for, but several folks that I know seem to like it a lot for gluing events together.
I'll check it out. Thanks for the great responses so far, too. :) -- Justin Warren <daedalus@eigenmagic.com>
Justin Warren wrote:
Any suggestions for people's favourite ways of doing this with twisted?
Perspective Broker, or Foolscap (aka newPB), or AMP. -- Nicola Larosa - http://www.tekNico.net/ Just being alive, It can really hurt And these moments given, Are a gift from time Just let us try, To give these moments back To those we love, To those who will survive -- Kate Bush, Moments Of Pleasure, Red Shoes, 1993
participants (5)
-
glyph＠divmod.com -
Jean-Paul Calderone -
Justin Warren -
Manuel Arias -
Nicola Larosa