[Twisted-Python] Is there a recommended way for a Service to request Application stop?
I have a hierarchy of Services some of which is MultiService and other parts are my own implementations of IServiceCollection - in some situations the a child Service may want to 'suggest' that the Application's job is done (error, or simply task completed) and I'm looking for some sort of standardized way to pass this info upstream. The idea being that I may using my implemented Services in a variety of Applications. In this type of situation, is it the general intention a child Service would use the Application directly, such that potential StopService()s could bubble down? Or is there a normal pattern here to have messages bubble up through the Services hierarchy? I'm not seeing anything like this in the examples I've found or looking through the sources, but I'm probably missing something. Cheers /dan -- Daniel Sutcliffe <dansut@gmail.com>
Services are just things that can be started and stopped. Application is just a top-level object that associates a thing-to-start with a few bits of global process-level state, like logging and pidfile settings. Therefore, the Service hierarchy abstraction is a poor fit for some code that needs to do some work and then exit; it's designed for long-running tools which can be started and stopped on demand. For example, what happens if two Service objects think that the Application's job is "done"? If you want to exit a process, calling `stop` on the reactor is generally the right way to go. But: talking about this in such vague, abstract terms is unlikely to be helpful. What, concretely, are you actually trying to do with the "Services in a variety of Applications"? -glyph
Thanks for your feedback Glyph, responses embedded below. On Oct 10, 2016, at 9:41 AM, Daniel Sutcliffe <dansut@gmail.com> wrote:
On Mon, Oct 10, 2016 at 5:51 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
This is perhap the type of thing I'm trying to code for; my goal is that the Services themselves do not have the final say in when the Application (or parent Service) is done. The event a Service feels is a problem may cause its parent to stop it, or it could simply call a method on it to remediate - depending on situation.
If you want to exit a process, calling `stop` on the reactor is generally the right way to go.
ie. specifically I would not want a Service to stop the reactor, as what it feels is a problem might only be a minor inconvenience for the Application as a whole; but in another Application, or Application state, it might be 'game over'.
I'll try to give one fairly concrete example of where I'd like to use this kind of pattern. The Services are long running polling ModBus clients whose configs are read from a DB by the parent, the child Service has no knowledge of where its config came from. Occasionally the child Services config may become totally unworkable (for a variety of reasons) and so they want to tell their parent the situation to give it a chance to reconfig, call child.stopService(), or...The parent will have many such Services, appropriate action may include a child reconfig, or if all children showing issues telling its parent the situation to ask for 'advice'. It seems to me in Twisted's Applications the job of calling stopService() (or similar) should always be the responsibility of the parent Service, but as a child how should I give my parent a clue I need attention? I can see implementing this with my own Interfaces so the child Services know more about their parent's Interfaces/attributes, or related objects, and can bubble information up through these, but I'd prefer my child Services to know as little about their parents as possible so they can be re-used in other simple Twisted apps. Basically, I didn't want to go implementing stuff when there were already tried and tested Twisted patterns for dealing with this kind of thing. Generically, can I somehow bubble up events through the Service hierarchy, or should I communicate with external objects to the hierarchy that can bubble down actions from higher up? Any suggestions, or pointers to similar examples will be most appreciated - I'm not stuck on this quandary, just wanting to write code which fits best with the Twisted way of doing things. Cheers /dan -- Daniel Sutcliffe <dansut@gmail.com>
Following the <https://en.wikipedia.org/wiki/Single_responsibility_principle <https://en.wikipedia.org/wiki/Single_responsibility_principle>>, the service hierarchy's job is just to make sure everything gets started up and shut down together. It sounds to me like you have a pretty well-defined hierarchy which seems like it fits into the service hierarchy because it's roughly parallel in terms of which objects participate; however, you have very application-specific semantics for this parallel hierarchy. For example, it's pretty unusual to have a super-service reconfigure a subordinate service in order to recover from an error condition, in my experience, unless you're talking about stuff like erlang supervision hierarchies, but that requires runtime support like the code being recovered running in a subprocess that doesn't share state. It often feels like abstractions are expensive so you should have as few of them as possible; but, in reality, *simple* abstractions are cheap, and what makes abstraction expensive is when you overload them. Make a new, simple abstraction that contains exactly the semantics you just described, and use composition to point at the appropriate point in the MultiService hierarchy. When it's time to "stop" a service, do setServiceParent(None); when it's time to "start" it, do setServiceParent(appropriateServiceParent). This should take care of keeping your services in the appropriate state. BTW, if you have stateful long-running services that have to self-modify based on changing circumstances, you might want to also check out https://github.com/glyph/automat <https://github.com/glyph/automat> to see if it can help you ensure that everything's in a consistent state. Good luck! -glyph
On Oct 10, 2016, at 4:38 PM, Daniel Sutcliffe <dansut@gmail.com> wrote:
On Mon, Oct 10, 2016 at 8:02 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Maybe this was my conceptual issue; the parallels were close enough that it just felt right to add the functionality to Twisted's Service Hierarchy - close enough that I was sucked into thoughts of modbusPollingClient IS-A Service... but keeping in mind the arguments of composition vs inheritance, I somehow finished up with modbusClientService HAS-A pollingLogic - whereas what I think you are recommending here is that that pollingLogic HAS-A modbusClientService and knows about a MultiService which it makes use of only to let the Application have overall control over starting and stopping all the active Services. Actually I just read this back through and am not sure this is actually really what you meant Glyph :-/ I do get that it seems I am grabbing the stick by the wrong end, I guess I'll just go back to reading more example code to see if I can find something that resonates with me as being close to what I'm trying to achieve.
I have to admit I had not even thought of using setServiceParent(None) to bring services down - looking at the source makes this concept clearer and potentially useful to me though
I'm still really just prototyping at this stage, trying to find an architecture that fits what I believe should be quite a simple application that is reasonably well suited to Twisted, but once I've passed this stage if I do find myself needing a FSM that is beyond totally simple then I'll certainly give this a look. Thanks. /dan -- Daniel Sutcliffe <dansut@gmail.com>
Services are just things that can be started and stopped. Application is just a top-level object that associates a thing-to-start with a few bits of global process-level state, like logging and pidfile settings. Therefore, the Service hierarchy abstraction is a poor fit for some code that needs to do some work and then exit; it's designed for long-running tools which can be started and stopped on demand. For example, what happens if two Service objects think that the Application's job is "done"? If you want to exit a process, calling `stop` on the reactor is generally the right way to go. But: talking about this in such vague, abstract terms is unlikely to be helpful. What, concretely, are you actually trying to do with the "Services in a variety of Applications"? -glyph
Thanks for your feedback Glyph, responses embedded below. On Oct 10, 2016, at 9:41 AM, Daniel Sutcliffe <dansut@gmail.com> wrote:
On Mon, Oct 10, 2016 at 5:51 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
This is perhap the type of thing I'm trying to code for; my goal is that the Services themselves do not have the final say in when the Application (or parent Service) is done. The event a Service feels is a problem may cause its parent to stop it, or it could simply call a method on it to remediate - depending on situation.
If you want to exit a process, calling `stop` on the reactor is generally the right way to go.
ie. specifically I would not want a Service to stop the reactor, as what it feels is a problem might only be a minor inconvenience for the Application as a whole; but in another Application, or Application state, it might be 'game over'.
I'll try to give one fairly concrete example of where I'd like to use this kind of pattern. The Services are long running polling ModBus clients whose configs are read from a DB by the parent, the child Service has no knowledge of where its config came from. Occasionally the child Services config may become totally unworkable (for a variety of reasons) and so they want to tell their parent the situation to give it a chance to reconfig, call child.stopService(), or...The parent will have many such Services, appropriate action may include a child reconfig, or if all children showing issues telling its parent the situation to ask for 'advice'. It seems to me in Twisted's Applications the job of calling stopService() (or similar) should always be the responsibility of the parent Service, but as a child how should I give my parent a clue I need attention? I can see implementing this with my own Interfaces so the child Services know more about their parent's Interfaces/attributes, or related objects, and can bubble information up through these, but I'd prefer my child Services to know as little about their parents as possible so they can be re-used in other simple Twisted apps. Basically, I didn't want to go implementing stuff when there were already tried and tested Twisted patterns for dealing with this kind of thing. Generically, can I somehow bubble up events through the Service hierarchy, or should I communicate with external objects to the hierarchy that can bubble down actions from higher up? Any suggestions, or pointers to similar examples will be most appreciated - I'm not stuck on this quandary, just wanting to write code which fits best with the Twisted way of doing things. Cheers /dan -- Daniel Sutcliffe <dansut@gmail.com>
Following the <https://en.wikipedia.org/wiki/Single_responsibility_principle <https://en.wikipedia.org/wiki/Single_responsibility_principle>>, the service hierarchy's job is just to make sure everything gets started up and shut down together. It sounds to me like you have a pretty well-defined hierarchy which seems like it fits into the service hierarchy because it's roughly parallel in terms of which objects participate; however, you have very application-specific semantics for this parallel hierarchy. For example, it's pretty unusual to have a super-service reconfigure a subordinate service in order to recover from an error condition, in my experience, unless you're talking about stuff like erlang supervision hierarchies, but that requires runtime support like the code being recovered running in a subprocess that doesn't share state. It often feels like abstractions are expensive so you should have as few of them as possible; but, in reality, *simple* abstractions are cheap, and what makes abstraction expensive is when you overload them. Make a new, simple abstraction that contains exactly the semantics you just described, and use composition to point at the appropriate point in the MultiService hierarchy. When it's time to "stop" a service, do setServiceParent(None); when it's time to "start" it, do setServiceParent(appropriateServiceParent). This should take care of keeping your services in the appropriate state. BTW, if you have stateful long-running services that have to self-modify based on changing circumstances, you might want to also check out https://github.com/glyph/automat <https://github.com/glyph/automat> to see if it can help you ensure that everything's in a consistent state. Good luck! -glyph
On Oct 10, 2016, at 4:38 PM, Daniel Sutcliffe <dansut@gmail.com> wrote:
On Mon, Oct 10, 2016 at 8:02 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Maybe this was my conceptual issue; the parallels were close enough that it just felt right to add the functionality to Twisted's Service Hierarchy - close enough that I was sucked into thoughts of modbusPollingClient IS-A Service... but keeping in mind the arguments of composition vs inheritance, I somehow finished up with modbusClientService HAS-A pollingLogic - whereas what I think you are recommending here is that that pollingLogic HAS-A modbusClientService and knows about a MultiService which it makes use of only to let the Application have overall control over starting and stopping all the active Services. Actually I just read this back through and am not sure this is actually really what you meant Glyph :-/ I do get that it seems I am grabbing the stick by the wrong end, I guess I'll just go back to reading more example code to see if I can find something that resonates with me as being close to what I'm trying to achieve.
I have to admit I had not even thought of using setServiceParent(None) to bring services down - looking at the source makes this concept clearer and potentially useful to me though
I'm still really just prototyping at this stage, trying to find an architecture that fits what I believe should be quite a simple application that is reasonably well suited to Twisted, but once I've passed this stage if I do find myself needing a FSM that is beyond totally simple then I'll certainly give this a look. Thanks. /dan -- Daniel Sutcliffe <dansut@gmail.com>
participants (2)
-
Daniel Sutcliffe -
Glyph Lefkowitz