[Twisted-Python] Looking for help dealing with ClientService reconnections
![](https://secure.gravatar.com/avatar/ad5d35adba8666840c8ce22c1a43ef1f.jpg?s=120&d=mm&r=g)
Hello all, my first post here - only been using Twisted for about a month and am also a relative newcomer to Python but have been coding professionally for 20+ years. I was attracted to Twisted and Python for a particular project purely because after research it seemed to be the best tool for the job, and have actually been enjoying both Python and Twisted much more than I ever thought I would. The project I am coding towards is creating a sensor data collection gateway. First iteration needs are simply pulling data from ModBus TCP slave PLCs and writing it to a MySQL database, but goals beyond that are making the source of the data and its destination(s) very flexible(pluggable). Therefore I am trying to create a good clean architecture from the outset so as I iterate forwards I don't finish up having to take too many steps backwards before heading forwards. I am using pymodbus to pull the data which works well for my devices, has a twisted async API, and have created more than a few prototypes that demonstrate all works as I expect. Where I am a bit stalled is getting to grips with a good architecture that fulfills my needs - my intention is that the application that meets my first goal will be a twistd plugin. The new ClientService class seems like it will fit my needs very closely but I am struggling with how to handle the reconnections... I have been using the whenConnected() method to grab the Protocol for the initial connection and then use a method of this to poll the connected slave. When the connection is lost I get an errback from this method's deferred which I use as a signal to abandon the Protocol and call whenConnected() again... at this point I have an issue though as the returned deferred immediately gives me a callback with the same Protocol which has just lost its connection, and thus loop... Before I got on this mailing list I posted this Q to stackoverflow with some example code: http://stackoverflow.com/q/37061807/3448214 but no solution or much attention there yet. As I say there, I realize I have probably just made a bad pattern choice for how to use this API, but I have not been able to work out a better choice which seems clean and fits my needs/understanding well. I have tried deriving my own Protocol/Factory and handling the polling there but this seems to get really messy once I start to add code to get the collected data to a destination at that level, involving giving the Protocol too much knowledge of how the data is to be handled. Any advice, good patterns, or pointers to other projects which do something similar is appreciated, Cheers /dan -- Daniel Sutcliffe <dansut@gmail.com>
![](https://secure.gravatar.com/avatar/12f8ad89084afabd2769ad2ef8c10e12.jpg?s=120&d=mm&r=g)
I'm still working through your code example and trying to get a better grasp of what, exactly, you're trying to implement as far as client behavior is concerned. In the meantime, it sounds, on the surface, like you are trying to implement a form of the ReconnectingClientFactory: http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Rec... . Your client protocol must be factory aware and call the resetDelay method on the factory when a connection is made, but the factory will handle reconnecting and generating a new protocol instance. Is that similar to what you are trying to accomplish? Maybe see also http://twistedmatrix.com/documents/current/core/howto/clients.html#reconnect... for a quick example of using the reconnecting factory. On Fri, May 6, 2016, 12:23 Daniel Sutcliffe <dansut@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 7, 2016, at 12:19 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
I'm still working through your code example and trying to get a better grasp of what, exactly, you're trying to implement as far as client behavior is concerned. In the meantime, it sounds, on the surface, like you are trying to implement a form of the ReconnectingClientFactory: http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Rec... <http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Rec...>.
Just to clear up this point: ReconnectingClientFactory is the old, bad way of doing things, that only works with connectTCP/connectSSL/connectUNIX et. al., and not with endpoints. ClientService <https://twistedmatrix.com/documents/16.1.1/api/twisted.application.internet....> is the new, good way, only available since the latest release. -glyph
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 7, 2016, at 4:02 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
Thanks, glyph, for the correction. Also, thanks for reminding me that I'm still living in twisted's past...
Just a reminder that there's always cool new stuff on the horizon :). In addition to just working with endpoints, the interface to ClientService is also a bit more flexible and testable, since the retry policy is implemented by a completely separate object from the service itself. -glyph
![](https://secure.gravatar.com/avatar/ad5d35adba8666840c8ce22c1a43ef1f.jpg?s=120&d=mm&r=g)
Thanks for the response Kevin, a few individual replies embedded below: On Sat, May 7, 2016 at 3:19 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
pymodbus actually has a pymodbus.client.async.ModbusClientFactory which is derived from twisted.internet.protocol.ReconnectingClientFactory http://pymodbus.readthedocs.io/en/latest/library/async-client.html
I had built working tests using the docs you link to and understand the way it works, but when Twisted 16.1 came out I decided that as I was writing fresh code I probably ought to use the suggested new API. After reading up and checking the source for ClientService it seemed just what I was after; offering the advantages that Glyph mentioned, and simplifying the amount of code I had to write (and maintain), especially as my intention was to use twistd and thus Services. It was only when I started to implement this that I felt my inexperience with Twisted's ways of doing things was holding me back and couldn't see a clear direction of how to build my architecture around it. Initially I used my own Protocol class derived from pymodbus.client.async.ModbusClientProtocol that handled the polling internally with a Factory that was aware of this class based on ClientFactory (not ReconnectingClientFactory) that stored the persistent info (poll rate, etc). This worked well with ClientService and reconnections happened as expected but once I started to try and evolve my code to consolidate (from many ClientServices polling various PLCs) and write out the data (to MySQL) the model felt like the focus was in the wrong place and that maybe I should be focussing more on making a derivation of the ClientService have more control of the polling and just using the Protocol provided by pymodbus as-is. After hitting the problem I described below, I decided it was time to ask those with more Twisted experience what the most appropriate way to approach this problem was. Am I better off going back to doing polling in Protocol with a Factory that provides ways to pump the data back upstream to where it needs to be consolidated, and not rolling my own ClientService to control it? or do I just need a better way of getting my ClientService to be aware of the latest connection? Maybe there's a pathway in between, or even something I haven't yet thought of? If anyone does have the time and desire to help me with this but feels they need more info from me or background then please just ask, and I will provide and be most appreciative :) Cheers /dan On Fri, May 6, 2016, 12:23 Daniel Sutcliffe <dansut@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
If you want a hook each time a new protocol is created, you're probably better off writing a wrapper protocol factory, and passing that to your ClientService, then doing any set-up work you want to do in your buildProtocol implementation, which delegates to the real, pymodbus implementation.
I am curious as to why you say that this is "messy".
Any advice, good patterns, or pointers to other projects which do something similar is appreciated,
I spent a while thinking about your question, and I'm sorry that I can't give a more thorough answer, but I think you need to be a bit more specific about what it is you don't like about your potential solution. It seems to me that having a delegating Factory, especially if all you need to do is set up some state on each Protocol that gets produced, should be sufficient... -glyph
![](https://secure.gravatar.com/avatar/ad5d35adba8666840c8ce22c1a43ef1f.jpg?s=120&d=mm&r=g)
Thanks Glyph, I think you have given me a push back in the 'right' direction - more thoughts and commentary embedded below if you, or anyone else, has the time. On May 6, 2016, at 10:19 AM, Daniel Sutcliffe <dansut@gmail.com> wrote: [...]
On Mon, May 9, 2016 at 6:19 PM, Glyph <glyph@twistedmatrix.com> wrote:
Understood, if this is the way the framework is intended to be used I realize doing anything else is going to be fighting against the flow. However, just to probe the situation I found myself in further, for the sake probing broken code to see how it might be fixed: Given the ClientService.whenConnected() method is intended to provide access to my connected Protocol through the deferred it returns, is it not a little unfriendly that this Protocol may turn out to be disconnected? OK occasionally due to timing but for this to be a possible condition which can loop with the same disconnected Protocol returned until the ClientService has its _currentConnection set to None, suggests to me that I can't safely use my Protocol from whenConnected() for much other than as a notification the first connection has occurred... but how do I avoid this? I have looked at the source and it seems to me the fact that the connection has been lost should bubble up to the ClientService through a t.a.i._DisconnectFactory and t.a.i._ReconnectingProtocolProxy once my Protocol's connectionLost() is called. My issue seems to be that I errback on a Protocol method's deferred returned to code at or above the ClientService level which gives up on that Protocol and calls whenConnected() to get the next one, only the Protocol's connectionLost() has yet to be called and then doesn't have chance to because my code is looping around calling whenConnected() and getting the same Protocol back. I hope that makes sense :-/ My Q on this is if I should be internally calling my Protocol's connectionLost() so it can bubble up to the ClientService before I errback on the Protocol method - whose responsibility is it to call this?
Honestly, this was just a gut feeling at the time, probably more sourced in my implementation from lack of experience in using Twisted; after reading around the subject, looking at many more examples, and your advice, I think I am convinced I need to back to looking at my own Protocol derived from the pymodbus one with a Factory that contains the persistent config and access to an interface to pump the polled data upstream.
Looking at the code again I think it just seemed to make sense to me at the time to have something that is (or has) a ClientService be the object I am calling a DataSource have more control over when it polls that data and what it does with it - and not to have to make the Protocol and its Factory aware of this at all. Does that make it any clearer? I am happy to push ahead with building this into my Factory/Protocol if that is more normal usage for Twisted as I am sure there will be benefits of encapsulating it here beyond the other side I was seeing during that moment of confusion. Thanks for the advice, I think it was enough to nudge me in a direction that will work better to get me going, and if with more experience other usage makes more sense I can always refactor, refactor, refactor :) Cheers /dan -- Daniel Sutcliffe <dansut@gmail.com>
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
whenConnected() is not intended to be used for "give me each Protocol as it is instantiated so that state can be set up", it is intended for API clients which want to send a message to the current connection to just retrieve the current connection so they can call a method on it. I'm not sure what you mean by "turn out to be disconnected". The physical reality of networking is that you might always encounter a transport which has been disconnected but which you haven't received notification of its disconnection yet.
Let me try to rephrase: you call a protocol method which returns a Deferred; you add an errback to that Deferred which calls whenConnected() to re-try, but since the protocol hasn't disconnected yet, you get the same protocol instance back, which is useless to you.
It's the framework's responsibility to call it. You should not call it yourself. Your Protocol's connectionLost isn't going to bubble up to ClientService anyway; you'd have to call your wrapper's connectionLost, which would confuse its internal state, since the framework would call it again right afterwards, and we definitely don't have test coverage for that, since the framework will normally only call it once. The right way to handle this would be to introduce a delay between re-tries. It's generally a good idea to have such a delay for lots of reasons; you don't want to overload your peer in the case of a transient failure. As a bonus, the fact that you've gone back up to the reactor loop to wait a while means that the transport will be properly disconnected and whenConnected() will do what you want.
Yup! Happy to help, sorry for the super long lag time on this reply, but my email queue has been pretty full lately :) -glyph
![](https://secure.gravatar.com/avatar/ad5d35adba8666840c8ce22c1a43ef1f.jpg?s=120&d=mm&r=g)
Delayed response here but hopefully it is still seen as worthwhile discussion, sorry if it is seems to just be beating a dead horse ;) On May 10, 2016, at 9:52 AM, Daniel Sutcliffe <dansut@gmail.com> wrote:
I appreciate now this 'protocol/connection init' isn't what the whenConnected() method was intended for, and it wasn't what I was using it for when I came across my issue. However there seems to me to be no documentary discouragement for using it this way... at least for someone who is not as familiar with Twisted's common usage patterns
The 'turns out to be disconnected' actually came from my acceptance that network connections drop/fail and I have little control of how or when this happens. Thus I may call a method on a Protocol at any time only to find that it is not in a good state to handle my request. My (misguided) goal was to call a method of the Protocol returned from whenConnected() every so often to give it a task to do; my expectation being that *when* the connection failed I would find out from the task method's returned deferred and could then just fire off another ClientService whenConnected() to get the next useful Protocol once it is established, and then use this to continue giving the Protocol tasks until it also ultimately fails...
Thanks Glyph, that seems like exactly what I was trying to say.
I'm glad that I got this mostly right in my head - it didn't feel right for me to be calling this, or any similar method I could find.
I didn't even consider this as my thinking was along the lines of the ClientService having the ability to handle the connection re-tries and delays to avoid any overloading of the service it wraps. It would be nice if there was something I could do in my Protocol task method such that it didn't fire the errback on its deferred until its Factory (and ultimately its Service in this case) had been made aware the connection had failed. Does this not have to be done elsewhere in Twisted or other projects using Twisted, or is the common pattern just to introduce a delay to avoid any possible race condition? My current intention is to have the Protocol's Factory handle the assigning of tasks to its Protocol which seems the 'more normal' direction to take, so any discussions above are purely for my education and intellectual curiosity - I'm glad of any and all feedback. Cheers /dan -- Daniel Sutcliffe <dansut@gmail.com>
![](https://secure.gravatar.com/avatar/12f8ad89084afabd2769ad2ef8c10e12.jpg?s=120&d=mm&r=g)
I'm still working through your code example and trying to get a better grasp of what, exactly, you're trying to implement as far as client behavior is concerned. In the meantime, it sounds, on the surface, like you are trying to implement a form of the ReconnectingClientFactory: http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Rec... . Your client protocol must be factory aware and call the resetDelay method on the factory when a connection is made, but the factory will handle reconnecting and generating a new protocol instance. Is that similar to what you are trying to accomplish? Maybe see also http://twistedmatrix.com/documents/current/core/howto/clients.html#reconnect... for a quick example of using the reconnecting factory. On Fri, May 6, 2016, 12:23 Daniel Sutcliffe <dansut@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 7, 2016, at 12:19 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
I'm still working through your code example and trying to get a better grasp of what, exactly, you're trying to implement as far as client behavior is concerned. In the meantime, it sounds, on the surface, like you are trying to implement a form of the ReconnectingClientFactory: http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Rec... <http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Rec...>.
Just to clear up this point: ReconnectingClientFactory is the old, bad way of doing things, that only works with connectTCP/connectSSL/connectUNIX et. al., and not with endpoints. ClientService <https://twistedmatrix.com/documents/16.1.1/api/twisted.application.internet....> is the new, good way, only available since the latest release. -glyph
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 7, 2016, at 4:02 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
Thanks, glyph, for the correction. Also, thanks for reminding me that I'm still living in twisted's past...
Just a reminder that there's always cool new stuff on the horizon :). In addition to just working with endpoints, the interface to ClientService is also a bit more flexible and testable, since the retry policy is implemented by a completely separate object from the service itself. -glyph
![](https://secure.gravatar.com/avatar/ad5d35adba8666840c8ce22c1a43ef1f.jpg?s=120&d=mm&r=g)
Thanks for the response Kevin, a few individual replies embedded below: On Sat, May 7, 2016 at 3:19 PM, Kevin Conway <kevinjacobconway@gmail.com> wrote:
pymodbus actually has a pymodbus.client.async.ModbusClientFactory which is derived from twisted.internet.protocol.ReconnectingClientFactory http://pymodbus.readthedocs.io/en/latest/library/async-client.html
I had built working tests using the docs you link to and understand the way it works, but when Twisted 16.1 came out I decided that as I was writing fresh code I probably ought to use the suggested new API. After reading up and checking the source for ClientService it seemed just what I was after; offering the advantages that Glyph mentioned, and simplifying the amount of code I had to write (and maintain), especially as my intention was to use twistd and thus Services. It was only when I started to implement this that I felt my inexperience with Twisted's ways of doing things was holding me back and couldn't see a clear direction of how to build my architecture around it. Initially I used my own Protocol class derived from pymodbus.client.async.ModbusClientProtocol that handled the polling internally with a Factory that was aware of this class based on ClientFactory (not ReconnectingClientFactory) that stored the persistent info (poll rate, etc). This worked well with ClientService and reconnections happened as expected but once I started to try and evolve my code to consolidate (from many ClientServices polling various PLCs) and write out the data (to MySQL) the model felt like the focus was in the wrong place and that maybe I should be focussing more on making a derivation of the ClientService have more control of the polling and just using the Protocol provided by pymodbus as-is. After hitting the problem I described below, I decided it was time to ask those with more Twisted experience what the most appropriate way to approach this problem was. Am I better off going back to doing polling in Protocol with a Factory that provides ways to pump the data back upstream to where it needs to be consolidated, and not rolling my own ClientService to control it? or do I just need a better way of getting my ClientService to be aware of the latest connection? Maybe there's a pathway in between, or even something I haven't yet thought of? If anyone does have the time and desire to help me with this but feels they need more info from me or background then please just ask, and I will provide and be most appreciative :) Cheers /dan On Fri, May 6, 2016, 12:23 Daniel Sutcliffe <dansut@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
If you want a hook each time a new protocol is created, you're probably better off writing a wrapper protocol factory, and passing that to your ClientService, then doing any set-up work you want to do in your buildProtocol implementation, which delegates to the real, pymodbus implementation.
I am curious as to why you say that this is "messy".
Any advice, good patterns, or pointers to other projects which do something similar is appreciated,
I spent a while thinking about your question, and I'm sorry that I can't give a more thorough answer, but I think you need to be a bit more specific about what it is you don't like about your potential solution. It seems to me that having a delegating Factory, especially if all you need to do is set up some state on each Protocol that gets produced, should be sufficient... -glyph
![](https://secure.gravatar.com/avatar/ad5d35adba8666840c8ce22c1a43ef1f.jpg?s=120&d=mm&r=g)
Thanks Glyph, I think you have given me a push back in the 'right' direction - more thoughts and commentary embedded below if you, or anyone else, has the time. On May 6, 2016, at 10:19 AM, Daniel Sutcliffe <dansut@gmail.com> wrote: [...]
On Mon, May 9, 2016 at 6:19 PM, Glyph <glyph@twistedmatrix.com> wrote:
Understood, if this is the way the framework is intended to be used I realize doing anything else is going to be fighting against the flow. However, just to probe the situation I found myself in further, for the sake probing broken code to see how it might be fixed: Given the ClientService.whenConnected() method is intended to provide access to my connected Protocol through the deferred it returns, is it not a little unfriendly that this Protocol may turn out to be disconnected? OK occasionally due to timing but for this to be a possible condition which can loop with the same disconnected Protocol returned until the ClientService has its _currentConnection set to None, suggests to me that I can't safely use my Protocol from whenConnected() for much other than as a notification the first connection has occurred... but how do I avoid this? I have looked at the source and it seems to me the fact that the connection has been lost should bubble up to the ClientService through a t.a.i._DisconnectFactory and t.a.i._ReconnectingProtocolProxy once my Protocol's connectionLost() is called. My issue seems to be that I errback on a Protocol method's deferred returned to code at or above the ClientService level which gives up on that Protocol and calls whenConnected() to get the next one, only the Protocol's connectionLost() has yet to be called and then doesn't have chance to because my code is looping around calling whenConnected() and getting the same Protocol back. I hope that makes sense :-/ My Q on this is if I should be internally calling my Protocol's connectionLost() so it can bubble up to the ClientService before I errback on the Protocol method - whose responsibility is it to call this?
Honestly, this was just a gut feeling at the time, probably more sourced in my implementation from lack of experience in using Twisted; after reading around the subject, looking at many more examples, and your advice, I think I am convinced I need to back to looking at my own Protocol derived from the pymodbus one with a Factory that contains the persistent config and access to an interface to pump the polled data upstream.
Looking at the code again I think it just seemed to make sense to me at the time to have something that is (or has) a ClientService be the object I am calling a DataSource have more control over when it polls that data and what it does with it - and not to have to make the Protocol and its Factory aware of this at all. Does that make it any clearer? I am happy to push ahead with building this into my Factory/Protocol if that is more normal usage for Twisted as I am sure there will be benefits of encapsulating it here beyond the other side I was seeing during that moment of confusion. Thanks for the advice, I think it was enough to nudge me in a direction that will work better to get me going, and if with more experience other usage makes more sense I can always refactor, refactor, refactor :) Cheers /dan -- Daniel Sutcliffe <dansut@gmail.com>
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
whenConnected() is not intended to be used for "give me each Protocol as it is instantiated so that state can be set up", it is intended for API clients which want to send a message to the current connection to just retrieve the current connection so they can call a method on it. I'm not sure what you mean by "turn out to be disconnected". The physical reality of networking is that you might always encounter a transport which has been disconnected but which you haven't received notification of its disconnection yet.
Let me try to rephrase: you call a protocol method which returns a Deferred; you add an errback to that Deferred which calls whenConnected() to re-try, but since the protocol hasn't disconnected yet, you get the same protocol instance back, which is useless to you.
It's the framework's responsibility to call it. You should not call it yourself. Your Protocol's connectionLost isn't going to bubble up to ClientService anyway; you'd have to call your wrapper's connectionLost, which would confuse its internal state, since the framework would call it again right afterwards, and we definitely don't have test coverage for that, since the framework will normally only call it once. The right way to handle this would be to introduce a delay between re-tries. It's generally a good idea to have such a delay for lots of reasons; you don't want to overload your peer in the case of a transient failure. As a bonus, the fact that you've gone back up to the reactor loop to wait a while means that the transport will be properly disconnected and whenConnected() will do what you want.
Yup! Happy to help, sorry for the super long lag time on this reply, but my email queue has been pretty full lately :) -glyph
![](https://secure.gravatar.com/avatar/ad5d35adba8666840c8ce22c1a43ef1f.jpg?s=120&d=mm&r=g)
Delayed response here but hopefully it is still seen as worthwhile discussion, sorry if it is seems to just be beating a dead horse ;) On May 10, 2016, at 9:52 AM, Daniel Sutcliffe <dansut@gmail.com> wrote:
I appreciate now this 'protocol/connection init' isn't what the whenConnected() method was intended for, and it wasn't what I was using it for when I came across my issue. However there seems to me to be no documentary discouragement for using it this way... at least for someone who is not as familiar with Twisted's common usage patterns
The 'turns out to be disconnected' actually came from my acceptance that network connections drop/fail and I have little control of how or when this happens. Thus I may call a method on a Protocol at any time only to find that it is not in a good state to handle my request. My (misguided) goal was to call a method of the Protocol returned from whenConnected() every so often to give it a task to do; my expectation being that *when* the connection failed I would find out from the task method's returned deferred and could then just fire off another ClientService whenConnected() to get the next useful Protocol once it is established, and then use this to continue giving the Protocol tasks until it also ultimately fails...
Thanks Glyph, that seems like exactly what I was trying to say.
I'm glad that I got this mostly right in my head - it didn't feel right for me to be calling this, or any similar method I could find.
I didn't even consider this as my thinking was along the lines of the ClientService having the ability to handle the connection re-tries and delays to avoid any overloading of the service it wraps. It would be nice if there was something I could do in my Protocol task method such that it didn't fire the errback on its deferred until its Factory (and ultimately its Service in this case) had been made aware the connection had failed. Does this not have to be done elsewhere in Twisted or other projects using Twisted, or is the common pattern just to introduce a delay to avoid any possible race condition? My current intention is to have the Protocol's Factory handle the assigning of tasks to its Protocol which seems the 'more normal' direction to take, so any discussions above are purely for my education and intellectual curiosity - I'm glad of any and all feedback. Cheers /dan -- Daniel Sutcliffe <dansut@gmail.com>
participants (3)
-
Daniel Sutcliffe
-
Glyph
-
Kevin Conway