[Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory
![](https://secure.gravatar.com/avatar/acf307afae4a1e72fa3cffe3596ed5f3.jpg?s=120&d=mm&r=g)
Hello community, First of all - thanks for an awesome platform! I'm brand new to this community, but have been using Twisted a couple years. Reason for posting: I've hit a condition with ReconnectingClientFactory that I'm not sure is per design. I have a work around right now, but need your perspective. Seems like there should be a better/right way to do this. Attempted design: I'd like to have long running TCP clients (forever until stopped), with a long running TCP server. When a long running client hits a problem with a dependency (database is down, kafka bus unavailable, external API not responding, etc), I want the client to go offline for a while and then come back online. an automated, self-recovery type action. Since it's not ok to start/stop/restart the Twisted Reactor, I am letting the client finish whatever it can do, disconnect from the service, destruct the dependencies, wait for a period of time, and then attempt a clean re-initialization of those dependencies along with reconnecting to the Twisted Server. Problem case: I'm using the ReconnectingClientFactory in my client. When the client hits a problem, it calls transport.loseConnection(). But whenever the client calls this, after the disconnect - it does not reconnect; stopFactory is called and everything exits. Work around: I noticed some Twisted source code that works off factory.numPorts. If numPorts is 1 and the client loses the connection, it goes to 0 and calls the cleanup. So I conditionally increase this number right before intentionally disconnecting, and then reset that after reconnecting. This solves the problem, but it's a hack. I'll attach the test scripts to this post (if attachments are allowed), but the main code is with these functions in the factory: def clientConnectionLost(self, connector, reason): print(' factory clientConnectionLost: reason: {}'.format(reason)) # if self.disconnectedOnPurpose: # ## Hack to keep reactor alive # print(' factory clientConnectionLost: increasing numPorts') # self.numPorts += 1 # self.numPortsChanged = True # self.disconnectedOnPurpose = False print(' ... simulate client going idle before attempting restart...') time.sleep(5) ReconnectingClientFactory.clientConnectionLost(self, connector, reason) print(' factory clientConnectionLost: end.\n') def clientConnectionMade(self): print(' factory clientConnectionMade: starting numPorts: {}'.format(self.numPorts)) # if self.numPortsChanged : # ## Resetting from hacked value # print(' factory clientConnectionMade: decreasing numPorts') # self.numPorts -= 1 # self.numPortsChanged = False print(' factory clientConnectionMade: finished numPorts: {}'.format(self.numPorts)) def cleanup(self): print('factory cleanup: calling loseConnection') if self.connectedClient is not None: self.connectedClient.transport.loseConnection() self.disconnectedOnPurpose = True With the above lines commented out, once the cleanup call does transport.loseConnection(), the factory stops at the end of clientConnectionLost. Sample scripts/logs: I've tried to create short test scripts and corresponding logs (with the client failing, and then with it restarting when I use the workaround). I've cut out several thousand lines to get down to something simple for the example test scripts, but I know the client is still a little long. Again, I'm not sure if attachments work on the mailing list, but I'll attempt to attach the client/server scripts with the corresponding pass/fail logs. Thanks! -Chris
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 22/03/2019 17:08, Chris Satterthwaite wrote:
def clientConnectionLost(self, connector, reason):
Without the rest of your class, it's difficult to see some of the potential problems... Could you put it on a gist somewhere?
This will block the reactor, so don't think you should be doing it... cheers, Chris
![](https://secure.gravatar.com/avatar/acf307afae4a1e72fa3cffe3596ed5f3.jpg?s=120&d=mm&r=g)
Hi Chris, The files I attached (with the full classes) made it through to my email, but I wondered if they would they go through to everyone. Here's a gist with the same scripts: https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f And yep, as you mentioned, a 'sleep' is definitely blocking. That's not in the production version; I just dropped it in here for the test script to simulate something. Thanks! -Chris -----Original Message----- From: Twisted-Python <twisted-python-bounces@twistedmatrix.com> On Behalf Of Chris Withers Sent: Friday, March 22, 2019 1:54 PM To: twisted-python@twistedmatrix.com Subject: Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory On 22/03/2019 17:08, Chris Satterthwaite wrote:
def clientConnectionLost(self, connector, reason):
Without the rest of your class, it's difficult to see some of the potential problems... Could you put it on a gist somewhere?
This will block the reactor, so don't think you should be doing it... cheers, Chris _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 22/03/2019 23:26, Chris Satterthwaite wrote:
Observations: - Your super call at https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file..., I'd expect that to be super(ServiceClientFactory, self).__init__(), but your spelling may be a python 3 only thing that works? - Those sleeps are going to cause you more problems than they solve. - What does this seek to achieve? https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file... - Why not use twisted logging instead of print? By setting it to debug, you'll get lots of into about what twisted is doing, and by using logging you won't need to do all that manual traceback printing. - when you say "When the client hits a problem, it calls transport.loseConnection()", where is that call to transport.loseConnection? - when you noticed some Twisted source code that works off factory.numPorts, where is that code? Can you provide a link? This doesn't sound right... Now, as to what your problem is, I suspect it's this call to stopTrying: https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file... The factory is stopped and started again by ReconnectingClientFactory, so you don't want that there as it means you stop trying every time there's a disconnection. cheers, Chris
![](https://secure.gravatar.com/avatar/acf307afae4a1e72fa3cffe3596ed5f3.jpg?s=120&d=mm&r=g)
Hi Chris, The print and sleep statements are just my efforts at converting something big down to something small for a test case. In the actual client I'm using twisted.logger for FilteringLogObserver's and Logger's. The looping call for system health was also a shorted version; I wasn't sure if looping calls were the culprit at the start. I appologise for the fluff and if I introduced confusion. I just figured that posting multiple classes and thousands of lines would get less response than an attempted short test sample.
when you say "When the client hits a problem, it calls transport.loseConnection()", where is that call to transport.loseConnection?
The transport.loseConnection call is on line 82 of the test client code; it's inside the cleanup() function.
Now, as to what your problem is, I suspect it's this call to stopTrying: https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file...
I may be missing this from the Twisted side, but I understood that stopFactory was only called when the reactor was cleaning up. So I added stopTrying in there for the ReconnectingClientFactory portion, amongst many other lines (that I removed for the example) for stopping external dependencies. All that seemed to work fine. When I called loseConnection with just a single established connection, stopFactory was called and ReconnectingClientFactory was not restarted or reconnected. Seemed like ReconnectingClientFactory was not expecting a new connection attempt after the loseConnection call. Additionally, it seemed like the factory expected to clean up and stop the reactor whenever loseConnection was called with that single connection. And of course in most use-cases, that makes sense. But my goal was to hang on to the factory after loseConnection, and continue work after external dependencies came back online.
when you noticed some Twisted source code that works off factory.numPorts, where is that code?
To troubleshoot, I used __dict__ to start investigating variable values with the factory and client. And I used that to start searching through the twisted code in site-packages. I noticed at least one function which conditionally worked off numPorts in order to shut things down (twisted.internet.protocol.AbstractDatagramProtocol.doStop). Here's the code: assert self.numPorts > 0 self.numPorts = self.numPorts - 1 self.transport = None if not self.numPorts: if self.noisy: log.msg("Stopping protocol %s" % self) self.stopProtocol() And so the work around I implemented was to conditionally increase this number while I controlled the disconnect/reconnect. Seemed to work fine in practice. Thanks again for the responses. Given all the benefits Glyph mentioned of ClientService - I've added that migration into my roadmap. -Chris -----Original Message----- From: Twisted-Python <twisted-python-bounces@twistedmatrix.com> On Behalf Of Chris Withers Sent: Tuesday, March 26, 2019 2:28 AM To: twisted-python@twistedmatrix.com Subject: Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory On 22/03/2019 23:26, Chris Satterthwaite wrote:
Observations: - Your super call at https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file..., I'd expect that to be super(ServiceClientFactory, self).__init__(), but your spelling may be a python 3 only thing that works? - Those sleeps are going to cause you more problems than they solve. - What does this seek to achieve? https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file... - Why not use twisted logging instead of print? By setting it to debug, you'll get lots of into about what twisted is doing, and by using logging you won't need to do all that manual traceback printing. - when you say "When the client hits a problem, it calls transport.loseConnection()", where is that call to transport.loseConnection? - when you noticed some Twisted source code that works off factory.numPorts, where is that code? Can you provide a link? This doesn't sound right... Now, as to what your problem is, I suspect it's this call to stopTrying: https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file... The factory is stopped and started again by ReconnectingClientFactory, so you don't want that there as it means you stop trying every time there's a disconnection. cheers, Chris _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
![](https://secure.gravatar.com/avatar/ea5f1293256e7758363e7fd7cd2da9cb.jpg?s=120&d=mm&r=g)
You may want to look at twisted.application.internet.ClientService <https://twistedmatrix.com/documents/18.7.0/api/twisted.application.internet....>. It uses the new endpoints instead of the `connectTCP()` stuff. Not sure if it applies in your situation, but it has all of the retry logic built in, so that may make it easier to work with. On Fri, Mar 22, 2019 at 10:08 AM Chris Satterthwaite <chris@cmsconstruct.com> wrote:
![](https://secure.gravatar.com/avatar/acf307afae4a1e72fa3cffe3596ed5f3.jpg?s=120&d=mm&r=g)
Hi Sean, Thanks for the suggestion; I’ll take a look. -Chris From: Twisted-Python <twisted-python-bounces@twistedmatrix.com> On Behalf Of Sean DiZazzo Sent: Friday, March 22, 2019 3:09 PM To: Twisted general discussion <twisted-python@twistedmatrix.com> Subject: Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory You may want to look at twisted.application.internet.ClientService <https://twistedmatrix.com/documents/18.7.0/api/twisted.application.internet....> . It uses the new endpoints instead of the `connectTCP()` stuff. Not sure if it applies in your situation, but it has all of the retry logic built in, so that may make it easier to work with.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I'd further note that ClientService is generally the new, good way to do things and ReconnectingClientFactory is the old, bad way. Our hope is to eventually deprecate ReconnectingClientFactory and most of the APIs that it uses, but this is a big project that we have not been able to make much progress on in the last, ahem, decade or so. -g
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 24/03/2019 04:30, Glyph wrote:
What's the big advantage(s) of ClientService over ReconnectingClientFactory? The Autobahn guys still show ReconnectingClientFactory in their docs, and I remember looking at this before and ending up going with ReconnectingClientFactory because it works and didn't look like it'd need much effort to integrated into an existing code base. cheers, Chris
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Let me count the ways. ReconnectingClientFactory is destined for deprecation, eventually. You should just adopt the "new" thing now so that if we get more energy to cycle the APIs and delete the old stuff, you'll have less hassle to deal with. ("New" is in quotes here since it's been around for well over 3 years at this point; Autobahn should update too, not just you.) ClientService works with endpoints, which means you can use it with any kind of transport, like SSH transports, subprocesses, etc. Most practically, it works with HostnameEndpoint which is a much better way to get TLS than connectSSL; ReconnectingClientFactory works, kind of accidentally, with TLS since connectSSL is on the reactor, but it won't use happy eyeballs and it won't connect over IPv6, so connections will be slower and less reliable. Conceptually, ClientService has a much clearer and more useful responsibility: its job is to maintain a state (i.e.: that there is a single connection, that it is connected) rather than to do a thing. For example, if you want to shut down a ReconnectingClientFactory: you have to call stopTrying, then uh... find the last protocol it built with buildProtocol, then grab its transport (hope it's saving that transport as '.transport', because it doesn't actually have to) call loseConnection remember to trap connectionLost so you can see when its done. if you want to shut down a ClientService call stopService wait for the Deferred that it returned to fire ClientService is (mostly) implemented using composition rather than inheritance, so much less of the guts of the internal implementation is hanging around where you might accidentally twiddle it and break its assumptions, so you can trust its guarantees more. other benefits of composition: you don't have to override attributes of your Protocol and thereby indulge in subclassing yourself to get notifications; consider 'prepareConnection', 'whenConnected'. the retry policy mechanics are better documented and much easier to customize it's backed by a formal state machine - not that I'm aware of any specific bugs in ReconnectingClientFactory but do you think it got all of these state transitions correct: https://gist.github.com/glyph/614be03151556333efe04b849fa05930 <https://gist.github.com/glyph/614be03151556333efe04b849fa05930> It's more testable because it just takes its clock and reactor as constructor parameters, rather than requiring post-hoc poorly-documented attribute patching to become testable. Hopefully at least some of this is convincing :) -g
![](https://secure.gravatar.com/avatar/d0e895e27d970995fff580593c1793c5.jpg?s=120&d=mm&r=g)
The Autobahn guys still show ReconnectingClientFactory in their docs,
Where did you find that? That would be a doc bug, but in the _docs_, there is no reference to ReconnectingClientFactory (cpy372_3) oberstet@intel-nuci7:~$ find ~/scm/crossbario/autobahn-python/docs/ -type f -exec grep -Hi "ReconnectingClientFactory" {} \; (cpy372_3) oberstet@intel-nuci7:~$ We do have some example code using ReconnectingClientFactory though: (cpy372_3) oberstet@intel-nuci7:~$ find ~/scm/crossbario/autobahn-python/examples/ -type f -exec grep -Hi "ReconnectingClientFactory" {} \; | wc -l 8
Autobahn will automatically use twisted.application.internet.ClientService for auto-reconnect when on Twisted 16.1.0+ https://github.com/crossbario/autobahn-python/blob/master/autobahn/twisted/w...
++1 (note: Autobahn is a culprit to the overuse of inheritance vs composition too .. but the so-called "new API" (for WAMP) is following composition + observer pattern, while the "old API" (that is inheriting from ApplicationSession) is still pretty much around. we have some legacy already ..)
all and each of them is convincing;)
-- Tobias Oberstein - phone +49 176 2375 2055 - tobias.oberstein@crossbario.com Crossbar.io GmbH - Waldstrasse 18 - 91054 Erlangen HRB 15870 - Amtsgericht Fuerth - Geschäftsfuehrer/CEO - Tobias Oberstein https://crossbar.io https://crossbario.com
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 26/03/2019 09:53, Tobias Oberstein wrote:
Yeah, this one: https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/w... Would be good to get that converted...
I'm not using WAMP, just raw websocket.
What's the simplest way to connect a ClientService to a websocket url (ie: ws://host/endpoint or wss://host/endpoint)
This hasn't proved to hard.
Why? Sending a WebSocket protocol-level close from the client seems to work just fine?
I've abstracted all that away into https://github.com/cjw296/carly. See https://github.com/cjw296/carly/blob/master/tests/test_autobahn_websocket_cl... for example. I do still have writing docs for carly on my list of things to do. It would be great if Twisted ever had a real testing reactor, but I think we all know that's never going to happen ;-) Besides, it's actually useful to test with real networking... Chris
![](https://secure.gravatar.com/avatar/7cedcd5fe799f836e4ea4f9560e873d0.jpg?s=120&d=mm&r=g)
yeah, agreed. unlikely to happen though (lack of time) unless someone steps up with a PR
What's the simplest way to connect a ClientService to a websocket url (ie: ws://host/endpoint or wss://host/endpoint)
you might adapt / start from this (again WAMP, but just a matter of using websocket rather than wamp factories/protocols): https://github.com/crossbario/autobahn-python/blob/master/examples/twisted/w...
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 22/03/2019 17:08, Chris Satterthwaite wrote:
def clientConnectionLost(self, connector, reason):
Without the rest of your class, it's difficult to see some of the potential problems... Could you put it on a gist somewhere?
This will block the reactor, so don't think you should be doing it... cheers, Chris
![](https://secure.gravatar.com/avatar/acf307afae4a1e72fa3cffe3596ed5f3.jpg?s=120&d=mm&r=g)
Hi Chris, The files I attached (with the full classes) made it through to my email, but I wondered if they would they go through to everyone. Here's a gist with the same scripts: https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f And yep, as you mentioned, a 'sleep' is definitely blocking. That's not in the production version; I just dropped it in here for the test script to simulate something. Thanks! -Chris -----Original Message----- From: Twisted-Python <twisted-python-bounces@twistedmatrix.com> On Behalf Of Chris Withers Sent: Friday, March 22, 2019 1:54 PM To: twisted-python@twistedmatrix.com Subject: Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory On 22/03/2019 17:08, Chris Satterthwaite wrote:
def clientConnectionLost(self, connector, reason):
Without the rest of your class, it's difficult to see some of the potential problems... Could you put it on a gist somewhere?
This will block the reactor, so don't think you should be doing it... cheers, Chris _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 22/03/2019 23:26, Chris Satterthwaite wrote:
Observations: - Your super call at https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file..., I'd expect that to be super(ServiceClientFactory, self).__init__(), but your spelling may be a python 3 only thing that works? - Those sleeps are going to cause you more problems than they solve. - What does this seek to achieve? https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file... - Why not use twisted logging instead of print? By setting it to debug, you'll get lots of into about what twisted is doing, and by using logging you won't need to do all that manual traceback printing. - when you say "When the client hits a problem, it calls transport.loseConnection()", where is that call to transport.loseConnection? - when you noticed some Twisted source code that works off factory.numPorts, where is that code? Can you provide a link? This doesn't sound right... Now, as to what your problem is, I suspect it's this call to stopTrying: https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file... The factory is stopped and started again by ReconnectingClientFactory, so you don't want that there as it means you stop trying every time there's a disconnection. cheers, Chris
![](https://secure.gravatar.com/avatar/acf307afae4a1e72fa3cffe3596ed5f3.jpg?s=120&d=mm&r=g)
Hi Chris, The print and sleep statements are just my efforts at converting something big down to something small for a test case. In the actual client I'm using twisted.logger for FilteringLogObserver's and Logger's. The looping call for system health was also a shorted version; I wasn't sure if looping calls were the culprit at the start. I appologise for the fluff and if I introduced confusion. I just figured that posting multiple classes and thousands of lines would get less response than an attempted short test sample.
when you say "When the client hits a problem, it calls transport.loseConnection()", where is that call to transport.loseConnection?
The transport.loseConnection call is on line 82 of the test client code; it's inside the cleanup() function.
Now, as to what your problem is, I suspect it's this call to stopTrying: https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file...
I may be missing this from the Twisted side, but I understood that stopFactory was only called when the reactor was cleaning up. So I added stopTrying in there for the ReconnectingClientFactory portion, amongst many other lines (that I removed for the example) for stopping external dependencies. All that seemed to work fine. When I called loseConnection with just a single established connection, stopFactory was called and ReconnectingClientFactory was not restarted or reconnected. Seemed like ReconnectingClientFactory was not expecting a new connection attempt after the loseConnection call. Additionally, it seemed like the factory expected to clean up and stop the reactor whenever loseConnection was called with that single connection. And of course in most use-cases, that makes sense. But my goal was to hang on to the factory after loseConnection, and continue work after external dependencies came back online.
when you noticed some Twisted source code that works off factory.numPorts, where is that code?
To troubleshoot, I used __dict__ to start investigating variable values with the factory and client. And I used that to start searching through the twisted code in site-packages. I noticed at least one function which conditionally worked off numPorts in order to shut things down (twisted.internet.protocol.AbstractDatagramProtocol.doStop). Here's the code: assert self.numPorts > 0 self.numPorts = self.numPorts - 1 self.transport = None if not self.numPorts: if self.noisy: log.msg("Stopping protocol %s" % self) self.stopProtocol() And so the work around I implemented was to conditionally increase this number while I controlled the disconnect/reconnect. Seemed to work fine in practice. Thanks again for the responses. Given all the benefits Glyph mentioned of ClientService - I've added that migration into my roadmap. -Chris -----Original Message----- From: Twisted-Python <twisted-python-bounces@twistedmatrix.com> On Behalf Of Chris Withers Sent: Tuesday, March 26, 2019 2:28 AM To: twisted-python@twistedmatrix.com Subject: Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory On 22/03/2019 23:26, Chris Satterthwaite wrote:
Observations: - Your super call at https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file..., I'd expect that to be super(ServiceClientFactory, self).__init__(), but your spelling may be a python 3 only thing that works? - Those sleeps are going to cause you more problems than they solve. - What does this seek to achieve? https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file... - Why not use twisted logging instead of print? By setting it to debug, you'll get lots of into about what twisted is doing, and by using logging you won't need to do all that manual traceback printing. - when you say "When the client hits a problem, it calls transport.loseConnection()", where is that call to transport.loseConnection? - when you noticed some Twisted source code that works off factory.numPorts, where is that code? Can you provide a link? This doesn't sound right... Now, as to what your problem is, I suspect it's this call to stopTrying: https://gist.github.com/codingadvocate/f732da79ddf6cef4b7a0b6b3679f519f#file... The factory is stopped and started again by ReconnectingClientFactory, so you don't want that there as it means you stop trying every time there's a disconnection. cheers, Chris _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
![](https://secure.gravatar.com/avatar/ea5f1293256e7758363e7fd7cd2da9cb.jpg?s=120&d=mm&r=g)
You may want to look at twisted.application.internet.ClientService <https://twistedmatrix.com/documents/18.7.0/api/twisted.application.internet....>. It uses the new endpoints instead of the `connectTCP()` stuff. Not sure if it applies in your situation, but it has all of the retry logic built in, so that may make it easier to work with. On Fri, Mar 22, 2019 at 10:08 AM Chris Satterthwaite <chris@cmsconstruct.com> wrote:
![](https://secure.gravatar.com/avatar/acf307afae4a1e72fa3cffe3596ed5f3.jpg?s=120&d=mm&r=g)
Hi Sean, Thanks for the suggestion; I’ll take a look. -Chris From: Twisted-Python <twisted-python-bounces@twistedmatrix.com> On Behalf Of Sean DiZazzo Sent: Friday, March 22, 2019 3:09 PM To: Twisted general discussion <twisted-python@twistedmatrix.com> Subject: Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory You may want to look at twisted.application.internet.ClientService <https://twistedmatrix.com/documents/18.7.0/api/twisted.application.internet....> . It uses the new endpoints instead of the `connectTCP()` stuff. Not sure if it applies in your situation, but it has all of the retry logic built in, so that may make it easier to work with.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
I'd further note that ClientService is generally the new, good way to do things and ReconnectingClientFactory is the old, bad way. Our hope is to eventually deprecate ReconnectingClientFactory and most of the APIs that it uses, but this is a big project that we have not been able to make much progress on in the last, ahem, decade or so. -g
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 24/03/2019 04:30, Glyph wrote:
What's the big advantage(s) of ClientService over ReconnectingClientFactory? The Autobahn guys still show ReconnectingClientFactory in their docs, and I remember looking at this before and ending up going with ReconnectingClientFactory because it works and didn't look like it'd need much effort to integrated into an existing code base. cheers, Chris
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Let me count the ways. ReconnectingClientFactory is destined for deprecation, eventually. You should just adopt the "new" thing now so that if we get more energy to cycle the APIs and delete the old stuff, you'll have less hassle to deal with. ("New" is in quotes here since it's been around for well over 3 years at this point; Autobahn should update too, not just you.) ClientService works with endpoints, which means you can use it with any kind of transport, like SSH transports, subprocesses, etc. Most practically, it works with HostnameEndpoint which is a much better way to get TLS than connectSSL; ReconnectingClientFactory works, kind of accidentally, with TLS since connectSSL is on the reactor, but it won't use happy eyeballs and it won't connect over IPv6, so connections will be slower and less reliable. Conceptually, ClientService has a much clearer and more useful responsibility: its job is to maintain a state (i.e.: that there is a single connection, that it is connected) rather than to do a thing. For example, if you want to shut down a ReconnectingClientFactory: you have to call stopTrying, then uh... find the last protocol it built with buildProtocol, then grab its transport (hope it's saving that transport as '.transport', because it doesn't actually have to) call loseConnection remember to trap connectionLost so you can see when its done. if you want to shut down a ClientService call stopService wait for the Deferred that it returned to fire ClientService is (mostly) implemented using composition rather than inheritance, so much less of the guts of the internal implementation is hanging around where you might accidentally twiddle it and break its assumptions, so you can trust its guarantees more. other benefits of composition: you don't have to override attributes of your Protocol and thereby indulge in subclassing yourself to get notifications; consider 'prepareConnection', 'whenConnected'. the retry policy mechanics are better documented and much easier to customize it's backed by a formal state machine - not that I'm aware of any specific bugs in ReconnectingClientFactory but do you think it got all of these state transitions correct: https://gist.github.com/glyph/614be03151556333efe04b849fa05930 <https://gist.github.com/glyph/614be03151556333efe04b849fa05930> It's more testable because it just takes its clock and reactor as constructor parameters, rather than requiring post-hoc poorly-documented attribute patching to become testable. Hopefully at least some of this is convincing :) -g
![](https://secure.gravatar.com/avatar/d0e895e27d970995fff580593c1793c5.jpg?s=120&d=mm&r=g)
The Autobahn guys still show ReconnectingClientFactory in their docs,
Where did you find that? That would be a doc bug, but in the _docs_, there is no reference to ReconnectingClientFactory (cpy372_3) oberstet@intel-nuci7:~$ find ~/scm/crossbario/autobahn-python/docs/ -type f -exec grep -Hi "ReconnectingClientFactory" {} \; (cpy372_3) oberstet@intel-nuci7:~$ We do have some example code using ReconnectingClientFactory though: (cpy372_3) oberstet@intel-nuci7:~$ find ~/scm/crossbario/autobahn-python/examples/ -type f -exec grep -Hi "ReconnectingClientFactory" {} \; | wc -l 8
Autobahn will automatically use twisted.application.internet.ClientService for auto-reconnect when on Twisted 16.1.0+ https://github.com/crossbario/autobahn-python/blob/master/autobahn/twisted/w...
++1 (note: Autobahn is a culprit to the overuse of inheritance vs composition too .. but the so-called "new API" (for WAMP) is following composition + observer pattern, while the "old API" (that is inheriting from ApplicationSession) is still pretty much around. we have some legacy already ..)
all and each of them is convincing;)
-- Tobias Oberstein - phone +49 176 2375 2055 - tobias.oberstein@crossbario.com Crossbar.io GmbH - Waldstrasse 18 - 91054 Erlangen HRB 15870 - Amtsgericht Fuerth - Geschäftsfuehrer/CEO - Tobias Oberstein https://crossbar.io https://crossbario.com
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 26/03/2019 09:53, Tobias Oberstein wrote:
Yeah, this one: https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/w... Would be good to get that converted...
I'm not using WAMP, just raw websocket.
What's the simplest way to connect a ClientService to a websocket url (ie: ws://host/endpoint or wss://host/endpoint)
This hasn't proved to hard.
Why? Sending a WebSocket protocol-level close from the client seems to work just fine?
I've abstracted all that away into https://github.com/cjw296/carly. See https://github.com/cjw296/carly/blob/master/tests/test_autobahn_websocket_cl... for example. I do still have writing docs for carly on my list of things to do. It would be great if Twisted ever had a real testing reactor, but I think we all know that's never going to happen ;-) Besides, it's actually useful to test with real networking... Chris
![](https://secure.gravatar.com/avatar/7cedcd5fe799f836e4ea4f9560e873d0.jpg?s=120&d=mm&r=g)
yeah, agreed. unlikely to happen though (lack of time) unless someone steps up with a PR
What's the simplest way to connect a ClientService to a websocket url (ie: ws://host/endpoint or wss://host/endpoint)
you might adapt / start from this (again WAMP, but just a matter of using websocket rather than wamp factories/protocols): https://github.com/crossbario/autobahn-python/blob/master/examples/twisted/w...
participants (6)
-
Chris Satterthwaite
-
Chris Withers
-
Glyph
-
Sean DiZazzo
-
Tobias Oberstein
-
Tobias Oberstein