Mailman 3 April 2019 - Twisted

[Twisted-Python] Twisted Conch on v19.2.1: now requires explicit installation of bcrypt
by Tobias Oberstein 13 Apr '19

13 Apr '19

Hi, I'm not sure if it considered a Twisted bug, but Twisted Conch on Twisted 19.2.1 now seems to require explicit installation of bcrypt, which wasn't the case on <19.2.1. Here is our (Crossbar.io) tracking issue: https://github.com/crossbario/crossbar/issues/1555 Not a big deal obviously - we'll add that (indirect) dependency to Crossbar.io .. just wanted to post here if anyone else runs into. Cheers, /Tobias PS: I did test the upcoming Twisted release with Crossbar.io - but only a "quick manual test", not our full CI - which now ran red (by being triggered from an unrelated PR). There is more fallout for us with the new Twisted release:( Analyzing that now. We (Crossbar.io) really should have run our _full_ CI on the upcoming release.

2 2

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory
by Tobias Oberstein 11 Apr '19

11 Apr '19

Am 02.04.19 um 09:03 schrieb Chris Withers: > On 02/04/2019 07:56, Tobias Oberstein wrote: >>> Would be good to get that converted... >> >> yeah, agreed. >> >> unlikely to happen though (lack of time) unless someone steps up with >> a PR > > Okay, but then don't be surprised when people get tripped up by it... well, I never claimed to be surprised. I was only triggered by the reference to our _docs_ as we've put quite some work improving on that. but honestly, lets not further drum that line;) it leads nowhere. I am with you: there is room for improvements. if you have a PR, please think about contributing! we'll be working with you / merging .. > >>> What's the simplest way to connect a ClientService to a websocket url >>> (ie: ws://host/endpoint or wss://host/endpoint) >> >> you might adapt / start from this (again WAMP, but just a matter of >> using websocket rather than wamp factories/protocols): >> >> https://github.com/crossbario/autobahn-python/blob/master/examples/twisted/… > > > Any way to get rid of the ugly duplication between the url in line 69 > and the endpoint on line 72? > > I seem to remember that ends up happening with or without ClientService. I am aware, but no, it's designed like this, because WebSocket requires a HTTP URL (with potentially path components, query parameters etc) in the opening handshake while the _transport_ does not necessarily have that. Autobahn allows you to run WebSocket over basically everything that is a bidirectional byte pipe: TCP of course, but also Unix domain socket, pipes, Tor onion services (via https://github.com/meejah/txtorcon) and even serial ports! Cheers, /Tobias > > Chris

3 5

[Twisted-Python] Twisted 19.2.0 Release Announcement
by Amber Brown 11 Apr '19

11 Apr '19

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 19.2! The highlights of this release are: * twisted.web.client.HostnameCachingHTTPSPolicy was added as a new contextFactory option. This reduces the performance overhead for making many TLS connections to the same host. * twisted.conch.ssh.keys can now read private keys in the new "openssh-key-v1" format, introduced in OpenSSH 6.5 and made the default in OpenSSH 7.8. * The sample code in the "Twisted Web In 60 Seconds" tutorial runs on Python 3. * DeferredLock and DeferredSemaphore can be used as asynchronous context managers on Python 3.5+. * twisted.internet.ssl.CertificateOptions now uses 32 random bytes instead of an MD5 hash for the ssl session identifier context. * twisted.python.failure.Failure.getTracebackObject now returns traceback objects whose frames can be passed into traceback.print_stack for better debugging of where the exception came from. * Much more! 20+ tickets closed overall. You can find the downloads at <https://pypi.python.org/pypi/Twisted> (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>). The NEWS file is also available at <https://github.com/twisted/twisted/blob/twisted-19.2.0/NEWS.rst>. Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted! Twisted Regards, Amber Brown (HawkOwl)

2 1

[Twisted-Python] how to get extReceived call from SSHCommandClientEndpoint
by Sereysethy TOUCH 07 Apr '19

07 Apr '19

Hello, I am implementing a program using SSHCommandClientEndpoint, the program works fine (dataReceived got calls) but when the server send an error message (stderr..), the function extReceived in my protocol never gets called, instead it only calls the one in the SSHChannel as I can see in the log (got extended data 1 b': No such file or directory\n'). How to override that? Or how to connect it to my protocol? I adapted the program found on Twisted website. Thanks, Sethy from twisted.conch.endpoints import SSHCommandClientEndpoint from twisted.internet.protocol import Factory, Protocol from twisted.python import log import sys from twisted.internet.defer import Deferred class NoiseProtocol(Protocol): def connectionMade(self): print("connectionMade") self.finished = Deferred() self.strings = ["bif", "pow", "zot"] self.sendNoise() def sendNoise(self): if self.strings: self.transport.write(self.strings.pop(0) + "\n") else: self.transport.loseConnection() def dataReceived(self, data): print("Server says:", data) self.sendNoise() def extReceived(self, dataType, data): print("extReceived") def connectionLost(self, reason): self.finished.callback(None) command = b"cat unknownfile" username = b"XXXX" password = b"XXXX" host = b"server" port = 22 endpoint = SSHCommandClientEndpoint.newConnection( reactor, command, username, host, port, password=password, agentEndpoint=None) factory = Factory() factory.protocol = NoiseProtocol d = endpoint.connect(factory) d.addCallback(lambda proto: proto.finished) log.startLogging(sys.stdout, setStdout=0) reactor.run()

4 5

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 07 Apr '19

07 Apr '19

1 0

[Twisted-Python] Twisted 19.2.0rc2 Release Candidate Announcement
by Amber Brown 04 Apr '19

04 Apr '19

Hello everyone! 19.2.0rc2 is now out. Sorry for the delay! This fixes the Cryptography dependency being set to 1.5, when we actually require at least 2.5. You can get the tarball and the NEWS file at https://twistedmatrix.com/Releases/rc/19.2.0rc2/ , or you can try it out from PyPI: python -m pip install Twisted==19.2.0rc2 Please test it, and let me know how your applications fare, good or bad! If nothing comes up, 19.2 will release in a week. Twisted regards, Amber Brown (hawkowl)

4 3

[Twisted-Python] A Proposal for reducing the burden of developing on Twisted by dropping Python 2 support
by Amber Brown 02 Apr '19

02 Apr '19

Hi everyone, Since the Python 2 EOL date is rapidly approaching, I thought it was time we consider dropping Python 2 support. I personally find that Python 2 compat adds a huge amount of overhead when working on and maintaining Twisted, and think that with the current maintainer availability, dropping it sooner rather than later would have a beneficial effect on how much work we spend on shims/compat, complexity, and our ability to ship new features, as well as onboarding people who are interested in the project, but have no interest (or experience!) in Python 2.7. It is basically summed up by doing a feature freeze on an agreed-upon version of Twisted, that will be the last version released for 2.7. It would be abnormal in that it would get security fixes (our current policy is to only release them for current versions) and critical bugfixes, but would otherwise be frozen. One of my rationales is that from some analysis of PyPI download statistics, the vast majority of Python 2 users are using old versions of Twisted, while nearly all our Python 3 users are on the latest version. As such, I believe freezing a version that will get security updates but no new features would not be a massive loss to those stuck on Python 2 for whatever reason. Twisted's compatibility policy would still apply, ensuring that Python 2/3 compatible software using Twisted would be able to use the older Twisted version on Python 2, and the newer version on Python 3, as you would usually expect. You can find the proposal here, in this handy-dandy Google Doc: https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN… Comments are welcome, as well as which timeline seems reasonable. - Amber

11 13

[Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory
by Chris Satterthwaite 02 Apr '19

02 Apr '19

Hello community, First of all - thanks for an awesome platform! I'm brand new to this community, but have been using Twisted a couple years. Reason for posting: I've hit a condition with ReconnectingClientFactory that I'm not sure is per design. I have a work around right now, but need your perspective. Seems like there should be a better/right way to do this. Attempted design: I'd like to have long running TCP clients (forever until stopped), with a long running TCP server. When a long running client hits a problem with a dependency (database is down, kafka bus unavailable, external API not responding, etc), I want the client to go offline for a while and then come back online. an automated, self-recovery type action. Since it's not ok to start/stop/restart the Twisted Reactor, I am letting the client finish whatever it can do, disconnect from the service, destruct the dependencies, wait for a period of time, and then attempt a clean re-initialization of those dependencies along with reconnecting to the Twisted Server. Problem case: I'm using the ReconnectingClientFactory in my client. When the client hits a problem, it calls transport.loseConnection(). But whenever the client calls this, after the disconnect - it does not reconnect; stopFactory is called and everything exits. Work around: I noticed some Twisted source code that works off factory.numPorts. If numPorts is 1 and the client loses the connection, it goes to 0 and calls the cleanup. So I conditionally increase this number right before intentionally disconnecting, and then reset that after reconnecting. This solves the problem, but it's a hack. I'll attach the test scripts to this post (if attachments are allowed), but the main code is with these functions in the factory: def clientConnectionLost(self, connector, reason): print(' factory clientConnectionLost: reason: {}'.format(reason)) # if self.disconnectedOnPurpose: # ## Hack to keep reactor alive # print(' factory clientConnectionLost: increasing numPorts') # self.numPorts += 1 # self.numPortsChanged = True # self.disconnectedOnPurpose = False print(' ... simulate client going idle before attempting restart...') time.sleep(5) ReconnectingClientFactory.clientConnectionLost(self, connector, reason) print(' factory clientConnectionLost: end.\n') def clientConnectionMade(self): print(' factory clientConnectionMade: starting numPorts: {}'.format(self.numPorts)) # if self.numPortsChanged : # ## Resetting from hacked value # print(' factory clientConnectionMade: decreasing numPorts') # self.numPorts -= 1 # self.numPortsChanged = False print(' factory clientConnectionMade: finished numPorts: {}'.format(self.numPorts)) def cleanup(self): print('factory cleanup: calling loseConnection') if self.connectedClient is not None: self.connectedClient.transport.loseConnection() self.disconnectedOnPurpose = True With the above lines commented out, once the cleanup call does transport.loseConnection(), the factory stops at the end of clientConnectionLost. Sample scripts/logs: I've tried to create short test scripts and corresponding logs (with the client failing, and then with it restarting when I use the workaround). I've cut out several thousand lines to get down to something simple for the example test scripts, but I know the client is still a little long. Again, I'm not sure if attachments work on the mailing list, but I'll attempt to attach the client/server scripts with the corresponding pass/fail logs. Thanks! -Chris

6 12