Mailman 3 April 2019 - Twisted

[Twisted-Python] Buildbot downtime
by Tom Prince March 3, 2022

March 3, 2022

I am plannig to upgrade twisted's buildbot in the near future. In preperation for that, I am going to take down the production buildbot on Tuesday, June 26, around 1600 UTC[1]. During that time, the result of any builds done will be lost. Thus, I'd ask that people avoid committing to trunk during that time. I will send a reminder before I take down the buildbot both here, and on #twisted and #twisted-dev. Tom [1] http://timeanddate.com/worldclock/meetingtime.html?iso=20120626&p1=80&p2=22…

2 8

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz Aug. 5, 2021

Aug. 5, 2021

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] magical and seamless asyncio support
by Glyph May 23, 2019

May 23, 2019

After filing https://github.com/tornadoweb/tornado/issues/2636 <https://github.com/tornadoweb/tornado/issues/2636> recently, I was reminded that Twisted should support asyncio seamlessly, and currently we have some quite-visible seams. There are three major use-cases for asyncio integration: I've got a large Twisted application. I run the reactor at startup. I find a cool asyncio lib. I want to use it. What do I do? I've got a large asyncio application. I run the main loop at startup. I find a cool twisted lib. I want to use it. What do I do? I'm noodling around in an environment, like a Jupyter notebook, which already happens to have an event loop but I don't really know which one it is. How do I have the fewest number of steps? What happens today in each case? Case 1: If I've got a large Twisted application, I either start off by doing Deferred.fromFuture or ensureDeferred. These give me Deferreds that never fire, because no asyncio loop is running, but they don't yell at me; they just hang. Now I have to switch my event loop over to be an AsyncioSelectorReactor. Except wait, my application is a GTK+ application, and the only GTK+ main loop I can get that implements the asyncio APIs is unmaintained! So I either give up my custom Twisted reactor or I give up my asyncio functionality or I run it all in a thread. None of these are great experiences. Case 2: if I've got an asyncio application, I have a similar problem; I need a reactor, but one isn't running, so all my Deferred.asFuture(get_event_loop())s just hang. So I need to run AsyncioSelectorReactor, but now I need to know a bunch of obscure Twisted trivia to boostrap all of this properly: Case 3: oh no, how do I even know which one I need to run? In all of these cases I need to know a bunch of really inane trivia: I need to call startRunning() on the reactor, but just once, when it gets set up, or threadpool invocations (such as name resolution) will hang. Unless I'm lucky enough to actually control the whole process startup, in which case I can replace my event loop's .run_forever() with Twisted's .run(). I need to know whether I need subprocess support, and from whom. If I need it in Twisted, I need to do startRunning(installSignalHandlers=True); if I need it in asyncio, I need to do installSignalHandlers=False. I don't think there's a way to have both. I'd better hope that this is not on Windows, because Twisted is going to use its POSIX socket I/O implementation no matter what. Then, once it's up and running, rather than just 'await'-ing Deferreds, I always need to await someDeferred().asFuture(loop=get_event_loop()). From a practical perspective, get_event_loop() is the only correct value that I'd ever want to pass here, but for some reason the library makes me pass it every time. I propose a series of changes that would make this seamless from either side. Make Deferreds awaitable by asyncio by just calling `get_event_loop` and lying about what loop they're connected to. My reading of this is that https://github.com/python/cpython/blob/c5c6cdada3d41148bdeeacfe7528327b481c… <https://github.com/python/cpython/blob/c5c6cdada3d41148bdeeacfe7528327b481c…> will totally let us do that, if we just have a get_loop method (or, gross, _loop property) on Deferred or whatever's returned from Deferred.__await__. It would be fine if this raised a warning or something, as long as it worked in the 80% case so people could get some initial success and a pointer as to how to succeed, and not just hitting a wall with "task got bad yield". Make the reactor automagical. Phase one of this change would be: when you do 'from twisted.internet import reactor', if there's already an asyncio loop installed and running, automatically select the asyncio integration. This only helps you if you're in a context like a Jupyter notebook where you're not doing it at the module level, but that's still interesting. Make 'twisted.internet.reactor' into a dynamic proxy object which forwards reactor calls to whichever the running reactor is at the moment of the method call (connectTCP, callLater, etc). This can move the reactor selection to whenever the "first touch" on the reactor is, rather than whenever it's imported. (This also fixes a ton of of annoying import-order stuff in Twisted itself, as a bonus.) Automatically call startRunning() as necessary if another loop is in charge. Fix the subprocess integration: As a simple first step, for UNIX, at least participate in the asyncio get_child_watcher() / set_child_watcher() protocol, so that at least someone trying to coordinate a Twisted loop and an asyncio loop can intentionally select which one gets child process termination notifications, and possibly even multiplex this. Fix subprocesses along with any platform-specific socket quirks, by doing the next step... Actually use the asyncio APIs in the "asyncio side down" integration, i.e. AsyncioSelectorReactor. Presently we implement everything in terms of the add_reader and add_writer APIs, which is both very low-level and also fairly UNIX-specific. We should instead be using loop.create_connection, loop.create_server, loop.subprocess_exec, loop.getaddrinfo, etc, and translating between asyncio protocol/transport APIs and our own. Implement the "twisted side down" integration with asyncio; i.e. instead of implementing the twisted APIs in terms of asyncio's interfaces, implement the asyncio APIs in terms of Twisted's interfaces, so we can use existing custom reactors, GUI loop integration, etc. This is all quite a bit of work but I think it would massively improve the experience of a novice trying to adopt Twisted in a modern Python stack. In particular, I think that in addition to being a good example of the general problem domain, Jupyter is quite specifically incredibly strategic, and having the ability to just grab Treq and start doing massively parallel I/O to ingest data, then just 'await' on it, would be a very powerful demonstration of Twisted's capabilities. Let me know what you all think! -glyph

2 2

[Twisted-Python] Real Viagra Professional - for real sex pros
by Tamika May 1, 2019

May 1, 2019

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <center> <a href="http://rqf.entercompassion.com"><img src="http://mediapix.ru/pics/685207b3dd437eff7f15c0744b5c1d3a.gif"> <title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" href="http://gfx8.nnt.com/mail/12.0.1187.0824/positioning.css" /> </head> <body style="overflow:auto;"> <div id="PrintShellToolbar"> <div class="ContainerPadding"> <span class="FloatRight"> <a href= > <img alt="" src="http://gfx1.nfi.com/mail/w2/pr02/ltr/glyph_close_rest.gif" /> <Close window> <a href= > <img alt="" src="http://gfx2.czb.com/mail/w2/pr02/ltr/i_print.gif" /> <Print> </a> </span> </div> </div> <div> <div class="BorderContainerOutline ContainerPadding"> <div class="ClearBoth"> <span class="HeaderData" style="font-size:large;"> <TuPostal.com Cambia de nombre?> </span> </div> <div class="ClearBoth"> <span class="HeaderLabel"> <From:> </span> <span class="HeaderData"> <b><TuPostal.com></b> </span> </div> <div class="ClearBoth"> <span class="HeaderLabel"> <Sent:> </span> <span class="HeaderData"> <div class="ClearBoth"> <span class="HeaderLabel"> To: </span> <span class="HeaderData"> Mikelelliott (mikelelliott(a)rlo.com) </span> </div> <br /> </div> </div> <div class="MessageBody" style=""> <br /> <div id="MessageBodyText" class="ExternalClass" style="border: 1px solid #fff ;"> <pre>Hola!<br> <br>Tupostal es ahora parte de Sonico!<br> <br>Para conocer nuestro nuevo sitio, haz clic en el siguiente link:<br><a href='http://www.gwpcom/index1.php' target='_blank'>http://www.bjncom/index1.php</a><br> <br>Sonico es un sitio donde puedes...<br>- Guardar fotos ilimitadas<br>- Re encontrarte con viejos amigos<br>- Recibir novedades de tus amigos.<br> <br>Te Esperamos! Ya somos mas de 21 millones de usuarios!!<br> <br>Sonico - Amigos Conectados<br> <br> <br>PD: Ahora tambien podras guardar tus postales<br> <br> <br>------------------------------------------<br>Haz recibido este email porque te encuentras registrado en fdm.com.<br>Si no deseas recibir mas mensajes como este, por favor haz clic en el siguiente link:<br><a href='http://www.cehcom/unsubscribe.php?type=2' target='_blank'>http://www.fyocom/unsubscribe.php?type=2</a><br> <br></pre> </div> </div> </div> </body> </html>

8 10

[Twisted-Python] Backward incompatible fixes to backward incompatible changes
by Jean-Paul Calderone May 1, 2019

May 1, 2019

Hello all, I'd like to discuss the project policy/process for dealing with incorrect backward incompatibilities that end up released. As a case study, we could look at https://twistedmatrix.com/trac/ticket/9410 <https://twistedmatrix.com/trac/ticket/9410#comment:17> / https://github.com/twisted/twisted/pull/1106 In Twisted 16.3.0 the behavior of Request.write was changed so that it raises an AttributeError if called after the connection has closed. Prior to that release, it silently did nothing in that case. Now, three years later, we have a PR which proposed restoring the behavior of silently ignoring the write to a closed connection. The original change was incompatible. The new change is incompatible. What should win out? Jean-Paul

3 3

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com April 28, 2019

April 28, 2019

1 0

[Twisted-Python] txsni + alpn + acme (letsencrypt)
by Daniel Holth April 23, 2019

April 23, 2019

Hello. Can you help me to learn to debug tls problems in twisted? I was disappointed that txacme, an automatic way to get certificates for twisted web, stopped working, so I'm trying to add a responder for the new challenge type. It sends a special certificate if the CA negotiates acme using alpn, proving domain control, then you get a certificate. Seems to work with openssl s_client but letsencrypt says "no application protocol". Anyone know where that error comes from? https://github.com/glyph/txsni/pull/26

2 14

[Twisted-Python] The Future of Twisted Spread
by Amber Brown April 22, 2019

April 22, 2019

twisted.spread is without a maintainer, and there were discussions in #twisted-dev about it's future. It has several flaws, mostly regarding security and resource usage that have not moved forward in a long time. These discussions mostly revolved around deprecating it, but Glyph pointed out existing uses of it and suggested looking for a maintainer first. As such, we're looking for a maintainer who can dedicate themselves to maintaining twisted.spread and giving Perspective Broker a bit of love. This would require someone that uses twisted.spread, and is committed to its long-term success. If nobody steps up, we'll declare it unmaintained and deprecate it in a future Twisted release. - Amber

1 0

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com April 21, 2019

April 21, 2019

1 0

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com April 14, 2019

April 14, 2019

1 0

[Twisted-Python] Twisted Conch on v19.2.1: now requires explicit installation of bcrypt
by Tobias Oberstein April 13, 2019

April 13, 2019

Hi, I'm not sure if it considered a Twisted bug, but Twisted Conch on Twisted 19.2.1 now seems to require explicit installation of bcrypt, which wasn't the case on <19.2.1. Here is our (Crossbar.io) tracking issue: https://github.com/crossbario/crossbar/issues/1555 Not a big deal obviously - we'll add that (indirect) dependency to Crossbar.io .. just wanted to post here if anyone else runs into. Cheers, /Tobias PS: I did test the upcoming Twisted release with Crossbar.io - but only a "quick manual test", not our full CI - which now ran red (by being triggered from an unrelated PR). There is more fallout for us with the new Twisted release:( Analyzing that now. We (Crossbar.io) really should have run our _full_ CI on the upcoming release.

2 2

Re: [Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory
by Tobias Oberstein April 11, 2019

April 11, 2019

Am 02.04.19 um 09:03 schrieb Chris Withers: > On 02/04/2019 07:56, Tobias Oberstein wrote: >>> Would be good to get that converted... >> >> yeah, agreed. >> >> unlikely to happen though (lack of time) unless someone steps up with >> a PR > > Okay, but then don't be surprised when people get tripped up by it... well, I never claimed to be surprised. I was only triggered by the reference to our _docs_ as we've put quite some work improving on that. but honestly, lets not further drum that line;) it leads nowhere. I am with you: there is room for improvements. if you have a PR, please think about contributing! we'll be working with you / merging .. > >>> What's the simplest way to connect a ClientService to a websocket url >>> (ie: ws://host/endpoint or wss://host/endpoint) >> >> you might adapt / start from this (again WAMP, but just a matter of >> using websocket rather than wamp factories/protocols): >> >> https://github.com/crossbario/autobahn-python/blob/master/examples/twisted/… > > > Any way to get rid of the ugly duplication between the url in line 69 > and the endpoint on line 72? > > I seem to remember that ends up happening with or without ClientService. I am aware, but no, it's designed like this, because WebSocket requires a HTTP URL (with potentially path components, query parameters etc) in the opening handshake while the _transport_ does not necessarily have that. Autobahn allows you to run WebSocket over basically everything that is a bidirectional byte pipe: TCP of course, but also Unix domain socket, pipes, Tor onion services (via https://github.com/meejah/txtorcon) and even serial ports! Cheers, /Tobias > > Chris

3 5

[Twisted-Python] Twisted 19.2.0 Release Announcement
by Amber Brown April 10, 2019

April 10, 2019

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 19.2! The highlights of this release are: * twisted.web.client.HostnameCachingHTTPSPolicy was added as a new contextFactory option. This reduces the performance overhead for making many TLS connections to the same host. * twisted.conch.ssh.keys can now read private keys in the new "openssh-key-v1" format, introduced in OpenSSH 6.5 and made the default in OpenSSH 7.8. * The sample code in the "Twisted Web In 60 Seconds" tutorial runs on Python 3. * DeferredLock and DeferredSemaphore can be used as asynchronous context managers on Python 3.5+. * twisted.internet.ssl.CertificateOptions now uses 32 random bytes instead of an MD5 hash for the ssl session identifier context. * twisted.python.failure.Failure.getTracebackObject now returns traceback objects whose frames can be passed into traceback.print_stack for better debugging of where the exception came from. * Much more! 20+ tickets closed overall. You can find the downloads at <https://pypi.python.org/pypi/Twisted> (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>). The NEWS file is also available at <https://github.com/twisted/twisted/blob/twisted-19.2.0/NEWS.rst>. Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted! Twisted Regards, Amber Brown (HawkOwl)

2 1

[Twisted-Python] how to get extReceived call from SSHCommandClientEndpoint
by Sereysethy TOUCH April 7, 2019

April 7, 2019

Hello, I am implementing a program using SSHCommandClientEndpoint, the program works fine (dataReceived got calls) but when the server send an error message (stderr..), the function extReceived in my protocol never gets called, instead it only calls the one in the SSHChannel as I can see in the log (got extended data 1 b': No such file or directory\n'). How to override that? Or how to connect it to my protocol? I adapted the program found on Twisted website. Thanks, Sethy from twisted.conch.endpoints import SSHCommandClientEndpoint from twisted.internet.protocol import Factory, Protocol from twisted.python import log import sys from twisted.internet.defer import Deferred class NoiseProtocol(Protocol): def connectionMade(self): print("connectionMade") self.finished = Deferred() self.strings = ["bif", "pow", "zot"] self.sendNoise() def sendNoise(self): if self.strings: self.transport.write(self.strings.pop(0) + "\n") else: self.transport.loseConnection() def dataReceived(self, data): print("Server says:", data) self.sendNoise() def extReceived(self, dataType, data): print("extReceived") def connectionLost(self, reason): self.finished.callback(None) command = b"cat unknownfile" username = b"XXXX" password = b"XXXX" host = b"server" port = 22 endpoint = SSHCommandClientEndpoint.newConnection( reactor, command, username, host, port, password=password, agentEndpoint=None) factory = Factory() factory.protocol = NoiseProtocol d = endpoint.connect(factory) d.addCallback(lambda proto: proto.finished) log.startLogging(sys.stdout, setStdout=0) reactor.run()

4 5

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com April 7, 2019

April 7, 2019

1 0

[Twisted-Python] Twisted 19.2.0rc2 Release Candidate Announcement
by Amber Brown April 4, 2019

April 4, 2019

Hello everyone! 19.2.0rc2 is now out. Sorry for the delay! This fixes the Cryptography dependency being set to 1.5, when we actually require at least 2.5. You can get the tarball and the NEWS file at https://twistedmatrix.com/Releases/rc/19.2.0rc2/ , or you can try it out from PyPI: python -m pip install Twisted==19.2.0rc2 Please test it, and let me know how your applications fare, good or bad! If nothing comes up, 19.2 will release in a week. Twisted regards, Amber Brown (hawkowl)

4 3

[Twisted-Python] A Proposal for reducing the burden of developing on Twisted by dropping Python 2 support
by Amber Brown April 2, 2019

April 2, 2019

Hi everyone, Since the Python 2 EOL date is rapidly approaching, I thought it was time we consider dropping Python 2 support. I personally find that Python 2 compat adds a huge amount of overhead when working on and maintaining Twisted, and think that with the current maintainer availability, dropping it sooner rather than later would have a beneficial effect on how much work we spend on shims/compat, complexity, and our ability to ship new features, as well as onboarding people who are interested in the project, but have no interest (or experience!) in Python 2.7. It is basically summed up by doing a feature freeze on an agreed-upon version of Twisted, that will be the last version released for 2.7. It would be abnormal in that it would get security fixes (our current policy is to only release them for current versions) and critical bugfixes, but would otherwise be frozen. One of my rationales is that from some analysis of PyPI download statistics, the vast majority of Python 2 users are using old versions of Twisted, while nearly all our Python 3 users are on the latest version. As such, I believe freezing a version that will get security updates but no new features would not be a massive loss to those stuck on Python 2 for whatever reason. Twisted's compatibility policy would still apply, ensuring that Python 2/3 compatible software using Twisted would be able to use the older Twisted version on Python 2, and the newer version on Python 3, as you would usually expect. You can find the proposal here, in this handy-dandy Google Doc: https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN… Comments are welcome, as well as which timeline seems reasonable. - Amber

11 13

[Twisted-Python] stop/start client connections with loseConnection in ReconnectingClientFactory
by Chris Satterthwaite April 2, 2019

April 2, 2019

Hello community, First of all - thanks for an awesome platform! I'm brand new to this community, but have been using Twisted a couple years. Reason for posting: I've hit a condition with ReconnectingClientFactory that I'm not sure is per design. I have a work around right now, but need your perspective. Seems like there should be a better/right way to do this. Attempted design: I'd like to have long running TCP clients (forever until stopped), with a long running TCP server. When a long running client hits a problem with a dependency (database is down, kafka bus unavailable, external API not responding, etc), I want the client to go offline for a while and then come back online. an automated, self-recovery type action. Since it's not ok to start/stop/restart the Twisted Reactor, I am letting the client finish whatever it can do, disconnect from the service, destruct the dependencies, wait for a period of time, and then attempt a clean re-initialization of those dependencies along with reconnecting to the Twisted Server. Problem case: I'm using the ReconnectingClientFactory in my client. When the client hits a problem, it calls transport.loseConnection(). But whenever the client calls this, after the disconnect - it does not reconnect; stopFactory is called and everything exits. Work around: I noticed some Twisted source code that works off factory.numPorts. If numPorts is 1 and the client loses the connection, it goes to 0 and calls the cleanup. So I conditionally increase this number right before intentionally disconnecting, and then reset that after reconnecting. This solves the problem, but it's a hack. I'll attach the test scripts to this post (if attachments are allowed), but the main code is with these functions in the factory: def clientConnectionLost(self, connector, reason): print(' factory clientConnectionLost: reason: {}'.format(reason)) # if self.disconnectedOnPurpose: # ## Hack to keep reactor alive # print(' factory clientConnectionLost: increasing numPorts') # self.numPorts += 1 # self.numPortsChanged = True # self.disconnectedOnPurpose = False print(' ... simulate client going idle before attempting restart...') time.sleep(5) ReconnectingClientFactory.clientConnectionLost(self, connector, reason) print(' factory clientConnectionLost: end.\n') def clientConnectionMade(self): print(' factory clientConnectionMade: starting numPorts: {}'.format(self.numPorts)) # if self.numPortsChanged : # ## Resetting from hacked value # print(' factory clientConnectionMade: decreasing numPorts') # self.numPorts -= 1 # self.numPortsChanged = False print(' factory clientConnectionMade: finished numPorts: {}'.format(self.numPorts)) def cleanup(self): print('factory cleanup: calling loseConnection') if self.connectedClient is not None: self.connectedClient.transport.loseConnection() self.disconnectedOnPurpose = True With the above lines commented out, once the cleanup call does transport.loseConnection(), the factory stops at the end of clientConnectionLost. Sample scripts/logs: I've tried to create short test scripts and corresponding logs (with the client failing, and then with it restarting when I use the workaround). I've cut out several thousand lines to get down to something simple for the example test scripts, but I know the client is still a little long. Again, I'm not sure if attachments work on the mailing list, but I'll attempt to attach the client/server scripts with the corresponding pass/fail logs. Thanks! -Chris

6 12