Mailman 3 September 2018 - Twisted

[Twisted-Python] Keeping logger from snatching stdout?
by Jeff Grimmett 15 Sep '18

15 Sep '18

I'm working on a Flask project using the flask_twisted package from PyPI and have encountered a mystery. I don't *have* to solve it to move on, but darn it, I can't let it go :) https://pypi.org/project/Flask-Twisted/ https://github.com/cravler/flask-twisted/ So here's the breakdown: when use the standard logging module to output to a logfile + stdout, everything is fine. I do see some logging output from Twisted as well (the web server part) but for the most part life is groovy. I'm going to be integrating in a wxPython windows app to this (previously it's been a shell app) and step 1 was to make my own stdout handler that eventually would pipe all that stuff to a window in the wx app. The first step of THAT (step 1.a) was to replace all the logging stuff with print() placeholders. At that point, things got confusing, as now ALL of my print()s are being handled by Twisted's logging handler. I brought flask-twisted in local to my code so I could monkey around with it. First thing I noticed was that it was using twisted.python.log. I messed around with that - commented it out, and now I get no stdout output at all. Set the parameter setStdout to False, same thing. The adapter code uses twisted.internet.reactor, threads, twisted.web.server.Site, twisted.web.wsgi.WSGIResource, and twisted.web.resource.Resource, any one which might be responsible. I spent yesterday evening digging around but haven't found anything yet. Any guidance / ideas? Regards, Jeff

2 3

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 09 Sep '18

09 Sep '18

1 0

Re: [Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels
by Kyle Altendorf 07 Sep '18

07 Sep '18

On 2018-09-07 01:58, Glyph wrote: >> On Sep 5, 2018, at 5:44 AM, Kyle Altendorf <sda(a)fstab.net> wrote: >> >> On 2018-09-05 03:25, Glyph wrote: >>>> On Aug 29, 2018, at 10:13 AM, Kyle Altendorf <sda(a)fstab.net> wrote: >>>> wheel links: >>>> https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 >>>> (and next comment) >>>> Now that I've got the wheel builds happening I figured it'd be good >>>> to try them out on 'real' machines. Turns out we get a failure on >>>> twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at >>>> least the two checks I've done so far (Windows and OSX). I haven't >>>> done more than a cursory look at that yet, but it's on the list to >>>> understand and resolve. More testing would of course be welcome. >>>> Real world, just trial Twisted's own tests, whatever would be >>>> appreciated if you are interested. >>>> failures with wheels: >>>> https://github.com/twisted/twisted/pull/1051#issuecomment-416977723 >>> It's not clear to me that we have any test environments that >>> accurately test what happens after you install from PyPI. If we >>> could build these wheels and then actually run tests on installing >>> them that would be a big step forward in test fidelity! But given >>> the current state of the art, I'd be happy if they just got built >>> from the same code that passed our tests in existing configurations. >> >> cibuildwheel does have an option to provide a test command. If we did >> that would we replace the existing tests or supplement? If >> supplementing that will presumably be a significant increase in build >> times. Especially if we did it for all wheels. > > Definitely replace. What would you want to even supplement them with? I was referring to supplementing the existing src tests with the wheel tests. > >> So, how much of a build time hit do we want to take? > > I was thinking that for starters here the goal would just be build > automation, independent of trying to get a more realistic test path; > only doing these builds when pushing a relevant tag. If it's easy to > just switch over everything to this then... great, I guess? :) But > let's not gild the lily here. Alrighty, another PR for swapping over tests from src to wheels. I would tend to always build the wheels. It's easier for people to test intermediate releases for bugs. It makes a release 'less special' so there is less likely to be a latent issue with the process or artifacts. Etc. But, let's see what the build time hit is. Travis: 4 minutes, about half the time of a single version test job AppVeyor: 7 minutes, about the same as a single version test job Circle OS X: 9 minutes, a bit over a single version test job Circle Linux: 5 minutes, a bit under a single (OS X) version test job Ok, so for about the cost of testing a single Python version we can get wheels. Especially if we want to switch over to testing wheels instead of source, I'd plan on building wheels for every commit. >> but sure, I didn't mean to argue against automation. > > Could we do a test run by uploading wheels to, say, > https://test.pypi.org <https://test.pypi.org/> first? Shall the process be build wheel, upload to test PyPI, install from test PyPI, test... :] But sure, I could see ending up with a test PyPI upload at least for a tag/release. Could technically do it for every build but that seems excessive, even given my comments above about exercising the process more often rather than less. >> Here's one spot I did automatic deployment off of Travis to PyPI. >> >> https://github.com/altendky/gitignoreio/blob/aeeb9546b6f27f411b191699d3625b… >> >> Pretty straightforward though I think it does the wheel build which >> isn't what we'd really want. > > That's pretty cool. I assumed you'd have to write a 'twine' command - > Travis just has a 'deploy' provider called "pypi"? What's actually > happening behind the scenes there, I wonder! > > (Also, you should probably upload an sdist as well as a bdist_wheel; > there are cases where wheels aren't always what you want.) Yeah, it's cool but then you have to know what is behind it... for each CI host... and how to make it use the files you want... So I'd tend towards a twine call. I made this silly little script the other day to handle this in another repo. https://github.com/altendky/pyqt5-tools/blob/master/twineontag.py >> I'd have to check on how to deploy existing wheels as well as those >> from OS X and Windows. I'd suggest a separate PR? I mean maybe it'd >> be easier to get one PR reviewed than two but that doesn't seem like a >> great reason to mash things together. > > Smaller PRs are always easier to review - they just take more latency > to build :). Ok, I'll make separate tickets for: * Testing wheels vs. src * Automatic deployment to PyPI Cheers, -kyle

1 0

[Twisted-Python] Issue with TLS failure modes
by Robert DiFalco 05 Sep '18

05 Sep '18

I'm having an issue properly handling TLS failure modes. For example consider the EchoServer and EchoClient code. If I use a TLS client with a TCP4 server, I do not get a handshake exception until I abort the connection. But I don't want to abort the connection unless I get a handshake error. What I'd like to do is to check the handshake status in my protocol before my client sends bytes to the server. I'd like my send message to be able to raise the <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'ssl23_read', 'ssl handshake failure'). But for some reason it seems to get lost until I abort the connection. Does this sound familiar to anyone? What I've done for now is setup a Timeout mixin so that after my client.send, if I do not get an ACK back (which my particular protocol does) within two minutes, I just abort the connection. This then calls connectionLost with the correct SSL.Error. But if it's in the error queue (and found during the course of abortConnection then isn't there a way to find it sooner? Like before my timeout and before I call send on the client protocol? Thanks!

2 2

[Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels
by Kyle Altendorf 05 Sep '18

05 Sep '18

Hi All, I am working on ticket #9217 / PR #1051 to add lots more wheel generation to the Twisted CI. I decided to give the cibuildwheel package a try and it made this process almost too easy (well... sort of :] ). I've got AppVeyor-Windows, Travis-Linux, and Circle-OSX all building a variety of wheels for the supported Python versions and bit depths. Travis doesn't save artifacts 'easily' so I went ahead and doubled up on Linux on Circle for now, though it's having some Docker issues at the moment and hasn't been successful yet. For some reason in this one case the project directory isn't getting mounted into the container as expected. wheel links: https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 (and next comment) Now that I've got the wheel builds happening I figured it'd be good to try them out on 'real' machines. Turns out we get a failure on twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at least the two checks I've done so far (Windows and OSX). I haven't done more than a cursory look at that yet, but it's on the list to understand and resolve. More testing would of course be welcome. Real world, just trial Twisted's own tests, whatever would be appreciated if you are interested. failures with wheels: https://github.com/twisted/twisted/pull/1051#issuecomment-416977723 Overall, it's a bit unclear what the intended use of the various CI hosts are for Twisted. I hear that Travis OSX builds were really slow, but from what I can see Circle isn't doing any OSX (other than what I added). There wasn't any artifact storage being used on Circle either. So, I'm not sure if there was a reason to use Circle that went away, or... But, not having to hook Travis up to S3 or somesuch for storage is quite nice so Circle wins at least in that category. Now that I've got something rough in place, are there any opinions about how this should work? I don't know the present release workflow so I don't know if we'd want an automatic push to PyPI on tags (probably not), or just artifacts on the build server to grab manually (would need some S3 or such for Travis, or Circle for Linux builds as well). Anything else? Do we want automated tests against the wheels? cibuildwheel does have a feature for that though I haven't done anything with it yet. Anyways... hello, thanks for Twisted, and I hope this work ends up saving some people some time. Cheers, -kyle

2 2

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 02 Sep '18

02 Sep '18

1 0