Mailman 3 August 2016 - Twisted

[Twisted-Python] Trial not operating on current working dir & Sunsetting Python 3.3
by Amber "Hawkie" Brown Aug. 28, 2016

Aug. 28, 2016

Hi everyone! Two things to let you all know in this email: 1. The new trial console script does not add the current directory to the PATH. Meaning, you can't use it to test a checkout of your code anymore! Instead, you should only use it for things that are actually on your path -- we recommend using something like tox to automate installing your package and testing your code. Hynek goes into detail here: https://hynek.me/articles/testing-packaging/ . If you really want to do this, you can use 'python -m twisted.trial', but you should really be testing installed versions of your code instead! 2. The next release of Twisted (tentatively named 16.5) will be the last to support Python 3.3. It's not in any LTS platforms, and the only reason we cared was PyPy3 -- but PyPy3.3.5 is not exactly production ready, and since they're working hard on the move to 3.5, I think there's no reason hanging around on 3.3. Hence, 16.5 will still support it in line with our compat policy, but 16.6 (the next release -- might be called 16.11 or 16.12 by that time!) will drop it completely. Enjoy 16.4.0! - Amber

1 0

[Twisted-Python] Waiting time for tests running on Travis CI and Buildbot
by Adi Roiban Aug. 24, 2016

Aug. 24, 2016

Hi, We now have 5 concurrent jobs on Travis-CI for the whole Twisted organization. If we want to reduce the waste of running push tests for a PR we should check that the other repos from the Twisted organization are doing the same. We now have in twisted/twisted 9 jobs per each build ... and for each push to a PR ... we run the tests for the push and for the PR merge... so those are 18 jobs for a commit. twisted/mantisa has 7 jobs per build, twisted/epsilon 3 jobs per build, twisted/nevow 14 jobs, twisted/axiom 6 jobs, twisted/txmongo 16 jobs .... so we are a bit over the limit of 5 jobs --------- I have asked Travis-CI how we can improve the waiting time for twisted/twisted jobs and for $6000 per year they can give us 15 concurrent jobs for the Twisted organization. This will not give us access to a faster waiting line for the OSX jobs. Also, I don't think that we can have twisted/twisted take priority inside the organization. If you think that we can raise $6000 per year for sponsoring our Travis-CI and that is worth increasing the queue size I can follow up with Travis-CI. ------------- I have also asked Circle CI for a free ride on their OSX builders, but it was put on hold as Glyph told me that Circe CI is slower than Travis. I have never used Circle CI. If you have a good experience with OSX on Circle CI I can continue the phone interview with Circle Ci so that we get the free access and see how it goes. -------- There are multiple ways in which we can improve the time a test takes to run on Travis-CI, but it will never be faster than buildbot with a slave which is always active and ready to start a job in 1 second and which already has 99% of the virtualev dependencies already installed. AFAIK the main concern with buildot, is that the slaves are always running so a malicious person could create a PR with some malware and then all our slaves will execute that malware. One way to mitigate this, is to use latent buildslaves and stop and reset a slave after each build, but this will also slow the build and lose the virtualenv ... which of docker based slave should not be a problem... but if we want Windows latent slaves it might increase the build time. What do you say if we protect our buildslaves with a firewall which only allows outgoing connections to buildmaster and github ... and have the slaves running only on RAX + Azure to simplify the firewall configuration? Will a malicious person still be interested of exploiting the slaves? I would be happy to help with buildbot configuration as I think that for TDD, buildbot_try with slaves which are always connected and virtualenv already created is the only acceptable CI system. Cheers -- Adi Roiban

6 17

[Twisted-Python] Regarding moving on with ticket #7934 (New Logging System)
by Eeshan Garg Aug. 24, 2016

Aug. 24, 2016

Hello again! I am fully aware that it has been a _very_ long time since I mentioned that I would get on with <https://twistedmatrix.com/trac/ticket/7934>, but I haven't really had the time to actually do anything about it. :-/ I would love to work on more tickets like < https://twistedmatrix.com/trac/ticket/7897> and help port Twisted to the new logging system and kind of get back into the flow of submitting patches and contributing to Twisted again (I miss it so much!!). My main concerns about moving on with #7934 after my last patch have already been reported by hawkowl in < https://twistedmatrix.com/trac/ticket/7983>. However, it was a bit unclear to me from the discussions in 7983 as to how to move forward. It would mean a lot to me if someone who is working on this right now (or glyph or hawkowl) could please give me a couple of pointers as to what should be done with the "self.log" issue for porting existing code to the new logging system. - Should I declare a log attribute for an entire module to use? - Should I rewrite my new patch around hawkowl's suggestion in 7983, use _loggerFor for everything? If so, how do you refer to the Logger instance to compare the log_logger in the event dict with what is expected (since asserting everything in the event dict, including log_logger is crucial for optimum test coverage)? - Or is there another way I should be searching for? Has anything changed since 7983 was filed? :-) I have already started working on the other suggestions put forth by the reviewers (such as context managers for capturing log events, etc.). Looking forward to contributing more! Regards, Eeshan Garg

1 1

[Twisted-Python] Tests on Windows with Python 3 now passing
by Craig Rodrigues Aug. 23, 2016

Aug. 23, 2016

I have fixed the tests that were failing on Windows with Python 3. If you look at: https://ci.appveyor.com/project/adiroiban/twisted/build/trunk-1008 We now have builds running on Python 3.3, 3.4, and 3.5 on Windows. These builds are also uploading coverage results to codecov.io. I looked at that build and on Python 3.5 we have: 9539 tests (2338 skipped, 7201 succeeded) -- Craig

3 2

[Twisted-Python] Please fix the broken buildbots
by Craig Rodrigues Aug. 23, 2016

Aug. 23, 2016

Hi, Someone has made changes in the past few days which have broken many of the buildbots. For example, in this build: https://buildbot.twistedmatrix.com/builders/debian8-py2.7/builds/1548/steps… There is this: python -m tox -r -e py27-tests "" in dir /buildslave/debian8-py2.7/Twisted (timeout 1200 secs) watching logfiles {'test.log': 'build/py27-tests/tmp/_trial_temp/test.log'} argv: ['python', '-m', 'tox', '-r', '-e', 'py27-tests', ''] The last argument being passed to python is an empty string. This is causing an empty string to be passed to trial, and consequently none of the tests are being run. This empty string is causing none of the tests to run, which is causing the build to fail. The fix is either to: (1) eliminate the last parameter (2) the last parameter should not be an empty string, but should be "twisted" so that trial is invoked as "trial twisted". I don't understand the buildbot scripts so don't know where to fix this. Can someone fix this? -- Craig

5 7

[Twisted-Python] Twisted 16.4.0 Prerelease 2 Announcement
by Amber "Hawkie" Brown Aug. 21, 2016

Aug. 21, 2016

Hi everyone! I bring you a new Twisted 16.4.0 prerelease, with the following fixes: - Twisted's HTTP server, when operating over TLS, would not cleanly close sockets, causing it to build up CLOSE_WAIT sockets until it would eventually run out of file descriptors. (from 16.3.2) - AMP no longer fails on floats on Python 3. For more information, check the NEWS file (link provided below). As usual, it's available for download -- go here (https://twistedmatrix.com/Releases/pre/16.4.0pre2/ <https://twistedmatrix.com/Releases/pre/16.4.0pre2/>) to get the prerelease tarballs and the full NEWS file. If you want to install it right away, run: pip install https://twistedmatrix.com/Releases/pre/16.4.0pre2/Twisted-16.4.0rc2.tar.bz2 <https://twistedmatrix.com/Releases/pre/16.4.0pre2/Twisted-16.4.0rc2.tar.bz2> Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :) Twisted Regards, Amber Brown (HawkOwl)

1 0

[Twisted-Python] setup.py broken in trunk
by Glyph Lefkowitz Aug. 21, 2016

Aug. 21, 2016

41482c4fe5442bcc7b046ab53bf15409f2031abd makes a change such that 'pip install -e .' (and indeed, 'pip install .') no longer installs 'twist'. I think we should revert this. Clearly we have a problem with our test suite, if we're not catching the binaries disappearing on us :-). -glyph

2 3

[Twisted-Python] SSL4ClientEndpoint not updating the transport's connected flag on connection lost
by Adi Roiban Aug. 20, 2016

Aug. 20, 2016

Hi, I have observed that when a SSL4ClientEndpoint was used, the protocol's transport `connected` attribute is not updated by the wrapper when the connection is lost. I have this code which will need your own server cert+key file: https://gist.github.com/adiroiban/8e0be840f81750f88a587e634ddfb194 I tried it on twisted/trunk and when SSL is used the output is server:client connected client:connected client:send-message server:data received client:connection-lost client:transport-status:1:wrapped-transport->0 server:client disconnected client:transport-status:1:wrapped-transport->0 while without SSL the output is: server:client connected client:connected client:send-message server:data received server:client disconnected client:connection-lost client:transport-status:0:wrapped-transport->0 client:transport-status:0:wrapped-transport->0 ------------ My current finding is that the reactor will call connection lost on t.i.abstract.FileDescriptor.connectionLost and not on t.protocols.policies.ProtocolWrapper so it seems that when SSL is used the wrapped protocol is registered instead of the wrapping protocol. ------ I am doing something wrong here? Is this the expected behavior or this is a bug? Thanks! -- Adi Roiban

2 3

[Twisted-Python] Twisted 16.3.2 Release Announcement
by Amber "Hawkie" Brown Aug. 19, 2016

Aug. 19, 2016

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 16.3.1. This is a security fix release, and is recommended for all users of Twisted. The fixes contained are: - Twisted's HTTP server, when operating over TLS, would not cleanly close sockets, causing it to build up CLOSE_WAIT sockets until it would eventually run out of file descriptors. For more information, check the NEWS file (link provided below). You can find the downloads at <https://pypi.python.org/pypi/Twisted <https://pypi.python.org/pypi/Twisted>> (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads <http://twistedmatrix.com/trac/wiki/Downloads>>). The NEWS file is also available at <https://github.com/twisted/twisted/blob/twisted-16.3.2/NEWS <https://github.com/twisted/twisted/blob/twisted-16.3.2/NEWS>>. Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted! Twisted Regards, Amber Brown (HawkOwl)

1 0

[Twisted-Python] Moving under a src directory, or: Sorry But I Broke All Your Pull Requests
by Amber "Hawkie" Brown Aug. 18, 2016

Aug. 18, 2016

Hi everyone, I have just merged the branch which puts twisted under a src/ directory. The reasons are many (see https://hynek.me/articles/testing-packaging/ for details as to why this is a Fantastic Idea), and it will require a bit of care with existing branches and pull requests. Thankfully, glyph has scripted all that your PR should need to do: python admin/fix-for-src-mv.py This will move everything under the src/ dir. If you have moved/renamed files in your PR, you may need to manually fix the conflicts. Happy hacking, everyone :) - Amber

1 0