Mailman 3 June 2016 - Twisted

[Twisted-Python] Deprecating classes and writing tests
by Adi Roiban 16 Jun '16

16 Jun '16

Hi, What is the best practice for deprecating a class in Twisted and writing tests for it ... and updating existing tests. I think that the part in which the deprecation code is written, is well understood and documented here http://twistedmatrix.com/documents/current/core/development/policy/compatib… What is missing, is the part talking about how to test these changes and how to update existing tests. The compatibility / deprecation documentation include a section about how to test deprecation code, but the section is brief. http://twistedmatrix.com/documents/current/core/development/policy/compatib… ---- This email is a follow up of a review done for this ticket https://twistedmatrix.com/trac/ticket/8368 Please also check the comments on that ticket. Please leave your feedback and we can try to apply and document it as part of the work for ticket #8368 Regards, -- Adi Roiban

2 2

[Twisted-Python] Twisted 16.3.0 Prerelease 1 Announcement
by Amber "Hawkie" Brown 14 Jun '16

14 Jun '16

Hello everyone! It's time for yet another Twisted release -- although not long after the last, we have had a bunch of good changes from the PyCon sprints, and some long-awaited work has landed! - The Git migration has happened, so we've updated our development documentation to match. We're now trialling accepting pull requests at github.com/twisted/twisted, so if you've ever wanted an excuse to contribute, now's the chance! - In our steady shedding of baggage, twisted.spread.ui, twisted.manhole (not to be confused with twisted.conch.manhole!), and a bunch of old and deprecated stuff from twisted.python.reflect and twisted.protocols.sip have been removed. - twisted.web's HTTP server now handles pipelined requests better -- it used to try and process them in parallel, but this was fraught with problems and now it processes them in series, which is less surprising to code that expects the Request's transport to not be buffered (e.g. WebSockets). - Twisted now has HTTP/2 support in its web server! This is currently not available by default -- you will need to install hyper-h2, which is available in the [h2] setuptools extras. If you want to play around with it, see the install link below. Upon release, you will be able to simply "pip install twisted[http2]". - 53 tickets closed overall, including cleanups that move us closer to a total Python 3 port. For more information, check the NEWS file (link provided below). As usual, it's available for download -- go here (https://twistedmatrix.com/Releases/pre/16.3.0pre1/) to get the prerelease tarballs and the full NEWS file. If you want to install it right away, run: pip install https://twistedmatrix.com/Releases/pre/16.3.0pre1/Twisted-16.3.0rc1.tar.bz2 If you would like to try out the newly-landed HTTP/2 support, run: pip install -U https://twistedmatrix.com/Releases/pre/16.3.0pre1/Twisted-16.3.0rc1.tar.bz2… This will download the new HTTP/2 dependencies and the TLS requirements as well. Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :) Twisted Regards, Amber Brown (HawkOwl)

1 0

[Twisted-Python] Porting PB to python3
by Craig Rodrigues 12 Jun '16

12 Jun '16

Hi, I reviewed your branch: https://github.com/wrohdewald/twisted/commits/spread-py3-7598/twisted In my opinion, that branch has two categories of Python 3 porting: Category 1: Minor -> these are "minor" fixos, such as dealing with xrange -> range Category 2: Major -> you have done a lot of work to deal with bytes versus unicode strings I hope you haven't lost interest in this, and are still willing to push forward. I would recommend that if you can separate out the minor fixes into separate GitHub pull requests and separate Trac tickets, then do that. If you see: https://github.com/twisted/twisted/commits/trunk you will see I have submitted a lot of pull requests + trac tickets of "minor" Python 3 fixes, and those have successfully been reviewed and merged by Twisted committers. Dealing with bytes versus unicode strings will take longer to review, because it is more advanced. That is why I recommend separating out the "minor" fixos if you can. The other think I would recommend is if you do periodic merges from trunk to your branch, since a lot of new Python 3 fixes have gone into trunk in the past few weeks. -- Craig

3 3

[Twisted-Python] Upcoming Buildbot Changes (Ubuntu version support)
by Amber "Hawkie" Brown 11 Jun '16

11 Jun '16

Hi everyone, I'm doing some builder work today; including changing the versions of Ubuntu we test on. From next week, our supported Ubuntu platforms will be: - Ubuntu 12.04 (Python 2.7) - Ubuntu 14.04 (Python 2.7) - Ubuntu 16.04 (Python 2.7, Python 3.4, Python 3.5, PyPy (unsupported)) These are all LTS versions right now, and should provide us pretty wide coverage of Ubuntus that are widely used. - Amber

2 2

[Twisted-Python] Using zope
by Craig Rodrigues 10 Jun '16

10 Jun '16

Hi, There are some places in the Twisted code which use zope.interface to do things like: class MyClass: inherits(IMyInterface) That works in Python 2, but the zope.interface library has code to hard fail that line on Python 3. The supported way of doing the same thing on Python 2 and 3 is: @implementer(IMyInterface) class MyClass This was discussed on the mailing list before by Barry Warsaw: https://twistedmatrix.com/pipermail/twisted-python/2013-January/026414.html I have submitted several patches for review (Trac ticket is in each one): http://bit.ly/1YhxKI2 For anyone who can help review the patches, the help would be appreciated. :) -- Craig

1 0

[Twisted-Python] Travis-CI integration work
by Adi Roiban 07 Jun '16

07 Jun '16

Hi, I just want to let you know that we are almost done with the work of integrating Travis-CI with Twisted GitHub PRs. The ticket is here https://twistedmatrix.com/trac/ticket/7229 If you have time, please take a look at the current work and suggest any changes or provide any feedback. The goal of this ticket is to have an initial set of tests executed for a patch, on behalf on non-Twisted commiters. In this way, I hope that contributors would be able to get a faster feedback regarding the quality of their patch. This should also reduce the work required to review a branch, as commiters will no longer have to merge and send to buildbot each new revision submitted by a contributor. Cheers, -- Adi Roiban

4 3

[Twisted-Python] inlinecallbacks, unhandled error....where?
by steven meier 07 Jun '16

07 Jun '16

hi, ive got these two functions: @inlineCallbacks def getUserList(): reload_interval = yield getReloadInterval() print "222", reload_interval target_url = host + "/?ReloaderUserOnline" body = urllib.urlencode({"reloadsequenz":reload_interval}) #body = '\r\n' #headers = {'Content-Type': 'application/x-www-form-urlencoded'} d = getPage(url = target_url, cookies=cookies, agent=useragent, postdata=body, method = 'POST', headers=headers) defer.returnValue(d) i can see the output: 222 somenumber, so apparently getReloadInterval works. it returns some magic number as expected now i call getUserList from this function: (response is 1, so d = getUserList gets evaluated) but there i get: 222 5821 Unhandled error in Deferred (short version, complete error at the end) Traceback (most recent call last): File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult def printLoginResult(response): #we are in, login accepted if response == ' 1': print "we are in, get user list" #printChat(response) def printResult(result): print repr(result) d = getUserList() d.addCallback(printResult) d.addErrback(printResult) why is printResult not run? Unhandled error in Deferred: Traceback (most recent call last): File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult _inlineCallbacks(r, g, deferred) File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1171, in _inlineCallbacks deferred.callback(e.value) File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 393, in callback self._startRunCallbacks(result) File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult _inlineCallbacks(r, g, deferred) File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1171, in _inlineCallbacks deferred.callback(e.value) File "/home/julius/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 392, in callback assert not isinstance(result, Deferred) exceptions.AssertionError:

1 0

[Twisted-Python] buildbot is back
by Glyph 06 Jun '16

06 Jun '16

When migrating it to a new VM, we seem to have somewhat underprovisioned buildbot.twistedmatrix.com. I've just moved it from a 1G to a 2G instance. Sorry about the downtime! Hopefully this is enough RAM for it to do its job. It seemed to happily consume only a few hundred megs when we were doing test builds, but there's some kind of load spike that happens periodically which was OOM killing it and I believe this change should fix it. -glyph

1 0

[Twisted-Python] Need clarification on reviews for Python 3 fixes for Twisted
by Craig Rodrigues 06 Jun '16

06 Jun '16

Hi, I have submitted some Python 3 patches for Twisted. 1. PATCHES REVIEWED AND COMMITTED TO TRUNK ============================================= Use new syntax for catching exceptions https://twistedmatrix.com/trac/ticket/8344 Use new syntax for raising exceptions https://twistedmatrix.com/trac/ticket/8345 Remove old syntax for octal literals https://twistedmatrix.com/trac/ticket/8347 Fix parentheses in list comprehensions https://twistedmatrix.com/trac/ticket/8351 2. PATCHES WHICH STILL NEED TO BE REVIEWED, NOT YET COMMITTED TO TRUNK ======================================================================= Change print to print() http://twistedmatrix.com/trac/ticket/5812 Change foo.has_key(bar) to "bar in foo" https://twistedmatrix.com/trac/ticket/8359 https://twistedmatrix.com/trac/ticket/8360 https://twistedmatrix.com/trac/ticket/8361 https://twistedmatrix.com/trac/ticket/8362 https://twistedmatrix.com/trac/ticket/8363 https://twistedmatrix.com/trac/ticket/8364 https://twistedmatrix.com/trac/ticket/8365 Eliminate the use of long literals https://twistedmatrix.com/trac/ticket/8366 Remove use of tuple parameter packing https://twistedmatrix.com/trac/ticket/8346 Adi has reviewed and committed the patches in 1. However, Adi has mentioned that in this document: http://twistedmatrix.com/trac/wiki/Plan/Python3, the strategy of submitting incremental Python3 fixes is not mentioned. Before doing any further reviews, Adi would like clarification that these types of reviews/patches are OK for submission and review. Are they OK? Would it be possible extend the Plan/Python3 document to accept incremental Python3 fixes as long as: * adheres to Twisted coding standards * works on Python 2.7 * passes existing tests * comes with new tests if functionality is changed that is not currently being tested My experience working with Python3 on other projects, is that incremental fixes is easier to review and get working, rather than an all or nothing approach. Some Python3 porting such as bytes/string/unicode or Python C API changes are very hard, while print vs. print() are very easy. Holding up the easy changes, until every hard change is also done is quite hard, and slows things down. -- Craig

5 10

[Twisted-Python] Why deferToThread is so slow?
by Nagy, Attila 03 Jun '16

03 Jun '16

Hi, I have a thread safe synchronous library, which I would like to use in a threadpool using deferToThread. Without using (deferTo)threads I get consistent 1-3 ms response times, with deferring to threadpool, I get 30-300, varying wildly. I've tried to construct a test, which is of course not perfect, but shows some differences, which widen with the number of threads (while directly calling that dumb function remains the same). I can see similar results with using the (database) library, but because it's an external dependency and performance gets worse with the number of queries (threads), I thought it would be best to leave that out of the picture, hence the dumb function. # python /tmp/deft.py 1 deferToThread: avg 316.17 us, sync: avg 1.38 us, 228.71x increase deferToThread: avg 312.92 us, sync: avg 1.38 us, 226.96x increase deferToThread: avg 320.22 us, sync: avg 1.39 us, 230.37x increase deferToThread: avg 317.33 us, sync: avg 1.35 us, 235.24x increase # python /tmp/deft.py 8 deferToThread: avg 2542.90 us, sync: avg 1.37 us, 1854.14x increase deferToThread: avg 2544.50 us, sync: avg 1.35 us, 1878.13x increase deferToThread: avg 2544.47 us, sync: avg 1.36 us, 1864.52x increase deferToThread: avg 2544.52 us, sync: avg 1.38 us, 1839.01x increase deferToThread: avg 2544.92 us, sync: avg 1.36 us, 1871.81x increase deferToThread: avg 2546.71 us, sync: avg 1.39 us, 1830.35x increase deferToThread: avg 2552.38 us, sync: avg 1.35 us, 1893.17x increase deferToThread: avg 2552.40 us, sync: avg 1.36 us, 1870.20x increase # python /tmp/deft.py 16 deferToThread: avg 4745.76 us, sync: avg 1.26 us, 3770.11x increase deferToThread: avg 4748.67 us, sync: avg 1.24 us, 3817.03x increase deferToThread: avg 4749.81 us, sync: avg 1.26 us, 3756.39x increase deferToThread: avg 4749.72 us, sync: avg 1.24 us, 3839.88x increase deferToThread: avg 4749.87 us, sync: avg 1.28 us, 3709.99x increase deferToThread: avg 4752.63 us, sync: avg 1.24 us, 3842.90x increase deferToThread: avg 4752.53 us, sync: avg 1.23 us, 3866.08x increase deferToThread: avg 4752.55 us, sync: avg 1.23 us, 3855.40x increase deferToThread: avg 4754.03 us, sync: avg 1.29 us, 3678.09x increase deferToThread: avg 4754.97 us, sync: avg 1.25 us, 3817.19x increase deferToThread: avg 4755.45 us, sync: avg 1.32 us, 3593.28x increase deferToThread: avg 4756.35 us, sync: avg 1.25 us, 3804.18x increase deferToThread: avg 4756.19 us, sync: avg 1.29 us, 3687.73x increase deferToThread: avg 4757.19 us, sync: avg 1.23 us, 3860.74x increase deferToThread: avg 4758.02 us, sync: avg 1.24 us, 3824.33x increase deferToThread: avg 4759.63 us, sync: avg 1.24 us, 3830.40x increase On 16 threads it takes nearly 5 ms(!) to call a local function... The test program is here: https://gist.github.com/bra-fsn/1fd481b44590a939e849cb9073ba1a41 cpython 2.7.11, Twisted 16.2.0 BTW, I've run this on pypy 5.1.1 with even worse results: https://mail.python.org/pipermail/pypy-dev/2016-June/014477.html

4 5