Mailman 3 July 2016 - Twisted

[Twisted-Python] Fixes for Twisted bindings in M2Crypto
by Craig Rodrigues 26 Jul '16

26 Jul '16

On Tue, Jul 26, 2016 at 8:47 AM, Matěj Cepl <mcepl(a)cepl.eu> wrote: > > I believe I have fixed all I can do without actually understanding Twisted > in > https://gitlab.com/mcepl/m2crypto/commit/6cd5f87b31e50016ebb7e44f3f2ae46610…. > So now, if Twisted is so transparent and perfectly understandable, could > you please suggest, what I do wrong, that the test ends in the endless loop > (https://travis-ci.org/mcepl/M2Crypto/builds/147175901)? > > In your initial e-mail, it would have been useful if you could have provided reproduction steps for your problem. Can you provide reproduction steps? Since I wasn't sure, I took a guess. I did the following inside a Python 3.6 virtual environment under OS X: git clone https://gitlab.com/mcepl/m2crypto.git m2crypto_test cd m2crypto_test git checkout python3 python setup.py build --openssl=/usr/local/opt/openssl python setup.py bdist python setup.py develop py.test -v -s -k test_twisted_wrapper tests/test_ssl.py I was able to reproduce the problem with something looping around: DEBUG:_encrypt:self.data = "b'GET / HTTP/1.0\n\n\r\n'" DEBUG:_decrypt:self.encrypted = "b''" DEBUG:_encrypt:self.data = "b'GET / HTTP/1.0\n\n\r\n'" DEBUG:_decrypt:self.encrypted = "b''" DEBUG:_encrypt:self.data = "b'GET / HTTP/1.0\n\n\r\n'" DEBUG:_decrypt:self.encrypted = "b''" DEBUG:_encrypt:self.data = "b'GET / HTTP/1.0\n\n\r\n'" DEBUG:_decrypt:self.encrypted = "b''" I took a look in the M2Crypto code which interfaces with Twisted, and found that the dataReceived() loop was never terminating because dataReceived() takes bytes, but the code was comparing to an emptry str. So, the loop was never terminating. I fixed this and other problems, and now the test_twisted_wrapper passes on Python 3. I submitted this merge request: https://gitlab.com/mcepl/m2crypto/merge_requests/3/commits While I was looking at the code, I also saw that the make_certs.py utility does not work on Python 3. I fixed that, and submitted this merge request: https://gitlab.com/mcepl/m2crypto/merge_requests/4/commits -- Craig

1 0

[Twisted-Python] Pydoc parameter formatting and explaining interfaces
by Daniel Sank 26 Jul '16

26 Jul '16

This is a branch from the thread with subject "Request for help with Twisted bindings in M2Crypt". Regarding my inability to read documentation: > This does at least point to a real problem with pydoctor in the way it presents types. > It should probably put them in their own colored box, not use the string 'type' or > parentheses to offset them, and put the type closer to (rather than farther from) the > parameter name. Would you mind filing a bug on pydoctor? Or commenting on > one if it already exists? :) Done: https://github.com/twisted/pydoctor/issues/121 >> Some years ago when I tried to understand Twisted's use of interfaces via Twisted's >> own documentation (which included something about hair dryers and voltage standards) >> I was puzzled by the fact that the examples didn't really show me how to solve a useful >> problem (or I was too stupid to understand that the examples did in fact do that) despite >> the fact that I knew what an interface was in general terms. It was a case of >> understanding the intent but none of the examples. > OK... it's a fair cop. I'm unfamiliar with that term. > Among other things, it's mainly trying to explain adaptation, which sort of puts the cart before > the horse Yes! That is definitely a big part of the problem. When I think "interface" I think "methods and their signatures an object promises to provided". Adaptation is a detail, so to speak. It's also somewhat confusing that the discussion begins with shapes as it goes over the basic idea of interfaces, and then switches to hair dryers when it comes time for an example. It would be easier to read if the examples were more consistent. > and automatic adaptation is increasingly considered spooky action-at-a-distance within > Twisted code. All the more reason to not use adaptation as the in-your-face example. > You're the perfect person to submit patches against this doc, by the way, since you have a > firm grasp of the whole "abstract interface" thing but also found it confusing. Perhaps. On the other hand I think it might be better to replace Twisted's own documentation with a link to zope's, or at least put the link at the top and say "read this before reading our examples about adaptation." We'll see if such a patch receives any love. -- Daniel Sank

3 2

[Twisted-Python] Request for help with Twisted bindings in M2Crypto
by Matěj Cepl 26 Jul '16

26 Jul '16

Hello, I took over a maintenance of (surprisingly) still quite popular M2Crypto project in the last year. I have just released 0.25.0 which is my fifth release during that time and I think we are slowly but surely moving towards porting to py3k, cleaning up the code, etc. I am now working on porting to py3k, but the biggest PITA for me (aside from the Windows Pain™ ;)) is the Twisted integration module (https://gitlab.com/m2crypto/m2crypto/blob/python3/M2Crypto/SSL/TwistedProto…). 1) I get bugs like https://gitlab.com/m2crypto/m2crypto/issues/111 which I have no idea how to solve, because I don't understand the deep magic which Twisted seems to me, and I am not even sure that M2Crypto is in fault here (not mentioning that I have a hard time to reproduce). 2) Even more pressing is that the Twisted module breaks my tests when porting to py3k (https://travis-ci.org/mcepl/M2Crypto/jobs/146633964). Given the opaque and complicated data types in Twisted, I see horribly complicated task of diving into it in front of me and I am not eager. 3) Moreover, I would like to know how much interest there is in maintaining the M2Crypto module for Twisted. I got some hope from http://twistedmatrix.com/trac/wiki/TransportLayerSecurity which seems like there is an interest in more complete OpenSSL bindings, but OTOH I see on the list that Twisted now seems to use more and more of Cryptography (why in the world somebody made such confusing name of their project ...). Obviously the most simple way for me to be cutting Twisted module from M2Crypto and let it be (although I am afraid I have still some legacy users who would like to see it maintained, and given that the legacy support is still the most important reason for maintaining M2Crypto, I don't want to give up lightly). Moreover, I am suspicious that for somebody who actually understands Twisted, most of my problems are trivial and they could be solved easily. So, before I start studying http://krondo.com/slow-poetry-and-the-apocalypse/ (is there some better tutorial from ground up for complete idiots?), I would like to solicit help here for help with this module. Would somebody raise up their hand to help me and help Twisted? Best, Matěj -- https://matej.ceplovi.cz/blog/, Jabber: mcepl(a)ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 Give a man a regular expression and he’ll match a string… teach him to make his own regular expressions and you’ve got a man with problems. -- yakugo in http://regex.info/blog/2006-09-15/247#comment-3022

5 16

[Twisted-Python] New users cannot subscribe to mailing list
by Craig Rodrigues 23 Jul '16

23 Jul '16

Hi, I have talked to two recent contributors to Twisted, and both mentioned to me that they cannot subscribe to the mailing list. The mailing list is essentially unusable to new subscribers. Is anyone working on fixing this? -- Craig

4 15

[Twisted-Python] buildbot.twistedmatrix.com is down a lot
by Craig Rodrigues 21 Jul '16

21 Jul '16

In the past few days, buildbot.twistedmatrix.com seems to be down all the time, and requires manual restarts. As I write this, it is down right now. Is there something wrong with the hardware involved with buildbot.twistedmatrix.com? -- Craig

5 13

[Twisted-Python] pywincffi status update and next steps
by Oliver Palmer 21 Jul '16

21 Jul '16

All (or at least those interested in replacing pywin32 in Twisted), pywincffi is getting very close to having all the API calls necessary to replace pywin32. By my count there should be one function left, GetOverlappedResult(). All of the other functions and constants in use today should be covered and ready to start swapping into Twisted. https://github.com/opalmer/pywincffi/issues/69 There's a couple of reasons I've been holding off on any integration work. First, Twisted itself has shifted development and testing workflows to public infrastructure which overall should make integration easier. The other issue right now seems to be some of the tests are not passing on AppVeyor and it's not testing across as many Python versions. These are not big issues for pywincffi itself but I think it's probably a good idea to do the integration on top of a known stable build so we can all be confident in the results. I plan to spend some time this weekend taking a look at the build starting off by helping Craig (see thread: "Windows buildbots and Appveyor") add support for more versions of Python. After that, unless someone is already working on it, I can start to take a look at the existing test failures on Windows. To wrap up, a big thank you to those who've reached out to help move pywincffi forward so far. Especially exvito who came up with a system to wrap cffi types, added functions, fixed some tests and contributed to several improvements on the development process. ---Oliver

3 2

[Twisted-Python] Twisted 16.3.0 Prerelease 2 Announcement
by Amber "Hawkie" Brown 21 Jul '16

21 Jul '16

Hi everyone, Here's another prerelease in the 16.3 series -- fixing a 16.2 regression in HTTP timeouts not working. For more information, check the NEWS file (link provided below). As usual, it's available for download -- go here (https://twistedmatrix.com/Releases/pre/16.3.0pre2/) to get the prerelease tarballs and the full NEWS file. If you want to install it right away, run: pip install https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2 A reminder that if you would like to try out the newly-landed HTTP/2 support, run: pip install -U https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2… This will download the new HTTP/2 dependencies and the TLS requirements as well. Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :) Twisted Regards, Amber Brown (HawkOwl)

8 28

[Twisted-Python] Python SSH Server(twisted.conch) set ChallengeResponseAuthentication to yes
by 陈健 20 Jul '16

20 Jul '16

hi: I wrote a SSH server with Twisted Conch. I want to support for the 'keyboard-interactive' authentication .ChallengeResponseAuthentication controls support for the 'keyboard-interactive' authentication scheme defined in RFC-4256. The 'keyboard-interactive' authentication scheme could, in theory, ask a user any number of multi-facited questions. In openssh the "ChallengeResponseAuthentication" option can be configured in "/etc/ssh/sshd_config". But in twisted i don't know how to do that enable the "ChallengeResponseAuthentication" option ? http://stackoverflow.com/questions/38364930/python-ssh-servertwisted-conch-… Best regards, ---jianchen

1 0

[Twisted-Python] Windows buildbots and Appveyor
by Craig Rodrigues 20 Jul '16

20 Jul '16

The Appveyor setup with Python 2.7 currently does not build any C extensions. This means that IOCP reactor is not currently built and none of the IOCP tests are run in Appveyor. I'm not sure offhand what the fix is, but I notice that quite a few Python projects import this script from Olivier Grisel: https://github.com/ogrisel/python-appveyor-demo/blob/master/appveyor/run_wi… which sets the environment correctly for the Windows compilers, and then uses the script from inside appveyor.yml with something like: https://github.com/ogrisel/python-appveyor-demo/blob/master/appveyor.yml It would be nice if we could get the C extensions buildin in Appveyor. The Appveyor setup does not have a Python 3 setup, so no Python 3 builds are done. There used to be a Python 3 buildbot, and although many of the tests were broken, it was producing some interesting logs. For example, I saw a lot of warnings in the SSL code where strings where being passed as arguments to API's which took bytes as arguments. Can we bring back at least one Windows Python 3 buildbot, and leave it as an unsupported, on-demand environment? I am working on getting more Windows Python 3 tests to pass, so this would be useful. -- Craig

2 3

[Twisted-Python] Twisted tests on public CI
by Adi Roiban 20 Jul '16

20 Jul '16

Hi, We now have Ubuntu / Windows Server and OS X running Twisted tests for each PR, using Travis-CI and Appveyor Non-`Twisted GitHub Team` members can now run a pretty extensive test suite for commits associated with a PR. The coverage report is 90.52% (+/- 0.01%) just by running tests on the public CI. As far as I can tell this is better than what we get with Buildbot builders (since we no longer have BSD slaves and Win7 coverage reporter is broken and Win2012 slave was removed) Some tests on Appveyor are still failing. Help here is much appreciated. We also got some tests failing on OS X but Amber has already started working to fix them https://twistedmatrix.com/trac/ticket/8639 OS X tests results will not block for now a merge, but you will have to wait for the OS X results. In this way we can see how much longer does a test run takes, now that we also have OS X. Appveyor was pretty fast in the last days... after the move to Rackspace. Travis-CI was a bit slow on the free queue... not sure how fast is the payed queue :) Please report any issues here or on Twisted Infrastructure tracker https://github.com/twisted-infra/braid/issues Regards, -- Adi Roiban

4 6