Mailman 3 November 2014 - Twisted

[Twisted-Python] twisted ssl server and client
by John Aherne 10 Nov '14

10 Nov '14

Hynek Well, thank you very much! Put it in and tested with request package and all seems to be working. Will test it later with the real system. That has saved me a lot of hassle. -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 1

[Twisted-Python] twisted ssl server and client
by John Aherne 10 Nov '14

10 Nov '14

<<You don’t need trustRoot for servers. You need to supply an extraCertChain which – as I’ve mentioned before – is easiest with the help of pem: https://github.com/hynek/pem#twisted>> Thanks. That link may be just what I need. I'll see how I get on. -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

1 0

[Twisted-Python] twisted ssl server and client
by John Aherne 10 Nov '14

10 Nov '14

So I am now looking at CertificateOptions in more detail. But I am stuck trying to figure out how to add my GoDaddy cert to trustRoot. Here is what I am doing so far to see what is happening. Bu I think I am getting further and further from what I should be doing. If anyone can point out what must be the obvious thing to do, GREAT! #!/usr/bin/env python # Copyright (c) Twisted Matrix Laboratories. # See LICENSE for details. import sys from twisted.internet import reactor, ssl, protocol, task, defer from twisted.python import log import echoserv log.startLogging(sys.stdout) ctx = ssl.CertificateOptions() certData2 = open('d:\\openssl\\ibook\\ibook2.pem', 'rb').read() certificate2 = ssl.PrivateCertificate.loadPEM(certData2) print 'CERT2', certificate2.options print 'CERT2', dir(certificate2) print 'CERT2 OPTIONS', certificate2.options() print 'CERT2OPTS', dir(certificate2.options) certGD = open('d:\\openssl\\ibook\\gd_bundle-g2-g1.crt', 'rb').read() #ctx.trustRoot = ssl.Certificate.loadPEM(certGD) certificate3 = certificate2.options(ssl.Certificate.loadPEM(certGD)) print 'cERT3', certificate3.privateKey print 'trustRoot2TROOT', certificate3.trustRoot #certificate2.options().trustRoot = ssl.Certificate.loadPEM(certGD) print 'trustRoot', certificate3 print 'trustRoot2type', type(certificate3.trustRoot) print 'trustRoot2TROOT', certificate3.trustRoot print 'trustRoot2PK', certificate3.privateKey print 'trustRoot2CERT', certificate3.certificate print 'trustRoot2VER', certificate3.verify #print 'trustRoot2', ctx.trustRoot factory = protocol.Factory.forProtocol(echoserv.Echo) reactor.listenSSL(8000, factory, certificate2.options(certificate3)) #reactor.listenSSL(8000, factory, certificate3) reactor.run() Thanks for any pointers. -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 1

[Twisted-Python] twisted ssl server and client
by John Aherne 09 Nov '14

09 Nov '14

Hynek Thanks for the information. Very interesting. I was wondering how to get the mozilla cert package. For the server I need to provide a certificate that can be verified by an external source. I used the requests package to test that it could verify my certificate without knowing exactly what it was doing. As far as I could see it verified the certificate and I then got an external source to also connect and send information having verified the certificate. That worked whereas before they had failed with SSL errors. So for the server, the certificate, key and godaddy bundle seem to be working. I found the DefaultOpenSSLContextFactory by googling for clues as to what to do. I then looked up the source to see what it required. And it seemed to do the trick. I have seen the documentation pointing at CertificateOptions but while I was trying to find out how I could get my godaddy bundle added in I found DefaultOpenSSlContextFactory which was easy to add my bundle to. It was not clear how I could do that with CertificateOptions. When I tried, I got unknown key argument because it seemed I could pass a key argument in but all my attempts failed. I have twisted 14.0.0 on one machine and twisted 14.0.2 on another and I bounce between them checking what works on one also works on the other. I have a lot of stuff running on the machine with 14.0.0 so I was not keen to upgrade until I had a clear idea of what I as doing. For the client I need to connect back to the external site. I can do this without verifying the certificate, but it would seem these days that is no longer sufficient. So I needed twisted.web.Agent to be able to verify the remote certificate. Trying that and once again googling for BrowserLikeContextFactory, I find that it has a real problem verifying hostnames. And your name features heavily in the list of people who have looked at the problem. I found that on windows 7 32 bit I could not get it to work. It always returned an SSL error. So for the time being I have abandoned that and am ignoring the problem. Just relying on finding and accepting any certificate. I need most of all to make sure that the server side is working properly, so your information is extremely welcome. I shall have to look at CertificateOptions again and see what the problem is. I have a feeling that everything is working with SSLv3 since all the error messages come back mentioning SSLv3. So I imagine at some point I need to be able to specify TLS and one of its variants. I hope I have explained a bit more clearly what I am trying to do. Regards -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

1 0

[Twisted-Python] twisted ssl server and client
by John Aherne 09 Nov '14

09 Nov '14

Thanks for the reply. In the end I took the examples in the docs and changed them to fit. So I have ended up with something that seems to work. But I wouldn't mind if someone can tell me if what I have done is miles wrong or spot on or could do with improvement or you have missed the point completely. Thanks for any pointers Below is the code with simple changes made #!/usr/bin/env python # Copyright (c) Twisted Matrix Laboratories. # See LICENSE for details. echoserv_ssl.py import sys from twisted.internet import reactor, ssl, protocol, task, defer from twisted.python import log import echoserv log.startLogging(sys.stdout) #certificate = ssl.PrivateCertificate.loadPEM(certData) certificate2 = ssl.DefaultOpenSSLContextFactory('d:\\openssl\\ibook\\ibookservernp.key', 'd:\\openssl\\ibook\\7044921f82b7.crt') print 'CERT2', certificate2 my_context2 = certificate2.getContext() my_context2.load_verify_locations('d:\\openssl\\ibook\\gd_bundle-g2-g1.crt') factory = protocol.Factory.forProtocol(echoserv.Echo) print 'CERT OPTIONS', dir(certificate2) reactor.listenSSL(8000, factory, certificate2) echoserv.py #!/usr/bin/env python # Copyright (c) Twisted Matrix Laboratories. # See LICENSE for details. from twisted.internet.protocol import Protocol, Factory from twisted.internet import reactor ### Protocol Implementation # This is just about the simplest possible protocol class Echo(Protocol): def dataReceived(self, data): """ As soon as any data is received, write it back. """ print 'MESSAGE', data self.transport.write(data) #'HELLO THERE?\r\n') def main(): f = Factory() f.protocol = Echo reactor.listenTCP(8000, f) reactor.run() if __name__ == '__main__': main() -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 1

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 09 Nov '14

09 Nov '14

1 0

[Twisted-Python] twisted ssl server and client
by John Aherne 06 Nov '14

06 Nov '14

Apologies in advance for the rather basic questions I have here, but I am a bit stuck. I am looking at ssl with twisted 14.0.0 I have loaded all the dependencies crypto, pycrypto, service_identity, six, idna, cffi, pyasn1, pyopenssl 0.14, openssl 1.0.1g I am running this on windows7, windows 2008r2 I have read through Using TLS in Twisted several times but still find myself not sure as to what I should do. I need both server and client set up. I have a server end where I have a GoDaddy certificate and certificate chain. So on the server end I need to pass to ssl.CertificateOptions the privatekey and certificate. I also need to pass in the location of the GoDaddy bundle. I have a set up using Cherrypy and this now works fine. But I need it working with Twisted. Now this is where I am not so clear. Should I pass the bundle as TrustRoot or as extraCertChain. The docs seem to say that on Windows there is no cert store to be used as trustRoot. If anyone can throw some light on this I would be very grateful. For the client using Twisted.web.Agent, I need to verify a different ssl certificate on another server system I connect to. At the moment, I can connect without verifying the certificate, but these days that is no longer acceptable. So I try the example in the docs for checking a certificate but it fails on all examples, including www.twistedmatrix.com. I assume this is because I do not have a default set of certificates in a store to check against. If I use the requests package I can get it to verify or not the server certificate, but when I turn to twisted it is not clear what I should be doing. The example specifies an 'authority' public.pem for the client to check against. So should I be looking for the way requests works where I do not specify an authority and it works by finding a bunch of certs somewhere. Or do I pass in a cert that is specific to the server I am connecting to and it will only check against that. This seems more specific and more secure. A final point. I need to be able to confirm which version of openssl I am connecting with. Is there a way to pin down which version pyopenssl is finding. Thanks for any information and pointers. -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 1

[Twisted-Python] The Path to Twisted 14.1
by HawkOwl 05 Nov '14

05 Nov '14

Hi everyone, To keep everyone in the loop... Since 14.0 came out months ago, and we really should have 14.1, I think it’s time we bit the bullet and cleaned out the regressions to get it out the door. Unfortunately, all of the currently open regressions are from the new logging system (#6750, plus #7548 and #7545) Since these regressions are introduced by a new ticket, we should revert them, and get the regressions fixed before it is remerged. I’ll make a new milestone called “New Twisted Logging” or similar so that they’re all still kept track of as a group. The only other big thing preventing 14.1 is https://twistedmatrix.com/trac/ticket/7647 , the patch that 14.0.1/14.0.2 was released to include. The patch fixes it, but not ideally, so it needs looking at. Once I’ve reverted those tickets and we can get #7647 refined and merged in some form, 14.1 will be out the door :) - hawkowl

4 10

[Twisted-Python] porting twisted.spread to Python3
by Wolfgang Rohdewald 03 Nov '14

03 Nov '14

I now have a local git branch with about 70 commits, always rebased onto current trunk. Those test suites pass with Python2 and Python3: test_banana, test_jelly, test_pb, test_pbfailure, test_persisted I also have a real app using PB (kajongg, a KDE game). Running the server and the client with any combination of Python 2 and Python 3 works but more testing is needed there. Especially transferring failures with non-ascii test, so I need to inject random errors. spread.banana is now fully covered by tests (was at 85% before). There is still a lot of code in twisted.spread which has no tests yet. I do not think I want to add missing tests for everything, only for those regions where I do changes or where it is not too difficult. And of course if an application triggers a bug yet uncovered by tests. Can you point me to other PB applications I could use for testing? Preferrably ones that are already mostly compatible with PY3 themselves, but that is not a _must_, I would try porting them unless it turns out to be too much work. And preferrably ones developed by people with a non-ascii native language. I do not yet trust Failures and logging with non-ascii. Below is the current list of commits. Everything you see there is still much in flux. This includes shuffling code between commits, combining and splitting commits, changing order. I want to open one ticket per commit. Simple things hopefully not needing much discussion first. Estimating one week per review/commit to trunk, this process will take more than one year. Often I can only add a new ticket after the previous one has been integrated because they depend on each other. So if it really seems to take that long, at some point I might just decide to put what I have to github and not care anymore. Which of course means I might have to distribute a patched twisted version with my application. I really want to leave Python2 behind, and only twisted is in the way. Latest versions of KDE do not have Python2 bindings anymore (KDE Frameworks 5). This log of course includes things being in tickets up for review. git log --pretty='format:%s' --reverse trunk..pb3 spread.banana: selecting an unsupported dialect now raises an Exception spread.banana now raises BananaError when receiving pb messages without pb being the selected dialect spread.banana: specific error message about unicode objects spread.banana: test coverage goes up from 85% to 100% test.test_banana: eliminate duplicate code twisted.python.reflect.safe_str on Python 3 converts utf-8 encoded bytes to clean str instead of "b'a'" test.test_{failure,banana,persisted,reflect}: consistently name all test methods test_* test.test_jelly: add tests for yet untested code spread.jelly: support the sets module only for Python 2 twisted.spread: Assume that Python always has unicode support enabled Ticket 7653. spread.jelly: deprecate unjellying instances spread.jelly can now transfer objects with new style classes cred.credentials: implements->implementer cred.checkers: implements->implementer test.test_pb: implements->implementer persisted.styles: add new tests persisted.styles.unpickleMethod: remove dead code spread.flavors: fix import paths, making them work with Python 3 too spread: Python 3: add from __future__ do not use argument tuple unpacking anymore, not available in Python 3 spread.publish and test.test_pb: replace file by open test.test_pb: use a constant for "BRAINS!" python.compat: get_im_func, get_im_self, get_im_class and use them in spread persisted.styles: simplify unpickleMethod persisted.styles: Port to Python 3 python.compat: for PY3, define long=int and xrange=range spread.banana: replace StringIO.StringIO by io.BytesIO python.compat.networkChar: New, and use it in spread.banana python.compat.networkString: improve Exception message spread.banana: change literals from "" to b"" where appropriate test.test_banana: fix a wrong assertion message test.test_banana: use itertools.chain test.test_banana: Python 3 does not know type long spread: Python 3 does not have sys.maxint spread.banana and test: Complete port to Python 3: handling of data spread.jelly: Unjellying instance is not supported with Python 3 spread.jelly: unify usage of the types module spread.jelly: consistently use _atom identifiers spread.jelly: complete port to Python 3 spread.pb, test.test_pbfailure: new syntax for except spread.pb: do not use argument tuple unpacking anymore, not available in Python 3 test.test_jelly now passes all tests with Python 3 cred.credentials.updateHash accepts unicode add ported modules to python/dist3.py spread.flavors: consistently use _atom identifiers spread.flavors: Python 3: eliminate __cmp__. Only support __eq__ and __ne__ spread: Use networkString/nativeString where appropriate spread.flavors: switch atoms from str to b"" test.test_pb: improve test_concurrentLogin: use a non-ascii password spread.pb: eliminate __cmp__, using FancyEqMixin test.test_pb: add comment to test_tooManyRefs: too slow spread.pb: introduce *_atom constants spread.pb: Python 3: use list() around some generators spread.pb.Broker: improve Exception message spread.pb and test: Python 3: xrange = range spread.pb and test: Python 3: use byte literals spread.pb: failure2Copyable: use _newInstance from jelly spread.pb: replace types.ListType with list spread.pb and test: use networkChar where appropriate test.test_pb: change from StringIO to io.BytesIO spread.pb: Python 3: make *_atom bytes python.logger.formatWithCall: support non-ascii strings. Why exactly does kajongg need this? python.compat: networkIdentifier, nativeIdentifier. Not yet used. SQUASH spread.pb: CopiedFailure now handles non-ascii strings. TEST MISSING spread.flavors: RemoteCopy supports non-ascii identifiers. TEST MISSING. Why does kajongg need this? python.reflect: _safeFormat must handle unicode python.util: new debug function stack(). REMOVE AGAIN python.compat: nativeIdentifier accepts non-unicode. NEEDED WHY? TEST CASE MISSING test.test_{banana,jelly,pb,pbfailure,persisted}: from __future__ test.test_pbfailure: make literal strings bytes test.test_pbfailure: replace StringIO with io.BytesIO spread.flavors: translate keys of received dict into nativeString. INCOMPLETE -- Wolfgang

4 8

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 02 Nov '14

02 Nov '14

1 0