Twisted March 2018

twisted@python.org

20 participants
24 discussions

[Twisted-Python] RunningTwisted tests on Windows and SSL
by John Aherne 19 Mar '18

19 Mar '18

When I run the test suite on windows I end up with the following errors and failures - see below. So my basic question is. Can anyone tell me what the latest state is for running tests on windows and in particular the state of running SSL on windows. When I look at the buildbots status by and large they all show the windows results as green for python 2.7 and 3.6. I assume then that the tests are passing. But maybe I misunderstand what the buildbot status is telling me. So should I be seeing these errors when the buildbots all show green. And if I should not, what am I doing wrong when running the tests. Another point is regarding SSL which is why I am running the tests in the first place. I am looking at https://twistedmatrix.com/trac/ticket/6371 with a view to getting the wincertstore working to load the system certificates. When I checked the standard library they are also loading the windows certificate store. So it seems best to stay in sync with them. However, I see the following message from the test results that imply that the SSL port to python3 has not been completed. That stops me in my tracks and I wonder whether the SSL port for python3 is waiting for some fundamental work to be finished off. [SKIPPED] Re-enable once the Python 3 SSL port is done. twisted.test.test_internet.PortStringificationTests.testSSL Nevertheless, when I add the patch 6371 for _sslverify and run the tests for it, all works fine. Then I run an https get using treq against www.google.com and twistedmatrix.com I get a 200 and the results back, If I run the treq test without wincertstore installed, it fails with a wincertstore not installed message. These simple tests imply that SSL is working, but I may be glossing over some important missing feature. Has anyone got some helpful suggestions? Below are the test results: ------------------------------- First for Python2.7.13 32bit Twisted 17.9.1 Windows 2012 R2 Server --------------------------------------------------------------------------------------------- The sslChainLoads test I can work around by fixing the line endings. The SSHCheckerTests I think is because conch plugin is not loaded. However I am under the impression that conch did not work on windows anyway. [ERROR] raise InvalidAuthType(authType) twisted.cred.strcred.InvalidAuthType: sshkey twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker =============================================================================== [ERROR] Traceback (most recent call last): exceptions.ValueError: Specified chain file 'c:\twistedssl\twisted\src\twisted\internet\test\fake_CAs\chain.pem' doesn't contain any valid certificates in PEM format. twisted.internet.test.test_endpoints.ServerStringTests.test_sslChainLoads ------------------------------------------------------------------------------- Ran 11883 tests in 629.094s FAILED (skips=2890, errors=2, successes=8991) --------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------- Second for python 3.6.3 32 bit twisted 17.9.1 on Windows 10 Pro Version 1709 (OS build 16229.309) --------------------------------------------------------------------------------------------------------------------- The StringIO errors are because it is not available on python3 The SSH check error is I imagine the same as for python2.7 The mail and test_hook errors I have not looked into yet. Python3.6-32 [SKIPPED] Re-enable once the Python 3 SSL port is done. twisted.test.test_internet.PortStringificationTests.testSSL [FAIL] Traceback (most recent call last): raise self.failureException(msg) twisted.trial.unittest.FailTest: True is not false : Module twisted.mail.alias exists twisted.python.test.test_dist3.ModulesToInstallTests.test_notexist =============================================================================== [ERROR] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\cred\strcred.py", line 114, in findCheckerFactory raise InvalidAuthType(authType) twisted.cred.strcred.InvalidAuthType: sshkey twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker =============================================================================== [ERROR] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\mail\test\test_mail.py", line 12, in <module> import StringIO builtins.ModuleNotFoundError: No module named 'StringIO' twisted.mail.test.test_mail =============================================================================== [ERROR] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\mail\test\test_mailmail.py", line 10, in <module> from StringIO import StringIO builtins.ModuleNotFoundError: No module named 'StringIO' twisted.mail.test.test_mailmail =============================================================================== [ERROR] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\mail\maildir.py", line 20, in <module> import StringIO builtins.ModuleNotFoundError: No module named 'StringIO' twisted.mail.test.test_options =============================================================================== [ERROR] Traceback (most recent call last): from email.Parser import Parser builtins.ModuleNotFoundError: No module named 'email.Parser' twisted.news.test.test_database =============================================================================== [ERROR] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\news\database.py", line 11, in <module> import StringIO builtins.ModuleNotFoundError: No module named 'StringIO' twisted.news.test.test_news =============================================================================== [ERROR] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\news\database.py", line 11, in <module> import StringIO builtins.ModuleNotFoundError: No module named 'StringIO' twisted.news.test.test_nntp =============================================================================== [ERROR] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\test\test_hook.py", line 89, in setUp BaseClass.__dict__.clear() builtins.AttributeError: 'mappingproxy' object has no attribute 'clear' twisted.test.test_hook.HookTests.testBaseHook twisted.test.test_hook.HookTests.testSubHook ------------------------------------------------------------------------------- Ran 11751 tests in 299.633s FAILED (skips=2770, failures=1, errors=9, successes=8971) -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 2

[Twisted-Python] we can stop depending on pypiwin32
by Glyph 18 Mar '18

18 Mar '18

As per the release announcement in https://mail.python.org/pipermail/python-win32/2018-February/014000.html <https://mail.python.org/pipermail/python-win32/2018-February/014000.html>, PyWin32 now does development on Github, and uploads binary wheels to PyPI, as you can see here: https://pypi.org/project/pywin32/223/#files <https://pypi.org/project/pywin32/223/#files>. So we can stop depending on my ill-fated pypiwin32 fork. If anybody is interested in doing some minor Windows maintenance, I would like very much to never think about PyPIWin32 again :). -g

1 0

[Twisted-Python] RunningTwisted tests on Windows and SSL
by John Aherne 18 Mar '18

18 Mar '18

This is useful info for anyone else running into these issues. At least they might save a bit of time not having to run down what is happening. Seems that the mail,email, test_hook and StringIO errors all come from not ported modules as seen in twisted\src\python\_setup.py. The _getEllipticCurve error << builtins.AttributeError: type object 'OpenSSLCertificateOptions' has no attribute '_getEllipticCurve' twisted.test.test_sslverify.OpenSSLOptionsTests.test_ecDoesNotBreakConstructor >> is because getattr will raise an AttributeError, which is what it is doing here. But it does not do this for python2.7. The test is not to break if the ECCurve is missing. It is missing since it is not found. Still looking at this. With wincertstore being used, we get a failing test as follows: [FAIL] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\web\test\test_agent.py", line 1568, in test_setsTrustRootOnContextToDefaultTrustRoot certificateOptions.trustRoot, ssl.OpenSSLDefaultPaths) File "c:\twistedssl\twisted\src\twisted\trial\_synctest.py", line 649, in assertIsInstance instance, classOrTuple, suffix)) twisted.trial.unittest.FailTest: <twisted.internet._sslverify.OpenSSLWindowsCertificateAuthorities object at 0x0CFDA810> is not an instance of <class 'twisted.internet._sslverify.OpenSSLDefaultPaths'> twisted.web.test.test_agent.WebClientContextFactoryTests.test_setsTrustRootOnContextToDefaultTrustRoot So a new test is needed to fix this. Also, the current patch only loads the ROOT certificates. The standard library loads ['CA', 'ROOT'], so we should be doing the same I imagine. -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

1 0

[Twisted-Python] Buildmaster is down
by Adi Roiban 18 Mar '18

18 Mar '18

Hi, Today, I tried to reconfigure the Buildbot master from braid/master and it failed. It looks like it is missing the configuration for some slaves. I have checked the private files, and it looks like newer slaves are missing. I guess that the changes were made outside braid or of the twisted-infra-secret repo and were overwritten when I have applied the braid changes. I don't know from where to get the latest list of slaves. I hope that someone can help:) Thanks! -- Adi Roiban

2 2

[Twisted-Python] New member of the twisted/twisted-contributors team.
by Adi Roiban 18 Mar '18

18 Mar '18

Hi, Just a quick announcement. I made Ilya Skriblovsky (https://github.com/IlyaSkriblovsky) a member of twisted/twisted-contributors team. The main reason is to make it easier to work with Ilya and allow him to run tests on Buildbot. Let me know if you want this change reverted. Regards, -- Adi Roiban

3 3

[Twisted-Python] RunningTwisted tests on Windows and SSL
by John Aherne 17 Mar '18

17 Mar '18

Some clarification on running the tests. For the tests on test_sslverify If I apply the patches from 6371 on python2.7.13 32bit windows2012R2, the test_sslverify runs without error or failure. TrustRootTests test_caCertsPlatformDefaults ... [OK] test_caCertsWindows ... [OK] test_trustRootPlatformRejectsUntrustedCA ... [OK] test_trustRootSpecificCertificate ... [OK] ============================================================ =================== [SKIPPED] NPN/ALPN is present on this platform twisted.test.test_sslverify.NPNAndALPNAbsentTests.test_ NPNAndALPNNotImplemented twisted.test.test_sslverify.NPNAndALPNAbsentTests.test_ NegotiatedProtocolReturns None twisted.test.test_sslverify.NPNAndALPNAbsentTests.test_ nextProtocolMechanismsNoN egotiationSupported ------------------------------------------------------------ ------------------- Ran 113 tests in 496.749s PASSED (skips=3, successes=110) ------------------------------------------------------------ --------------------------------------------------------------------------- If I apply the patch on python3.6.3 32bit Windows10Pro, the test_sslverify produces 1 error and 1 failure. The test_caCertsWindows passes Looking at the Error on test_ecDoesNotBreakConstructor, I don't see where I can find _getEllipticCurve. I can't see it in OpenSSLCertificateOptions. And so far google does not help ============================================================ =================== [FAIL] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\test\test_sslverify.py", line 1292, in test_ecSuccessWithRealBindings self.assertEqual(defaultCurve, opts._ecCurve) File "c:\twistedssl\twisted\src\twisted\trial\_synctest.py", line 432, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "c:\program files (x86)\python36-32\Lib\unittest\case.py", line 829, in assertEqual assertion_func(first, second, msg=msg) File "c:\program files (x86)\python36-32\Lib\unittest\case.py", line 822, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: <Curve 'prime256v1'> != <twisted.internet._sslverify._OpenSSLECCurve object at 0x06B4FDF0> twisted.test.test_sslverify.OpenSSLOptionsTests.test_ ecSuccessWithRealBindings ============================================================ =================== [ERROR] Traceback (most recent call last): File "c:\twistedssl\twisted\src\twisted\test\test_sslverify.py", line 1254, in test_ecDoesNotBreakConstructor sslverify.OpenSSLCertificateOptions, "_getEllipticCurve", missing) File "c:\twistedssl\twisted\src\twisted\trial\_synctest.py", line 1086, in patch monkeyPatch.patch() File "c:\twistedssl\twisted\src\twisted\python\monkey.py", line 53, in patch self._originals.append((obj, name, getattr(obj, name))) builtins.AttributeError: type object 'OpenSSLCertificateOptions' has no attribute '_getEllipticCurve' twisted.test.test_sslverify.OpenSSLOptionsTests.test_ ecDoesNotBreakConstructor ------------------------------------------------------------ ------------------- Ran 113 tests in 185.166s FAILED (skips=3, failures=1, errors=1, successes=108) The following is what monkeypatch is working with test_ecDoesNotBreakConstructor ... *******PATCH******* <class 'twisted.internet._sslverify.OpenSSLCertificateOptions'> *******PATCH******* _getEllipticCurve [ERROR] test_caCertsPlatformDefaults ... [OK] test_caCertsWindows ... [OK] test_trustRootPlatformRejectsUntrustedCA ... [OK] Any pointers would be welcome John Aherne -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

1 0

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 17 Mar '18

17 Mar '18

1 0

[Twisted-Python] INCOMPATIBLE CHANGE: Removal of t.c.ssh.filetransfer.FileTransferClient.wasAFile
by Wim Lewis 17 Mar '18

17 Mar '18

In PR 949 ("Remove “terrible hack” in t.c.ssh.filetransfer.FileTransferClient", trac bug 9362) I'm removing an attribute of FileTransferClient which I'm pretty sure was never intended to be public. Adi Roiban points out that this is technically a compatibility-breaking change, though, and so it needs more public review. My reasoning for thinking that this will not break any user code: the attribute, although not prefixed with an underscore, was undocumented; anyone relying on it would have had to read the source code to discover what it does, and in the source it was commented as a "TERRIBLE HACK". I'd hope this would be enough of a warning. (Also, I can't think of a use for it.) trac: https://twistedmatrix.com/trac/ticket/9362 pr: https://github.com/twisted/twisted/pull/949

3 2

[Twisted-Python] List of required builds before a merge
by Adi Roiban 13 Mar '18

13 Mar '18

Hi, It is not clear to me what builders need to pass before we can merge something. I expect that all supported "platforms" need to pass, but it is not clear what are the currently supported platforms. We have this info in the wiki but it does not help. https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:Howtomergethechan… In GitHub I can see Travis / Appveyor and OSX from Buildot as "Required" Is that all? ---------- If I check the "supported" group in Buildbot, I see many more builders. The problem is that a significant number of slaves are down and those builders are not available. -------- Is Fedora still supported and required? ------- I suggest to use GitHub "Required" marker to document what platforms are supported. We don't have time to maintain the infrastructure, so I suggest to drop support for anything that is not supported by Travis and Appveyor. I know that this might be disruptive. I think that we need it in order to raise awareness that supporting a platform is not easy. If someone (including me) cares about a platform they should find a way to help to project supporting that platform. What do you think? -- Adi Roiban

4 7

[Twisted-Python] Waiting for a contended resource
by Richard van der Hoff 12 Mar '18

12 Mar '18

Hi folks, I thought I'd poll the list on the best way to approach a problem in Twisted. The background is that we have a number of resources which can be requested by a REST client, and which are calculated on demand. The calculation is moderately expensive (can take multiple seconds), so the results of the calculation are cached so multiple lookups of the same resource are more efficient. The problem comes in trying to handle multiple clients requesting the same resource at once. Obviously if 200 clients all request the same resource at the same time, we don't want to fire off 200 calculation requests. The approach we adopted was, effectively, to maintain a lock for each resource: > lock = defer.DeferredLock() > cached_result = None > > @defer.inlineCallbacks > def getResource(): > yield lock.acquire() > try: > if cached_result is None: > cached_result = yield do_expensive_calculation() > defer.returnValue(cached_result) > finally: > lock.release() (Of course one can optimise the above to avoid getting the lock if we already have the cached result - I've omitted that for simplicity.) That's all very well, but it falls down when we get more than about 200 requests for the same resource: once the calculation completes, we can suddenly serve all the requests, and the Deferreds returned by DeferredLock end up chaining together in a way that overflows the stack. I reported this as http://twistedmatrix.com/trac/ticket/9304 and, at the time, worked around it by adding a call to reactor.callLater(0) into our implementation. However, Jean-Paul's comments on that bug implied that we were approaching the problem in completely the wrong way, and instead we should be avoiding queuing up work like this in the first place. It's worth reiterating that the requests arrive from REST clients which we have no direct control over. We *could* keep track of the number of waiting clients, and make the API respond with a 5xx error or similar if that number gets too high, with the expectation that the client retries - but one concern would be that the load from the additional HTTP traffic would outweigh any efficiency gained by not stacking up Deferreds. So, I'd welcome any advice on better ways to approach the problem. Richard

4 5

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

Twisted March 2018