Twisted March 2018

twisted@python.org

20 participants
24 discussions

Start a nNew thread

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 11 Mar '18

11 Mar '18

1 0

[Twisted-Python] Circular references in TLSMemoryBIOProtocol
by Ilya Skriblovsky 10 Mar '18

10 Mar '18

Hello, I have the Twisted app that serves tons of short-lived TLS connections using TLSMemoryBIOFactory. I usually set loosened garbage collector thresholds in production environment for the sake of performance. But I've noticed that this app's RAM usage quickly grows up to unreasonable values. Digging into the issue using pdb and objgraph showed that protocol instances are still living long after they were closed. I found two circular dependencies which are created for each TLS connection: 1. Between twisted.protocols.policies.ProtocolWrapper and its self.wrappedProtocol 2. Between twisted.protocols.tls.TLSMemoryBIOProtocol and its self._tlsConnection Both of them cause protocol instance to not be deleted when the connection is closed. So all OpenSSL-related objects and all business-related data attached to that protocol instance are still living untill the next GC collection. This affects both RAM usage and performance (due to much more often GC collections) I've tried to fix both circular dependencies: replaced https://github.com/twisted/twisted/blob/trunk/src/twisted/protocols/policie… by self.wrappedProtocol.makeConnection(weakref.proxy(self)) and replaced https://github.com/twisted/twisted/blob/trunk/src/twisted/protocols/tls.py#… by: self._tlsConnection = self.factory._createConnection(weakref.proxy(self)) Memory usage pattern changed drastically after this change. I've created demo script that makes 10k TLS loopback connections with GC disabled and measures the number of objects are still living after the work is done and total resident RAM consumption: https://gist.github.com/IlyaSkriblovsky/4dd3abfd5f67c64b13f1c673f56466f9 Output without the fix: N = 10000 , K = 100 objects before 50136 DummyServerProtocols still living 10000 objects after 439919 mem 778 mb Output with the fix: N = 10000 , K = 100 objects before 50133 DummyServerProtocols still living 0 objects after 159919 mem 96 mb So using weakrefs makes all protocol instances and instances of TLSMemoryBIOProtocol to be deleted right after a connection is closed. Less circular-dependent objects → less GC invocations → better performance. And I see much nicer RAM usage pattern in my app. Is it possible to fix circular deps in some more clean way? Can this be solved at all while user's code is able to try to touch both sides of circular dep after connection is closed? Please advice Thanks for consideration Best regards, Ilya

4 15

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 04 Mar '18

04 Mar '18

1 0

[Twisted-Python] Does Twisted support ECC for TLS?
by Justin Myles Holmes 04 Mar '18

04 Mar '18

I'm trying to bend a little bit of custom TLS - one possible use case for me is a HendrixDeploy object which uses an ethereum keypair to self-sign a certificate. So I'm wondering: is it currently possible to use an ECC keypair for TLS with Twisted? Here's what I've discovered: twisted.internet.sssl.ContextFactory has a method, use_privatekey(). This thing wants an OpenSSL.crypto.PKey object. And, lo and behold, PKey offers a facility, from_cryptography_key(), which attempts to use a key from cryptography.io, from whence I'm generating keys anyway. However, it expects an RSA or DSA key, not an EC or ECDSA key. Glyph suggested that, instead of trying to handle PKeys myself, I might try loading PEM files with txsni or the like. I can actually get txsni to work with my cert/keypair, but I don't seem to be able to get a client to connect. For example, Firefox tells me "SSL_ERROR_NO_CYPHER_OVERLAP". I tried the same things with SSL4ServerEndpoint, and I get exactly the same thing - my protocol's dataReceived method is never run, no output appears in the console, but the client gets this same error. I notice that there's an issue on PyOpenSSL which appears to address this: -- Justin Myles Holmes justinholmes.com thisisthebus.com github.com/jMyles/

2 3

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

Twisted March 2018