-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/28/2011 05:02 PM, Christopher Armstrong wrote:
On Mon, Mar 28, 2011 at 9:44 AM, John Arbash Meinel <john@arbash-meinel.com <mailto:john@arbash-meinel.com>> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I'm doing some performance testing of one of our Twisted applications. And what I came across was a surprising amount of time being spent in twisted.python.failure.Failure.__getstate__
So under profiling, we spent 408ms in __getstate__. I then changed Failure.cleanFailure to just 'return', and I saw a real-world improvement of ~480ms down to about 240ms. I then restored cleanFailure, but changed Failure.__init__ to always set 'tb=None' before it does anything. And in that case the time went down to 180ms. (And when I dug into it, 150ms of that is time spent waiting for an XMLRPC response.)
Just for the record, changing cleanFailure to not do its operation is likely to lead to big memory leaks. The second thing you did, setting self.tb to None, makes more sense to speed things up.
Right. I agree that it is a better solution. I was wondering about having a whitelist of Exception classes that aren't considered interesting enough to keep a traceback.
I'm wondering if there is a tasteful way to disable Traceback processing in a production machine. I realize you never know when you are going to need the state in order to figure out what went wrong. But it is costing 2-5x runtime speed. (The other answer is to write all of our code to avoid Deferred.addErrback...)
I hate to mention it (since it's unlikely I'd be able to work on it), but I wonder how much a C extension would improve this without going the full step of throwing out the debug data.
A large portion of the time spent is in "safe_repr", which presumably is turning all 30k objects into 'str' representations. I would imagine that a lot of this is redundant across state. Meaning that the objects up at the top frame probably don't change very often. I think part of the issue is that when you have an unhandled exception, that's when you really want the traceback, but you only know that long after the traceback is gone. John =:-> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2QpSwACgkQJdeBCYSNAANHhwCgj3J5a6/FgV1n/O2Zwm8U9pm7 eTgAoKWemYqgHnvMwnCrQA+uFZK/+Zeo =x5Xm -----END PGP SIGNATURE-----