[Twisted-Python] New TLS Code *Could* Cause Problems
Hi All,
I just wanted to fire a note about our experiences testing Twisted trunk (after _newtls.py landed) on some servers in our cluster, on PyOpenSSL 0.12.
In our application (a twisted.web REST api endpoint), we make a lot of https calls to a SimpleDB Amazon AWS endpoint (sdb.amazonaws.com). At some point after a deployed test on some of our machines, we started getting our internel DNS requests to the sdb endpoint throttled (resulting in failures in most of our requests). Since every call to SimpleDB was made to the sdb endpoint, each call would be resolved, bombarding amazon's dns servers with requests. After diving in, I learned all Twisted connect* calls go through socket.gethostbyname, which apparently does not do any DNS caching (much to my surprise) unless something like nscd or a local caching nameserver is running (we ended up using BIND).
Now, since we never ran a local caching nameserver before I wondered what changed. Could it have been that using the _newtls code (with PyOpenSSL 0.12) removed some element of DNS caching that OpenSSL was performing behind the scene? Is this possible? If so, then I just wanted to make sure everyone was aware that if you make a lot of DNS calls in a Twisted server running this code, you too could experience problems if you're not running a local caching nameserver.
Thanks, Reza
On 07/19/2011 10:51 AM, Reza Lotun wrote:
would be resolved, bombarding amazon's dns servers with requests. After diving in, I learned all Twisted connect* calls go through socket.gethostbyname, which apparently does not do any DNS caching (much to my surprise) unless something like nscd or a local caching nameserver is running (we ended up using BIND).
socket.gethostbyname is just a wrapper around the libc implementation, and you're correct that it does no caching unless the NSS layer does, or the DNS query you're talking to does.
Now, since we never ran a local caching nameserver before I wondered
How did anything work then if you had no local DNS?
what changed. Could it have been that using the _newtls code (with PyOpenSSL 0.12) removed some element of DNS caching that OpenSSL was performing behind the scene? Is this possible? If so, then I just wanted
A quick 10-line connectSSL script on my box running 10.1 does a DNS lookup per-connection. Are you sure this behaviour has changed?
to make sure everyone was aware that if you make a lot of DNS calls in a Twisted server running this code, you too could experience problems if you're not running a local caching nameserver.
Well, TBH if you're not running a local caching nameserver, I am failing to see how anything works!
On Tue, Jul 19, 2011 at 12:16 PM, Phil Mayers p.mayers@imperial.ac.ukwrote:
Well, TBH if you're not running a local caching nameserver, I am failing to see how anything works!
Hmm, it turns out I may have been a bit too hasty in blaming the newtls code perhaps. We're running tests on resolver calls on the old setup vs. the new one, and the behaviour is the same on on our pre-trunk (standard) config. We may have just hit some threshold where it has suddenly started to matter, I suppose.
Anyway, I suppose the real lesson is the non-caching of gethostbyname calls. Not being a DNS guru, I had no idea calls weren't cached locally. Let that be a lesson, I guess.
Thanks, Reza
On Jul 19, 2011, at 8:36 AM, Reza Lotun wrote:
Anyway, I suppose the real lesson is the non-caching of gethostbyname calls. Not being a DNS guru, I had no idea calls weren't cached locally. Let that be a lesson, I guess.
You were probably confused by the fact that glibc caches /etc/resolv.conf in each process and doesn't always reload it when it should. Depending on what patches your platform has decided to include in its libc, you may or may not have to call res_init() to get an up-to-date resolv.conf. It doesn't ever cache the results of name lookups themselves though, that would lead to all kinds of mayhem.
False alarm or not, thanks for testing trunk versions of Twisted against your application. It's great to get these kinds of issues investigated _before_ a release :).
-glyph
participants (3)
-
Glyph Lefkowitz -
Phil Mayers -
Reza Lotun