Re: web client & "TimeoutError: User timeout caused connection failure"
--- Lenny G Arbage <alengarbage@yahoo.com> wrote:
(I can produce a short example that reproduces the behavior if needed)
'ere it is: import sys from twisted.python import failure from twisted.internet import reactor, defer from twisted.web.client import getPage CONCURRENT=800 def itersuccess(res, message): print "itersuccess: %s" % message return res def itererror(failure, message): print "itererror message: %s" % message failure.printTraceback() return failure def testGetPage(url, num=CONCURRENT): print "testGetPage started..." dlist = [] for i in range(num): deferred = getPage(url) deferred.addCallback(itersuccess, "succeeded at testGetPage %d" % i) deferred.addErrback(itererror, "failed at testGetPage %d" % i) dlist.append(deferred) return defer.DeferredList(dlist, consumeErrors=True) def runTests(url): d1 = testGetPage(url) d1.addBoth(cleanup) reactor.run() def cleanup(res): reactor.stop() if __name__ == '__main__': runTests(sys.argv[1]) I can run this against a local version of apache, and get a bunch of successes until somewhere in the 600s: itersuccess: succeeded at testGetPage 679 itersuccess: succeeded at testGetPage 657 itersuccess: succeeded at testGetPage 683 itererror message: failed at testGetPage 638 Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure. itererror message: failed at testGetPage 639 Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure. Your results, of course, will vary depending on hardware and software. With other servers I only need to get in 50-60s before I start getting the TimeoutError messages. Is the answer to just catch these messages and keep retrying (after a short pause)? Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
This may be known to some of you, but google sure doesn't want to divulge it upon searching for the "TimeoutError" message described in the title, so I thought I'd document what I've found so far. Failures begin to occur exactly 30 seconds after all the initial deferreds were started, which explains why you need lots of requests if the responses are fairly quick and less if the responses take a bit of time to come back. If I add a line to HTTPClientFactory.clientConnectionFailed(): print "connect failed: %s" % _.timeout The connector object confesses that its timeout was indeed set at 30 seconds. In my case, I'd like to be able to set the connector's timeout a bit higher, but I don't see how to go about it. This isn't the 'timeout=' parameter passed into HTTPClientFactory, which is set to 0. I think it comes from HTTPPageGetter or protocol.http with the TimeoutMixin, but it's all a bit unclear to me -- am I on the right track? Any pointers? Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Sorry in advance for pestering. I've figured it out. The 30 second timeout is the default value for reactor.connectTCP(). I just needed to pass in a new timeout=X value in that call (which means writing a custom getPage() and downloadPage() or using HTTPClientFactory directly, but I'm already doing that for other reasons anyway). Thanks, Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
participants (1)
-
Lenny G Arbage