--- Lenny G Arbage <alengarbage@yahoo.com> wrote:
(I can produce a short example that reproduces the behavior if needed)
'ere it is: import sys from twisted.python import failure from twisted.internet import reactor, defer from twisted.web.client import getPage CONCURRENT=800 def itersuccess(res, message): print "itersuccess: %s" % message return res def itererror(failure, message): print "itererror message: %s" % message failure.printTraceback() return failure def testGetPage(url, num=CONCURRENT): print "testGetPage started..." dlist = [] for i in range(num): deferred = getPage(url) deferred.addCallback(itersuccess, "succeeded at testGetPage %d" % i) deferred.addErrback(itererror, "failed at testGetPage %d" % i) dlist.append(deferred) return defer.DeferredList(dlist, consumeErrors=True) def runTests(url): d1 = testGetPage(url) d1.addBoth(cleanup) reactor.run() def cleanup(res): reactor.stop() if __name__ == '__main__': runTests(sys.argv[1]) I can run this against a local version of apache, and get a bunch of successes until somewhere in the 600s: itersuccess: succeeded at testGetPage 679 itersuccess: succeeded at testGetPage 657 itersuccess: succeeded at testGetPage 683 itererror message: failed at testGetPage 638 Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure. itererror message: failed at testGetPage 639 Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure. Your results, of course, will vary depending on hardware and software. With other servers I only need to get in 50-60s before I start getting the TimeoutError messages. Is the answer to just catch these messages and keep retrying (after a short pause)? Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com