On Tue, Oct 6, 2009 at 10:40 PM, Steve Steiner (listsin) <listsin@integrateddevcorp.com> wrote:
Should I limit the number of "in-flight" pages?
Currently, I'm scanning sites that have upwards of 5000 pages and it
seems that, when I get too many deferred's in flight, the app
*appears* to crash.
I'm not sure whether it's actually going out to lunch or just appears
that way and, before I go instrumenting the app to death, can anyone
tell me whether there is some sort of practical limit to how many "in-
flight" deferreds might start to cause issues, just due to the sheer
number?
If your app is doing something strange that you don't understand, you should instrument it until you understand it. Regardless of any practical advice you may receive as a temporary stopgap, there's always a chance that something else is going wrong, and by reducing the number of concurrent requests you're just decreasing its likelihood rather than properly fixing it.
It's highly unlikely that it's actually the number of Deferreds. A Deferred is just a Python object, so if you've got the RAM to store them and their associated callbacks, you should be fine. It's more likely that it has something to do with long callback chains, or hitting some kind of file-descriptor limit (what version of Twisted are you using?) or perhaps that 5000 pages is just a lot of pages to request and you might need to wait a while.
Good luck,
-Glyph