On Oct 6, 2009, at 10:57 PM, Glyph Lefkowitz wrote:
Cool, I didn't know about that, I'll give it a look. Thanks!
If your app is doing something strange that you don't understand, you should instrument it until you understand it.
It's not that I don't understand what's supposed to be happening, or that it's doing something strange, it just seems that sites up to about 2000 pages work fine, then things get dicey.
I was just looking for some guidance on the "max-deferred" that has been found in practical experience more than anything else.
Regardless of any practical advice you may receive as a temporary stopgap, there's always a chance that something else is going wrong, and by reducing the number of concurrent requests you're just decreasing its likelihood rather than properly fixing it.
I understand, and agree. I'm not looking for a stopgap, just maybe a ballpark of "don't set more than 2000 in-flight deferreds at one time" type of guideline.
I understand that every situation is different, I'm working to limit my in-filght requests to a manageable number,
It's highly unlikely that it's actually the number of Deferreds. A Deferred is just a Python object, so if you've got the RAM to store them and their associated callbacks, you should be fine.
Yes, I understand that, thank you for clarifying.
It's more likely that it has something to do with long callback chains, or hitting some kind of file-descriptor limit
The callback chains are short, and I'm not getting a file-descriptor limit exception, or any exception that's getting percolated up.
(what version of Twisted are you using?)
Sorry for not including this earlier...
# python -V
Python 2.6.1
>>> import twisted
>>> twisted.__version__
'8.2.0'
I'm running right out of the release versions for these tests since that's what my users will have installed.
or perhaps that 5000 pages is just a lot of pages to request and you might need to wait a while.
Yes, it is a lot of stuff...
What I'm working on determining is whether limiting the number of "in-flight" URL getters would beneficial.
Thanks,
S
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python