Async Client with 1K connections?

[William Chang]

aahz at pythoncraft.com (Aahz) wrote:
> Note that while you-know-who makes extensive use of Python, I don't
> think they're using it for spidering/searching.  I do have some
> background writing a spider in Python, using Verity's engine for
> indexing/retrieval, but we were using threading rather than
> asyncore-style operations.

Interesting, did you try maxing out the number of threads/connections?
On an UltraSparc with hardware thread/lwp support, a thousand threads
can co-exist reliably, at least for computations and disk I/O.  Linux
is another matter entirely.

--William