asynchronous downloading

[Paul Rubin]

Plumo <richardbp at gmail.com> writes:
> What do you recommend? 

Threads.

> And why is there poor support for asynchronous execution?

The freenode #python crowd seems to hate threads and prefer twisted,
which seems to have the features you want and probably handles very
large #'s of connections better than POSIX threads do.  But I find the
whole event-driven model to be an annoying abstraction inversion and
threads to be simpler, so I've stayed with threads.  I keep hearing
boogieman stories about the evil hazards of race conditions etc. but
none of that stuff has ever happened to me (yet) as far as I can tell.
The main thing is to avoid sharing mutable data between threads to the
extent that you can.  Keep the threads isolated from each other except
for communication through Queues and not too much can go wrong.  The
last program I wrote had around 20 threads and one or two condition
variables and I don't think any significant bugs resulted from that.

FWIW, the Erlang language is built around the above concept, it uses
super-lightweight userland threads so it can handle millions of them
concurrently, and it's used successfully for ultra-high-reliability
phone switches and similar applications that are not allowed to fail, so
it must be doing something right.

There are a few schemes like Camaelia (sp?) implementing concurrency
with Python generators or coroutines, but I think they're not widely
used, and Python coroutines are kind of crippled because they don't
carry any stack below their entry point.