Asynchronous urllib (urllib+asyncore)?
jgardner at jonathangardner.net
Tue Feb 26 21:49:11 CET 2008
So, I ran into a problem that I would like to write as little code as
possible to solve.
The problem is that I would like to send out a bunch of HTTP requests
simultaneously, using asynchronous techniques, and then do stuff with
the results in parallel. Think of something like Google's map-reduce.
It sounds like this would be something someone else has done before. I
searched google, but I don't think there is any discussion anywhere of
anyone doing something like this.
I am sure Twisted folks can code this up in 2 lines or less, or it
probably already exists there by default. But I need this to exist in
a non-asynchronous environment. I can't let Twisted take over
So, has someone done something like this? Is it something where I can
bring the urllib and asyncore together? Or is it much more involved?
More information about the Python-list