how to get 20000 html pages content quickly from one server?

Larry Bates larry.bates at websafe.com
Wed Mar 15 17:33:16 CET 2006


JuHui wrote:
> Hi
>   I want to get 200000 html pages content from one server, you know
> urllib.urlopen need construct network connection, it will be very
> slowly, how to speed up this function?
>  I try to using multi-thread, it speed up, but I want to quickly more,
> any idea about it?
> Thanks!
> 
You are most likely constrained by the speed of your Internet
connection.  If that is the case, there is nothing you can do.

-Larry Bates



More information about the Python-list mailing list