[Chicago] how to use multithread to download?

[Jesse London]

I would only add to that that, if you're requesting multiple resources from
the same host, you might be able to speed up the networking portion using
the urllib3 library, which supports connection pooling —
http://code.google.com/p/urllib3/ (Threading would be another story.)

On Fri, Jun 17, 2011 at 11:30 AM, Brian Ray <brianhray at gmail.com> wrote:

> 2011/6/17 守株待兔 <1248283536 at qq.com>:
>
> > it's slow , would  you  mind to revise it with multithread??
> >
>
> Generally, I am not sure making anything multithreaded in cPython will
> make it faster, per see. However, in this case I can see where if some
> web requests are slow you could go on with other ones so perhaps.
>
> As always, pin point why slow. Is it network communication, I/O, cpu
> time... I guess if is slow due to I/O the cPython might make a
> difference; however, usually it will not so much.  You can add more
> logging to get some idea.
>
> If you do want to make multi threaded it is pretty simple. Create a
> class that inherits from threading.Thread. Send in the constructor the
> url you want it to grab. overwrite the run() method to do the heavy
> lifting. Instantiate an instance of your class, and start(). from the
> caller you want to check on completion before you exit.
>
> There are other choices of approaches to threading like making
> sub-process or using async tasks. It would be interesting to take a
> couple and test with your simple example. Then present at chipy your
> results ;)
>
> You may want to think about async tasks with a framework like twisted
> or something. In fact, this somewhat amusing non scientific experiment
> uses twisted to run different implementations.
> http://morepypy.blogspot.com/2010/03/hello.html . also, inside these
> tests they show some differences based on python implementations.
>
> Again, before you start it really would help if you take a closer look
> at the problem your trying to solve. First take what you have and try
> to figure out if it always takes the same amount of time overall / per
> piece ..
>
> Regards, Brian
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/chicago/attachments/20110617/68480a3f/attachment.html>