[Chicago] how to use multithread to download?

[Brian Ray]

2011/6/17 守株待兔 <1248283536 at qq.com>:

> it's slow , would  you  mind to revise it with multithread??
>

Generally, I am not sure making anything multithreaded in cPython will
make it faster, per see. However, in this case I can see where if some
web requests are slow you could go on with other ones so perhaps.

As always, pin point why slow. Is it network communication, I/O, cpu
time... I guess if is slow due to I/O the cPython might make a
difference; however, usually it will not so much.  You can add more
logging to get some idea.

If you do want to make multi threaded it is pretty simple. Create a
class that inherits from threading.Thread. Send in the constructor the
url you want it to grab. overwrite the run() method to do the heavy
lifting. Instantiate an instance of your class, and start(). from the
caller you want to check on completion before you exit.

There are other choices of approaches to threading like making
sub-process or using async tasks. It would be interesting to take a
couple and test with your simple example. Then present at chipy your
results ;)

You may want to think about async tasks with a framework like twisted
or something. In fact, this somewhat amusing non scientific experiment
uses twisted to run different implementations.
http://morepypy.blogspot.com/2010/03/hello.html . also, inside these
tests they show some differences based on python implementations.

Again, before you start it really would help if you take a closer look
at the problem your trying to solve. First take what you have and try
to figure out if it always takes the same amount of time overall / per
piece ..

Regards, Brian