is it safe to stop a thread?

Dustin Boswell boswell at ugcs.caltech.edu
Tue May 22 20:01:51 EDT 2001


I'm writing a client that needs to access various web sites
periodically, but the
problem is that every once in a while a network glitch will happen so
that urlopen() hangs
forever (hitting CTRL-C shows it was on a read() ).  This is similar to
using Netscape and
having to click "stop" and then "reload" on a page that won't fully
load.

I would like to keep using urllib to do my web interface (as opposed to
doing low level
asynchronous sockets) and so my first thought is to use threads.

The "threading" module doesn't allow stopping, so that seems useless.
But the "thread" module DOES.  Am I going to get myself into trouble
starting and
stopping threads that are busy doing network i/o?  If the number of
parallel connections
remains small (like around 10), would you maybe suggest separate
processes (which
CAN be stopped)?

Has anyone out there written a robust web crawler in Python that got
around this problem?

Thanks for any advice.
-Dustin.




More information about the Python-list mailing list