Threads in python?

[Andrae Muys]

Peter Hansen <peter at engcorp.com> wrote in message news:<3C7AD7F1.AFCADA42 at engcorp.com>...
> "Fernando Rodríguez" wrote:
> > 
> > I'm going to start coding a little utility that needs to retrieve webpages and
> > process them.
> > 
> > However, I worried that the whole app might stall if one server takes too long
> > to respond or is offline.
> > 
> > Does Python support threads? "Learning Python" doesn't mention it...
> 
> See http://www.python.org/doc/current/lib/module-threading.html .
> 
> (Yes, Python supports threads.  There may, however, be simpler and/or
> more robust approaches to what you are doing, although I personally
> would just jump on threads since I grok them better than, for example,
> asyncore.)
> 

And I personally wouldn't bother with asyncore either.  As you are
wanting to download the webpages, not just confirm their availability
(as in linkchecker), I would split the application in two, one that
does the downloading, and one that does the processing.  Under an OS
with a lightweight fork(), I would be inclined to use multi-processing
for the first stage.  For OS's with heavier weight fork() (ie. MS),
the front end will probably require threading.  OTOH as none of the
fetch threads would need to interact in anyway this is effectively
multi-processing without the safety of seperate processes.

Similar considerations apply to the processing stage, but precisely
where the balance falls depends on what processing you're doing.

Good Luck,

Andrae Muys