simple httplib and urllib timeout question

[Michal Wallace (sabren)]

On Tue, 11 Apr 2000 darrell at rubina.funkomite.com wrote:

> Just a quick question, im writing a small webbot for a python
> learning experience and it comes across some sites that will make it
> hang for a while, if the site is down or extremely slow etc, is
> there a way to set a faster timeout for httplib . Or is there a
> better way to handle it ?

Yes. The better way is to use asyncore whenever httplib/urllib uses a
normal socket. Unfortunately, this requires breaking urllib into
smaller pieces, and nobody seems to have done it yet. I'm actually
working on the exact same problem (and posting regular updates
at http://www.linkwatcher.com/ ) 

I sent an email to Jeremy Hylton, who's working on urllib2. He's
swamped, but willing to accept a patch that would add this feature
to urllib2.. I think I'll wind up taking a crack at it tonight.

There's also a partial solution using a different method if you look
at...  http://www.linkwatcher.com/src/blogbot/timeout.py

there's an example script called blogbot.py there, too.. it uses
timeout.py and another hack with signal.alarm(), but neither really
works all that well..

hope that helps.

meanwhile... HEY OLEG! -> have you done anything w/ this?

Cheers,

- Michal
-------------------------------------------------------------------------
http://www.manifestation.com/         http://www.linkwatcher.com/metalog/
-------------------------------------------------------------------------