pehr at pehr.net
Fri Sep 1 07:17:28 CEST 2000
For this sort of spider app you might want to try timeoutsocket
which is an overlay on conventional python sockets that
causes them to timeout after a specified number of seconds.
Being tied to an IP stack, your computer has a maximum of 65535
active sockets. With the normal 2 hour timeout for IP sockets,
your spider probably doesn't take long to fill up this entire space!
Check this by running netstat on linux. I don't know if windows
has a diagnostic tool for this...
David Arnold wrote:
> -->"Aahz" == Aahz Maruch <aahz at netcom.com> writes:
> In article <8odh9e$qqf$1 at nnrp1.deja.com>, <karlsass at my-deja.com>
> >> I have a set of spiders running on my system, The spiders
> >> communicate with a urlserver & a processing server via internal
> >> sockets. The spiders are the only ones using Inet sockets. Yet
> >> after a while of say 1 or 2 spiders, a urlserver and process
> >> server running , say for 3 hours, then all of a sudden the system
> >> disallows any connections to the outside, eg inet sockets as well
> >> as Netscape cannot connect to other web located resources.
> Aahz> Yeah, sounds like you're not properly closing down the sockets
> Aahz> after you use them.
> i had a bug yesterday, on WinNT (it was on someone else's system so
> i'm not sure about this), where using socket.close() does not close
> the underlying socket, byt socket.shutdown() does.
> might be worth a go?
More information about the Python-list