pehr anderson pehr at
Fri Sep 1 07:17:28 CEST 2000

For this sort of spider app you might want to try timeoutsocket
which is an overlay on conventional python sockets that 
causes them to timeout after a specified number of seconds.

Being tied to an IP stack, your computer has a maximum of 65535 
active sockets. With the normal 2 hour timeout for IP sockets, 
your spider probably doesn't take long to fill up this entire space!
Check this by running netstat on linux. I don't know if windows
has a diagnostic tool for this...


David Arnold wrote:
> -->"Aahz" == Aahz Maruch <aahz at> writes:
> In article <8odh9e$qqf$1 at>, <karlsass at>
> wrote:
>   >>  I have a set of spiders running on my system, The spiders
>   >> communicate with a urlserver & a processing server via internal
>   >> sockets. The spiders are the only ones using Inet sockets. Yet
>   >> after a while of say 1 or 2 spiders, a urlserver and process
>   >> server running , say for 3 hours, then all of a sudden the system
>   >> disallows any connections to the outside, eg inet sockets as well
>   >> as Netscape cannot connect to other web located resources.
>   Aahz> Yeah, sounds like you're not properly closing down the sockets
>   Aahz> after you use them.
> i had a bug yesterday, on WinNT (it was on someone else's system so
> i'm not sure about this), where using socket.close() does not close
> the underlying socket, byt socket.shutdown() does.
> might be worth a go?
> d

More information about the Python-list mailing list