urllib2 - closing sockets
John J. Lee
jjl at pobox.com
Sat Jun 5 18:57:02 CEST 2004
pythonguy at Hotpop.com (Anand Pillai) writes:
> I recently noted that urllib2.urlopen(...) for http:// urls
> does not make an explicit call to close the underlying
> HTTPConnection socket once the data from the socket is read.
> This might not be required since the garbage collector will
> close & collect open sockets that are not closed, but it might
> cause the system to run out of socket memory if there are
> multiple threads, each opening a socket and the gc not running
> in between.
> This specifically happens in my HarvestMan program which uses
> multiple threads to achieve fast offline web downloads.
Does this cause trouble in your app?
I tried using urllib2 with threads a while back, and failed miserably.
Do you reckon the problem you've found could cause deadlock? Seems
unlikely, but I was at the point of having to read the threading
module's code to get any further with my deadlock bug, so I'm
clutching at straws...
> A patch to fix this in urllib2.py would be nice.
More information about the Python-list