urllib2 - closing sockets

[John J. Lee]

pythonguy at Hotpop.com (Anand Pillai) writes:

> I recently noted that urllib2.urlopen(...) for http:// urls
> does not make an explicit call to close the underlying 
> HTTPConnection socket once the data from the socket is read.
> 
> This might not be required since the garbage collector will
> close & collect open sockets that are not closed, but it might
> cause the system to run out of socket memory if there are 
> multiple threads, each opening a socket and the gc not running
> in between.
> 
> This specifically happens in my HarvestMan program which uses
> multiple threads to achieve fast offline web downloads.

Does this cause trouble in your app?

I tried using urllib2 with threads a while back, and failed miserably.
Do you reckon the problem you've found could cause deadlock?  Seems
unlikely, but I was at the point of having to read the threading
module's code to get any further with my deadlock bug, so I'm
clutching at straws...


> A patch to fix this in urllib2.py would be nice.

if-wishes-were-horses-then-beggars-would-ride-<wink>-ly y'rs


John