[issue19524] ResourceWarning when urlopen() forgets the HTTPConnection object

[Martin Panter]

New submission from Martin Panter:

The AbstractHTTPHandler.do_open() method creates a HTTPConnection object but does not save it anywhere. This means a ResourceWarning is eventually triggered, at least when the HTTP server leaves the the connection open.

Demonstration code:

from urllib.request import urlopen
with urlopen("http://localhost") as response: response.read()

When I used the above code, the warning did not trigger until I forced a garbage collection:

import gc; gc.collect()

Output:

__main__:1: ResourceWarning: unclosed <socket.socket object, fd=3, family=2, type=1, proto=6>

Alternatively, you can add a line to the bottom of the do_open() method:

        r.msg = r.reason
        del h; import gc; gc.collect()  # Add this to force warning
        return r

When the warning happens without forced garbage collection, it tends to happen here:

/usr/lib/python3.3/socket.py:370: ResourceWarning: unclosed <socket.socket object, fd=3, family=2, type=1, proto=6>
  self._sock = None

I tested by using the “socat” CLI program and supplying a chunked HTTP response, without immediately closing the connection at the server end. Using the Content-length header also seems to trigger the issue.

$ sudo socat -d TCP-LISTEN:www,reuseaddr,crnl READLINE
GET / HTTP/1.1
Accept-Encoding: identity
Host: localhost
Connection: close
User-Agent: Python-urllib/3.3

<HTTP response start>
HTTP/1.1 200 OK
Transfer-encoding: chunked

0

<HTTP response end>

If the server leaves the connection open, it only seems to get closed when Python garbage-collects the socket and closes it. Perhaps the connection should be explicitly closed when the urlopen() response object is closed. But I guess that would require wrapping the HTTPResponse object to add to the close behaviour.

----------
components: Library (Lib)
messages: 202404
nosy: vadmium
priority: normal
severity: normal
status: open
title: ResourceWarning when urlopen() forgets the HTTPConnection object
versions: Python 3.2, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue19524>
_______________________________________