[issue12955] urllib2.build_opener().open() is not friendly to "with ... as:"
New submission from Valery Khamenya <khamenya@gmail.com>: The following intuitive construction with urllib2.build_opener().open() as: ... leads to AttributeError: addinfourl instance has no attribute '__exit__' http://docs.python.org/library/urllib2.html says almost nothing about concept of closing. Could it be possible to add a canonical "with ... as:" example for urllib2.build_opener() ? Thanks, Valery ---------- assignee: docs@python components: Documentation, IO, Library (Lib) messages: 143836 nosy: Valery.Khamenya, docs@python priority: normal severity: normal status: open title: urllib2.build_opener().open() is not friendly to "with ... as:" type: resource usage versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Changes by Ezio Melotti <ezio.melotti@gmail.com>: ---------- nosy: +ezio.melotti, orsenthil _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Terry J. Reedy <tjreedy@udel.edu> added the comment: As I understand this, you are asking that 2.7 urllib2.build_opener().open(), which in 3.x is urllib.request.build_opener().open(), be upgraded to return an object that works as a context manager. Unless the docs say that this should already be the case, this is a feature request for 3.3. I am unable to test whether this feature is already present (in 3.2.2). Your example line "with urllib2.build_opener().open() as:" has an obvious syntax error. When I correct that and adjust for 3.x import urllib.request as ur with ur.build_opener().open() as f: pass # TypeError: open() takes at least 2 arguments (1 given) The doc for build_opener says it returns an OpenerDirector instance. help(ur.OpenerDirector.open) just says it needs a 'fullurl'. But when I add 'http:www.python.org' as an argument, I get urllib.error.URLError: <urlopen error no host given> I do not know what else is needed. Please copy and paste the ACTUAL (minimal) code you ran to get the AttributeError. ---------- components: -Documentation, IO nosy: +terry.reedy stage: -> test needed type: resource usage -> feature request versions: -Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Senthil Kumaran <senthil@uthcode.com> added the comment: Just as a quick guideline. If you are talking about context manager support for urlopen, it is available and present in 3.x but not on 2.7. And I fear, it is late to make it available on 2.7, because it is a feature. In any case, as Terry requested, a simple snippet would help us understand the problem and original poster's expectation. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Valery Khamenya <khamenya@gmail.com> added the comment: Terry, Senthil, thanks, for replying to this ticket. OK, to the question: 1. @Terry, here is the full example as for CPython 2.7 I am talking about and the output: ################# from urllib2 import Request, build_opener request = Request('http://example.com') with build_opener().open(request) as f: txt = f.read() print '%d characters fetched' % len(txt) #################### Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: addinfourl instance has no attribute '__exit__' 2. @Senthil, regarding the statement being a feature. I assume, that to open a connection, to read from a connection and to close it -- are the fundamental functions for what urllib2 was in particular created for. I was looking in docs for some hints of "canonical" way of doing this using urllib2.opener. After I have failed to find any guidance in docs, I've created this ticket. That is, I assume that no new feature is needed, but just a documented 5-lines example about a typical way of doing the above operations, especially *closing* the connection. regards, Valery ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Terry J. Reedy <tjreedy@udel.edu> added the comment: On 3.2.2, your example (adapted) produces
2945 characters fetched
So, as Senthil said, the requested feature already exists. But it cannot be added to the 2.7 series; the Python 2.7 *language* is feature frozen. 2.7.z bug fixes serve to make the implementation better match the intended language. Prior to the addition of with..., the standard way to close things was with explicit .close() or implicit closing when the function returned or the program exited. In the 3.2 docs, "20.5.21. Examples", there are several examples with f = <some opener> <do something with f> (closing ignored) Where possible, these could and should be changed in 3.2+ docs to with <some opener> as f: <do something with f> to promote the new idiom. This had been done in other chapters. Perhaps there is also a need for 'X supports the context manager protocol' to be added here or there. But showing that in the examples would make the point. I have changed the title of this issue to match. ---------- components: +Documentation -Library (Lib) stage: test needed -> needs patch title: urllib2.build_opener().open() is not friendly to "with ... as:" -> urllib.request example should use "with ... as:" versions: +Python 3.2 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Valery Khamenya <khamenya@gmail.com> added the comment: Guys, in my item 2 the simplistic goal was stated clearly: open, read and close. Do you confirm that this basic sequence is not supported by urllib2 under 2.7 ? (I just requested for a tiny documentation update entry) regards, Valery ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Senthil Kumaran <senthil@uthcode.com> added the comment: Valery, yes. I shall update 2.7 documentation with this known limitation and 3.x documentation with the example usage scenarios. Thanks! ---------- assignee: docs@python -> orsenthil _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Éric Araujo <merwok@netwok.org> added the comment: [Terry]
But when I add 'http:www.python.org' as an argument, I get urllib.error.URLError: <urlopen error no host given>
Your URI lacks a host (netloc, in RFC parlance) component:
urllib.parse.urlparse('http:python.org') ParseResult(scheme='http', netloc='', path='python.org', params='', query='', fragment='')
---------- nosy: +eric.araujo _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Terry J. Reedy <tjreedy@udel.edu> added the comment: Senthil: http://docs.python.org/dev/library/contextlib.html#contextlib.closing currently has this example: from urllib.request import urlopen with closing(urlopen('http://www.python.org')) as page: which is misleading in that the object returned from urlopen <class 'http.client.HTTPResponse'> has __enter__ and __exit__ methods and therefore is a c.m. in itself and does not need to be wrapped in closing(). I did not really understand from your comment whether there is any need to use closing() with anything returned in urllib. At the moment, shelves are not C.M.s, and would make better examples, but #13896 suggests 'fixing' that also. ---------- nosy: +ncoghlan _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Éric Araujo <merwok@netwok.org> added the comment: Either we find a commonly used stdlib class that is not a context manager but has a close method and is not going to become a context manager (I can’t see why such a thing would be), or we can add something like: “closing is useful with code that you can’t change to add context management, for example urllib.urlopen before Python 3.3”. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Changes by Alexander Boyd <alex@opengroove.org>: ---------- nosy: +javawizard _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Changes by Terry J. Reedy <tjreedy@udel.edu>: ---------- versions: +Python 3.4, Python 3.5 -Python 3.2, Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Martin Panter added the comment: Issue 22755 is about the example arms race for contextlib.closing(). ---------- nosy: +vadmium _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Martin Panter added the comment: Here is a patch to change the urlopen() examples to use context managers where appropriate. There were also a few examples of handling HTTPError which I didn’t touch, because the whole file object versus exception object thing is probably a separate can of worms. ---------- keywords: +patch Added file: http://bugs.python.org/file38049/urlopen-with.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Changes by Berker Peksag <berker.peksag@gmail.com>: ---------- nosy: +berker.peksag stage: needs patch -> patch review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Roundup Robot added the comment: New changeset 6d5336a193cc by Berker Peksag in branch '3.4': Issue #12955: Change the urlopen() examples to use context managers where appropriate. https://hg.python.org/cpython/rev/6d5336a193cc New changeset 08adaaf08697 by Berker Peksag in branch 'default': Issue #12955: Change the urlopen() examples to use context managers where appropriate. https://hg.python.org/cpython/rev/08adaaf08697 ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
Berker Peksag added the comment: Great patch. Thanks Martin. ---------- resolution: -> fixed stage: patch review -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue12955> _______________________________________
participants (9)
-
Alexander Boyd -
Berker Peksag
-
Ezio Melotti
-
Martin Panter
-
Roundup Robot
-
Senthil Kumaran
-
Terry J. Reedy
-
Valery Khamenya
-
Éric Araujo