Re: [Python-ideas] __enter__ and __exit__ on generators

On 6 July 2016 at 01:17, Guido van Rossum wrote:

Hm, it's probably just an oversight. Context managers and closable generators were developed around the same time (PEP 342 and 343!) for Python 2.5; it probably took a while before either of them became part of the popular culture, so to speak.

No, it's deliberate - if generators natively implemented the context management protocol, accidentally leaving out "@contextmanager" would silently call the underlying generator's close() method without ever executing the generator body instead of giving you an immediate exception when you tried to use it in a with statement. When you get the exception, if you actually *do* want to just close the generator, you can wrap it in contextlib.closing, while if you meant to write a custom context manager, you can add the missing decorator. I thought we'd actually added an entry for the design FAQ about that, but it turns out it got stuck on the issue tracker: https://bugs.python.org/issue13814 It does occur to me that it might be worth customising the error reported, though - it's currently genuinely non-obvious why they don't implement the protocol, but it might be clearer if the error was "TypeError: generator requires 'contextlib.contextmanager' or 'contextlib.closing' to support context management" Regards, Nick.

On Tue, Jul 5, 2016 at 7:56 AM, Nathaniel Smith wrote:

...

Hi all,

This might be too trivial for python-ideas even, but... I recently stumbled slightly over the way that generators define a close method (the one that injects a GeneratorExit), but don't directly implement the context manager protocol. So if you want a generator's internal resources to be cleaned up promptly, you can't write

with genfunc() as it: for obj in it: ...

but instead have to write

with contextlib.closing(genfunc()) as it: for obj in it: ...

Is there any particular reason for this, or is it just an oversight?

(Why is it important to close generators? Consider a generator that holds an open file descriptor:

def genfunc(): with open(...) as f: for line in f: yield ...

Python these days very much encourages you to use a "with" statement for the file open, with ResourceWarnings and all that. But in this case, the file won't be closed until someone calls generator.close(), so the with statement above is only really effective if genfunc's caller also uses a "with" statement. Unfortunately, having the "with" statement inside genfunc *is* enough to unconditionally hide the ResourceWarning, even if it's otherwise ineffective, because the warning is triggered when file.__del__ calls file.__close__, and here we have generator.__del__ calling file.__close__.)

-n

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia