[Python-ideas] __enter__ and __exit__ on generators

[Nick Coghlan]

On 6 July 2016 at 01:17, Guido van Rossum <guido at python.org> wrote:
> Hm, it's probably just an oversight. Context managers and closable
> generators were developed around the same time (PEP 342 and 343!) for
> Python 2.5; it probably took a while before either of them became part
> of the popular culture, so to speak.

No, it's deliberate - if generators natively implemented the context
management protocol, accidentally leaving out "@contextmanager" would
silently call the underlying generator's close() method without ever
executing the generator body instead of giving you an immediate
exception when you tried to use it in a with statement.

When you get the exception, if you actually *do* want to just close
the generator, you can wrap it in contextlib.closing, while if you
meant to write a custom context manager, you can add the missing
decorator.

I thought we'd actually added an entry for the design FAQ about that,
but it turns out it got stuck on the issue tracker:
https://bugs.python.org/issue13814

It does occur to me that it might be worth customising the error
reported, though - it's currently genuinely non-obvious why they don't
implement the protocol, but it might be clearer if the error was
"TypeError: generator requires 'contextlib.contextmanager' or
'contextlib.closing' to support context management"

Regards,
Nick.


>
> On Tue, Jul 5, 2016 at 7:56 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> Hi all,
>>
>> This might be too trivial for python-ideas even, but... I recently stumbled
>> slightly over the way that generators define a close method (the one that
>> injects a GeneratorExit), but don't directly implement the context manager
>> protocol. So if you want a generator's internal resources to be cleaned up
>> promptly, you can't write
>>
>>   with genfunc() as it:
>>       for obj in it:
>>           ...
>>
>> but instead have to write
>>
>>   with contextlib.closing(genfunc()) as it:
>>       for obj in it:
>>           ...
>>
>> Is there any particular reason for this, or is it just an oversight?
>>
>> (Why is it important to close generators? Consider a generator that holds an
>> open file descriptor:
>>
>>   def genfunc():
>>       with open(...) as f:
>>           for line in f:
>>               yield ...
>>
>> Python these days very much encourages you to use a "with" statement for the
>> file open, with ResourceWarnings and all that. But in this case, the file
>> won't be closed until someone calls generator.close(), so the with statement
>> above is only really effective if genfunc's caller also uses a "with"
>> statement. Unfortunately, having the "with" statement inside genfunc *is*
>> enough to unconditionally hide the ResourceWarning, even if it's otherwise
>> ineffective, because the warning is triggered when file.__del__ calls
>> file.__close__, and here we have generator.__del__ calling file.__close__.)
>>
>> -n
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia