[Python-ideas] Make "yield" inside a with statement a SyntaxError

[Nathaniel Smith]

On Tue, Aug 7, 2018 at 11:14 PM, Ken Hilton <kenlhilton at gmail.com> wrote:
> This mostly springs off of a comment I saw in some thread.
>
> The point of a with statement is that it ensures that some resource will be
> disposed of, yes? For example, this:
>
>     with open(filename) as f:
>         contents = f.read()
>
> is better than this:
>
>     contents = open(filename).read()
>
> because the former definitely closes the file while the latter relies on
> garbage collection?
>
> The point of a yield expression is to suspend execution. This is nice for
> efficient looping because instead of having to hold all results in memory,
> each result can be consumed immediately, yes? Therefore this:
>
>     def five_to_one():
>         for i in range(4):
>             yield 5 - i
>
> is better than this:
>
>     def five_to_one():
>         result = []
>         for i in range(4):
>             result.append(5 - i)
>         return result
>
> because the former suspends execution of "five_to_one" while the latter
> holds all five results in memory?
>
> Now, let's take a look at the following scenario:
>
>     def read_multiple(*filenames):
>         for filename in filenames:
>             with open(filename) as f:
>                 yield f.read()
>
> Can you spot the problem? The "with open(filename)" statement is supposed to
> ensure that the file object is disposed of properly. However, the "yield
> f.read()" statement suspends execution within the with block, so if this
> happened:
>
>     for contents in read_multiple('chunk1', 'chunk2', 'chunk3'):
>         if contents == 'hello':
>             break
>
> and the contents of "chunk2" were "hello" then the loop would exit, and
> "chunk2" would never be closed! Yielding inside a with block, therefore,
> doesn't make sense and can only lead to obscure bugs.

This is a real problem. (Well, technically the 'with' block's __exit__
function *will* eventually close the file, when the generator is
garbage-collected – see PEP 342 for details – but this is not exactly
satisfying, because the whole purpose of the 'with' block is to close
the file *without* relying on the garbage collector.)

Unfortunately, your proposal for solving it is a non-starter -- there
are lots of cases where 'yield' inside a 'with' block is not only
used, but is clearly the right thing to do. A notable one is when
defining a next contextmanager in terms of a pre-existing
contextmanager:

@contextmanager
def tempfile():
    # This is an insecure way of making a temp file but good enough
for an example
    tempname = pick_random_filename()
    with open(tempname, "w") as f:
        yield f

Here are some links for previous discussions around these kinds of
issues, none of which have really gone anywhere but might help you get
a sense of the landscape of options:

https://www.python.org/dev/peps/pep-0533/
https://www.python.org/dev/peps/pep-0521/
https://www.python.org/dev/peps/pep-0568/
https://github.com/python-trio/trio/issues/264

One small step that might be doable would be to start issuing
ResourceWarning whenever a generator that was suspended inside a
'with' or 'try' block is GC'ed.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org