[Python-ideas] With clauses for generator expressions

[Andrew Barnert]

From: Masklinn <masklinn at masklinn.net>
Sent: Thu, November 15, 2012 1:29:46 AM


> On 2012-11-15, at 04:44 , Andrew Barnert wrote:
> 
> > Here's the  suggestion:
> > 
> >    upperlines = (lines.upper() for line  in file with open('foo', 'r') as 
>file)
> > 
> > This would be equivalent  to:
> > 
> >    def foo():
> >        with open('foo', 'r') as file:
> >            for line in file:
> >                yield line.upper()
> >    upperlines = foo()
> > 
> > The  motivation is that there is no way to write this properly using a with 
> >  statement and a generator expression—in fact, the only way to get this 
>right 
>is 
>
> > with the generator function above.
> 
> Actually, it's extremely  debatable that the generator function is
> correct: if the generator is not  fully consumed (terminating iteration
> on the file) I'm pretty sure the file  will *not* get closed save by the
> GC doing a pass on all dead objects maybe.  This means this function is
> *not safe* as a lazy source to an arbitrary  client, as that client may
> very well use itertools.slice or  itertools.takewhile and only partially
> consume the generator.

Well, yes, *no possible object* is safe as a lazy source to an arbitrary client 
that might not fully consume, close, or destroy it. By definition, the object 
must stay alive as long as an arbitrary client might use it, so a client that 
never finishes using it means the object must stay alive forever. And, 
similarly, in the case of a client that does finish using it, but the only way 
to detect that is by GCing the client, the object must stay alive until the GC 
collects the client. So, the correct thing for the generator function to do in 
that case is… exactly what it does.

Of course in that case, it would arguably be just as correct to just do "ms = 
Manager()" or "file = open('foo', 'r')" instead of "with Manager() as ms:" or 
"with open('foo', 'r') as file:".

The difference is that, in cases where the client does fully consume, close, or 
destroy the iterator deterministically, the with version will still do the 
right 
thing, while the leaky version will not. You can test this very easily by 
adding an "f.close()" to the end of bar, or changing "f = foo()" to "with 
closing(foo()) as f:", and compare the two versions of the generator function.

Put another way, if your point is an argument against with clauses, it's also 
an 
argument against with statements, and manual resource cleanup, and in fact 
anything but a magical GC.

> This is  in fact one of the huge issues with faking dynamic scopes via
> threadlocals  and context managers (as e.g. Flask might do, I'm not sure
> what actual  strategy it uses), they interact rather weirdly with
> generators (it's also  why I think Python should support actually
> dynamically scoped variables, it  would also fix the thread-broken
> behavior of e.g.  warnings.catch_warnings)


This is an almost-unrelated side issue. A generator used in a single thread 
defines a fully deterministic dynamic scope, one that can and often should be 
used for cleanup. The fact that sometimes it's not the right scope for some 
cleanups, or that you can use them in multithreaded programs in a way that 
makes 
them indeterministic, isn't an argument that it should be hard to use them for 
cleanup when appropriate, is it?