[Python-ideas] With clauses for generator expressions
abarnert at yahoo.com
Thu Nov 15 11:08:27 CET 2012
From: Masklinn <masklinn at masklinn.net>
Sent: Thu, November 15, 2012 1:29:46 AM
> On 2012-11-15, at 04:44 , Andrew Barnert wrote:
> > Here's the suggestion:
> > upperlines = (lines.upper() for line in file with open('foo', 'r') as
> > This would be equivalent to:
> > def foo():
> > with open('foo', 'r') as file:
> > for line in file:
> > yield line.upper()
> > upperlines = foo()
> > The motivation is that there is no way to write this properly using a with
> > statement and a generator expression—in fact, the only way to get this
> > with the generator function above.
> Actually, it's extremely debatable that the generator function is
> correct: if the generator is not fully consumed (terminating iteration
> on the file) I'm pretty sure the file will *not* get closed save by the
> GC doing a pass on all dead objects maybe. This means this function is
> *not safe* as a lazy source to an arbitrary client, as that client may
> very well use itertools.slice or itertools.takewhile and only partially
> consume the generator.
Well, yes, *no possible object* is safe as a lazy source to an arbitrary client
that might not fully consume, close, or destroy it. By definition, the object
must stay alive as long as an arbitrary client might use it, so a client that
never finishes using it means the object must stay alive forever. And,
similarly, in the case of a client that does finish using it, but the only way
to detect that is by GCing the client, the object must stay alive until the GC
collects the client. So, the correct thing for the generator function to do in
that case is… exactly what it does.
Of course in that case, it would arguably be just as correct to just do "ms =
Manager()" or "file = open('foo', 'r')" instead of "with Manager() as ms:" or
"with open('foo', 'r') as file:".
The difference is that, in cases where the client does fully consume, close, or
destroy the iterator deterministically, the with version will still do the
thing, while the leaky version will not. You can test this very easily by
adding an "f.close()" to the end of bar, or changing "f = foo()" to "with
closing(foo()) as f:", and compare the two versions of the generator function.
Put another way, if your point is an argument against with clauses, it's also
argument against with statements, and manual resource cleanup, and in fact
anything but a magical GC.
> This is in fact one of the huge issues with faking dynamic scopes via
> threadlocals and context managers (as e.g. Flask might do, I'm not sure
> what actual strategy it uses), they interact rather weirdly with
> generators (it's also why I think Python should support actually
> dynamically scoped variables, it would also fix the thread-broken
> behavior of e.g. warnings.catch_warnings)
This is an almost-unrelated side issue. A generator used in a single thread
defines a fully deterministic dynamic scope, one that can and often should be
used for cleanup. The fact that sometimes it's not the right scope for some
cleanups, or that you can use them in multithreaded programs in a way that
them indeterministic, isn't an argument that it should be hard to use them for
cleanup when appropriate, is it?
More information about the Python-ideas