[pypy-dev] RFC: draft idea for making for loops automatically close iterators

[William ML Leslie]

A quick definition:  A resource is something, typically represented as
a program object, whose timely disposal is relevant for the correct
operation of the program.  Typically, it can't be closed too late (in
the case of files, the process may end before all writes are flushed),
but it often can't be closed too soon.

This has a few obvious corallaries:

* If the correct functioning of the program relies on prompt disposal,
then something must be responsible for that disposal.  If you place it
in a container and hand that container off to a library without
linearity or borrow checking, you're saying you have no interest in
its disposal.

* The property of "being a resource" is transitive over ownership
(Hopwood's Corallary?): if X requires explicit disposal, and that
disposal is managed by Y, then Y is also a resource (proof of this is
easy via contradiction).  Typically, there is no special magic for Y;
the documentation for Y must either describe how to dispose of it, or
whatever code constructed Y must also maintain control over the
lifetime of X.

* Are files/sockets open for reading a resource?  They typically
aren't, especially on GNU and BSD derived systems.  Some programs will
never run out of them because they will never open enough.  At the
same time, given that "the correct operation of the program" is
specific to the actual program and its usage, the answer isn't
typically clear.

The first point we should notice is that explicit lifetime management
is /required/ - it's a direct consequence of the definition of
resource!

On 22 October 2016 at 01:20, Alex S. <alex0player at gmail.com> wrote:
> That’s a good point, as it means there’s probably no safe & portable way to
> ensure that kind of stuff. «Trying to collect» something doesn’t really fall
> short of an actual collection, I believe (finding referers is hard).
> But I believe iterclose() defined appropriately on derived iterators would
> solve that?..

That now places the onus on every user of iterators to ensure that
their iterators are either consumed or closed; but many of the
iterators we create that are resources are then handed to library
functions which just can't be relied upon to maintain resource
transitivity.

The second point, then, is that requiring everyone to rewrite any code
they have that makes or consumes generators or iterables is probably
not tractible.  Even if everyone decided that would be better than
fixing their use of files, you'd still stumble across library code
that didn't make the effort.

We might have been able to do something like this if we'd had
something like (dynamic-wind) when generators first arrived in python,
but it's probably beyond us now without a good helping of SA and/or
ugly magic.

So it really is easier to rewrite file usage than it is to rewrite
iterator usage, especially if we can only detect and fix a handful of
obvious cases in the runtime.

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.