[Python-ideas] Deterministic iterator cleanup

[Stephen J. Turnbull]

Chris Barker wrote:
 > Nick Coghlan wrote:

 >> Chris, would you be open to trying a thought experiment with some of
 >> your students looking at ways to introduce function-scoped
 >> deterministic resource management *before* introducing with
 >> statements?

I'm with Chris, I think: this seems inappropriate to me.  A student
has to be rather sophisticated to understand resource management at
all in Python.  Eg, generators and closures can hang on to resources
between calls, yet there's no syntactic marker at the call site.

 >> The idea here is to change the requirement for new developers from
 >> "telling the interpreter what to *do*" (which is the situation we have
 >> for context managers) to "telling the interpreter what we *want*"
 >> (which is for it to link a managed resource with the lifecycle of the
 >> currently running function call, regardless of interpreter
 >> implementation details)

I think this attempt at a distinction is spurious.  On the syntactic
side,

    with open("file") as f:
        results = read_and_process_lines(f)

the with statement effectively links management of the file resource
to the lifecycle of read_and_process_lines.  (Yes, I know what you
mean by "link" -- will "new developers"?)  On the semantic side,
constructs like closures and generators (which they may be cargo-
culting!) mean that it's harder to link resource management to
(syntactic) function calls than a new developer might think.  (Isn't
that Nathaniel's motivation for the OP?)  And then there's the loop
that may not fully consume an iterator problem: that must be
explicitly decided -- the question for language designers is which of
"close generators on loop exit" or "leave generators open on loop
exit" should be marked with explicit syntax -- and what if you've got
two generators involved, and want different decisions for both?

Chris:

 > I can see that, but I'm not sure newbies will -- it either case,
 > you have to think about what you want -- which is the complexity
 > I'm trying to avoid at this stage.

Indeed.

 > Until much later, when I get into weak references, I can pretty
 > much tell people that python will take care of itself with regards
 > to resource management.

I hope you phrase that very carefully.  Python takes care of itself,
but does not take care of the use case.  That's the programmer's
responsibility.  In a very large number of use cases, including the
novice developer's role in a large project, that is a distinction that
makes no difference.  But the "close generators on loop exit" (or
maybe not!) use case makes it clear that in general the developer must
explicitly manage resources.

 > That's what context mangers are for, in fact. YOU can use:
 > 
 > with open(...) as infile:
 >     .....
 > 
 > Without needing to know what actually has to be "cleaned up" about
 > a file.  In the case of files, it's a close() call, simple enough
 > (in the absence of Exceptions...), but with a database connection
 > or something, it could be a lot more complex, and it's nice to know
 > that it will simply be taken care of for you by the context
 > manager.

But somebody has to write that context manager.  I suppose in the
organizational context imagined here, it was written for the project
by the resource management wonk in the group, and the new developer
just cargo-cults it at first.

 > > The big refactoring benefit that this feature would offer over
 > > with statements is that it doesn't require a structural change to
 > > the code - it's just wrapping an existing expression in a new
 > > function call that says "clean this up promptly when the function
 > > terminates, even if it's still part of a reference cycle, or
 > > we're not using a reference counting GC".
 > 
 > hmm -- that would be simpler in one sense, but wouldn't it require
 > a new function to be defined for everything you might want to do
 > this with?  rather than the same "with" syntax for everything?

Even if it can be done with a single "ensure_cleanup" function, Python
isn't Haskell.  I think context management deserves syntax to mark it.

After all, from the "open and read one file" scripting standpoint,
there's really not a difference between

    f = open("file")
    process(f)

and

    with open("file") as f:
        process(f)

(see "taking care of Python ~= taking care of use case" above).  But
the with statement and indentation clearly mark the call to process as
receiving special treatment.  As Chris says, the developer doesn't
need to know anything but that the object returned by the with
expression participates "appropriately" in the context manager
protocol (which she may think of as the "with protocol"!, ie, *magic*)
and gets the "special treatment" it needs.

So (for me) this is full circle: "with" context management is what we
need, but it interacts poorly with stateful "function" calls -- and
that's what Nathaniel proposes to deal with.