[Python-ideas] Deterministic iterator cleanup

[Chris Barker]

On Fri, Oct 21, 2016 at 12:12 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> Portability across Pythons... if all Pythons performed exactly the same,
> why would we need multiple implementations? The way I see it,
> non-deterministic cleanup is the cost you pay for a non-reference
> counting implementation, for those who care about the garbage collection
> implementation. (And yes, ref counting is garbage collection.)
>

Hmm -- and yet "with" was added, and I an't imageine that its largest
use-case is with ( ;-) ) open:

with open(filename, mode) as my_file:
    ....
    ....

And yet for years I happily counted on reference counting to close my
files, and was particularly happy with:

data = open(filename, mode).read()

I really liked that that file got opened, read, and closed and cleaned up
right off the bat.

And then context managers were introduced. And it seems to be there is a
consensus in the Python community that we all should be using them when
working on files, and I myself have finally started routinely using them,
and teaching newbies to use them -- which is kind of a pain, 'cause I want
to have them do basic file reading stuff before I explain what a "context
manager" is.

Anyway, my point is that the broader Python community really has been
pretty consistent about making it easy to write code that will work the
same way (maybe not with the same performance) across python
implementations. Ans specifically  with deterministic resource management.

On my system, I can open 1000+ files as a regular user. I can't even
> comprehend opening a tenth of that as an ordinary application, although
> I can imagine that if I were writing a server application things would
> be different.


well, what you can image isn't really the point -- I've bumped into that
darn open file limit in my work, which was not a server application (though
it was some pretty serious number crunching...). And I'm sure I'm not
alone. OK, to be fair that was a poorly designed library, not an issue with
determinism of resource management (through designing the lib well WOULD
depend on that)

But then I don't expect to write server applications in
> quite the same way as I do quick scripts or regular user applications.
>

Though data analysts DO write "quick scripts" that might need to do things
like access 100s of files...


> So it seems to me that a leaked file handler or two normally shouldn't
> be a problem in practice. They'll be friend when the script or
> application closes, and in the meantime, you have hundreds more
> available. 90% of the time, using `with file` does exactly what we want,
> and the times it doesn't (because we're writing a generator that isn't
> closed promptly) 90% of those times it doesn't matter.


that was the case with "with file" from the beginning -- particularly on
cPython. And yet we all thought it was a great idea.


> So (it seems to
> me) that you're talking about changing the behaviour of for-loops to
> suit only a small proportion of cases: maybe 10% of 10%.
>

I don't see what the big overhead is here. for loops would get a new
feature, but it would only be used by the objects that chose to implement
it. So no huge change.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20161021/743650bb/attachment.html>