RFC: Proposal: Deterministic Object Destruction

[Steven D'Aprano]

On Sat, 03 Mar 2018 08:02:35 -0800, ooomzay wrote:

[...]
>> > But I am not! On the contrary RAII frees the programmer from even
>> > having to remember to close the file. The poster asked what would
>> > happen if the resource was deliberately kept open by storing a
>> > reference at global scope.
>> 
>> You say that as if it were difficult to do, requiring the programmer to
>> take extraordinary steps of heroic proportion. It doesn't. It is
>> unbelievably easy to store the reference at global scope, which means
>> that the programmer needs to remember to close the file explicitly and
>> RAII doesn't help.
> 
> Can you expand on what you mean by "unbelievably easy to store the
> reference at global scope". I will assume you are claiming that it is
> easy to _inadvertently_ store a global reference.

No.

Copy and paste this line:

--- %< ---------

x = 1

--- %< ---------

into a text file, and save it as "demo_global.py". Congratulations, 
you've just created a reference in the global scope!

I told you it was easy.

Do you put your imports at the top of the module, as recommended? Every 
one of those imported modules are references in the global scope.

Do you define classes and functions at the top level of your module? 
Those too are references in the global scope.

I'll accept that, *typically*, imported modules, functions and classes 
are unlikely to involve the sorts of resources that we care about closing 
in a timely fashion. But other kinds of top-level instances may.

The point is, you want to *mandate* RAII as a *language feature* rather 
than an implementation detail which Python interpreters are free to 
choose or not choose on their own. This (probably?) will put a VERY high 
burden on Jython, IronPython and Stackless, as well as anyone else who 
tries making a Python interpreter. And for what benefit? Let's see:

- RAII doesn't make reference cycles go away, so we still need a 
  garbage collector that can break cycles;

- RAII rarely helps for objects in the module global namespace, and
  many Python objects are in the global namespace;

- to make RAII work for heap objects, every implementation needs to
  implement their own "smart pointers" or similar, otherwise the entire
  idea is a bust; this adds significant complexity to interpreters that
  aren't written in C++ (i.e. all of them apart from Nuitka);

- to make the RAII guarantees of timely resource closure work, you have
  to impose significant mental burdens on the programmer: 

  * no cycles (avoid them, or use weak references);
  * no globals; 
  * if you must use a global, then you are responsible for 
    manually deleting it in order to allow RAII to operate;

- to make the RAII guarantees work *well*, you need to introduce new
  scoping rules (new to Python, not C++ ) where each indented block 
  (loops, if...else blocks etc) are new scopes; otherwise the lifetime
  of a function scope could easily be too long.



> This problem with destruction of globals on exit is not unique to
> python. There are patterns to manage this for code you control, such as
> putting them all in one global "bucket" object and explicitly deleting
> it before exit.

The problem we have is, we want the resource to be closed in a timely 
manner once we're done with it. And your solution is to put it in a "God 
Object" that collects a whole lot of unrelated resources, then wait until 
you are done with the last of the them, before closing any of them?

That's not a pattern managing the problem, that's a work-around that 
fails to come even close to fixing the problem.


[...]
>> But as you said yourself, if the resource is held open in a global
>> reference, it will stay open indefinitely. And remember, global in this
>> context doesn't just mean the main module of your application, but
>> *every* module you import.
> 
> Can you give an example of such a case where an application-managed
> (acquired/opened & released/closed) resource is referenced strongly by a
> 3rd party module at global scope? ...in my experience this pattern
> usually smells.

Regardless of whether it smells or not, it is allowed.


>> I think you have just put your finger on the difference between what
>> RAII *claims* to do and what it *actually* can do.
> 
> I can assure you that RAII does what it says on the tin and is relied on
> in many critical systems to release resources robustly ... given the
> pre-requisite deterministic destruction.

Given the execution environment and design constraints of C++, I'm sure 
RAII works fine in C++ (and any other language which may copy it).

But it is not a panacea that makes resource management go away. You 
yourself have stated at least one limitation: if you hold a global 
reference to the object then RAII won't help you.

So your proposal requires major changes to not just one but all (present 
and future) Python interpreters, AND major changes to the way people 
write Python code, in order to solve a problem which is already solved by 
context managers and the with statement.



-- 
Steve