On 09. 11. 21 0:54, Gregory P. Smith wrote:

On Mon, Nov 8, 2021 at 1:16 PM Victor Milovanov <lostfreeman@gmail.com> wrote:

...

Hi,

TL;DR; Is it safe to Py_DecRef an object that was created before Py_Finalize after Py_Initialize was called to restart Python runtime?

TL;DR: objects are part if the structures that Py_Finalize tears down; if you want to keep objects you need to keep the interpreter state around as well.

gut take: No. Because there should be no valid PyObject's remaining after Py_Finalize.

That's correct.

After finalize the entire garbage collection mechanism should be considered to be in an undefined state.

That is not: The garbage collection should be in a state where there are no old objects remaining. myObj should be deallocated at this point. If it isn't/wasn't, that's a bug in extension code or in Python. (I wouldn't be surprised if there are bugs like this in Python.)

The garbage collection mechanism state should be fine, but calling Py_DecRef (or anything else) on a deallocated object is undefined behavior.

...

I am debugging 3.9+ support in https://github.com/pythonnet/pythonnet

The issue I originally got is validate_list from gcmodule.c fails at

assert(prev == GC_PREV(head));

In debugging this, I sprinkled my code with calls to validate_list, and found that the assertion fails in the following scenario:

 Py_Initialize();
 ...
 PyObject* myObj = PyObject_Call(simple_class, empty_tuple, NULL);
 Py_Finalize();
 ...
 Py_Initialize();
 ...
 validate_list(gc.generations[2], collecting_clear_unreachable_clear);

// OK Py_DecRef(myObj); validate_list(gc.generations[2], collecting_clear_unreachable_clear); // EXPLOSION

In general, tearing down the interpreter and creating a new one within a single process is not well supported in CPython. Global state still exists and may not be detected and cleaned up properly on Py_Finalize. Especially true for extension modules. Work is ongoing to improve this situation in bits and pieces but I do not believe any released version of CPython to date makes this safe. (The multi-phase init PEP 489 adoption work usually moves things in the right direction, though that doesn't solve the problem on its own)

"not well supported" means that this is not a well used/tested area of the interpreter, so you're likely to find bugs. If you do find any, please report them.

Some third-party extensions might not support reinitialization at all, but Python and the standard library should. (Eventually. There's not a lot of people working on this.)

I'm not saying that C code doing this does not exist... Just that it tends to be tripped up by the existing bugs eventually even if it _appeared_ to work at one point.

-gps

Now I have not yet tried to narrow down the steps further, because I

...

realised that I am unsure about the question in TL;DR;. E.g. is this use of myObj supported by embedding API in general, or is this expected, that after reinitializing the runtime Py_DecRef on old objects would break GC?

Good call!

...

In my scenario simple_class is defined in Python as inheriting from object, and only has two attributes set at runtime. One has value builtins.StopIteration, and the other one is an instance of traceback.

I suspect the traceback could cause a reference loop that's keeping the object from being garbage-collected.

There's a bunch of caveats listed in Py_FinalizeEx docs: https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx (Granted, these aren't precise at all.) You code might be relying on the "Memory tied up in circular references between objects is not freed" part. This is a bug; don't rely on it. It might have been fixed in Python 3.9 for your case; that would, ironically, break your code.