Deallocating objects from previous run
Hi,
TL;DR; Is it safe to Py_DecRef an object that was created before Py_Finalize after Py_Initialize was called to restart Python runtime?
I am debugging 3.9+ support in https://github.com/pythonnet/pythonnet
The issue I originally got is validate_list from gcmodule.c fails at
assert(prev == GC_PREV(head));
In debugging this, I sprinkled my code with calls to validate_list, and found that the assertion fails in the following scenario:
Py_Initialize();
...
PyObject* myObj = PyObject_Call(simple_class, empty_tuple, NULL);
Py_Finalize();
...
Py_Initialize();
...
validate_list(gc.generations[2], collecting_clear_unreachable_clear); // OK
Py_DecRef(myObj);
validate_list(gc.generations[2], collecting_clear_unreachable_clear); // EXPLOSIONNow I have not yet tried to narrow down the steps further, because I realised that I am unsure about the question in TL;DR;. E.g. is this use of myObj supported by embedding API in general, or is this expected, that after reinitializing the runtime Py_DecRef on old objects would break GC?
In my scenario simple_class is defined in Python as inheriting from object, and only has two attributes set at runtime. One has value builtins.StopIteration, and the other one is an instance of traceback.
Regards, Victor
On Mon, Nov 8, 2021 at 1:16 PM Victor Milovanov <lostfreeman@gmail.com> wrote:
gut take: No. Because there should be no valid PyObject's remaining after Py_Finalize. After finalize the entire garbage collection mechanism should be considered to be in an undefined state.
In general, tearing down the interpreter and creating a new one within a single process is not well supported in CPython. Global state still exists and may not be detected and cleaned up properly on Py_Finalize. Especially true for extension modules. Work is ongoing to improve this situation in bits and pieces but I do not believe any released version of CPython to date makes this safe. (The multi-phase init PEP 489 adoption work usually moves things in the right direction, though that doesn't solve the problem on its own)
I'm not saying that C code doing this does not exist... Just that it tends to be tripped up by the existing bugs eventually even if it _appeared_ to work at one point.
-gps
Now I have not yet tried to narrow down the steps further, because I
On 09. 11. 21 0:54, Gregory P. Smith wrote:
TL;DR: objects are part if the structures that Py_Finalize tears down; if you want to keep objects you need to keep the interpreter state around as well.
gut take: No. Because there should be no valid PyObject's remaining after Py_Finalize.
That's correct.
After finalize the entire garbage collection mechanism should be considered to be in an undefined state.
That is not: The garbage collection should be in a state where there are no old objects remaining. myObj should be deallocated at this point. If it isn't/wasn't, that's a bug in extension code or in Python. (I wouldn't be surprised if there are bugs like this in Python.)
The garbage collection mechanism state should be fine, but calling Py_DecRef (or anything else) on a deallocated object is undefined behavior.
"not well supported" means that this is not a well used/tested area of the interpreter, so you're likely to find bugs. If you do find any, please report them.
Some third-party extensions might not support reinitialization at all, but Python and the standard library should. (Eventually. There's not a lot of people working on this.)
Good call!
I suspect the traceback could cause a reference loop that's keeping the object from being garbage-collected.
There's a bunch of caveats listed in Py_FinalizeEx docs: https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx (Granted, these aren't precise at all.) You code might be relying on the "Memory tied up in circular references between objects is not freed" part. This is a bug; don't rely on it. It might have been fixed in Python 3.9 for your case; that would, ironically, break your code.
At the end of each run, Dask workers are automatically cleared. Objects created in previous runs will be removed from memory and workers will be left with a blank slate for the next time you submit jobs. This is currently done by calling the client. restart() which clears all workers and resubmits them with the same configuration as before. In the future, there may be more fine-grained control over this process. Beginners can also hire https://bestonlinewritingservice.com/coursework-writing-services/ for getting the required help. The objects that are already there are ones that were already in your scene before you ran it. If you click on any of those, they'll be highlighted in blue in the scene view.
On Mon, Nov 8, 2021 at 1:16 PM Victor Milovanov <lostfreeman@gmail.com> wrote:
gut take: No. Because there should be no valid PyObject's remaining after Py_Finalize. After finalize the entire garbage collection mechanism should be considered to be in an undefined state.
In general, tearing down the interpreter and creating a new one within a single process is not well supported in CPython. Global state still exists and may not be detected and cleaned up properly on Py_Finalize. Especially true for extension modules. Work is ongoing to improve this situation in bits and pieces but I do not believe any released version of CPython to date makes this safe. (The multi-phase init PEP 489 adoption work usually moves things in the right direction, though that doesn't solve the problem on its own)
I'm not saying that C code doing this does not exist... Just that it tends to be tripped up by the existing bugs eventually even if it _appeared_ to work at one point.
-gps
Now I have not yet tried to narrow down the steps further, because I
On 09. 11. 21 0:54, Gregory P. Smith wrote:
TL;DR: objects are part if the structures that Py_Finalize tears down; if you want to keep objects you need to keep the interpreter state around as well.
gut take: No. Because there should be no valid PyObject's remaining after Py_Finalize.
That's correct.
After finalize the entire garbage collection mechanism should be considered to be in an undefined state.
That is not: The garbage collection should be in a state where there are no old objects remaining. myObj should be deallocated at this point. If it isn't/wasn't, that's a bug in extension code or in Python. (I wouldn't be surprised if there are bugs like this in Python.)
The garbage collection mechanism state should be fine, but calling Py_DecRef (or anything else) on a deallocated object is undefined behavior.
"not well supported" means that this is not a well used/tested area of the interpreter, so you're likely to find bugs. If you do find any, please report them.
Some third-party extensions might not support reinitialization at all, but Python and the standard library should. (Eventually. There's not a lot of people working on this.)
Good call!
I suspect the traceback could cause a reference loop that's keeping the object from being garbage-collected.
There's a bunch of caveats listed in Py_FinalizeEx docs: https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx (Granted, these aren't precise at all.) You code might be relying on the "Memory tied up in circular references between objects is not freed" part. This is a bug; don't rely on it. It might have been fixed in Python 3.9 for your case; that would, ironically, break your code.
At the end of each run, Dask workers are automatically cleared. Objects created in previous runs will be removed from memory and workers will be left with a blank slate for the next time you submit jobs. This is currently done by calling the client. restart() which clears all workers and resubmits them with the same configuration as before. In the future, there may be more fine-grained control over this process. Beginners can also hire https://bestonlinewritingservice.com/coursework-writing-services/ for getting the required help. The objects that are already there are ones that were already in your scene before you ran it. If you click on any of those, they'll be highlighted in blue in the scene view.
participants (4)
-
Gregory P. Smith -
nisos70722＠sartess.com -
Petr Viktorin -
Victor Milovanov