I tried to test a very simple module:
static void myFree(void *arg) { printf("it's the free function\n"); }
static struct PyModuleDef myModulemodule = { PyModuleDef_HEAD_INIT, .m_name = "myModule", .m_doc = NULL, .m_size = -1, .m_methods = NULL, .m_slots = NULL, .m_traverse = NULL, .m_clear = NULL, .m_free = myFree };
PyMODINIT_FUNC PyInit_myModule(void) { return PyModule_Create(&myModulemodule); }
And in Python script, I did what you proposed:
import sys import myModule print(sys.modules) myModule = None del sys.modules['myModule'] print(sys.modules)
import myModule print(sys.modules) myModule = None del sys.modules['myModule'] print(sys.modules)
In my test, the module can be deleted. I check sys.modules after "import myModule", myModule is added to sys.modules, and after deleting module, it is indeed deleted from sys.modules. The only different is that the message "it's the free function" of m_free is not printed the second time.
I tried to add a simple method in the module: static PyObject* print_str(PyObject* self, PyObject* args) { char* content; if (!PyArg_ParseTuple(args, "s", &content)) return NULL; printf("we want to print %s\n", content); Py_INCREF(Py_None); return Py_None; }
static PyMethodDef MyModuleMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} }; The message "it's the free function" of m_free is not printed at all, but the module can be deleted as well.
Is it normal? Or I did something wrong?
Best, Kun
Victor Stinner vstinner@python.org 于2021年4月13日周二 下午7:48写道:
To test your C extension, try to import it, then delete it. And then import it again.
Example:
import sys
# load the C extension once import _abc _abc = None del sys.modules['_abc']
# reload the C extension import _abc _abc = None del sys.modules['_abc']
Last time I checked, it was basically not possible to unload a C extension written using the legacy C API. You should use the multi-phase initialization API if you care about subinterpreters.
Victor
On Tue, Apr 13, 2021 at 7:11 PM Kun He hekun19890913@gmail.com wrote:
Hi Victor,
I don't think that my C extension uses the multi-phase initialization.
Instead of calling PyModuleDef_Init(&_abcmodule), I call PyModule_Create(&_abcmodule). If the C extension uses single-phase initialization, m_free will not be called? And m_free is indeed for finalizing?
I did some tests. (In all tests I used PyModule_Create()) I defined "myFree" function just to see whether m_free is called or not:
static void myFree(void *arg) { printf("it's the m_free function\n"); }
static struct PyModuleDef myModulemodule = { PyModuleDef_HEAD_INIT, .m_name = "myModule", .m_doc = NULL, .m_size = -1, .m_methods = MyModuleMethods, .m_slots = NULL, .m_traverse = NULL, .m_clear = NULL, .m_free = myFree };
PyMODINIT_FUNC PyInit_myModule(void) { return PyModule_Create(&myModulemodule); }
"it's the m_free function" is printed in the end. I think that means
m_free is called.
Then I did some tests in my code, I found that if the reference count
does not reach zero in the end, m_free will not be called.
So, no matter which initialization I use, PyModuleDef_Init() or
PyModule_Create(), only make sure that the reference count reaches zero in the end, m_free will be called. Am I right??
Best, Kun
Victor Stinner vstinner@python.org 于2021年4月13日周二 下午4:15写道:
m_free is called if your C extension uses the multi-phase initialization API (PEP 489), is it the case?
Something like that:
static struct PyModuleDef _abcmodule = { PyModuleDef_HEAD_INIT, .m_name = "_abc", .m_doc = _abc__doc__, .m_size = sizeof(_abcmodule_state), .m_methods = _abcmodule_methods, .m_slots = _abcmodule_slots, .m_traverse = _abcmodule_traverse, .m_clear = _abcmodule_clear, .m_free = _abcmodule_free, };
PyMODINIT_FUNC PyInit__abc(void) { return PyModuleDef_Init(&_abcmodule); }
See the "Multiphase initialization API" section of my article: https://vstinner.github.io/isolate-subinterpreters.html
Victor
On Mon, Apr 12, 2021 at 2:59 PM Kun He hekun19890913@gmail.com wrote:
I saw this issue36225. Thanks a lot.
In fact, I tried to put Py_EndInterpreter() and all the other
Best, Kun
Victor Stinner vstinner@python.org 于2021年4月12日周一 下午2:04写道:
As I see in the document, this function can destroy all
sub-interpreters
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He hekun19890913@gmail.com
wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling
Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the
I suppose that I can call Py_EndInterpreter() manually before
Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner vstinner@python.org 于2021年4月12日周一 上午10:17写道: > > You need to call PyGILState_Ensure()/PyGILState_Release() around
> the call to the C extension function. These function creates a Python > thread state to the newly created thread. > > > So when calling Py_NewInterpreter() I don't have the GIL. > > Sorry, but this is wrong. All functions of the C API require to hold > the GIL. If it's not the case, it's a bug in your code. > > Victor > > On Fri, Apr 9, 2021 at 6:06 PM Kun He hekun19890913@gmail.com wrote: > > > > Hi Victor, > > > > Thank you for handling my problem. > > > > In fact, in my actually processed C interface, I need to call a C extension function which calls Py_NewInterpreter() in new thread, and I
> > > > Best regards, > > Kun > > > > Victor Stinner vstinner@python.org 于2021年4月9日周五 下午3:57写道: > >> > >> Aha, I reproduce your issue: https://bugs.python.org/issue43793 > >> > >> Currently, I understand that you must first acquire the GIL using > >> PyGILState_Ensure()/PyGILState_Release() in the newly created
> >> to be able to call Py_NewInterpreter(). See: > >> https://docs.python.org/dev/c-api/init.html#non-python-created-threads > >> > >> Maybe tomorrow it will be possible to call Py_NewInterpreter() on a > >> newly created thread which has no thread state. > >> > >> Victor > >> > >> On Fri, Apr 9, 2021 at 2:26 PM Kun He hekun19890913@gmail.com wrote: > >> > > >> > Hi Victor, > >> > > >> > Thanks for your response. I'm sorry that the attachment is not available. > >> > Anyway, I copy the C program as you can see below: > >> > > >> > #include
> >> > #include > >> > > >> > #include > >> > > >> > #define PY_SSIZE_T_CLEAN > >> > #include > >> > > >> > static PyObject* print_str(PyObject* self, PyObject* args) > >> > { > >> > char* content; > >> > > >> > if (!PyArg_ParseTuple(args, "s", &content)) > >> > return NULL; > >> > > >> > printf("we want to print %s\n", content); > >> > > >> > Py_INCREF(Py_None); > >> > return Py_None; > >> > } > >> > > >> > static PyMethodDef NewinterMethods[] = { > >> > {"print_str", print_str, METH_VARARGS, "print the input string"}, > >> > {NULL, NULL, 0, NULL} > >> > }; > >> > > >> > static struct PyModuleDef newintermodule = { > >> > PyModuleDef_HEAD_INIT, > >> > "newinter", > >> > NULL, > >> > -1, > >> > NewinterMethods > >> > }; > >> > > >> > pthread_t tid; > >> > > >> > > >> > void *thr_func(void *arg) > >> > { > >> > Py_NewInterpreter(); > >> > return ((void *)0); > >> > } > >> > > >> > PyMODINIT_FUNC > >> > PyInit_newinter(void) > >> > { > >> > PyObject *m; > >> > > >> > m = PyModule_Create(&newintermodule); > >> > > >> > #if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) > >> > PyEval_InitThreads(); > >> > #endif > >> > > >> > int err = pthread_create(&tid, NULL, thr_func, NULL); > >> > if(err != 0) > >> > { > >> > printf("can't create thread\n"); > >> > return NULL; > >> > } > >> > else > >> > printf("create new thread\n"); > >> > > >> > return m; > >> > } > >> > > >> > And for test, a very simple Python script is used as below: > >> > import newinter > >> > newinter.print_str("test") > >> > > >> > > >> > Sincerely, > >> > Kun > >> > > >> > Victor Stinner vstinner@python.org 于2021年4月9日周五 下午2:01写道: > >> >> > >> >> Hi Kun, > >> >> > >> >> Can you please try to provide a short C program reproducing your > >> >> issue? Sadly, I cannot see your attachement. Try to copy it shutdown functions of my code in freefun m_free of struct PyModuleDef, but it seems that m_free is not called in the end. So I would like to ask which function should we call for finalizing? program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens? the put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL. thread directly
> >> >> in the email body. > >> >> > >> >> Victor > >> >> > >> >> On Fri, Apr 9, 2021 at 12:09 PM Kun He < hekun19890913@gmail.com> wrote: > >> >> > > >> >> > Hello, > >> >> > > >> >> > I'm working on a C extension interface of Python. I want to create a new > >> >> > interpreter by using the function Py_NewInterpreter() in a new thread, > >> >> > which is created by pthread_create (my test files are in attachment), but > >> >> > there are always errors when calling Py_NewInterpreter() such as "failed: > >> >> > object already tracked by the garbage collector". > >> >> > > >> >> > I would like to ask how to solve the problem and create a new interpreter > >> >> > in multi thread in Python C extension? > >> >> > > >> >> > Sincerely, > >> >> > Kun > >> >> > _______________________________________________ > >> >> > capi-sig mailing list -- capi-sig@python.org > >> >> > To unsubscribe send an email to capi-sig-leave@python.org > >> >> > https://mail.python.org/mailman3/lists/capi-sig.python.org/ > >> >> > Member address: vstinner@python.org > >> >> > >> >> > >> >> > >> >> -- > >> >> Night gathers, and now my watch begins. It shall not end until my death. > >> > >> > >> > >> -- > >> Night gathers, and now my watch begins. It shall not end until my death. > > > > -- > Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.