To test your C extension, try to import it, then delete it. And then import it again.
Example:
import sys
# load the C extension once import _abc _abc = None del sys.modules['_abc']
# reload the C extension import _abc _abc = None del sys.modules['_abc']
Last time I checked, it was basically not possible to unload a C extension written using the legacy C API. You should use the multi-phase initialization API if you care about subinterpreters.
Victor
On Tue, Apr 13, 2021 at 7:11 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
I don't think that my C extension uses the multi-phase initialization. Instead of calling PyModuleDef_Init(&_abcmodule), I call PyModule_Create(&_abcmodule). If the C extension uses single-phase initialization, m_free will not be called? And m_free is indeed for finalizing?
I did some tests. (In all tests I used PyModule_Create()) I defined "myFree" function just to see whether m_free is called or not:
static void myFree(void *arg) { printf("it's the m_free function\n"); }
static struct PyModuleDef myModulemodule = { PyModuleDef_HEAD_INIT, .m_name = "myModule", .m_doc = NULL, .m_size = -1, .m_methods = MyModuleMethods, .m_slots = NULL, .m_traverse = NULL, .m_clear = NULL, .m_free = myFree };
PyMODINIT_FUNC PyInit_myModule(void) { return PyModule_Create(&myModulemodule); }
"it's the m_free function" is printed in the end. I think that means m_free is called.
Then I did some tests in my code, I found that if the reference count does not reach zero in the end, m_free will not be called.
So, no matter which initialization I use, PyModuleDef_Init() or PyModule_Create(), only make sure that the reference count reaches zero in the end, m_free will be called. Am I right??
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月13日周二 下午4:15写道：
m_free is called if your C extension uses the multi-phase initialization API (PEP 489), is it the case?
Something like that:
static struct PyModuleDef _abcmodule = { PyModuleDef_HEAD_INIT, .m_name = "_abc", .m_doc = _abc__doc__, .m_size = sizeof(_abcmodule_state), .m_methods = _abcmodule_methods, .m_slots = _abcmodule_slots, .m_traverse = _abcmodule_traverse, .m_clear = _abcmodule_clear, .m_free = _abcmodule_free, };
PyMODINIT_FUNC PyInit__abc(void) { return PyModuleDef_Init(&_abcmodule); }
See the "Multiphase initialization API" section of my article: https://vstinner.github.io/isolate-subinterpreters.html
Victor
On Mon, Apr 12, 2021 at 2:59 PM Kun He <hekun19890913@gmail.com> wrote:
I saw this issue36225. Thanks a lot.
In fact, I tried to put Py_EndInterpreter() and all the other shutdown functions of my code in freefun m_free of struct PyModuleDef, but it seems that m_free is not called in the end. So I would like to ask which function should we call for finalizing?
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 下午2:04写道：
As I see in the document, this function can destroy all sub-interpreters
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens?
I suppose that I can call Py_EndInterpreter() manually before Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道：
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
> So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com> wrote: > > Hi Victor, > > Thank you for handling my problem. > > In fact, in my actually processed C interface, I need to call a C extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL. > > Best regards, > Kun > > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道： >> >> Aha, I reproduce your issue: https://bugs.python.org/issue43793 >> >> Currently, I understand that you must first acquire the GIL using >> PyGILState_Ensure()/PyGILState_Release() in the newly created thread >> to be able to call Py_NewInterpreter(). See: >> https://docs.python.org/dev/c-api/init.html#non-python-created-threads >> >> Maybe tomorrow it will be possible to call Py_NewInterpreter() on a >> newly created thread which has no thread state. >> >> Victor >> >> On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote: >> > >> > Hi Victor, >> > >> > Thanks for your response. I'm sorry that the attachment is not available. >> > Anyway, I copy the C program as you can see below: >> > >> > #include <stdio.h> >> > #include <string.h> >> > >> > #include <pthread.h> >> > >> > #define PY_SSIZE_T_CLEAN >> > #include <Python.h> >> > >> > static PyObject* print_str(PyObject* self, PyObject* args) >> > { >> > char* content; >> > >> > if (!PyArg_ParseTuple(args, "s", &content)) >> > return NULL; >> > >> > printf("we want to print %s\n", content); >> > >> > Py_INCREF(Py_None); >> > return Py_None; >> > } >> > >> > static PyMethodDef NewinterMethods[] = { >> > {"print_str", print_str, METH_VARARGS, "print the input string"}, >> > {NULL, NULL, 0, NULL} >> > }; >> > >> > static struct PyModuleDef newintermodule = { >> > PyModuleDef_HEAD_INIT, >> > "newinter", >> > NULL, >> > -1, >> > NewinterMethods >> > }; >> > >> > pthread_t tid; >> > >> > >> > void *thr_func(void *arg) >> > { >> > Py_NewInterpreter(); >> > return ((void *)0); >> > } >> > >> > PyMODINIT_FUNC >> > PyInit_newinter(void) >> > { >> > PyObject *m; >> > >> > m = PyModule_Create(&newintermodule); >> > >> > #if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) >> > PyEval_InitThreads(); >> > #endif >> > >> > int err = pthread_create(&tid, NULL, thr_func, NULL); >> > if(err != 0) >> > { >> > printf("can't create thread\n"); >> > return NULL; >> > } >> > else >> > printf("create new thread\n"); >> > >> > return m; >> > } >> > >> > And for test, a very simple Python script is used as below: >> > import newinter >> > newinter.print_str("test") >> > >> > >> > Sincerely, >> > Kun >> > >> > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道： >> >> >> >> Hi Kun, >> >> >> >> Can you please try to provide a short C program reproducing your >> >> issue? Sadly, I cannot see your attachement. Try to copy it directly >> >> in the email body. >> >> >> >> Victor >> >> >> >> On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote: >> >> > >> >> > Hello, >> >> > >> >> > I'm working on a C extension interface of Python. I want to create a new >> >> > interpreter by using the function Py_NewInterpreter() in a new thread, >> >> > which is created by pthread_create (my test files are in attachment), but >> >> > there are always errors when calling Py_NewInterpreter() such as "failed: >> >> > object already tracked by the garbage collector". >> >> > >> >> > I would like to ask how to solve the problem and create a new interpreter >> >> > in multi thread in Python C extension? >> >> > >> >> > Sincerely, >> >> > Kun >> >> > _______________________________________________ >> >> > capi-sig mailing list -- capi-sig@python.org >> >> > To unsubscribe send an email to capi-sig-leave@python.org >> >> > https://mail.python.org/mailman3/lists/capi-sig.python.org/ >> >> > Member address: vstinner@python.org >> >> >> >> >> >> >> >> -- >> >> Night gathers, and now my watch begins. It shall not end until my death. >> >> >> >> -- >> Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.