
Hi Victor,
I don't think that my C extension uses the multi-phase initialization. Instead of calling PyModuleDef_Init(&_abcmodule), I call PyModule_Create(&_abcmodule). If the C extension uses single-phase initialization, m_free will not be called? And m_free is indeed for finalizing?
I did some tests. (In all tests I used PyModule_Create()) I defined "myFree" function just to see whether m_free is called or not:
static void myFree(void *arg) { printf("it's the m_free function\n"); }
static struct PyModuleDef myModulemodule = { PyModuleDef_HEAD_INIT, .m_name = "myModule", .m_doc = NULL, .m_size = -1, .m_methods = MyModuleMethods, .m_slots = NULL, .m_traverse = NULL, .m_clear = NULL, .m_free = myFree };
PyMODINIT_FUNC PyInit_myModule(void) { return PyModule_Create(&myModulemodule); }
"it's the m_free function" is printed in the end. I think that means m_free is called.
Then I did some tests in my code, I found that if the reference count does not reach zero in the end, m_free will not be called.
So, no matter which initialization I use, PyModuleDef_Init() or PyModule_Create(), only make sure that the reference count reaches zero in the end, m_free will be called. Am I right??
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月13日周二 下午4:15写道:
m_free is called if your C extension uses the multi-phase initialization API (PEP 489), is it the case?
Something like that:
static struct PyModuleDef _abcmodule = { PyModuleDef_HEAD_INIT, .m_name = "_abc", .m_doc = _abc__doc__, .m_size = sizeof(_abcmodule_state), .m_methods = _abcmodule_methods, .m_slots = _abcmodule_slots, .m_traverse = _abcmodule_traverse, .m_clear = _abcmodule_clear, .m_free = _abcmodule_free, };
PyMODINIT_FUNC PyInit__abc(void) { return PyModuleDef_Init(&_abcmodule); }
See the "Multiphase initialization API" section of my article: https://vstinner.github.io/isolate-subinterpreters.html
Victor
On Mon, Apr 12, 2021 at 2:59 PM Kun He <hekun19890913@gmail.com> wrote:
I saw this issue36225. Thanks a lot.
In fact, I tried to put Py_EndInterpreter() and all the other shutdown
functions of my code in freefun m_free of struct PyModuleDef, but it seems that m_free is not called in the end. So I would like to ask which function should we call for finalizing?
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 下午2:04写道:
As I see in the document, this function can destroy all
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He <hekun19890913@gmail.com>
wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling
Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the
I suppose that I can call Py_EndInterpreter() manually before
Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道:
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com>
wrote:
Hi Victor,
Thank you for handling my problem.
In fact, in my actually processed C interface, I need to call a C
extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL.
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道: > > Aha, I reproduce your issue: https://bugs.python.org/issue43793 > > Currently, I understand that you must first acquire the GIL using > PyGILState_Ensure()/PyGILState_Release() in the newly created
> to be able to call Py_NewInterpreter(). See: > https://docs.python.org/dev/c-api/init.html#non-python-created-threads > > Maybe tomorrow it will be possible to call Py_NewInterpreter() on a > newly created thread which has no thread state. > > Victor > > On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote: > > > > Hi Victor, > > > > Thanks for your response. I'm sorry that the attachment is not available. > > Anyway, I copy the C program as you can see below: > > > > #include <stdio.h> > > #include <string.h> > > > > #include <pthread.h> > > > > #define PY_SSIZE_T_CLEAN > > #include <Python.h> > > > > static PyObject* print_str(PyObject* self, PyObject* args) > > { > > char* content; > > > > if (!PyArg_ParseTuple(args, "s", &content)) > > return NULL; > > > > printf("we want to print %s\n", content); > > > > Py_INCREF(Py_None); > > return Py_None; > > } > > > > static PyMethodDef NewinterMethods[] = { > > {"print_str", print_str, METH_VARARGS, "print the input string"}, > > {NULL, NULL, 0, NULL} > > }; > > > > static struct PyModuleDef newintermodule = { > > PyModuleDef_HEAD_INIT, > > "newinter", > > NULL, > > -1, > > NewinterMethods > > }; > > > > pthread_t tid; > > > > > > void *thr_func(void *arg) > > { > > Py_NewInterpreter(); > > return ((void *)0); > > } > > > > PyMODINIT_FUNC > > PyInit_newinter(void) > > { > > PyObject *m; > > > > m = PyModule_Create(&newintermodule); > > > > #if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) > > PyEval_InitThreads(); > > #endif > > > > int err = pthread_create(&tid, NULL, thr_func, NULL); > > if(err != 0) > > { > > printf("can't create thread\n"); > > return NULL; > > } > > else > > printf("create new thread\n"); > > > > return m; > > } > > > > And for test, a very simple Python script is used as below: > > import newinter > > newinter.print_str("test") > > > > > > Sincerely, > > Kun > > > > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道: > >> > >> Hi Kun, > >> > >> Can you please try to provide a short C program reproducing your > >> issue? Sadly, I cannot see your attachement. Try to copy it
sub-interpreters program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens? thread directly
> >> in the email body. > >> > >> Victor > >> > >> On Fri, Apr 9, 2021 at 12:09 PM Kun He < hekun19890913@gmail.com> wrote: > >> > > >> > Hello, > >> > > >> > I'm working on a C extension interface of Python. I want to create a new > >> > interpreter by using the function Py_NewInterpreter() in a new thread, > >> > which is created by pthread_create (my test files are in attachment), but > >> > there are always errors when calling Py_NewInterpreter() such as "failed: > >> > object already tracked by the garbage collector". > >> > > >> > I would like to ask how to solve the problem and create a new interpreter > >> > in multi thread in Python C extension? > >> > > >> > Sincerely, > >> > Kun > >> > _______________________________________________ > >> > capi-sig mailing list -- capi-sig@python.org > >> > To unsubscribe send an email to capi-sig-leave@python.org > >> > https://mail.python.org/mailman3/lists/capi-sig.python.org/ > >> > Member address: vstinner@python.org > >> > >> > >> > >> -- > >> Night gathers, and now my watch begins. It shall not end until my death. > > > > -- > Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.