Can't create new interpreter in multi thread
Hello,
I'm working on a C extension interface of Python. I want to create a new interpreter by using the function Py_NewInterpreter() in a new thread, which is created by pthread_create (my test files are in attachment), but there are always errors when calling Py_NewInterpreter() such as "failed: object already tracked by the garbage collector".
I would like to ask how to solve the problem and create a new interpreter in multi thread in Python C extension?
Sincerely, Kun
Just an FYI: Attachments are pointless as only you have access to them and that's because you have access to the original sent email.
Hi Kun,
Can you please try to provide a short C program reproducing your issue? Sadly, I cannot see your attachement. Try to copy it directly in the email body.
Victor
On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote:
Hello,
I'm working on a C extension interface of Python. I want to create a new interpreter by using the function Py_NewInterpreter() in a new thread, which is created by pthread_create (my test files are in attachment), but there are always errors when calling Py_NewInterpreter() such as "failed: object already tracked by the garbage collector".
I would like to ask how to solve the problem and create a new interpreter in multi thread in Python C extension?
Sincerely, Kun
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
Hi Victor,
Thanks for your response. I'm sorry that the attachment is not available. Anyway, I copy the C program as you can see below:
#include <stdio.h> #include <string.h>
#include <pthread.h>
#define PY_SSIZE_T_CLEAN #include <Python.h>
static PyObject* print_str(PyObject* self, PyObject* args) { char* content;
if (!PyArg_ParseTuple(args, "s", &content))
return NULL;
printf("we want to print %s\n", content);
Py_INCREF(Py_None);
return Py_None;}
static PyMethodDef NewinterMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} };
static struct PyModuleDef newintermodule = { PyModuleDef_HEAD_INIT, "newinter", NULL, -1, NewinterMethods };
pthread_t tid;
void *thr_func(void *arg) { Py_NewInterpreter(); return ((void *)0); }
PyMODINIT_FUNC PyInit_newinter(void) { PyObject *m;
m = PyModule_Create(&newintermodule);#if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) PyEval_InitThreads(); #endif
int err = pthread_create(&tid, NULL, thr_func, NULL);
if(err != 0)
{
printf("can't create thread\n");
return NULL;
}
else
printf("create new thread\n");
return m;}
And for test, a very simple Python script is used as below: import newinter newinter.print_str("test")
Sincerely, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道：
Hi Kun,
Can you please try to provide a short C program reproducing your issue? Sadly, I cannot see your attachement. Try to copy it directly in the email body.
Victor
On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote:
Hello,
I'm working on a C extension interface of Python. I want to create a new interpreter by using the function Py_NewInterpreter() in a new thread, which is created by pthread_create (my test files are in attachment), but there are always errors when calling Py_NewInterpreter() such as "failed: object already tracked by the garbage collector".
I would like to ask how to solve the problem and create a new interpreter in multi thread in Python C extension?
Sincerely, Kun
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
Aha, I reproduce your issue: https://bugs.python.org/issue43793
Currently, I understand that you must first acquire the GIL using PyGILState_Ensure()/PyGILState_Release() in the newly created thread to be able to call Py_NewInterpreter(). See: https://docs.python.org/dev/c-api/init.html#non-python-created-threads
Maybe tomorrow it will be possible to call Py_NewInterpreter() on a newly created thread which has no thread state.
Victor
On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thanks for your response. I'm sorry that the attachment is not available. Anyway, I copy the C program as you can see below:
#include <stdio.h> #include <string.h>
#include <pthread.h>
#define PY_SSIZE_T_CLEAN #include <Python.h>
static PyObject* print_str(PyObject* self, PyObject* args) { char* content;
if (!PyArg_ParseTuple(args, "s", &content)) return NULL; printf("we want to print %s\n", content); Py_INCREF(Py_None); return Py_None;}
static PyMethodDef NewinterMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} };
static struct PyModuleDef newintermodule = { PyModuleDef_HEAD_INIT, "newinter", NULL, -1, NewinterMethods };
pthread_t tid;
void *thr_func(void *arg) { Py_NewInterpreter(); return ((void *)0); }
PyMODINIT_FUNC PyInit_newinter(void) { PyObject *m;
m = PyModule_Create(&newintermodule);#if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) PyEval_InitThreads(); #endif
int err = pthread_create(&tid, NULL, thr_func, NULL); if(err != 0) { printf("can't create thread\n"); return NULL; } else printf("create new thread\n"); return m;}
And for test, a very simple Python script is used as below: import newinter newinter.print_str("test")
Sincerely, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道：
Hi Kun,
Can you please try to provide a short C program reproducing your issue? Sadly, I cannot see your attachement. Try to copy it directly in the email body.
Victor
On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote:
Hello,
I'm working on a C extension interface of Python. I want to create a new interpreter by using the function Py_NewInterpreter() in a new thread, which is created by pthread_create (my test files are in attachment), but there are always errors when calling Py_NewInterpreter() such as "failed: object already tracked by the garbage collector".
I would like to ask how to solve the problem and create a new interpreter in multi thread in Python C extension?
Sincerely, Kun
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
Hi Victor,
Thank you for handling my problem.
In fact, in my actually processed C interface, I need to call a C extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL.
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道：
Aha, I reproduce your issue: https://bugs.python.org/issue43793
Currently, I understand that you must first acquire the GIL using PyGILState_Ensure()/PyGILState_Release() in the newly created thread to be able to call Py_NewInterpreter(). See: https://docs.python.org/dev/c-api/init.html#non-python-created-threads
Maybe tomorrow it will be possible to call Py_NewInterpreter() on a newly created thread which has no thread state.
Victor
On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thanks for your response. I'm sorry that the attachment is not available. Anyway, I copy the C program as you can see below:
#include <stdio.h> #include <string.h>
#include <pthread.h>
#define PY_SSIZE_T_CLEAN #include <Python.h>
static PyObject* print_str(PyObject* self, PyObject* args) { char* content;
if (!PyArg_ParseTuple(args, "s", &content)) return NULL; printf("we want to print %s\n", content); Py_INCREF(Py_None); return Py_None;}
static PyMethodDef NewinterMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} };
static struct PyModuleDef newintermodule = { PyModuleDef_HEAD_INIT, "newinter", NULL, -1, NewinterMethods };
pthread_t tid;
void *thr_func(void *arg) { Py_NewInterpreter(); return ((void *)0); }
PyMODINIT_FUNC PyInit_newinter(void) { PyObject *m;
m = PyModule_Create(&newintermodule);#if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION <
PyEval_InitThreads();#endif
int err = pthread_create(&tid, NULL, thr_func, NULL); if(err != 0) { printf("can't create thread\n"); return NULL; } else printf("create new thread\n"); return m;}
And for test, a very simple Python script is used as below: import newinter newinter.print_str("test")
Sincerely, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道：
Hi Kun,
Can you please try to provide a short C program reproducing your issue? Sadly, I cannot see your attachement. Try to copy it directly in the email body.
Victor
On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote:
Hello,
I'm working on a C extension interface of Python. I want to create a
new
interpreter by using the function Py_NewInterpreter() in a new thread, which is created by pthread_create (my test files are in attachment), but there are always errors when calling Py_NewInterpreter() such as "failed: object already tracked by the garbage collector".
I would like to ask how to solve the problem and create a new interpreter in multi thread in Python C extension?
Sincerely, Kun
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thank you for handling my problem.
In fact, in my actually processed C interface, I need to call a C extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL.
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道：
Aha, I reproduce your issue: https://bugs.python.org/issue43793
Currently, I understand that you must first acquire the GIL using PyGILState_Ensure()/PyGILState_Release() in the newly created thread to be able to call Py_NewInterpreter(). See: https://docs.python.org/dev/c-api/init.html#non-python-created-threads
Maybe tomorrow it will be possible to call Py_NewInterpreter() on a newly created thread which has no thread state.
Victor
On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thanks for your response. I'm sorry that the attachment is not available. Anyway, I copy the C program as you can see below:
#include <stdio.h> #include <string.h>
#include <pthread.h>
#define PY_SSIZE_T_CLEAN #include <Python.h>
static PyObject* print_str(PyObject* self, PyObject* args) { char* content;
if (!PyArg_ParseTuple(args, "s", &content)) return NULL; printf("we want to print %s\n", content); Py_INCREF(Py_None); return Py_None;}
static PyMethodDef NewinterMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} };
static struct PyModuleDef newintermodule = { PyModuleDef_HEAD_INIT, "newinter", NULL, -1, NewinterMethods };
pthread_t tid;
void *thr_func(void *arg) { Py_NewInterpreter(); return ((void *)0); }
PyMODINIT_FUNC PyInit_newinter(void) { PyObject *m;
m = PyModule_Create(&newintermodule);#if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) PyEval_InitThreads(); #endif
int err = pthread_create(&tid, NULL, thr_func, NULL); if(err != 0) { printf("can't create thread\n"); return NULL; } else printf("create new thread\n"); return m;}
And for test, a very simple Python script is used as below: import newinter newinter.print_str("test")
Sincerely, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道：
Hi Kun,
Can you please try to provide a short C program reproducing your issue? Sadly, I cannot see your attachement. Try to copy it directly in the email body.
Victor
On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote:
Hello,
I'm working on a C extension interface of Python. I want to create a new interpreter by using the function Py_NewInterpreter() in a new thread, which is created by pthread_create (my test files are in attachment), but there are always errors when calling Py_NewInterpreter() such as "failed: object already tracked by the garbage collector".
I would like to ask how to solve the problem and create a new interpreter in multi thread in Python C extension?
Sincerely, Kun
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens?
I suppose that I can call Py_EndInterpreter() manually before Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道：
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thank you for handling my problem.
In fact, in my actually processed C interface, I need to call a C
extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL.
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道：
Aha, I reproduce your issue: https://bugs.python.org/issue43793
Currently, I understand that you must first acquire the GIL using PyGILState_Ensure()/PyGILState_Release() in the newly created thread to be able to call Py_NewInterpreter(). See: https://docs.python.org/dev/c-api/init.html#non-python-created-threads
Maybe tomorrow it will be possible to call Py_NewInterpreter() on a newly created thread which has no thread state.
Victor
On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thanks for your response. I'm sorry that the attachment is not
Anyway, I copy the C program as you can see below:
#include <stdio.h> #include <string.h>
#include <pthread.h>
#define PY_SSIZE_T_CLEAN #include <Python.h>
static PyObject* print_str(PyObject* self, PyObject* args) { char* content;
if (!PyArg_ParseTuple(args, "s", &content)) return NULL; printf("we want to print %s\n", content); Py_INCREF(Py_None); return Py_None;}
static PyMethodDef NewinterMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} };
static struct PyModuleDef newintermodule = { PyModuleDef_HEAD_INIT, "newinter", NULL, -1, NewinterMethods };
pthread_t tid;
void *thr_func(void *arg) { Py_NewInterpreter(); return ((void *)0); }
PyMODINIT_FUNC PyInit_newinter(void) { PyObject *m;
m = PyModule_Create(&newintermodule);#if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) PyEval_InitThreads(); #endif
int err = pthread_create(&tid, NULL, thr_func, NULL); if(err != 0) { printf("can't create thread\n"); return NULL; } else printf("create new thread\n"); return m;}
And for test, a very simple Python script is used as below: import newinter newinter.print_str("test")
Sincerely, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道：
Hi Kun,
Can you please try to provide a short C program reproducing your issue? Sadly, I cannot see your attachement. Try to copy it directly in the email body.
Victor
On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com>
wrote:
Hello,
I'm working on a C extension interface of Python. I want to create
a new
interpreter by using the function Py_NewInterpreter() in a new
available. thread,
which is created by pthread_create (my test files are in attachment), but there are always errors when calling Py_NewInterpreter() such as "failed: object already tracked by the garbage collector".
I would like to ask how to solve the problem and create a new interpreter in multi thread in Python C extension?
Sincerely, Kun
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
As I see in the document, this function can destroy all sub-interpreters
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens?
I suppose that I can call Py_EndInterpreter() manually before Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道：
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thank you for handling my problem.
In fact, in my actually processed C interface, I need to call a C extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL.
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道：
Aha, I reproduce your issue: https://bugs.python.org/issue43793
Currently, I understand that you must first acquire the GIL using PyGILState_Ensure()/PyGILState_Release() in the newly created thread to be able to call Py_NewInterpreter(). See: https://docs.python.org/dev/c-api/init.html#non-python-created-threads
Maybe tomorrow it will be possible to call Py_NewInterpreter() on a newly created thread which has no thread state.
Victor
On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thanks for your response. I'm sorry that the attachment is not available. Anyway, I copy the C program as you can see below:
#include <stdio.h> #include <string.h>
#include <pthread.h>
#define PY_SSIZE_T_CLEAN #include <Python.h>
static PyObject* print_str(PyObject* self, PyObject* args) { char* content;
if (!PyArg_ParseTuple(args, "s", &content)) return NULL; printf("we want to print %s\n", content); Py_INCREF(Py_None); return Py_None;}
static PyMethodDef NewinterMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} };
static struct PyModuleDef newintermodule = { PyModuleDef_HEAD_INIT, "newinter", NULL, -1, NewinterMethods };
pthread_t tid;
void *thr_func(void *arg) { Py_NewInterpreter(); return ((void *)0); }
PyMODINIT_FUNC PyInit_newinter(void) { PyObject *m;
m = PyModule_Create(&newintermodule);#if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) PyEval_InitThreads(); #endif
int err = pthread_create(&tid, NULL, thr_func, NULL); if(err != 0) { printf("can't create thread\n"); return NULL; } else printf("create new thread\n"); return m;}
And for test, a very simple Python script is used as below: import newinter newinter.print_str("test")
Sincerely, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道：
Hi Kun,
Can you please try to provide a short C program reproducing your issue? Sadly, I cannot see your attachement. Try to copy it directly in the email body.
Victor
On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote: > > Hello, > > I'm working on a C extension interface of Python. I want to create a new > interpreter by using the function Py_NewInterpreter() in a new thread, > which is created by pthread_create (my test files are in attachment), but > there are always errors when calling Py_NewInterpreter() such as "failed: > object already tracked by the garbage collector". > > I would like to ask how to solve the problem and create a new interpreter > in multi thread in Python C extension? > > Sincerely, > Kun > _______________________________________________ > capi-sig mailing list -- capi-sig@python.org > To unsubscribe send an email to capi-sig-leave@python.org > https://mail.python.org/mailman3/lists/capi-sig.python.org/ > Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
I saw this issue36225. Thanks a lot.
In fact, I tried to put Py_EndInterpreter() and all the other shutdown functions of my code in freefun m_free of struct PyModuleDef, but it seems that m_free is not called in the end. So I would like to ask which function should we call for finalizing?
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 下午2:04写道：
As I see in the document, this function can destroy all sub-interpreters
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling
Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the
program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens?
I suppose that I can call Py_EndInterpreter() manually before
Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道：
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thank you for handling my problem.
In fact, in my actually processed C interface, I need to call a C
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道：
Aha, I reproduce your issue: https://bugs.python.org/issue43793
Currently, I understand that you must first acquire the GIL using PyGILState_Ensure()/PyGILState_Release() in the newly created thread to be able to call Py_NewInterpreter(). See:
https://docs.python.org/dev/c-api/init.html#non-python-created-threads
Maybe tomorrow it will be possible to call Py_NewInterpreter() on a newly created thread which has no thread state.
Victor
On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com>
wrote:
Hi Victor,
Thanks for your response. I'm sorry that the attachment is not
available.
Anyway, I copy the C program as you can see below:
#include <stdio.h> #include <string.h>
#include <pthread.h>
#define PY_SSIZE_T_CLEAN #include <Python.h>
static PyObject* print_str(PyObject* self, PyObject* args) { char* content;
if (!PyArg_ParseTuple(args, "s", &content)) return NULL; printf("we want to print %s\n", content); Py_INCREF(Py_None); return Py_None;}
static PyMethodDef NewinterMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} };
static struct PyModuleDef newintermodule = { PyModuleDef_HEAD_INIT, "newinter", NULL, -1, NewinterMethods };
pthread_t tid;
void *thr_func(void *arg) { Py_NewInterpreter(); return ((void *)0); }
PyMODINIT_FUNC PyInit_newinter(void) { PyObject *m;
m = PyModule_Create(&newintermodule);#if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) PyEval_InitThreads(); #endif
int err = pthread_create(&tid, NULL, thr_func, NULL); if(err != 0) { printf("can't create thread\n"); return NULL; } else printf("create new thread\n"); return m;}
And for test, a very simple Python script is used as below: import newinter newinter.print_str("test")
Sincerely, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道： > > Hi Kun, > > Can you please try to provide a short C program reproducing your > issue? Sadly, I cannot see your attachement. Try to copy it
> in the email body. > > Victor > > On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote: > > > > Hello, > > > > I'm working on a C extension interface of Python. I want to create a new > > interpreter by using the function Py_NewInterpreter() in a new
extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL. directly thread,
> > which is created by pthread_create (my test files are in attachment), but > > there are always errors when calling Py_NewInterpreter() such as "failed: > > object already tracked by the garbage collector". > > > > I would like to ask how to solve the problem and create a new interpreter > > in multi thread in Python C extension? > > > > Sincerely, > > Kun > > _______________________________________________ > > capi-sig mailing list -- capi-sig@python.org > > To unsubscribe send an email to capi-sig-leave@python.org > > https://mail.python.org/mailman3/lists/capi-sig.python.org/ > > Member address: vstinner@python.org > > > > -- > Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
m_free is called if your C extension uses the multi-phase initialization API (PEP 489), is it the case?
Something like that:
static struct PyModuleDef _abcmodule = { PyModuleDef_HEAD_INIT, .m_name = "_abc", .m_doc = _abc__doc__, .m_size = sizeof(_abcmodule_state), .m_methods = _abcmodule_methods, .m_slots = _abcmodule_slots, .m_traverse = _abcmodule_traverse, .m_clear = _abcmodule_clear, .m_free = _abcmodule_free, };
PyMODINIT_FUNC PyInit__abc(void) { return PyModuleDef_Init(&_abcmodule); }
See the "Multiphase initialization API" section of my article: https://vstinner.github.io/isolate-subinterpreters.html
Victor
On Mon, Apr 12, 2021 at 2:59 PM Kun He <hekun19890913@gmail.com> wrote:
I saw this issue36225. Thanks a lot.
In fact, I tried to put Py_EndInterpreter() and all the other shutdown functions of my code in freefun m_free of struct PyModuleDef, but it seems that m_free is not called in the end. So I would like to ask which function should we call for finalizing?
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 下午2:04写道：
As I see in the document, this function can destroy all sub-interpreters
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens?
I suppose that I can call Py_EndInterpreter() manually before Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道：
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
Thank you for handling my problem.
In fact, in my actually processed C interface, I need to call a C extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL.
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道：
Aha, I reproduce your issue: https://bugs.python.org/issue43793
Currently, I understand that you must first acquire the GIL using PyGILState_Ensure()/PyGILState_Release() in the newly created thread to be able to call Py_NewInterpreter(). See: https://docs.python.org/dev/c-api/init.html#non-python-created-threads
Maybe tomorrow it will be possible to call Py_NewInterpreter() on a newly created thread which has no thread state.
Victor
On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote: > > Hi Victor, > > Thanks for your response. I'm sorry that the attachment is not available. > Anyway, I copy the C program as you can see below: > > #include <stdio.h> > #include <string.h> > > #include <pthread.h> > > #define PY_SSIZE_T_CLEAN > #include <Python.h> > > static PyObject* print_str(PyObject* self, PyObject* args) > { > char* content; > > if (!PyArg_ParseTuple(args, "s", &content)) > return NULL; > > printf("we want to print %s\n", content); > > Py_INCREF(Py_None); > return Py_None; > } > > static PyMethodDef NewinterMethods[] = { > {"print_str", print_str, METH_VARARGS, "print the input string"}, > {NULL, NULL, 0, NULL} > }; > > static struct PyModuleDef newintermodule = { > PyModuleDef_HEAD_INIT, > "newinter", > NULL, > -1, > NewinterMethods > }; > > pthread_t tid; > > > void *thr_func(void *arg) > { > Py_NewInterpreter(); > return ((void *)0); > } > > PyMODINIT_FUNC > PyInit_newinter(void) > { > PyObject *m; > > m = PyModule_Create(&newintermodule); > > #if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) > PyEval_InitThreads(); > #endif > > int err = pthread_create(&tid, NULL, thr_func, NULL); > if(err != 0) > { > printf("can't create thread\n"); > return NULL; > } > else > printf("create new thread\n"); > > return m; > } > > And for test, a very simple Python script is used as below: > import newinter > newinter.print_str("test") > > > Sincerely, > Kun > > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道： >> >> Hi Kun, >> >> Can you please try to provide a short C program reproducing your >> issue? Sadly, I cannot see your attachement. Try to copy it directly >> in the email body. >> >> Victor >> >> On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote: >> > >> > Hello, >> > >> > I'm working on a C extension interface of Python. I want to create a new >> > interpreter by using the function Py_NewInterpreter() in a new thread, >> > which is created by pthread_create (my test files are in attachment), but >> > there are always errors when calling Py_NewInterpreter() such as "failed: >> > object already tracked by the garbage collector". >> > >> > I would like to ask how to solve the problem and create a new interpreter >> > in multi thread in Python C extension? >> > >> > Sincerely, >> > Kun >> > _______________________________________________ >> > capi-sig mailing list -- capi-sig@python.org >> > To unsubscribe send an email to capi-sig-leave@python.org >> > https://mail.python.org/mailman3/lists/capi-sig.python.org/ >> > Member address: vstinner@python.org >> >> >> >> -- >> Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
Hi Victor,
I don't think that my C extension uses the multi-phase initialization. Instead of calling PyModuleDef_Init(&_abcmodule), I call PyModule_Create(&_abcmodule). If the C extension uses single-phase initialization, m_free will not be called? And m_free is indeed for finalizing?
I did some tests. (In all tests I used PyModule_Create()) I defined "myFree" function just to see whether m_free is called or not:
static void myFree(void *arg) { printf("it's the m_free function\n"); }
static struct PyModuleDef myModulemodule = { PyModuleDef_HEAD_INIT, .m_name = "myModule", .m_doc = NULL, .m_size = -1, .m_methods = MyModuleMethods, .m_slots = NULL, .m_traverse = NULL, .m_clear = NULL, .m_free = myFree };
PyMODINIT_FUNC PyInit_myModule(void) { return PyModule_Create(&myModulemodule); }
"it's the m_free function" is printed in the end. I think that means m_free is called.
Then I did some tests in my code, I found that if the reference count does not reach zero in the end, m_free will not be called.
So, no matter which initialization I use, PyModuleDef_Init() or PyModule_Create(), only make sure that the reference count reaches zero in the end, m_free will be called. Am I right??
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月13日周二 下午4:15写道：
m_free is called if your C extension uses the multi-phase initialization API (PEP 489), is it the case?
Something like that:
static struct PyModuleDef _abcmodule = { PyModuleDef_HEAD_INIT, .m_name = "_abc", .m_doc = _abc__doc__, .m_size = sizeof(_abcmodule_state), .m_methods = _abcmodule_methods, .m_slots = _abcmodule_slots, .m_traverse = _abcmodule_traverse, .m_clear = _abcmodule_clear, .m_free = _abcmodule_free, };
PyMODINIT_FUNC PyInit__abc(void) { return PyModuleDef_Init(&_abcmodule); }
See the "Multiphase initialization API" section of my article: https://vstinner.github.io/isolate-subinterpreters.html
Victor
On Mon, Apr 12, 2021 at 2:59 PM Kun He <hekun19890913@gmail.com> wrote:
I saw this issue36225. Thanks a lot.
In fact, I tried to put Py_EndInterpreter() and all the other shutdown
functions of my code in freefun m_free of struct PyModuleDef, but it seems that m_free is not called in the end. So I would like to ask which function should we call for finalizing?
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 下午2:04写道：
As I see in the document, this function can destroy all
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He <hekun19890913@gmail.com>
wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling
Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the
I suppose that I can call Py_EndInterpreter() manually before
Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道：
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com>
wrote:
Hi Victor,
Thank you for handling my problem.
In fact, in my actually processed C interface, I need to call a C
extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL.
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道： > > Aha, I reproduce your issue: https://bugs.python.org/issue43793 > > Currently, I understand that you must first acquire the GIL using > PyGILState_Ensure()/PyGILState_Release() in the newly created
> to be able to call Py_NewInterpreter(). See: > https://docs.python.org/dev/c-api/init.html#non-python-created-threads > > Maybe tomorrow it will be possible to call Py_NewInterpreter() on a > newly created thread which has no thread state. > > Victor > > On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote: > > > > Hi Victor, > > > > Thanks for your response. I'm sorry that the attachment is not available. > > Anyway, I copy the C program as you can see below: > > > > #include <stdio.h> > > #include <string.h> > > > > #include <pthread.h> > > > > #define PY_SSIZE_T_CLEAN > > #include <Python.h> > > > > static PyObject* print_str(PyObject* self, PyObject* args) > > { > > char* content; > > > > if (!PyArg_ParseTuple(args, "s", &content)) > > return NULL; > > > > printf("we want to print %s\n", content); > > > > Py_INCREF(Py_None); > > return Py_None; > > } > > > > static PyMethodDef NewinterMethods[] = { > > {"print_str", print_str, METH_VARARGS, "print the input string"}, > > {NULL, NULL, 0, NULL} > > }; > > > > static struct PyModuleDef newintermodule = { > > PyModuleDef_HEAD_INIT, > > "newinter", > > NULL, > > -1, > > NewinterMethods > > }; > > > > pthread_t tid; > > > > > > void *thr_func(void *arg) > > { > > Py_NewInterpreter(); > > return ((void *)0); > > } > > > > PyMODINIT_FUNC > > PyInit_newinter(void) > > { > > PyObject *m; > > > > m = PyModule_Create(&newintermodule); > > > > #if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) > > PyEval_InitThreads(); > > #endif > > > > int err = pthread_create(&tid, NULL, thr_func, NULL); > > if(err != 0) > > { > > printf("can't create thread\n"); > > return NULL; > > } > > else > > printf("create new thread\n"); > > > > return m; > > } > > > > And for test, a very simple Python script is used as below: > > import newinter > > newinter.print_str("test") > > > > > > Sincerely, > > Kun > > > > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道： > >> > >> Hi Kun, > >> > >> Can you please try to provide a short C program reproducing your > >> issue? Sadly, I cannot see your attachement. Try to copy it
sub-interpreters program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens? thread directly
> >> in the email body. > >> > >> Victor > >> > >> On Fri, Apr 9, 2021 at 12:09 PM Kun He < hekun19890913@gmail.com> wrote: > >> > > >> > Hello, > >> > > >> > I'm working on a C extension interface of Python. I want to create a new > >> > interpreter by using the function Py_NewInterpreter() in a new thread, > >> > which is created by pthread_create (my test files are in attachment), but > >> > there are always errors when calling Py_NewInterpreter() such as "failed: > >> > object already tracked by the garbage collector". > >> > > >> > I would like to ask how to solve the problem and create a new interpreter > >> > in multi thread in Python C extension? > >> > > >> > Sincerely, > >> > Kun > >> > _______________________________________________ > >> > capi-sig mailing list -- capi-sig@python.org > >> > To unsubscribe send an email to capi-sig-leave@python.org > >> > https://mail.python.org/mailman3/lists/capi-sig.python.org/ > >> > Member address: vstinner@python.org > >> > >> > >> > >> -- > >> Night gathers, and now my watch begins. It shall not end until my death. > > > > -- > Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
To test your C extension, try to import it, then delete it. And then import it again.
Example:
import sys
# load the C extension once import _abc _abc = None del sys.modules['_abc']
# reload the C extension import _abc _abc = None del sys.modules['_abc']
Last time I checked, it was basically not possible to unload a C extension written using the legacy C API. You should use the multi-phase initialization API if you care about subinterpreters.
Victor
On Tue, Apr 13, 2021 at 7:11 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
I don't think that my C extension uses the multi-phase initialization. Instead of calling PyModuleDef_Init(&_abcmodule), I call PyModule_Create(&_abcmodule). If the C extension uses single-phase initialization, m_free will not be called? And m_free is indeed for finalizing?
I did some tests. (In all tests I used PyModule_Create()) I defined "myFree" function just to see whether m_free is called or not:
static void myFree(void *arg) { printf("it's the m_free function\n"); }
static struct PyModuleDef myModulemodule = { PyModuleDef_HEAD_INIT, .m_name = "myModule", .m_doc = NULL, .m_size = -1, .m_methods = MyModuleMethods, .m_slots = NULL, .m_traverse = NULL, .m_clear = NULL, .m_free = myFree };
PyMODINIT_FUNC PyInit_myModule(void) { return PyModule_Create(&myModulemodule); }
"it's the m_free function" is printed in the end. I think that means m_free is called.
Then I did some tests in my code, I found that if the reference count does not reach zero in the end, m_free will not be called.
So, no matter which initialization I use, PyModuleDef_Init() or PyModule_Create(), only make sure that the reference count reaches zero in the end, m_free will be called. Am I right??
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月13日周二 下午4:15写道：
m_free is called if your C extension uses the multi-phase initialization API (PEP 489), is it the case?
Something like that:
static struct PyModuleDef _abcmodule = { PyModuleDef_HEAD_INIT, .m_name = "_abc", .m_doc = _abc__doc__, .m_size = sizeof(_abcmodule_state), .m_methods = _abcmodule_methods, .m_slots = _abcmodule_slots, .m_traverse = _abcmodule_traverse, .m_clear = _abcmodule_clear, .m_free = _abcmodule_free, };
PyMODINIT_FUNC PyInit__abc(void) { return PyModuleDef_Init(&_abcmodule); }
See the "Multiphase initialization API" section of my article: https://vstinner.github.io/isolate-subinterpreters.html
Victor
On Mon, Apr 12, 2021 at 2:59 PM Kun He <hekun19890913@gmail.com> wrote:
I saw this issue36225. Thanks a lot.
In fact, I tried to put Py_EndInterpreter() and all the other shutdown functions of my code in freefun m_free of struct PyModuleDef, but it seems that m_free is not called in the end. So I would like to ask which function should we call for finalizing?
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 下午2:04写道：
As I see in the document, this function can destroy all sub-interpreters
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens?
I suppose that I can call Py_EndInterpreter() manually before Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道：
You need to call PyGILState_Ensure()/PyGILState_Release() around the the call to the C extension function. These function creates a Python thread state to the newly created thread.
> So when calling Py_NewInterpreter() I don't have the GIL.
Sorry, but this is wrong. All functions of the C API require to hold the GIL. If it's not the case, it's a bug in your code.
Victor
On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com> wrote: > > Hi Victor, > > Thank you for handling my problem. > > In fact, in my actually processed C interface, I need to call a C extension function which calls Py_NewInterpreter() in new thread, and I put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL. > > Best regards, > Kun > > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道： >> >> Aha, I reproduce your issue: https://bugs.python.org/issue43793 >> >> Currently, I understand that you must first acquire the GIL using >> PyGILState_Ensure()/PyGILState_Release() in the newly created thread >> to be able to call Py_NewInterpreter(). See: >> https://docs.python.org/dev/c-api/init.html#non-python-created-threads >> >> Maybe tomorrow it will be possible to call Py_NewInterpreter() on a >> newly created thread which has no thread state. >> >> Victor >> >> On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote: >> > >> > Hi Victor, >> > >> > Thanks for your response. I'm sorry that the attachment is not available. >> > Anyway, I copy the C program as you can see below: >> > >> > #include <stdio.h> >> > #include <string.h> >> > >> > #include <pthread.h> >> > >> > #define PY_SSIZE_T_CLEAN >> > #include <Python.h> >> > >> > static PyObject* print_str(PyObject* self, PyObject* args) >> > { >> > char* content; >> > >> > if (!PyArg_ParseTuple(args, "s", &content)) >> > return NULL; >> > >> > printf("we want to print %s\n", content); >> > >> > Py_INCREF(Py_None); >> > return Py_None; >> > } >> > >> > static PyMethodDef NewinterMethods[] = { >> > {"print_str", print_str, METH_VARARGS, "print the input string"}, >> > {NULL, NULL, 0, NULL} >> > }; >> > >> > static struct PyModuleDef newintermodule = { >> > PyModuleDef_HEAD_INIT, >> > "newinter", >> > NULL, >> > -1, >> > NewinterMethods >> > }; >> > >> > pthread_t tid; >> > >> > >> > void *thr_func(void *arg) >> > { >> > Py_NewInterpreter(); >> > return ((void *)0); >> > } >> > >> > PyMODINIT_FUNC >> > PyInit_newinter(void) >> > { >> > PyObject *m; >> > >> > m = PyModule_Create(&newintermodule); >> > >> > #if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) >> > PyEval_InitThreads(); >> > #endif >> > >> > int err = pthread_create(&tid, NULL, thr_func, NULL); >> > if(err != 0) >> > { >> > printf("can't create thread\n"); >> > return NULL; >> > } >> > else >> > printf("create new thread\n"); >> > >> > return m; >> > } >> > >> > And for test, a very simple Python script is used as below: >> > import newinter >> > newinter.print_str("test") >> > >> > >> > Sincerely, >> > Kun >> > >> > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道： >> >> >> >> Hi Kun, >> >> >> >> Can you please try to provide a short C program reproducing your >> >> issue? Sadly, I cannot see your attachement. Try to copy it directly >> >> in the email body. >> >> >> >> Victor >> >> >> >> On Fri, Apr 9, 2021 at 12:09 PM Kun He <hekun19890913@gmail.com> wrote: >> >> > >> >> > Hello, >> >> > >> >> > I'm working on a C extension interface of Python. I want to create a new >> >> > interpreter by using the function Py_NewInterpreter() in a new thread, >> >> > which is created by pthread_create (my test files are in attachment), but >> >> > there are always errors when calling Py_NewInterpreter() such as "failed: >> >> > object already tracked by the garbage collector". >> >> > >> >> > I would like to ask how to solve the problem and create a new interpreter >> >> > in multi thread in Python C extension? >> >> > >> >> > Sincerely, >> >> > Kun >> >> > _______________________________________________ >> >> > capi-sig mailing list -- capi-sig@python.org >> >> > To unsubscribe send an email to capi-sig-leave@python.org >> >> > https://mail.python.org/mailman3/lists/capi-sig.python.org/ >> >> > Member address: vstinner@python.org >> >> >> >> >> >> >> >> -- >> >> Night gathers, and now my watch begins. It shall not end until my death. >> >> >> >> -- >> Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
I tried to test a very simple module:
static void myFree(void *arg) { printf("it's the free function\n"); }
static struct PyModuleDef myModulemodule = { PyModuleDef_HEAD_INIT, .m_name = "myModule", .m_doc = NULL, .m_size = -1, .m_methods = NULL, .m_slots = NULL, .m_traverse = NULL, .m_clear = NULL, .m_free = myFree };
PyMODINIT_FUNC PyInit_myModule(void) { return PyModule_Create(&myModulemodule); }
And in Python script, I did what you proposed:
import sys import myModule print(sys.modules) myModule = None del sys.modules['myModule'] print(sys.modules)
import myModule print(sys.modules) myModule = None del sys.modules['myModule'] print(sys.modules)
In my test, the module can be deleted. I check sys.modules after "import myModule", myModule is added to sys.modules, and after deleting module, it is indeed deleted from sys.modules. The only different is that the message "it's the free function" of m_free is not printed the second time.
I tried to add a simple method in the module: static PyObject* print_str(PyObject* self, PyObject* args) { char* content; if (!PyArg_ParseTuple(args, "s", &content)) return NULL; printf("we want to print %s\n", content); Py_INCREF(Py_None); return Py_None; }
static PyMethodDef MyModuleMethods[] = { {"print_str", print_str, METH_VARARGS, "print the input string"}, {NULL, NULL, 0, NULL} }; The message "it's the free function" of m_free is not printed at all, but the module can be deleted as well.
Is it normal? Or I did something wrong?
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月13日周二 下午7:48写道：
To test your C extension, try to import it, then delete it. And then import it again.
Example:
import sys
# load the C extension once import _abc _abc = None del sys.modules['_abc']
# reload the C extension import _abc _abc = None del sys.modules['_abc']
Last time I checked, it was basically not possible to unload a C extension written using the legacy C API. You should use the multi-phase initialization API if you care about subinterpreters.
Victor
On Tue, Apr 13, 2021 at 7:11 PM Kun He <hekun19890913@gmail.com> wrote:
Hi Victor,
I don't think that my C extension uses the multi-phase initialization.
Instead of calling PyModuleDef_Init(&_abcmodule), I call PyModule_Create(&_abcmodule). If the C extension uses single-phase initialization, m_free will not be called? And m_free is indeed for finalizing?
I did some tests. (In all tests I used PyModule_Create()) I defined "myFree" function just to see whether m_free is called or not:
static void myFree(void *arg) { printf("it's the m_free function\n"); }
static struct PyModuleDef myModulemodule = { PyModuleDef_HEAD_INIT, .m_name = "myModule", .m_doc = NULL, .m_size = -1, .m_methods = MyModuleMethods, .m_slots = NULL, .m_traverse = NULL, .m_clear = NULL, .m_free = myFree };
PyMODINIT_FUNC PyInit_myModule(void) { return PyModule_Create(&myModulemodule); }
"it's the m_free function" is printed in the end. I think that means
m_free is called.
Then I did some tests in my code, I found that if the reference count
does not reach zero in the end, m_free will not be called.
So, no matter which initialization I use, PyModuleDef_Init() or
PyModule_Create(), only make sure that the reference count reaches zero in the end, m_free will be called. Am I right??
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月13日周二 下午4:15写道：
m_free is called if your C extension uses the multi-phase initialization API (PEP 489), is it the case?
Something like that:
static struct PyModuleDef _abcmodule = { PyModuleDef_HEAD_INIT, .m_name = "_abc", .m_doc = _abc__doc__, .m_size = sizeof(_abcmodule_state), .m_methods = _abcmodule_methods, .m_slots = _abcmodule_slots, .m_traverse = _abcmodule_traverse, .m_clear = _abcmodule_clear, .m_free = _abcmodule_free, };
PyMODINIT_FUNC PyInit__abc(void) { return PyModuleDef_Init(&_abcmodule); }
See the "Multiphase initialization API" section of my article: https://vstinner.github.io/isolate-subinterpreters.html
Victor
On Mon, Apr 12, 2021 at 2:59 PM Kun He <hekun19890913@gmail.com> wrote:
I saw this issue36225. Thanks a lot.
In fact, I tried to put Py_EndInterpreter() and all the other
Best, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 下午2:04写道：
As I see in the document, this function can destroy all
sub-interpreters
Sadly, I don't think that it's the case. You must end each sub-interpreter manually. https://bugs.python.org/issue36225
Victor
On Mon, Apr 12, 2021 at 12:38 PM Kun He <hekun19890913@gmail.com>
wrote:
Hi Victor,
I added PyGILState_Ensure()/PyGILState_Release() when calling
Py_NewInterpreter(), new interpreters can be created.
But during the finalizing state, Py_FinalizeEx() is called by the
I suppose that I can call Py_EndInterpreter() manually before
Py_FinalizeEx() is called, but I don't know which function is used for finalizing (is there a function like PyInit_modulename but reverse)?
Best regards, Kun
Victor Stinner <vstinner@python.org> 于2021年4月12日周一 上午10:17写道： > > You need to call PyGILState_Ensure()/PyGILState_Release() around
> the call to the C extension function. These function creates a Python > thread state to the newly created thread. > > > So when calling Py_NewInterpreter() I don't have the GIL. > > Sorry, but this is wrong. All functions of the C API require to hold > the GIL. If it's not the case, it's a bug in your code. > > Victor > > On Fri, Apr 9, 2021 at 6:06 PM Kun He <hekun19890913@gmail.com> wrote: > > > > Hi Victor, > > > > Thank you for handling my problem. > > > > In fact, in my actually processed C interface, I need to call a C extension function which calls Py_NewInterpreter() in new thread, and I
> > > > Best regards, > > Kun > > > > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午3:57写道： > >> > >> Aha, I reproduce your issue: https://bugs.python.org/issue43793 > >> > >> Currently, I understand that you must first acquire the GIL using > >> PyGILState_Ensure()/PyGILState_Release() in the newly created
> >> to be able to call Py_NewInterpreter(). See: > >> https://docs.python.org/dev/c-api/init.html#non-python-created-threads > >> > >> Maybe tomorrow it will be possible to call Py_NewInterpreter() on a > >> newly created thread which has no thread state. > >> > >> Victor > >> > >> On Fri, Apr 9, 2021 at 2:26 PM Kun He <hekun19890913@gmail.com> wrote: > >> > > >> > Hi Victor, > >> > > >> > Thanks for your response. I'm sorry that the attachment is not available. > >> > Anyway, I copy the C program as you can see below: > >> > > >> > #include <stdio.h> > >> > #include <string.h> > >> > > >> > #include <pthread.h> > >> > > >> > #define PY_SSIZE_T_CLEAN > >> > #include <Python.h> > >> > > >> > static PyObject* print_str(PyObject* self, PyObject* args) > >> > { > >> > char* content; > >> > > >> > if (!PyArg_ParseTuple(args, "s", &content)) > >> > return NULL; > >> > > >> > printf("we want to print %s\n", content); > >> > > >> > Py_INCREF(Py_None); > >> > return Py_None; > >> > } > >> > > >> > static PyMethodDef NewinterMethods[] = { > >> > {"print_str", print_str, METH_VARARGS, "print the input string"}, > >> > {NULL, NULL, 0, NULL} > >> > }; > >> > > >> > static struct PyModuleDef newintermodule = { > >> > PyModuleDef_HEAD_INIT, > >> > "newinter", > >> > NULL, > >> > -1, > >> > NewinterMethods > >> > }; > >> > > >> > pthread_t tid; > >> > > >> > > >> > void *thr_func(void *arg) > >> > { > >> > Py_NewInterpreter(); > >> > return ((void *)0); > >> > } > >> > > >> > PyMODINIT_FUNC > >> > PyInit_newinter(void) > >> > { > >> > PyObject *m; > >> > > >> > m = PyModule_Create(&newintermodule); > >> > > >> > #if PY_MAJOR_VERSION < 3 || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION < 9) > >> > PyEval_InitThreads(); > >> > #endif > >> > > >> > int err = pthread_create(&tid, NULL, thr_func, NULL); > >> > if(err != 0) > >> > { > >> > printf("can't create thread\n"); > >> > return NULL; > >> > } > >> > else > >> > printf("create new thread\n"); > >> > > >> > return m; > >> > } > >> > > >> > And for test, a very simple Python script is used as below: > >> > import newinter > >> > newinter.print_str("test") > >> > > >> > > >> > Sincerely, > >> > Kun > >> > > >> > Victor Stinner <vstinner@python.org> 于2021年4月9日周五 下午2:01写道： > >> >> > >> >> Hi Kun, > >> >> > >> >> Can you please try to provide a short C program reproducing your > >> >> issue? Sadly, I cannot see your attachement. Try to copy it
shutdown functions of my code in freefun m_free of struct PyModuleDef, but it seems that m_free is not called in the end. So I would like to ask which function should we call for finalizing? program itself. As I see in the document, this function can destroy all sub-interpreters, but it is not the case in my code. There is an error "Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters". I don't know why it happens? the put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around this function. So when calling Py_NewInterpreter() I don't have the GIL. thread directly
> >> >> in the email body. > >> >> > >> >> Victor > >> >> > >> >> On Fri, Apr 9, 2021 at 12:09 PM Kun He < hekun19890913@gmail.com> wrote: > >> >> > > >> >> > Hello, > >> >> > > >> >> > I'm working on a C extension interface of Python. I want to create a new > >> >> > interpreter by using the function Py_NewInterpreter() in a new thread, > >> >> > which is created by pthread_create (my test files are in attachment), but > >> >> > there are always errors when calling Py_NewInterpreter() such as "failed: > >> >> > object already tracked by the garbage collector". > >> >> > > >> >> > I would like to ask how to solve the problem and create a new interpreter > >> >> > in multi thread in Python C extension? > >> >> > > >> >> > Sincerely, > >> >> > Kun > >> >> > _______________________________________________ > >> >> > capi-sig mailing list -- capi-sig@python.org > >> >> > To unsubscribe send an email to capi-sig-leave@python.org > >> >> > https://mail.python.org/mailman3/lists/capi-sig.python.org/ > >> >> > Member address: vstinner@python.org > >> >> > >> >> > >> >> > >> >> -- > >> >> Night gathers, and now my watch begins. It shall not end until my death. > >> > >> > >> > >> -- > >> Night gathers, and now my watch begins. It shall not end until my death. > > > > -- > Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
I'm afraid that I misunderstood your previous mail. When you said " it was basically not possible to unload a C extension.", it means that we can't delete the module from sys.modules (del sys.modules['_abc'])?
Kun
Victor Stinner <vstinner@python.org> 于2021年4月14日周三 上午12:35写道：
The message "it's the free function" of m_free is not printed at all, but the module can be deleted as well.
Maybe it's not deleted immediately and you must call gc.collect() to force a GC collection.
Victor
On Wed, 14 Apr 2021, 9:03 am Kun He, <hekun19890913@gmail.com> wrote:
I'm afraid that I misunderstood your previous mail. When you said " it was basically not possible to unload a C extension.", it means that we can't delete the module from sys.modules (del sys.modules['_abc'])?
Extension modules that use the legacy initialisation API have a hidden copy of their top level namespace created.
If you delete them from sys.modules, reimporting them will create a new module object, but it won't recreate the *contents* of the module - those will be taken from the hidden copy rather than executing the module initialisation again.
Only extension modules that use the newer multi-phase initialisation API can be truly unloaded and reloaded.
Cheers, Nick.
I see. Thank you for the explanation. I use multi-phase initialisation now, but there is another problem. Last time, I found that Py_FinalizeEx() can't destroy all sub-interpreters, Victor told me that I must end each sub-interpreter manually ( https://bugs.python.org/issue36225). I tried to do it, like this (I leave out the function m_methods and m_slots below):
sem_t sem_start; sem_t sem_created; sem_t sem_end; pthread_t tid;
static void newinterFree(void *arg) { sem_post(&sem_start); printf("it's the free function\n"); Py_BEGIN_ALLOW_THREADS sem_wait(&sem_end); Py_END_ALLOW_THREADS }
static struct PyModuleDef newintermodule = { PyModuleDef_HEAD_INIT, .m_name = "newinter", .m_doc = NULL, .m_size = 0, .m_methods = NewinterMethods, .m_slots = mySlots, .m_traverse = NULL, .m_clear = NULL, .m_free = newinterFree };
void *thr_func(void *arg) { PyThreadState *thread_state, *new_thread_state;
PyGILState_STATE state = PyGILState_Ensure();
thread_state = PyGILState_GetThisThreadState();
new_thread_state = Py_NewInterpreter();
PyThreadState_Swap(thread_state);
PyGILState_Release(state);
PyThreadState_Swap(new_thread_state);
sem_post(&sem_created);
sem_wait(&sem_start);
state = PyGILState_Ensure();
Py_EndInterpreter(new_thread_state);
PyGILState_Release(state);
sem_post(&sem_end);
return ((void *)0);}
PyMODINIT_FUNC PyInit_newinter(void) { PyObject *m; m= PyModuleDef_Init(&newintermodule);
sem_init(&sem_created, 0, 0);
sem_init(&sem_start, 0, 0);
sem_init(&sem_end, 0, 0);
pthread_create(&tid, NULL, thr_func, NULL);
Py_BEGIN_ALLOW_THREADS
sem_wait(&sem_created);
Py_END_ALLOW_THREADS
return m;}
but the program is blocked when acquiring the GIL for Py_EndInterpreter, since at that time, the thread is already exiting. If I check that pthread_exit is called or not, I get the backtrace: #0 __pthread_exit (value=value@entry=0x0) at pthread_exit.c:25 #1 0x00005555556dcfc1 in PyThread_exit_thread () at ../Python/thread_pthread.h:361 #2 0x000055555567f072 in take_gil (tstate=tstate@entry=0x7ffff0000b30) at ../Python/ceval_gil.h:224 #3 0x000055555567f7f5 in PyEval_RestoreThread (tstate=tstate@entry=0x7ffff0000b30) at ../Python/ceval.c:467 #4 0x00005555556c9c15 in PyGILState_Ensure () at ../Python/pystate.c:1378
I saw the issue https://bugs.python.org/issue36479, it seems very close to my problem. I would like to ask how can I solve this problem? Can we set a thread to non-daemon thread?
Best, Kun
Nick Coghlan <ncoghlan@gmail.com> 于2021年4月17日周六 下午4:34写道：
On Wed, 14 Apr 2021, 9:03 am Kun He, <hekun19890913@gmail.com> wrote:
I'm afraid that I misunderstood your previous mail. When you said " it was basically not possible to unload a C extension.", it means that we can't delete the module from sys.modules (del sys.modules['_abc'])?
Extension modules that use the legacy initialisation API have a hidden copy of their top level namespace created.
If you delete them from sys.modules, reimporting them will create a new module object, but it won't recreate the *contents* of the module - those will be taken from the hidden copy rather than executing the module initialisation again.
Only extension modules that use the newer multi-phase initialisation API can be truly unloaded and reloaded.
Cheers, Nick.
participants (4)
-
Kun He -
Nick Coghlan -
Victor Stinner -
William Pickard