Hello,
I want to embed Python in an application and use an API of the application from Python. The application uses a library that creates several threads and I the users of the application will write Python scripts handling this events.
The problem is that I having problems with threads. I saw that I have to PyEval_InitThreads and then PyThreadState_New and PyThreadState_Swap from the new thread but this way to solve the problem doesn't work for me because I need to let 2 or more threads run at the SAME time. The interpreter lock doesn't let them run at the same time so I'm looking for another solution. I saw Py_NewInterpreter and I tried to use it but it doesn't work if I don't keep the lock.
Can anyone help me to solve this issue or tell me 'Forget it!'?
Thanks on advance, Pablo Yabo
"Pablo Yabo" <pablo.yabo@gmail.com> writes:
The problem is that I having problems with threads. I saw that I have to PyEval_InitThreads and then PyThreadState_New and PyThreadState_Swap from the new thread but this way to solve the problem doesn't work for me because I need to let 2 or more threads run at the SAME time.
I don't know if this is still relevant, but it occurs to me that what you want might be accomplished. If your external library is thread-safe, and if you're only using Python to manipulate data managed by the library, you can simply Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS when entering the library. That way all calls from Python into the library will run with threading enabled, while the part spent in the interpreter (outside the library entry points) will continue to run single thread at a time. Python sources often use this pattern when dealing with IO-bound (e.g. the file and socket API) or CPU-bound code.
Is such a solution acceptable for you?
Hello Hrvoje,
Thank you for your response, it's difficult to get an answer on this subject.
I 'solved' the issue in that way. When I enter to any function of my library I release the interpreter lock and then I get it again. It's not the best solution but it's acceptable. The problem is that Python code cannot use any native wait or suspend function.
I wonder if why this issue is not fixed.
Regards, Pablo Yabo
On 9/4/07, Hrvoje Niksic <hniksic@xemacs.org> wrote:
"Pablo Yabo" <pablo.yabo@gmail.com> writes:
The problem is that I having problems with threads. I saw that I have to PyEval_InitThreads and then PyThreadState_New and PyThreadState_Swap from the new thread but this way to solve the problem doesn't work for me because I need to let 2 or more threads run at the SAME time.
I don't know if this is still relevant, but it occurs to me that what you want might be accomplished. If your external library is thread-safe, and if you're only using Python to manipulate data managed by the library, you can simply Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS when entering the library. That way all calls from Python into the library will run with threading enabled, while the part spent in the interpreter (outside the library entry points) will continue to run single thread at a time. Python sources often use this pattern when dealing with IO-bound (e.g. the file and socket API) or CPU-bound code.
Is such a solution acceptable for you?
"Pablo Yabo" <pablo.yabo@gmail.com> writes:
I 'solved' the issue in that way. When I enter to any function of my library I release the interpreter lock and then I get it again.
OK, that is what I proposed.
It's not the best solution but it's acceptable. The problem is that Python code cannot use any native wait or suspend function.
What do you mean by "native wait or suspend"? When entering blocking OS calls such as os.wait, or even the stdio IO code, Python tries to release the interpreter lock, to enable other threads to run at the same time.
I wonder if why this issue is not fixed.
Because it is very hard to fix without slowing down the interpreter for the single-thread case.
It's not the best solution but it's acceptable. The problem is that Python code cannot use any native wait or suspend function.
What do you mean by "native wait or suspend"? When entering blocking OS calls such as os.wait, or even the stdio IO code, Python tries to release the interpreter lock, to enable other threads to run at the same time.
Great, I didn´t know that. I mean that if a thread waits for a condition of another thread or sleeps but I think that they should release interpreter lock also.
Thanks for you help.
Pablo Yabo
"Pablo Yabo" <pablo.yabo@gmail.com> writes:
Great, I didn´t know that. I mean that if a thread waits for a condition of another thread or sleeps but I think that they should release interpreter lock also.
And you're right. After all, time.sleep is an easy and obvious target for releasing the GIL, and releasing it when waiting for a condition is necessary to avoid deadlock.
Hi, Id like to start a C/API cookbook and collect generic useful utility functions there that can be a resource for people who are newer to the C api.
This page dosnt really ave any sections that match a C/API cookbook, and you need a login to do so. http://wiki.python.org/moin/
We could start a wikibook here at http://en.wikibooks.org
A while back I started one for Blender3D/Python http://en.wikibooks.org/wiki/Blender_3D:_Blending_Into_Python/Cookbook
There are quite a few generic functions in Blender3D's python API that Id add in there and Im sure others here have some code they could contribute too.
- Cam
-- Campbell J Barton (ideasman42)
On 9/6/07, Campbell Barton <cbarton@metavr.com> wrote:
Hi, Id like to start a C/API cookbook and collect generic useful utility functions there that can be a resource for people who are newer to the C api.
Good idea.
This page dosnt really ave any sections that match a C/API cookbook, and you need a login to do so. http://wiki.python.org/moin/
The Python wiki does *not* require a login. I'd suggest that this should be the preferred place to start such an effort unless there's a compelling reason it won't work. I don't see one at this point.
-Fred
-- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Fred Drake wrote:
On 9/6/07, Campbell Barton <cbarton@metavr.com> wrote:
Hi, Id like to start a C/API cookbook and collect generic useful utility functions there that can be a resource for people who are newer to the C api.
Good idea.
This page dosnt really ave any sections that match a C/API cookbook, and you need a login to do so. http://wiki.python.org/moin/
The Python wiki does *not* require a login. I'd suggest that this should be the preferred place to start such an effort unless there's a compelling reason it won't work. I don't see one at this point.
-Fred
Started the cookbook here, http://wiki.python.org/moin/CAPI_CookBook
Its linked from this page http://wiki.python.org/moin/CategoryDocumentation
Need to come up with a prefix for helper functions like this... so far Iv left the prefix we had in blender.
Please add helper functions from your own API projects. ;)
Hi, this compiles fine on linux but on windows with MSVC8 it gives an error on this line.
&PyFloat_Type, /* tp_base */saying that it cant initialize because &PyFloat_Type is not a constant.
Has anyone else had this problem or know a workaround?
Heres teh pytype, source file is here http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.c?root=bf-blender&view=markup http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.h?root=bf-blender&view=markup
PyTypeObject BPyFloat_Type = { PyObject_HEAD_INIT(NULL) 0, "BPyFloat", sizeof(BPyFloatObject), 0, (destructor)BPyFloat_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ BPyFloat_methods, /* tp_methods */ 0, /* tp_members */ BPyFloat_getset, /* tp_getset */ &PyFloat_Type, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)BPyFloat_init, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ };
found the problem from re-reading pythons source. they have to be zero because some compilers complain,
The values need to be assigned in the module initializer before PyType_Ready is called,
xxsubtype.c uses a macro DEFERRED_ADDRESS so the value is documented but not compiled in.
Campbell Barton wrote:
Hi, this compiles fine on linux but on windows with MSVC8 it gives an error on this line.
&PyFloat_Type, /* tp_base */ saying that it cant initialize because &PyFloat_Type is not a constant.
Has anyone else had this problem or know a workaround?
Heres teh pytype, source file is here http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.c?root=bf-blender&view=markup http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.h?root=bf-blender&view=markup
The work-around for a similar problem with tp_new is described here:
http://docs.python.org/ext/dnt-basics.html
"On some platforms or compilers, we can't statically initialize a structure member with a function defined in another C module, so, instead, we'll assign the tp_new slot in the module initialization function just before calling PyType_Ready()"
So in BPyFloatType_Init(), you put the initialization for "BPyFloat_Type.tp_base = &PyFloat_Type;"
Ken
Campbell Barton wrote:
Hi, this compiles fine on linux but on windows with MSVC8 it gives an error on this line.
&PyFloat_Type, /* tp_base */ saying that it cant initialize because &PyFloat_Type is not a constant.
Has anyone else had this problem or know a workaround?
Heres teh pytype, source file is here http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.c?root=bf-blender&view=markup http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.h?root=bf-blender&view=markup
PyTypeObject BPyFloat_Type = { PyObject_HEAD_INIT(NULL) 0, "BPyFloat", sizeof(BPyFloatObject), 0, (destructor)BPyFloat_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ BPyFloat_methods, /* tp_methods */ 0, /* tp_members */ BPyFloat_getset, /* tp_getset */ &PyFloat_Type, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)BPyFloat_init, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */ };
capi-sig mailing list capi-sig@python.org http://mail.python.org/mailman/listinfo/capi-sig
participants (5)
-
Campbell Barton -
Fred Drake -
Hrvoje Niksic -
Ken Hughes -
Pablo Yabo