From paul at anilabsys.com Thu Jul 1 23:36:17 2010 From: paul at anilabsys.com (Paul Grunau) Date: Thu, 1 Jul 2010 14:36:17 -0700 (PDT) Subject: [capi-sig] intermittent failure of threading Message-ID: <282763.59015.qm@web302.biz.mail.mud.yahoo.com> I have a problem with threading using the Python/C API. I have an extension that implements a timer, and the C++ timer callback function calls a Python function. The relevant code looks like this: staticPyObject *timer_setmodname( PyObject *pSelf, PyObject *pArgs ) { char *b; PyArg_ParseTuple( pArgs, "s", &b ); mod = PyImport_ImportModule(b); if( mod == NULL ) { printf("Could not import %s\n",b); return Py_None; } modsetFlag = TRUE; return Py_None; } staticPyObject *timer_setprocname( PyObject *pSelf, PyObject *pArgs ) { char *b; if( !modsetFlag )return Py_None; PyArg_ParseTuple( pArgs, "s", &b ); attr = PyObject_GetAttrString(mod,b); if( attr == NULL ) { printf("Could not import %s\n",b); return Py_None; } attrsetFlag = TRUE; return Py_None; } staticvoid CALLBACK PeriodicTimer(UINT wTimerID, UINT msg, DWORD dwUser, DWORD dw1, DWORD dw2) { PyGILState_STATE pgs; pgs = PyGILState_Ensure(); if(attrsetFlag) { pres = PyObject_CallFunction(attr,NULL); if( pres == NULL )printf("CallFunction failed!\n"); } PyGILState_Release( pgs ); } The Python code that sets this up looks like this: fetimer.setmodname("Timeslice3") fetimer.setprocname("Timetester") print "\n Program Waiting for Time Slice" while True: time.sleep(0.010) and the module Timeslice3.py looks like this: #Timeslice3.py def Timetester(): pass When I run this stuff, it works fine for hundreds, often even thousands, of timer ticks (I?ve been testing with about thirty ticks per second, but it doesn?t matter ? it still crashes at ten or fewer ticks per second). Sometimes it runs for only a few seconds, sometimes for ten minutes or so. But it always eventually crashes Python. Usually it gives no error message. Sometimes, though, it does give an error message, but not always the same one. I?ve noted three that it has given in my testing so far: Fatal Python Error: This thread state must be current when releasing Fatal Python Error: PyThreadState_DeleteCurrent: no current tstate Fatal Python Error: PyEval_SaveThread: NULL tstate Can anybody help me make this code stable, so that it works all the time? I?m using Python 2.6.5 under Windows Vista, but it crashes under Windows XP as well. From hniksic at xemacs.org Fri Jul 2 08:37:07 2010 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Fri, 02 Jul 2010 08:37:07 +0200 Subject: [capi-sig] intermittent failure of threading In-Reply-To: <282763.59015.qm@web302.biz.mail.mud.yahoo.com> (Paul Grunau's message of "Thu, 1 Jul 2010 14:36:17 -0700 (PDT)") References: <282763.59015.qm@web302.biz.mail.mud.yahoo.com> Message-ID: <87vd8y9yp8.fsf@busola.homelinux.net> Paul Grunau writes: > char *b; > PyArg_ParseTuple( pArgs, "s", &b ); > mod = PyImport_ImportModule(b); > if( mod == NULL ) > { > printf("Could not import %s\n",b); > return Py_None; > } > modsetFlag = TRUE; > return Py_None; Python functions are expected to return a new reference, so you should Py_INCREF None (or any other returned Python object) before returning it. This applies to both setmodname and setprocname. > pres = PyObject_CallFunction(attr,NULL); > if( pres == NULL )printf("CallFunction failed!\n"); PyObject_CallFunction returns a new reference, so you should Py_DECREF pres in the case when it is not NULL and you no longer need it. Also, posting a complete code sample in C, that we can build and test ourselves, might help debugging the specific problem you are having, which may or may not be related to the above reference-counting errors. From mail at timgolden.me.uk Fri Jul 2 11:14:33 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 02 Jul 2010 10:14:33 +0100 Subject: [capi-sig] DECREF when using PyDict_SetItem... Message-ID: <4C2DADF9.7060002@timgolden.me.uk> I'm using PyDict_SetItemString to add a long from a C struct into a Python dictionary. The full code is in the usn2dict function in here: http://svn.timgolden.me.uk/extensions/change-journal/_usn.c but the sort of thing I'm talking about is this: PyDict_SetItemString ( dict, "RecordLength", PyLong_FromLongLong (usn_record->RecordLength) ); My understanding is that PyLong_FromLongLong passes its reference to my function, which then passes it to PyDict_Set... which INCREFs it. That means, I think, that my function should DECREF it before exit since I'm only creating it to store it in the dict. Is that correct? In other words, should my code really do this? record_length = PyLong_FromLongLong (usn_record->RecordLength); PyDict_SetItemString (dict, "RecordLength", record_length); Py_DECREF (record_length) Thanks TJG From stefan_ml at behnel.de Fri Jul 2 11:44:21 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 11:44:21 +0200 Subject: [capi-sig] DECREF when using PyDict_SetItem... In-Reply-To: <4C2DADF9.7060002@timgolden.me.uk> References: <4C2DADF9.7060002@timgolden.me.uk> Message-ID: <4C2DB4F5.6050000@behnel.de> Tim Golden, 02.07.2010 11:14: > I'm using PyDict_SetItemString to add a long from a C struct > into a Python dictionary. The full code is in the usn2dict > function in here: > > http://svn.timgolden.me.uk/extensions/change-journal/_usn.c > > but the sort of thing I'm talking about is this: > > PyDict_SetItemString ( > dict, > "RecordLength", > PyLong_FromLongLong (usn_record->RecordLength) > ); > > My understanding is that PyLong_FromLongLong passes its reference > to my function, which then passes it to PyDict_Set... which INCREFs > it. That means, I think, that my function should DECREF it before > exit since I'm only creating it to store it in the dict. Is that correct? > > In other words, should my code really do this? > > record_length = PyLong_FromLongLong (usn_record->RecordLength); > PyDict_SetItemString (dict, "RecordLength", record_length); > Py_DECREF (record_length) Yes. Stefan From hniksic at xemacs.org Fri Jul 2 12:02:38 2010 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Fri, 02 Jul 2010 12:02:38 +0200 Subject: [capi-sig] DECREF when using PyDict_SetItem... In-Reply-To: <4C2DADF9.7060002@timgolden.me.uk> (Tim Golden's message of "Fri, 02 Jul 2010 10:14:33 +0100") References: <4C2DADF9.7060002@timgolden.me.uk> Message-ID: <87wrtew69t.fsf@busola.homelinux.net> Tim Golden writes: > My understanding is that PyLong_FromLongLong passes its reference > to my function, which then passes it to PyDict_Set... which INCREFs > it. Your understanding is correct. Note that the same reasoning doesn't apply to, for instance, PyList_SetItem, which actually "steals" a reference to the value it receives. Fortunately, functions that behave in this fashion are very rare, and clearly documented to do so. > That means, I think, that my function should DECREF it before exit > since I'm only creating it to store it in the dict. Is that correct? > > In other words, should my code really do this? > > record_length = PyLong_FromLongLong (usn_record->RecordLength); > PyDict_SetItemString (dict, "RecordLength", record_length); > Py_DECREF (record_length) Yes, and don't forget to check for PyLong_FromLongLong returning NULL: if (!record_length) { Py_DECREF(dict); /* presumably */ return NULL; } (And of course the same goes for PyDict_SetItemString.) From mail at timgolden.me.uk Fri Jul 2 12:16:19 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 02 Jul 2010 11:16:19 +0100 Subject: [capi-sig] DECREF when using PyDict_SetItem... In-Reply-To: <87wrtew69t.fsf@busola.homelinux.net> References: <4C2DADF9.7060002@timgolden.me.uk> <87wrtew69t.fsf@busola.homelinux.net> Message-ID: <4C2DBC73.3070605@timgolden.me.uk> On 02/07/2010 11:02, Hrvoje Niksic wrote: > Tim Golden writes: > >> My understanding is that PyLong_FromLongLong passes its reference >> to my function, which then passes it to PyDict_Set... which INCREFs >> it. > > Your understanding is correct. > > Note that the same reasoning doesn't apply to, for instance, > PyList_SetItem, which actually "steals" a reference to the value it > receives. Fortunately, functions that behave in this fashion are very > rare, and clearly documented to do so. > >> That means, I think, that my function should DECREF it before exit >> since I'm only creating it to store it in the dict. Is that correct? >> >> In other words, should my code really do this? >> >> record_length = PyLong_FromLongLong (usn_record->RecordLength); >> PyDict_SetItemString (dict, "RecordLength", record_length); >> Py_DECREF (record_length) > > Yes, and don't forget to check for PyLong_FromLongLong returning NULL: > > if (!record_length) { > Py_DECREF(dict); /* presumably */ > return NULL; > } > > (And of course the same goes for PyDict_SetItemString.) Thanks for that, Hrvoje TJG From paul at anilabsys.com Mon Jul 5 20:36:05 2010 From: paul at anilabsys.com (Paul Grunau) Date: Mon, 5 Jul 2010 11:36:05 -0700 (PDT) Subject: [capi-sig] Threading failure Message-ID: <648128.84773.qm@web307.biz.mail.mud.yahoo.com> I asked this question here a few days ago, and someone suggested I ask again, providing all the code, so that it can be actually tried. So here it is, and I?m still stumped. Original post: I have a problem with threading using the Python/C API. I have an extension that implements a timer, and the C++ timer callback function calls a Python function. The code looks like this: // fetimer.cpp : Defines the exported functions for timer function. #define_WIN32_WINNT 0x502 #include"stdafx.h" #include"mmsystem.h" #include"python.h" #defineTARGET_RESOLUTION 1 UINT wTimerRes,fooval; BOOL LEDflag,timeFlag; BOOL timerActive = FALSE; BOOL timerSet = FALSE; BOOL funcsetFlag = FALSE; BOOL modsetFlag = FALSE; BOOL attrsetFlag = FALSE; UINT wTimerID,timeval,setVal = 2000; HANDLE porthandle; TIMECAPS tc; UINT userval,*pVal; PyObject *mod, *attr, *pargs, *pres; UINT SetTimerCallback( UINT wTimrID, UINT msInterval ); staticvoid CALLBACK PeriodicTimer(UINT wTimerID, UINT msg, DWORD dwUser, DWORD dw1, DWORD dw2); PyObject *ProcessTimer( void ); PyObject *pymod; PyObject *pattr; PyObject *foop; staticvoid (*pFunc)(UINT wTimerID, UINT msg, DWORD dwUser, DWORD dw1, DWORD dw2); staticPyObject *timer_setup( PyObject *pSelf, PyObject *pArgs ) { if (timeGetDevCaps(&tc, sizeof(TIMECAPS)) != TIMERR_NOERROR) { } wTimerRes = min(max(tc.wPeriodMin, TARGET_RESOLUTION), tc.wPeriodMax); timeBeginPeriod(wTimerRes); LEDflag = FALSE; timeFlag = FALSE; timerActive = FALSE; userval = 123; pVal = &userval; Py_Initialize(); fooval = 0; return Py_None; } staticPyObject *timer_start( PyObject *pSelf, PyObject *pArgs ) { int tval = 2000; if( timerActive ){ timeKillEvent( wTimerID ); } if( timerSet ){ tval = setVal; } UINT retval = SetTimerCallback( wTimerID, tval ); timerActive = TRUE; return Py_None; } UINT SetTimerCallback(UINT wTimrID, // sequencer data UINT msInterval) // event interval { wTimerID = timeSetEvent( msInterval, // delay wTimerRes, // resolution (global variable) (LPTIMECALLBACK)PeriodicTimer, // callback function wTimrID, // user data TIME_PERIODIC|TIME_CALLBACK_FUNCTION ); if(!wTimerID) { return 999; } else{ return 0; } } staticvoid CALLBACK PeriodicTimer(UINT wTimerID, UINT msg, DWORD dwUser, DWORD dw1, DWORD dw2) { PyGILState_STATE pgs; pgs = PyGILState_Ensure(); if(attrsetFlag) { pres = PyObject_CallFunction(attr,NULL); if( pres == NULL )printf("CallFunction failed!\n"); } PyGILState_Release( pgs ); } staticPyObject *timer_kill( PyObject *pSelf, PyObject *pArgs ) { timeKillEvent( wTimerID ); timerSet = FALSE; timerActive = FALSE; return Py_None; } staticPyObject *timer_settime( PyObject *pSelf, PyObject *pArgs ) { UINT t; PyArg_ParseTuple( pArgs, "i", &t ); setVal = t; timerSet = TRUE; return Py_None; } staticPyObject *timer_setmodname( PyObject *pSelf, PyObject *pArgs ) { char *b; PyArg_ParseTuple( pArgs, "s", &b ); mod = PyImport_ImportModule(b); if( mod == NULL ) { printf("Could not import %s\n",b); return Py_None; } modsetFlag = TRUE; return Py_None; } staticPyObject *timer_setprocname( PyObject *pSelf, PyObject *pArgs ) { char *b; if( !modsetFlag )return Py_None; PyArg_ParseTuple( pArgs, "s", &b ); attr = PyObject_GetAttrString(mod,b); if( attr == NULL ) { printf("Could not import %s\n",b); return Py_None; } attrsetFlag = TRUE; return Py_None; } staticPyMethodDef fetimer_methods[] = { {"timer", timer_setup, METH_VARARGS, "blah"}, {"start", timer_start, METH_VARARGS, "start timer" }, {"kill", timer_kill, METH_VARARGS, "kill timer" }, {"settime", timer_settime, METH_VARARGS, "set timer" }, {"setmodname", timer_setmodname, METH_VARARGS, "set module name" }, {"setprocname", timer_setprocname, METH_VARARGS, "set procedure name" }, {NULL, NULL} }; PyMODINIT_FUNC initfetimer(void) { Py_InitModule("fetimer", fetimer_methods); } The Python code (Timetest3.py) that sets this up looks like this: #Time Test 3.py import fetimer #import feserial import time Hit = 0 L = 0 fetimer.timer() fetimer.settime(30) fetimer.setmodname("Timeslice3") fetimer.setprocname("Timetester") #feserial.open(4) #feserial.dtr(0) print "\n Program Waiting for Time Slice" fetimer.start() while True: time.sleep(0.010) and the module Timeslice3.py looks like this: #Timeslice3.py def Timetester(): pass The application should run by entering "python timetest3.py" at the command prompt. When I run this stuff, it works fine for hundreds, often even thousands, of timer ticks (I?ve been testing with about thirty ticks per second, but it doesn?t matter ? it still crashes at ten or fewer ticks per second). Sometimes it runs for only a few seconds, sometimes for ten minutes or so. But it always eventually crashes Python. Usually it gives no error message. Sometimes, though, it does give an error message, but not always the same one. I?ve noted three that it has given in my testing so far: Fatal Python Error: This thread state must be current when releasing Fatal Python Error: PyThreadState_DeleteCurrent: no current tstate Fatal Python Error: PyEval_SaveThread: NULL tstate Can anybody help me make this code stable, so that it works all the time? I?m using Python 2.6.5 under Windows Vista, but it crashes under Windows XP as well. From chensahenc39hensa at yahoo.com Sat Jul 3 18:20:15 2010 From: chensahenc39hensa at yahoo.com (Gary) Date: Sun, 4 Jul 2010 00:20:15 +0800 Subject: [capi-sig] =?gb2312?b?U29mdHdhcmUgRGV2ZWxvcG1lbnQgLSBBcHBsaWNh?= =?gb2312?b?dGlvbiBEZXZlbG9wbWVudCAtIEdhbWUgRGV2ZWxvcG1lbnQ=?= Message-ID: <201007031120772.SM00648@dani> You are receiving this email because we wish you to use our cost-effective IT services. We are a China based Custom Software Application Development provider. We offer full cycle custom software programming services, from product idea, software development to support and enhancement. We employ a large pool of software engineers coming from different backgrounds. We are able to balance product development efforts and project duration to your business needs. Core Services: Custom Application Development Custom Software Development iPhone Application Development iPad Application Development Mobile Application Development Android Mobile Application Development Soft Product Development Game Design & Development Game Testing & Quality Assurance Cloud Computing Solutions: CRM Software Application Development ERP Software Application Development Enterprise Portal Solution Sales Force Automation Solution Game Design & Development Knowledge Management Solution Workflow Management Solution SharePoint Development Services Microsoft Online Services Microsoft Azure Services Best regards, Gary ITVIATSA Software Development Contact: ibsoftware at yeah.net Pls send address to larryremove123 at msn.com for remove From davegb at pobox.com Tue Jul 6 17:46:13 2010 From: davegb at pobox.com (Dave Brotherstone) Date: Tue, 6 Jul 2010 17:46:13 +0200 Subject: [capi-sig] Threading failure In-Reply-To: <648128.84773.qm@web307.biz.mail.mud.yahoo.com> References: <648128.84773.qm@web307.biz.mail.mud.yahoo.com> Message-ID: On Mon, Jul 5, 2010 at 8:36 PM, Paul Grunau wrote: > I asked this question here a few days ago, and someone suggested I ask > again, > providing all the code, so that it can be actually tried. So here it is, > and I?m > still stumped. > > Although I've never written a pure extension, I have just implemented an embedded & extended python, that works with multiple C threads, and I can't see anything glaringly obviously wrong in your code. What I might suggest is that you try with the debug version of python, and then when the program crashes you can take a dump, allowing you to see at least the stack trace for the different threads. Maybe that will provide some clues. I found that useful when debugging mine. I'd also say that as you suggest that it only crashes when the timer ticks very often, it could be a re-entrant problem? The docs on timeSetEvent don't give much information, but if the one timer callback event hasn't finished by the time the next one starts, does that start a new thread, wait for the last to finish then callback again on the same thread? Hopefully someone with more experience can spot something. Just a suggestion. Dave. From skippy.hammond at gmail.com Mon Jul 19 03:15:32 2010 From: skippy.hammond at gmail.com (Mark Hammond) Date: Mon, 19 Jul 2010 11:15:32 +1000 Subject: [capi-sig] Threading failure In-Reply-To: <648128.84773.qm@web307.biz.mail.mud.yahoo.com> References: <648128.84773.qm@web307.biz.mail.mud.yahoo.com> Message-ID: <4C43A734.4000808@gmail.com> On 6/07/2010 4:36 AM, Paul Grunau wrote: > I asked this question here a few days ago, and someone suggested I ask again, > providing all the code, so that it can be actually tried. So here it is, and I?m > still stumped. It isn't clear that this is your problem, but all your methods which return None fail to increment it's reference count. Also, consider reducing your code down to the smallest possible sample which demonstrates the problem and attach the source files to the message - you may find that if you can get it down to "a few" lines of code people which people can use without trying to copy/paste then fix the line wrapping, people will actually try it rather than speculate about it. HTH, Mark From stefan_ml at behnel.de Mon Jul 19 07:10:33 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 19 Jul 2010 07:10:33 +0200 Subject: [capi-sig] Threading failure In-Reply-To: <4C43A734.4000808@gmail.com> References: <648128.84773.qm@web307.biz.mail.mud.yahoo.com> <4C43A734.4000808@gmail.com> Message-ID: <4C43DE49.9060701@behnel.de> Mark Hammond, 19.07.2010 03:15: > On 6/07/2010 4:36 AM, Paul Grunau wrote: >> I asked this question here a few days ago, and someone suggested I ask >> again, >> providing all the code, so that it can be actually tried. So here it >> is, and I?m still stumped. > > It isn't clear that this is your problem, but all your methods which > return None fail to increment it's reference count. Good call. Since returning None is so common, there's even a macro for that, called Py_RETURN_NONE. A suggestion to the OP at this point: using Cython instead of C makes it easy to get the ref-counting right, as you don't have to care about it there. Another very common thing to get wrong: if the callback comes within a completely new thread that was not created by Python, you need to set up Python's thread state first. PyGILState_Ensure() is not enough in this case. See the C-API docs. Stefan From ideasman42 at gmail.com Fri Jul 30 07:43:57 2010 From: ideasman42 at gmail.com (Campbell Barton) Date: Fri, 30 Jul 2010 15:43:57 +1000 Subject: [capi-sig] Best practice for new namespace (from C/API) In-Reply-To: References: Message-ID: Posted to the python developer list but suggested I mail this here instead. --- Hi, I'm writing because I'm working on a project where the user can run scripts that don't reference a file but use internal application text data. Otherwise we are not doing anything tricky, only that the scripts should each run independently (no cruft left from the previous scripts namespace, sharing sys.modules etc is fine). Something which is unclear to me even after looking over pythonrun.c is the correct way to setup a namespace. For years we have been doing this and it seemed to work fine... ?PyObject *d = PyDict_New(); // new namespace ?PyDict_SetItemString(d, "__builtins__", PyEval_GetBuiltins()); ?PyDict_SetItemString(d, "__file__", PyString_FromString(filename)); // fake, avoids sys.argv[0] being used for warnings. /* --- snip ---*/ ?PyEval_EvalCode(compiled_text, d, d); Recently a developer reported a bug where pickle wasn't working, it turns out that in a few places python expects the __main__ modules namespace to match that if the running script: _pickle.c's save_global() in this case eg: ?>>> spam = 10 ?>>> print(__import__("__main__").__dict__["spam"]) ?... 10 Once I found this was the problem it was simple to use __main__'s namespace however there are still things that are not clear about exactly how this should be done. Simplified code... ?PyObject *item, *dict= PyModule_GetDict(PyImport_AddModule("__main__")); ?PyDict_Clear(dict); ?PyDict_SetItemString(dict, "__builtins__", PyImport_AddModule("builtins")); ?item = PyUnicode_FromString( "__main__" ); ?PyDict_SetItemString( dict, "__name__", item ); ?Py_DECREF(item); ?PyDict_SetItemString(d, "__file__", PyString_FromString(filename)); // fake, avoids sys.argv[0] being used for warnings. /* --- snip ---*/ ?PyEval_EvalCode(compiled_text, dict, dict); Still this leaves me with the following questions... - Whats the best way to manage the namespace for running multiple scripts one after another? clear and initialize __main__'s dict each time?, keep a copy and restore it after each run? - should the original __main__ namespace be restored after running a script? - pickle expects: __import__("__main__").__dict__ == ***the current namespace***, is this apart of the python spec or just something specific to pickle? - PyDict_SetItemString(d, "__builtins__", PyEval_GetBuiltins()) acts differently to PyDict_SetItemString(dict, "__builtins__", PyImport_AddModule("builtins")), using the PyEval_GetBuiltins() dict adds every member of __builtins__ when running: print(__import__("__main__").__dict__.keys()), rather then just showing __builtins__. -- - Campbell