[Pythonmac-SIG] ThreadSafety and Callbacks
Donovan Preston
dp@ulaluma.com
Fri, 12 Apr 2002 03:05:06 -0700
--Apple-Mail-1--503482814
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
format=flowed
Jack and Just (and anyone else),
As you know, I have been keenly interested in issues surrounding
CarbonEvent programming for quite a while now, since I see the Carbon
Event Manager as an elegant solution to event-based programming.
Unfortunately, thread safety issues aren't quite as elegant.
I hate to throw more unfinished patches onto the fire, but I have been
working on exactly the issue you have been discussing in "Threading and
FrameWork.Application", and also in the thread on Python-dev. I thought
it would be better to submit my unfinished solution to share my learning
experience than to try to finish it in silence while someone else did
the same work in an unrelated way.
Attached find some patches to pymactoolbox, mactoolboxglue, macmain, and
CarbonEventsupport. Take a look at the attached patches first, they are
quite short. Here's the basic theory behind it, currently:
In reference to Martin v. Loewis' message, Very similar to the 1)
_tkinter approach, where we guarantee that the call-back appears to come
from within the context of the same thread that released the GIL. Within
the context of my applications, RunApplicationEventLoop releases the GIL
(by calling Py_BEGIN_ALLOW_THREADS) and any Carbon Event callback
reacquires the GIL (By inserting the macro
Py_GENERATE_THREAD_STATE_AND_ACQUIRE_LOCK (by lock I am referring to the
GIL)), and releases it when done.
Note that this code currently ensures that each callback will run to
completion before a new callback begins by requiring each callback to
first acquire a MPCriticalRegion mutex
(MPEnterCriticalRegion(_criticalRegion, kDurationForever)) This may
cause deadlocks in some situations where the Carbon Event calls back
into the toolbox, thus generating another Callback which tries to
acquire the critical region... (unless ALL toolbox calls release the GIL
and the CriticalRegion, (and ALL callbacks acquire them) which I am
currently thinking may be the best solution)
Also note that since this code currently guarantees callback atomicity,
it is NOT necessary to create a new PyThreadState every time a callback
comes in, and this is a bunch of wasted effort and may be removed.
However, not removing it leads to the next solution, which I have also
tried, which is 2) the free threading approach:
If you instead remove the MPCriticalRegion mutex, and leave the
PyThreadState_New call, you will then get one new Python thread per
Carbon Event callback. This works quite well in some cases, however,
incurs the additional overhead of creating and destroying a thread state
for every callback, and you have to be very careful if you want to be
able to call Toolbox calls from an arbitrary event handler, especially
when there is the potential for multiple event handler threads to be
running at once.
I'm sorry if this isn't incredibly coherent, I have just worked a 12
hour day and am composing this message at three in the morning. I will
try to clarify both the code and my explanation tomorrow (I have the day
off, woohoo!)
Donovan
--Apple-Mail-1--503482814
Content-Disposition: attachment;
filename=mactoolboxglue.diff
Content-Transfer-Encoding: 7bit
Content-Type: application/octet-stream;
x-unix-mode=0644;
name="mactoolboxglue.diff"
Index: Python/mactoolboxglue.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Python/mactoolboxglue.c,v
retrieving revision 1.9
diff -c -r1.9 mactoolboxglue.c
*** Python/mactoolboxglue.c 4 Dec 2001 01:11:32 -0000 1.9
--- Python/mactoolboxglue.c 10 Apr 2002 05:40:23 -0000
***************
*** 29,35 ****
--- 29,91 ----
#ifdef WITHOUT_FRAMEWORKS
#include <Script.h>
#include <Resources.h>
+ #else
+ #include <Carbon/Carbon.h>
#endif
+
+ #if TARGET_API_MAC_OSX
+ static PyThreadState *_savedThreadState = NULL;
+ static PyThreadState *_originalThreadState = NULL;
+ static PyInterpreterState *_savedInterpreterState = NULL;
+ static MPCriticalRegionID _criticalRegion = NULL;
+
+ static pthread_t currentThreadID = NULL;
+ static int recursiveCount = 0;
+
+ void PyMac_SetupCallbackThreadGeneration(void) {
+ PyEval_InitThreads();
+ _originalThreadState = PyThreadState_Get();
+ _savedInterpreterState = _originalThreadState->interp;
+ _savedThreadState = PyThreadState_New(_savedInterpreterState);
+ MPCreateCriticalRegion(&_criticalRegion);
+ printf("hello threadstate original %d new %d\n", _originalThreadState, _savedThreadState);
+ }
+
+ void PyMac_DestroyCallbackThreadGeneration(void) {
+ printf("Acquire lock to destroy lock %d\n", _originalThreadState);
+ MPEnterCriticalRegion(_criticalRegion, kDurationForever);
+ PyThreadState_Swap(_originalThreadState);
+ MPExitCriticalRegion(_criticalRegion);
+ }
+
+ PyThreadState* PyMac_GenerateThreadStateAndAcquireInterpreterLock() {
+ PyThreadState *newCallbackThreadState;
+
+ printf("Acquiring Critical Region: ");
+
+ MPEnterCriticalRegion(_criticalRegion, kDurationForever);
+ PyEval_AcquireLock();
+ newCallbackThreadState = PyThreadState_New(_savedInterpreterState);
+ _savedThreadState = PyThreadState_Swap(newCallbackThreadState);
+ printf("Saved: %d, New: %d\n", _savedThreadState, newCallbackThreadState);
+ return newCallbackThreadState;
+ }
+
+ void PyMac_DestroyThreadStateAndReleaseInterpreterLock(PyThreadState *inState) {
+ printf("Exiting Critical Region: ");
+ if (_savedThreadState != inState) {
+ printf("Restored: %d, Destroyed: %d\n", _savedThreadState, inState);
+ PyThreadState_Swap(_savedThreadState);
+ PyThreadState_Clear(inState);
+ PyThreadState_Delete(inState);
+ } else {
+ printf("xxx exited thread recursion\n");
+ }
+ PyEval_ReleaseLock();
+ MPExitCriticalRegion(_criticalRegion);
+ }
+ #endif /* TARGET_API_MAC_OSX */
+
/*
** Find out what the current script is.
--Apple-Mail-1--503482814
Content-Disposition: attachment;
filename=pymactoolbox.diff
Content-Transfer-Encoding: 7bit
Content-Type: application/octet-stream;
x-unix-mode=0644;
name="pymactoolbox.diff"
Index: Include/pymactoolbox.h
===================================================================
RCS file: /cvsroot/python/python/dist/src/Include/pymactoolbox.h,v
retrieving revision 1.4
diff -c -r1.4 pymactoolbox.h
*** Include/pymactoolbox.h 5 Nov 2001 14:39:22 -0000 1.4
--- Include/pymactoolbox.h 10 Apr 2002 05:40:15 -0000
***************
*** 26,31 ****
--- 26,54 ----
#include <QuickTime/QuickTime.h>
#endif
+ #if TARGET_API_MAC_OSX
+ /* Prototypes */
+ void PyMac_SetupCallbackThreadGeneration(void);
+ void PyMac_DestroyCallbackThreadGeneration(void);
+ PyThreadState* PyMac_GenerateThreadStateAndAcquireInterpreterLock(void);
+ void PyMac_DestroyThreadStateAndReleaseInterpreterLock(PyThreadState *inState);
+
+ /* Macros */
+ #define PYMAC_SETUP_CALLBACK_THREAD_GENERATION PyMac_SetupCallbackThreadGeneration();
+ #define PYMAC_DESTROY_CALLBACK_THREAD_GENERATION PyMac_DestroyCallbackThreadGeneration();
+ #define PYMAC_GENERATE_THREAD_STATE_AND_ACQUIRE_LOCK { \
+ PyThreadState *_save; \
+ _save = PyMac_GenerateThreadStateAndAcquireInterpreterLock();
+ #define PYMAC_DESTROY_THREAD_STATE_AND_RELEASE_LOCK PyMac_DestroyThreadStateAndReleaseInterpreterLock(_save); \
+ }
+ #else
+ /* On OS 9, these get reduced to nothing */
+ #define PYMAC_SETUP_CALLBACK_THREAD_GENERATION
+ #define PYMAC_DESTROY_CALLBACK_THREAD_GENERATION
+ #define PYMAC_GENERATE_THREAD_STATE_AND_ACQUIRE_LOCK(x)
+ #define PYMAC_DESTROY_THREAD_STATE_AND_RELEASE_LOCK
+ #endif
+
/*
** Helper routines for error codes and such.
*/
--Apple-Mail-1--503482814
Content-Disposition: attachment;
filename=macmain.diff
Content-Transfer-Encoding: 7bit
Content-Type: application/octet-stream;
x-unix-mode=0644;
name="macmain.diff"
Index: Mac/Python/macmain.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Mac/Python/macmain.c,v
retrieving revision 1.75
diff -c -r1.75 macmain.c
*** Mac/Python/macmain.c 29 Mar 2002 14:43:50 -0000 1.75
--- Mac/Python/macmain.c 12 Apr 2002 09:56:26 -0000
***************
*** 24,36 ****
/* Python interpreter main program */
#include "Python.h"
#include "pythonresources.h"
#include "import.h"
#include "marshal.h"
#include "macglue.h"
-
- #ifdef WITHOUT_FRAMEWORKS
#include <Memory.h>
#include <Resources.h>
#include <stdio.h>
--- 24,35 ----
/* Python interpreter main program */
+ #ifdef WITHOUT_FRAMEWORKS
#include "Python.h"
#include "pythonresources.h"
#include "import.h"
#include "marshal.h"
#include "macglue.h"
#include <Memory.h>
#include <Resources.h>
#include <stdio.h>
***************
*** 51,56 ****
--- 50,60 ----
#endif /* USE_APPEARANCE */
#else
#include <Carbon/Carbon.h>
+ #include <Python/Python.h>
+ #include <Python/pythonresources.h>
+ #include <Python/import.h>
+ #include <Python/marshal.h>
+ #include <Python/macglue.h>
#endif /* WITHOUT_FRAMEWORKS */
#ifdef __MWERKS__
***************
*** 655,660 ****
--- 659,666 ----
#endif
Py_Initialize();
+
+ PYMAC_SETUP_CALLBACK_THREAD_GENERATION
PyUnicode_SetDefaultEncoding(PyMac_getscript());
***************
*** 675,680 ****
--- 681,688 ----
if ( filename != NULL || command != NULL )
sts = (run_inspect() || sts);
+
+ PYMAC_DESTROY_CALLBACK_THREAD_GENERATION
Py_Exit(sts);
/*NOTREACHED*/
--Apple-Mail-1--503482814
Content-Disposition: attachment;
filename=CarbonEvtsupport.diff
Content-Transfer-Encoding: 7bit
Content-Type: application/octet-stream;
x-unix-mode=0644;
name="CarbonEvtsupport.diff"
Index: Mac/Modules/carbonevt/CarbonEvtsupport.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Mac/Modules/carbonevt/CarbonEvtsupport.py,v
retrieving revision 1.10
diff -c -r1.10 CarbonEvtsupport.py
*** Mac/Modules/carbonevt/CarbonEvtsupport.py 8 Jan 2002 11:49:31 -0000 1.10
--- Mac/Modules/carbonevt/CarbonEvtsupport.py 12 Apr 2002 09:27:58 -0000
***************
*** 89,102 ****
return; \
}} while(0)
-
- #define USE_MAC_MP_MULTITHREADING 0
-
- #if USE_MAC_MP_MULTITHREADING
- static PyThreadState *_save;
- static MPCriticalRegionID reentrantLock;
- #endif /* USE_MAC_MP_MULTITHREADING */
-
extern int CFStringRef_New(CFStringRef *);
extern int CFStringRef_Convert(PyObject *, CFStringRef *);
--- 89,94 ----
***************
*** 172,181 ****
PyObject *retValue;
int status;
! #if USE_MAC_MP_MULTITHREADING
! MPEnterCriticalRegion(reentrantLock, kDurationForever);
! PyEval_RestoreThread(_save);
! #endif /* USE_MAC_MP_MULTITHREADING */
retValue = PyObject_CallFunction((PyObject *)outPyObject, "O&O&",
EventHandlerCallRef_New, handlerRef,
--- 164,170 ----
PyObject *retValue;
int status;
! PYMAC_GENERATE_THREAD_STATE_AND_ACQUIRE_LOCK
retValue = PyObject_CallFunction((PyObject *)outPyObject, "O&O&",
EventHandlerCallRef_New, handlerRef,
***************
*** 194,203 ****
Py_DECREF(retValue);
}
! #if USE_MAC_MP_MULTITHREADING
! _save = PyEval_SaveThread();
! MPExitCriticalRegion(reentrantLock);
! #endif /* USE_MAC_MP_MULTITHREADING */
return status;
}
--- 183,189 ----
Py_DECREF(retValue);
}
! PYMAC_DESTROY_THREAD_STATE_AND_RELEASE_LOCK
return status;
}
***************
*** 337,357 ****
EventRefobject.add(f)
runappeventloop = """
! #if USE_MAC_MP_MULTITHREADING
! if (MPCreateCriticalRegion(&reentrantLock) != noErr) {
! PySys_WriteStderr("lock failure\\n");
! return NULL;
! }
! _save = PyEval_SaveThread();
! #endif /* USE_MAC_MP_MULTITHREADING */
RunApplicationEventLoop();
! #if USE_MAC_MP_MULTITHREADING
! PyEval_RestoreThread(_save);
!
! MPDeleteCriticalRegion(reentrantLock);
! #endif /* USE_MAC_MP_MULTITHREADING */
Py_INCREF(Py_None);
--- 323,333 ----
EventRefobject.add(f)
runappeventloop = """
! Py_BEGIN_ALLOW_THREADS
RunApplicationEventLoop();
! Py_END_ALLOW_THREADS
Py_INCREF(Py_None);
--Apple-Mail-1--503482814--