[Python-Dev] Making python C-API thread safe (try 2)
fuerte at sci.fi
Thu Sep 11 20:53:10 EDT 2003
Sorry about the previous post! :-)
Background: I'm new to Python. I just embedded Python into a
multi-threaded application, and extended Python by callbacks as well.
During this process I realised that Python is not internally as
beautiful as externally. It is not thread safe. This surprised me,
because Python is a quite young programming language. I started thinking
how to fix this, how to make Python thread safe. Currently Python does
not support multiprocessor machines or hyperthreading processors. It is
only a matter of time when this has to be fixed, better soon than later.
How to make C-API thread safe: It's quite simple, in fact. Thread should
and use this thread state pointer in every following API call. This
thread state pointer should point to a memory structure, that holds
everything that Python interpreter needs. There should be no global
variables, not even None, except
const char* Py_GetVersion()
and such. So every Python function should have this thread state as the
int PyRun_SimpleString(PyThreadState*tstate, char *command)
and so on. Also callbacks should have it, about every API function.
Advantages: The following functions would not be needed any more, it
would make the API much simpler:
void PyEval_InitThreads( )
void PyEval_AcquireLock( )
void PyEval_ReleaseLock( )
void PyEval_AcquireThread( PyThreadState *tstate)
void PyEval_ReleaseThread( PyThreadState *tstate)
PyThreadState* PyEval_SaveThread( )
void PyEval_RestoreThread( PyThreadState *tstate)
PyInterpreterState* PyInterpreterState_New( )
void PyInterpreterState_Clear( PyInterpreterState *interp)
void PyInterpreterState_Delete( PyInterpreterState *interp)
PyThreadState* PyThreadState_New( PyInterpreterState *interp)
void PyThreadState_Clear( PyThreadState *tstate)
void PyThreadState_Delete( PyThreadState *tstate)
PyThreadState* PyThreadState_Get( )
PyThreadState* PyThreadState_Swap( PyThreadState *tstate)
The current documentation says: "In order to support multi-threaded
Python programs, the interpreter regularly releases and reacquires the
lock -- by default, every 100 bytecode instructions". This would not be
needed anymore. It would make the interpreter faster.
Making this change would take a couple of weeks, if the job would be
divided between modules to different persons. The change is trivial, but
it would have to be made everywhere.
Disadvantages: Everything old would break. Thread safe Python should
have a version number significantly higher, like Python 3. Old
applications would work with old Python versions, with old Python DLLs,
but they would need to be rewritten for the new API. It would be a
trivial task, but still.
Disclaimer: I already got mail saying: "
This issue gets discussed regularly on comp.lang.python; python-dev is
for discussion of specific changes to the Python language and libraries.
Until you are familiar with the way threading works in Python and have
some concrete proposal to make, please keep the discussion on c.l.py."
The current situation is very unfortunate, having a single threaded implementation in the modern computer age. I have a concrete proposal, which would break everything :-) but I think that is the only solution.
Thanks for reading,
More information about the Python-Dev