[Python-Dev] Making python C-API thread safe (try 2)

[Harri Pesonen]

Sorry about the previous post! :-)

Background: I'm new to Python. I just embedded Python into a 
multi-threaded application, and extended Python by callbacks as well. 
During this process I realised that Python is not internally as 
beautiful as externally. It is not thread safe. This surprised me, 
because Python is a quite young programming language. I started thinking 
how to fix this, how to make Python thread safe. Currently Python does 
not support multiprocessor machines or hyperthreading processors. It is 
only a matter of time when this has to be fixed, better soon than later.

How to make C-API thread safe: It's quite simple, in fact. Thread should 
first call

PyThreadState* Py_NewInterpreter()

and use this thread state pointer in every following API call. This 
thread state pointer should point to a memory structure, that holds 
everything that Python interpreter needs. There should be no global 
variables, not even None, except

const char* Py_GetVersion()

and such. So every Python function should have this thread state as the 
first argument:

int PyRun_SimpleString(PyThreadState*tstate, char *command)

and so on. Also callbacks should have it, about every API function.

Advantages: The following functions would not be needed any more, it 
would make the API much simpler:

void PyEval_InitThreads( )
void PyEval_AcquireLock( )
void PyEval_ReleaseLock( )
void PyEval_AcquireThread( PyThreadState *tstate)
void PyEval_ReleaseThread( PyThreadState *tstate)
PyThreadState* PyEval_SaveThread( )
void PyEval_RestoreThread( PyThreadState *tstate)
Py_BEGIN_ALLOW_THREADS
Py_END_ALLOW_THREADS
Py_BLOCK_THREADS
Py_UNBLOCK_THREADS
PyInterpreterState* PyInterpreterState_New( )
void PyInterpreterState_Clear( PyInterpreterState *interp)
void PyInterpreterState_Delete( PyInterpreterState *interp)
PyThreadState* PyThreadState_New( PyInterpreterState *interp)
void PyThreadState_Clear( PyThreadState *tstate)
void PyThreadState_Delete( PyThreadState *tstate)
PyThreadState* PyThreadState_Get( )
PyThreadState* PyThreadState_Swap( PyThreadState *tstate)

The current documentation says: "In order to support multi-threaded 
Python programs, the interpreter regularly releases and reacquires the 
lock -- by default, every 100 bytecode instructions". This would not be 
needed anymore. It would make the interpreter faster.

Making this change would take a couple of weeks, if the job would be 
divided between modules to different persons. The change is trivial, but 
it would have to be made everywhere.

Disadvantages: Everything old would break. Thread safe Python should 
have a version number significantly higher, like Python 3. Old 
applications would work with old Python versions, with old Python DLLs, 
but they would need to be rewritten for the new API. It would be a 
trivial task, but still.

Disclaimer: I already got mail saying: "

This issue gets discussed regularly on comp.lang.python; python-dev is
for discussion of specific changes to the Python language and libraries.
Until you are familiar with the way threading works in Python and have
some concrete proposal to make, please keep the discussion on c.l.py."

The current situation is very unfortunate, having a single threaded implementation in the modern computer age. I have a concrete proposal, which would break everything :-) but I think that is the only solution.

Thanks for reading,

Harri