[Python-Dev] Re: Extension modules, Threading, and the GIL

[Mark Hammond]

[David Abrahams]

> I can't really imagine what you're suggesting here.  Code samples
> help.

OK :)  Here is some code demonstrating my "pluggable" idea, but working
backwards <wink>.

(Please don't pick on the names, or where I will need to pass a param, or
where I forgot to cast, etc <wink>)

First, let's assume we come up with a high-level API similar to:

/*** Python "auto-thread-state" API ***/

typedef void *PyATS_HANDLE;

/* Get's a "cookie" for use in all subsequent auto-thread-state
   calls.  Generally called once per application/extension.
   Not strictly necessary, but a vehicle to abstract
   PyInterpreterState should people care enough.
*/
PyATS_HANDLE PyAutoThreadState_Init();

/* Ensure we have Python ready to rock.  This is the "slow" version
   that assumes nothing about Python's state, other than the handle
   is valid.
*/
int PyAutoThreadState_Ensure(PyATS_HANDLE);

/* Notify the auto-thread-state mechanism that we are done - there should
   be one Release() per Ensure().  Again, maybe not necessary if we are
   super clever, but for the sake of argument ...<wink>
*/
void PyAutoThreadState_Release(PyATS_HANDLE);

/* end of definitions */
This is almost the "holy grail" for me.  Your module/app init code does:
PyATS_HANDLE myhandle = PyAutoThreadState_Init()

And your C function entry points do a PyAutoThreadStateEnsure()/Release()
pair.  That is it!

Your Python extension functions generally need take no special action,
including releasing the lock, as PyAutoThreadStateEnsure() is capable of
coping with the fact the lock is already held by this thread.

So, to my mind, that sketches out the high-level API we are discussing.
Underneath the covers, Python will need TLS to implement this.  We have 2
choices for the TLS:

* Implement it inside Python as part of the platform threading API.  This
works fine in most scenarios, but may potentially let down e.g. some Mozilla
xpcom users - users where Python is ported, but this TLS API is not.  Those
platforms could not use this new AutoThreadState API, even though the
application has a functional TLS implemention provided by xpcom.

* Allow the extension author to provide "pluggable" TLS.  This would expand
the API like so:

/* Back in the "auto thread state" header
struct PyTLS_FUNCS = {
	/* Save a PyThreadState pointer in TLS */
	int (*pfnSaveThreadState)(PyThreadState *p);
	/* Release the pointer for the thread (as the thread dies) */
	void (*pfnReleaseThreadState)();
	/* Get the saved pointer for this thread */
	PyThreadState *(*pfnGetThreadState)();
}

/* For the Win32 extensions, I would provide the following code in my
extension */
DWORD dwTlsIndex = 0;

// The TLS functions we "export" back to Python.
int MyTLS_SaveThreadState(PyThreadState *ts)
{
	// allocate space for the pointer in the platform TLS
	PyThreadState **p = (ThreadData **)malloc(sizeof(PyThreadState *));
	if (!p) return -1;
	*p = ts;
	TlsSetValue(dwTlsIndex, p);
	return 0;
}

void PyThreadState MyTLS_DropThreadState()
{
	PyThreadState **p = (PyThreadState**)TlsGetValue(dwTlsIndex);
	if (!p) return;
	TlsSetValue(dwTlsIndex, NULL);
	free(p);
}

PyThreadState *MyTLS_FetchThreadState()
{
	return (PyThreadState *)TlsGetValue(dwTlsIndex);
}

// A structure of function pointers defined by Python.
Py_AutoThreadStateFunctions myfuncs = {
	MyTLS_SaveThreadState,
	MyTLS_DropThreadState,
	MyTLS_FetchThreadState
}

/* End of Win32 code */

The XPCOM code would look almost identical, except spelt
PR_GetThreadPrivate, PR_SetThreadPrivate etc.  I assume pthreads can also
fit into this scheme.

> > But yeah, as I said before, happy to YAGNI it.
>
> Not sure what "it" is supposed to be here, either.

I'm happy to YAGNI the pluggable TLS idea.  I see that the number of users
who would actually benefit is tiny.  Keeping the TLS api completely inside
Python is fine with me.

Mark.