[Python-ideas] Add faster locks to the threading module?

[Sturla Molden]

Thread synchronization with threading.Lock can be expensive. But 
consider this: Why should the active thread need to aquire a mutex, when 
it already holds one? That would be the GIL.

Instead of acqiring a lock (and possibly inducing thread swiching etc.), 
it could just deny the other threads access to the GIL for a while. The 
cost of that synchronization method would be completely amortized, as 
check intervals happen anyway.

Here is what a very naïve implementation would look like in ctypes (real 
code could use C instead, or perhaps should not attempt this at all...):

from contextlib import contextmanager
import ctypes
_Py_Ticker = ctypes.c_int.in_dll(ctypes.pythonapi,"_Py_Ticker")

@contextmanager
def threadsafe():
     tmp = _Py_Ticker.value
     _Py_Ticker.value = 0x7fffffff
     yield
     _Py_Ticker.value = tmp


Now we can do this:

with threadsafe():
     # the GIL is mine,
     # for as long as I want
     pass

The usecase for this "gillock" is about the same as for a spinlock C. We 
want synchronization for a breif period of time, but don't want the 
overhead of aquiring a mutex.

In Python this gillock has one big advantage over a spinlock: We don't 
have to wait, so we don't risk a tread switch on __enter__/aquire. But 
there can be only one instance of this lock, as there is only one GIL. 
That is the drawback compared to a spinlock.

Therefore I think both a spinlock and a gillock should be added to the 
threading module. These are synchronization methods that should be 
available.

P.S. A gillock working like the ctypes code above is of course very 
dangerous. If a C extension releases the GIL while _Py_Ticker is 
astronomic, we have a very bad situation... But real code could try to 
safeguard against this. E.g. Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS could be defined to do nothing if _Py_Ticker is 
above some ridiculous threshold.

P.P.S. Yes I know about the newgil. But I have not thought about how to 
achieve similar effect with that.



Sturla






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100721/6125bd53/attachment.html>