[Python-Dev] shm + Win32 + docs (was: Adding library modules to the core)
Vladimir Marangozov
Vladimir.Marangozov@inrialpes.fr
Thu, 17 Aug 2000 04:12:18 +0200 (CEST)
Eric S. Raymond wrote:
>
> Vladimir, I suggest that the most useful thing you could do to advance
> the process at this point would be to document shm in core-library style.
Eric, I'm presently suffering from chronic lack of time (as you probably
do too) so if you could write the docs for me and take all associated
credits for them, please do so (shouldn't be that hard, after all -- the
web page and the comments are self-explanatory :-). I'm willing to "unblock"
you on this, but I can hardly make the time for it -- it's low-priority on
my dynamic task schedule. :(
I'd also love to assemble the win32 bits on the matter (what's in win32event
for the semaphore interface + my Windows book) to add shm and sem for
Windows and rewrite the interface, but I have no idea on when this could
happen.
I will disappear from the face of the World sometime soon and it's
unclear on when I'll be able to reappear (nor how soon I'll disappear, btw)
So, be aware of that. I hope to be back again before 2.1 so if we can
wrap up a Unix + win32 shm, that would be much appreciated!
>
> At the moment, core Python has nothing (with the weak and nonportable
> exception of open(..., O_EXCL)) that can do semaphores properly. Thus
> shm would address a real gap in the language.
Indeed. This is currently being discussed on the french Python list,
where Richard Gruet (rgruet@ina.fr) posted the following code for
inter-process locks: glock.py
I don't have the time to look at it in detail, just relaying here
for food and meditation :-)
--
Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
------------------------------[ glock.py ]----------------------------
#!/usr/bin/env python
#----------------------------------------------------------------------------
# glock.py: Global mutex
#
# Prerequisites:
# - Python 1.5.2 or newer (www.python.org)
# - On windows: win32 extensions installed
# (http://www.python.org/windows/win32all/win32all.exe)
# - OS: Unix, Windows.
#
# History:
# -22 Jan 2000 (R.Gruet): creation
#
# Limitations:
# TODO:
#----------------------------------------------------------------------------
''' This module defines the class GlobalLock that implements a global
(inter-process) mutex that works on Windows and Unix, using
file-locking on Unix (I also tried this approach on Windows but got
some tricky problems so I ended using Win32 Mutex)..
See class GlobalLock for more details.
'''
__version__ = 1,0,2
__author__ = ('Richard Gruet', 'rgruet@ina.fr')
# Imports:
import sys, string, os
# System-dependent imports for locking implementation:
_windows = (sys.platform == 'win32')
if _windows:
try:
import win32event, win32api, pywintypes
except ImportError:
sys.stderr.write('The win32 extensions need to be installed!')
else: # assume Unix
try:
import fcntl
except ImportError:
sys.stderr.write("On what kind of OS am I ? (Mac?) I should be on "
"Unix but can't import fcntl.\n")
raise
import threading
# Exceptions :
# ----------
class GlobalLockError(Exception):
''' Error raised by the glock module.
'''
pass
class NotOwner(GlobalLockError):
''' Attempt to release somebody else's lock.
'''
pass
# Constants
# ---------:
true=-1
false=0
#----------------------------------------------------------------------------
class GlobalLock:
#----------------------------------------------------------------------------
''' A global mutex.
*Specification:
-------------
The lock must act as a global mutex, ie block between different
candidate processus, but ALSO between different candidate
threads of the same process.
It must NOT block in case of recursive lock request issued by
the SAME thread.
Extraneous unlocks should be ideally harmless.
*Implementation:
--------------
In Python there is no portable global lock AFAIK.
There is only a LOCAL/ in-process Lock mechanism
(threading.RLock), so we have to implement our own solution.
Unix: use fcntl.flock(). Recursive calls OK. Different process OK.
But <> threads, same process don't block so we have to
use an extra threading.RLock to fix that point.
Win: We use WIN32 mutex from Python Win32 extensions. Can't use
std module msvcrt.locking(), because global lock is OK, but
blocks also for 2 calls from the same thread!
'''
def __init__(self, fpath, lockInitially=false):
''' Creates (or opens) a global lock.
@param fpath Path of the file used as lock target. This is also
the global id of the lock. The file will be created
if non existent.
@param lockInitially if true locks initially.
'''
if _windows:
self.name = string.replace(fpath, '\\', '_')
self.mutex = win32event.CreateMutex(None, lockInitially, self.name)
else: # Unix
self.name = fpath
self.flock = open(fpath, 'w')
self.fdlock = self.flock.fileno()
self.threadLock = threading.RLock()
if lockInitially:
self.acquire()
def __del__(self):
#print '__del__ called' ##
try: self.release()
except: pass
if _windows:
win32api.CloseHandle(self.mutex)
else:
try: self.flock.close()
except: pass
def __repr__(self):
return '<Global lock @ %s>' % self.name
def acquire(self):
''' Locks. Suspends caller until done.
On windows an IOError is raised after ~10 sec if the lock
can't be acquired.
@exception GlobalLockError if lock can't be acquired (timeout)
'''
if _windows:
r = win32event.WaitForSingleObject(self.mutex, win32event.INFINITE)
if r == win32event.WAIT_FAILED:
raise GlobalLockError("Can't acquire mutex.")
else:
# Acquire 1st the global (inter-process) lock:
try:
fcntl.flock(self.fdlock, fcntl.LOCK_EX) # blocking
except IOError: #(errno 13: perm. denied,
# 36: Resource deadlock avoided)
raise GlobalLockError('Cannot acquire lock on "file" %s\n' %
self.name)
#print 'got file lock.' ##
# Then acquire the local (inter-thread) lock:
self.threadLock.acquire()
#print 'got thread lock.' ##
def release(self):
''' Unlocks. (caller must own the lock!)
@return The lock count.
@exception IOError if file lock can't be released
@exception NotOwner Attempt to release somebody else's lock.
'''
if _windows:
try:
win32event.ReleaseMutex(self.mutex)
except pywintypes.error, e:
errCode, fctName, errMsg = e.args
if errCode == 288:
raise NotOwner("Attempt to release somebody else's lock")
else:
raise GlobalLockError('%s: err#%d: %s' % (fctName, errCode,
errMsg))
else:
# Acquire 1st the local (inter-thread) lock:
try:
self.threadLock.release()
except AssertionError:
raise NotOwner("Attempt to release somebody else's lock")
# Then release the global (inter-process) lock:
try:
fcntl.flock(self.fdlock, fcntl.LOCK_UN)
except IOError: # (errno 13: permission denied)
raise GlobalLockError('Unlock of file "%s" failed\n' %
self.name)
#----------------------------------------------------------------------------
# M A I N
#----------------------------------------------------------------------------
def main():
# unfortunately can't test inter-process lock here!
lockName = 'myFirstLock'
l = GlobalLock(lockName)
if not _windows:
assert os.path.exists(lockName)
l.acquire()
l.acquire() # rentrant lock, must not block
l.release()
l.release()
if _windows:
try: l.release()
except NotOwner: pass
else: raise Exception('should have raised a NotOwner exception')
# Check that <> threads of same process do block:
import threading, time
thread = threading.Thread(target=threadMain, args=(l,))
print 'main: locking...',
l.acquire()
print ' done.'
thread.start()
time.sleep(3)
print '\nmain: unlocking...',
l.release()
print ' done.'
time.sleep(0.1)
del l # to close file
print 'tests OK.'
def threadMain(lock):
print 'thread started(%s).' % lock
print 'thread: locking (should stay blocked for ~ 3 sec)...',
lock.acquire()
print 'thread: locking done.'
print 'thread: unlocking...',
lock.release()
print ' done.'
print 'thread ended.'
if __name__ == "__main__":
main()