[Patches] per module locking patch for import.c
eric jones
ej@ee.duke.edu
Wed, 10 May 2000 23:25:01 -0400
This is a multi-part message in MIME format.
------=_NextPart_000_01F0_01BFBAD6.F6E2F560
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Hello,
This is a patch to import.c that allows per module locking. It stems from
an issue I have run into when trying to import and run wxPython in a second
thread. The central issue has been discussed lately on the thread-sig under
the title:
RE: [Thread-SIG] thread behavior with join() and import
I've included one of the pertinent messages at the bottom. Mark Hammond
tracked the issue to the import lock. His comments follow:
[begin comments]
Damn. The problem is clearly the import lock.
* Main script is imported - this acquires the import lock.
* Main script starts new thread
* New thread attempts import - blocks on the import lock.
* Main thread does a join() on the new thread; still holds the import
lock.
* Deadlocked.
Hrm. We have the situation that _any_ time code run via an import,
creates
threads which themselves do imports, we have a problem.
[snip]
The only other, much harder alternative, is to make the import lock more
fine-grained - eg, effectively attach a lock to each module, so the
threads
are only locked out when the _same_ module is being imported by 2
threads
(rather than _any_ module).
[snip]
[end comments]
I've tried my hand at writing the patch for this so that the module lock is
applied on a per module bases instead of a single lock for all imports. I
looked through the import.c file, and, best I can tell, the lock is totally
encapsulated within import_lock() and import_unlock(). These functions were
re-written and a very slight change was made to one other function. I am
new to looking at the python core and the INCREF/DECREF stuff, so I hope I
got it right. I can not think of any global side-affects to what I've done,
but it is possible I overlooked something.
The approach is as follows:
Locking:
1. A lock structure is created for each module the request a lock. A
pointer to this structure is stored in a standard python dictionary.
2. When modules request a lock, this dictionary is checked for that modules
entry. If it does not exist, a new structure is allocated and its pointer
is added to the dictionary. Otherwise, the lock info is pulled from the
dictionary.
3. The rest of locking proceeds in the same way as the current locking
code, but using this per module lock info.
Unlocking:
1. Read the module lock pointer from the dictionary. Failuer is fatal.
2. If lock_level is 0, do the standard stuff, and then delete the
dictionary
entry for this module.
3. If the dictionary is empty, delete the dictionary.
These last two steps could really be saved until the shutdown of python.
This would be more efficient (I'm pretty sure), but it would also
necessitate altering other code. Including them here keeps the entire
process encapsulated in the lock functions. The only other change is that
the module name has to be passed to import_lock(char *mod_name) and
import_unlock(char *mod_name). They are only called once each (in
import.c
The patches are against the CVS tree for 1.6 as of 8:00 pm on 5/10/00. It
has been tested on NT 4.0 and RH6.1. The "make test" on RH 6.1 hangs on
"test_fork1.py" **with or without** the patch. After removing this test, it
passed all test tried(52 passed, 21 skipped).
thanks,
eric
#### Disclaimer Text ####
I confirm that, to the best of my knowledge and belief, this
contribution is free of any claims of third parties under
copyright, patent or other rights or interests ("claims"). To
the extent that I have any such claims, I hereby grant to CNRI a
nonexclusive, irrevocable, royalty-free, worldwide license to
reproduce, distribute, perform and/or display publicly, prepare
derivative versions, and otherwise use this contribution as part
of the Python software and its related documentation, or any
derivative versions thereof, at no cost to CNRI or its licensed
users, and to authorize others to do so.
I acknowledge that CNRI may, at its sole discretion, decide
whether or not to incorporate this contribution in the Python
software and its related documentation. I further grant CNRI
permission to use my name and other identifying information
provided to CNRI by me for use in connection with the Python
software and its related documentation.
#### THREAD SIG message detailing the issue. ####
>> I ask a question about threadin with import.
> Tim responds: Huh! Works fine for me under Win95 -- get a real OS <wink>.
> D:\Python>python misc/ttest.py
> and code runs correctly
Hmmm. Thats odd. It actually works for me when I run it this way. The
problem occurs if you run it like this:
D:\Python> python
PythonWin 1.5.2 (#0, Apr 13 1999, 10:51:12) [MSC 32 bit (Intel)] on
win32
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import ttest
worker: without done
main: without import done
main: with import done
code blocks!
I've re-written the sample to illustrate the strangeness a little better.
Running the thread_test.py code at the end of the message from the command
line causes the problem.
>>> import thread_test
worker: without done
main: without import done
main: with import done
! and then an endless block
If I comment out the call to test() at the bottom of the module, and then
run:
>>> import thread_test
>>> thread_test.test()
worker: without done
main: without import done
main: with import done
worker: with done
worker: with done
main: with import done
ALL TESTS PASS
So the problem only appears when the thread is executed during a module
import from the command line.
thanks,
eric
------------- thread_test.py -----------------------------
import threading ,time
class without_import(threading.Thread):
def run(self):
time.sleep(.5)
print 'worker: without done'
class with_import(threading.Thread):
def run(self):
import time
time.sleep(.5)
print 'worker: with done'
def test():
############# this works fine
wo = without_import()
wo.start()
wo.join()
print 'main: without import done'
######## without join(), thread also terminates normally
w = with_import()
w.start()
print 'main: with import done'
##### this blocks indefinitely
w = with_import()
w.start()
w.join()
print 'main: with import done'
print 'ALL TESTS PASS'
test()
------=_NextPart_000_01F0_01BFBAD6.F6E2F560
Content-Type: application/octet-stream;
name="import.diff"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="import.diff"
*** /home/emagserver/ej/wrk/python/dist/src/Python/import.c Wed May 10 =
22:17:38 2000=0A=
--- import.c Wed May 10 22:10:34 2000=0A=
***************=0A=
*** 165,214 ****=0A=
=0A=
#include "pythread.h"=0A=
=0A=
! static PyThread_type_lock import_lock =3D 0;=0A=
! static long import_lock_thread =3D -1;=0A=
! static int import_lock_level =3D 0;=0A=
=0A=
static void=0A=
! lock_import()=0A=
{=0A=
! long me =3D PyThread_get_thread_ident();=0A=
! if (me =3D=3D -1)=0A=
return; /* Too bad */=0A=
! if (import_lock =3D=3D NULL)=0A=
! import_lock =3D PyThread_allocate_lock();=0A=
! if (import_lock_thread =3D=3D me) {=0A=
! import_lock_level++;=0A=
! return;=0A=
}=0A=
! if (import_lock_thread !=3D -1 || !PyThread_acquire_lock(import_lock, =
0)) {=0A=
PyThreadState *tstate =3D PyEval_SaveThread();=0A=
! PyThread_acquire_lock(import_lock, 1);=0A=
PyEval_RestoreThread(tstate);=0A=
}=0A=
! import_lock_thread =3D me;=0A=
! import_lock_level =3D 1;=0A=
}=0A=
=0A=
static void=0A=
! unlock_import()=0A=
{=0A=
! long me =3D PyThread_get_thread_ident();=0A=
! if (me =3D=3D -1)=0A=
return; /* Too bad */=0A=
! if (import_lock_thread !=3D me)=0A=
Py_FatalError("unlock_import: not holding the import lock");=0A=
! import_lock_level--;=0A=
! if (import_lock_level =3D=3D 0) {=0A=
! import_lock_thread =3D -1;=0A=
! PyThread_release_lock(import_lock);=0A=
}=0A=
}=0A=
=0A=
#else=0A=
=0A=
! #define lock_import()=0A=
! #define unlock_import()=0A=
=0A=
#endif=0A=
=0A=
--- 165,254 ----=0A=
=0A=
#include "pythread.h"=0A=
=0A=
! static PyObject *import_locks =3D NULL;=0A=
! typedef struct {=0A=
! PyThread_type_lock lock;=0A=
! long thread;=0A=
! int level;=0A=
! } mod_lock;=0A=
=0A=
static void=0A=
! lock_import(char* mod_name)=0A=
{=0A=
! PyObject* py_lock =3D NULL;=0A=
! mod_lock *lock =3D NULL; =0A=
! long current_thread =3D PyThread_get_thread_ident();=0A=
! =0A=
! if (current_thread =3D=3D -1)=0A=
return; /* Too bad */=0A=
! =0A=
! if (import_locks =3D=3D NULL) {=0A=
! import_locks =3D PyDict_New();=0A=
! if( !import_locks)=0A=
! Py_FatalError("lock_import: failed to create lock list");=0A=
! }=0A=
! py_lock =3D PyDict_GetItemString(import_locks,mod_name); =0A=
! if (!py_lock) { /* First import of this module */=0A=
! lock =3D malloc(sizeof(mod_lock));=0A=
! lock->lock =3D PyThread_allocate_lock();=0A=
! lock->thread =3D -1;=0A=
! lock->level =3D 0;=0A=
! py_lock =3D PyInt_FromLong((long)lock);=0A=
! if(!py_lock)=0A=
! Py_FatalError("lock_import: failed to create lock");=0A=
! PyDict_SetItemString(import_locks,mod_name,py_lock); =0A=
! }=0A=
! else /* Module already has been locked before */=0A=
! lock =3D (mod_lock*)PyInt_AsLong(py_lock);=0A=
! if( lock->thread =3D=3D current_thread ) {=0A=
! lock->level++;=0A=
! return; /* ??what do we need to do to clean up REFs */=0A=
}=0A=
! if (lock->thread !=3D 1 || !PyThread_acquire_lock(lock->lock,0)) {=0A=
PyThreadState *tstate =3D PyEval_SaveThread();=0A=
! PyThread_acquire_lock(lock->lock, 1);=0A=
PyEval_RestoreThread(tstate);=0A=
}=0A=
! lock->thread =3D current_thread;=0A=
! lock->level =3D 1;=0A=
}=0A=
=0A=
static void=0A=
! unlock_import(char* mod_name)=0A=
{=0A=
! PyObject* py_lock =3D NULL;=0A=
! mod_lock *lock =3D NULL; =0A=
! long current_thread =3D PyThread_get_thread_ident();=0A=
! =0A=
! if (current_thread =3D=3D -1)=0A=
return; /* Too bad */=0A=
! if (!import_locks)=0A=
! Py_FatalError("unlock_import: no import lock list");=0A=
! =0A=
! py_lock =3D PyDict_GetItemString(import_locks,mod_name); =0A=
! if (!py_lock)=0A=
! Py_FatalError("unlock_import: module was never locked");=0A=
! =0A=
! lock =3D (mod_lock*)PyInt_AsLong(py_lock); =0A=
! if (lock->thread !=3D current_thread)=0A=
Py_FatalError("unlock_import: not holding the import lock");=0A=
! lock->level--;=0A=
! if (lock->level =3D=3D 0) {=0A=
! lock->thread =3D -1; /* not necessary */=0A=
! PyThread_release_lock(lock->lock);=0A=
! PyDict_DelItemString(import_locks,mod_name);=0A=
! free(lock);=0A=
! if(!PyDict_Size(import_locks)) {=0A=
! Py_DECREF(import_locks);=0A=
! import_locks =3D NULL;=0A=
! }=0A=
}=0A=
}=0A=
=0A=
#else=0A=
=0A=
! #define lock_import(char* mod_name)=0A=
! #define unlock_import(char* mod_name)=0A=
=0A=
#endif=0A=
=0A=
***************=0A=
*** 1527,1535 ****=0A=
PyObject *fromlist;=0A=
{=0A=
PyObject *result;=0A=
! lock_import();=0A=
result =3D import_module_ex(name, globals, locals, fromlist);=0A=
! unlock_import();=0A=
return result;=0A=
}=0A=
=0A=
--- 1567,1575 ----=0A=
PyObject *fromlist;=0A=
{=0A=
PyObject *result;=0A=
! lock_import(name);=0A=
result =3D import_module_ex(name, globals, locals, fromlist);=0A=
! unlock_import(name);=0A=
return result;=0A=
}=0A=
=0A=
------=_NextPart_000_01F0_01BFBAD6.F6E2F560--