Import lock knowledge required!
I've found a bug related to the import lock causing a deadlock in 2.3. The bug can be found at http://www.python.org/sf/683658 The bug is fairly self-explanatory, so I won't repeat it here. The proposed patch exposes the import lock to the Python C API, allowing optionally a non-blocking attempt to acquire the lock. I should note that the only reason I propose this is because I can see no other option! I would appreciate it if people knowledgeable about the import lock and its implications could have a quick look at this bug, and comment on my proposed patch. Thanks, Mark.
Mark Hammond wrote:
I've found a bug related to the import lock causing a deadlock in 2.3. The bug can be found at http://www.python.org/sf/683658
The bug is fairly self-explanatory, so I won't repeat it here. The proposed patch exposes the import lock to the Python C API, allowing optionally a non-blocking attempt to acquire the lock. I should note that the only reason I propose this is because I can see no other option!
I would appreciate it if people knowledgeable about the import lock and its implications could have a quick look at this bug, and comment on my proposed patch.
I think we're running into a similar bootstrap problem as we did when we had exceptions coded in Python. The solution back then was to recode it in C. Perhaps the warnings module should get the same treatment ?! -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 15 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 45 days left EuroPython 2003, Charleroi, Belgium: 129 days left
On zaterdag, feb 15, 2003, at 16:49 Europe/Amsterdam, M.-A. Lemburg wrote:
I think we're running into a similar bootstrap problem as we did when we had exceptions coded in Python. The solution back then was to recode it in C. Perhaps the warnings module should get the same treatment ?!
There are other modules with the same potential problem. For instance, the code that prints MacOS error messages imports a Python module to get at the texts of the messages. I haven't had any reports of deadlocks, but I haven't looked for them either, so it may be possible this code gets called with the import lock held. And at first glance code using cobject's PyCobject_import() may be at risk too (only at first glance: I didn't check to see whether such code is ever executed with with import lock held). -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -
Jack Jansen wrote:
On zaterdag, feb 15, 2003, at 16:49 Europe/Amsterdam, M.-A. Lemburg wrote:
I think we're running into a similar bootstrap problem as we did when we had exceptions coded in Python. The solution back then was to recode it in C. Perhaps the warnings module should get the same treatment ?!
There are other modules with the same potential problem. For instance, the code that prints MacOS error messages imports a Python module to get at the texts of the messages. I haven't had any reports of deadlocks, but I haven't looked for them either, so it may be possible this code gets called with the import lock held.
And at first glance code using cobject's PyCobject_import() may be at risk too (only at first glance: I didn't check to see whether such code is ever executed with with import lock held).
I wonder whether a general lock such as the one used in import is such a good idea. Perhaps it should only lock the importing of a specific module, keeping the locks in a dictionary indexed by module name instead of a static C variable ?! Then again it's hard to know the real name of the module being searched before finding it... I see a more general problem here: the lock prevent starting up threaded applications which use client-server logic between the threads. If the application's main thread starts a client thread as a result of an import which then tries to import other Python modules, you have a deadlock. (At least that's how I understand the current implementation.) Don't know about others, but I frequently use the idiom of placing the server's main code in a separate module and then have small startup script importing this module. This kind of setup is also advertised for CGI programs, so it may not be uncommon out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 16 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 44 days left EuroPython 2003, Charleroi, Belgium: 128 days left
[MAL]
I wonder whether a general lock such as the one used in import is such a good idea. Perhaps it should only lock the importing of a specific module, keeping the locks in a dictionary indexed by module name instead of a static C variable ?!
I've been thinking along the same lines. We could make the import locking much finer-grained, and limit the blocking only to threads that are importing a module that is in the middle of being loaded by some other thread. But this is hard work, and I suggest that we put this off until Python 2.4 so we can do it right.
Then again it's hard to know the real name of the module being searched before finding it...
There could be a short-lived lock for that problem.
I see a more general problem here: the lock prevent starting up threaded applications which use client-server logic between the threads. If the application's main thread starts a client thread as a result of an import which then tries to import other Python modules, you have a deadlock. (At least that's how I understand the current implementation.)
Correct.
Don't know about others, but I frequently use the idiom of placing the server's main code in a separate module and then have small startup script importing this module. This kind of setup is also advertised for CGI programs, so it may not be uncommon out there.
We had this problem with Zope2 -- I don't know why the import lock didn't bite us before, but we decided to change the Zope startup code so that you have to import Zope first and then, separately, make a call to start it. You could do the same for your application, and I recommend that Mark does the same for his. --Guido van Rossum (home page: http://www.python.org/~guido/)
[MAL]
I wonder whether a general lock such as the one used in import is such a good idea. Perhaps it should only lock the importing of a specific module, keeping the locks in a dictionary indexed by module name instead of a static C variable ?!
I am interested in this also.
[Guido] I've been thinking along the same lines. We could make the import locking much finer-grained, and limit the blocking only to threads that are importing a module that is in the middle of being loaded by some other thread.
But this is hard work, and I suggest that we put this off until Python 2.4 so we can do it right.
I ran into this exact thing when trying to get wxPython windows (data plots) to co-exist peacefully on the screen in parallel with an interactive shell. The module is called gui_thread, and we use it in SciPy. http://www.scipy.org/site_content/tutorials/gui_thread The command line took the main thread, and a wxPython app was started in the background thread. The first import of wxPython had to occur in the background thread for wxPython's to be happy. Initially, I put a lock in the import of gui_thread which waited for the wxPython import to complete before allowing the gui_thread import to complete. If 'import gui_thread' is the first statement executed, this guarantees that wxPython is always imported first in the background thread. Unfortunately, it also causes deadlock because of the import lock. A discussion that occurred about this problem on the thread-sig with is summarized here: http://www.scipy.org/site_content/tutorials/import_thread_lock_discussio n It includes an alternative version of several import.c routines (compared to 1.5.2 I believe) that solved my problem as proposed by MAL above and passed all the regression tests at the time. I just plugged the code into the 2.3CVS, and it mainly works. But, there are some import related errors in the regression tests for: test_threaded_import test_loggingllbacks I could spend some time on these if there is interest in getting this in the current release. Also, this code would need review by someone that is an expert on the subtleties of the import code to make sure it is sound. Let me know, and I'll submit a patch with of the current code. eric
participants (5)
-
eric jones -
Guido van Rossum -
Jack Jansen -
M.-A. Lemburg -
Mark Hammond