[ python-Bugs-689895 ] Imports can deadlock

[SourceForge.net]

Bugs item #689895, was opened at 2003-02-20 00:39
Message generated for change (Comment added) made by sitbon
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=689895&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: M.-A. Lemburg (lemburg)
Assigned to: Nobody/Anonymous (nobody)
Summary: Imports can deadlock

Initial Comment:
[Just copying some discussion details from python-dev
into this
bug report so that we can reopen it to Python 2.4]

[MAL]

>> I wonder whether a general lock such as the one used
in import
>> is such a good idea. Perhaps it should only lock the
importing
>> of a specific module, keeping the locks in a
dictionary indexed by
>> module name instead of a static C variable ?!


I've been thinking along the same lines.  We could make
the import
locking much finer-grained, and limit the blocking only
to threads
that are importing a module that is in the middle of
being loaded by
some other thread.

But this is hard work, and I suggest that we put this
off until Python
2.4 so we can do it right.


>> Then again it's hard to know the real name of the
module being
>> searched before finding it...


There could be a short-lived lock for that problem.


>> I see a more general problem here: the lock prevent
starting
>> up threaded applications which use client-server
logic between
>> the threads. If the application's main thread starts
a client
>> thread as a result of an import which then tries to
import
>> other Python modules, you have a deadlock. (At least
that's how
>> I understand the current implementation.)


Correct.


>> Don't know about others, but I frequently use the
idiom of
>> placing the server's main code in a separate module
and then
>> have small startup script importing this module.
This kind
>> of setup is also advertised for CGI programs, so it
may not
>> be uncommon out there.


We had this problem with Zope2 -- I don't know why the
import lock
didn't bite us before, but we decided to change the
Zope startup code
so that you have to import Zope first and then,
separately, make a
call to start it.  You could do the same for your
application, and I
recommend that Mark does the same for his.

--Guido van Rossum (home page:
http://www.python.org/~guido/)



----------------------------------------------------------------------

Comment By: Phillip Sitbon (sitbon)
Date: 2005-12-08 13:50

Message:
Logged In: YES 
user_id=1303233

Has anyone looked further into this? I seem to be having the same problem with a simple multithreaded program that imports from within thread functions. I didn't have the problem (at least not too often, but still in some cases) with 2.4.1 but now 2.4.2 has me deadlocking on the imports every time. It's a frustrating problem!

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg)
Date: 2003-02-20 00:40

Message:
Logged In: YES 
user_id=38388

[eric at enthought.com]

>[MAL]
>
>>> > I wonder whether a general lock such as the one used
in import
>>> > is such a good idea. Perhaps it should only lock the
importing
>>> > of a specific module, keeping the locks in a
dictionary indexed by
>>> > module name instead of a static C variable ?!


I am interested in this also.


>> 
>> [Guido]
>> I've been thinking along the same lines.  We could make
the import
>> locking much finer-grained, and limit the blocking only
to threads
>> that are importing a module that is in the middle of
being loaded by
>> some other thread.
>> 
>> But this is hard work, and I suggest that we put this off
until Python
>> 2.4 so we can do it right.


I ran into this exact thing when trying to get wxPython
windows (data
plots) to co-exist peacefully on the screen in parallel with an
interactive shell.  The module is called gui_thread, and we
use it in
SciPy.

http://www.scipy.org/site_content/tutorials/gui_thread
  
The command line took the main thread, and a wxPython app
was started in
the background thread.  The first import of wxPython had to
occur in the
background thread for wxPython's to be happy.  Initially, I
put a lock
in the import of gui_thread which waited for the wxPython
import to
complete before allowing the gui_thread import to complete.
 If 'import
gui_thread' is the first statement executed, this guarantees
that
wxPython is always imported first in the background thread.
Unfortunately, it also causes deadlock because of the import
lock.

A discussion that occurred about this problem on the
thread-sig with is
summarized here:

http://www.scipy.org/site_content/tutorials/import_thread_lock_discussio
n

It includes an alternative version of several import.c routines
(compared to 1.5.2 I believe) that solved my problem as
proposed by MAL
above and passed all the regression tests at the time.  I
just plugged
the code into the 2.3CVS, and it mainly works.  But, there
are some
import related errors in the regression tests for: 

test_threaded_import
test_loggingllbacks


I could spend some time on these if there is interest in
getting this in
the current release.  Also, this code would need review by
someone that
is an expert on the subtleties of the import code to make
sure it is
sound.  Let me know, and I'll submit a patch with of the
current code.

eric


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=689895&group_id=5470