[ python-Bugs-1590864 ] import deadlocks when using PyObjC threads

Mon Jul 9 10:15:28 CEST 2007

Bugs item #1590864, was opened at 2006-11-05 17:06
Message generated for change (Comment added) made by ronaldoussoren
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1590864&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Michael Tsai (michaeltsai)
Assigned to: Nobody/Anonymous (nobody)
Summary: import deadlocks when using PyObjC threads

Initial Comment:
When I use subprocess.py from a child thread, sometimes it deadlocks. I 
determined that the new process is blocked during an import:

#0  0x90024427 in semaphore_wait_signal_trap ()
#1  0x90028414 in pthread_cond_wait ()
#2  0x004c77bf in PyThread_acquire_lock (lock=0x3189a0, waitflag=1)  
at Python/thread_pthread.h:452
#3  0x004ae2a6 in lock_import () at Python/import.c:266
#4  0x004b24be in PyImport_ImportModuleLevel (name=0xaad74 "errno",  
globals=0xbaed0, locals=0x502aa0, fromlist=0xc1378, level=-1) at  
Python/import.c:2054
#5  0x0048d2e2 in builtin___import__ (self=0x0, args=0x53724c90,  
kwds=0x0) at Python/bltinmodule.c:47
#6  0x0040decb in PyObject_Call (func=0xa94b8, arg=0x53724c90,  
kw=0x0) at Objects/abstract.c:1860

and that the code in question is in os.py:

def _execvpe(file, args, env=None):
     from errno import ENOENT, ENOTDIR

I think the problem is that since exec (the C function) hasn't yet been 
called in the new process, it's inherited from the fork a lock that's already 
held. The main process will eventually release its copy of the lock, but this 
will not unlock it in the new process, so it deadlocks.

If I change os.py so that it imports the constants outside of  
_execvpe, the new process no longer blocks in this way.

This is on Mac OS X 10.4.8.

----------------------------------------------------------------------

>Comment By: Ronald Oussoren (ronaldoussoren)
Date: 2007-07-09 10:15

Message:
Logged In: YES 
user_id=580910
Originator: NO

Do you have sample code that reproduces this problem? (Not necessarily
code that has this problem 100% of the time)

----------------------------------------------------------------------

Comment By: Kosuha (kosuha)
Date: 2007-02-27 17:54

Message:
Logged In: YES 
user_id=1684258
Originator: NO

I confirm that problem with deadlock on execution of
PyImport_ImportModuleLevel exists. Here is a working example:

1) Python was embbeded with C++ application using Python for scripting
support:

------------------------------------------------------------------
PyThreadState * InitThreadScripting()
{
	ASSERT_KOBOLD( g_pyMainThreadState );

	// get the global lock
	PyEval_AcquireLock();
	// get a reference to the PyInterpreterState
	PyInterpreterState * mainInterpreterState = g_pyMainThreadState->interp;
	ASSERT_KOBOLD( mainInterpreterState );
	// create a thread state object for this thread
	PyThreadState * myThreadState = PyThreadState_New(mainInterpreterState);
	// free the lock
	PyEval_ReleaseLock();

	return myThreadState;
}

---------------------------------------------------------------------
void DeadLock()
{
Py_Initialize();
PyEval_InitThreads();
g_AiGlobals     = new Py::Dict();
g_pyInterpState = PyInterpreterState_New();

// save a pointer to the main PyThreadState object
g_pyMainThreadState = PyThreadState_Get();
ASSERT_KOBOLD( g_pyMainThreadState );

// release the lock
PyEval_ReleaseLock();

g_pyWorldThreadState = InitThreadScripting();

// import sys
// sys.path.append ("./scripts")
// 
PyObject	*p = PyImport_ImportModuleEx ("sys", **g_AiGlobals, NULL, NULL);
Py::Module	mod_sys (p);
Py::List	path = mod_sys.getAttr ("path");
path.append (Py::String ("scripts"));
path.append (Py::String ("scripts/sys"));
path.append (Py::String ("../../scripts"));
path.append (Py::String ("../../scripts/sys"));
Py_XDECREF (p);

// HERE IT OCCURS //
Log.ScriptsSrc("Python", "Running startup python scripts...");
PyObject *p = PyImport_ImportModuleEx ("startup", **g_AiGlobals, NULL,
NULL); // <<< Here
if (reload) PyImport_ReloadModule (p);
Py::Module	module (p);
Py_XDECREF (p);
}

Execution locks right on PyImport_ImportModuleEx. 

Code from sturtup.py:
------------------------------------------------------------------------------------
# This module is sturtup script.
# Here we are redirecting output and checking for server version.
################################################################################

import sys
from consts import *   # Import of constants
import config as cfg   # Import of configuration constants
reload(cfg)  

################################################################################

#
----------------------------------------------------------------------------------------------------------------------------------------------------------------
class OurLogStdErr:
    def write (self, txt):
        printLog (txt, PRINT_ERROR)

#
----------------------------------------------------------------------------------------------------------------------------------------------------------------
class OurLogStdOut:
    def write (self, txt):
        printLog (txt)

#
----------------------------------------------------------------------------------------------------------------------------------------------------------------
def CheckServerVersion():
    # Checking for server build
    if GetServerBuild() < MIN_SERVER_BUILD:
        printLog( "YOU ARE TRYING TO RUN PYTHON SCRIPTS ON OUTDATED SERVER
BUILD!\
\nREQUIRED SERVER BUILD: %s\
\nPlease Update your server core before running server!\
\nScripting Engine will be Shut Down!"\
        % (MIN_SERVER_BUILD), PRINT_ERROR )

        killScripting()

#
----------------------------------------------------------------------------------------------------------------------------------------------------------------
def GetScriptsVersion():
	return SCRIPTS_VERSION

################################################################################
# Startup code here:

# Redirecting errors:
sys.stderr = OurLogStdErr()

# Redirecting output:
sys.stdout = OurLogStdOut()

---------------------------------------------------------------------------------------

----------------------------------------------------------------------

Comment By: Peter Åstrand (astrand)
Date: 2007-01-13 23:42

Message:
Logged In: YES 
user_id=344921
Originator: NO

Since both the reporter and I believes that this is not a bug in the
subprocess module, I'm stepping back. 

----------------------------------------------------------------------

Comment By: Michael Tsai (michaeltsai)
Date: 2007-01-07 18:09

Message:
Logged In: YES 
user_id=817528
Originator: YES

I don't have time at the moment to write sample code that reproduces this.
But, FYI, I was using PyObjC to create the threads. It might not happen
with "threading" threads. And second, I think it's a bug in os.py, not in
subprocess.py. Sorry for the confusion.

----------------------------------------------------------------------

Comment By: Peter Åstrand (astrand)
Date: 2007-01-07 15:10

Message:
Logged In: YES 
user_id=344921
Originator: NO

Can you provide a test case or sample code that demonstrates this problem?

I'm a bit unsure of if this really is a subprocess bug or a more general
Python bug. 

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1590864&group_id=5470