[ python-Bugs-921077 ] embedding in multi-threaded & multi sub-interpreter environ

[SourceForge.net]

Bugs item #921077, was opened at 2004-03-22 07:39
Message generated for change (Comment added) made by bcannon
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=921077&group_id=5470

>Category: Extension Modules
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Atul (atulkshirsagar)
>Assigned to: Brett Cannon (bcannon)
Summary: embedding in multi-threaded & multi sub-interpreter environ

Initial Comment:
I am embedding python in my C++ application. I am 
using Python *2.3.2* with a C++ extention DLL in multi-
threaded environment. I am using SWIG-1.3.19 to 
generate C++ to Python interface.

Now to explain it in details,
1. Python initialization [Py_Initialize()] and finalization 
[Py_Finalize()] is done in the *main* thread.
2. For each new thread I create a separate sub-
interpreter [Py_NewInterpreter()]. 
3. Using PyRun_String("import myModule"...) before 
execution of python script, extention module is imported.
4. Each thread executes *multiple* python script using 
PyEval_EvalCode() using the class objects in my 
extention DLL.
5. Each sub-interpreter is destroyed [Py_EndInterpreter
()] at the end of that particular thread.

I am observing that;
As explained above when multiple threads are running. 
And as one of these threads finishes, in other running 
threads I start getting 
"TypeError: 'NoneType' object is not callable" error on 
the methods called on class objects in extention module. 

The same code *works fine* with Python 2.2.2.

I have found these links more or less talking about the 
same problem
migrating from 2.2 to 2.3.
http://mail.python.org/pipermail/python-dev/2003-
September/038237.html
http://mail.python.org/pipermail/python-list/2004-
February/206851.html
http://mail.python.org/pipermail/python-list/2004-
January/204040.html

I *guess* what is happening is global variables are 
zapped to "NoneType"
when one thread finishes and other thread trying to 
access them through the
Python script (step 4.) this error is generated. But it 
*works* sometimes
when(*guess*) the running thread is at step 3. and by 
importing the  module
the global variables are re-initialized for "Type" 
information.

I tried using reload(myModule) to solve the problem but 
that is generating
big memory leak every time it is called.

Is this a know issue with 2.3 (interpreter?) ? Or is there 
a change for 2.3 in the way embedding should be done 
in a multi-threaded and multi-sub-interpreter 
environment ?


----------------------------------------------------------------------

>Comment By: Brett Cannon (bcannon)
Date: 2004-08-07 12:46

Message:
Logged In: YES 
user_id=357491

Have you tried this with the fixes added in 2.3.4 or 2.4?  Should print out 
proper tracebacks now.

----------------------------------------------------------------------

Comment By: Atul (atulkshirsagar)
Date: 2004-04-05 13:16

Message:
Logged In: YES 
user_id=1003784

Hello,
Did you get a chance to take a look at this yet ?
Thanks,
Atul

----------------------------------------------------------------------

Comment By: Atul (atulkshirsagar)
Date: 2004-04-01 12:12

Message:
Logged In: YES 
user_id=1003784

Hello Martin,

I am attaching the sample program which demonstrates the 
issue;
1. I compiled it on Windows 2000 with VC++ compiler.
2. There is a "Extension Module" [flpythonmodulesu(d).dll] 
which is used by "Main" program

=========================================
Extension Module (extModule.dsp project) 
generates : (*u.* - release, *ud.* - debug)
1.  dll _flpythonmodulesud.dll
2. python script flpythonmodulesud.py 
------------------------------------------------------
1. Module has class FlPythonString and class Record defined
2. Using interface file include\flpythonmodulesud.i, SWIG-
1.3.19 generates include\flpythonmodulesud_wrap.cxx 
===========================================


=========================================
Main sample program (multiSub.dsp project) 
generates : 
1.  executable multiSub.exe
------------------------------------------------------
1. Links in the extension DLL
2. Initializes and De-initializes python in main thread
3. Main thread creates 2 separate threads: 
   (a) Each thread creating a new sub-interpreter
   (b) importing extension module
   (c) Calling following EXPRESSION using PyEval_EvalCode(),
       Thread 1:
myVar = FlPythonStringPtr(STRObject)
myRecord = myVar.NewRecord()
Thread=str(myVar.Get())
record=str(myRecord.GetValue())
print 'Inside EXPRESSION ' + Thread + ':' + 
record;myVar.DeleteRecord(myRecord)
       
    Thread 2:
myVar = FlPythonStringPtr(STRObject)
for recordNum in range(1, 11):
 myRecord = myVar.NewRecord()
 Thread=str(myVar.Get())
 record=str(myRecord.GetValue())
 print 'Inside EXPRESSION ' + Thread + ':' + 
record;myVar.DeleteRecord(myRecord)
      (d) Destroying sub-interpreter.

Comments in the code explain details about each step.
===========================================

==========================================
Console Output messages explanations
(Attaching a sample console output for the issue
[consoleop.txt])
-----------------------------------------------------
-Thread *ID*:IMPORT Module Called:*ITERATION*
-Inside EXPRESSION *Thread*:*Record::GetValue()
[hardcoded to 1000]*
-Thread *ID*:EXPRESSION Done:*ITERATION*
-Thread *ID*:ENDINTERPRETER Called
-Thread *ID*:EXPRESSION Failed!:*ITERATION*

Check the error message demonstrating the issue:
-----------------------------------------------
"Thread 2:EXPRESSION Failed!:3
Traceback (most recent call last):
  File "EXPRESSION", line 4, in ?
  
File "E:\VssLocal\Isis\python23_issue\multiSub\Debug\flpython
modulesud.py", line 69, in Get
    def Get(*args): return apply
(_flpythonmodulesud.FlPythonString_Get,args)
  
File "E:\VssLocal\Isis\python23_issue\multiSub\Debug\flpython
modulesud.py", line 51, in __init__
    _swig_setattr(self, Record, 'this', this)
TypeError: 'NoneType' object is not callable"

==========================================



Martin, Thanks for taking a look at the issue which has proved 
to be a major hurdle for our release and made us switch back 
to Python 2.2.2. Python 2.2.2 in itself has some issues for us; 
so if this can get fixed we would desperately be seeking a 
patch on Python 2.3.

Let me know if you need any more inputs from my side.

Thanks again,
Atul

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2004-03-29 14:28

Message:
Logged In: YES 
user_id=21627

Can you provide an example that demonstrates the supposed bug?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=921077&group_id=5470