Mailman 3 March 2005 - Python.NET

ANNOUNCE: Python for .NET beta 5 now available
by Brian Lloyd June 2, 2005

June 2, 2005

Python for .NET 1.0 beta 5 is now available - you can download it from: http://www.zope.org/Members/Brian/PythonNet/ Python for .NET is a near-seamless integration of Python with the .NET common language runtime. For more details, see the README: http://www.zope.org/Members/Brian/PythonNet/README.html The beta 5 release features a refactoring of thread management support, making the runtime stable for multi-threaded applications, as well as several improvements and bug-fixes. Special thanks to all who sent in thread-related scenarios and test cases! I have tried to incorporate as many of these as possible into the unit tests - if you still have problems that you suspect may be related to threading or interpreter lock issues, please let me know asap. There is also now a mailing list for discussion, questions and issues related to Python for .NET at: pythondotnet(a)python.org. To subscribe to the mailing list or read the online archives, see: http://mail.python.org/mailman/listinfo/pythondotnet Brian Lloyd brian(a)zope.com V.P. Engineering 540.361.1716 Zope Corporation http://www.zope.com

4 14

[PythonNet] 8/ 4 Resolve "Reentrancy bug"
by Collector: Python for .NET Issue ... March 31, 2005

March 31, 2005

Issue #8 Update (Resolve) "Reentrancy bug" Status Resolved, General/bug medium To followup, visit: http://www.zope.org/Members/Brian/PythonNet/Collector/8 ============================================================== = Resolve - Entry #4 by Brian on Mar 31, 2005 1:21 pm Status: Pending => Resolved Great - thanks for the followup. I'm going to work on a 'tutorial for embedders' for the next release, as this isn't nearly well-documented enough... -BL ________________________________________ = Comment - Entry #3 by Anonymous User on Mar 31, 2005 12:59 pm You're right -- the bug is indeed caused by the lack of a lock acquisition in that code. Thanks for the tip. This item should be closed. > = Comment - Entry #2 by Brian on Mar 31, 2005 9:45 am > > Note that to be thread-safe and reentrancy-safe you > also need to make sure that any managed code you have > that uses Python.Runtime objects or methods is acquiring > the Python lock first. For example: > > public class Example { > > public static int MethodThatCallsPython() { > // have to acquire the GIL to safely call back into > // Python!! > IntPtr state = PythonEngine.AcquireLock(); > > PythonEngine.RunSimpleString("print 'hello'"); > > // done using python for now, so release the GIL... > PythonEngine.ReleaseLock(state); > } > > } > > Forgetting to manage the lock around a callback into Python > is almost certain to cause a NullReferenceException (because > there will be no valid thread state when the callback is > made). > > ________________________________________ > = Request - Entry #1 by Anonymous User on Mar 30, 2005 8:10 pm > > There is a reentrancy problem when calling CLR code from Python that > calls back into Python. The sequence I use to reproduce the problem is > fairly simple: > 1) Execute some Python code either through PyObject.InvokeMethod, or > PythonEngine.RunSimpleString. > 2) From the executed Python code, call into a CLR class. > 3) From the CLR class, execute Python code again. > > Here is some example code that demonstrates the issue: > using System; > using Python.Runtime; > > namespace PDNBug { > public class Example { > [STAThread] > static void Main(string[] args) { > PythonEngine.Initialize(); > PythonEngine.RunSimpleString("from CLR.PDNBug import > Example\nExample.FirstCall()"); > } > > public static void FirstCall () { > Console.WriteLine("First call succeeded."); > PythonEngine.RunSimpleString("from CLR.PDNBug import > Example\nExample.SecondCall()"); > } > > public static void SecondCall () { > Console.WriteLine("Second call succeeded."); > } > } > } > > The output generated is: > First call succeeded. > Traceback (most recent call last): > File "<string>", line 2, in ? > System.NullReferenceException: Object reference not set to an instance of > an object. > at Python.Runtime.Runtime.PyRun_SimpleString(String code) > at Python.Runtime.PythonEngine.RunSimpleString(String code) > at PDNBug.Example.FirstCall() in example.cs:line 15: Object reference > not set to an instance of an object. > > The expected output is: > First call succeeded. > Second call succeeded. > > This issue was previously posted to the mailing list > (http://mail.python.org/pipermail/pythondotnet/2005-January/000235.html). ________________________________________ = Comment - Entry #2 by Brian on Mar 31, 2005 9:45 am Note that to be thread-safe and reentrancy-safe you also need to make sure that any managed code you have that uses Python.Runtime objects or methods is acquiring the Python lock first. For example: public class Example { public static int MethodThatCallsPython() { // have to acquire the GIL to safely call back into // Python!! IntPtr state = PythonEngine.AcquireLock(); PythonEngine.RunSimpleString("print 'hello'"); // done using python for now, so release the GIL... PythonEngine.ReleaseLock(state); } } Forgetting to manage the lock around a callback into Python is almost certain to cause a NullReferenceException (because there will be no valid thread state when the callback is made). ________________________________________ = Request - Entry #1 by Anonymous User on Mar 30, 2005 8:10 pm There is a reentrancy problem when calling CLR code from Python that calls back into Python. The sequence I use to reproduce the problem is fairly simple: 1) Execute some Python code either through PyObject.InvokeMethod, or PythonEngine.RunSimpleString. 2) From the executed Python code, call into a CLR class. 3) From the CLR class, execute Python code again. Here is some example code that demonstrates the issue: using System; using Python.Runtime; namespace PDNBug { public class Example { [STAThread] static void Main(string[] args) { PythonEngine.Initialize(); PythonEngine.RunSimpleString("from CLR.PDNBug import Example\nExample.FirstCall()"); } public static void FirstCall () { Console.WriteLine("First call succeeded."); PythonEngine.RunSimpleString("from CLR.PDNBug import Example\nExample.SecondCall()"); } public static void SecondCall () { Console.WriteLine("Second call succeeded."); } } } The output generated is: First call succeeded. Traceback (most recent call last): File "<string>", line 2, in ? System.NullReferenceException: Object reference not set to an instance of an object. at Python.Runtime.Runtime.PyRun_SimpleString(String code) at Python.Runtime.PythonEngine.RunSimpleString(String code) at PDNBug.Example.FirstCall() in example.cs:line 15: Object reference not set to an instance of an object. The expected output is: First call succeeded. Second call succeeded. This issue was previously posted to the mailing list (http://mail.python.org/pipermail/pythondotnet/2005-January/000235.html). ==============================================================

1 0

[PythonNet] 6/ 2 Resolve "Hang caused by multithreaded program that invokes .NET CLR"
by Collector: Python for .NET Issue ... March 31, 2005

March 31, 2005

Issue #6 Update (Resolve) "Hang caused by multithreaded program that invokes .NET CLR" Status Resolved, Threading/bug critical To followup, visit: http://www.zope.org/Members/Brian/PythonNet/Collector/6 ============================================================== = Resolve - Entry #2 by Brian on Mar 31, 2005 9:54 am Status: Pending => Resolved This issue is resolved in beta 5 - thanks for the test case (I put a condensed version of it into the unit tests). -BL ________________________________________ = Request - Entry #1 by Anonymous User on Mar 25, 2005 1:04 pm Hang caused by multithreaded program accessing CLR methods. I successfully reproduced the state with a very simple application. Here is very dumb python program that uses the CLR: from CLR.System import String class cSharpCaller( object ): def __init__( self ): print "Created a cSharpCaller instance" self.__cSharpString = None def createCSharpString( self, str ): self.__cSharpString = String( str ) def returnCSharpString( self ): return self.__cSharpString.ToString() And if I test this with this (dumb) threethreaded testprogram: import threading, thread, time from cSharpCaller import cSharpCaller def start_threads(amount=5): for i in range(amount): thread = threading.Thread(target=process_thread ) print "starting thread ", thread.getName() thread.start() def process_thread( ): print "thread started " for i in range( 2 ): print "Hi, I'm a thread" time.sleep( 1 ) cs = cSharpCaller( ) cs.createCSharpString( "created from a thread" ) print "cs.returnCSharpString() = ", cs.returnCSharpString() print "thread %s ended" start_threads( 3 ) I get this output: starting thread thread0 thread started Hi, I'm thread %s starting thread thread1 thread started Hi, I'm thread %s starting thread thread2 thread started Hi, I'm thread %s Hi, I'm thread %s Hi, I'm thread %s Hi, I'm thread %s Created a cSharpCaller instance Created a cSharpCaller instance cs.returnCSharpString() = and I have to CTRL+Pause to get out of the hang. Clearly hangs on the first String.ToString() method call. ==============================================================

1 0

application crash.
by Stan Pinte March 31, 2005

March 31, 2005

hello, the following application (python/C#) is working properly, when I do not run simpy, a python framework. Once I enable, I get an immediate crash. See enclosed for runnable/compilable example. (run "python test.py"). To run, it needs the Simpy framework, GPL, that you can get there: http://simpy.sourceforge.net/ Simpy makes heavy uses of the "yield" construct. as soon as I uncomment that line in the script: #simulate() I get an immediate crash. I see no reason for this crash. I will open a tracker item. Stan.

2 3

Python 2.4 hang when calling .NET dll function
by Torgeir Johansen March 25, 2005

March 25, 2005

Great, Brian. I successfully reproduced the state with a very simple application. Here is very dumb python program that uses the CLR: from CLR.System import String class cSharpCaller( object ): def __init__( self ): print "Created a cSharpCaller instance" self.__cSharpString = None def createCSharpString( self, str ): self.__cSharpString = String( str ) def returnCSharpString( self ): return self.__cSharpString.ToString() And if I test this with this (dumb) threethreaded testprogram: import threading, thread, time from cSharpCaller import cSharpCaller def start_threads(amount=5): for i in range(amount): thread = threading.Thread(target=process_thread ) print "starting thread ", thread.getName() thread.start() def process_thread( ): print "thread started " for i in range( 2 ): print "Hi, I'm a thread" time.sleep( 1 ) cs = cSharpCaller( ) cs.createCSharpString( "created from a thread" ) print "cs.returnCSharpString() = ", cs.returnCSharpString() print "thread %s ended" start_threads( 3 ) I get this output: starting thread thread0 thread started Hi, I'm thread %s starting thread thread1 thread started Hi, I'm thread %s starting thread thread2 thread started Hi, I'm thread %s Hi, I'm thread %s Hi, I'm thread %s Hi, I'm thread %s Created a cSharpCaller instance Created a cSharpCaller instance cs.returnCSharpString() = and I have to CTRL+Pause to get out of the hang. Clearly hangs on the first String.ToString() method call. Regards, Torgeir

1 0

Python 2.4 hang when calling .NET dll function
by Torgeir Johansen March 25, 2005

March 25, 2005

I have been able to successfully use my .NET DLL (written in C#) using python 2.4 and the Python for .NET DLLs. I basically have one python module implementing a class that is using methods from my C# DLLs. My test application (singlethreaded) ran great, but when integrating the python module that uses the .NET DLLs in a multithreaded program, one call to a .NET function just hangs. All threads are hanging, and I get no response using CTRL+C. I have debugged my application using ntsd, and I do not see any bad behaviour from my called DLLs. Are there some issues with locking/unlocking mechanism when calling functions from .NET DLLs ? Regards Torgeir

3 2

deadlock
by Stan Pinte March 23, 2005

March 23, 2005

hello, I am running Simpy (python simulation framework) within pythondotnet, and, even though this process is single-thread, it hangs misteriously, in an unpredictable way... Python console cease to respond to Ctrl-C events... Here is the current Thread status: Thread Start Address: Symbol Name: Line Number: PC: mscoree!_CorExeMain() + 0x0 --- 7917D08C Thread Stack: ntdll ! KiFastSystemCallRet() + 0x KERNEL32 ! WaitForSingleObject() + 0x12 python24 ! PySys_WriteStderr() + 0x14d python24 ! PyTuple_Type() + 0x0 as the entry point in the hanging thread (higher on stack) is PyTuple_Type() and as PyTuple is defined in C# (src/runtime/PyTuple.cs), I suspect this might be the cause of my problem. [PythonNet-1.0-beta4]> grep -nr "PyTuple_Type" . Binary file ./DLLs/_socket.pyd matches Binary file ./python24.dll matches [PythonNet-1.0-beta4]> However, I would like to be able to go higher in the stack, to see what caused this deadlock. Any proposed strategy to guess what happened, or to track down the problem? thanks a lot, Stan.

1 3

build a debuggable version of python dot net
by Stan Pinte March 23, 2005

March 23, 2005

hello, when trying to debug a pythondotnet program in WinDbg, it gives me the following error: ModLoad: 1e000000 1e1ca000 C:\export\PythonNet-1.0-beta4\python24.dll HEAP[python.exe]: Invalid Address specified to RtlFreeHeap( 00140000, 071B5994 ) (ecc.cc8): Break instruction exception - code 80000003 (first chance) eax=071b598c ebx=071b598c ecx=7c91eb05 edx=0012e7d6 esi=00140000 edi=071b598c eip=7c901230 esp=0012e9e0 ebp=0012e9e4 iopl=0 nv up ei pl nz na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 ntdll!DbgBreakPoint: 7c901230 cc int 3 --> Any idea how I can build a debuggable version of pythondotnet? thanks, Stan.

1 0

(no subject)
by Stan Pinte March 23, 2005

March 23, 2005

hello, I am running Simpy (python simulation framework) within pythondotnet, and, even though this process is single-thread, it hangs misteriously, in an unpredictable way... Python console cease to respond to Ctrl-C events... Here is the current Thread status: Thread Start Address: Symbol Name: Line Number: PC: mscoree!_CorExeMain() + 0x0 --- 7917D08C Thread Stack: ntdll ! KiFastSystemCallRet() + 0x KERNEL32 ! WaitForSingleObject() + 0x12 python24 ! PySys_WriteStderr() + 0x14d python24 ! PyTuple_Type() + 0x0 as the entry point in the hanging thread (higher on stack) is PyTuple_Type() and as PyTuple is defined in C# (src/runtime/PyTuple.cs), I suspect this might be the cause of my problem. [PythonNet-1.0-beta4]> grep -nr "PyTuple_Type" . Binary file ./DLLs/_socket.pyd matches Binary file ./python24.dll matches [PythonNet-1.0-beta4]> However, I would like to be able to go higher in the stack, to see what caused this deadlock. Any proposed strategy to guess what happened, or to track down the problem? thanks a lot, Stan.

1 0

from CLR.mynamespace import * doesn't work
by Stan Pinte March 22, 2005

March 22, 2005

hello, from CLR.mynamespace import * doesn't work. I have to import every class manually. Any hint? Note: would'it be worth opening a bugzilla-like system for pythondotnet? thanks a lot, Stan.

2 1