[IronPython] IronPython 2.6.1 & ctypes, C library, and debugging with VS2008

[TP]

On Sun, Jul 18, 2010 at 10:47 PM, TP <wingusr at gmail.com> wrote:
> On Sun, Jul 18, 2010 at 2:38 PM, Dino Viehland <dinov at microsoft.com> wrote:
>> TP wrote:
>>> I'm using IronPython 2.6.1 for Net 2.0 and VS2008 on Windows XP SP3.
>>>
>>> I have a python script that lets me access the Leptonica Image
>>> Processing C library (http://leptonica.com/) using the ctypes module.
>>>
>>> Everything seems to work fine with cpython 2.6. I can correctly load
>>> leptonlib.dll, create leptonica PIX images, manipulate them, and even
>>> display them using PyQT. I can use VS2008 to put breakpoints on
>>> leptonica C functions and step through the C code in the VS2008
>>> debugger.
>>>
>>> I thought I'd give IronPython a try since it also has ctypes, but I
>>> ran into a number of problems.
>>>
>>> The leptonlib.dll must be loading since I am able to correctly get
>>> back the library's version string. That is the following works:
>>>
>>>     _getLeptonlibVersion = _leptonlib.getLeptonlibVersion
>>>     _getLeptonlibVersion.restype = ctypes.c_void_p
>>>     addr = _getLeptonlibVersion()
>>>     version = ctypes.string_at(addr)
>>>
>>> However the following code doesn't seem to work under IronPython 2.6.1:
>>>
>>>     pix = Pix(dimensions=(10,20,1))
>>>
>>> where essentially the following is called:
>>>
>>>     _pix = ctypes.c_void_p()
>>>     _pix.value = _leptonlib.pixCreate(width, height, depth)
>>>     self._pix = _pix
>>>
>>> and I want to treat _pix as an opaque ptr I just hand back to
>>> leptonica whenever it needs it. For example:
>>>
>>>     def height(self):
>>>         """Get height of pix in pixels."""
>>>         return _leptonlib.pixGetHeight(self._pix)
>>>
>>> Which works with cpython but returns something other than 20 with
>>> IronPython.
>>>
>>> By specifying "-X:Debug -X:FullFrames -X:Tracing leptonica.py" as the
>>> arguments to C:\Program Files\IronPython 2.6\ipy.exe I can get
>>> IronPython to correctly stop at places in my script where I put:
>>>
>>>     import pdb
>>>     pdb.set_trace()
>>>
>>> when I debug leptonlibd.dll with VS2008. However, my breakpoints on
>>> leptonica's C functions never seem to get hit. This works fine if I
>>> instead use python26.exe as command to launch when debugging.
>>
>> You should only need -X:Debug to get debugging under VS.  pdb uses a
>> more Pythonic form of debugging but there's no support for it in VS.
>
> Let me be clear here. I am using the VS2008 Solution that I use to
> create leptonlib.dll. I am debugging that dll by right-clicking its
> project and setting its Configuration Properties | Debugging tab to:
>
>   Command: C:\Program Files\IronPython 2.6\ipy.exe
>   Arguments: -X:Debug -X:FullFrames -X:Tracing -i leptonica.py
>   Working Directory: C:\leptonica\
>
> If I only use -X:Debug as you suggest I get the following error:
>
>   Traceback (most recent call last):
>     File "leptonica.py", line 458, in <module>
>     File "leptonica.py", line 176, in __init__
>     File "C:\Program Files\IronPython 2.6\Lib\pdb.py", line 1220, in set_trace
>   AttributeError: 'module' object has no attribute '_getframe'>>>
>
> Googling, I determined that I needed to add at least -X:FullFrames,
> and by trial and error I found I also needed -X:Tracing. (I wasn't
> able to find any documentation on ipy.exe's command line switches? I
> just ran "ipy.exe -h" to dump out the short help description and took
> a wild guess at which might be useful)
>
>>>
>>> Ideally I like to have the VS Debugger stop in leptonlibd.dll's
>>> pixCreate() C function just before it returns its value so I can
>>> compare that to what I get back on the IronPython side.
>>>
>>> So how's does one use VS2008 to step through C functions in DLLs that
>>> are loaded by IronPython and the ctypes module?
>>
>> Do you have symbols (.PDB files) for leptonlibd.dll?  You'll need symbols
>> to be able to step through the C code.  You can still probably set a breakpoint
>> in the function because it's DLL exported - I think you can debug->new
>> breakpoint and enter leptonlibd!pixCreate.  Even w/o symbols you could
>> step through the assembly.
>
> leptonlibd.dll is created with the C7 Compatible (/Z7) compiler
> switch. This embeds the debugging info directly in the DLL so no .pdb
> file is produced. It also does NOT use pre-compiled headers.
>
> I might also point out that the same exact .dll will hit breakpoints
> if debugged with python26.exe rather than ipy.exe. Perhaps since
> ipy.exe is built on top of .NET the VS2008 debugger handles any .dll's
> loaded by it differently? I know for example that the VS2008 debugger
> didn't like trying to run the IronPython for NET 4.0 version of
> ipy.exe (I gather you have to use VS2010 if you want to do that).
>
> Trying to set a breaking at leptonlibd!pixCreate didn't work.
>
>>>
>>> Secondly, am I using ctypes wrong, and does my code only work by
>>> happenstance for cpython.
>>
>> I don't see anything particularly wrong on your side and this looks like
>> a pretty simple call.  I would assume the C functions are defined as:
>>
>> void* pixCreate(int width, int height, int depth);
>> int pixGetHeight(void* pixel);
>
> That's correct.
>
>>
>> I would hope we're getting all of this right as it seems like simple stuff
>> that should be tested somewhere.  My first guess would be maybe we're
>> getting the calling convention wrong.  Maybe it needs to be a WinDLL instead
>> of a CDLL?  Maybe there's a check in the Python code for sys.platform which
>> is looking for win32 and uses WinDLL to open the DLL instead of CDLL on
>> Windows?
>
> More information on the problem:
>
> I decided to explicitly set the restype even though it's the default
> return type. I get the same exact behavior as my original code. Works
> with cpython, doesn't work with IronPython 2.6.1.
>
> If I have with the following python code:
>
>   _pix = ctypes.c_void_p()
>   _pixCreate = _leptonlib.pixCreate
>   _pixCreate.restype = ctypes.c_int
>   import pdb
>   pdb.set_trace()
>
>   _pix.value = _pixCreate(width, height, depth)
>
> I can do the following in the Command Window after pdb breaks into the script:
>
>   > leptonica.py(180)__init__()
>   -> _pix.value = _pixCreate(width, height, depth)
>   (Pdb) n
>   > leptonica.py(181)__init__()
>   -> self._pix = _pix
>   (Pdb) _pix.value
>   *** AttributeError: 'cell' object has no attribute 'value'
>
> This is a bit strange since I just assigned to _pix.value.
> Investigating further:
>
>   (Pdb) _pix
>   <cell at 43: c_void_p object at 44>
>
> So instead of assigning to _pix.value (that is changing an attribute
> of _pix), IronPython is clobbering _pix?
>
> If I instead separately assign the result of pixCreate() to a
> temporary variable:
>
>   width, height, depth = dimensions
>   _pix = ctypes.c_void_p()
>   _pixCreate = _leptonlib.pixCreate
>   _pixCreate.restype = ctypes.c_int
>   import pdb
>   pdb.set_trace()
>
>   addr = _pixCreate(width, height, depth)
>   _pix.value = addr
>
> Once pdb stops the script:
>
>   > leptonica.py(178)__init__()
>   -> addr = _pixCreate(width, height, depth)
>   (Pdb) n
>   > leptonica.py(179)__init__()
>   -> _pix.value = addr
>   (Pdb) addr
>   <cell at 43: int object at 44>
>   (Pdb) type(addr)
>   <type 'cell'>
>   (Pdb) dir(addr)
>   ['__class__', '__cmp__', '__delattr__', '__doc__', '__eq__', '__format__',
>    '__getattribute__', '__hash__', '__init__', '__ne__', '__new__',
>    '__reduce__', '__reduce_ex__', '__repr__', '__setattr__',
>    '__sizeof__', '__str__', '__subclasshook__', 'cell_contents']
>   (Pdb) type(addr.cell_contents)
>   <type 'int'>
>   (Pdb) addr.cell_contents
>   77491296
>
> So instead of returning an int, ctypes under IronPython is returning a
> "cell" in this case for some reason?
>
> Compare this to what I get with cpython. If I change the leptonlib.dll
> project Configuration Properties | Debugging tab to:
>
>   Command: c:\Python26\python26.exe
>   Arguments: -i leptonica.py
>   Working Directory: C:\leptonica\
>
> Debugging by typing F5 (or choosing Debug > Start Debugging) without
> changing leptonica.py or leptonlibd.dll in any way, I then hit all my
> breakpoints set in leptonlibd.dll. I get warning messages about
> python.exe having stopped for each breakpoint and I just click the
> Continue button to close them. After I type F5 to get past all my C
> function breakpoints, pdb stops my script with:
>
>   > leptonica.py(178)__init__()
>   -> addr = _pixCreate(width, height, depth)
>   (Pdb) n
>   > leptonica.py(179)__init__()
>   -> _pix.value = addr
>   (Pdb) addr
>   19492760
>   (Pdb) hex(addr)
>   '0x1296f98'
>
> and 0x1296f98 matches what I see for the return value of pixCreate()
> from the VS2008 debugger.
>
> I also tried debugging my leptonica.py script by following the
> directions in the IronPython tutorial. I made a new Solution with the
> following Debugging properties:
>
>   Command: C:\Program Files\IronPython 2.6\ipy.exe
>   Arguments: -X:Debug -i leptonica.py
>   Working Directory: C:\leptonica\
>
> I added my leptonlib.vcprog file to this solution by right-clicking it
> and choosing Add > Existing Project. I again set a breakpoint at the C
> function pixCreate().
>
> I also set various breakpoints in my leptonica.py script.
>
> Pressing F5 to Start Debugging, I now hit my leptonica.py breakpoints
> but still don't hit any C function breakpoints.
>
> From the locals window I can see that addr is:
>
>   addr 0x00bf6ea0      object {int}
>
> Looking in a Memory window at "addr" I see:
>
>   0x06AAB590  79332d70 00bf6ea0 00000000 793042f4 00000001 33b4c9bc
> p-3y n¿.....ôB0y.....É´3
>
> So there's something else at "addr", but the next int is 0x00bf6ea0 again.
>
> The memory at 0x00bf6ea0 looks correct:
>
>   0x00BF6EA0  0a 00 00 00 14 00 00 00 01 00 00 00 01 00 00 00 01 00
> 00 00 00 00  ......................
>   0x00BF6EB6  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f8 6e bf 00  ..................øn¿.
>   0x00BF6ECC  fd fd fd fd 10 00 0b 00 f2 01 0c 00 80 6e bf 00 00 00
> 00 00 00 00  ýýýý....ò...€n¿.......
>   0x00BF6EE2  00 00 00 00 00 00 50 00 00 00 01 00 00 00 4d 00 00 00
> fd fd fd fd  ......P.......M...ýýýý
>
> At least it looks to me like those are indeed ints for width, height &
> depth which is what a PIX starts out with.
>
> After:
>
>   _pix.value = addr
>
> I see this in the locals window:
>
>   _pix "ctypes.c_void_p instance"      object
> {IronPython.NewTypes.IronPython.Modules.SimpleCData_4$4}
>
> In the Immediate window I get:
>
>   _pix
>   "ctypes.c_void_p instance"
>       base {IronPython.Modules.CTypes.SimpleCData}: "ctypes.c_void_p instance"
>       .class: PythonType: "c_void_p"
>       .dict: null
>       .slots_and_weakref: null
>
> But I am unable to figure out how to tell what the "value" of _pix is?
>

Since there seems to be something strange going on with using
IronPython, pdb & VS2008, I decided to just put in some printf's in
leptonlib, rebuild it, and forget about using debuggers.

Here's the results:

   [C:\leptonica]python26 leptonica.py
   pixCreate() returns 01293340
   addr=01293340
   _pix.value = 01293340
   ctypes.addressof(_pix) = 00c78878
   pixGetWidth() addr=01293340
   width = 10
   height = 20
   depth = 1

   [C:\leptonica]ipy leptonica.py
   pixCreate() returns 046d4f60
   addr=046d4f60
   _pix.value = 046d4f60
   ctypes.addressof(_pix) = 0353c698
   pixGetWidth() addr=0353c698
   width = 74272608
   height = 1442168
   depth = 131074

So despite what it looks like from pdb, IronPython is correctly
setting the ctypes.c_void_p. The problem is that when it passes the
ctypes.c_void_p back to a C function it gives the address of the
ctypes.c_void_p, instead of the ptr that it contains.

I'd still like to know:

   Why can't I use VS2008 to set breakpoints on C functions in DLLs
that are loaded by IronPython and the ctypes module?

   How to see the value of ctypes.c_void_p instances in the VS2008 debugger.