[IronPython] IronPython 2.6.1 & ctypes, C library, and debugging with VS2008
TP
wingusr at gmail.com
Tue Jul 20 12:03:32 CEST 2010
On Mon, Jul 19, 2010 at 12:18 AM, TP <wingusr at gmail.com> wrote:
> On Sun, Jul 18, 2010 at 10:47 PM, TP <wingusr at gmail.com> wrote:
>> On Sun, Jul 18, 2010 at 2:38 PM, Dino Viehland <dinov at microsoft.com> wrote:
>>> TP wrote:
>>>> I'm using IronPython 2.6.1 for Net 2.0 and VS2008 on Windows XP SP3.
>>>>
>>>> I have a python script that lets me access the Leptonica Image
>>>> Processing C library (http://leptonica.com/) using the ctypes module.
>>>>
>>>> Everything seems to work fine with cpython 2.6. I can correctly load
>>>> leptonlib.dll, create leptonica PIX images, manipulate them, and even
>>>> display them using PyQT. I can use VS2008 to put breakpoints on
>>>> leptonica C functions and step through the C code in the VS2008
>>>> debugger.
>>>>
>>>> I thought I'd give IronPython a try since it also has ctypes, but I
>>>> ran into a number of problems.
>>>>
>>>> The leptonlib.dll must be loading since I am able to correctly get
>>>> back the library's version string. That is the following works:
>>>>
>>>> _getLeptonlibVersion = _leptonlib.getLeptonlibVersion
>>>> _getLeptonlibVersion.restype = ctypes.c_void_p
>>>> addr = _getLeptonlibVersion()
>>>> version = ctypes.string_at(addr)
>>>>
>>>> However the following code doesn't seem to work under IronPython 2.6.1:
>>>>
>>>> pix = Pix(dimensions=(10,20,1))
>>>>
>>>> where essentially the following is called:
>>>>
>>>> _pix = ctypes.c_void_p()
>>>> _pix.value = _leptonlib.pixCreate(width, height, depth)
>>>> self._pix = _pix
>>>>
>>>> and I want to treat _pix as an opaque ptr I just hand back to
>>>> leptonica whenever it needs it. For example:
>>>>
>>>> def height(self):
>>>> """Get height of pix in pixels."""
>>>> return _leptonlib.pixGetHeight(self._pix)
>>>>
>>>> Which works with cpython but returns something other than 20 with
>>>> IronPython.
>>>>
>>>> By specifying "-X:Debug -X:FullFrames -X:Tracing leptonica.py" as the
>>>> arguments to C:\Program Files\IronPython 2.6\ipy.exe I can get
>>>> IronPython to correctly stop at places in my script where I put:
>>>>
>>>> import pdb
>>>> pdb.set_trace()
>>>>
>>>> when I debug leptonlibd.dll with VS2008. However, my breakpoints on
>>>> leptonica's C functions never seem to get hit. This works fine if I
>>>> instead use python26.exe as command to launch when debugging.
>>>
>>> You should only need -X:Debug to get debugging under VS. pdb uses a
>>> more Pythonic form of debugging but there's no support for it in VS.
>>
>> Let me be clear here. I am using the VS2008 Solution that I use to
>> create leptonlib.dll. I am debugging that dll by right-clicking its
>> project and setting its Configuration Properties | Debugging tab to:
>>
>> Command: C:\Program Files\IronPython 2.6\ipy.exe
>> Arguments: -X:Debug -X:FullFrames -X:Tracing -i leptonica.py
>> Working Directory: C:\leptonica\
>>
>> If I only use -X:Debug as you suggest I get the following error:
>>
>> Traceback (most recent call last):
>> File "leptonica.py", line 458, in <module>
>> File "leptonica.py", line 176, in __init__
>> File "C:\Program Files\IronPython 2.6\Lib\pdb.py", line 1220, in set_trace
>> AttributeError: 'module' object has no attribute '_getframe'>>>
>>
>> Googling, I determined that I needed to add at least -X:FullFrames,
>> and by trial and error I found I also needed -X:Tracing. (I wasn't
>> able to find any documentation on ipy.exe's command line switches? I
>> just ran "ipy.exe -h" to dump out the short help description and took
>> a wild guess at which might be useful)
>>
>>>>
>>>> Ideally I like to have the VS Debugger stop in leptonlibd.dll's
>>>> pixCreate() C function just before it returns its value so I can
>>>> compare that to what I get back on the IronPython side.
>>>>
>>>> So how's does one use VS2008 to step through C functions in DLLs that
>>>> are loaded by IronPython and the ctypes module?
>>>
>>> Do you have symbols (.PDB files) for leptonlibd.dll? You'll need symbols
>>> to be able to step through the C code. You can still probably set a breakpoint
>>> in the function because it's DLL exported - I think you can debug->new
>>> breakpoint and enter leptonlibd!pixCreate. Even w/o symbols you could
>>> step through the assembly.
>>
>> leptonlibd.dll is created with the C7 Compatible (/Z7) compiler
>> switch. This embeds the debugging info directly in the DLL so no .pdb
>> file is produced. It also does NOT use pre-compiled headers.
>>
>> I might also point out that the same exact .dll will hit breakpoints
>> if debugged with python26.exe rather than ipy.exe. Perhaps since
>> ipy.exe is built on top of .NET the VS2008 debugger handles any .dll's
>> loaded by it differently? I know for example that the VS2008 debugger
>> didn't like trying to run the IronPython for NET 4.0 version of
>> ipy.exe (I gather you have to use VS2010 if you want to do that).
>>
>> Trying to set a breaking at leptonlibd!pixCreate didn't work.
>>
>>>>
>>>> Secondly, am I using ctypes wrong, and does my code only work by
>>>> happenstance for cpython.
>>>
>>> I don't see anything particularly wrong on your side and this looks like
>>> a pretty simple call. I would assume the C functions are defined as:
>>>
>>> void* pixCreate(int width, int height, int depth);
>>> int pixGetHeight(void* pixel);
>>
>> That's correct.
>>
>>>
>>> I would hope we're getting all of this right as it seems like simple stuff
>>> that should be tested somewhere. My first guess would be maybe we're
>>> getting the calling convention wrong. Maybe it needs to be a WinDLL instead
>>> of a CDLL? Maybe there's a check in the Python code for sys.platform which
>>> is looking for win32 and uses WinDLL to open the DLL instead of CDLL on
>>> Windows?
>>
>> More information on the problem:
>>
>> I decided to explicitly set the restype even though it's the default
>> return type. I get the same exact behavior as my original code. Works
>> with cpython, doesn't work with IronPython 2.6.1.
>>
>> If I have with the following python code:
>>
>> _pix = ctypes.c_void_p()
>> _pixCreate = _leptonlib.pixCreate
>> _pixCreate.restype = ctypes.c_int
>> import pdb
>> pdb.set_trace()
>>
>> _pix.value = _pixCreate(width, height, depth)
>>
>> I can do the following in the Command Window after pdb breaks into the script:
>>
>> > leptonica.py(180)__init__()
>> -> _pix.value = _pixCreate(width, height, depth)
>> (Pdb) n
>> > leptonica.py(181)__init__()
>> -> self._pix = _pix
>> (Pdb) _pix.value
>> *** AttributeError: 'cell' object has no attribute 'value'
>>
>> This is a bit strange since I just assigned to _pix.value.
>> Investigating further:
>>
>> (Pdb) _pix
>> <cell at 43: c_void_p object at 44>
>>
>> So instead of assigning to _pix.value (that is changing an attribute
>> of _pix), IronPython is clobbering _pix?
>>
>> If I instead separately assign the result of pixCreate() to a
>> temporary variable:
>>
>> width, height, depth = dimensions
>> _pix = ctypes.c_void_p()
>> _pixCreate = _leptonlib.pixCreate
>> _pixCreate.restype = ctypes.c_int
>> import pdb
>> pdb.set_trace()
>>
>> addr = _pixCreate(width, height, depth)
>> _pix.value = addr
>>
>> Once pdb stops the script:
>>
>> > leptonica.py(178)__init__()
>> -> addr = _pixCreate(width, height, depth)
>> (Pdb) n
>> > leptonica.py(179)__init__()
>> -> _pix.value = addr
>> (Pdb) addr
>> <cell at 43: int object at 44>
>> (Pdb) type(addr)
>> <type 'cell'>
>> (Pdb) dir(addr)
>> ['__class__', '__cmp__', '__delattr__', '__doc__', '__eq__', '__format__',
>> '__getattribute__', '__hash__', '__init__', '__ne__', '__new__',
>> '__reduce__', '__reduce_ex__', '__repr__', '__setattr__',
>> '__sizeof__', '__str__', '__subclasshook__', 'cell_contents']
>> (Pdb) type(addr.cell_contents)
>> <type 'int'>
>> (Pdb) addr.cell_contents
>> 77491296
>>
>> So instead of returning an int, ctypes under IronPython is returning a
>> "cell" in this case for some reason?
>>
>> Compare this to what I get with cpython. If I change the leptonlib.dll
>> project Configuration Properties | Debugging tab to:
>>
>> Command: c:\Python26\python26.exe
>> Arguments: -i leptonica.py
>> Working Directory: C:\leptonica\
>>
>> Debugging by typing F5 (or choosing Debug > Start Debugging) without
>> changing leptonica.py or leptonlibd.dll in any way, I then hit all my
>> breakpoints set in leptonlibd.dll. I get warning messages about
>> python.exe having stopped for each breakpoint and I just click the
>> Continue button to close them. After I type F5 to get past all my C
>> function breakpoints, pdb stops my script with:
>>
>> > leptonica.py(178)__init__()
>> -> addr = _pixCreate(width, height, depth)
>> (Pdb) n
>> > leptonica.py(179)__init__()
>> -> _pix.value = addr
>> (Pdb) addr
>> 19492760
>> (Pdb) hex(addr)
>> '0x1296f98'
>>
>> and 0x1296f98 matches what I see for the return value of pixCreate()
>> from the VS2008 debugger.
>>
>> I also tried debugging my leptonica.py script by following the
>> directions in the IronPython tutorial. I made a new Solution with the
>> following Debugging properties:
>>
>> Command: C:\Program Files\IronPython 2.6\ipy.exe
>> Arguments: -X:Debug -i leptonica.py
>> Working Directory: C:\leptonica\
>>
>> I added my leptonlib.vcprog file to this solution by right-clicking it
>> and choosing Add > Existing Project. I again set a breakpoint at the C
>> function pixCreate().
>>
>> I also set various breakpoints in my leptonica.py script.
>>
>> Pressing F5 to Start Debugging, I now hit my leptonica.py breakpoints
>> but still don't hit any C function breakpoints.
>>
>> From the locals window I can see that addr is:
>>
>> addr 0x00bf6ea0 object {int}
>>
>> Looking in a Memory window at "addr" I see:
>>
>> 0x06AAB590 79332d70 00bf6ea0 00000000 793042f4 00000001 33b4c9bc
>> p-3y n¿.....ôB0y.....É´3
>>
>> So there's something else at "addr", but the next int is 0x00bf6ea0 again.
>>
>> The memory at 0x00bf6ea0 looks correct:
>>
>> 0x00BF6EA0 0a 00 00 00 14 00 00 00 01 00 00 00 01 00 00 00 01 00
>> 00 00 00 00 ......................
>> 0x00BF6EB6 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> f8 6e bf 00 ..................øn¿.
>> 0x00BF6ECC fd fd fd fd 10 00 0b 00 f2 01 0c 00 80 6e bf 00 00 00
>> 00 00 00 00 ýýýý....ò...€n¿.......
>> 0x00BF6EE2 00 00 00 00 00 00 50 00 00 00 01 00 00 00 4d 00 00 00
>> fd fd fd fd ......P.......M...ýýýý
>>
>> At least it looks to me like those are indeed ints for width, height &
>> depth which is what a PIX starts out with.
>>
>> After:
>>
>> _pix.value = addr
>>
>> I see this in the locals window:
>>
>> _pix "ctypes.c_void_p instance" object
>> {IronPython.NewTypes.IronPython.Modules.SimpleCData_4$4}
>>
>> In the Immediate window I get:
>>
>> _pix
>> "ctypes.c_void_p instance"
>> base {IronPython.Modules.CTypes.SimpleCData}: "ctypes.c_void_p instance"
>> .class: PythonType: "c_void_p"
>> .dict: null
>> .slots_and_weakref: null
>>
>> But I am unable to figure out how to tell what the "value" of _pix is?
>>
>
> Since there seems to be something strange going on with using
> IronPython, pdb & VS2008, I decided to just put in some printf's in
> leptonlib, rebuild it, and forget about using debuggers.
>
> Here's the results:
>
> [C:\leptonica]python26 leptonica.py
> pixCreate() returns 01293340
> addr=01293340
> _pix.value = 01293340
> ctypes.addressof(_pix) = 00c78878
> pixGetWidth() addr=01293340
> width = 10
> height = 20
> depth = 1
>
> [C:\leptonica]ipy leptonica.py
> pixCreate() returns 046d4f60
> addr=046d4f60
> _pix.value = 046d4f60
> ctypes.addressof(_pix) = 0353c698
> pixGetWidth() addr=0353c698
> width = 74272608
> height = 1442168
> depth = 131074
>
> So despite what it looks like from pdb, IronPython is correctly
> setting the ctypes.c_void_p. The problem is that when it passes the
> ctypes.c_void_p back to a C function it gives the address of the
> ctypes.c_void_p, instead of the ptr that it contains.
>
> I'd still like to know:
>
> Why can't I use VS2008 to set breakpoints on C functions in DLLs
> that are loaded by IronPython and the ctypes module?
>
> How to see the value of ctypes.c_void_p instances in the VS2008 debugger.
>
I installed Visual C++ 2010 Express edition. It is able to debug C
functions in DLLs that are loaded by IronPython 2.6.1 (for Net 4.0 or
Net 2.0) just fine. So the problem was with using VS2008.
I created a simple test dll & python script to demonstrate
IronPython's bug when passing ctypes.cvoid_p objects back to C. The
dll source, cvoidPBug.c is:
#include <stdlib.h>
#include <malloc.h>
__declspec(dllexport) void *makeIntPtr(void)
{
int *p = (int *) malloc(sizeof(int));
*p = 42;
return p;
}
__declspec(dllexport) int getInt(void *p)
{
int i = *(int *) p;
return i;
}
The python script is:
import sys
import ctypes
print(sys.version)
cvoidPBugLib = ctypes.cdll.LoadLibrary(r"Debug\cvoidPBug.dll")
addr=cvoidPBugLib.makeIntPtr()
print("type of addr=%s" % type(addr))
print("int from direct addr=%d" % cvoidPBugLib.getInt(addr))
voidptr = ctypes.c_voidp(cvoidPBugLib.makeIntPtr())
print("type of voidptr=%s" % type(voidptr))
print("int from initialized voidptr=%d" % cvoidPBugLib.getInt(voidptr))
voidptr = ctypes.c_voidp()
addr = cvoidPBugLib.makeIntPtr()
voidptr.value = addr
print("type of voidptr=%s" % type(voidptr))
print("int from voidptr made by assigning value=%d" %
cvoidPBugLib.getInt(voidptr))
getInt = cvoidPBugLib.getInt
getInt.argtypes = [ctypes.c_voidp]
print("int from declared voidptr arg made by assigning value=%d" %
getInt(voidptr))
makeIntPtr = cvoidPBugLib.makeIntPtr
makeIntPtr.restype = ctypes.c_voidp
voidptr = cvoidPBugLib.makeIntPtr()
print("type of voidptr=%s" % type(voidptr))
print("int from voidptr after declaring restype=%d" %
cvoidPBugLib.getInt(voidptr))
Here's the results:
2.6.4 (r264:75706, Jan 22 2010, 16:41:54) [MSC v.1500 32 bit (Intel)]
type of addr=<type 'int'>
int from direct addr=42
type of voidptr=<class 'ctypes.c_void_p'>
int from initialized voidptr=42
type of voidptr=<class 'ctypes.c_void_p'>
int from voidptr made by assigning value=42
int from declared voidptr arg made by assigning value=42
type of voidptr=<type 'int'>
int from voidptr after declaring restype=42
2.6.1 (IronPython 2.6.1 (2.6.10920.0) on .NET 2.0.50727.3607)
type of addr=<type 'int'>
int from direct addr=42
type of voidptr=<class 'ctypes.c_void_p'>
int from initialized voidptr=70274912
type of voidptr=<class 'ctypes.c_void_p'>
int from voidptr made by assigning value=70274960
int from declared voidptr arg made by assigning value=70274960
type of voidptr=<type 'int'>
int from voidptr after declaring restype=42
2.6.1 (IronPython 2.6.1 (2.6.10920.0) on .NET 4.0.30319.1)
type of addr=<type 'int'>
int from direct addr=42
type of voidptr=<class 'ctypes.c_void_p'>
int from initialized voidptr=63983480
type of voidptr=<class 'ctypes.c_void_p'>
int from voidptr made by assigning value=63983528
int from declared voidptr arg made by assigning value=63983528
type of voidptr=<type 'int'>
int from voidptr after declaring restype=42
As can be seen any time the type of voidptr is actually <class
'ctypes.c_void_p'> (rather than <type 'int'>) the wrong value is
passed to C in IronPython.
For my particular case, I've just decided to store void * ptrs
returned from C as ints and not bother with ctypes.c_void_p, This
solves my problem but I still think the current IronPython behavior is
a bug.
More information about the Ironpython-users
mailing list