[IronPython] IronPython 2.6.1 & ctypes, C library, and debugging with VS2008
Dino Viehland
dinov at microsoft.com
Tue Jul 20 12:11:24 CEST 2010
I agree this sounds like a bug - if you can open a bug on CodePlex that'd
be great. I'll have a look at it after I'm back from EuroPython.
> -----Original Message-----
> From: users-bounces at lists.ironpython.com [mailto:users-
> bounces at lists.ironpython.com] On Behalf Of TP
> Sent: Tuesday, July 20, 2010 11:04 AM
> To: Discussion of IronPython
> Subject: Re: [IronPython] IronPython 2.6.1 & ctypes, C library, and
> debugging with VS2008
>
> On Mon, Jul 19, 2010 at 12:18 AM, TP <wingusr at gmail.com> wrote:
> > On Sun, Jul 18, 2010 at 10:47 PM, TP <wingusr at gmail.com> wrote:
> >> On Sun, Jul 18, 2010 at 2:38 PM, Dino Viehland <dinov at microsoft.com>
> wrote:
> >>> TP wrote:
> >>>> I'm using IronPython 2.6.1 for Net 2.0 and VS2008 on Windows XP
> SP3.
> >>>>
> >>>> I have a python script that lets me access the Leptonica Image
> >>>> Processing C library (http://leptonica.com/) using the ctypes
> module.
> >>>>
> >>>> Everything seems to work fine with cpython 2.6. I can correctly
> load
> >>>> leptonlib.dll, create leptonica PIX images, manipulate them, and
> even
> >>>> display them using PyQT. I can use VS2008 to put breakpoints on
> >>>> leptonica C functions and step through the C code in the VS2008
> >>>> debugger.
> >>>>
> >>>> I thought I'd give IronPython a try since it also has ctypes, but
> I
> >>>> ran into a number of problems.
> >>>>
> >>>> The leptonlib.dll must be loading since I am able to correctly get
> >>>> back the library's version string. That is the following works:
> >>>>
> >>>> _getLeptonlibVersion = _leptonlib.getLeptonlibVersion
> >>>> _getLeptonlibVersion.restype = ctypes.c_void_p
> >>>> addr = _getLeptonlibVersion()
> >>>> version = ctypes.string_at(addr)
> >>>>
> >>>> However the following code doesn't seem to work under IronPython
> 2.6.1:
> >>>>
> >>>> pix = Pix(dimensions=(10,20,1))
> >>>>
> >>>> where essentially the following is called:
> >>>>
> >>>> _pix = ctypes.c_void_p()
> >>>> _pix.value = _leptonlib.pixCreate(width, height, depth)
> >>>> self._pix = _pix
> >>>>
> >>>> and I want to treat _pix as an opaque ptr I just hand back to
> >>>> leptonica whenever it needs it. For example:
> >>>>
> >>>> def height(self):
> >>>> """Get height of pix in pixels."""
> >>>> return _leptonlib.pixGetHeight(self._pix)
> >>>>
> >>>> Which works with cpython but returns something other than 20 with
> >>>> IronPython.
> >>>>
> >>>> By specifying "-X:Debug -X:FullFrames -X:Tracing leptonica.py" as
> the
> >>>> arguments to C:\Program Files\IronPython 2.6\ipy.exe I can get
> >>>> IronPython to correctly stop at places in my script where I put:
> >>>>
> >>>> import pdb
> >>>> pdb.set_trace()
> >>>>
> >>>> when I debug leptonlibd.dll with VS2008. However, my breakpoints
> on
> >>>> leptonica's C functions never seem to get hit. This works fine if
> I
> >>>> instead use python26.exe as command to launch when debugging.
> >>>
> >>> You should only need -X:Debug to get debugging under VS. pdb uses
> a
> >>> more Pythonic form of debugging but there's no support for it in
> VS.
> >>
> >> Let me be clear here. I am using the VS2008 Solution that I use to
> >> create leptonlib.dll. I am debugging that dll by right-clicking its
> >> project and setting its Configuration Properties | Debugging tab to:
> >>
> >> Command: C:\Program Files\IronPython 2.6\ipy.exe
> >> Arguments: -X:Debug -X:FullFrames -X:Tracing -i leptonica.py
> >> Working Directory: C:\leptonica\
> >>
> >> If I only use -X:Debug as you suggest I get the following error:
> >>
> >> Traceback (most recent call last):
> >> File "leptonica.py", line 458, in <module>
> >> File "leptonica.py", line 176, in __init__
> >> File "C:\Program Files\IronPython 2.6\Lib\pdb.py", line 1220, in
> set_trace
> >> AttributeError: 'module' object has no attribute '_getframe'>>>
> >>
> >> Googling, I determined that I needed to add at least -X:FullFrames,
> >> and by trial and error I found I also needed -X:Tracing. (I wasn't
> >> able to find any documentation on ipy.exe's command line switches? I
> >> just ran "ipy.exe -h" to dump out the short help description and
> took
> >> a wild guess at which might be useful)
> >>
> >>>>
> >>>> Ideally I like to have the VS Debugger stop in leptonlibd.dll's
> >>>> pixCreate() C function just before it returns its value so I can
> >>>> compare that to what I get back on the IronPython side.
> >>>>
> >>>> So how's does one use VS2008 to step through C functions in DLLs
> that
> >>>> are loaded by IronPython and the ctypes module?
> >>>
> >>> Do you have symbols (.PDB files) for leptonlibd.dll? You'll need
> symbols
> >>> to be able to step through the C code. You can still probably set
> a breakpoint
> >>> in the function because it's DLL exported - I think you can debug-
> >new
> >>> breakpoint and enter leptonlibd!pixCreate. Even w/o symbols you
> could
> >>> step through the assembly.
> >>
> >> leptonlibd.dll is created with the C7 Compatible (/Z7) compiler
> >> switch. This embeds the debugging info directly in the DLL so no
> .pdb
> >> file is produced. It also does NOT use pre-compiled headers.
> >>
> >> I might also point out that the same exact .dll will hit breakpoints
> >> if debugged with python26.exe rather than ipy.exe. Perhaps since
> >> ipy.exe is built on top of .NET the VS2008 debugger handles any
> .dll's
> >> loaded by it differently? I know for example that the VS2008
> debugger
> >> didn't like trying to run the IronPython for NET 4.0 version of
> >> ipy.exe (I gather you have to use VS2010 if you want to do that).
> >>
> >> Trying to set a breaking at leptonlibd!pixCreate didn't work.
> >>
> >>>>
> >>>> Secondly, am I using ctypes wrong, and does my code only work by
> >>>> happenstance for cpython.
> >>>
> >>> I don't see anything particularly wrong on your side and this looks
> like
> >>> a pretty simple call. I would assume the C functions are defined
> as:
> >>>
> >>> void* pixCreate(int width, int height, int depth);
> >>> int pixGetHeight(void* pixel);
> >>
> >> That's correct.
> >>
> >>>
> >>> I would hope we're getting all of this right as it seems like
> simple stuff
> >>> that should be tested somewhere. My first guess would be maybe
> we're
> >>> getting the calling convention wrong. Maybe it needs to be a
> WinDLL instead
> >>> of a CDLL? Maybe there's a check in the Python code for
> sys.platform which
> >>> is looking for win32 and uses WinDLL to open the DLL instead of
> CDLL on
> >>> Windows?
> >>
> >> More information on the problem:
> >>
> >> I decided to explicitly set the restype even though it's the default
> >> return type. I get the same exact behavior as my original code.
> Works
> >> with cpython, doesn't work with IronPython 2.6.1.
> >>
> >> If I have with the following python code:
> >>
> >> _pix = ctypes.c_void_p()
> >> _pixCreate = _leptonlib.pixCreate
> >> _pixCreate.restype = ctypes.c_int
> >> import pdb
> >> pdb.set_trace()
> >>
> >> _pix.value = _pixCreate(width, height, depth)
> >>
> >> I can do the following in the Command Window after pdb breaks into
> the script:
> >>
> >> > leptonica.py(180)__init__()
> >> -> _pix.value = _pixCreate(width, height, depth)
> >> (Pdb) n
> >> > leptonica.py(181)__init__()
> >> -> self._pix = _pix
> >> (Pdb) _pix.value
> >> *** AttributeError: 'cell' object has no attribute 'value'
> >>
> >> This is a bit strange since I just assigned to _pix.value.
> >> Investigating further:
> >>
> >> (Pdb) _pix
> >> <cell at 43: c_void_p object at 44>
> >>
> >> So instead of assigning to _pix.value (that is changing an attribute
> >> of _pix), IronPython is clobbering _pix?
> >>
> >> If I instead separately assign the result of pixCreate() to a
> >> temporary variable:
> >>
> >> width, height, depth = dimensions
> >> _pix = ctypes.c_void_p()
> >> _pixCreate = _leptonlib.pixCreate
> >> _pixCreate.restype = ctypes.c_int
> >> import pdb
> >> pdb.set_trace()
> >>
> >> addr = _pixCreate(width, height, depth)
> >> _pix.value = addr
> >>
> >> Once pdb stops the script:
> >>
> >> > leptonica.py(178)__init__()
> >> -> addr = _pixCreate(width, height, depth)
> >> (Pdb) n
> >> > leptonica.py(179)__init__()
> >> -> _pix.value = addr
> >> (Pdb) addr
> >> <cell at 43: int object at 44>
> >> (Pdb) type(addr)
> >> <type 'cell'>
> >> (Pdb) dir(addr)
> >> ['__class__', '__cmp__', '__delattr__', '__doc__', '__eq__',
> '__format__',
> >> '__getattribute__', '__hash__', '__init__', '__ne__', '__new__',
> >> '__reduce__', '__reduce_ex__', '__repr__', '__setattr__',
> >> '__sizeof__', '__str__', '__subclasshook__', 'cell_contents']
> >> (Pdb) type(addr.cell_contents)
> >> <type 'int'>
> >> (Pdb) addr.cell_contents
> >> 77491296
> >>
> >> So instead of returning an int, ctypes under IronPython is returning
> a
> >> "cell" in this case for some reason?
> >>
> >> Compare this to what I get with cpython. If I change the
> leptonlib.dll
> >> project Configuration Properties | Debugging tab to:
> >>
> >> Command: c:\Python26\python26.exe
> >> Arguments: -i leptonica.py
> >> Working Directory: C:\leptonica\
> >>
> >> Debugging by typing F5 (or choosing Debug > Start Debugging) without
> >> changing leptonica.py or leptonlibd.dll in any way, I then hit all
> my
> >> breakpoints set in leptonlibd.dll. I get warning messages about
> >> python.exe having stopped for each breakpoint and I just click the
> >> Continue button to close them. After I type F5 to get past all my C
> >> function breakpoints, pdb stops my script with:
> >>
> >> > leptonica.py(178)__init__()
> >> -> addr = _pixCreate(width, height, depth)
> >> (Pdb) n
> >> > leptonica.py(179)__init__()
> >> -> _pix.value = addr
> >> (Pdb) addr
> >> 19492760
> >> (Pdb) hex(addr)
> >> '0x1296f98'
> >>
> >> and 0x1296f98 matches what I see for the return value of pixCreate()
> >> from the VS2008 debugger.
> >>
> >> I also tried debugging my leptonica.py script by following the
> >> directions in the IronPython tutorial. I made a new Solution with
> the
> >> following Debugging properties:
> >>
> >> Command: C:\Program Files\IronPython 2.6\ipy.exe
> >> Arguments: -X:Debug -i leptonica.py
> >> Working Directory: C:\leptonica\
> >>
> >> I added my leptonlib.vcprog file to this solution by right-clicking
> it
> >> and choosing Add > Existing Project. I again set a breakpoint at the
> C
> >> function pixCreate().
> >>
> >> I also set various breakpoints in my leptonica.py script.
> >>
> >> Pressing F5 to Start Debugging, I now hit my leptonica.py
> breakpoints
> >> but still don't hit any C function breakpoints.
> >>
> >> From the locals window I can see that addr is:
> >>
> >> addr 0x00bf6ea0 object {int}
> >>
> >> Looking in a Memory window at "addr" I see:
> >>
> >> 0x06AAB590 79332d70 00bf6ea0 00000000 793042f4 00000001 33b4c9bc
> >> p-3y n¿.....ôB0y.....É´3
> >>
> >> So there's something else at "addr", but the next int is 0x00bf6ea0
> again.
> >>
> >> The memory at 0x00bf6ea0 looks correct:
> >>
> >> 0x00BF6EA0 0a 00 00 00 14 00 00 00 01 00 00 00 01 00 00 00 01 00
> >> 00 00 00 00 ......................
> >> 0x00BF6EB6 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> f8 6e bf 00 ..................øn¿.
> >> 0x00BF6ECC fd fd fd fd 10 00 0b 00 f2 01 0c 00 80 6e bf 00 00 00
> >> 00 00 00 00 ýýýý....ò...€n¿.......
> >> 0x00BF6EE2 00 00 00 00 00 00 50 00 00 00 01 00 00 00 4d 00 00 00
> >> fd fd fd fd ......P.......M...ýýýý
> >>
> >> At least it looks to me like those are indeed ints for width, height
> &
> >> depth which is what a PIX starts out with.
> >>
> >> After:
> >>
> >> _pix.value = addr
> >>
> >> I see this in the locals window:
> >>
> >> _pix "ctypes.c_void_p instance" object
> >> {IronPython.NewTypes.IronPython.Modules.SimpleCData_4$4}
> >>
> >> In the Immediate window I get:
> >>
> >> _pix
> >> "ctypes.c_void_p instance"
> >> base {IronPython.Modules.CTypes.SimpleCData}: "ctypes.c_void_p
> instance"
> >> .class: PythonType: "c_void_p"
> >> .dict: null
> >> .slots_and_weakref: null
> >>
> >> But I am unable to figure out how to tell what the "value" of _pix
> is?
> >>
> >
> > Since there seems to be something strange going on with using
> > IronPython, pdb & VS2008, I decided to just put in some printf's in
> > leptonlib, rebuild it, and forget about using debuggers.
> >
> > Here's the results:
> >
> > [C:\leptonica]python26 leptonica.py
> > pixCreate() returns 01293340
> > addr=01293340
> > _pix.value = 01293340
> > ctypes.addressof(_pix) = 00c78878
> > pixGetWidth() addr=01293340
> > width = 10
> > height = 20
> > depth = 1
> >
> > [C:\leptonica]ipy leptonica.py
> > pixCreate() returns 046d4f60
> > addr=046d4f60
> > _pix.value = 046d4f60
> > ctypes.addressof(_pix) = 0353c698
> > pixGetWidth() addr=0353c698
> > width = 74272608
> > height = 1442168
> > depth = 131074
> >
> > So despite what it looks like from pdb, IronPython is correctly
> > setting the ctypes.c_void_p. The problem is that when it passes the
> > ctypes.c_void_p back to a C function it gives the address of the
> > ctypes.c_void_p, instead of the ptr that it contains.
> >
> > I'd still like to know:
> >
> > Why can't I use VS2008 to set breakpoints on C functions in DLLs
> > that are loaded by IronPython and the ctypes module?
> >
> > How to see the value of ctypes.c_void_p instances in the VS2008
> debugger.
> >
>
> I installed Visual C++ 2010 Express edition. It is able to debug C
> functions in DLLs that are loaded by IronPython 2.6.1 (for Net 4.0 or
> Net 2.0) just fine. So the problem was with using VS2008.
>
> I created a simple test dll & python script to demonstrate
> IronPython's bug when passing ctypes.cvoid_p objects back to C. The
> dll source, cvoidPBug.c is:
>
> #include <stdlib.h>
> #include <malloc.h>
>
> __declspec(dllexport) void *makeIntPtr(void)
> {
> int *p = (int *) malloc(sizeof(int));
> *p = 42;
> return p;
> }
>
> __declspec(dllexport) int getInt(void *p)
> {
> int i = *(int *) p;
> return i;
> }
>
> The python script is:
>
> import sys
> import ctypes
>
> print(sys.version)
> cvoidPBugLib = ctypes.cdll.LoadLibrary(r"Debug\cvoidPBug.dll")
>
> addr=cvoidPBugLib.makeIntPtr()
> print("type of addr=%s" % type(addr))
> print("int from direct addr=%d" % cvoidPBugLib.getInt(addr))
>
> voidptr = ctypes.c_voidp(cvoidPBugLib.makeIntPtr())
> print("type of voidptr=%s" % type(voidptr))
> print("int from initialized voidptr=%d" %
> cvoidPBugLib.getInt(voidptr))
>
> voidptr = ctypes.c_voidp()
> addr = cvoidPBugLib.makeIntPtr()
> voidptr.value = addr
> print("type of voidptr=%s" % type(voidptr))
> print("int from voidptr made by assigning value=%d" %
> cvoidPBugLib.getInt(voidptr))
>
> getInt = cvoidPBugLib.getInt
> getInt.argtypes = [ctypes.c_voidp]
> print("int from declared voidptr arg made by assigning value=%d" %
> getInt(voidptr))
>
> makeIntPtr = cvoidPBugLib.makeIntPtr
> makeIntPtr.restype = ctypes.c_voidp
> voidptr = cvoidPBugLib.makeIntPtr()
> print("type of voidptr=%s" % type(voidptr))
> print("int from voidptr after declaring restype=%d" %
> cvoidPBugLib.getInt(voidptr))
>
> Here's the results:
>
> 2.6.4 (r264:75706, Jan 22 2010, 16:41:54) [MSC v.1500 32 bit
> (Intel)]
> type of addr=<type 'int'>
> int from direct addr=42
>
> type of voidptr=<class 'ctypes.c_void_p'>
> int from initialized voidptr=42
>
> type of voidptr=<class 'ctypes.c_void_p'>
> int from voidptr made by assigning value=42
> int from declared voidptr arg made by assigning value=42
>
> type of voidptr=<type 'int'>
> int from voidptr after declaring restype=42
>
> 2.6.1 (IronPython 2.6.1 (2.6.10920.0) on .NET 2.0.50727.3607)
> type of addr=<type 'int'>
> int from direct addr=42
>
> type of voidptr=<class 'ctypes.c_void_p'>
> int from initialized voidptr=70274912
>
> type of voidptr=<class 'ctypes.c_void_p'>
> int from voidptr made by assigning value=70274960
> int from declared voidptr arg made by assigning value=70274960
>
> type of voidptr=<type 'int'>
> int from voidptr after declaring restype=42
>
> 2.6.1 (IronPython 2.6.1 (2.6.10920.0) on .NET 4.0.30319.1)
> type of addr=<type 'int'>
> int from direct addr=42
>
> type of voidptr=<class 'ctypes.c_void_p'>
> int from initialized voidptr=63983480
>
> type of voidptr=<class 'ctypes.c_void_p'>
> int from voidptr made by assigning value=63983528
> int from declared voidptr arg made by assigning value=63983528
>
> type of voidptr=<type 'int'>
> int from voidptr after declaring restype=42
>
> As can be seen any time the type of voidptr is actually <class
> 'ctypes.c_void_p'> (rather than <type 'int'>) the wrong value is
> passed to C in IronPython.
>
> For my particular case, I've just decided to store void * ptrs
> returned from C as ints and not bother with ctypes.c_void_p, This
> solves my problem but I still think the current IronPython behavior is
> a bug.
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
More information about the Ironpython-users
mailing list