[Cython] Fwd: [cython-users] can pointers be stored in Python's dict or list?

[Robert Bradshaw]

Given the CObject vs Capsule differences in Py2 vs Py3, and the
clumsiness in using them (see below), any thoughts on automatically
converting void* to/from a Python object? There is the sticky issue of
pointer lifetime management, but we could assume the lifetime is
managed entirely in C in most cases.

Note that <void*>object already has a specific meaning that we can't
hijack, but perhaps non-void* would be fine to do implicitly? Also, if
we built this into the language, could we provide type safety via the
name/description attribute? Something like this could be really handy
to have and isn't as easily done as a library (especially given the
2/3 differences).

- Robert


---------- Forwarded message ----------
From: Robert Bradshaw <robertwb at gmail.com>
Date: Fri, Sep 14, 2012 at 1:11 PM
Subject: Re: [cython-users] can pointers be stored in Python's dict or list?
To: Mark Summerfield <list at qtrac.plus.com>
Cc: cython-users at googlegroups.com


On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield <list at qtrac.plus.com> wrote:
> Hi Robert,
>
> On Fri, 14 Sep 2012 10:45:06 -0700
> Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <list at qtrac.plus.com>
>> wrote:
>> > Hi,
>> >
>> > (1)
>> >
>> > I am using Cython to provide a Python wrapper to a C library.
>> > The C library has a function which gives a handle (a pointer) which is
>> > then used to call other functions.
>> >
>> > I want to be able to store a dictionary of name-pointer pairs so that I
>> > can resuse handles.
>> >
>> > This doesn't seem to work with dict or list because of the
>> >     Cannot convert 'Thing*' to Python object
>> > error.
>> >
>> > Right now I have a static array and a dictionary:
>> >
>> > cdef Thing* pointers[30]
>> > indexForThingName = {}
>> >
>> > but I'd really like to be able to store an arbitrary number of
>> > pointers, ideally in a dict.
>> >
>> > Is this possible?
>>
>> Sounds like you want
>> http://docs.python.org/release/3.1.5/c-api/capsule.html
>
> But that means I have to drop down to C. I want to stay in Python and
> Cython.

You're already "in C" if you're dealing with pointers.

> I was hoping that Cython offered some kind of wrapper type,
> something that at the Python level would look like:
>
>     class Pointer(object):
>         def __init__(self, void *pointer):
>             cdef void *self.pointer = pointer # Doesn't work
>
> that could be put in a dict or list. Then I could do things like this:
>
>     Pointer p = <void*>c_func_that_returns_a_pointer()
>     mylist.append(p)
>     ...
>     Foo *q = <Foo*>mylist.pop()

How about (untested)

    from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr

    cdef void* p = <void*>c_func_that_returns_a_pointer()
    mylist.append(PyCObject_FromVoidPtr(p, NULL)
    ...
    Foo *q = <Foo*>PyCObject_AsVoidPtr(mylist.pop())

This should work for Python 2, use capsules (at the provided link) for
Python 3+. For a portable solution, you could cast the void* to a
intptr_t (or long long, if that's sufficient) which converts back and
forth from Python just fine (but is more hackish).

> Oh well, I'll see if I can use libcpp's map...
>
>> > (2)
>> >
>> > Also, I tried doing
>> >
>> >     from libc.stdio cimport printf
>> >
>> > and then in my code:
>> >
>> >     printf("%p", pointer)
>> >
>> > but although it compiled without error, nothing was printed.
>>
>> Did it perhaps not flush? You also might want to try adding a newline.
>
> I changed it to printf("%p\n", pointer) and now I get
>     Cannot convert 'void *' to Python object

Are you still cimporting printf?

- Robert