[Cython] Fwd: [cython-users] can pointers be stored in Python's dict or list?

[Stefan Behnel]

Robert Bradshaw, 15.09.2012 00:39:
> On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn wrote:
>> Isn't there a case for converting to/from ctypes pointers rather than
>> capsules? And if capsules, what would the secret word be? Hmm...
> 
> +1, ctypes would be even better. It's not in the standard library
> 'till 2.5, but I don't think that's a huge blocker as it can be
> installed earlier.

I'm not entirely sure I see the use case for starting to support the ctypes
type system. Is there more to it than just passing pointers through Python
code? A capsule (or even a special Cython extension type) sounds like a
better option. For example, it's rather unlikely that non-Cython user code
will start to support passing in ctypes types, so it would rather be a
Cython-to-Cython-only thing anyway.

Do you assume that users would want to access C internals through the
values that Cython returns to them? That sounds risky.


>> Robert Bradshaw wrote:
>>> Given the CObject vs Capsule differences in Py2 vs Py3, and the
>>> clumsiness in using them (see below), any thoughts on automatically
>>> converting void* to/from a Python object? There is the sticky issue of
>>> pointer lifetime management, but we could assume the lifetime is
>>> managed entirely in C in most cases.
>>>
>>> Note that <void*>object already has a specific meaning that we can't
>>> hijack, but perhaps non-void* would be fine to do implicitly? Also, if
>>> we built this into the language, could we provide type safety via the
>>> name/description attribute? Something like this could be really handy
>>> to have and isn't as easily done as a library (especially given the
>>> 2/3 differences).

We could support both use cases by adding both types to the type system.
For example:

    from cython import capsule

    cap = <capsule[description]>some_ptr

or, likely nicer:

    cap = capsule(some_ptr, description)

with auto-fallback to CObject on older Py2 versions. I'm not sure about the
way back. Maybe a function like "decapsule(cap, description)" would work here?

As for ctypes, we might get away with providing a generic <ctypes> cast and
do the type matching ourselves for the forward way. For the way back into
Cython types, however, we should require an explicitly typed ctypes
variable. Just allowing a cast from anything to a Cython C type would let
our type conversion code explode.

Not sure how much work it would be to implement this, though. The way from
Cython to ctypes may just be a simple mapping of type names, whereas the
other way might require some utility code overhead.

Stefan