Fwd: [cython-users] can pointers be stored in Python's dict or list?
Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases. Note that <void*>object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences). - Robert ---------- Forwarded message ---------- From: Robert Bradshaw <robertwb@gmail.com> Date: Fri, Sep 14, 2012 at 1:11 PM Subject: Re: [cython-users] can pointers be stored in Python's dict or list? To: Mark Summerfield <list@qtrac.plus.com> Cc: cython-users@googlegroups.com On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield <list@qtrac.plus.com> wrote:
Hi Robert,
On Fri, 14 Sep 2012 10:45:06 -0700 Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <list@qtrac.plus.com> wrote:
Hi,
(1)
I am using Cython to provide a Python wrapper to a C library. The C library has a function which gives a handle (a pointer) which is then used to call other functions.
I want to be able to store a dictionary of name-pointer pairs so that I can resuse handles.
This doesn't seem to work with dict or list because of the Cannot convert 'Thing*' to Python object error.
Right now I have a static array and a dictionary:
cdef Thing* pointers[30] indexForThingName = {}
but I'd really like to be able to store an arbitrary number of pointers, ideally in a dict.
Is this possible?
Sounds like you want http://docs.python.org/release/3.1.5/c-api/capsule.html
But that means I have to drop down to C. I want to stay in Python and Cython.
You're already "in C" if you're dealing with pointers.
I was hoping that Cython offered some kind of wrapper type, something that at the Python level would look like:
class Pointer(object): def __init__(self, void *pointer): cdef void *self.pointer = pointer # Doesn't work
that could be put in a dict or list. Then I could do things like this:
Pointer p = <void*>c_func_that_returns_a_pointer() mylist.append(p) ... Foo *q = <Foo*>mylist.pop()
How about (untested) from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr cdef void* p = <void*>c_func_that_returns_a_pointer() mylist.append(PyCObject_FromVoidPtr(p, NULL) ... Foo *q = <Foo*>PyCObject_AsVoidPtr(mylist.pop()) This should work for Python 2, use capsules (at the provided link) for Python 3+. For a portable solution, you could cast the void* to a intptr_t (or long long, if that's sufficient) which converts back and forth from Python just fine (but is more hackish).
Oh well, I'll see if I can use libcpp's map...
(2)
Also, I tried doing
from libc.stdio cimport printf
and then in my code:
printf("%p", pointer)
but although it compiled without error, nothing was printed.
Did it perhaps not flush? You also might want to try adding a newline.
I changed it to printf("%p\n", pointer) and now I get Cannot convert 'void *' to Python object
Are you still cimporting printf? - Robert
Isn't there a case for converting to/from ctypes pointers rather than capsules? And if capsules, what would the secret word be? Hmm... (sorry for the top post) DS -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Robert Bradshaw <robertwb@gmail.com> wrote: Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases. Note that <void*>object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences). - Robert ---------- Forwarded message ---------- From: Robert Bradshaw <robertwb@gmail.com> Date: Fri, Sep 14, 2012 at 1:11 PM Subject: Re: [cython-users] can pointers be stored in Python's dict or list? To: Mark Summerfield <list@qtrac.plus.com> Cc: cython-users@googlegroups.com On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield <list@qtrac.plus.com> wrote:
Hi Robert,
On Fri, 14 Sep 2012 10:45:06 -0700 Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <list@qtrac.plus.com> wrote:
Hi,
(1)
I am using Cython to provide a Python wrapper to a C library. The C library has a function which gives a handle (a pointer) which is then used to call other functions.
I want to be able to store a dictionary of name-pointer pairs so that I can resuse handles.
This doesn't seem to work with dict or list because of the Cannot convert 'Thing*' to Python object error.
Right now I have a static array and a dictionary:
cdef Thing* pointers[30] indexForThingName = {}
but I'd really like to be able to store an arbitrary number of pointers, ideally in a dict.
Is this possible?
Sounds like you want http://docs.python.org/release/3.1.5/c-api/capsule.html
But that means I have to drop down to C. I want to stay in Python and Cython.
You're already "in C" if you're dealing with pointers.
I was hoping that Cython offered some kind of wrapper type, something that at the Python level would look like:
class Pointer(object): def __init__(self, void *pointer): cdef void *self.pointer = pointer # Doesn't work
that could be put in a dict or list. Then I could do things like this:
Pointer p = <void*>c_func_that_returns_a_pointer() mylist.append(p) ... Foo *q = <Foo*>mylist.pop()
How about (untested) from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr cdef void* p = <void*>c_func_that_returns_a_pointer() mylist.append(PyCObject_FromVoidPtr(p, NULL) ... Foo *q = <Foo*>PyCObject_AsVoidPtr(mylist.pop()) This should work for Python 2, use capsules (at the provided link) for Python 3+. For a portable solution, you could cast the void* to a intptr_t (or long long, if that's sufficient) which converts back and forth from Python just fine (but is more hackish).
Oh well, I'll see if I can use libcpp's map...
(2)
Also, I tried doing
from libc.stdio cimport printf
and then in my code:
printf("%p", pointer)
but although it compiled without error, nothing was printed.
Did it perhaps not flush? You also might want to try adding a newline.
I changed it to printf("%p\n", pointer) and now I get Cannot convert 'void *' to Python object
Are you still cimporting printf? - Robert _____________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
Isn't there a case for converting to/from ctypes pointers rather than capsules? And if capsules, what would the secret word be? Hmm...
+1, ctypes would be even better. It's not in the standard library 'till 2.5, but I don't think that's a huge blocker as it can be installed earlier.
Robert Bradshaw <robertwb@gmail.com> wrote:
Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases.
Note that <void*>object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences).
- Robert
---------- Forwarded message ---------- From: Robert Bradshaw <robertwb@gmail.com> Date: Fri, Sep 14, 2012 at 1:11 PM Subject: Re: [cython-users] can pointers be stored in Python's dict or list? To: Mark Summerfield <list@qtrac.plus.com> Cc: cython-users@googlegroups.com
On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield <list@qtrac.plus.com> wrote:
Hi Robert,
On Fri, 14 Sep 2012 10:45:06 -0700 Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <list@qtrac.plus.com> wrote:
Hi,
(1)
I am using Cython to provide a Python wrapper to a C library. The C library has a function which gives a handle (a pointer) which is then used to call other functions.
I want to be able to store a dictionary of name-pointer pairs so that I can resuse handles.
This doesn't seem to work with dict or list because of the Cannot convert 'Thing*' to Python object error.
Right now I have a static array and a dictionary:
cdef Thing* pointers[30] indexForThingName = {}
but I'd really like to be able to store an arbitrary number of pointers, ideally in a dict.
Is this possible?
Sounds like you want http://docs.python.org/release/3.1.5/c-api/capsule.html
But that means I have to drop down to C. I want to stay in Python and Cython.
You're already "in C" if you're dealing with pointers.
I was hoping that Cython offered some kind of wrapper type, something that at the Python level would look like:
class Pointer(object): def __init__(self, void *pointer): cdef void *self.pointer = pointer # Doesn't work
that could be put in a dict or list. Then I could do things like this:
Pointer p = <void*>c_func_that_returns_a_pointer() mylist.append(p) ... Foo *q = <Foo*>mylist.pop()
How about (untested)
from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr
cdef void* p = <void*>c_func_that_returns_a_pointer() mylist.append(PyCObject_FromVoidPtr(p, NULL) ... Foo *q = <Foo*>PyCObject_AsVoidPtr(mylist.pop())
This should work for Python 2, use capsules (at the provided link) for Python 3+. For a portable solution, you could cast the void* to a intptr_t (or long long, if that's sufficient) which converts back and forth from Python just fine (but is more hackish).
Oh well, I'll see if I can use libcpp's map...
(2)
Also, I tried doing
from libc.stdio cimport printf
and then in my code:
printf("%p", pointer)
but although it compiled without error, nothing was printed.
Did it perhaps not flush? You also might want to try adding a newline.
I changed it to printf("%p\n", pointer) and now I get Cannot convert 'void *' to Python object
Are you still cimporting printf?
- Robert ________________________________
cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Robert Bradshaw, 15.09.2012 00:39:
On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn wrote:
Isn't there a case for converting to/from ctypes pointers rather than capsules? And if capsules, what would the secret word be? Hmm...
+1, ctypes would be even better. It's not in the standard library 'till 2.5, but I don't think that's a huge blocker as it can be installed earlier.
I'm not entirely sure I see the use case for starting to support the ctypes type system. Is there more to it than just passing pointers through Python code? A capsule (or even a special Cython extension type) sounds like a better option. For example, it's rather unlikely that non-Cython user code will start to support passing in ctypes types, so it would rather be a Cython-to-Cython-only thing anyway. Do you assume that users would want to access C internals through the values that Cython returns to them? That sounds risky.
Robert Bradshaw wrote:
Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases.
Note that <void*>object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences).
We could support both use cases by adding both types to the type system. For example: from cython import capsule cap = <capsule[description]>some_ptr or, likely nicer: cap = capsule(some_ptr, description) with auto-fallback to CObject on older Py2 versions. I'm not sure about the way back. Maybe a function like "decapsule(cap, description)" would work here? As for ctypes, we might get away with providing a generic <ctypes> cast and do the type matching ourselves for the forward way. For the way back into Cython types, however, we should require an explicitly typed ctypes variable. Just allowing a cast from anything to a Cython C type would let our type conversion code explode. Not sure how much work it would be to implement this, though. The way from Cython to ctypes may just be a simple mapping of type names, whereas the other way might require some utility code overhead. Stefan
On 15 September 2012 06:39, Stefan Behnel <stefan_ml@behnel.de> wrote:
Robert Bradshaw, 15.09.2012 00:39:
On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn wrote:
Isn't there a case for converting to/from ctypes pointers rather than capsules? And if capsules, what would the secret word be? Hmm...
+1, ctypes would be even better. It's not in the standard library 'till 2.5, but I don't think that's a huge blocker as it can be installed earlier.
I'm not entirely sure I see the use case for starting to support the ctypes type system. Is there more to it than just passing pointers through Python code? A capsule (or even a special Cython extension type) sounds like a better option. For example, it's rather unlikely that non-Cython user code will start to support passing in ctypes types, so it would rather be a Cython-to-Cython-only thing anyway.
Actually, I am working on Numba, in which I get ctypes pointers and pointers as integers. It's a usecase, but I'm not sure it should be special-cased, although I do like the automatic type safety it would bring. It's not a big issue for me, I just make Cython accept a Py_uintptr_t and pass in an integer, and I convert that to a (function) pointer using <signature *> x, which is simple enough (but provides zero type-safety).
Do you assume that users would want to access C internals through the values that Cython returns to them? That sounds risky.
Robert Bradshaw wrote:
Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases.
Note that <void*>object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences).
We could support both use cases by adding both types to the type system. For example:
from cython import capsule
cap = <capsule[description]>some_ptr
or, likely nicer:
cap = capsule(some_ptr, description)
with auto-fallback to CObject on older Py2 versions. I'm not sure about the way back. Maybe a function like "decapsule(cap, description)" would work here?
As for ctypes, we might get away with providing a generic <ctypes> cast and do the type matching ourselves for the forward way. For the way back into Cython types, however, we should require an explicitly typed ctypes variable. Just allowing a cast from anything to a Cython C type would let our type conversion code explode.
Not sure how much work it would be to implement this, though. The way from Cython to ctypes may just be a simple mapping of type names, whereas the other way might require some utility code overhead.
Yeah, it sounds like buffer format string parsing all over again :) Maybe you can build at runtime the Cython type you expect and compare it with the object's _type_? I agree it's probably not trivial, and the use case not common enough.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Don't know how I missed this earlier. On Fri, Sep 14, 2012 at 10:39 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Robert Bradshaw, 15.09.2012 00:39:
On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn wrote:
Isn't there a case for converting to/from ctypes pointers rather than capsules? And if capsules, what would the secret word be? Hmm...
+1, ctypes would be even better. It's not in the standard library 'till 2.5, but I don't think that's a huge blocker as it can be installed earlier.
I'm not entirely sure I see the use case for starting to support the ctypes type system. Is there more to it than just passing pointers through Python code? A capsule (or even a special Cython extension type) sounds like a better option. For example, it's rather unlikely that non-Cython user code will start to support passing in ctypes types, so it would rather be a Cython-to-Cython-only thing anyway.
My assumption is that it would allow more than just Cython-to-Cython translation. Essentially ctypes would be the lingua franca of, well, external C types like pointers and any library that needs (say) and int*, whether in Cython or not, could communicate this via Python space. I'ts just an idea, I haven't pursued it that far. The other option is forgoing type safety or rolling our own (which might be lightweight enough to just do).
Do you assume that users would want to access C internals through the values that Cython returns to them? That sounds risky.
True, but it could be useful as well.
Robert Bradshaw wrote:
Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases.
Note that <void*>object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences).
We could support both use cases by adding both types to the type system. For example:
from cython import capsule
cap = <capsule[description]>some_ptr
or, likely nicer:
cap = capsule(some_ptr, description)
with auto-fallback to CObject on older Py2 versions. I'm not sure about the way back. Maybe a function like "decapsule(cap, description)" would work here?
As for ctypes, we might get away with providing a generic <ctypes> cast and do the type matching ourselves for the forward way. For the way back into Cython types, however, we should require an explicitly typed ctypes variable. Just allowing a cast from anything to a Cython C type would let our type conversion code explode.
Not sure how much work it would be to implement this, though. The way from Cython to ctypes may just be a simple mapping of type names, whereas the other way might require some utility code overhead.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
participants (4)
-
Dag Sverre Seljebotn -
mark florisson -
Robert Bradshaw -
Stefan Behnel