Multidimensional indexing of C++ objects
Hi everyone, I'm a GSOC student working to make a Cython API for DyND. DyND <https://github.com/libdynd/libdynd> is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead. Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice. Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this: a(irange(), 1).vals() = 0; Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this? I'm willing to spend some time implementing a new feature if we can reach a consensus on a good way to deal with this. Here are some possible solutions I've thought of: 1. We could allow assignment to C++ method and function calls that return references. This has the advantage that it matches the existing syntax in C++ for dealing with C++ objects. Though Cython is a Python-like language, the ability to manipulate C++ objects directly is a key part of its feature set. Since the native way to do things like multidimensional indexing in C++ is via the call operator, it seems sensible to allow assignment to C++-level call operations in Cython as well. This could be enabled via a Cython compiler directive and be disabled by default. Using a compiler directive like this would result in an interface similar to the one already used for cdivision, wrap-around indexing, and index bounds checking. The user would avoid unexpected results by default, but be able to get the needed functionality simply by enabling it. 2. We could recommend that all assignment operations of this nature be wrapped in a fake method that wraps the assignment in it's c-level name. This has the advantage that it works in current and past versions of Cython, but it is a rather unusual hack. For example, something like the following would work right now: # declared as a method in a pxd file: void assign "vals() = "(int value) except + # used in a pyx file to perform assignment to a slice of an array a: a(irange(), 1).assign(0) For DyND, at least for now, this would be a workable solution since the difference lies primarily in the placement of the parenthesis and the presence of the assignment operator. The syntax is less clear than it could be, but it would work. On the other hand, other libraries may not be so lucky since this involves replacing assignment to a slice with a method call. For example, the expression template libraries Eigen and Blaze-lib would encounter incompatibility to varying degrees if someone were to try using them within Cython. This method also has the disadvantage that it creates an interface that is fundamentally different from both the Python and C++ interfaces. I have also considered, writing a proxy class that can serve as an effective temporary value while a multidimensional index is constructed from a series of calls to operator[]. This is a reasonable approach, but it leads to unnecessary code bloat. It also complicates the interface exposed to users, since operator[] would be needed for left hand values and operator() would be needed for right hand values. This would also make it so that users that want to use these C++ classes in Cython would have to include and link against another set of headers and libraries to be able to use the proxy class. The burden of maintainability for Python bindings created in this way would be greater as well. This also isn't a viable approach for using any C++ class that overloads both operators. Another option I have considered is allowing Cython's indexing operator to dispatch to a different function. Currently, user-defined cname entries for overloaded operators are not used. If this were changed for the indexing operator, indexing could be performed at the C++ level using some other method. This doesn't look like a viable approach though, since, for this to really work, users would need some way to call different methods when a C++ object is being indexed and when it is being assigned to. Using operator[] for left-hand values and operator() for right-hand values is a possible solution, but that isn't a very consistent interface. Doing this would also increase the complexity of the existing code for indexing in the Cython compiler and could lead to name collisions for classes that overload both operator[] and operator(). Are any of these acceptable ways to go forward? Does anyone have any better ideas? My preference would definitely be toward allowing C++ calls returning references to be used as lvalues, but I'd really appreciate any alternative solutions. Thanks! -Ian Henriksen
Hi Ian! Ian Henriksen schrieb am 04.07.2015 um 00:43:
I'm a GSOC student working to make a Cython API for DyND. DyND <https://github.com/libdynd/libdynd> is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead.
Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice. Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this:
a(irange(), 1).vals() = 0;
Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this?
Just an idea, don't know how feasible this is, but we could allow inline special methods in C++ class declarations that implement Python protocols. Example: cdef extern from ...: cppclass Array2D: int operator[] except + int getItemAt(ssize_t x, ssize_t y) except + cdef inline __getitem__(self, Py_ssize_t x, Py_ssize_t y): return self.getItemAt(x, y) def test(): cdef Array2D a return a[1, 2] Cython could then translate an item access on an Array2D instance into the corresponding special "method" call. Drawbacks: 1) The example above would conflict with the C++ [] operator, so it would be ambiguous which one is being used in Cython code. Not sure if there's a use case for making both available to Cython code, but that would be difficult to achieve if the need arises. 2) It doesn't solve the general problem of assigning to C++ expressions, especially because it does not extend the syntax allowed by Cython which would still limit what you can do in these fake special methods. Regarding your proposals, I'd be happy if we could avoid adding syntax support for assigning to function calls. And I agree that the cname assignment hack is really just a big hack. It shouldn't be relied on. Stefan
Drawback 1) could be solved by declaring the operator as cdef inline operator []() instead of cdef inline __getitem__()
Michael Enßlin schrieb am 04.07.2015 um 13:18:
Drawback 1) could be solved by declaring the operator as
cdef inline operator []()
instead of
cdef inline __getitem__()
But that would even shadow the operator entirely. The __getitem__ method is meant to provide a Python/Cython-level operator implementation, whereas operator[] declares the C++ level operation. The inline method might even want to use the C++ operator, or do its entirely own thing. Both are really separate levels. OTOH, if users actually provide their own implementation, I guess their intention is rather clear about *not* wanting the original C++ operator for some reason, even if it's implemented. So maybe shadowing isn't actually wrong here. Can't say if it really makes a difference which of the two inline methods above we allow or which one would appear cleaner. Adapting the operator interfaces in Cython should be possible in both cases. Stefan
On Sat, Jul 4, 2015 at 12:43 AM Stefan Behnel <stefan_ml@behnel.de> wrote:
Hi Ian!
Ian Henriksen schrieb am 04.07.2015 um 00:43:
I'm a GSOC student working to make a Cython API for DyND. DyND <https://github.com/libdynd/libdynd> is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead.
Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice. Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this:
a(irange(), 1).vals() = 0;
Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this?
Just an idea, don't know how feasible this is, but we could allow inline special methods in C++ class declarations that implement Python protocols. Example:
cdef extern from ...: cppclass Array2D: int operator[] except + int getItemAt(ssize_t x, ssize_t y) except +
cdef inline __getitem__(self, Py_ssize_t x, Py_ssize_t y): return self.getItemAt(x, y)
def test(): cdef Array2D a return a[1, 2]
Cython could then translate an item access on an Array2D instance into the corresponding special "method" call.
Drawbacks:
1) The example above would conflict with the C++ [] operator, so it would be ambiguous which one is being used in Cython code. Not sure if there's a use case for making both available to Cython code, but that would be difficult to achieve if the need arises.
2) It doesn't solve the general problem of assigning to C++ expressions, especially because it does not extend the syntax allowed by Cython which would still limit what you can do in these fake special methods.
Regarding your proposals, I'd be happy if we could avoid adding syntax support for assigning to function calls. And I agree that the cname assignment hack is really just a big hack. It shouldn't be relied on.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Yes, both this idea and the modified version that redefines operator[] are similar to the idea I had about respecting the cname entries for operator[]. This method would certainly expose a more flexible API for modules that want to do this. It may work in my case, but I worry that getting this into Cython would further complicate the (already lengthy) indexing logic. I'm still uneasy about exporting an API that is fundamentally different from the existing Python and C++ APIs, but making a way to use Python's syntax could help with that. Is there a good way to make a method like this accept Python-like indexing syntax? It would be confusing to put a code definition like this inside an extern block too. Could this syntax be adapted to work outside the extern block while still showing its connection to the original cppclass? Thanks for looking at it! -Ian
Ian Henriksen schrieb am 08.07.2015 um 03:50:
On Sat, Jul 4, 2015 at 12:43 AM Stefan Behnel wrote:
Ian Henriksen schrieb am 04.07.2015 um 00:43:
I'm a GSOC student working to make a Cython API for DyND. DyND <https://github.com/libdynd/libdynd> is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead.
Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice. Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this:
a(irange(), 1).vals() = 0;
Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this?
Just an idea, don't know how feasible this is, but we could allow inline special methods in C++ class declarations that implement Python protocols. Example:
cdef extern from ...: cppclass Array2D: int operator[] except + int getItemAt(ssize_t x, ssize_t y) except +
cdef inline __getitem__(self, Py_ssize_t x, Py_ssize_t y): return self.getItemAt(x, y)
def test(): cdef Array2D a return a[1, 2]
Cython could then translate an item access on an Array2D instance into the corresponding special "method" call.
Drawbacks:
1) The example above would conflict with the C++ [] operator, so it would be ambiguous which one is being used in Cython code. Not sure if there's a use case for making both available to Cython code, but that would be difficult to achieve if the need arises.
2) It doesn't solve the general problem of assigning to C++ expressions, especially because it does not extend the syntax allowed by Cython which would still limit what you can do in these fake special methods.
Regarding your proposals, I'd be happy if we could avoid adding syntax support for assigning to function calls. And I agree that the cname assignment hack is really just a big hack. It shouldn't be relied on.
Yes, both this idea and the modified version that redefines operator[] are similar to the idea I had about respecting the cname entries for operator[]. This method would certainly expose a more flexible API for modules that want to do this. It may work in my case, but I worry that getting this into Cython would further complicate the (already lengthy) indexing logic.
The main problem with the logic in IndexNode is that it predates the infrastructure change that allows node replacements in the analyse_types() methods. It should eventually be split into separate nodes that do different things, e.g. integer indexing into C arrays, Python object item access, C++ operator[] usage, buffer/memory view indexing, memory view slicing, you name it. In any case, adding new functionality can now be done by creating a new node rather than complicating the type analysis code. And any further refactoring would be warmly appreciated. :)
I'm still uneasy about exporting an API that is fundamentally different from the existing Python and C++ APIs, but making a way to use Python's syntax could help with that. Is there a good way to make a method like this accept Python-like indexing syntax? It would be confusing to put a code definition like this inside an extern block too. Could this syntax be adapted to work outside the extern block while still showing its connection to the original cppclass?
The feature of providing inline functions in .pxd files already exists, as does the feature of adding functionality to external extension types by implementing special methods in their declaration. See, for example, the buffer protocol support for old NumPy arrays that we implemented in numpy/__init__pxd (look for "__getbuffer__") or the helper functions in cpython/array.pxd. Allowing to override __getitem__() in an extern C++ class declaration would really only be one step further. The question is whether __getitem__() is the right abstraction to use here as it also only accepts a single argument as input. That would be a tuple in Python for multi-dimensional lookup. It would be nice if the index arguments (e.g. x,y,z for 3 dimensions) could be explicit in the method signature instead, potentially using default arguments if less dimensions should be allowed. Stefan
On Wed, Jul 8, 2015 at 11:20 AM Stefan Behnel <stefan_ml@behnel.de> wrote:
On Sat, Jul 4, 2015 at 12:43 AM Stefan Behnel wrote:
Ian Henriksen schrieb am 04.07.2015 um 00:43:
I'm a GSOC student working to make a Cython API for DyND. DyND <https://github.com/libdynd/libdynd> is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead.
Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice. Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this:
a(irange(), 1).vals() = 0;
Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this?
Just an idea, don't know how feasible this is, but we could allow inline special methods in C++ class declarations that implement Python
Example:
cdef extern from ...: cppclass Array2D: int operator[] except + int getItemAt(ssize_t x, ssize_t y) except +
cdef inline __getitem__(self, Py_ssize_t x, Py_ssize_t y): return self.getItemAt(x, y)
def test(): cdef Array2D a return a[1, 2]
Cython could then translate an item access on an Array2D instance into
corresponding special "method" call.
Drawbacks:
1) The example above would conflict with the C++ [] operator, so it would be ambiguous which one is being used in Cython code. Not sure if
Ian Henriksen schrieb am 08.07.2015 um 03:50: protocols. the there's a
use case for making both available to Cython code, but that would be difficult to achieve if the need arises.
2) It doesn't solve the general problem of assigning to C++ expressions, especially because it does not extend the syntax allowed by Cython which would still limit what you can do in these fake special methods.
Regarding your proposals, I'd be happy if we could avoid adding syntax support for assigning to function calls. And I agree that the cname assignment hack is really just a big hack. It shouldn't be relied on.
Yes, both this idea and the modified version that redefines operator[] are similar to the idea I had about respecting the cname entries for operator[]. This method would certainly expose a more flexible API for modules that want to do this. It may work in my case, but I worry that getting this into Cython would further complicate the (already lengthy) indexing logic.
The main problem with the logic in IndexNode is that it predates the infrastructure change that allows node replacements in the analyse_types() methods. It should eventually be split into separate nodes that do different things, e.g. integer indexing into C arrays, Python object item access, C++ operator[] usage, buffer/memory view indexing, memory view slicing, you name it.
In any case, adding new functionality can now be done by creating a new node rather than complicating the type analysis code. And any further refactoring would be warmly appreciated. :)
I'm still uneasy about exporting an API that is fundamentally different from the existing Python and C++ APIs, but making a way to use Python's syntax could help with that. Is there a good way to make a method like this accept Python-like indexing syntax? It would be confusing to put a code definition like this inside an extern block too. Could this syntax be adapted to work outside the extern block while still showing its connection to the original cppclass?
The feature of providing inline functions in .pxd files already exists, as does the feature of adding functionality to external extension types by implementing special methods in their declaration. See, for example, the buffer protocol support for old NumPy arrays that we implemented in numpy/__init__pxd (look for "__getbuffer__") or the helper functions in cpython/array.pxd.
Allowing to override __getitem__() in an extern C++ class declaration would really only be one step further. The question is whether __getitem__() is the right abstraction to use here as it also only accepts a single argument as input. That would be a tuple in Python for multi-dimensional lookup. It would be nice if the index arguments (e.g. x,y,z for 3 dimensions) could be explicit in the method signature instead, potentially using default arguments if less dimensions should be allowed.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Okay, I see what you mean in the numpy pxd file. With regards to which abstraction to use, __getitem__ and __setitem__ would work best, but we would need to allow a greater number of arguments. We could force users to use something like a C++ vector or tuple, but that creates a bunch of useless temporaries (which we can only hope the C++ compiler will handle properly) and will cause all kinds of grief when mixing code compiled with different standard libraries. That's not the end of the world, but I'd prefer to avoid incompatibilities like that wherever possible. Using __getitem__ and __setitem__ does have the advantage that it would maintain a separation between using an indexing operation on the right or left of an assignment so that libraries could handle the two cases separately. That has the potential to make the Cython API exposed to users simpler than the C++ API in some cases, though overwriting operator[] comes with its own set of problems. Using operator[] as a name for a Cython-level indexing operation would work too. Since we're already overloading the syntax, the name is available. On the other hand, there don't appear to be any additional benefits other than leaving __getitem__ and __setitem__ alone. I feel like using the same name as is used in C++ could be a potential point of confusion. Rewriting a C++ class's handling of indexing in code that primarily provides a wrapper is plenty confusing on its own. Using different names would at least help accentuate the change. With regards to defining these special methods, how would one go about using the C++ version of the syntax? I've been trying to make it so that all the Python compatibility functions in dynd-python can be loaded dynamically via cimport so that the users will only have to find and include the headers from the original C++ library. With the additional special methods like this, the only way I can see to do this is to define C++ functions that operate on the C++ objects, provide their declarations and make Cython shims in a pyx file, then use the Cython shims inside the pxd file to define the special __getitem__ and __setitem__ methods. Even though most users won't see it, that's an awful lot of code for any library writer that wants to expose an interface like this. Do you see any better solution? Thanks for the great suggestions! -Ian Henriksen
participants (3)
-
Ian Henriksen -
Michael Enßlin -
Stefan Behnel