State-of-the-art to use a C/C++ library from Python
Hi all There are several ways on how to use C/C++ code from Python with NumPy, as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11. I'm not quite sure which approach is state-of-the-art as of 2016. How would you do it if you had to make a C/C++ library available in Python right now? In my case, I have a C library with some scientific functions on matrices and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize. Best regards, Michael
On Wed, Aug 31, 2016 at 12:28 PM, Michael Bieri <mibieri@gmail.com> wrote:
Hi all
There are several ways on how to use C/C++ code from Python with NumPy,
as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11.
I'm not quite sure which approach is state-of-the-art as of 2016. How
would you do it if you had to make a C/C++ library available in Python right now?
In my case, I have a C library with some scientific functions on matrices
and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize. I usually reach for Cython: http://cython.org/ http://docs.cython.org/en/latest/src/userguide/memoryviews.html -- Robert Kern
Michael Bieri wrote:
Hi all
There are several ways on how to use C/C++ code from Python with NumPy, as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11.
I'm not quite sure which approach is state-of-the-art as of 2016. How would you do it if you had to make a C/C++ library available in Python right now?
In my case, I have a C library with some scientific functions on matrices and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize.
Best regards, Michael
I prefer ndarray: https://github.com/ndarray/ndarray
On Wed, Aug 31, 2016 at 2:28 PM, Michael Bieri <mibieri@gmail.com> wrote:
Hi all
There are several ways on how to use C/C++ code from Python with NumPy, as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11.
I'm not quite sure which approach is state-of-the-art as of 2016. How would you do it if you had to make a C/C++ library available in Python right now?
In my case, I have a C library with some scientific functions on matrices and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize.
I have been delighted with Cython for this purpose. Great integration with NumPy (you can access numpy arrays directly as C arrays), very python like syntax and amazing performance. Good luck, David
I just wanted to follow up on the C++ side of OP email - Cython has quite a few difficulties working with C++ code at the moment. It's really more of a C solution most of the time and you must split things up into a mostly C call interface (that is the C code Cython can call) and limit exposure/complications with templates and complex C++11+ constructs. This may change in the longer term but in the near, that is the state. I used to use Boost.Python but I'm getting my feet wet with Pybind (which is basically the same api but works more as you expect it to with it's signature/type plumbing (including std::shared_ptr islanding), with some other C++11 based improvements, and is header only + submodule friendly!). I also remembered ndarray thanks to Neal's post but I haven't figured out how to leverage it better than pybind, at the moment. I'd be interested to see ndarray gain support for pybind interoperability... -Jason On Wed, Aug 31, 2016 at 1:08 PM, David Morris <othalan@othalan.net> wrote:
On Wed, Aug 31, 2016 at 2:28 PM, Michael Bieri <mibieri@gmail.com> wrote:
Hi all
There are several ways on how to use C/C++ code from Python with NumPy, as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11.
I'm not quite sure which approach is state-of-the-art as of 2016. How would you do it if you had to make a C/C++ library available in Python right now?
In my case, I have a C library with some scientific functions on matrices and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize.
I have been delighted with Cython for this purpose. Great integration with NumPy (you can access numpy arrays directly as C arrays), very python like syntax and amazing performance.
Good luck,
David
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
We use Cython very heavily in DyND's Python bindings. It has worked well for us even when working with some very modern C++. That said, a lot depends on exactly which C++ features you want to expose as a part of the interface. Interfaces that require things like non-type template parameters or variadic templates will often require a some extra C++ code to work them in to something that Cython can understand. In my experience, those particular limitations haven't been that hard to work with. Best, Ian Henriksen On Wed, Aug 31, 2016 at 12:20 PM Jason Newton <nevion@gmail.com> wrote:
I just wanted to follow up on the C++ side of OP email - Cython has quite a few difficulties working with C++ code at the moment. It's really more of a C solution most of the time and you must split things up into a mostly C call interface (that is the C code Cython can call) and limit exposure/complications with templates and complex C++11+ constructs. This may change in the longer term but in the near, that is the state.
I used to use Boost.Python but I'm getting my feet wet with Pybind (which is basically the same api but works more as you expect it to with it's signature/type plumbing (including std::shared_ptr islanding), with some other C++11 based improvements, and is header only + submodule friendly!). I also remembered ndarray thanks to Neal's post but I haven't figured out how to leverage it better than pybind, at the moment. I'd be interested to see ndarray gain support for pybind interoperability...
-Jason
On Wed, Aug 31, 2016 at 1:08 PM, David Morris <othalan@othalan.net> wrote:
On Wed, Aug 31, 2016 at 2:28 PM, Michael Bieri <mibieri@gmail.com> wrote:
Hi all
There are several ways on how to use C/C++ code from Python with NumPy, as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11.
I'm not quite sure which approach is state-of-the-art as of 2016. How would you do it if you had to make a C/C++ library available in Python right now?
In my case, I have a C library with some scientific functions on matrices and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize.
I have been delighted with Cython for this purpose. Great integration with NumPy (you can access numpy arrays directly as C arrays), very python like syntax and amazing performance.
Good luck,
David
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Hey Ian - I hope I gave Cython a fair comment, but I have to add the disclaimer that your capability to understand/implement those solutions/workarounds in that project is greatly enhanced from your knowing the innards of Cython from being core developer on the Cython project. This doesn't detract from DyDN's accomplishments (if nothing it means Cython users should look there for how to use C++ with Cython and the workarounds used + shortcomings) but I would not expect not everyone would want to jump through those hoops to get things working without a firm understanding of Cython's edges, and all this potential for special/hack adaption code is still something to keep in mind when comparing to something more straight forward and easier to understand coming from a more pure C/C++ side, where things are a bit more dangerous and fairly more verbose but make play with the language and environment first-class (like Boost.Python/pybind). Since this thread is a survey over state and options it's my intent just to make sure readers have something bare in mind for current pros/cons of the approaches. -Jason On Wed, Aug 31, 2016 at 2:17 PM, Ian Henriksen < insertinterestingnamehere@gmail.com> wrote:
We use Cython very heavily in DyND's Python bindings. It has worked well for us even when working with some very modern C++. That said, a lot depends on exactly which C++ features you want to expose as a part of the interface. Interfaces that require things like non-type template parameters or variadic templates will often require a some extra C++ code to work them in to something that Cython can understand. In my experience, those particular limitations haven't been that hard to work with. Best, Ian Henriksen
On Wed, Aug 31, 2016 at 12:20 PM Jason Newton <nevion@gmail.com> wrote:
I just wanted to follow up on the C++ side of OP email - Cython has quite a few difficulties working with C++ code at the moment. It's really more of a C solution most of the time and you must split things up into a mostly C call interface (that is the C code Cython can call) and limit exposure/complications with templates and complex C++11+ constructs. This may change in the longer term but in the near, that is the state.
I used to use Boost.Python but I'm getting my feet wet with Pybind (which is basically the same api but works more as you expect it to with it's signature/type plumbing (including std::shared_ptr islanding), with some other C++11 based improvements, and is header only + submodule friendly!). I also remembered ndarray thanks to Neal's post but I haven't figured out how to leverage it better than pybind, at the moment. I'd be interested to see ndarray gain support for pybind interoperability...
-Jason
On Wed, Aug 31, 2016 at 1:08 PM, David Morris <othalan@othalan.net> wrote:
On Wed, Aug 31, 2016 at 2:28 PM, Michael Bieri <mibieri@gmail.com> wrote:
Hi all
There are several ways on how to use C/C++ code from Python with NumPy, as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11.
I'm not quite sure which approach is state-of-the-art as of 2016. How would you do it if you had to make a C/C++ library available in Python right now?
In my case, I have a C library with some scientific functions on matrices and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize.
I have been delighted with Cython for this purpose. Great integration with NumPy (you can access numpy arrays directly as C arrays), very python like syntax and amazing performance.
Good luck,
David
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Wed, Aug 31, 2016, at 13:57, Jason Newton wrote:
Hey Ian - I hope I gave Cython a fair comment, but I have to add the disclaimer that your capability to understand/implement those solutions/workarounds in that project is greatly enhanced from your knowing the innards of Cython from being core developer on the Cython project. This doesn't detract from DyDN's accomplishments (if nothing it means Cython users should look there for how to use C++ with Cython and the workarounds used + shortcomings) but I would not expect not everyone would want to jump through those hoops to get things working without a firm understanding of Cython's edges, and all this potential for special/hack adaption code is still something to keep in mind when comparing to something more straight forward and easier to understand coming from a more pure C/C++ side, where things are a bit more dangerous and fairly more verbose but make play with the language and environment first-class (like Boost.Python/pybind). Since this thread is a survey over state and options it's my intent just to make sure readers have something bare in mind for current pros/cons of the approaches.
There are many teaching resources available for Cython, after which exposure to sharp edges may be greatly reduced. See, e.g., https://github.com/stefanv/teaching/blob/master/2014_assp_split_cython/slide... and accompanying problems and exercises at https://github.com/stefanv/teaching/tree/master/2014_assp_split_cython Stéfan
On Wed, Aug 31, 2016 at 3:57 PM Jason Newton <nevion@gmail.com> wrote:
Hey Ian - I hope I gave Cython a fair comment, but I have to add the disclaimer that your capability to understand/implement those solutions/workarounds in that project is greatly enhanced from your knowing the innards of Cython from being core developer on the Cython project. This doesn't detract from DyDN's accomplishments (if nothing it means Cython users should look there for how to use C++ with Cython and the workarounds used + shortcomings) but I would not expect not everyone would want to jump through those hoops to get things working without a firm understanding of Cython's edges, and all this potential for special/hack adaption code is still something to keep in mind when comparing to something more straight forward and easier to understand coming from a more pure C/C++ side, where things are a bit more dangerous and fairly more verbose but make play with the language and environment first-class (like Boost.Python/pybind). Since this thread is a survey over state and options it's my intent just to make sure readers have something bare in mind for current pros/cons of the approaches.
-Jason
No offense taken at all. I'm actually not a Cython developer, just a frequent contributor. That said, knowing the compiler internals certainly helps when finding workarounds and building intermediate interfaces. My main point was just that, in my experience, Cython has worked well for many things beyond plain C interfaces and that workarounds (hackery entirely aside) for any missing features are usually manageable. Given that my perspective is a bit different in that regard, it seemed worth chiming in on the discussion. I suppose the moral of the story is that there's still not a clear cut "best" way of building wrappers and that your mileage may vary depending on what features you need. Thanks, Ian Henriksen
Jason Newton wrote:
I just wanted to follow up on the C++ side of OP email - Cython has quite a few difficulties working with C++ code at the moment. It's really more of a C solution most of the time and you must split things up into a mostly C call interface (that is the C code Cython can call) and limit exposure/complications with templates and complex C++11+ constructs. This may change in the longer term but in the near, that is the state.
I used to use Boost.Python but I'm getting my feet wet with Pybind (which is basically the same api but works more as you expect it to with it's signature/type plumbing (including std::shared_ptr islanding), with some other C++11 based improvements, and is header only + submodule friendly!). I also remembered ndarray thanks to Neal's post but I haven't figured out how to leverage it better than pybind, at the moment. I'd be interested to see ndarray gain support for pybind interoperability...
-Jason
On Wed, Aug 31, 2016 at 1:08 PM, David Morris <othalan@othalan.net> wrote:
On Wed, Aug 31, 2016 at 2:28 PM, Michael Bieri <mibieri@gmail.com> wrote:
Hi all
There are several ways on how to use C/C++ code from Python with NumPy, as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11.
I'm not quite sure which approach is state-of-the-art as of 2016. How would you do it if you had to make a C/C++ library available in Python right now?
In my case, I have a C library with some scientific functions on matrices and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize.
I have been delighted with Cython for this purpose. Great integration with NumPy (you can access numpy arrays directly as C arrays), very python like syntax and amazing performance.
Good luck,
David
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
pybind11 looks very nice. My problem is that the numpy API exposed by pybind11 is fairly weak at this point, as far as I can see from the docs. ndarray exposes a lot of functionality through the Array object, including convenient indexing and slicing. AFAICT, the interface in pybind11 is pretty low level - just pointers. There is also some functionality exposed by pybind11 using eigen. Personally, I find eigen rather baroque, and only use it when I see no alternative.
+1 on pybind11. Sylvain On Wed, Aug 31, 2016 at 1:28 PM, Michael Bieri <mibieri@gmail.com> wrote:
Hi all
There are several ways on how to use C/C++ code from Python with NumPy, as given in http://docs.scipy.org/doc/numpy/user/c-info.html . Furthermore, there's at least pybind11.
I'm not quite sure which approach is state-of-the-art as of 2016. How would you do it if you had to make a C/C++ library available in Python right now?
In my case, I have a C library with some scientific functions on matrices and vectors. You will typically call a few functions to configure the computation, then hand over some pointers to existing buffers containing vector data, then start the computation, and finally read back the data. The library also can use MPI to parallelize.
Best regards, Michael
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (8)
-
David Morris -
Ian Henriksen -
Jason Newton -
Michael Bieri -
Neal Becker -
Robert Kern -
Stefan van der Walt -
Sylvain Corlay