[Numpy-discussion] Best way to expose std::vector to be used with numpy

[John Zwinck]

On Tue, Oct 14, 2014 at 6:19 PM, Daniele Nicolodi <daniele at grinta.net> wrote:
>>     On Mo, 2014-10-13 at 13:35 +0200, Daniele Nicolodi wrote:
>>     > I have a C++ application that collects float, int or complex data in a
>>     > possibly quite large std::vector. The application has some SWIG
>>     > generated python wrappers that expose this vector to python. However,
>>     > the standard way in which SWIG exposes the data is to create a touple
>>     > and pass this to python, where it is very often converted to a numpy
>>     > array for processing. Of course this is not efficient.
>
> Boost Python may be an option as the codebase already depends on Boost,
> but probably not yet on Boost Python. Can you point me to the relevant
> documentation, and maybe to an example? One of the problems I have is
> that the current wrapping is done auto-magically with SWIG and I would
> like to deviate the less possible from that patter.

Some time ago I needed to do something similar.  I fused the NumPy C
API and Boost.Python with a small bit of code which I then
open-sourced as part of a slightly larger library.  The most relevant
part for you is here:
https://github.com/jzwinck/pccl/blob/master/NumPyArray.hpp

In particular, it offers this function:

  // use already-allocated storage for the array
  // (it will not be initialized, and its format must match the given dtype)
  boost::python::object makeNumPyArrayWithData(
    boost::python::list const& dtype, unsigned count, void* data);

What is dtype?  For that you can use another small widget in my
library: https://github.com/jzwinck/pccl/blob/master/NumPyDataType.hpp

So the outline for your use case: you have a C array or C++ vector of
contiguous plain old data.  You create a NumPyDataType(), call
append() on it with your data type, then pass all of the above to
makeNumPyArrayWithData().  The result is a boost::python::object which
you can return from C++ to Python with zero copies made of your data.

Even if you don't decide to use Boost.Python, maybe you will find the
implementation instructive.  The functions I described take only a few
lines.

John Zwinck