[Numpy-discussion] creating/working NumPy-ndarrays in C++

[Dag Sverre Seljebotn]

On 04/12/2012 01:02 PM, Holger Herrlich wrote:
>
> On 04/09/2012 09:19 PM, Dag Sverre Seljebotn wrote:
>> On 04/08/2012 08:25 PM, Holger Herrlich wrote:
>>>
>>> That all sounds like no option -- sad. Cython is no solution cause,
>>> all I want is to leave Python Syntax in favor for strong OOP design
>>> patterns.
>>
>> I'm sorry, I'm trying and trying to make heads and tails of this
>> paragraph, but I don't manage to. If you could rephrase it that would
>> be very helpful. (But I'm afraid that if you believe that C++ is more
>> object-oriented than Python, you'll find most people disagree.
>> Perhaps you meant that you want static typing?)
>
> Yes and further, design patterns known to me are very java (static)
> specific. In Python you can often work around the "rules" (or intension).

Most design patterns I know work well in Python as well, but may 
sometimes be overkill because Python has simpler mechanisms to do the 
same compared to Java.

E.g., I often use the visitor pattern in Python.

Yes, the Python philosophy is that we don't waste time on assuming that 
the user of our code doesn't abuse it. "If you break it, you get to keep 
the pieces."


>
>> Any wrapping tool (Cython, ctypes, probably SWIG but don't know) will
>> allow you to pass the pointer to the data array, the npy_intp* shape
>> array, and the npy_intp* strides array.
>>
>> Really, that's all you need. And given those three, writing a simple
>> C++ class wrapping the arrays and allowing you to conveniently index
>> into the array is done in 15 minutes.
>
> On the scipy-pages I miss C/C++ samples. I got some idea of what to
> include and what to link to by running distutils, but wonder why here is
> no single one gcc command line. (Anyway, read further.)

It's difficult to provide such examples because either you know enough 
technical details that what to do is "obvious", or the amount of 
education needed (about how C works and how CPython works) is too much 
to have in such a place, and it is not specific to the SciPy project.

There is documentation on docs.scipy.org about the C interface to NumPy.

>
>> If you need more than that -- that is, what you want is essentially
>> to "use NumPy from C++", which slicing and ufuncs and reductions and
>> so on -- then you should probably look into a C++ array library (such
>> as Eigen or Blitz++ or the array stuff in boost). Then, pass the
>> aforementioned data and shape/stride arrays to the array library.
>
> I see a bit clearer now that the task can be splited up. For first a
> shared library will be used by Python (embedding C in Python). Ctypes
> might do that. SWIG also. Second, to benefit from: slicing, in sito
> calculation (ufunc?) and using the histogram[2D]() functions as I do in
> Python. It's not necessary, for the second approach, to use Python at
> all. It should be possible to compile a shared library.
>
> Both will become mixed up if the ndarray crosses the library boundary,
> maybe towards Python. More complicated SWIG. Taking care of deallocation.
>
> So I will have a look on Blitz++, Boost, or native C++ arrays (here
> performance will come to play), but I also consider the concept of using
> NumPy-arrays. Or the concept of converting the array at an output layer.

Look at eigen too. And be advised that "native C++ arrays" (if there is 
such a think) should only be used in the 1D form; i.e. where you to 
access element (i, j) in a 2D array do "arr[n * j + i]" on a "double*", 
NOT where you do arr[i][j] on a "double**" (the latter simply doesn't 
work well with any scientific codes).

> I'm not that familiar yet with the C-API of NumPy, but do I see it right
> that in using NumPy a whole Python is embedded to the Program?

Yes (of course). Remember that large parts of the NumPy package is 
written in Python code too.

NumPy and Python. "Using NumPy (the library) from C++" doesn't really 
make sense and no tools will let you do that easily; the idea is to 
transfer the *data* = what the NumPy array wraps.

If you want to use C++, it's because you want to use C++ tools anyway, 
right?

If you just want to use "Python with types" then Cython is really the 
only option.

> Inclusively the garbage collector. Is this, why distutils allways link
> with pthread (linux).

I believe distutils isn't too smart; it always uses the flags that were 
used to compile Python, and Python was compiled with pthread on your 
machine.

Dag