[C++-sig] Publish a big C++ vector to Python

[Nat Goodspeed]

I'm interested by the recent discussion about the new indexing suite:

http://mail.python.org/pipermail/c++-sig/2006-June/010830.html

I will soon want to be able to pass very large volumes of numeric data
from C++ into a Python module. From the Python point of view, we can
treat it as read-only.

The C++ code internally uses std::vector<Vector3>, where each Vector3
object contains -- surprise -- double c[3].

We'd like to be able to export it to various different file formats. So
I expect the Python code to have to make at least one pass over the
data.

But I want to avoid requiring the Python code to make an additional pass
solely for the purpose of copying to another memory structure. If we
must do that, we should do it on the C++ side beforehand. (Yes, the
memory penalty is of some concern as well.)

What approach would you recommend?

I imagine a few different possibilities, none of which I've researched
yet:

1. Use the existing indexing suite to publish std::vector<Vector3>, if
it would suffice for our purposes.

2. Use the new indexing suite to publish std::vector<Vector3>. Perhaps
fortunately, we're still using Boost 1.31. But it sounds as if there are
other hurdles to jump?

3. Make a preliminary C++ pass to construct a Python array object
containing doubles, three array entries per Vector3. (I don't yet know
how to construct a Python 'array' of appropriate size in C++.)

4. Make a preliminary C++ pass to construct a NumPy array containing
doubles, as above. (I don't yet know how to do that either.)

5. Abstract away the whole problem by publishing a little proxy class
with access methods. (Could that possibly be as fast, within an order of
magnitude? For our purposes, "fast enough" means that memory access time
is still swamped by disk-write time.)

6. Other suggestions?

We'll be adding this to a large existing hand-coded boost::python
module, so -- although I'm intrigued -- converting to pyplusplus doesn't
seem like the short path.

Thanks for your suggestions!