exposing C array to python namespace: NumPy and array module.

[Craig Ringer]

On Sat, 2005-01-01 at 08:18, Bo Peng wrote:

> Python's array module is built-in, easy to use, but *without* a 
> FromLenAndData function! Even the buffer interface provides only 'get 
> buffer' but no 'set buffer' functions. Could anyone tell me how I can 
> create an array object from existing data?

Python has no array objects in the core language, only lists. The
distinction is important when discussing numarray etc, because Python
lists and NumPy etc arrays are very different.

While you can build a Python list from a subsection of your C array,
changes made in Python won't be pushed back to the C array it was
created from. If this is OK, you can probably build the list using just
a for loop - I'm not sure if there are any more efficient methods for
variable length lists.

If the Python user needs to be able to change the underlying array, I'd
probably drop the use of the built-in list class entirely and write my
own class that looks like a list (and smells like a list, and tastes
like a list - lucky we didn't step in it!). It can be pretty simple,
providing as few of the list protocol methods as:

__getitem__   (a PyList_GetItem equivalent)
__setitem__   (a PyList_SetItem equivalent)

and preferably:

__len__
__iter__

or as much of the list protocol as documented on the Python/C API page
as you need.

I'd probably implement the class in Python, and have my extension module
provide a couple of simple functions to the underlying C array. These
could be considered private to your list class. That'd make writing
things like the __iter__ method much nicer, while still letting you
implement __len__, __getitem__, __setitem__, etc in C. For example, I
might write:

class CArray(object):
    def __init__(self, ...):
        ...

    def __getitem__(self, index):
        _carray_getitem(self, index)

    def __len__(self):
        _carray_len(self, index)

    def __iter__(self):
        # build and return an interator using Python
        ...


If you want to write part of your extension module in Python and part in
C, there are two main ways to do it. The usual way is to write a
'wrapper' in Python that imports the C parts, wraps them where necessary
or just pushes them into its own namespace, etc.

The less common way is to import __builtins__ and __main__ into your C
extension module's namespace then PyRun_String() python code in it to
set things up. I find this approach MUCH more useful when embedding
Python in an app and I only want to write small bits of my module in
Python.


The other alternative is to code your class entirely in C, implementing
all the __methods__ as C functions. Unattractive as far as I'm
concerned, but then I find constructing classes using Python's C API
irritating and less clear than it could be.



Here's the code -- hideously reformatted to avoid wrapping in the mail -
in my initmodule() function that I use to set up the module so that
Python code can execute in its namespace. You can ignore the
const_cast<> stuff, chances are your compiler will ignore the const
problems.

----
// 'd' is the dictionary of the extension module, as obtained
// with PyModule_GetDict(module)

PyObject* builtinModule = PyImport_ImportModuleEx(
    const_cast<char*>("__builtin__"),
    d, d, Py_BuildValue(const_cast<char*>("[]"))
    );
if (builtinModule == NULL)
{
// Error handling will not be shown; it'll depend on your module anyway.
}
PyDict_SetItemString(d, const_cast<char*>("__builtin__"),
                     builtinModule);

PyObject* exceptionsModule = PyImport_ImportModuleEx(
     const_cast<char*>("exceptions"), d, d,
     Py_BuildValue(const_cast<char*>("[]"))
     );
if (exceptionsModule == NULL) {}
PyDict_SetItemString(d, const_cast<char*>("exceptions"),
                     exceptionsModule);

// We can now run Python code in the module's namespace. For
// example (untested), as my real examples wouldn't do you any
// good, they're too bound to the internal API of my module:

QString python_code = "";
python_code += "def sample_function():\n";
python_code += "    print \"See, it worked\"\n";
// My app sets sysdefaultencoding to utf-8, hence:
char* python_code_cstring = python_code.utf8();

// Note that we pass our module dictionary as both
// locals and globals. This makes the code effectively
// run "in" the extension module, as if it was being
// run during loading of a Python module after an 
// 'import' statement.
PyObject* result = PyRun_String(python_code_cstring,
                                Py_file_input,
                                d,d);
if (result == NULL)
{
    qDebug("Python code to declare sample_function failed!");
    PyErr_Print(); // also clears the exception
}
// Because 'result' may be NULL, not a PyObject*, we must call PyXDECREF
not Py_DECREF
Py_XDECREF(result);


--

Ugh - I'd forgotten how ugly C code limited to 80 cols and without
syntax highlighting really was. Especially when the reformatting is done
as badly as I've done it. I hope you can make some sense out of that,
anyway. Note that once the setup is done you can run as many python code
snippets as you want, for declaring variables, functions, classes, etc.

In my case, its easier to execute snippets as shown above than it is to
worry about the module search path and wrapping things using a Python
module. If you're doing substantial amounts of Python coding for your
module, you'll almost certainly be better off writing a Python module
that uses your C module internally (see PIL for a good example of this).

--
Craig Ringer