[PYTHON MATRIX-SIG] Calling NumPy from C

[Nigel O'Brian]

Sorry if this is a bit long -- it is more a suggestion than a request.

In NumPy1.0a4 the routine 
------------------------------------------------------------------------
  array_from_existing_data
    = PyArray_FromDimsAndData(n_dims, dims[n_dims], item_type, old_data);
------------------------------------------------------------------------
is declared in arraymodule.h but, *not recommended*. But it seems to me that
this ought to be a useful feature of the C API.

Checking through the archive, I see this matter was raised about
a year ago by Paul Dubois, but apparently dismissed --  e.g.
------------------------------------------------------------------------
From: hinsenk@ere.umontreal.ca (Hinsen Konrad)
Subject: Re: [PYTHON MATRIX-SIG] A related idea

KH: As for C, the best way would be to leave allocation completely to
KH: Python. The matrix module would provide functions to allocate and free
KH: matrices that can be called from the C code just like malloc() and
KH: free(). These functions would take care of the reference from the C
KH: code.
KH: 
KH: Of course the C code is also free to handle its own matrices and
KH: manage them completely (e.g. workspace in library functions). These
KH: would never be accessed from Python; in fact, Python wouldn't even
KH: know about their existence.
------------------------------------------------------------------------
But suppose I have a canned optimization routine in C, which is
finding the minimum of an `objective function' F of N real variables.

This routine expects to be given an array of N doubles, containing the
initial guess as input and in which it leaves the minimizing solution
as output.  The routine also gets passed the calling address of the
objective function F.

Inside the routine, the objective function F gets called with an argument
specifying a (possibly different) array of N doubles on which F is to
be evaluated. This array may be workspace allocated by the optimization
routine itself, unknown to Python.

Suppose one wants a Python interface to the optimization routine,
which can set parameters and allocate an array for input and output.
This is easy enough, but a difficulty arises if one wants to define
the objective function in Python, so the C/Python interface needs a
*callback* to the Python interpreter (using PyEval_CallObject, I
guess). Then F would be a C wrapper for this callback.

But suppose the optimization routine just gives F a pointer to a block
of data allocated by the routine itself. For this to make any sense to
the Numerical Python routines it needs to be `wrapped' inside an array
object and `FromDimsAndData' is the obvious way to do this.

The problem then arises (and this seems to be the source of the
warning) of how to delete the Python array structure without also
freeing the enclosed data (to the surprise of the optimization routine
which allocated it).

This seems to be a general problem which arises when one wants
to fully wrap numerical routines which have functions as parameters -
optimization routines, differential equation integrators, etc.

Question: Is it possible to place a flag in the PyArray structure to
indicate the ownership of the enclosed data? If the array is created
from pre-existing data it would then be possible for Python, or the
C API, to delete the enclosing structure without freeing the data.

I can easily work around this problem by just copying the data into an
existing Python array on each function call, which is probably not a
serious performance overhead. Just less elegant.

Possibly Related Question: If B is a subarray of A, created by e.g. B
= A[3:6].  How does B get deleted without attempting to free the data
it shares with A?

Or is there a better way to look at this whole thing?
--------------------------------------------------------------------
Nigel O'Brian                             obrian_n@maths.usyd.edu.au
School of Mathematics                     Ph: (02)-9351-4083
University of Sydney                      FAX: (02)-9351-4534                
Sydney NSW 2006, Australia

=================
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================