Efficient data flow between orthogonal C extensions?

[Pierre Schnizer]

mlh at furu.idi.ntnu.no (Magnus Lie Hetland) writes:

> In article <87el29rpmv.fsf at smtp.gsi.de>, Pierre Schnizer wrote:
> >mlh at furu.idi.ntnu.no (Magnus Lie Hetland) writes:
> >
> [snip]
> >If you are using callbacks,
> 
> That was, at least, my original idea, although a more centralized
> version would also be possible.
> 
> >you can wrap the callbacks with swig.
> 
> Sounds good....
> 
> >While swig will not generate helpers to call python functions from C,
> >you can pass pointers to C functions.
> >http://www.swig.org/Doc1.3/SWIG.html#n30
Yes but as far as I understand the callbacks must be implemented in C.
> 
> Hm. I don't quite get this... This piece of documentation does seem to
> work with Python-implemented callbacks... And the name-based approach
> strikes me as a bit impractical. A callback has to have a given name
> to be applicable -- or am I misunderstanding? Even though this doesn't
> mean that the extensions need to know about each other, conceptually,
> the would need to know what the methods of the other extensions are
> called...
> 
> It would be nice to be able to supply the extension with an arbitrary
> object (e.g. a PyCObject) and have it cast it and use it itself. Of
> course, assigning a function to a variable with the correct name (if
> that would work) isn't too much of a nuisance.
> 
> Any input on this? Am I misunderstanding the functionality? And... I
> suppose the declared callbacks would be in the namespace of the
> extension module? (Otherwise, how would it be imported?) In that case,
> wouldn't it be necessary to assign a function pointer to it anyway?
> Hm...
> 
> I guess I'll have to look into whether this is the way to go (and
> perhaps actually try it before speculating too much ;). Thanks for the
> input.
> 
Perhaps I should have been more explicit. If I get your viewpoint correctly, I
can answer: SWIG generally can do the job you described above. 

Here now the explanation in more detail. You can wrap your C Function (which 
in your
case happens to live in an other module). SWIG will export that function to
Python as a string or a PyCObject, as you wish. You pass this "variable" to
the module. The SWIG wrapper will convert the PyCObject into the approbriate
callback and call your C extension. 


What I wanted to stress and what misled you, (I guess), was that SWIG does
not help you to automatise writting a C callback, which will call the python
function. (At least I do not know about ...).  For example f2py can parses the
code of a Fortran ODE solver and will automatically generate a wrapper and a
callback wrapper. Thus one can use it to integrate a function written in
Python. In the SWIG case you have to write the callback wrapper by hand. But
as you want to use C Functions in both cases that is not too much of concern
for you.

Perhaps a example. Consider the gsl root solver (partly taken from 
http://pygsl.sf.net).
The swig interface part for the root solver looks like that:

<snip file = src/callback/gsl_roots.i, line = 42>
extern int gsl_root_fsolver_set (gsl_root_fsolver * s, gsl_function * BUFFER,
 double X_LOWER, double X_UPPER)
<snip>

This is the C description from the header. gsl_function is the interesting
part. This is the function to integrate. If you had a C function you would
wrap this function and then pass the wrapped "variable" from python as a 
variable.

Here it is interesting to be able to integrate a python function.
Therefore a C wrapper was written. This function is called by the gsl roots
solver.
<snip>
 double 
 PyGSL_function_wrap(double x, void * params)
 {
      double result;
      callback_function_params *p = NULL;
 
      p = (callback_function_params *) params;
      tmp = PyEval_CallObject(p->function, p->arguments);
      result = PyFloat_AsDouble(tmp);	
      return result;
 }
<snip>

On the entry side, there is an other wrapper which puts the callback and its
arguments in the p struct and sets PyGSL_function_wrap as callback for the
root solver. 


I hope that cleared the point a bit what I wanted to express.
Regards
	Pierre