[Python-Dev] instances as buffers

[Greg Stein]

David Ascher wrote:
>...
> I'm using buffer objects to pass pointers to structs back and forth
> between Python and Windows (Win32's GUI scheme involves sending messages
> to functions with, oftentimes, addresses of structs as arguments, and
> expect the called function to modify the struct directly -- similarly, I
> must call Win32 functions w/ pointers to memory that Windows will modify,
> and be able to read the modified memory). With 'raw' buffer object
> manipulation (after exposing the PyBuffer_FromMemoryReadWrite call to
> Python), this works fine [*].  So far, no instances.

How do you manage the lifetimes of the memory and objects?
PyBuffer_FromReadWriteMemory() creates a buffer object that points to
memory. You need to ensure that the memory exists as long as the buffer
does.

Would it make more sense to use PyBuffer_New(size)?

Note: PyBuffer_FromMemory() (read-only) was built primarily for the case
where you have static constants in an extension module (strings, code
objects, etc) and want to expose them to Python without copying them
into the heap. Currently, stuff like this must be copied into a dynamic
string object to be exposed to Python. The
PyBuffer_FromReadWriteMemory() is there for symmetry, but it can be very
dangerous to use because of the lifetime problem.

PyBuffer_New() allocates its own memory, so the lifetimes are managed
properly. PyBuffer_From*Object maintains a reference to the target
object so that the target object can be kept around at least as long as
the buffer.

> I also have a class which allows the user to describe the buffer memory
> layout in a natural way given the C struct, and manipulate the buffer
> layout w/ getattr/setattr.  For example:

This is a very cool class. Mark and I had discussed doing something just
like this (a while back) for some of the COM stuff. Basically, we'd want
to generate these structures from type libraries.

>...
> The only hitch is that to send the buffer to the SWIG'ed function call, I
> have three options, none ideal:
> 
>    1) define a __str__ method which makes a string of the buffer and pass
>       that to the function which expects an "s#" argument.  This send
>       a copy of the data, not the address.  As a result, this works
>       well for structs which I create from scratch as long as I don't need
>       to see any changes that Windows might have performed on the memory.

Note that "s#" can be used directly against the buffer object. You could
pass it directly rather than via __str__.

>    2) send the instance but make up my own 'get-the-instance-as-buffer'
>       API -- complicates extension module code.
> 
>    3) send the buffer attribute of the instance instead of the instance --
>       complicates Python code, and the C code isn't trivial because there
>       is no 'buffer' typecode for PyArg_ParseTuple().
> 
> If I could define an
> 
>   def __aswritebuffer__
> 
> and if there was a PyArg_ParseTuple() typecode associated with read/write
> buffers (I nominate 'w'!), I believe things would be simpler -- I could
> then send the instance, specify in the PyArgParse_Tuple that I want a
> pointer to memory, and I'd be golden.
> 
> What did I miss?

You can do #3 today since there is a buffer typecode present ("w" or
"w#"). It will complicate Python code a bit since you need to pass the
buffer, but it is the simplest of the three options.

Allowing instances to return buffers does seem to make sense, although
it exposes a lot of underlying machinery at the Python level. It might
be nicer to find a better semantic for this than just exposing the
buffer interface slots.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/