[Python-Dev] An updated extended buffer PEP

[Carl Banks]

Carl Banks wrote:
> Here's a concrete example of where it would be useful: consider a 
> ByteBufferSlice object.  Basically, the object represents a 
> shared-memory slice of a 1-D array of bytes (for example, Python 3000 
> bytes object, or an mmap object).
> 
> Now, if the ByteBufferSlice object could tell the consumer that someone 
> else is managing the buffer, then it wouldn't have to keep track of 
> views, thus simplifying things.
> 
> P.S. In thinking about this, it occurred to me that there should be a 
> way to lock the buffer without requesting details.  ByteBufferSlice 
> would already know the details of the buffer, but it would need to 
> increment the original buffer's lock count.  Thus, I propose new fuction:
> 
> typedef int (*lockbufferproc)(PyObject* self);


And, because real examples are better than philosophical speculations, 
here's a skeleton implementation of the ByteBufferSlice array, sans 
boilerplate and error checking, and with some educated guessing about 
future details:


typedef struct  {
   PyObject_HEAD
   PyObject* releaser;
   unsigned char* buf;
   Py_ssize_t length;
}
ByteBufferSliceObject;


PyObject* ByteBufferSlice_new(PyObject* bufobj, Py_ssize_t start, 
Py_ssize_t end) {
   ByteBufferSliceObject* self;
   BufferInfoObject* bufinfo;

   self = (ByteBufferSliceObject*)type->tp_alloc(type, 0);
   bufinfo = PyObject_GetBuffer(bufobj);

   self->releaser = bufinfo->releaser;
   self->buf = bufinfo->buf + start;
   self->length = end-start;

   /* look how soon we're done with this information */
   Py_DECREF(bufinfo);

   return self;
}


PyObject* ByteBufferSlice_dealloc(PyObject* self) {
   PyObject_ReleaseBuffer(self->releaser);
   self->ob_type->tp_free((PyObject*)self);
}


PyObject* ByteBufferSlice_getbuffer(PyObject* self, int flags) {
   BufferInfoObject* bufinfo;
   static Py_ssize_t stridesarray[] = { 1 };

   bufinfo = BufferInfo_New();
   bufinfo->releaser = self->releaser;
   bufinfo->writable = 1;
   bufinfo->buf = self->buf;
   bufinfo->length = self->length;
   bufinfo->ndims = 1;
   bufinfo->strides = stridesarray;
   bufinfo->size = &self->length;
   bufinfo->subbufoffsets = NULL;

   /* Before we go, increase the original buffer's lock count */
   PyObject_LockBuffer(self->releaser);

   return bufinfo;
}


/* don't define releasebuffer or lockbuffer */
/* only objects that manage buffers themselves would define these */


/* Now look how easy this is */
/* Everything works out if ByteBufferSlice reexports the buffer */

PyObject* ByteBufferSlice_getslice(PyObject* self, Py_ssize_t start, 
Py_ssize_t end) {
   return ByteBufferSlice_new(self,start,end);
}


The implementation of this is very straightforward, and it's easy to see 
why and how "bufinfo->release" works, and why it'd be useful.

It's almost like there's two protocols here: a buffer exporter protocol 
(getbuffer) and a buffer manager protocol (lockbuffer and 
releasebuffer).  Some objects would support only exporter protocol; 
others both.


Carl Banks