Re: [Python-Dev] An updated extended buffer PEP
Carl Banks wrote:
Travis E. Oliphant wrote:
I think we are getting closer. What do you think about Greg's idea of basically making the provider the bufferinfo structure and having the exporter handle copying memory over for shape and strides if it wants to be able to change those before the lock is released.
It seems like it's just a different way to return the data. You could do it by setting values through pointers, or do it by returning a structure. Which way you choose is a minor detail in my opinion. I'd probably favor returning the information in a structure.
I would consider adding two fields to the structure:
size_t structsize; /* size of the structure */ Why is this necessary? can't you get that by sizeof(bufferinfo)?
PyObject* releaser; /* the object you need to call releasebuffer on */ Is this so that another object could be used to manage releases if desired?
-Travis
Travis Oliphant wrote:
Carl Banks wrote:
I think we are getting closer. What do you think about Greg's idea of basically making the provider the bufferinfo structure and having the exporter handle copying memory over for shape and strides if it wants to be able to change those before the lock is released. It seems like it's just a different way to return the data. You could do it by setting values through pointers, or do it by returning a structure. Which way you choose is a minor detail in my opinion. I'd
Travis E. Oliphant wrote: probably favor returning the information in a structure.
I would consider adding two fields to the structure:
size_t structsize; /* size of the structure */ Why is this necessary? can't you get that by sizeof(bufferinfo)?
In case you want to add something later. Though if you did that, it would be a different major release, meaning you'd have to rebuild anyway. They rashly add fields to the PyTypeObject in the same way. :) So never mind.
PyObject* releaser; /* the object you need to call releasebuffer on */ Is this so that another object could be used to manage releases if desired?
Yes, that was a use case I saw for a different "view" object. I don't think it's crucially important to have it, but for exporting objects that delegate management of the buffer to another object, then it would be very helpful if the exporter could tell consumers that the other object is managing the buffer. Suppose A is an exporting object, but it uses a hidden object R to manage the buffer memory. Thus you have A referring to R, like this: A -> R Now object B takes a view of A. If we don't have this field, then B will have to hold a reference to A, like this: B -> A -> R A would be responsible for keeping track of views, and A could not be garbage collected until B disappears. If we do have this field, then A could tell be B to hold a reference to R instead: B -> R A -> R A is no longer obliged to keep track of views, and it can be garbage collected even if B still exists. Here's a concrete example of where it would be useful: consider a ByteBufferSlice object. Basically, the object represents a shared-memory slice of a 1-D array of bytes (for example, Python 3000 bytes object, or an mmap object). Now, if the ByteBufferSlice object could tell the consumer that someone else is managing the buffer, then it wouldn't have to keep track of views, thus simplifying things. P.S. In thinking about this, it occurred to me that there should be a way to lock the buffer without requesting details. ByteBufferSlice would already know the details of the buffer, but it would need to increment the original buffer's lock count. Thus, I propose new fuction: typedef int (*lockbufferproc)(PyObject* self); Carl Banks
Carl Banks wrote:
Here's a concrete example of where it would be useful: consider a ByteBufferSlice object. Basically, the object represents a shared-memory slice of a 1-D array of bytes (for example, Python 3000 bytes object, or an mmap object).
Now, if the ByteBufferSlice object could tell the consumer that someone else is managing the buffer, then it wouldn't have to keep track of views, thus simplifying things.
P.S. In thinking about this, it occurred to me that there should be a way to lock the buffer without requesting details. ByteBufferSlice would already know the details of the buffer, but it would need to increment the original buffer's lock count. Thus, I propose new fuction:
typedef int (*lockbufferproc)(PyObject* self);
And, because real examples are better than philosophical speculations, here's a skeleton implementation of the ByteBufferSlice array, sans boilerplate and error checking, and with some educated guessing about future details: typedef struct { PyObject_HEAD PyObject* releaser; unsigned char* buf; Py_ssize_t length; } ByteBufferSliceObject; PyObject* ByteBufferSlice_new(PyObject* bufobj, Py_ssize_t start, Py_ssize_t end) { ByteBufferSliceObject* self; BufferInfoObject* bufinfo; self = (ByteBufferSliceObject*)type->tp_alloc(type, 0); bufinfo = PyObject_GetBuffer(bufobj); self->releaser = bufinfo->releaser; self->buf = bufinfo->buf + start; self->length = end-start; /* look how soon we're done with this information */ Py_DECREF(bufinfo); return self; } PyObject* ByteBufferSlice_dealloc(PyObject* self) { PyObject_ReleaseBuffer(self->releaser); self->ob_type->tp_free((PyObject*)self); } PyObject* ByteBufferSlice_getbuffer(PyObject* self, int flags) { BufferInfoObject* bufinfo; static Py_ssize_t stridesarray[] = { 1 }; bufinfo = BufferInfo_New(); bufinfo->releaser = self->releaser; bufinfo->writable = 1; bufinfo->buf = self->buf; bufinfo->length = self->length; bufinfo->ndims = 1; bufinfo->strides = stridesarray; bufinfo->size = &self->length; bufinfo->subbufoffsets = NULL; /* Before we go, increase the original buffer's lock count */ PyObject_LockBuffer(self->releaser); return bufinfo; } /* don't define releasebuffer or lockbuffer */ /* only objects that manage buffers themselves would define these */ /* Now look how easy this is */ /* Everything works out if ByteBufferSlice reexports the buffer */ PyObject* ByteBufferSlice_getslice(PyObject* self, Py_ssize_t start, Py_ssize_t end) { return ByteBufferSlice_new(self,start,end); } The implementation of this is very straightforward, and it's easy to see why and how "bufinfo->release" works, and why it'd be useful. It's almost like there's two protocols here: a buffer exporter protocol (getbuffer) and a buffer manager protocol (lockbuffer and releasebuffer). Some objects would support only exporter protocol; others both. Carl Banks
Carl Banks wrote:
/* don't define releasebuffer or lockbuffer */ /* only objects that manage buffers themselves would define these */
That's an advantage, but it's a pretty small one -- the releasebuffer implementation would be very simple in this case. I'm bothered that the releaser field makes the protocol asymmetrical and thus harder to reason about. It would cost me more mental effort to convince myself that a releasebuffer implementation wasn't needed in any particular case than it would to write the one-line implementation otherwise required. -- Greg
Carl Banks wrote:
Now object B takes a view of A. If we don't have this field, then B will have to hold a reference to A, like this:
B -> A -> R
A would be responsible for keeping track of views,
A isn't keeping track of views, it's keeping track of the single object R, which it has to keep a reference to anyway.
and A could not be garbage collected until B disappears.
I'm not convinced that this would be a serious problem. An object that's using a different object to manage the buffer is probably quite small, so it doesn't matter much if it stays around.
Here's a concrete example of where it would be useful: consider a ByteBufferSlice object. Basically, the object represents a shared-memory slice of a 1-D array of bytes (for example, Python 3000 bytes object, or an mmap object).
And this would be a very small object, not worth the trouble of caring whether it stays around a bit longer than needed, IMO.
P.S. In thinking about this, it occurred to me that there should be a way to lock the buffer without requesting details.
Perhaps you could do this by calling getbuffer with NULL for the bufferinfo pointer, and similarly call releasebuffer with NULL to unlock it. -- Greg
participants (3)
-
Carl Banks -
Greg Ewing -
Travis Oliphant