Buffer interface in abstract.c?
Hi all, Im trying to slowly wean myself over to the buffer interfaces. My exploration so far indicates that, for most cases, simply replacing "PyString_FromStringAndSize" with "PyBuffer_FromMemory" handles the vast majority of cases, and is preferred when the data contains arbitary bytes. PyArg_ParseTuple("s#", ...) still works correctly as we would hope. However, performing this explicitly is a pain. Looking at getargs.c, the code to achieve this is a little too convoluted to cut-and-paste each time. Therefore, I would like to propose these functions to be added to abstract.c: int PyObject_GetBufferSize(); void *PyObject_GetReadWriteBuffer(); /* or "char *"? */ const void *PyObject_GetReadOnlyBuffer(); Although equivalent functions exist for the buffer object, I can't see the equivalent abstract implementations - ie, that work with any object supporting the protocol. Im willing to provide a patch if there is agreement a) the general idea is good, and b) my specific spelling of the idea is OK (less likely - PyBuffer_* seems better, but loses any implication of being abstract?). Thoughts? Mark.
Mark Hammond wrote:
... Therefore, I would like to propose these functions to be added to abstract.c:
int PyObject_GetBufferSize(); void *PyObject_GetReadWriteBuffer(); /* or "char *"? */ const void *PyObject_GetReadOnlyBuffer();
Although equivalent functions exist for the buffer object, I can't see the equivalent abstract implementations - ie, that work with any object supporting the protocol.
Im willing to provide a patch if there is agreement a) the general idea is good, and b) my specific spelling of the idea is OK (less likely - PyBuffer_* seems better, but loses any implication of being abstract?).
Marc-Andre proposed exactly the same thing back at the end of March (to me and Guido). The two of us hashed out some of the stuff and M.A. came up with a full patch for the stuff. Guido was relatively non-committal at the point one way or another, but said they seemed fine. It appears the stuff never made it into source control. If Marc-Andre can resurface the final proposal/patch, then we'd be set. Until then: use the bufferprocs :-) Cheers, -g -- Greg Stein, http://www.lyra.org/
Greg Stein wrote:
Mark Hammond wrote:
... Therefore, I would like to propose these functions to be added to abstract.c:
int PyObject_GetBufferSize(); void *PyObject_GetReadWriteBuffer(); /* or "char *"? */ const void *PyObject_GetReadOnlyBuffer();
Although equivalent functions exist for the buffer object, I can't see the equivalent abstract implementations - ie, that work with any object supporting the protocol.
Im willing to provide a patch if there is agreement a) the general idea is good, and b) my specific spelling of the idea is OK (less likely - PyBuffer_* seems better, but loses any implication of being abstract?).
Marc-Andre proposed exactly the same thing back at the end of March (to me and Guido). The two of us hashed out some of the stuff and M.A. came up with a full patch for the stuff. Guido was relatively non-committal at the point one way or another, but said they seemed fine. It appears the stuff never made it into source control.
If Marc-Andre can resurface the final proposal/patch, then we'd be set.
Below is the code I currently use. I don't really remember if this is what Greg and I discussed a while back, but I'm sure he'll correct me ;-) Note that you the buffer length is implicitly returned by these APIs. /* Takes an arbitrary object which must support the character (single segment) buffer interface and returns a pointer to a read-only memory location useable as character based input for subsequent processing. buffer and buffer_len are only set in case no error occurrs. Otherwise, -1 is returned and an exception set. */ static int PyObject_AsCharBuffer(PyObject *obj, const char **buffer, int *buffer_len) { PyBufferProcs *pb = obj->ob_type->tp_as_buffer; const char *pp; int len; if ( pb == NULL || pb->bf_getcharbuffer == NULL || pb->bf_getsegcount == NULL ) { PyErr_SetString(PyExc_TypeError, "expected a character buffer object"); goto onError; } if ( (*pb->bf_getsegcount)(obj,NULL) != 1 ) { PyErr_SetString(PyExc_TypeError, "expected a single-segment buffer object"); goto onError; } len = (*pb->bf_getcharbuffer)(obj,0,&pp); if (len < 0) goto onError; *buffer = pp; *buffer_len = len; return 0; onError: return -1; } /* Same as PyObject_AsCharBuffer() except that this API expects a readable (single segment) buffer interface and returns a pointer to a read-only memory location which can contain arbitrary data. buffer and buffer_len are only set in case no error occurrs. Otherwise, -1 is returned and an exception set. */ static int PyObject_AsReadBuffer(PyObject *obj, const void **buffer, int *buffer_len) { PyBufferProcs *pb = obj->ob_type->tp_as_buffer; void *pp; int len; if ( pb == NULL || pb->bf_getreadbuffer == NULL || pb->bf_getsegcount == NULL ) { PyErr_SetString(PyExc_TypeError, "expected a readable buffer object"); goto onError; } if ( (*pb->bf_getsegcount)(obj,NULL) != 1 ) { PyErr_SetString(PyExc_TypeError, "expected a single-segment buffer object"); goto onError; } len = (*pb->bf_getreadbuffer)(obj,0,&pp); if (len < 0) goto onError; *buffer = pp; *buffer_len = len; return 0; onError: return -1; } /* Takes an arbitrary object which must support the writeable (single segment) buffer interface and returns a pointer to a writeable memory location in buffer of size buffer_len. buffer and buffer_len are only set in case no error occurrs. Otherwise, -1 is returned and an exception set. */ static int PyObject_AsWriteBuffer(PyObject *obj, void **buffer, int *buffer_len) { PyBufferProcs *pb = obj->ob_type->tp_as_buffer; void*pp; int len; if ( pb == NULL || pb->bf_getwritebuffer == NULL || pb->bf_getsegcount == NULL ) { PyErr_SetString(PyExc_TypeError, "expected a writeable buffer object"); goto onError; } if ( (*pb->bf_getsegcount)(obj,NULL) != 1 ) { PyErr_SetString(PyExc_TypeError, "expected a single-segment buffer object"); goto onError; } len = (*pb->bf_getwritebuffer)(obj,0,&pp); if (len < 0) goto onError; *buffer = pp; *buffer_len = len; return 0; onError: return -1; } -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 150 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
Greg Stein writes:
Until then: use the bufferprocs :-)
Greg, On the topic of the buffer interface: Have you written documentation for this that I can include in the API reference? Bugging you about this is on my to-do list. ;-) -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
Mark Hammond wrote:
Hi all, Im trying to slowly wean myself over to the buffer interfaces.
OK, I'll bite. Where is the buffer interface documented? I found references to it in various places (e.g. built-in buffer()) but didn't find the interface itself. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
Jim Fulton wrote:
Mark Hammond wrote:
Hi all, Im trying to slowly wean myself over to the buffer interfaces.
OK, I'll bite. Where is the buffer interface documented? I found references to it in various places (e.g. built-in buffer()) but didn't find the interface itself.
I guess it's a read-the-source feature :-) Objects/bufferobject.c and Include/object.h provide a start. Objects/stringobject.c has a "sample" implementation. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 150 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
participants (5)
-
Fred L. Drake -
Greg Stein -
Jim Fulton -
M.-A. Lemburg -
Mark Hammond