[Python-Dev] PEP 298, final (?) version
Thomas Heller
thomas.heller@ion-tof.com
Thu, 1 Aug 2002 11:07:42 +0200
Here is PEP 298 in it's near final version (not yet checked in).
It seems to me we have to end the discussion and I'm quite
happy with it. If accepted in this form, I can start the
implementation right after the end of my vacation, second half
of August.
The only thing I consider worth changing is to rename the
whole stuff from 'fixed buffer interface' to 'locked buffer
interface', which makes more sense at the current state.
Thomas
-----
PEP: 298
Title: The Fixed Buffer Interface
Version: $Revision: 1.4 $
Last-Modified: $Date: 2002/07/31 18:48:36 $
Author: Thomas Heller <theller@python.net>
Status: Draft
Type: Standards Track
Created: 26-Jul-2002
Python-Version: 2.3
Post-History: 30-Jul-2002, 1-Aug-2002
Abstract
This PEP proposes an extension to the buffer interface called the
'fixed buffer interface'.
The fixed buffer interface fixes the flaws of the 'old' buffer
interface [1] as defined in Python versions up to and including
2.2, and has the following semantics:
The lifetime of the retrieved pointer is clearly defined and
controlled by the client.
The buffer size is returned as a 'size_t' data type, which
allows access to large buffers on platforms where sizeof(int)
!= sizeof(void *).
(Guido comments: This second sounds like a change we could also
make to the "old" buffer interface, if we introduce another flag
bit that's *not* part of the default flags.)
Specification
The fixed buffer interface exposes new functions which return the
size and the pointer to the internal memory block of any python
object which chooses to implement this interface.
Retrieving a buffer from an object puts this object in a locked
state during which the buffer may not be freed, resized, or
reallocated.
The object must be unlocked again by releasing the buffer if it's
no longer used by calling another function in the fixed buffer
interface. If the object never resizes or reallocates the buffer
during it's lifetime, this function may be NULL. Failure to call
this function (if it is != NULL) is a programming error and may
have unexpected results.
The fixed buffer interface omits the memory segment model which is
present in the old buffer interface - only a single memory block
can be exposed.
Implementation
Define a new flag in Include/object.h:
/* PyBufferProcs contains bf_acquirefixedreadbuffer,
bf_acquirefixedwritebuffer, and bf_releasefixedbuffer */
#define Py_TPFLAGS_HAVE_FIXEDBUFFER (1L<<15)
This flag would be included in Py_TPFLAGS_DEFAULT:
#define Py_TPFLAGS_DEFAULT ( \
....
Py_TPFLAGS_HAVE_FIXEDBUFFER | \
....
0)
Extend the PyBufferProcs structure by new fields in
Include/object.h:
typedef size_t (*acquirefixedreadbufferproc)(PyObject *,
const void **);
typedef size_t (*acquirefixedwritebufferproc)(PyObject *,
void **);
typedef void (*releasefixedbufferproc)(PyObject *);
typedef struct {
getreadbufferproc bf_getreadbuffer;
getwritebufferproc bf_getwritebuffer;
getsegcountproc bf_getsegcount;
getcharbufferproc bf_getcharbuffer;
/* fixed buffer interface functions */
acquirefixedreadbufferproc bf_acquirefixedreadbuffer;
acquirefixedwritebufferproc bf_acquirefixedwritebuffer;
releasefixedbufferproc bf_releasefixedbuffer;
} PyBufferProcs;
The new fields are present if the Py_TPFLAGS_HAVE_FIXEDBUFFER
flag is set in the object's type.
The Py_TPFLAGS_HAVE_FIXEDBUFFER flag implies the
Py_TPFLAGS_HAVE_GETCHARBUFFER flag.
The acquirefixedreadbufferproc and acquirefixedwritebufferproc
functions return the size in bytes of the memory block on success,
and fill in the passed void * pointer on success. If these
functions fail - either because an error occurs or no memory block
is exposed - they must set the void * pointer to NULL and raise an
exception. The return value is undefined in these cases and
should not be used.
If calls to these functions succeed, eventually the buffer must be
released by a call to the releasefixedbufferproc, supplying the
original object as argument. The releasefixedbufferproc cannot
fail.
Usually these functions aren't called directly, they are called
through convenience functions declared in Include/abstract.h:
int PyObject_AquireFixedReadBuffer(PyObject *obj,
const void **buffer,
size_t *buffer_len);
int PyObject_AcquireFixedWriteBuffer(PyObject *obj,
void **buffer,
size_t *buffer_len);
void PyObject_ReleaseFixedBuffer(PyObject *obj);
The former two functions return 0 on success, set buffer to the
memory location and buffer_len to the length of the memory block
in bytes. On failure, or if the fixed buffer interface is not
implemented by obj, they return -1 and set an exception.
The latter function doesn't return anything, and cannot fail.
Backward Compatibility
The size of the PyBufferProcs structure changes if this proposal
is implemented, but the type's tp_flags slot can be used to
determine if the additional fields are present.
Reference Implementation
Will be uploaded to the SourceForge patch manager by the author.
Additional Notes/Comments
Python strings, unicode strings, mmap objects, and array objects
would expose the fixed buffer interface.
mmap and array objects would actually enter a locked state while
the buffer is active, this is not needed for strings and unicode
objects. Resizing locked array objects is not allowed and will
raise an exception. Whether closing a locked mmap object is an
error or will only be deferred until the lock count reaches zero
is an implementation detail.
Community Feedback
Greg Ewing doubts the fixed buffer interface is needed at all, he
thinks the normal buffer interface could be used if the pointer is
(re)fetched each time it's used. This seems to be dangerous,
because even innocent looking calls to the Python API like
Py_DECREF() may trigger execution of arbitrary Python code.
The first version of this proposal didn't have the release
function, but it turned out that this would have been too
restrictive: mmap and array objects wouldn't have been able to
implement it, because mmap objects can be closed anytime if not
locked, and array objects could resize or reallocate the buffer.
Credits
Scott Gilbert came up with the name 'fixed buffer interface'.
References
[1] The buffer interface
http://mail.python.org/pipermail/python-dev/2000-October/009974.html
[2] The Buffer Problem
http://www.python.org/peps/pep-0296.html
Copyright
This document has been placed in the public domain.
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End: