[Python-Dev] PEP 298 - the Fixed Buffer Interface

[Thomas Heller]

Here is PEP 298 - the Fixed Buffer Interface, posted to
get feedback from the Python community.
Enjoy!

Thomas

PS: I'll going to a 2 weeks vacation at the end of this week,
so don't hold your breath on replies from me if you post
after, let's say, thursday.

-----
PEP: 298
Title: The Fixed Buffer Interface
Version: $Revision: 1.3 $
Last-Modified: $Date: 2002/07/30 16:52:53 $
Author: Thomas Heller <theller@python.net>
Status: Draft
Type: Standards Track
Created: 26-Jul-2002
Python-Version: 2.3
Post-History:


Abstract

    This PEP proposes an extension to the buffer interface called the
    'fixed buffer interface'.

    The fixed buffer interface fixes the flaws of the 'old' buffer
    interface as defined in Python versions up to and including 2.2,
    see [1]:

        The lifetime of the retrieved pointer is clearly defined.

        The buffer size is returned as a 'size_t' data type, which
        allows access to large buffers on platforms where sizeof(int)
        != sizeof(void *).


Specification

    The fixed buffer interface exposes new functions which return the
    size and the pointer to the internal memory block of any python
    object which chooses to implement this interface.

    The size and pointer returned must be valid as long as the object
    is alive (has a positive reference count).  So, only objects which
    never reallocate or resize the memory block are allowed to
    implement this interface.

    The fixed buffer interface omits the memory segment model which is
    present in the old buffer interface - only a single memory block
    can be exposed.


Implementation

    Define a new flag in Include/object.h:

        /* PyBufferProcs contains bf_getfixedreadbuffer
           and bf_getfixedwritebuffer */
        #define Py_TPFLAGS_HAVE_GETFIXEDBUFFER (1L<<15)


    This flag would be included in Py_TPFLAGS_DEFAULT:

        #define Py_TPFLAGS_DEFAULT  ( \
                             ....
                             Py_TPFLAGS_HAVE_GETFIXEDBUFFER | \
                             ....
                            0)


    Extend the PyBufferProcs structure by new fields in
    Include/object.h:

        typedef size_t (*getfixedreadbufferproc)(PyObject *, void **);
        typedef size_t (*getfixedwritebufferproc)(PyObject *, void **);

        typedef struct {
                getreadbufferproc bf_getreadbuffer;
                getwritebufferproc bf_getwritebuffer;
                getsegcountproc bf_getsegcount;
                getcharbufferproc bf_getcharbuffer;
                /* fixed buffer interface functions */
                getfixedreadbufferproc bf_getfixedreadbufferproc;
                getfixedwritebufferproc bf_getfixedwritebufferproc;
        } PyBufferProcs;


    The new fields are present if the Py_TPFLAGS_HAVE_GETFIXEDBUFFER
    flag is set in the object's type.

    The Py_TPFLAGS_HAVE_GETFIXEDBUFFER flag implies the
    Py_TPFLAGS_HAVE_GETCHARBUFFER flag.

    The getfixedreadbufferproc and getfixedwritebufferproc functions
    return the size in bytes of the memory block on success, and fill
    in the passed void * pointer on success.  If these functions fail
    - either because an error occurs or no memory block is exposed -
    they must set the void * pointer to NULL and raise an exception.
    The return value is undefined in these cases and should not be
    used.

    Usually the getfixedwritebufferproc and getfixedreadbufferproc
    functions aren't called directly, they are called through
    convenience functions declared in Include/abstract.h:

        int PyObject_AsFixedReadBuffer(PyObject *obj,
                                      void **buffer,
                                      size_t *buffer_len);

        int PyObject_AsFixedWriteBuffer(PyObject *obj,
                                       void **buffer,
                                       size_t *buffer_len);

    These functions return 0 on success, set buffer to the memory
    location and buffer_len to the length of the memory block in
    bytes. On failure, or if the fixed buffer interface is not
    implemented by obj, they return -1 and set an exception.


Backward Compatibility

    The size of the PyBufferProcs structure changes if this proposal
    is implemented, but the type's tp_flags slot can be used to
    determine if the additional fields are present.


Reference Implementation

    Will be uploaded to the SourceForge patch manager by the author.


Additional Notes/Comments

    Python strings, Unicode strings, mmap objects, and maybe other
    types would expose the fixed buffer interface, but the array type
    would *not*, because its memory block may be reallocated during
    its lifetime.


Community Feedback

    Greg Ewing doubts the fixed buffer interface is needed at all, he
    thinks the normal buffer interface could be used if the pointer is
    (re)fetched each time it's used.  This seems to be dangerous,
    because even innocent looking calls to the Python API like
    Py_DECREF() may trigger execution of arbitrary Python code.

    Neil Hodgson wants to expose pointers to memory blocks with
    limited lifetime: do some kind of lock operation on the object,
    retrieve the pointer, use it, and unlock the object again.  While
    the author sees the need for this, it cannot be addressed by this
    proposal.  Beeing required to call a function after not using the
    pointer received by the getfixedbufferprocs any more seems too
    error prone.


Credits

    Scott Gilbert came up with the name 'fixed buffer interface'.


References

    [1] The buffer interface
        http://mail.python.org/pipermail/python-dev/2000-October/009974.html

    [2] The Buffer Problem
        http://www.python.org/peps/pep-0296.html


Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End: