[Python-Dev] buffer interface (again)

[Thomas Heller]

[no need to cc me directly]
> > I would like to use (again) the buffer interface,
> > and I have some suggestions/questions.
> > 
> > - Only extension types can implement the buffer interface,
> > no way for a python class. Maybe a magic method __buffer__(self)
> > could be invented for this purpose?
> 
> How could it be made safe?  The buffer interface makes raw memory
> addresses available.  Letting a Python class decide on what addresses
> to use opens a big hole for abuse.
class C:
    ...
    def __buffer__(self):
        # self.my_buffer is an object exposing the buffer interface
        return buffer(self.my_buffer)
or:
        return self.my_buffer
> 
> > - Why does the builtin function buffer() always return
> > readonly buffers, even for read/write objects? Shouldn't
> > there also be a readwrite_buffer() function, or should
> > buffer() return read/write buffers for read/write objects?
> 
> It's a lifetime issue.  You can hold on to the object returned by
> buffer() long after the actual memory it points to is recycled for
> some other purpose, and while *reading* that memory is not such a big
> deal (assuming it is still part of your address space, you'll just get
> garbage), allowing it to be *written* is again an invitation to
> disaster.
Lifetimes are managed by refcounts, so it's a refcount issue,
at least as long as every object exposing the buffer interface
_guarantees_ that the memory address and size does not change.
(Which does not seem the case for array objects).

> 
> > - My bug report 216405 on sourceforge is still open
> > (well, it is marked as closed, but it went into PEP 42)
> > http://sourceforge.net/tracker/index.php?func=detail&aid=216405&group_id=5470&atid=105470
> 
> I still feel that your bug report is based on the wrong assumption
> about what the buffer API should do.
I only tried to fix the refcount issue.

> 
> Thomas, please first explain what you want to *do* with the buffer
> interface.  Some of the buffer API was a mistake.  It *appears* to be
> an interface for allocating and managing buffers, while in actuality
> it is only intended to provide access to buffered data that is managed
> by some C code.  You're probably better off using the array module to
> manage buffers.

I want to expose memory blocks to C, wrapped in python objects/classes.
These memory blocks could come from builtin python types: strings,
unicode strings, array objects, mmap'd files, ...

Thomas