[Python-Dev] An updated extended buffer PEP

[Carl Banks]

Travis Oliphant wrote:
> Carl Banks wrote:
>> Travis E. Oliphant wrote:
>>> I think we are getting closer.   What do you think about Greg's idea 
>>> of basically making the provider the bufferinfo structure and having 
>>> the exporter handle copying memory over for shape and strides if it 
>>> wants to be able to change those before the lock is released.
>> It seems like it's just a different way to return the data.  You could 
>> do it by setting values through pointers, or do it by returning a 
>> structure.  Which way you choose is a minor detail in my opinion.  I'd 
>> probably favor returning the information in a structure.
>>
>> I would consider adding two fields to the structure:
>>
>> size_t structsize; /* size of the structure */
> Why is this necessary?  can't you get that by sizeof(bufferinfo)?

In case you want to add something later.  Though if you did that, it 
would be a different major release, meaning you'd have to rebuild 
anyway.  They rashly add fields to the PyTypeObject in the same way. :) 
  So never mind.


>> PyObject* releaser; /* the object you need to call releasebuffer on */ 
> Is this so that another object could be used to manage releases if desired?

Yes, that was a use case I saw for a different "view" object.  I don't 
think it's crucially important to have it, but for exporting objects 
that delegate management of the buffer to another object, then it would 
be very helpful if the exporter could tell consumers that the other 
object is managing the buffer.

Suppose A is an exporting object, but it uses a hidden object R to 
manage the buffer memory.  Thus you have A referring to R, like this:

A -> R

Now object B takes a view of A.  If we don't have this field, then B 
will have to hold a reference to A, like this:

B -> A -> R

A would be responsible for keeping track of views, and A could not be 
garbage collected until B disappears.  If we do have this field, then A 
could tell be B to hold a reference to R instead:

B -> R
A -> R

A is no longer obliged to keep track of views, and it can be garbage 
collected even if B still exists.


Here's a concrete example of where it would be useful: consider a 
ByteBufferSlice object.  Basically, the object represents a 
shared-memory slice of a 1-D array of bytes (for example, Python 3000 
bytes object, or an mmap object).

Now, if the ByteBufferSlice object could tell the consumer that someone 
else is managing the buffer, then it wouldn't have to keep track of 
views, thus simplifying things.

P.S. In thinking about this, it occurred to me that there should be a 
way to lock the buffer without requesting details.  ByteBufferSlice 
would already know the details of the buffer, but it would need to 
increment the original buffer's lock count.  Thus, I propose new fuction:

typedef int (*lockbufferproc)(PyObject* self);


Carl Banks