[Python-3000] PEP Draft: Enhancing the buffer protcol

[Travis E. Oliphant]

Daniel Stutzbach wrote:
> I know I'm joining this discussion late in the game, so I apologize if
> my look through the list archives was not sufficiently exhaustive and
> this has been proposed and shot down before...

No, I don't think you are late.  But this discussion has been going on 
off and on for at least 10 years :-)  We don't all remember all the 
issues, though.

> 
> What if the locking mechanism were put into the array's memory instead
> of the container's memory?  

Basically, my first proposal was to have a single view object and you 
would get at the memory through it.  But, having a light-weight API that 
returns a pointer to memory like the current one does is desirable.

> If the array-memory is a PyObject, then
> the existing reference counting mechanism can be used, instead of
> inventing a new one.  We can introduce a new type, PyArray, that is
> pretty much opaque (bare minimum of methods).  A PyArray is just a
> PyObject_HEAD (one type pointer plus the reference counter) followed
> by the data that would normally be there.
> 

The original object still needs to distinguish between normal references 
and "view-based references."  Thus, even with your proposal it seems you 
will need another counter on the objects that wish to track 
buffer-interface views.


> When an array-like container allocates memory, it allocates a PyArray
> to store the actual data in.  When a caller request a view, the
> container increments the PyArray's reference counter and returns a
> pointer to the PyArray.  The caller is responsible for decrementing
> the reference counter when it is done with the view, so
> bf_releasebuffer becomes unnecessary.

Maybe I'm not understanding you correctly.  Perhaps what you are saying 
is that we should have all memory allocation go through a light-weight 
memory-object.  Then, you would get this object + an offset when you 
wanted a pointer into memory.

This way, the memory would never be deallocated until nothing was 
referencing it.  I think this approach would work.   However, you could 
still have the case, where an object reallocated memory while another 
object which thought it had a view of that object ended up with a 
"out-dated" view.   You just wouldn't segfault in that case.

You could check the reference count on the memory object, before 
reallocating, I suppose. But I've heard that the reference counts on 
Python objects can be larger than 1 in some cases (even though there 
isn't really anything "viewing" the memory).


> 
> The container cannot reallocate the memory unless the reference
> counter on the PyArray is exactly 1.

I'm not sure we can guarantee this would work.  It seems like for 
various reasons depending on the state of the interpreter, reference 
counts increase.

> 
> Basically, I'm wondering if it makes sense to move the new reference
> counter into the buffered memory rather than putting it in the
> container, so that there is only one reference counter implementation.
> 

This is an idea I've thought of too, but we would be enforcing a 
"use-python for all shared memory allocations" restriction.


> Different question: what is a container supposed to do if a view is
> locking its memory and it needs to reallocate to complete some
> operation?  I assume it would raise an exception, but it would be nice
> to spell this out in the PEP.

It would raise an exception.

-Travis