[Python-3000] PEP Draft: Enhancing the buffer protcol
Travis E. Oliphant
oliphant.travis at ieee.org
Wed Feb 28 05:05:23 CET 2007
Daniel Stutzbach wrote:
> I know I'm joining this discussion late in the game, so I apologize if
> my look through the list archives was not sufficiently exhaustive and
> this has been proposed and shot down before...
No, I don't think you are late. But this discussion has been going on
off and on for at least 10 years :-) We don't all remember all the
> What if the locking mechanism were put into the array's memory instead
> of the container's memory?
Basically, my first proposal was to have a single view object and you
would get at the memory through it. But, having a light-weight API that
returns a pointer to memory like the current one does is desirable.
> If the array-memory is a PyObject, then
> the existing reference counting mechanism can be used, instead of
> inventing a new one. We can introduce a new type, PyArray, that is
> pretty much opaque (bare minimum of methods). A PyArray is just a
> PyObject_HEAD (one type pointer plus the reference counter) followed
> by the data that would normally be there.
The original object still needs to distinguish between normal references
and "view-based references." Thus, even with your proposal it seems you
will need another counter on the objects that wish to track
> When an array-like container allocates memory, it allocates a PyArray
> to store the actual data in. When a caller request a view, the
> container increments the PyArray's reference counter and returns a
> pointer to the PyArray. The caller is responsible for decrementing
> the reference counter when it is done with the view, so
> bf_releasebuffer becomes unnecessary.
Maybe I'm not understanding you correctly. Perhaps what you are saying
is that we should have all memory allocation go through a light-weight
memory-object. Then, you would get this object + an offset when you
wanted a pointer into memory.
This way, the memory would never be deallocated until nothing was
referencing it. I think this approach would work. However, you could
still have the case, where an object reallocated memory while another
object which thought it had a view of that object ended up with a
"out-dated" view. You just wouldn't segfault in that case.
You could check the reference count on the memory object, before
reallocating, I suppose. But I've heard that the reference counts on
Python objects can be larger than 1 in some cases (even though there
isn't really anything "viewing" the memory).
> The container cannot reallocate the memory unless the reference
> counter on the PyArray is exactly 1.
I'm not sure we can guarantee this would work. It seems like for
various reasons depending on the state of the interpreter, reference
> Basically, I'm wondering if it makes sense to move the new reference
> counter into the buffered memory rather than putting it in the
> container, so that there is only one reference counter implementation.
This is an idea I've thought of too, but we would be enforcing a
"use-python for all shared memory allocations" restriction.
> Different question: what is a container supposed to do if a view is
> locking its memory and it needs to reallocate to complete some
> operation? I assume it would raise an exception, but it would be nice
> to spell this out in the PEP.
It would raise an exception.
More information about the Python-3000