[Python-Dev] Assertion in _PyManagedBuffer_FromObject()

[Stefan Behnel]

Stefan Krah, 02.03.2012 12:53:
> Stefan Behnel wrote:
>>     if (PyObject_GetBuffer(base, &mbuf->master, PyBUF_FULL_RO) < 0) {
>>         /* mbuf->master.obj must be NULL. */
>>         Py_DECREF(mbuf);
>>         return NULL;
>>     }
>>
>>     /* Assume that master.obj is a new reference to base. */
>>     assert(mbuf->master.obj == base);
> 
> 
>> I'm not saying that this is likely to happen, but I could imagine code that
>> wants to use a different object for the cleanup than itself, possibly for
>> keeping a certain kind of state when it delivers more than one buffer, or
>> for remembering what kind of allocation was used, or ...
> 
> I /think/ a different cleanup object would be possible, but memoryview now
> has the m.obj attribute that let's you see easily which object the view
> actually references. That attribute would then point to the cleanup handler.
> 
> Note that the complexity is such that I would have to go through the whole
> code again to be *sure* that it's possible.
> 
> So I'd rather see that people just don't use such schemes (unless there
> is a storm of protest).
> 
> The assumption is clearly documented in:
> 
> http://docs.python.org/dev/c-api/buffer.html#Py_buffer
> http://docs.python.org/dev/c-api/typeobj.html#buffer-object-structures
> 
> Since the Py_buffer.obj filed was undocumented in 3.2, I think we're within
> out rights to restrict the field to the exporter.

Careful. There are tons of code out there that use the buffer interface,
and the "obj" field has been the way to handle the buffer release ever
since the interface actually worked (somewhere around the release of Py3.0,
IIRC).

Personally, I never read the documentation above (which was written way
after the design and implementation of the buffer interface). I initially
looked at the (outdated) PEP, and then switched to reading the code once it
started to divert substantially from the PEP. I'm sure there are many users
out there who have never seen the second link above, and still some who
aren't aware that the "exporting object" in the first link is required to
be identical with the one that "__getbuffer__()" was called on. Just think
of an object that acts as a façade to different buffers.

I'm well aware of the complexity of the implementation. However, even if
the assert was (appropriately, as Nick noted) replaced by an exception,
it's still not all that unlikely that it breaks user code (assuming that it
currently works). The decision to enforce this restriction should not be
taken lightly.

Stefan