[Python-Dev] buffer interface considered harmful

[Greg Stein]

Fredrik Lundh wrote:
> 
> > I think the buffer interface was introduced in 1.5 (by Jack?). I added
> > the 8-bit character buffer slot and buffer objects in 1.5.2.
> >
> > > from array import array
> > >
> > > a = array("f", [0]*8192)
> > >
> > > b = buffer(a)
> > >
> > > for i in range(1000):
> > >     a.append(1234)
> > >
> > > print b
> > >
> > > in other words, the buffer interface should
> > > be redesigned, or removed.
> >
> > I don't understand what you believe is weird here.
> 
> did you run that code?

Yup. It printed nothing.

> it may work, it may bomb, or it may generate bogus
> output. all depending on your memory allocator, the
> phase of the moon, etc. just like back in the C/C++
> days...

It probably appeared as an empty string because the construction of the
array filled it with zeroes (at least the first byte).

Regardless, I'd be surprised if it crashed the interpreter. The print
command is supposed to do a str() on the object, which creates a
PyStringObject from the buffer contents. Shouldn't be a crash there.

> imo, that's not good enough for a core feature.

If it crashed, then sure. But I'd say that indicates a bug rather than a
design problem. Do you have a stack trace from a crash?

Ah. I just worked through, in my head, what is happening here. The
buffer object caches the pointer returned by the array object. The
append on the array does a realloc() somewhere, thereby invalidating the
pointer inside the buffer object.

Icky. Gotta think on this one... As an initial thought, it would seem
that the buffer would have to re-query the pointer for each operation.
There are performance implications there, of course, but that would
certainly fix the problem.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/