[Python-Dev] How to fix the buffer object's broken char buffer support

[Brett Cannon]

On 6/6/06, Guido van Rossum <guido at python.org> wrote:
> On 6/6/06, Brett Cannon <brett at python.org> wrote:
> > If you run ``import array; int(buffer(array.array('c')))`` the
> > interpreter will segfault.  While investigating this I discovered that
> > buffer objects, for their tp_as_buffer->bf_getcharbuffer, return the
> > result by calling the wrapped object bf_getreadbuffer or
> > bf_getwritebuffer.  This is wrong since it is essentially redirecting
> > the expected call to the wrong tp_as_buffer slot for the wrapped
> > object.  Plus it doesn't have Py_TPFLAGS_HAVE_GETCHARBUFFER defined.
> >
> > I see two options here.  One is to remove the bf_getcharbuffer slot
> > from the buffer object.  The other option is to fix it so that it only
> > returns bf_getcharbuffer and doesn't redirect improperly (this also
> > brings up the issue if Py_TPFLAGS_HAVE_GETCHARBUFFER should then also
> > be defined for buffer objects).
> >
> > Since I don't use buffer objects I don't know if it is better to fix
> > this or just rip it out.
>
> How ironic. the charbuffer slot was added late in the game -- now we'd
> be ripping it out...
>
> I suspect that there's a reason for it; but in Py3k it will
> *definitely* be ripped out. Buffers will purely deal in byte then,
> never in characters; you won't be able to get a buffer from a
> (unicode) string at all.
>
> Unhelpfully,

I actually figured out a reasonable way to integrate it into the
buffer object so that it won't be a huge issue.  Just took a while to
make sure there was not a ton of copy-and-paste and deciphering the
docs (have a separate patch going for clarifying those).

So buffer objects will properly support charbuffer in 2.5 (won't
backport since it is a semantic change).  Hopefully it won't break too
much stuff.  =)

-Brett