[Python-Dev] PEP: Adding data-type objects to Python

[Alexander Belopolsky]

Paul Moore <p.f.moore <at> gmail.com> writes:

> Somewhat. My understanding is that the python-level buffer object is
> frowned upon as not good practice, and is scheduled for removal at
> some point (Py3K, quite possibly?) Hence, any code that uses buffer()
> feels like it "needs" to be replaced by something "more acceptable".

Python 2.x buffer object serves two distinct purposes.  First, it is a
"mutable string" object and this is definitely not going away being
replaced by the bytes object. (Interestingly, this functionality is not
exposed to python, but C extension modules can call
PyBuffer_New(size) to create a buffer.)  Second, it is a "view" into any
object supporting buffer protocol.  For a while this usage was indeed
frowned upon because buffer objects held the pointer obtained from
bf_get*buffer for too long causing memory errors in situations like
this:

>>> a = array('c', "x"*10)
>>> b = buffer(a, 5, 2)
>>> a.extend('x'*1000)
>>> str(b)
'xx'

This problem was fixed more than two years ago. 

------
r35400 | nascheme | 2004-03-10 

Make buffer objects based on mutable objects (like array) safe.
------

Even though it was suggested in the past that buffer *object*
should be deprecated as unsafe, I don't remember seeing a call
to deprecate the buffer protocol.   


> So although I understand the use you suggest, it's not compelling to
> me because I am left with the feeling that I wish I knew "the way to
> do it that didn't need the buffer object" (even though I realise
> intellectually that such a way may not exist).
> 

As I explained in another post,  I used buffer object as an example of
an object that supports buffer protocol, but does not export type
information in the form usable by numpy.

Here is another way to illustrate the problem:

>>> a = numpy.array(array.array('H', [1,2,3]))
>>> b = numpy.array([1,2,3],dtype='H')
>>> a.dtype == b.dtype
False

With the extended buffer protocol it will be possible for numpy.array(..)
to realize that array.array('H', [1,2,3]) is a sequence of unsigned short
integers and convert it accordingly.  Currently numpy has to go through
the sequence protocol to create a numpy.array from an array.array and
loose the type information.