[Python-Dev] PEP 3118: Extended buffer protocol (new version)

[Carl Banks]

Travis Oliphant wrote:
> Carl Banks wrote:
>> Ok, I've thought quite a bit about this, and I have an idea that I 
>> think will be ok with you, and I'll be able to drop my main 
>> objection.  It's not a big change, either.  The key is to explicitly 
>> say whether the flag allows or requires.  But I made a few other 
>> changes as well.
> I'm good with using an identifier to differentiate between an "allowed" 
> flag and a "require" flag.   I'm not a big fan of 
> VERY_LONG_IDENTIFIER_NAMES though.  Just enough to understand what it 
> means but not so much that it takes forever to type and uses up 
> horizontal real-estate.

That's fine with me.  I'm not very particular about spellings, as long
as they're not misleading.

>> Now, here is a key point: for these functions to work (indeed, for 
>> PyObject_GetBuffer to work at all), you need enough information in 
>> bufinfo to figure it out.  The bufferinfo struct should be 
>> self-contained; you should not need to know what flags were passed to 
>> PyObject_GetBuffer in order to know exactly what data you're looking at.
> Naturally.
> 
>> Therefore, format must always be supplied by getbuffer.  You cannot 
>> tell if an array is contiguous without the format string.  (But see 
>> below.)
> 
> No, I don't think this is quite true.   You don't need to know what 
> "kind" of data you are looking at if you don't get strides.  If you use 
> the SIMPLE interface, then both consumer and exporter know the object is 
> looking at "bytes" which always has an itemsize of 1.

But doesn't this violate the above maxim?  Suppose these are the
contents of bufinfo:

ndim = 1
len = 20
shape = (10,)
strides = (2,)
format = NULL

How does it know whether it's looking at contiguous array of 10 two-byte
objects, or a discontiguous array of 10 one-byte objects, without having
at least an item size?  Since item size is now in the mix, it's moot, of
course.

The idea that Py_BUF_SIMPLE implies bytes is news to me.  What if you
want a contiguous, one-dimensional array of an arbitrary type?  I was
thinking this would be acceptable with Py_BUF_SIMPLE.  It seems you want
to require Py_BUF_FORMAT for that, which would suggest to me that
Py_BUF_ALLOW_ND amd Py_BUF_ALLOW_NONCONTIGUOUS, etc., would imply
Py_BUF_FORMAT?  IOW, pretty much anything that's not SIMPLE implies FORMAT?

If that's the case, then most of the issues I brought up about item size
don't apply.  Also, if that's the case, you're right that Py_BUF_FORMAT
makes more sense than Py_BUF_DONT_NEED_FORAMT.

But it now it seems even more unnecessary than it did before.  Wouldn't
any consumer that just wants to look at a chunk of bytes always use
Py_BUF_FORMAT, especially if there's danger of a presumptuous exporter
raising an exception?


> I'll update the PEP with my adaptation of your suggestions in a little 
> while.

Ok.  Thanks for taking the lead, and for putting up with my verbiose
nitpicking. :)


Carl Banks