[Python-Dev] PEP 3118: Extended buffer protocol (new version)

[Jim Jewett]

Reading this message without the entire PEP in front of me showed some
confusing usage.  (Details below)  Most (but not all) I could resolve
from the PEP itself, but they could be clarified with different
constant names.

Counter Proposal at bottom, and specific questions in between.

Travis Oliphant wrote:
> Carl Banks wrote:

>> Some of the flags RESTRICT the kind of buffers that can be
>> exported (Py_BUF_WRITABLE); other flags EXPAND the
>> kind of buffers that can be exported (Py_BUF_INDIRECT).
>> That is highly confusing and I'm -1
>> on any proposal that includes both behaviors.

> Basically, every flag corresponds to a different property of
> the buffer  that the consumer is requesting:

I had trouble seeing it like that.  Could you at least rename them
something like SHAPE_VALID?  But I would prefer the change suggested
at the bottom.

> Py_BUF_SIMPLE  --- you are requesting the simplest possible  (0x00)

??? Does this automatically assume a readable buffer?  (No hardware
write buffers?)  (This one I couldn't tell from the PEP)

> Py_BUF_WRITEABLE --  get a writeable buffer   (0x01)
> Py_BUF_READONLY --  get a read-only buffer    (0x02)

Do Py_BUF_READONLY promise that *this* consumer won't try to change
it, or request that it be immutable (and no one else will change it
either)?  (From the PEP itself, I think you wanted IMMUTABLE.)

When it comes back, does it mean "you, the consumer, promised not to
change this", or "I, the exporter, won't authorize anyone to change
it, including myself."

If it is really a request for immutability, and the exporter can't
make that promise, then should the buffer protocol itself make and
return an immutable copy?

For this grouping to make sense, I have to assume that you really mean

    Py_BUF_READABLE  (0 - no way to mark it write-only)
    Py_BUF_WRITABLE (1 - meaning *this* consumer can write)
    Py_BUF_IMMUTABLE (2 - meaning no one can change it.)
                    so 3 => compile-time error

but that still doesn't mesh with your later statement that:

> ... most people request read-or-write  buffers
> i.e. Py_BUF_SIMPLE.


> Py_BUF_FORMAT --  get a "formatted" buffer.   (0x04)

Is this saying "I the consumer will respect your formatting" or "I the
consumer will fail if you don't tell me the formatting"?

If this flag comes back, does that mean that understanding the
formatting is mandatory, or is it just informational?

??? To make this concrete, there are libraries that sniff and guess a
format.  Should they pass this flag or not?

> Py_BUF_SHAPE -- get a buffer with shape information  (0x08)

Wait -- what is the difference between these two again?  Is "format"
the internal format of a single element, and shape the dimensions of
an array?  Should Py_BUF_FORMAT be Py_BUF_ELT_FORMAT?

Do you pass Py_BUF_SHAPE to indicate that you'll accept N-Dim arrays,
or to say that you prefer them or somehow need them?

??? If Py_BUF_SHAPE was requested, but the buffer really is
1-dimensional, should this still be set on the way back?  (Presumably
setting the "shape" variable to point to a (one-element) array
containing len/itemsize?

> Py_BUF_STRIDES --  get a buffer with stride information (and shape)  (0x18)
> Py_BUF_OFFSET -- get a buffer with suboffsets (and strides and shape) (0x38)

I assume that Py_BUF_OFFSET doesn't make sense without Py_BUF_STRIDES,
and Py_BUF_STRIDES doesn't make sense without Py_BUF_SHAPE.

What happens if someone *does* pass 0x20?

If you want to avoid that, it might make sense to treat this as 4
enumerated values (1-D, n-D, n-D with strides, n-D with strides and
offsets) instead of 3 flags.

If Py_BUF_OFFSET was requested, but the return value is a single
continuous 1D array, should Py_BUF_OFFSET still come back but have the
zeros filled in?

Also, why are all negative offsets invalid?  It seems like they might
be useful for some re-orderings, and using an offset of 0 has the same
effect as marking the offset unused.

> For me, the most restrictive requests are

> PY_BUF_WRITEABLE | Py_BUF_FORMAT and
> Py_BUF_READONLY |  Py_BUF_FORMAT

> The most un-restrictive request (the largest circle in my mental Venn
> diagram) is

> Py_BUF_OFFSETS followed by Py_BUF_STRIDES followed by Py_BUF_SHAPE

I think this is what he meant when he said that the flags had opposite meanings.
Ideally, 0 and 0xFF should be shortcuts to either the most restrictive
or the least restrictive.

> Think of them as turning "on" members of the bufferinfo structure.

If the senses can't be reversed, could you at least rename them to
indicate this?  Something like Py_BUF_VALID_SUBOFFSETS?

================================

Counter Proposal:


0x00: Py_BUF_RWLOCK:  "Consumer is a new owner.  It can read and
write; no other code can." (most restrictive)

0x01: Py_BUF_READONLY:  "Consumer doesn't need to (cannot) write."
0x02: Py_BUF_MUTABLE:  "Other code can write"

(implies)
0x03 = PY_BUF_UR:
    Py_BUF_READONLY | Py_BUF_MUTABLE : "Unconfirmed Read: Consumer
won't write, but other code might."


(skipping 0x04 just to get the shapes all in the second digit)

0x08: Py_BUF_ELT_FORMAT:  "If you send the element format, I'll look at it."
	

??? What does this mean in a response?  Just that the *format isn't NULL?


# Note that these are four shapes alternatives,
# rather than two orthogonal flags

0x0n: Py_BUF_1D:  "No shape information."  (in response, means none
required, -- should the shape pointer still be set?)

0x1n: Py_BUF_SHAPE:  "continous n-D arrays acceptable (to consumer) or
used (by exporter)"

0x2n: Py_BUF_STRIDE: " ? How to best describe strides?  Srides
acceptable (to consumer) or used (by exporter)"

0x3n: Py_BUF_OFFSET: " ? How to best describe offsets?  offests
acceptable (to consumer) or used (by exporter)"


-jJ