[Numpy-discussion] PyBUF_SIMPLE requests

[Stefan Krah]

Hello,

I'm working on the completion of the PEP-3118 (buffer protocol) implementation
in Python core.


I was wondering how a PyBUF_SIMPLE buffer request sent to an object with
a complex structure should behave.  The current Python C-API documentation
for a PyBUF_SIMPLE buffer request says:

"This is the default flag. The returned buffer exposes a read-only memory
 area. The format of data is assumed to be raw unsigned bytes, without any
 particular structure. This is a “stand-alone” flag constant. It never needs
 to be ‘|’d to the others. The exporter will raise an error if it cannot
 provide such a contiguous buffer of bytes."


To me a memory view "without any particular structure" means that the
returned buffer should always expose the whole memory area.


But NumPy behaves differently (see below for the printbuffer() hack):

Python 3.3.0a0 (default:1dd6908df8f5, Jul 16 2011, 11:16:00)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> x = ndarray(buffer=bytearray([1,2,3,4,5,6,7,8,9,10]), shape=[10], strides=[-1], dtype="B", offset=9)
>>> x
array([10,  9,  8,  7,  6,  5,  4,  3,  2,  1], dtype='uint8')
>>> x.printbuffer()
PyBUF_FULL:

obj: 0x7f527e946240
buf: 0x7f5281268c79
buf[0]: 10
len: 10
itemsize: 1
readonly: 0
ndim: 1
format: B
shape: 10
strides: -1

PyBUF_SIMPLE:

obj: 0x7f527e946240
buf: 0x7f5281268c79
buf[0]: 10
len: 10
itemsize: 1
readonly: 0
ndim: 0
format: (null)
shape:
strides:


The PyBUF_FULL request returns a reasonable buffer, because consumers that
follow the strides will behave correctly.


I do not understand the PyBUF_SIMPLE result. According to the C-API docs
a consumer would be allowed to access buf[9], which would be invalid.

Or is a consumer supposed to observe ndim = 0 and treat the result
as a scalar? If so, there is a problem with this approach. For example,
consumers like the struct module do not care about ndim and would walk
right out of the array.


Stefan Krah



printbuffer() hack:

============================================================================
diff --git a/numpy/core/src/multiarray/methods.c b/numpy/core/src/multiarray/methods.c
index 68f697a..1cd260a 100644
--- a/numpy/core/src/multiarray/methods.c
+++ b/numpy/core/src/multiarray/methods.c
@@ -2234,6 +2234,51 @@ array_newbyteorder(PyArrayObject *self, PyObject *args)

 }

+static int
+do_print(PyObject *self, int flags, const char *flagstring)
+{
+    Py_buffer view;
+    int i;
+
+    if (PyObject_GetBuffer(self, &view, flags) < 0)
+        return -1;
+
+    fprintf(stderr, "%s:\n\n", flagstring);
+    fprintf(stderr, "obj: %p\n", view.obj);
+    fprintf(stderr, "buf: %p\n", view.buf);
+    fprintf(stderr, "buf[0]: %d\n", ((char *)(view.buf))[0]);
+    fprintf(stderr, "len: %zd\n", view.len);
+    fprintf(stderr, "itemsize: %zd\n", view.itemsize);
+    fprintf(stderr, "readonly: %d\n", view. readonly);
+    fprintf(stderr, "ndim: %d\n", view.ndim);
+    fprintf(stderr, "format: %s\n", view.format);
+
+    fprintf(stderr, "shape: ");
+    for (i = 0; i < view.ndim; i++)
+        fprintf(stderr, "%zd ", view.shape[i]);
+    fprintf(stderr, "\n");
+
+    fprintf(stderr, "strides: ");
+    for (i = 0; i < view.ndim; i++)
+        fprintf(stderr, "%zd ", view.strides[i]);
+    fprintf(stderr, "\n\n");
+
+    PyBuffer_Release(&view);
+
+    return 0;
+}
+
+static PyObject *
+array_print_getbuf(PyObject *self, PyObject *dummy)
+{
+    if (do_print(self, PyBUF_FULL, "PyBUF_FULL") < 0)
+        return NULL;
+    if (do_print(self, PyBUF_SIMPLE, "PyBUF_SIMPLE") < 0)
+        return NULL;
+
+    Py_RETURN_NONE;
+}
+
 NPY_NO_EXPORT PyMethodDef array_methods[] = {

     /* for subtypes */
@@ -2426,6 +2471,9 @@ NPY_NO_EXPORT PyMethodDef array_methods[] = {
     {"view",
         (PyCFunction)array_view,
         METH_VARARGS | METH_KEYWORDS, NULL},
+    {"printbuffer",
+        (PyCFunction)array_print_getbuf,
+        METH_NOARGS, NULL},
     {NULL, NULL, 0, NULL}           /* sentinel */
 };