[Python-Dev] An updated extended buffer PEP

[Carl Banks]

Travis Oliphant wrote:
> Travis Oliphant wrote:
>> Hi Carl and Greg,
>>
>> Here is my updated PEP which incorporates several parts of the 
>> discussions we have been having.
> 
> And here is the actual link:
> 
> http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt 


What's the purpose of void** segments in PyObject_GetBuffer?  It seems 
like it's leftover from an older incarnation?

I'd hope after more recent discussion, we'll end up simplifying 
releasebuffer.  It seems like it'd be a nightmare to keep track of what 
you've released.


Finally, the isptr thing.  It's just not sufficient.  Frankly, I'm 
having doubts whether it's a good idea to support multibuffer at all. 
Sure, it brings generality, but I'm thinking its too hard to explain and 
too hard to get one's head around, and will lead to lots of 
misunderstanding and bugginess.  OTOH, it really doen't burden anyone 
except those who want to export multi-buffered arrays, and we only have 
one shot to do it.  I just hope it doesn't confuse everyone so much that 
no one bothers.

Here's how I would update the isptr thing.  I've changed "derefoff" to 
"subbufferoffsets" to describe it better.


typedef PyObject *(*getbufferproc)(PyObject *obj, void **buf,
                                    Py_ssize_t *len, int *writeable,
                                    char **format, int *ndims,
                                    Py_ssize_t **shape,
                                    Py_ssize_t **strides,
                                    Py_ssize_t **subbufferoffsets);


subbufferoffsets

   Used to export information about multibuffer arrays.  It is an
   address of a ``Py_ssize_t *`` variable that will be set to point at
   an array of ``Py_ssize_t`` of length ``*ndims``.

   [I don't even want to try a verbal description.]

   To demonstrate how subbufferoffsets works, here is am example of a
   function that returns a pointer to an element of ANY N-dimensional
   array, single- or multi-buffered.

    void* get_item_pointer(int ndim, void* buf, Py_ssize_t* strides,
                         Py_ssize_t* subarrayoffs, Py_ssize_t *indices) {
         char* pointer = (char*)buf;
         int i;
         for (i = 0; i < ndim; i++) {
             pointer += strides[i]*indices[i];
             if (subarraysoffs[i] >= 0) {
                 pointer = *(char**)pointer + subarraysoffs[i];
             }
         }
         return (void*)pointer;
     }

   For single buffers, subbufferoffsets is negative in every dimension
   and it reduces to normal single-buffer indexing.  For multi-buffers,
   subbufferoffsets indicates when to dereference the pointer and switch
   to the new buffer, and gives the offset into the buffer to start at.
   In most cases, the subbufferoffset would be zero (indicating it should
   start at the beginning of the new buffer), but can be a positive
   number if the following dimension has been sliced, and thus the 0th
   entry in that dimension would not be at the beginning of the new
   buffer.



Other than that, looks good. :)


Carl Banks