[Numpy-discussion] [Fwd: python-dev Summary for 2006-10-16 through 2006-10-31]

[Colin J. Williams]

Numpy readers will likely be interested in this excerpt.

Colin W.

-------- Original Message --------
Subject: python-dev Summary for 2006-10-16 through 2006-10-31
Date: Mon, 27 Nov 2006 04:47:23 +0000 (GMT)
From: steven.bethard at gmail.com
Reply-To: python-list at python.org
Newsgroups: gmane.comp.python.announce

python-dev Summary for 2006-10-16 through 2006-10-31
++++++++++++++++++++++++++++++++++++++++++++++++++++

.. Excerpt::

[The HTML version of this Summary is available at
http://www.python.org/dev/summary/2006-10-16_2006-10-31]

=========
Summaries
=========

---------------------------------------------------------------
The buffer protocol and communicating binary format information
---------------------------------------------------------------

Travis E. Oliphant presented a pre-PEP for adding a standard way to 
describe the shape and intended types of binary-formatted data. It was 
accompanied by a pre-PEP for extending the buffer protocol to handle 
such shapes and types. Under the proposal, a new ``datatype`` object 
would describe binary-formatted data with an API like::

     datatype((float, (3,2))
     # describes a 3*2*8=48 byte block of memory that should be interpreted
     # as 6 doubles laid out as arr[0,0], arr[0,1], ... a[2,0], a[1,2]

     datatype([( ([1,2],'coords'), 'f4', (3,6)), ('address', 'S30')])
     # describes the structure
     #     float coords[3*6]   /* Has [1,2] associated with this field */
     #     char  address[30]

Alexander Belopolsky provided a nice example of why you might want to 
extend the buffer protocol along these lines. Currently, there's not 
much you can do with a basic buffer object. If you want to pass it to 
numpy_, you have to provide the type and shape information yourself::

     >>> b = buffer(array('d', [1,2,3]))
     >>> numpy.ndarray(shape=(3,), dtype=float, buffer=b)
     array([ 1.,  2.,  3.])

By extending the buffer protocol appropriately so that the necessary 
information can be provided, you should be able to pass the buffer 
directly to numpy_ and have it understand the format itself::

     >>> numpy.array(b)

People were uncomfortable with the many ``datatype`` variants -- the 
constructor accepted types, strings, lists or dicts, each of which could 
specify the structure in a different way. Also, a number of people 
questioned why the existing ``ctypes`` mechanisms for describing binary 
data couldn't be used instead, particularly since ``ctypes`` could 
already describe things like function pointers and recursive types, 
which the pre-PEP could not. Travis said he was looking for a way to 
unify the data formats of all the ``array``, ``struct``, ``numpy`` and 
``ctypes`` modules, and felt like using the ``ctypes`` approach was too 
verbose for use in the other modules. In particular, he felt like the 
``ctypes`` use of type objects as binary-format specifiers was 
problematic because type objects were harder to manipulate at the C level.

The discussion continued on into the next fortnight.

.. _numpy:


Contributing threads:

- `PEP: Adding data-type objects to Python 
<http://mail.python.org/pipermail/python-dev/2006-October/069602.html>`__
- `PEP: Extending the buffer protocol to share array information. 
<http://mail.python.org/pipermail/python-dev/2006-October/069681.html>`__