[Python-Dev] buffer interface considered harmful

[David Ascher]

On Mon, 16 Aug 1999, Jim Fulton wrote:

[ Aside:

  > It seems to me that NumPy is sufficiently broad enogh to encompass
  > image processing.

  Well, I'll just say that you could have been right, but w/ the current
  NumPy, I don't blame F/ for having developed his own data structures.  
  NumPy is messy, and some of its design decisions are wrong for image
  things (memory handling, casting rules, etc.).  It's all water under the
  bridge at this point.
]

Back to the main topic:

You say:

> [Multi-segment buffers] seem to strip away an awful lot of information
> hiding.

My impression of the buffer notion was that it is intended to *provide*
information hiding, by giving a simple API to byte arrays which could be
stored in various ways.  I do agree that whether those bytes should be
shared or not is a decision which should be weighted carefully.

> My main concern is having two systems rely on some low-level "shared
> memory" mechanism to achiev effiecient communication.

I don't particularly care about the specific buffer interface (the
low-level nature of which is what I think you object to). I do care about
having a well-defined mechanism for sharing memory between objects, and I
think there is value in defining such an interface generically.  Maybe the
notion of segmented arrays of bytes is too low-level, and instead we
should think of the data spaces as segmented arrays of chunks, where a
chunk can be one or more bytes?  Or do you object to any 'generic'
interface?

Just for fun, here's the list of things which either currently do or have
been talked about possibly in the future supporting some sort of buffer
interface, and my guesses as to chunk size, segmented status and
writeability):

  - strings  (1 byte, single-segment, r/o)
  - unicode strings (2 bytes, single-segment, r/o)
  - struct.pack() things (1 byte, single-segment,r/o)
  - arrays (1-4? bytes, single-segment, r/w)
  - NumPy arrays (1-8 bytes, multi-segment, r/w)
  - PIL images (1-? bytes, multi-segment, r/w)
  - CObjects (1-byte, single-segment, r/?)
  - mmapfiles (1-byte, multi-segment?, r/w)
  - non-python-owned memory (1-byte, single-segment, r/w)

--david