[Python-Dev] buffer interface considered harmful

Greg Stein gstein@lyra.org
Mon, 16 Aug 1999 02:56:31 -0700


Jack Jansen wrote:
>...
> I've just look at it, and I'm not sure I understand the full intentions of the
> buffer object. Buffer objects can either behave as the "buffer-aspect" of the
> object behind them (without the rest of their functionality) or as array
> objects, and if they start out life as the first they can evolve into the
> second, is that right?
> 
> Is there a rationale behind this design, or is it just something that
> happened?

The object doesn't change. You create it as a reference to an existing
object's buffer (as exported via the buffer interface), or you create it
as a reference to some arbitrary memory.

The buffer object provides (optionally read/write) string-like behavior
to any object that supports buffer behavior. It can also be used to make
lightweight slices of another object. For example:

>>> a = "abcdefghi"
>>> b = buffer(a, 3, 3)
>>> print b
def
>>>

In the above example, there is only one copy of "def" (the portion
inside of the string object referenced by <a>).

The string-like behavior can be quite nice for memory-mapped files.
Andrew's mmapfile module's file objects export the buffer interface.
This means that you can open a file, wrap a buffer around it, and
perform quick and easy random-access on the thing. You could even select
slices of the file and pass them around as if they were strings, without
loading anything into the process heap. (I want to try mmap'ing a .pyc
and create code objects that have buffer-based bytecode streams; it will
be interesting to see if this significantly reduces memory consumption
(in terms of the heap size; the mmap'd .pyc can be shared across
processes)).

Cheers,
-g

--
Greg Stein, http://www.lyra.org/