[Python-Dev] marshal (was:Buffer interface in abstract.c? )

[Fredrik Lundh]

Guido van Rossum wrote:
> > btw, how about adding support for buffer access
> > to data that have strange internal formats (like cer-
> > tain PIL image memories) or isn't directly accessible
> > (like "virtual" and "abstract" image buffers in PIL 1.1).
> > something like:
> > 
> > int initbuffer(PyObject* obj, void** context);
> > int exitbuffer(PyObject* obj, void* context);
> > 
> > and corresponding context arguments to the
> > rest of the functions...
> 
> Can you explain this idea more?  Without more understanding of PIL I
> have no idea what you're talking about...

in code:

    void* context;

    // this can be done at any time
    segments = pb->getsegcount(obj, NULL, context);

    if (!pb->bf_initbuffer(obj, &context))
        ... failed to initialise buffer api ...
    
    ... allocate segment size buffer ...

    pb->getsegcount(obj, &bytes, context);
    ... calculate total buffer size and allocate buffer ...

    for (i = offset = 0; i < segments; i++) {
        n = pb->getreadbuffer(obj, i, &p, context);
        if (n < 0)
            ... failed to fetch a given segment ...
        memcpy(buf + offset, p, n); // or write to file, or whatevef
        offset = offset + n;
   }

   pb->bf_exitbuffer(obj, context);

in other words, this would given the target object a
chance to keep some local context (like a temporary
buffer) during a sequence of buffer operations...

for PIL, this would make it possible to

1) store required metadata (size, mode, palette)
along with the actual buffer contents.

2) possibly pack formats that use extra internal
storage for performance reasons -- RGB pixels
are stored as 32-bit integers, for example.

3) access virtual image memories (that can only
be accessed via a buffer-like interface in them-
selves -- given an image object, you acquire an
access handle, and use a getdata method to
access the actual data.  without initbuffer,
there's no way to do two buffer access in
parallel.  without exitbuffer, there's no way
to release the access handle.  without the
context variable, there's nowhere to keep
the access handle between calls.)

4) access abstract image memories (like virtual
memories, but they reside outside PIL, like on
a remote server, or inside another image pro-
cessing library, or on a hardware device).

5) convert to external formats on the fly:

    fp.write(im.buffer("JPEG"))

and probably a lot more.  as far as I can tell,
nothing of this can be done using the current
design...

...

besides, what about buffers and threads?  if you
return a pointer from getreadbuf, wouldn't it be
good to know exactly when Python doesn't need
that pointer any more?  explicit initbuffer/exitbuffer
calls around each sequence of buffer operations
would make that a lot safer...

</F>