[Python-Dev] pre-PEP: The Safe Buffer Interface

[Greg Ewing]

Scott Gilbert <xscottg@yahoo.com>:

> We haven't seen a semi-thorough use case where the locking behavior is
> beneficial yet. ... If there is no realizable benefit to the
> acquire/release semantics of the new interface, then this is just extra
> burden too.

The proposer of the original safe-buffer interface claimed to have a
use case where the existing buffer interface is not safe enough,
involving asynchronous I/O. I've been basing my comments on the
assumption that he does actually have a need for it.

The original proposal was restricted to non-resizable objects. I
suggested a small extension which would remove this restriction, at
what seems to me quite a small cost.

It may turn out that the restriction is easily lived with. On the
other hand, we might decide later that it's a nuisance. What worries
me is if we design a restricted safe-buffer interface now, and start
using it, and later decide that we want an unrestricted safe-buffer
interface, we'll then have two different safe-buffer interfaces
around, with lots of code that will only accept non-resizable objects
for no reason other than that it's using the old interface.

So I think it's worth putting in some thought and getting it as
right as we can from the beginning.

> I'm concerned that this is very much like the segment count features
> of the current PyBufferProcs.  It was apparently designed for more
> generality, and while no one uses it, everyone has to check that the
> segment count is one or raise an exception.

It's not as bad as that! My version of the proposal would impose *no*
burden on implementations that did not require locking, for the
following reasons:

1) Locking is an optional task performed by the getxxxbuffer
routines. Objects which do not require locking just don't
do it.

2) For objects not requiring locking, the releasebuffer
operation is a no-op. Such an object can simply not
implement this routine, and the type machinery can fill
it in with a stub.

It does place one extra burden on users of the interface, namely
calling the release routine. But I believe that this could even be
beneficial, in a way. The user is going to have to think about the
lifetime of the pointer, and be sure to keep a reference to the
underlying Python object as long as the pointer is needed.  Having to
keep it around so that you can call the release routine on it would
help to bring this into sharp focus.

> The extension releases the GIL so that another
> thread can work on the array object.

Hey, whoa right there! If you have two threads accessing this array
object simulaneously, you should be using a mutex or semaphore or
something to coordinate them. As I pointed out before, thread
synchronisation is outside the scope of my proposal.

The only purpose of the locking, in my proposal, is to ensure that an
exception occurs instead of a crash if the programmer screws up and
tries to resize an object whose internals are being messed with. It's
up to the programmer to do whatever is necessary to ensure that he
doesn't do that.

> If extend() is called while thread 1 has the array locked, it can:
> 
>    A) raise an exception or return an error

Yes. (Raise an exception.)

> Case A is troublesome because depending on thread scheduling/disk
> performance, you will or won't get the exception.

As I said before, you should be synchronising your threads
somehow *before* they operate on the object! If you don't,
you deserve whatever you get.

Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+