[Python-Dev] pre-PEP: The Safe Buffer Interface

[Scott Gilbert]

--- Greg Ewing <greg@cosc.canterbury.ac.nz> wrote:
> Scott Gilbert <xscottg@yahoo.com>:
> 
> > We haven't seen a semi-thorough use case where the locking behavior is
> > beneficial yet. ... If there is no realizable benefit to the
> > acquire/release semantics of the new interface, then this is just extra
> > burden too.
> 
> The proposer of the original safe-buffer interface claimed to have a
> use case where the existing buffer interface is not safe enough,
> involving asynchronous I/O. I've been basing my comments on the
> assumption that he does actually have a need for it.
> 

I believe Thomas Heller's needs were met without making locking part of the
interface, but that he was willing to bend to please you and Neil.  His
original proposal did not include any notion of locking.  Nor does his
current since Guido has taken a stand on this issue.


> 
> So I think it's worth putting in some thought and getting it as
> right as we can from the beginning.
> 

Absolutely.  I just wanted to make sure that there is at least one sensible
use case before adding the complexity.  Moreover, if the sensible use cases
for locking are few and far between, then I'm still inclined to leave it
out since you can add the locking semantics at a different level.

It looks like Neil has sufficiently defined an example where it's useful. 
His use case is a bit complicated though, and I think he could get every
bit of that functionality by putting the locking in a smarter object
tailored for his application, and working with temporary "snapshot" objects
with an explicit release() method.  

What if Neil decides he needs Reader/Writer locks?  This is completely
justifiable too, since multiple threads can read an object without
interfering, but only one should be writing it.  We shouldn't arbitrarily
add complexity for the exceptional cases.


>
> > I'm concerned that this is very much like the segment count features
> > of the current PyBufferProcs.  It was apparently designed for more
> > generality, and while no one uses it, everyone has to check that the
> > segment count is one or raise an exception.
> 
> It's not as bad as that! My version of the proposal would impose *no*
> burden on implementations that did not require locking, for the
> following reasons:
>

Your use of the word *no* is different than mine.  :-)  I could similarly
claim that the segment count puts no burden on implementations that don't
need it.


> 
> 1) Locking is an optional task performed by the getxxxbuffer
> routines. Objects which do not require locking just don't
> do it.
> 
> 2) For objects not requiring locking, the releasebuffer
> operation is a no-op. Such an object can simply not
> implement this routine, and the type machinery can fill
> it in with a stub.
> 

I believe it will be a no-op in enough places that extension writers will
do it wrong without even knowing.


>
> > The extension releases the GIL so that another
> > thread can work on the array object.
> 
> Hey, whoa right there! If you have two threads accessing this array
> object simulaneously, you should be using a mutex or semaphore or
> something to coordinate them. As I pointed out before, thread
> synchronisation is outside the scope of my proposal.
> 

This is exactly Neil's use case.  He's got two threads reading it
simultaneously.  One thread (not really a thread, but the asynchronous I/O
operation) is writing to disk, and the other thread is keeping the user
interface updated.  There is no problem until the user tries to enter text
(which forces a resize) before the asynchronous I/O is complete.  Neil has
a solution for this, but I think it's less than typical.


>
> The only purpose of the locking, in my proposal, is to ensure that an
> exception occurs instead of a crash if the programmer screws up and
> tries to resize an object whose internals are being messed with. It's
> up to the programmer to do whatever is necessary to ensure that he
> doesn't do that.
> 
> > If extend() is called while thread 1 has the array locked, it can:
> > 
> >    A) raise an exception or return an error
> 
> Yes. (Raise an exception.)
> 

Which exception?  Would you introduce a standard exception that should be
raised when the user tries to do an operation that currently isn't allowed
because the buffer is locked?



Truthfully, now that Neil has given his explanation, I'm beginning to bend
on this a bit.  You're right in that it's not that much burden (however,
it's more than *no* burden :-), and someone might find it useful.  I still
think it's going to be pretty uncommon, and I still believe the locking can
be added on top of the simpler interface as needed.


Cheers,
    -Scott







__________________________________________________
Do You Yahoo!?
Yahoo! Health - Feel better, live better
http://health.yahoo.com