[Python-Dev] PEP-298: buffer interface widening too much?

[Todd Miller]

Martin v. Löwis wrote:

>>I've been following the PEP-298 discussion a little, because I fear 
>>eventually IAGNI.   But when I look at the proposed changes,  I can't 
>>help but get the feeling that the "too wide" buffer interface just got 
>>wider.  By "too wide",  I am referring to the extraneous segmentation 
>>and character buffer aspects of the existing interface.  Adding three 
>>more calls for the locking problem bugs me.  Doesn't anything ever get 
>>simpler?
>>    
>>
>
>The locked buffer interface is simpler than the original buffer interface.
>It does not support segmentation.
>  
>
I understand that,  it just seemed to me that the complexity of the 
PEP-298 solution is rising out of proportion with the problem: no locking.  

>  
>
>>It seems to me that the existing (3!) calls for 
>>acquiring the buffer pointer can do just fine.   Except...
>>
>>In the case of existing extensions,  there will certainly be no call to 
>>the release function.
>>    
>>
>
>This is a minor problem. The major problem is that the existing protocol
>does not guarantee locking. So if you find an object that supports the buffer
>interface, you still don't know how long you can use the pointer 
>you've received. It might become invalid after calling nearly any Python
>API function.
>
I'm suggesting that we should change the protocol to guarantee locking. 
  The consequences of doing that appear acceptable to me, and I was 
hoping to stir up some discussion on why they're *not*, or to simplify 
the PEP enough that the problem gets fixed rather than ignored.

>
>The PEP authors believe that this can't be retroactively fixed 
>by declaring the existing interfaces to be locked, and I tend to agree:
>changing the docs doesn't change the code "out there".
>  
>
I was trying to look at this from the perspective of what *really* 
breaks if we change the protocol to include locking implicitly.   If the 
consequences of "obsolescent extension code not calling the buffer 
release" are unnecessary exceptions,  that might be better than the 
current situation with no locking at all, where the consequences are a 
segfault or worse.   I'm also arguing that anywhere there are 
"consequences",  the current application is already in jeopardy  because 
it must be (a) using the buffer protocol and (b) attempting to modify 
the buffer address while the buffer is in use.   So,  either the 
application already has serious issues, or changing the protocol and the 
subsequent "dangling locks" won't hurt it at all.  Here I am assuming 
that locking does not affect refcount,  nor block deallocation.  (I'm 
also assuming that an object refcount cannot drop to zero during an 
extension function call w/o a bug somewhere else.  Help me out if I'm 
wrong here.)

>  
>
>>But is it really necessary to mess with the reference 
>>counts?   I'm having a hard time imagining a situation in which the 
>>reference count for a buffer goes to zero while it is locked which is 
>>not a bug somewhere else.  What am I missing?  
>>    
>>
>
>Not sure what you are referring to here. If it is the PEP: it doesn't
>tell whether reference counts should be increased. If you are referring
>to my messages: I'm not saying that the reference counts should be increased.
>I'm saying that the PEP does not specify what the interaction of locking
>and reference counting is.
>  
>
I was referring to the recent python-dev messages between yourself and 
Scott Gilbert.

>  
>
>>So,  in the end,  what I'm wondering is whether there's a simpler middle 
>>ground between dropping the PEP and implementing it as it stands now.   
>>    
>>
>
>There certainly is. The advantage of the PEP process is that there always
>is a specific proposal to discuss. We only have this proposal at the moment,
>so we can't discuss anything else. If you want to make a different proposal,
>please propose specific wording (either as a new PEP, or as a modification
>of the existing one). I can't picture precisely what it is that you are
>proposing, nor what its consequences would be.
>
Point taken.  What I'm really proposing is this:

1.  Add a buffer lock release call to the bufferprocs struct and an 
indicator flag bit.   Thomas's call is fine already.  The indicator bit 
could also remain the same,  but the new meaining would simply be that a 
"release" routine is supported.  The locking buffer pointer access calls 
should be dropped.

2.  Advise objects with support the buffer API to modify their *current* 
buffer pointer fetch routines to implement locking if they need it. 
 They must also fill in the release function pointer and set the bit to 
indicate they have it.  They need to modify their "mutator methods" to 
check the lock.

3. Add a buffer release wrapper function the C-API to hide the nasty 
details of checking for and calling the release function.  If there is 
none,  the release wrapper, is a no-op.   Thomas's existing release 
wrapper is fine.  The locking buffer pointer access calls should be dropped.

4. Advise extensions to treat all objects as implementing only a locking 
interface.  Get the locked pointer,   and release it later.   If the 
object chooses to ignore support for locking,  its the object's problem. 
 But there should be only *one* way to write an extension.  

5. Advise objects supporting the buffer API *not* to mess with refcounts 
during buffer protocol get/release.  This prevents probable (certain) 
memory leaks at the cost of improbable segfaults from obsolescent 
extensions.  Any extension that this is a problem for should be happy to 
have a solution and be willing to update to support locking.   I see no 
*new* risk here.

6. Advise objects which are asked to change a locked buffer to raise an 
exception rather than blocking.

7. Advise objects which are asked to deallocate while locked to 
deallocate normally.

>
>Regards,
>Martin
>  
>
Todd