[Python-Dev] PEP-298: buffer interface widening too much?

[Todd Miller]

Martin v. Löwis wrote:

>>I've used this aspect of PyArg_ParseTuple,  but forgot about it.    I 
>>was thinking that you're right,  the original object is "gone", we're 
>>stuck.   But...  it's still (in all probability) lying around in the arg 
>>tuple,  so some sort of PyArg_ReleaseBuffers is possible.   
>>    
>>
>
>That looks clumsy, 
>
Well, yes, especially in light of PEP-286.  :)

>and also too easy to forget unless done systematically,
>
I really liked the idea of doing the buffer release behind the scenes, 
 as suggested by PEP-286 and below.

>in which case it will add a considerable performance cost. I think the
>
But won't *any* systematic solution have a performance cost?  Argtuples 
don't sound "free".  I'm really playing devil's advocate here,  because 
I like the concept of PEP-286.

>locked buffer interface is much simpler in comparison.
>
I disagree here.   I didn't see PEP-298 address PyArg_ParseTuple at all, 
 so there's no real basis for comparison on this point.    I'll grant 
you that PyArg_ReleaseBuffers is probably a lame solution to a problem 
better solved by PEP-286;  that said,  because use of 
PyArg_ReleaseBuffers is explicit,  there is no problem distributing the 
performance penalty to places where there is a corresponding benefit.  I 
was not suggesting dropping the existing release call;   merely adding a 
layer which could make sense out of PyArg_ParseTuple specs.

>
>If Jack ever manages to design an alternative parse tuple interface, I
>assume it will arrange to automatically release the buffer when the function
>returns (or, more precisely, when the buffer is deallocated.
>
>In fact, my PEP 286 could address this issue: the locked buffers would be
>associated with the enhanced tuple, so that releasing the tuple would unlock
>the buffers. 
>
That sounds pretty cool.

>This *still* has the potential for breaking code, since now
>the buffers are locked while the function runs (i.e. callbacks couldn't 
>expand the array anymore); it is likely that such breakage would uncover
>a bug in the locking assumptions of the existing code.
>  
>
I agree.  But the means exist to work around such problem (release and 
re-aquire),  and I would guess they are even rarer than the existing 
"buffer safety" issues.

>Your attempt to fix your proposal indicates one of the strengths of the PEP
>process: most initial proposals have flaws, and, unless completely spelled
>out, readers will typically miss the flaws on initial reading. The weakness,
>of course, is that a PEP might be sitting around for a long time, and nobody
>would be reviewing it.
>
>  
>
>>Incidentally,  does PEP-298 solve this now?
>>    
>>
>
>Depends on what "this" is. Existing code will continue to work as before:
>No locking in ParseTuple happens, as a result, the arguments are unlocked
>when the function returns; this is desirable. Since the buffers aren't 
>locked, relocation may happen. If such relocation is ever observed, people
>would have to replace the "s" parser with an "O" parser, and perform the
>locking themselves.
>
I'm just saying that we don't need 2 extra "locking calls" to achieve 
this.  

If the default behavior for the existing Python objects (array, mmap, 
...) is a onetime warning when a lock is abused,  where's the issue? 
 And with the warning framework,  the one time warning could be mutated 
into an exception or supressed by a variety of means.

My thinking is that *all* buffer pointers should be "logically" locked. 
  Objects that don't need locks don't need to change.   All extensions 
should be written and/or updated to handle objects which require 
locking.  Extensions which aren't updated will trigger warnings which 
could be supressed or converted into exceptions.

For me, that's simpler than saying "correct extensions" lock the buffer, 
 but "obsolete ones" may still segfault w/o warning.

Perhaps saying that "a warning is enough" is too limiting,  but this is 
a fringe issue,  so I think maybe it is.

>
>So, yes, PEP-298 addresses this - although modification of existing code is
>necessary to make use of this feature (as is in your fixed proposal).
>
>  
>
>>Well,  this does sound bad,  but I was assuming the core and library 
>>would be "fixed" at the first introduction of locking, and the only 
>>problem would be with non-conforming extensions.    I now think that 
>>"lock violations" might be better handled using the warning framework, 
>>defaulting to a one time warning.  
>>    
>>
>
>I think we should strive for a solution that keeps the function parameters
>locked while the function runs, and unlocks them automatically at function 
>return time.
>
I like this idea,  but wonder if it won't inflict a performance penalty 
on all function calls.

>
>This shows another flaw in the wording of PEP 298: It implies without saying
>that the expectation is that you should have as many release calls as you
>have lock calls. 
>
I was certainly assuming this was the case.

>
>Regards,
>Martin
>
>  
>
Regards,
Todd