[Python-Dev] PEP-298: buffer interface widening too much?

[Todd Miller]

Martin v. Löwis wrote:

>I understand that. But please compare
>
>  if (!PyArg_ParseTuple(args,"sss",&string1, &string2, &string3))
>     return NULL;
>  if(!do_some_stuff(string1)){
>     PyArg_Release(args,"sss",string1,string2,string3);
>     return NULL;
>  }
>  if(!do_more_stuff(string2)){
>     PyArg_Release(args,"sss",string1,string2,string3);
>     return NULL;
>  }
>  compute_results(string3);
>
>  PyArg_Release(args,"sss",string1,string2,string3);
>  return result;
>
>to  
>
>if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3))
>     return NULL;
>  if(!do_some_stuff(string1)){
>     return NULL;
>  }
>  if(!do_more_stuff(string2)){
>     return NULL;
>  }
>  PyBuffer_Lock(obj3, &string3);
>  compute_results(string3);
>  PyBuffer_Unlock(obj3);
>
>  return result
>
>In general, you won't have to lock everything, so you can avoid
>a lot of clutter if you use just the PEP-298 interfaces, together
>with an "O" parser.
>  
>
Well,   my initial proposal would have read more like:

 PyObject *result = NULL;

  if (!PyArg_ParseTuple(args,"ssO",&string1, &string2, &obj3))
     goto fail_or_exit;
  if(!do_some_stuff(string1))
     goto fail_or_exit;
  if(!do_more_stuff(string2))
     goto fail_or_exit;

 result = compute_results(string3);

fail_or_exit:
  PyBuffer_Unlock(obj3);
  return result;

I'll concede that I'm not 100% comfortable with exit-gotos and that my 
implicit locking approach makes them necessary.

>>If the default behavior for the existing Python objects (array, mmap, 
>>...) is a onetime warning when a lock is abused,  where's the issue? 
>>    
>>
>
>It's unclear. I would expect that every application that uses arrays will
>trigger the warning. When you get the warning, you will have no clue what
>you did wrong, and most likely, the error is in the Python core. Who is
>
True.  Isolating the missing release call is hard.  Harder still will be 
isolating missing locks.

>going to fix the hundreds and hundreds of functions that use the "s" parser,
>and who is not going to be terribly bored by this, knowing that 99% of them
>will never see an array.
>  
>
I guess that's the beauty of PEP-286...   I now see that (or something 
like it) as the only possibility for really correcting the buffer problem. 

>  
>
>>My thinking is that *all* buffer pointers should be "logically" locked. 
>>    
>>
>
>That is the intent of the PEP, indeed. For backwards compatibility, it
>can't change the existing interface, but needs to create a new one. After
>some time, the core could require that objects support it, much more later,
>it could remove support for the old interface.
>  
>
I think that kind of backwards compatability just paves the wave for 
extensions to stay broken and complicates the object structures as well. 
  I guess PEP-298 does create the possibility for creating correct 
extensions,  I just hate the thought of 5 getpointer calls where there 
are now 3 and should be 2.

>>  Objects that don't need locks don't need to change.   All extensions
>>should be written and/or updated to handle objects which require 
>>locking.  Extensions which aren't updated will trigger warnings which 
>>could be supressed or converted into exceptions.
>>    
>>
>
>The problem is that you are talking about virtually *all* extensions.
>I bet you won't find a single extension that doesn't use an "s" parser.
>
Most of our (numarray) extensions don't use "s" parsers,  but I'll 
accept that it's not feasible to update all extensions.  

>>For me, that's simpler than saying "correct extensions" lock the buffer, 
>> but "obsolete ones" may still segfault w/o warning.
>>    
>>
>
>But, with your warning, this would still happen?
>  
>
Yes.  But it's a (minor) improvement because you have an immediate clue 
why the segfault occurred.

>  
>
>>I like this idea,  but wonder if it won't inflict a performance penalty 
>>on all function calls.
>>    
>>
>
>I hope it won't. There is no implementation of enhanced argument tuples, yet
>(and well may not appear for Python 2.3); but if this is done, a 
>PyArg_ParseTuple call that doesn't allocate any resources will leave a NULL
>pointer in the slot for the resources - on deallocation, all you pay is the
>null pointer test.
>  
>
Sounds good to me.  I hope you implement it!  If PEP-286 were here now, 
 a simplified PEP-298 would be possible.  Without PEP-286,  I doubt the 
buffer locking problem will ever be adequately fixed and life is going 
to get more complicated.  Fortunately for me,  it's all still 
theoretical...  

>Regards,
>Martin
>
>  
>
Regards,
Todd