[Python-Dev] Use of Cython

[Ronald Oussoren]

> On 5 Aug 2018, at 03:15, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 5 August 2018 at 00:46, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Antoine Pitrou schrieb am 04.08.2018 um 15:57:
>>> Actually, I think testing the C API is precisely the kind of area where
>>> you don't want to involve a third-party, especially not a moving target
>>> (Cython is actively maintained and generated code will vary after each
>>> new Cython release).  Besides, Cython itself calls the C API, which
>>> means you might end up involuntarily testing the C API against itself.
>>> 
>>> If anything, testing the C API using ctypes or cffi would probably be
>>> more reasonable... assuming we get ctypes / cffi to compile everywhere,
>>> which currently isn't the case.
>> 
>> I agree that you would rather not want to let Cython (or another tool)
>> generate the specific code that tests a specific C-API call, but you could
>> still use Cython to get around writing the setup, validation and unittest
>> boilerplate code in C. Basically, a test could then look something like
>> this (probably works, although I didn't test it):
>> 
>>    from cpython.object cimport PyObject
>>    from cpython.list cimport PyList_Append
>> 
>>    def test_PyList_Append_on_empty_list():
>>        # setup code
>>        l = []
>>        assert len(l) == 0
>>        value = "abc"
>>        pyobj_value = <PyObject*> value
>>        refcount_before = pyobj_value.ob_refcnt
>> 
>>        # conservative test call, translates to the expected C code,
>>        # although with exception propagation if it returns -1:
>>        errcode = PyList_Append(l, value)
>> 
>>        # validation
>>        assert errcode == 0
>>        assert len(l) == 1
>>        assert l[0] is value
>>        assert pyobj_value.ob_refcnt == refcount_before + 1
>> 
>> 
>> If you don't want the exception handling, you can define your own
>> declaration of PyList_Append() that does not have it. But personally, I'd
>> rather use try-except in my test code than manually taking care of cleaning
>> up (unexpected) exceptions.
> 
> Exactly, that's the kind of thing I had in mind. At the moment,
> writing a new dedicated C API test requires designing 4 things:
> 
> 1. The test case itself (what action to take, which assertions to make about it)
> 2. The C code to make the API call you want to test
> 3. The Python->C interface for the test case from 1 to pass test
> values in to the code from 2
> 4. The C->Python interface to get state of interest from 2 back to the
> test case from 1
> 
> If we were able to use Cython to handle 3 & 4 rather than having to
> hand craft it for every test, then I believe it would significantly
> lower the barrier to testing the C API directly rather than only
> testing it indirectly through the CPython implementation.
> 
> Having such a test suite available would then hopefully make it easier
> for other implementations to provide robust emulations of the public C
> API.
> 
> ctypes & cffi likely wouldn't help as much in the case, since they
> don't eliminate the need to come up with custom code for parts 3 & 4,
> they just let you write that logic in Python rather than C.

I’m not sure if I understand this, ctypes and cffi are used to access C APIs without writing C code including the CPython API (see for example <https://github.com/abarnert/superhackyinternals/blob/master/internals.py <https://github.com/abarnert/superhackyinternals/blob/master/internals.py>>). 

The code code below should be mostly equivalent to the Cython example posted earlier:

import unittest
import ctypes
from ctypes import pythonapi

class PyObject(ctypes.Structure):
    _fields_ = (
        ('ob_refcnt', ctypes.c_ssize_t),
    )

pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]

def refcount(v):
    return PyObject.from_address(id(v)).ob_refcnt


def test_PyList_Append_on_empty_list():
       # setup code
       l = []
       assert len(l) == 0
       value = "abc"


       refcount_before = refcount(value)

       errcode = pythonapi.PyList_Append(l, value)

       assert errcode == 0
       assert len(l) == 1
       assert l[0] is value
       assert refcount(value) == refcount_before + 1

I write “mostly” because I rarely use ctypes and am not 100% sure that I use the API correctly.

A problem with using ctypes is that this tests the ABI and to the API, which for example means you cannot test C macros this way.

Ronald

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180805/344db74a/attachment.html>