[Python-Dev] Use of Cython

[Nick Coghlan]

On 5 August 2018 at 00:46, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Antoine Pitrou schrieb am 04.08.2018 um 15:57:
>> Actually, I think testing the C API is precisely the kind of area where
>> you don't want to involve a third-party, especially not a moving target
>> (Cython is actively maintained and generated code will vary after each
>> new Cython release).  Besides, Cython itself calls the C API, which
>> means you might end up involuntarily testing the C API against itself.
>>
>> If anything, testing the C API using ctypes or cffi would probably be
>> more reasonable... assuming we get ctypes / cffi to compile everywhere,
>> which currently isn't the case.
>
> I agree that you would rather not want to let Cython (or another tool)
> generate the specific code that tests a specific C-API call, but you could
> still use Cython to get around writing the setup, validation and unittest
> boilerplate code in C. Basically, a test could then look something like
> this (probably works, although I didn't test it):
>
>     from cpython.object cimport PyObject
>     from cpython.list cimport PyList_Append
>
>     def test_PyList_Append_on_empty_list():
>         # setup code
>         l = []
>         assert len(l) == 0
>         value = "abc"
>         pyobj_value = <PyObject*> value
>         refcount_before = pyobj_value.ob_refcnt
>
>         # conservative test call, translates to the expected C code,
>         # although with exception propagation if it returns -1:
>         errcode = PyList_Append(l, value)
>
>         # validation
>         assert errcode == 0
>         assert len(l) == 1
>         assert l[0] is value
>         assert pyobj_value.ob_refcnt == refcount_before + 1
>
>
> If you don't want the exception handling, you can define your own
> declaration of PyList_Append() that does not have it. But personally, I'd
> rather use try-except in my test code than manually taking care of cleaning
> up (unexpected) exceptions.

Exactly, that's the kind of thing I had in mind. At the moment,
writing a new dedicated C API test requires designing 4 things:

1. The test case itself (what action to take, which assertions to make about it)
2. The C code to make the API call you want to test
3. The Python->C interface for the test case from 1 to pass test
values in to the code from 2
4. The C->Python interface to get state of interest from 2 back to the
test case from 1

If we were able to use Cython to handle 3 & 4 rather than having to
hand craft it for every test, then I believe it would significantly
lower the barrier to testing the C API directly rather than only
testing it indirectly through the CPython implementation.

Having such a test suite available would then hopefully make it easier
for other implementations to provide robust emulations of the public C
API.

ctypes & cffi likely wouldn't help as much in the case, since they
don't eliminate the need to come up with custom code for parts 3 & 4,
they just let you write that logic in Python rather than C.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia