We use Cython a lot to wrap and test C and C++ libraries at work and is true that makes some aspects more simple and easy to express. Also you can leverage integration with other testing tools (pytest, mocks...etc) at to some extent. But I humbly think that Cython is not the best choice for this particular project. One of the reasons is that Cython makes use of the C-API itself, so we can have unwanted and surprising interactions between the usage of the C-APi from Cython and the tested code (I know that this is unlikely, but possible). Another reason is that Cython will auto generate code that has proven to be much more difficult to debug than handcrafted code when the issue is in the translation layer. (Think for example when you pass a string to a function that needs a char* and that function takes ownership of that buffer. In this case when the temporary goes away the memory held by the function is invalid. This is wrong, obviously, but for the person unaware of the issue can be very confusing and somewhat difficult to debug).

I think that Cython should build on top of the C-API and therefore the C-API itself should not depend on Cython even for testing purposes.

On Mon, 3 Sep 2018 at 09:40, Victor Stinner <vstinner@redhat.com> wrote:

Le sam. 1 sept. 2018 à 08:20, Stefan Behnel <python_capi@behnel.de> a écrit :

...

Totally agree with Jeroen. Cython makes it trivial to integrate with the current CPython testing infrastructure (as I mentioned somewhere else, we're even using doctests for Cython itself), while giving you any freedom you need for fine grained C-level testing.

I don't know. For me, Cython adds a big layer on top of the C API, and if something bad happens, I would be able to know if the issue comes from the C API, Cython or something else. I like unit tests written in C, because you can debug them directly using gdb: the code that you see in gdb is the code that you debug. Moreover, I'm not sure that Cython does not do anything "magic" behind my back.

...

Here's a simple PyMalloc test that might be relevant for the case at hand:

https://github.com/cython/cython/blob/master/tests/run/cpython_capi.pyx

The advantage of these tests is the try/finally.

Check is designed for the good case: when the test pass. I don't see how to release resources when a test fails. But since Check does fork by default, side effects of a failing test should be limited. Moreover, the side effects only concern tests which fail, and I'm optimistic: tests should pass in the normal case :-)

Apart of that, it doesn't seem *to me* that tests written with Check are more verbose:

https://github.com/pythoncapi/cpython/blob/pythoncapi/capi_tests/test_tuple....

...

And some doctest intensive generator tests:

https://github.com/cython/cython/blob/master/tests/run/generators.pyx

Hum, IMHO doctests are too high level for the C API. If possible, I would prefer to not execute Python code in C tests. But it's likely that I will need some Python somewhere ;-)

...

I find doctests a really nice way to write parametrised C tests.

Using Check, parametrised tests are a litlte bit verbose, I agree:

static void check_PyTuple_New(size_t size) { /* ... */ }

START_TEST(test_PyTuple_New) { check_PyTuple_New(0); check_PyTuple_New(1); check_PyTuple_New(5);

PyObject *tuple = PyTuple_New(-1);
ck_assert_ptr_null(tuple);

} END_TEST

...

...

Of course you may have other reasons to not want Cython...

Cython adds another dependency, I would prefer to limit dependencies.

Well, I don't want to take decisions alone. What do other people think about writing tests using Cython instead of Check?

Victor

---

capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org

On Mon., 3 Sep. 2018, 9:44 pm Victor Stinner, <vstinner@redhat.com> wrote:

Le lun. 3 sept. 2018 à 13:09, Pablo Galindo Salgado <pablogsal@gmail.com> a écrit :

...

We use Cython a lot to wrap and test C and C++ libraries at work and is true that makes some aspects more simple and easy to express. Also you can leverage integration with other testing tools (pytest, mocks...etc) at to some extent. But I humbly think that Cython is not the best choice for this particular project. (...)

One compromise would be to use unit tests in C for the most basic tests, but use Cython for "integration" tests" and more complex tests.

One way to implement such a split would be to write the Cython based tests as a separate project that tested across multiple CPython branches (and potentially even across multiple Python implementations).

The Check based tests could then focus on unit testing of individual API calls, while the Cython ones would check for unintended interactions and explore edge cases (perhaps with the aid of Hypothesis).

Cheers, Nick.