Currently C API is not completely covered by tests. Tests for particular parts of C API are scattered through different files. There are files completely purposed for testing C API (like test_capi.py, test_getargs2.py), there are classes (usually having "CAPI" in the name) in different files for testing C API specific for unicode, marshal. Argument parsing tests are split between two files, test_capi.py, test_getargs2.py. I need to add new tests for new features, and I'm going to add new tests for existing C API. But first I'm going to reorganize tests. Add a new directory Lib/test/test_capi, and move all C API tests into it, grouped by function prefixes. test_getargs.py for testing PyArg_*(), test_unicode.py for testing PyUnicode_*(), etc. Tests that use the _testcapi module, but don't test specific C API, will left on place. The benefit is that it will be easier to run all C API tests at once, and only them, and it will be clearer what C API is covered by tests. The disadvantage is that you will need to run several files for testing marshal for example. What are your thoughts?
On 29Jul2018 1253, Serhiy Storchaka wrote:
Can we make the regular tests import and also run the related C API tests? So that a normal run wouldn't normally include the entire C API test directory, but would include test classes in the related Python test modules? (Maybe there's a way to decorate the test classes for this?) I agree with the intent, but also think that's quite a disadvantage. It would be good to avoid it. Cheers, Steve
29.07.18 15:39, Steve Dower пише:
There are many ways of running tests: ./python -m test test_capi ./python -m test.test_capi ./python -m unittest test.test_capi ./python -m unittest discover Lib/test/test_capi/ They need different solutions for making them disabled by default. Seems that the simplest way is to move test_capi out of the test directory. But I think that in any case this will complicate testing code.
I agree with the intent, but also think that's quite a disadvantage. It would be good to avoid it.
Actually this disadvantage is not very large. There are not much C API tests for now. Testing unicode requires running not just test_unicode, but test_codecs, test_codeccallbacks, test_format, and yet few test. test_bytes itself is a mess, it needs significant rewriting. Marshal C API is outdated (it is based on FILE*), it is mostly unused in CPython. In any case Python tests should be enough for testing Python API.
On Sun, Jul 29, 2018, 06:44 Serhiy Storchaka, <storchaka@gmail.com> wrote:
For now, but if are successful, Serhiy, there will be a lot more tests. 😉 Testing unicode requires running not just test_unicode,
I think it depends on how you mentally group tests. Do you want to run all tests relating to marshal when you run test_marshal, or should you run test_marshal just for the Python code and test_capi.test_marshal for just the C API depending on what you changed? One perk of putting the C API tests in under test_capi is it makes it easier for other implementations to ignore those tests if they want to (although I'm assuming all the tests will also be marked as appropriate as CPython-specific).
I prefer the current organization that keeps the various tests together with the category being tested. I almost never need to run the C API tests all at once, but I do need to see all the tests for an object in one place. When maintaining something like marshal, it would be easy to miss some of the tests if they are in a separate file. IMO, the proposed change would hinder future maintenance and fly in the face of our traditional code organization. Raymond
30.07.18 09:46, Raymond Hettinger пише:
I prefer the current organization that keeps the various tests together with the category being tested. I almost never need to run the C API tests all at once, but I do need to see all the tests for an object in one place. When maintaining something like marshal, it would be easy to miss some of the tests if they are in a separate file. IMO, the proposed change would hinder future maintenance and fly in the face of our traditional code organization.
What about moving just test_capi.py, test_getargs2.py and test_structmembers.py into Lib/test/test_capi? They are not related to specific types or modules. For references: An issue on the tracker: https://bugs.python.org/issue34272. A PR that moves C API into the test_capi subdirectory: https://github.com/python/cpython/pull/8551. Currently it just moves test_capi.py, test_getargs2.py and test_structmembers.py. An initial version that moved also other C API tests: https://github.com/python/cpython/pull/8551/commits/eb16b9ee9eb36c9965c2d852....
Currently C API is not completely covered by tests. Tests for particular
Buildbots have a timeout of 15 min per test. I suggest to use multiple test_capi_<name>.py files rather than a directory which behaves as a single test. Or regrtest should be modified to implement timeout differently. Victor Le dimanche 29 juillet 2018, Serhiy Storchaka <storchaka@gmail.com> a écrit : parts of C API are scattered through different files. There are files completely purposed for testing C API (like test_capi.py, test_getargs2.py), there are classes (usually having "CAPI" in the name) in different files for testing C API specific for unicode, marshal. Argument parsing tests are split between two files, test_capi.py, test_getargs2.py.
I need to add new tests for new features, and I'm going to add new tests
for existing C API. But first I'm going to reorganize tests. Add a new directory Lib/test/test_capi, and move all C API tests into it, grouped by function prefixes. test_getargs.py for testing PyArg_*(), test_unicode.py for testing PyUnicode_*(), etc. Tests that use the _testcapi module, but don't test specific C API, will left on place.
The benefit is that it will be easier to run all C API tests at once, and
only them, and it will be clearer what C API is covered by tests. The disadvantage is that you will need to run several files for testing marshal for example.
https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
I need to add new tests for new features, and I'm going to add new tests
for existing C API. But first I'm going to reorganize tests. Add a new
Or maybe test_<name>_capi.py so you can more easily discover test_unicode_cami while working on Unicode. You can use -m "test_*_capi" to run all C API tests. Victor Le lundi 30 juillet 2018, Victor Stinner <vstinner@redhat.com> a écrit : parts of C API are scattered through different files. There are files completely purposed for testing C API (like test_capi.py, test_getargs2.py), there are classes (usually having "CAPI" in the name) in different files for testing C API specific for unicode, marshal. Argument parsing tests are split between two files, test_capi.py, test_getargs2.py. directory Lib/test/test_capi, and move all C API tests into it, grouped by function prefixes. test_getargs.py for testing PyArg_*(), test_unicode.py for testing PyUnicode_*(), etc. Tests that use the _testcapi module, but don't test specific C API, will left on place.
On 30 July 2018 at 21:23, Victor Stinner <vstinner@redhat.com> wrote:
Missing word there: -m test "test*_capi" I think between this approach and expanding test_capi to be a test package, not just a test file, I think it would be possible to make these parts of the test a fair bit more discoverable and maintainable. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Actually, I mean -m test --match 'test_*_capi' where --match can also be written -m. It may catch false positive since the filter is also applied to test case names and test method names. Maybe 'test_*_capi.*' is better, I didn't try. You may using use "ls Lib/test/test_*_capi.py > tests; ./python -m test --fromfile tests". There are different options. By the way, running the full test suite just takes 5 min on my laptop, it isn't so long ;-) Victor Le lundi 30 juillet 2018, Nick Coghlan <ncoghlan@gmail.com> a écrit :
participants (6)
-
Brett Cannon -
Nick Coghlan -
Raymond Hettinger -
Serhiy Storchaka -
Steve Dower -
Victor Stinner