[pypy-dev] Improving the documentation on how we test

[Carl Friedrich Bolz]

Hi Manuel,

we require and have always required app-level tests for every new
feature up to the finer details. The CPython test suite is often not
very thorough, and we often work under the assumption that if our own
tests about a feature work, the feature works.

The coding guide states that already, but is maybe not forceful or not
detailed enough:

"adding features requires adding appropriate tests."

I am open to suggestions how to make this more explicit.


Now, about integration vs. unit-tests: we indeed have been somewhat
flexible on this question. In my mind it comes down to how complicated
the logic is. For rather simple logic, an integration test only is
probably fine. For more intricate features it's probably better to also
add careful unit tests.

For your concrete case: There are intermediate approaches between just
adding an integration test to test_simple and super careful unit tests.
I am sure there are more user-visible effects of the PEP, e.g. shouldn't
there also be ast module tests necessary?

Cheers,

Carl Friedrich

On 10/12/17 18:14, Manuel Jacob wrote:
> Hi,
> 
> I'm currently implementing PEP 526 [1], which requires additions to the
> interpreter from the parser down to the bytecode interpreter.  Since
> this is a common task, I want to expand the documentation a bit, adding
> a "how to extend the interpreter" recipe.
> 
> One thing which I wasn't 100% sure about myself is how to test it. 
> Clearly every addition has to come with a test, preferably written
> before implementing anything.  But a few things are a bit unclear:
> 
> For every change in CPython there is (hopefully) a test case in
> lib-python/*/test.  So technically there's already a test case when
> modifying the interpreter.  One reason to add app-level tests is to
> avoid re-translations.  But other than this convenience, how much
> app-level testing to we *require*?  Should we test everything up to the
> last detail or are we fine with testing the basic functionality and
> leaving the rest to the CPython test suite?
> 
> A related question is which style of testing — integration-test-style or
> unit-test-style — we prefer.  Different people — also among the core
> developers — have different preferences, so we have a bit of a mix.  So
> in this concrete example, it is tempting to create a single test like
> AppTestFunctionAnnotations.test_simple from
> pypy/interpreter/test/test_syntax.py [2] and then add all the
> functionality to the interpreter until the test passes (this would be
> integration-style testing).  From the unit-style testing perspective,
> this test is not very good, because it doesn't test only the syntax,
> like the file name "test_syntax.py" would suggest.  The alternative
> would be to add a unit test for every affected part of the interpreter
> (parser, AST builder, AST validator, bytecode generation, symtable,
> stack effect computation, bytecode interpreter).
> 
> What do you think?  Feel free to braindump your general thoughts (also
> outside of the syntax addition case), so I can extend other relevant
> parts of the documentation as well.  This will hopefully make it easier
> for new (but also more advanced) contributors and ensure a consistent
> test suite quality all over our code base.
> 
> -Manuel
> 
> [1] https://www.python.org/dev/peps/pep-0526/
> [2]
> https://bitbucket.org/pypy/pypy/src/e30b20325b579b5a92cba08d326d2e385caba2d9/pypy/interpreter/test/test_syntax.py?fileviewer=file-view-default#test_syntax.py-685
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev