[pypy-dev] Improving the documentation on how we test

[Manuel Jacob]

Hi,

I'm currently implementing PEP 526 [1], which requires additions to the 
interpreter from the parser down to the bytecode interpreter.  Since 
this is a common task, I want to expand the documentation a bit, adding 
a "how to extend the interpreter" recipe.

One thing which I wasn't 100% sure about myself is how to test it.  
Clearly every addition has to come with a test, preferably written 
before implementing anything.  But a few things are a bit unclear:

For every change in CPython there is (hopefully) a test case in 
lib-python/*/test.  So technically there's already a test case when 
modifying the interpreter.  One reason to add app-level tests is to 
avoid re-translations.  But other than this convenience, how much 
app-level testing to we *require*?  Should we test everything up to the 
last detail or are we fine with testing the basic functionality and 
leaving the rest to the CPython test suite?

A related question is which style of testing — integration-test-style or 
unit-test-style — we prefer.  Different people — also among the core 
developers — have different preferences, so we have a bit of a mix.  So 
in this concrete example, it is tempting to create a single test like 
AppTestFunctionAnnotations.test_simple from 
pypy/interpreter/test/test_syntax.py [2] and then add all the 
functionality to the interpreter until the test passes (this would be 
integration-style testing).  From the unit-style testing perspective, 
this test is not very good, because it doesn't test only the syntax, 
like the file name "test_syntax.py" would suggest.  The alternative 
would be to add a unit test for every affected part of the interpreter 
(parser, AST builder, AST validator, bytecode generation, symtable, 
stack effect computation, bytecode interpreter).

What do you think?  Feel free to braindump your general thoughts (also 
outside of the syntax addition case), so I can extend other relevant 
parts of the documentation as well.  This will hopefully make it easier 
for new (but also more advanced) contributors and ensure a consistent 
test suite quality all over our code base.

-Manuel

[1] https://www.python.org/dev/peps/pep-0526/
[2] 
https://bitbucket.org/pypy/pypy/src/e30b20325b579b5a92cba08d326d2e385caba2d9/pypy/interpreter/test/test_syntax.py?fileviewer=file-view-default#test_syntax.py-685