[pypy-dev] How pypy & all Python implentations test for 'correctness'? Are there standard unit tests somewhere?

[Carl Friedrich Bolz]

Hi Chris!

Christian Seberino wrote:
> I'm curious how the various python implementations like JPython,
> IronPython, Pypy, ??? can VERIFY they are not deviating from
> 'correctness' (whatever that means).

You forgot CPython. There are some very obscure details where PyPy is 
actually more correct than CPython: in CPython you cannot subclass str 
or tuple while adding slots, for no good reason, while you can do that 
in PyPy.

> Is there a set of unit tests somewhere?

Yes, there is CPython's compliance-testsuite:
http://codespeak.net/svn/pypy/dist/lib-python/2.4.1/test/

It has several drawbacks. One of the biggest is that it does not only 
test compliance but also some obscure implementation details. Example: 
The itertools tests contain a test that checks whether izip reuses 
tuples (which involves mutating them) if izip is the only holder of a 
reference to that tuple. Such a thing is only reasonable if you have a 
reference counting garbage collector (which is not part of the language 
specification).

To counter this problem we had to modify several compliance tests to be 
more "pure". These modified tests can be found at:
http://codespeak.net/svn/pypy/dist/lib-python/modified-2.4.1/test/


> Is that the best way to approach the problem?

Depends on what you mean with "the problem". If you mean the problem of 
verifying language implementation correctness: given a good enough set 
of unit tests it would probably be a good approach. But the current 
tests are not really good enough.

If you mean the problem of writing new Python implementations: The 
answer is "no", of course. The best way to write a new Python 
implementation in the future (remember that you are in a PyPy mailing 
list) is to write another backend for PyPy's translation toolchain :-)

Cheers,

Carl Friedrich