On Thu, Oct 30, 2008 at 12:16 PM, Armin Rigo <arigo@tunes.org> wrote:

Hi all,

Here is a first step towards classifying the Python test suite into "real language tests" and "implementation details":

http://bugs.python.org/issue4242

I've actually implemented something like this along with more general test skipping capabilities in my Bazaar testing branch. [1] It adds a decorator @cpython_only that can be applied to tests.

If the general approach seems acceptable to people, I would be willing to port more of PyPy's test suite patches. The net result would be to move the test suite towards being more directly useful for alternate Python implementations. (Right now, all of them have some custom tests plus their own similarly patched copy of the stdlib test suite.)

That would be great to see!

Note that the patch above is against 2.7 trunk; the 2.x line is what all non-CPython implementations currently target. The general idea (and to a large extend the patches) can easily be forward-ported to 3.x when it makes sense.

Also, the actual decision of what is a "real" or a "detail" test is of course up to discussion. The patch above does the classification for test_descr. It was obtained by running it with PyPy, and for each failure either fixing PyPy, or argumenting (by adding a comment) the reason for which it is a detail -- usually referring to "well-known agreement" expressed on python-dev about the matter. (Of course, "detail" tests still run normally on top of CPython.)

[1] http://code.python.org/python/users/benjamin.peterson/new_testing/main -- Cheers, Benjamin Peterson "There's nothing quite as beautiful as an oboe... except a chicken stuck in a vacuum cleaner."