[Python-Dev] Classifying language vs. impl-detail tests

[Benjamin Peterson]

On Thu, Oct 30, 2008 at 12:16 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi all,
>
> Here is a first step towards classifying the Python test suite into
> "real language tests" and "implementation details":
>
>    http://bugs.python.org/issue4242

I've actually implemented something like this along with more general
test skipping capabilities in my Bazaar testing branch. [1] It adds a
decorator @cpython_only that can be applied to tests.

>
> If the general approach seems acceptable to people, I would be willing
> to port more of PyPy's test suite patches.  The net result would be to
> move the test suite towards being more directly useful for alternate
> Python implementations.  (Right now, all of them have some custom tests
> plus their own similarly patched copy of the stdlib test suite.)

That would be great to see!

>
> Note that the patch above is against 2.7 trunk; the 2.x line is what all
> non-CPython implementations currently target.  The general idea (and to
> a large extend the patches) can easily be forward-ported to 3.x when it
> makes sense.
>
> Also, the actual decision of what is a "real" or a "detail" test is of
> course up to discussion.  The patch above does the classification for
> test_descr.  It was obtained by running it with PyPy, and for each
> failure either fixing PyPy, or argumenting (by adding a comment) the
> reason for which it is a detail -- usually referring to "well-known
> agreement" expressed on python-dev about the matter.  (Of course,
> "detail" tests still run normally on top of CPython.)

[1] http://code.python.org/python/users/benjamin.peterson/new_testing/main




-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."