[Python-Dev] PEP: Frequently-requested additional features for the `unittest` module

[Ben Finney]

Significant changes: Add a new 'TestLoader.load_tests_from_collection'
method, with full reference implementation. This makes the 'run_tests'
reference implementation straightforward.


:PEP:           XXX
:Title:         Frequently-requested additional features for the `unittest` module
:Version:       0.4
:Last-Modified: 2008-07-16
:Author:        Ben Finney <ben+python at benfinney.id.au>
:Status:        Draft
:Type:          Standards Track
:Content-Type:  test/x-rst
:Requires:      PEP XXX (Consolidating names in the `unittest` module)
:Created:       2008-07-14
:Post-History:


..  contents::


Abstract
========

This PEP proposes frequently-requested additions to the standard
library `unittest` module that are natural extensions of its existing
functionality.


Motivation
==========

The `unittest` module is functionally complete. Nevertheless, many
users request and devise extensions to perform functions that are so
common they deserve to have a standard implementation.


Specification
=============

New condition tests
-------------------

The following test methods will be added to the ``TestCase`` class.
The function signature is part of this specification. The body is to
be treated as a reference implementation only; any functionally
identical implementation also meets this specification.

::

    import operator
    import re

    class TestCase(object):
        # …

        def assert_compare_true(op, first, second, msg=None):
            if msg is None:
                msg = "%(first)r %(op)r %(second)" % vars()
            if not op(first, second):
                raise self.failure_exception(msg)

        def assert_in(container, member, msg=None):
            op = operator.__contains__
            self.assert_compare_true(op, container, member, msg)

        def assert_is(first, second, msg=None):
            op = operator.is_
            self.assert_compare_true(op, first, second, msg)

        def assert_less_than(first, second, msg=None):
            op = operator.lt
            self.assert_compare_true(op, first, second, msg)

        def assert_greater_than(first, second, msg=None):
            op = operator.gt
            self.assert_compare_true(op, first, second, msg)

        def assert_less_than_or_equal(first, second, msg=None):
            op = operator.le
            self.assert_compare_true(op, first, second, msg)

        def assert_greater_than_or_equal(first, second, msg=None):
            op = operator.ge
            self.assert_compare_true(op, first, second, msg)

        def assert_members_equal(first, second, msg=None):
            self.assert_equal(set(first), set(second), msg)

        def assert_sequence_equal(first, second, msg=None):
            self.assert_equal(list(first), list(second), msg)

        def assert_raises_with_message_regex(
            exc_class, message_regex, callable_obj, *args, **kwargs):
            exc_name = exc_class.__name__
            message_pattern = re.compile(message_regex)
            try:
                callable_obj(*args, **kwargs)
            except exc_class, exc:
                exc_message = str(exc)
                if not message_pattern.match(exc_message):
                    msg = (
                        "%(exc_name)s raised"
                        " without message matching %(message_regex)r"
                        " (got message %(exc_message)r)"
                        ) % vars()
                    raise self.failure_exception(msg)
            else:
                msg = "%(exc_name)s not raised" % vars()
                raise self.failure_exception(msg)

The following test methods are also added. Their implementation in
each case is simply the logical inverse of a corresponding method
above.

::

        def assert_compare_false(op, first, second, msg=None):
            # Logical inverse of assert_compare_true

        def assert_not_in(container, member, msg=None):
            # Logical inverse of assert_in

        def assert_is_not(first, second, msg=None)
            # Logical inverse of assert_is

        def assert_not_less_than(first, second, msg=None)
            # Logical inverse of assert_less_than

        def assert_not_greater_than(first, second, msg=None)
            # Logical inverse of assert_greater_than

        def assert_not_less_than_or_equal(first, second, msg=None)
            # Logical inverse of assert_less_than_or_equal

        def assert_not_greater_than_or_equal(first, second, msg=None)
            # Logical inverse of assert_greater_than_or_equal

        def assert_members_not_equal(first, second, msg=None)
            # Logical inverse of assert_members_equal

        def assert_sequence_not_equal(first, second, msg=None)
            # Logical inverse of assert_sequence_equal

Enhanced failure message for equality tests
-------------------------------------------

The equality tests will change their behaviour such that the message
always, even if overridden with a specific message when called,
includes extra information:

* For both ``assert_equal`` and ``assert_not_equal``: the ``repr()``
  output of the objects that were compared.

* For ``assert_equal`` comparisons of ``basestring`` instances that
  are multi-line text: the output of ``diff`` comparing the two texts.

* For membership comparisons with ``assert_*_equal``: the ``repr()``
  output of the members that were not equal in each collection. (This
  change is not done for the corresponding ``assert_*_not_equal``
  tests, which only fail if the collection members are equal.)

Simple invocation of test collection
------------------------------------

To allow simpler loading and running of test cases from an arbitrary
collection, the following new functionality will be added to the
`unittest` module. The function signatures are part of this
specification; the implementation is to be considered a reference
implementation only.

::

    class TestLoader(object):
        # …

        def load_tests_from_collection(self, collection):
            """ Return a suite of all test cases found in `collection`

                :param collection:
                    One of the following:
                    * a `TestSuite` instance
                    * a `TestCase` subclass
                    * a module
                    * an iterable producing any of these types
                :return:
                    A `TestSuite` instance containing all the test
                    cases found recursively within `collection`.

                """
            suite = None
            if isinstance(collection, TestSuite):
                suite = collection
            elif isinstance(collection, type):
                if issubclass(collection, TestCase):
                    suite = self.load_tests_from_test_case(collection)
            elif isinstance(collection, types.ModuleType):
                suite = self.load_tests_from_module(collection)
            elif hasattr(collection, '__iter__'):
                suite = self.suite_class()
                for item in collection:
                    suite.add_test(self.load_tests_from_collection(item))
            else:
                msg = "not a test collection: %(collection)r" % vars()
                raise TypeError(msg)
            return suite


    def run_tests(
        tests,
        loader_class=TestLoader, runner_class=TextTestRunner):
        """ Run a collection of tests with a test runner

            :param tests:
                A test collection as defined by the `loader_class`
                method `load_tests_from_collection`.
            :param loader_class:
                The type of test loader to use for collecting tests
                from the `tests` collection.
            :param runner_class:
                The type of test runner to instantiate for running the
                collected test suite.
            :return:
                None.

            """
        loader = loader_class()
        suite = loader.load_tests_from_collection(tests)
        runner = runner_class()
        runner.run(suite)


Rationale
=========

Names for logical-inverse tests
-------------------------------

The simple pattern established by ``assert_foo`` having a logical
inverse named ``assert_not_foo`` sometimes results in gramatically
awkward names. The following names were chosen in exception of this
pattern, in the interest of the API names being more intuitive:

* ``assert_is_not``
* ``assert_members_not_equal``
* ``assert_sequence_not_equal``

Order of method parameters
--------------------------

The methods ``assert_in``, ``assert_not_in`` have the container as the
first parameter. This makes the grammatical object of the function
name come immediately after the function name: "Assert in
`container`". This matches the convention set by the existing
``assert_raises(exception, callable_obj, …)`` "(Assert the code
raises `exception`").


Backwards Compatibility
=======================

This PEP proposes only additional features. There are no
backward-incompatible changes.


Reference Implementation
========================

None yet.


Copyright
=========

This document is hereby placed in the public domain by its author.


..
    Local Variables:
    mode: rst
    coding: utf-8
    End:
    vim: filetype=rst :