[Python-Dev] PEP: Frequently-requested additional features for the `unittest` module (version 0.5)

[Ben Finney]

Significant changes: targeting Python 3.1, removal of separate
{lt,gt,le,ge} comparison tests, implementation of enhanced-information
failure message, reference to BDFL pronouncement.

I won't be working on this further; someone else should feel free to
champion this further if they wish.


:PEP:               XXX
:Title:             Frequently-requested additional features for the `unittest` module
:Version:           0.5
:Last-Modified:     2008-07-16
:Author:            Ben Finney <ben+python at benfinney.id.au>
:Status:            Draft
:Type:              Standards Track
:Content-Type:      test/x-rst
:Requires:          PEP XXX (Consolidating names in the `unittest` module)
:Created:           2008-07-14
:Python-Version:    3.1
:Post-History:


..  contents::


Abstract
========

This PEP proposes frequently-requested additions to the standard
library `unittest` module that are natural extensions of its existing
functionality.


Motivation
==========

The `unittest` module is functionally complete. Nevertheless, many
users request and devise extensions to perform functions that are so
common they deserve to have a standard implementation.


Specification
=============

New condition tests
-------------------

The following test methods will be added to the ``TestCase`` class.
The function signature is part of this specification. The body is to
be treated as a reference implementation only; any functionally
identical implementation also meets this specification.

::

    import operator
    import re

    class TestCase(object):
        # …

        def assert_compare_true(op, first, second, msg=None):
            fail_detail = "%(first)r %(op)r %(second)r" % vars()
            if msg is None:
                msg = fail_detail
            else:
                msg = "%(fail_detail)s: %(msg)s" % vars()
            if not op(first, second):
                raise self.failure_exception(msg)

        def assert_in(container, member, msg=None):
            op = operator.__contains__
            self.assert_compare_true(op, container, member, msg)

        def assert_is(first, second, msg=None):
            op = operator.is_
            self.assert_compare_true(op, first, second, msg)

        def assert_members_equal(first, second, msg=None):
            op = operator.eq
            self.assert_compare_true(op, set(first), set(second), msg)

        def assert_sequence_equal(first, second, msg=None):
            op = operator.eq
            self.assert_compare_true(op, list(first), list(second), msg)

        def assert_raises_with_message_regex(
            exc_class, message_regex, callable_obj, *args, **kwargs):
            exc_name = exc_class.__name__
            message_pattern = re.compile(message_regex)
            try:
                callable_obj(*args, **kwargs)
            except exc_class, exc:
                exc_message = str(exc)
                if not message_pattern.match(exc_message):
                    msg = (
                        "%(exc_name)s raised"
                        " without message matching %(message_regex)r"
                        " (got message %(exc_message)r)"
                        ) % vars()
                    raise self.failure_exception(msg)
            else:
                msg = "%(exc_name)s not raised" % vars()
                raise self.failure_exception(msg)

The following test methods are also added. Their implementation in
each case is simply the logical inverse of a corresponding method
above.

::

        def assert_compare_false(op, first, second, msg=None):
            # Logical inverse of assert_compare_true

        def assert_not_in(container, member, msg=None):
            # Logical inverse of assert_in

        def assert_is_not(first, second, msg=None)
            # Logical inverse of assert_is

        def assert_members_not_equal(first, second, msg=None)
            # Logical inverse of assert_members_equal

        def assert_sequence_not_equal(first, second, msg=None)
            # Logical inverse of assert_sequence_equal

Enhanced failure message for comparison tests
---------------------------------------------

The comparison tests will change their behaviour such that the message
always, even if overridden with a specific message when called,
includes extra information:

* For both ``assert_equal`` and ``assert_not_equal``: the ``repr()``
  output of the objects that were compared. This matches the behaviour
  of ``assert_compare_true`` and ``assert_compare_false``, above.

* For ``assert_equal`` comparisons of ``basestring`` instances that
  are multi-line text: the output of ``diff`` comparing the two texts.

* For membership comparisons with ``assert_members_equal`` and
  ``assert_sequence_equal``: the ``repr()`` output of the members that
  were not equal in each collection. (This change is not done for the
  corresponding ``assert_*_not_equal`` tests, which only fail if the
  collection members are equal.)

Simple invocation of test collection
------------------------------------

To allow simpler loading and running of test cases from an arbitrary
collection, the following new functionality will be added to the
`unittest` module. The function signatures are part of this
specification; the implementation is to be considered a reference
implementation only.

::

    class TestLoader(object):
        # …

        def load_tests_from_collection(self, collection):
            """ Return a suite of all test cases found in `collection`

                :param collection:
                    One of the following:
                    * a `TestSuite` instance
                    * a `TestCase` subclass
                    * a module
                    * an iterable producing any of these types
                :return:
                    A `TestSuite` instance containing all the test
                    cases found recursively within `collection`.

                """
            suite = None
            if isinstance(collection, TestSuite):
                suite = collection
            elif isinstance(collection, type):
                if issubclass(collection, TestCase):
                    suite = self.load_tests_from_test_case(collection)
            elif isinstance(collection, types.ModuleType):
                suite = self.load_tests_from_module(collection)
            elif hasattr(collection, '__iter__'):
                suite = self.suite_class()
                for item in collection:
                    suite.add_test(self.load_tests_from_collection(item))
            else:
                msg = "not a test collection: %(collection)r" % vars()
                raise TypeError(msg)
            return suite


    def run_tests(
        tests,
        loader_class=TestLoader, runner_class=TextTestRunner):
        """ Run a collection of tests with a test runner

            :param tests:
                A test collection as defined by the `loader_class`
                method `load_tests_from_collection`.
            :param loader_class:
                The type of test loader to use for collecting tests
                from the `tests` collection.
            :param runner_class:
                The type of test runner to instantiate for running the
                collected test suite.
            :return:
                None.

            """
        loader = loader_class()
        suite = loader.load_tests_from_collection(tests)
        runner = runner_class()
        runner.run(suite)


Rationale
=========

BDFL pronouncement
------------------

The BDFL pronounced [#vanrossum-1]_ a set of boundaries within which
changes to the `unittest` module need to operate. This PEP may
currently violate some of those, making it currently unacceptable.

Names for logical-inverse tests
-------------------------------

The simple pattern established by ``assert_foo`` having a logical
inverse named ``assert_not_foo`` sometimes results in gramatically
awkward names. The following names were chosen in exception of this
pattern, in the interest of the API names being more intuitive:

* ``assert_is_not``
* ``assert_members_not_equal``
* ``assert_sequence_not_equal``

Order of method parameters
--------------------------

The methods ``assert_in``, ``assert_not_in`` have the container as the
first parameter. This makes the grammatical object of the function
name come immediately after the function name: "Assert in
`container`". This matches the convention set by the existing
``assert_raises(exception, callable_obj, …)`` "(Assert the code
raises `exception`").

Set of additional tests
-----------------------

Test methods that were considered but failed to gain significant
support include:

* ``assert_less_than``, ``assert_greater_than``,
  ``assert_less_than_or_equal``, ``assert_greater_than_or_equal``, and
  logical inverses.

  These, and other less-used boolean comparison tests, can all be
  covered adequately with the new ``assert_compare_true`` and
  ``assert_compare_false`` methods with an appropriate comparison
  operator function.


Backwards Compatibility
=======================

This PEP proposes only additional features. There are no
backward-incompatible changes.


Reference Implementation
========================

None yet.


Copyright
=========

This document is hereby placed in the public domain by its author.


..  [#vanrossum-1] http://mail.python.org/pipermail/python-dev/2008-July/081263.html


..
    Local Variables:
    mode: rst
    coding: utf-8
    End:
    vim: filetype=rst :