[Python-Dev] Using more specific methods in Python unit tests

Nick Coghlan ncoghlan at gmail.com
Sun Feb 16 01:22:04 CET 2014

On 16 February 2014 09:20, Ned Deily <nad at acm.org> wrote:
> In article
> <1392492250.26338.83831085.39A5ED08 at webmail.messagingengine.com>,
>  Benjamin Peterson <benjamin at python.org> wrote:
>> On Sat, Feb 15, 2014, at 10:12 AM, Serhiy Storchaka wrote:
>> > Although Raymond approved a patch for test_bigmem [2], his expressed the
>> > insistent recommendation not to do this. So I stop committing new
>> > reviewed patches. Terry recommended to discuss this in Python-Dev. What
>> > are your thoughts?
>> I tend to agree with Raymond. I think such changes are very welcome when
>> the module or tests are otherwise being changed, but on their on
>> constitute unnecessary churn.
> +1
> Integrity of the test suite and minimizing code churn top any benefits
> of more specific messages on failures.  The expectation is that most
> tests will never fail so their changed messages will never be seen.  For
> the rare cases when a test does fail, quite often the test was written
> in a way that will require examination of the code to understand exactly
> what the test case was intending to test and why it failed.  Having a
> more specific exception message wouldn't help for many tests without
> further modifications; the key point is to know that the test failed.

Right, there are a few key problems with large scale style changes to
the test suite:

1. The worst case scenario is where we subtly change a test so that it
is no longer testing what it is supposed to be testing, allowing the
future introduction of an undetected regression. This isn't
particularly *likely*, but a serious problem if it happens.

2. If there are pending patches for that module that include new
tests, then the style change may cause the patches to no longer apply
cleanly, require rework of bug fix and new feature patches to
accommodate the style change.

3. Merges between branches may become more complicated (for reasons
similar to 2), unless the style change is also applied to the
maintenance branches (which is problematic due to 1).

The practical benefits of this kind of change in the test suite are
also highly dubious, because they *only help if the test fails at some
point in the future*. At that point, whoever caused the test to fail
will switch into debugging mode, and a couple of relevant points

* the cause of the problem may be immediately obvious anyway, since
it's likely in whatever code they just changed

* if they need more information, then they can refactor the failing
test to use richer (and perhaps additional) assertions as part of the
debugging exercise

Now, debugging a failing test isn't a skill most programming courses
teach, so it may be worth our while to actually provide some pointers
on doing so effectively in the devguide (including a note about
checking that the failing test is using rich assertions and
potentially updating it if it isn't), but the general policy against
large scale changes to take advantage of new features still applies to
the test suite just as much as it does to the rest of the standard


Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list