unittest: assertRaises() with an instance instead of a type

[Steven D'Aprano]

On Thu, 29 Mar 2012 09:08:30 +0200, Ulrich Eckhardt wrote:
> Am 28.03.2012 20:07, schrieb Steven D'Aprano:

>> Secondly, that is not the right way to do this unit test. You are
>> testing two distinct things, so you should write it as two separate
>> tests:
> [..code..]
>> If foo does *not* raise an exception, the unittest framework will
>> handle the failure for you. If it raises a different exception, the
>> framework will also handle that too.
>>
>> Then write a second test to check the exception code:
> [...]
>> Again, let the framework handle any unexpected cases.
> 
> Sorry, you got it wrong, it should be three tests: 1. Make sure foo()
> raises an exception. 2. Make sure foo() raises the right exception. 3.
> Make sure the errorcode in the exception is right.
> 
> Or maybe you should in between verify that the exception raised actually
> contains an errorcode? And that the errorcode can be equality-compared
> to the expected value? :>

Of course you are free to slice it even finer if you like:

testFooWillRaiseSomethingButIDontKnowWhat
testFooWillRaiseMyException
testFooWillRaiseMyExceptionWithErrorcode
testFooWillRaiseMyExceptionWithErrorcodeWhichSupportsEquality
testFooWillRaiseMyExceptionWithErrorcodeEqualToFooError

Five tests :)

To the degree that the decision of how finely to slice tests is a matter 
of personal judgement and/or taste, I was wrong to say "that is not the 
right way". I should have said "that is not how I would do that test".

I believe that a single test is too coarse, and three or more tests is 
too fine, but two tests is just right. Let me explain how I come to that 
judgement.

If you take a test-driven development approach, the right way to test 
this is to write testFooWillFail once you decide that foo() should raise 
MyException but before foo() actually does so. You would write the test, 
the test would fail, and you would fix foo() to ensure it raises the 
exception. Then you leave the now passing test in place to detect 
regressions.

Then you do the same for the errorcode. Hence two tests.

Since running tests is (usually) cheap, you never bother going back to 
remove tests which are made redundant by later tests. You only remove 
them if they are made redundant by chances to the code. So even though 
the first test is made redundant by the second (if the first fails, so 
will the second), you don't remove it.

Why not? Because it guards against regressions. Suppose I decide that 
errorcode is no longer needed, so I remove the test for errorcode. If I 
had earlier also removed the independent test for MyException being 
raised, I've now lost my only check against regressions in foo().

So: never remove tests just because they are redundant. Only remove them 
when they are obsolete due to changes in the code being tested.

Even when I don't actually write the tests in advance of the code, I 
still write them as if I were. That usually makes it easy for me to 
decide how fine grained the tests should be: since there was never a 
moment when I thought MyException should have an errorcode attribute, but 
not know what that attribute would be, I don't need a *separate* test for 
the existence of errorcode.

(I would only add such a separate test if there was a bug that sometimes 
the errorcode does not exist. That would be a regression test.)

The question of the exception type is a little more subtle. There *is* a 
moment when I knew that foo() should raise an exception, but before I 
decided what that exception would be. ValueError? TypeError? Something 
else? I can write the test before making that decision:

def testFooRaises(self):
    try:
        foo()
    except:  # catch anything
        pass
    else:
        self.fail("foo didn't raise")


However, the next step is broken: I have to modify foo() to raise an 
exception, and there is no "raise" equivalent to the bare "except", no 
way to raise an exception without specifying an exception type.

I can use a bare raise, but only in response to an existing exception. So 
to raise an exception at all, I need to decide what exception that will 
be. Even if I start with a placeholder "raise BaseException", and test 
for that, when I go back and change the code to "raise MyException" I 
should change the test, not create a new test.

Hence there is no point is testing for "any exception, I don't care what" 
since I can't write code corresponding to that test case. Hence, I end up 
with two tests, not three and certainly not five.




-- 
Steven