From: exarkun@twistedmatrix.com

Subject: Re: [Twisted-Python] Adding mock as a test suite dependency

On 03:00 am, glyph@twistedmatrix.com wrote:

...

Also I don't particularly like the testing style associated with Mock. I think it might discourage us yet further from writing verified fakes, i.e. supported in-memory implementations of things like IReactorTCP, that have somewhat intricate behavior that's tedious to emulate with Mock.

I'm also not a huge fan of the *unverified* mock style of testing. I don't think anything says that mocks *have* to be unverified, though it seems they're often used that way.

The mock library that got added to the stdlib has the notion of constructing a mock using another object as a template. I haven't used this feature, but it seems like the intent is to at least take a step towards verification. It'd be nice if someone who knows more about the features of this library could give some examples.

In case anyone isn't clear, the problem with unverified fakes is that they either start out incompatible with the objects they're fakes of, or else they become incompatible with them over time. Once they're incompatible, the tests that use them become significantly less useful, since they demonstrate little or nothing about what will happen when you try to use the code for real.

Verified fakes solve this problem by adding assertions that objects and their fakes have the necessary overlap in either interface or functionality in order for the tests using them to be valid.

Beyond that, considering the particular example presented, I wouldn't actually use mocks to test this. The real object, the debugger, should be perfectly usable in unit tests. It doesn't allocate or depend on expensive resources, it doesn't do network I/O, etc. Mocks are perhaps an attractive nuisance that distract from coming up with a better test.

Well it seems I sidestepped behavior vs. state but fell into isolationist vs integration :). The reason I think even this was a reasonable example is because in the code that this will test, there is no dependence on an actual debugger whatsoever. What this test should be testing is that an object whose interface is irrelevant for the purpose of the test was handed off to another method in the case that that is expected. In short, the isolationist view as I understand and have come to appreciate says that mocks (in a broader sense here since these aren't true mocks I guess) aren't just for cases where the real object is expensive or annoying to create – they also remove irrelevant details from the body of the test. That being said though, I'm still looking (read: I have not yet looked but will do so when I get home) for actual examples in the test suite I can point to and say that real, actual mocks would have helped if that's what you'd be looking for. To go back to your first point about verification, mock has a bunch of things there of which if I'm truthful I only use some of them with any regularity. The thing that sounds like what you're referring to is likely the `spec` argument, which will do something like:

...

...

import mock from twisted.trial.itrial import ITestCase testCase = mock.Mock(spec=ITestCase.names()) testCase.run <Mock name='mock.run' id='4319309392'> testCase.jump Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/mock.py", line 647, in __getattr__ raise AttributeError("Mock object has no attribute %r" % name) AttributeError: Mock object has no attribute 'jump'

As someone mentioned you also can hand it an implementor of ITestCase too but it will then pull off all attributes that that object has. There's plenty more granularity, but like anything else the design of the test requires common sense. If the test is meant to test how a specific method on the testCase mock is being used then assertions on how it was called generally seem most natural. As an alternate approach, I've learned (from a few places I think) that in cases where I want to verify that the places I've mocked are being used in a way that isn't going to differ from their actual use in the code, that a much wider scoped test that does integrate but doesn't need to be a unit test can be helpful, such that there are a whole bunch of isolated tests for each piece of logic and then one integrated test that actually does push the real object through such that anyone who does change the way that the two objects interacts still will have something telling them that the tests need updating as they would if all of the tests integrated. Jean-Paul

...

Personally I'm -0. Don't let that stop you from cooking up a patch that would include it though, I might be in the minority here.

-glyph