[Twisted-Python] Adding mock as a test suite dependency
Hi. I'd like to gauge any interest in (or resistance towards) adding mock (http://www.voidspace.org.uk/python/mock/mock.html) as a dependency for the test suite. In short, while I don't necessarily feel strongly about behavior vs. state based verification being preferred in a general sense, I think both have their place when used with sensibility. (Sorry, I know that's not saying too much of substance, but I'm trying to ward off a discussion on whether or not it has a place at all if possible). The *benefit* though for me in having mock present is that it decreases the lines of code necessary to write stubs and mocks. While doing so is not really that difficult anyhow, it *is* just a bit more clutter to do so without mock, and the extra 3 or 4 lines mean that in more than one instance I have found myself pick a different strategy than I would have because of the extra lines of code that clutter the test method. To give a specific example, as I write this I'm writing tests for #6143 (which will hopefully let trial drop into a debugger on an error or failure rather than at the outset of a test run). To do so I want to write tests that look like: def test_itDebugsOnError(self): result = mock.Mock() exc_type, exc_value, tb = mock.Mock(), mock.Mock(), mock.Mock() decorated = DebugDecorator(result) with mock.patch.object(decorated, "debug") as debug: decorated.addError(exc_type, exc_value, tb) debug.assert_called_once_with(tb) result.addError.assert_called_once_with(exc_type, exc_value, tb) where I want to test that the decorated test result is being used as I expect it to, and that debug is being called. Another test will test that debug does what it's supposed to, also in isolation. Of course none of the objects I used there were very interesting, but mock has similarly terse syntax for creating mocks with attributes, return values, etc. amongst a bunch of other useful features which I'll be happy to elaborate on if anyone would like, I just had the above example at hand. To add a few more notes: - mock is in the stdlib starting with 3.3 as unittest.mock, which as I understand it is the first version of Py3 for which support is planned. So really all that'd be needed would be to add mock as a dependency on 2.X. - it's a single file with no external deps, so installation is not difficult, and it'd only be a dep for running the tests (and presumably only a subset thereof that chose to use it) - there are other mocking libraries, but I don't like any of the others as much personally for various reasons. - I'd be willing to do the small amount of work to add it if there'd be interest :) Cheers, Julian
On Oct 21, 2012, at 7:45 PM, Julian Berman <julian@grayvines.com> wrote:
This seems like a pretty small benefit; adding a new dependency affects lots of people and introduces a new point of failure in the installation process, especially for Windows users who already have a devil of a time getting Twisted installed. Also I don't particularly like the testing style associated with Mock. I think it might discourage us yet further from writing verified fakes, i.e. supported in-memory implementations of things like IReactorTCP, that have somewhat intricate behavior that's tedious to emulate with Mock. Personally I'm -0. Don't let that stop you from cooking up a patch that would include it though, I might be in the minority here. -glyph
On Sun, Oct 21, 2012 at 11:00 PM, Glyph <glyph@twistedmatrix.com> wrote:
A testing dependency isn't quite as problematic as a regular dependency; you'd only need to install it if you wanted to run Twisted's test suite, which probably most users do not bother with. -- Itamar Turner-Trauring, Future Foundries LLC http://futurefoundries.com/ — Twisted consulting, training and support.
On Oct 22, 2012, at 5:47 AM, Itamar Turner-Trauring <itamar@futurefoundries.com> wrote:
A testing dependency isn't quite as problematic as a regular dependency; you'd only need to install it if you wanted to run Twisted's test suite, which probably most users do not bother with.
This actually strikes me as a strong counter-argument for adding the dependency and as I'm considering it is pushing me from -0 to -1. We already have a couple of problems running the tests when Twisted is system-installed and read-only to the user running the tests. Running the tests is not a special thing that only developers should do in development configurations. It is (well, ought to be, anyway) the way that you figure out if Twisted works on your system. Every user should be encouraged to run the full test suite when submitting a bug report; every system integrator should run the tests before declaring their Twisted package working. If we do add a dependency on mock, I would rather users have a slightly more inconvenient installation experience than have them unable to run the tests to figure out what is going on if they have a problem. It should be imported in twisted/__init__.py to make sure that nobody accidentally forgets to install it along with Twisted. -glyph
On Sun, Oct 21, 2012 at 11:00 PM, Glyph <glyph@twistedmatrix.com> wrote:
Is there anything we can do to make the dependency stuff nicer to install on Windows, other than throwing out all niceties that we can depend on? It seems that the issue here is our lack of good installation story on Windows.
-- Jasper
On 03:00 am, glyph@twistedmatrix.com wrote:
I'm also not a huge fan of the *unverified* mock style of testing. I don't think anything says that mocks *have* to be unverified, though it seems they're often used that way. The mock library that got added to the stdlib has the notion of constructing a mock using another object as a template. I haven't used this feature, but it seems like the intent is to at least take a step towards verification. It'd be nice if someone who knows more about the features of this library could give some examples. In case anyone isn't clear, the problem with unverified fakes is that they either start out incompatible with the objects they're fakes of, or else they become incompatible with them over time. Once they're incompatible, the tests that use them become significantly less useful, since they demonstrate little or nothing about what will happen when you try to use the code for real. Verified fakes solve this problem by adding assertions that objects and their fakes have the necessary overlap in either interface or functionality in order for the tests using them to be valid. Beyond that, considering the particular example presented, I wouldn't actually use mocks to test this. The real object, the debugger, should be perfectly usable in unit tests. It doesn't allocate or depend on expensive resources, it doesn't do network I/O, etc. Mocks are perhaps an attractive nuisance that distract from coming up with a better test. Jean-Paul
So, mock does have an autospec mode, which is not the default, but does result in eg methods undefined on the parent raising AttributeError. I don't know if there's a way to generate a mock from an interface: particularly the fact that interface methods don't have "self" in their signature might trip it up. cheers lvh
On Mon, Oct 22, 2012 at 9:57 AM, Laurens Van Houtven <_@lvh.cc> wrote:
So, mock does have an autospec mode, which is not the default, but does result in eg methods undefined on the parent raising AttributeError.
It does, but it won't for instance know that for: mock_agent = mock.Mock(Agent) that mock_agent.request should return a thing that looks like a Deferred. So you still end up with a bunch of setup code to enforce return types and things.
Yeah, this doesn't work at all. You end up having to always pass a thing which provides the interface. Or an unspecified Mock. Of course, this isn't actually a "verified" Mock. Because you still have to specify all the behavior and constructing that behavior by setting side_effect and return_value is often difficult and confusing. So though I use mock quite a bit in my personal projects and at work I am definitely against using it for twisted style tests. I'd rather see some energy put into things like a verified fake Request object and a ResourceTraversingAgent. -David
On Sun, Oct 21, 2012 at 10:00 PM, Glyph <glyph@twistedmatrix.com> wrote:
I never have much trouble, of course I always have a C compiler installed and never use the Windows installer for Twisted any more. The main issue I have with installing Twisted, is that in order to use any of the command like tools I have to go in and muck with the files (I think they just have to be renamed...been a while since I've done it), since Twisted uses the old distutils "script" method of installing them, rather than using setuptools/sitribute or distutils2/packaging or whatever. This means the various command line tools get installed (IIRC and if my info is up to date) without file extensions, which is fine in a Unixy environment, but don't work a'tall on Windows.
My experience with mock is that when you need it it's really really obvious, and if you don't, you shouldn't start using it, as it starts to become a crutch.
Kevin Horn
On Sun, Oct 21, 2012 at 9:45 PM, Julian Berman <julian@grayvines.com> wrote:
I like mock, but for me, using mocks is a method of last resort. I don't see a lot of need for it in internal Twisted tests (except maybe in adbapi? I don't know what the tests for that currently look like, but it seems a logical place). Kevin Horn
On Oct 21, 2012, at 7:45 PM, Julian Berman <julian@grayvines.com> wrote:
This seems like a pretty small benefit; adding a new dependency affects lots of people and introduces a new point of failure in the installation process, especially for Windows users who already have a devil of a time getting Twisted installed. Also I don't particularly like the testing style associated with Mock. I think it might discourage us yet further from writing verified fakes, i.e. supported in-memory implementations of things like IReactorTCP, that have somewhat intricate behavior that's tedious to emulate with Mock. Personally I'm -0. Don't let that stop you from cooking up a patch that would include it though, I might be in the minority here. -glyph
On Sun, Oct 21, 2012 at 11:00 PM, Glyph <glyph@twistedmatrix.com> wrote:
A testing dependency isn't quite as problematic as a regular dependency; you'd only need to install it if you wanted to run Twisted's test suite, which probably most users do not bother with. -- Itamar Turner-Trauring, Future Foundries LLC http://futurefoundries.com/ — Twisted consulting, training and support.
On Oct 22, 2012, at 5:47 AM, Itamar Turner-Trauring <itamar@futurefoundries.com> wrote:
A testing dependency isn't quite as problematic as a regular dependency; you'd only need to install it if you wanted to run Twisted's test suite, which probably most users do not bother with.
This actually strikes me as a strong counter-argument for adding the dependency and as I'm considering it is pushing me from -0 to -1. We already have a couple of problems running the tests when Twisted is system-installed and read-only to the user running the tests. Running the tests is not a special thing that only developers should do in development configurations. It is (well, ought to be, anyway) the way that you figure out if Twisted works on your system. Every user should be encouraged to run the full test suite when submitting a bug report; every system integrator should run the tests before declaring their Twisted package working. If we do add a dependency on mock, I would rather users have a slightly more inconvenient installation experience than have them unable to run the tests to figure out what is going on if they have a problem. It should be imported in twisted/__init__.py to make sure that nobody accidentally forgets to install it along with Twisted. -glyph
On Sun, Oct 21, 2012 at 11:00 PM, Glyph <glyph@twistedmatrix.com> wrote:
Is there anything we can do to make the dependency stuff nicer to install on Windows, other than throwing out all niceties that we can depend on? It seems that the issue here is our lack of good installation story on Windows.
-- Jasper
On 03:00 am, glyph@twistedmatrix.com wrote:
I'm also not a huge fan of the *unverified* mock style of testing. I don't think anything says that mocks *have* to be unverified, though it seems they're often used that way. The mock library that got added to the stdlib has the notion of constructing a mock using another object as a template. I haven't used this feature, but it seems like the intent is to at least take a step towards verification. It'd be nice if someone who knows more about the features of this library could give some examples. In case anyone isn't clear, the problem with unverified fakes is that they either start out incompatible with the objects they're fakes of, or else they become incompatible with them over time. Once they're incompatible, the tests that use them become significantly less useful, since they demonstrate little or nothing about what will happen when you try to use the code for real. Verified fakes solve this problem by adding assertions that objects and their fakes have the necessary overlap in either interface or functionality in order for the tests using them to be valid. Beyond that, considering the particular example presented, I wouldn't actually use mocks to test this. The real object, the debugger, should be perfectly usable in unit tests. It doesn't allocate or depend on expensive resources, it doesn't do network I/O, etc. Mocks are perhaps an attractive nuisance that distract from coming up with a better test. Jean-Paul
So, mock does have an autospec mode, which is not the default, but does result in eg methods undefined on the parent raising AttributeError. I don't know if there's a way to generate a mock from an interface: particularly the fact that interface methods don't have "self" in their signature might trip it up. cheers lvh
On Mon, Oct 22, 2012 at 9:57 AM, Laurens Van Houtven <_@lvh.cc> wrote:
So, mock does have an autospec mode, which is not the default, but does result in eg methods undefined on the parent raising AttributeError.
It does, but it won't for instance know that for: mock_agent = mock.Mock(Agent) that mock_agent.request should return a thing that looks like a Deferred. So you still end up with a bunch of setup code to enforce return types and things.
Yeah, this doesn't work at all. You end up having to always pass a thing which provides the interface. Or an unspecified Mock. Of course, this isn't actually a "verified" Mock. Because you still have to specify all the behavior and constructing that behavior by setting side_effect and return_value is often difficult and confusing. So though I use mock quite a bit in my personal projects and at work I am definitely against using it for twisted style tests. I'd rather see some energy put into things like a verified fake Request object and a ResourceTraversingAgent. -David
On Sun, Oct 21, 2012 at 10:00 PM, Glyph <glyph@twistedmatrix.com> wrote:
I never have much trouble, of course I always have a C compiler installed and never use the Windows installer for Twisted any more. The main issue I have with installing Twisted, is that in order to use any of the command like tools I have to go in and muck with the files (I think they just have to be renamed...been a while since I've done it), since Twisted uses the old distutils "script" method of installing them, rather than using setuptools/sitribute or distutils2/packaging or whatever. This means the various command line tools get installed (IIRC and if my info is up to date) without file extensions, which is fine in a Unixy environment, but don't work a'tall on Windows.
My experience with mock is that when you need it it's really really obvious, and if you don't, you shouldn't start using it, as it starts to become a crutch.
Kevin Horn
On Sun, Oct 21, 2012 at 9:45 PM, Julian Berman <julian@grayvines.com> wrote:
I like mock, but for me, using mocks is a method of last resort. I don't see a lot of need for it in internal Twisted tests (except maybe in adbapi? I don't know what the tests for that currently look like, but it seems a logical place). Kevin Horn
participants (8)
-
David Reid -
exarkun＠twistedmatrix.com -
Glyph -
Itamar Turner-Trauring -
Jasper St. Pierre -
Julian Berman -
Kevin Horn -
Laurens Van Houtven