[Python-Dev] setUpClass and setUpModule in unittest

[Michael Foord]

On 15/02/2010 20:27, Glyph Lefkowitz wrote:
>
> On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:
>
>> On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz
>> <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
>>> On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
>>>
>>> For what it's worth, I am a big fan of abusing test frameworks in 
>>> generally, and pyunit specifically, to perform every possible kind 
>>> of testing.  In fact, I find setUpClass more hostile to *other* 
>>> kinds of testing, because this convenience for simple integration 
>>> tests makes more involved, performance-intensive integration tests 
>>> harder to write and manage.
>>
>> That sounds odd, as if the presence of this convenience would prohibit
>> you from also implement other features.
>
> Well, that is the main point I'm trying to make.  There are ways to 
> implement setUpClass that *do* make the implementation of other 
> features effectively impossible, by breaking the integration 
> mechanisms between tests and framework, and between multiple testing 
> frameworks.
>
> And I am pretty sure this is not just my over-reaction; Michael still 
> appears to be wrestling with the problems I'm describing.
And I appreciate your input.
>  In a recent message he was talking about either breaking 
> compatibility with TestSuite implementations that override run(), or 
> test-reordering - both of which I consider important, core features of 
> the unittest module.

Well, by "breaking compatibility with custom TestSuite implementations 
that override run" I mean that is one possible place to put the 
functionality. Code that does override it will *not* stop working, it 
just won't support the new features.

If we chose this implementation strategy there would be no compatibility 
issues for existing tests / frameworks that don't use the new features. 
If tests do want to use the new features then the framework authors will 
need to ensure they are compatible with them. This seems like a 
reasonable trade-off to me. We can ensure that it is easy to write 
custom TestSuite objects that work with earlier versions of unittest but 
are also compatible with setUpClass in 2.7 (and document the recipe - 
although I expect it will just mean that TestSuite.run should call a 
single method if it exists).

Perhaps a better idea might be to also add startTest and stopTest 
methods to TestSuite so that frameworks can build in features like 
timing tests (etc) without having to override run itself. This is 
already possible in the TestResult of course, which is a more common 
extensibility point in *my* experience.

All the best,

Michael


>
>>> I tried to write about this problem a while ago 
>>> <http://glyf.livejournal.com/72505.html> - the current extensibility 
>>> API (which is mostly just composing "run()") is sub-optimal in many 
>>> ways, but it's important not to break it.
>>
>> I expect that *eventually* something will come along that is so much
>> better than unittest that, once matured, we'll want it in the stdlib.
>
> I'm not sure what point you're trying to make here.  I was saying 
> "it's not perfect, but we should be careful not to break it, because 
> it's all we've got".  Are you saying that we shouldn't worry about 
> unittest's composition API, because it's just a stopgap until 
> something better comes along?
>
>> (Or, alternatively, eventually stdlib inclusion won't be such a big
>> deal any more since distros mix and match. But then inclusion in a
>> distro would become every package developer's goal -- and then the
>> circle would be round, since distros hardly move faster than Python
>> releases...)
>>
>> But in the mean time I believe evolving unittest is the right thing to
>> do. Adding new methods is relatively easy. Adding whole new paradigms
>> (like testresources) is a lot harder, eventually in the light of the
>> latter's relative immaturity.
>
> I disagree with your classification of the solutions.
>
> First and foremost: setUpClass is not a "new method", it's a pile of 
> new code to call that method, to deal with ordering that method, etc. 
>  Code which has not yet been written or tested or tried in the real 
> world. It is beyond simply immature, it's hypothetical.  We do have an 
> implementation of this code in Twisted, but as I have said, it's an 
> albatross we are struggling to divest ourselves of, not something we'd 
> like to propose for inclusion in the standard library.  (Nose has this 
> feature as well, but I doubt their implementation would be usable, 
> since their idea of a 'test' isn't really TestCase based.)
>
> testresources, by contrast, is a tested, existing package, which 
> people are already using, using a long-standing integration mechanism 
> that has been part of unittest since its first implementation. 
>  Granted, I would not contest that it is "immature"; it is still 
> fairly new, and doesn't have a huge number of uses, but it's odd to 
> criticize it on grounds of maturity when it's so much *more* mature 
> than the alternative.
>
> While superficially the programming interface to testresources is 
> slightly more unusual, this is only because programmers don't think to 
> hard about what unittest actually does with your code, and 
> testresources requires a little more familiarity with that.
>
>>> And setUpClass does inevitably start to break those integration 
>>> points down, because it implies certain things, like the fact that 
>>> classes and modules are suites, or are otherwise grouped together in 
>>> test ordering.
>
>> I expect that is what the majority of unittest users already believe.
>
> Yes, but they're wrong, and enforcing this misconception doesn't help 
> anyone.  There are all kinds of assumptions that most python 
> developers have about how Python works which are vaguely incorrect 
> abstractions over the actual behavior.
>
>>> This makes it difficult to create custom suites, to do custom 
>>> ordering, custom per-test behavior (like metrics collection before 
>>> and after run(), or gc.collect() after each test, or looking for 
>>> newly-opened-but-not-cleaned-up external resources like file 
>>> descriptors after each tearDown).
>>
>> True, the list never ends.
>>
>>> Again: these are all concrete features that *users* of test 
>>> frameworks want, not just idle architectural fantasy of us framework 
>>> hackers.
>>
>> I expect that most bleeding edge users will end up writing a custom
>> framework, or at least work with a bleeding edge framework that change
>> change rapidly to meet their urgent needs.
>
> Yes, exactly.  Users who want an esoteric feature like setUpClass 
> should use a custom framework :).  The standard library's job, in my 
> view, is to provide an integration point for those more advanced 
> framework so that different "bleeding edge" frameworks have a 
> mechanism to communicate, and users have some level of choice about 
> what tools they use to run their tests.
>
> I have not always thought this way.  Originally, my intention was for 
> twisted's test framework to be a complete departure from the standard 
> library and just do its own thing.  But, both users of the framework 
> and more savvy test developers have gradually convinced me that it's 
> really useful to be able to load stdlib tests with a variety of tools.
>
> This doesn't mean that I think the stdlib should stop changing 
> completely, but I do think changes which potentially break 
> compatibility with _basic_ test framework things like re-ordering 
> tests or overriding a core method like 'run' need to be done extremely 
> carefully.
>
>>> I haven't had the opportunity to read the entire thread, so I don't 
>>> know if this discussion has come to fruition, but I can see that 
>>> some attention has been paid to these difficulties.  I have no 
>>> problem with setUpClass or tearDownClass hooks *per se*, as long as 
>>> they can be implemented in a way which explicitly preserves 
>>> extensibility.
>>
>> That's good to know. I have no doubt they (and setUpModule c.s.) can
>> be done in a clean, extensible way. And that doesn't mean we couldn't
>> also add other features -- after all, not all users have the same
>> needs. (If you read the Zen of Python, you'll see that TOOWTDI has
>> several qualifications. :-)
>
> Recent messages indicate that this is still a problem.  But perhaps it 
> will be solved!  I can yell at Michael in person in a few days, anyway :).
>
> <snip a bunch of questions about behavior of the feature>
>
> All those answers were pretty reasonable, so I will avoid retreading them.
>
>>> testresources very neatly sidesteps this problem by just providing 
>>> an API to say "this test case depends on that test resource", 
>>> without relying on the grouping of tests within classes, modules, or 
>>> packages.  Of course you can just define a class-level or 
>>> module-level resource and then have all your tests depend on it, 
>>> which gives you the behavior of setUpClass and setUpModule in a more 
>>> general way.
>>
>> I wish it was always a matter of "resources". I've seen use cases for
>> module-level setup that were much messier than that (e.g. fixing
>> import paths). I expect it will be a while before the testresources
>> design has been shaken out sufficiently for it to be included in the
>> stdlib.
>
> Why isn't loadable code a "resource" like anything else?  I haven't 
> used testresources specifically for this, but I've definitely written 
> fixture setup and teardown code that dealt with sys.path.
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100215/2ca81a3e/attachment-0001.htm>