[Python-Dev] setUpClass and setUpModule in unittest

[Glyph Lefkowitz]

On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:

> On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz
> <glyph at twistedmatrix.com> wrote:
>> On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
>> 
>> For what it's worth, I am a big fan of abusing test frameworks in generally, and pyunit specifically, to perform every possible kind of testing.  In fact, I find setUpClass more hostile to *other* kinds of testing, because this convenience for simple integration tests makes more involved, performance-intensive integration tests harder to write and manage.
> 
> That sounds odd, as if the presence of this convenience would prohibit
> you from also implement other features.

Well, that is the main point I'm trying to make.  There are ways to implement setUpClass that *do* make the implementation of other features effectively impossible, by breaking the integration mechanisms between tests and framework, and between multiple testing frameworks.

And I am pretty sure this is not just my over-reaction; Michael still appears to be wrestling with the problems I'm describing.  In a recent message he was talking about either breaking compatibility with TestSuite implementations that override run(), or test-reordering - both of which I consider important, core features of the unittest module.

>> I tried to write about this problem a while ago <http://glyf.livejournal.com/72505.html> - the current extensibility API (which is mostly just composing "run()") is sub-optimal in many ways, but it's important not to break it.
> 
> I expect that *eventually* something will come along that is so much
> better than unittest that, once matured, we'll want it in the stdlib.

I'm not sure what point you're trying to make here.  I was saying "it's not perfect, but we should be careful not to break it, because it's all we've got".  Are you saying that we shouldn't worry about unittest's composition API, because it's just a stopgap until something better comes along?

> (Or, alternatively, eventually stdlib inclusion won't be such a big
> deal any more since distros mix and match. But then inclusion in a
> distro would become every package developer's goal -- and then the
> circle would be round, since distros hardly move faster than Python
> releases...)
> 
> But in the mean time I believe evolving unittest is the right thing to
> do. Adding new methods is relatively easy. Adding whole new paradigms
> (like testresources) is a lot harder, eventually in the light of the
> latter's relative immaturity.

I disagree with your classification of the solutions.

First and foremost: setUpClass is not a "new method", it's a pile of new code to call that method, to deal with ordering that method, etc.  Code which has not yet been written or tested or tried in the real world. It is beyond simply immature, it's hypothetical.  We do have an implementation of this code in Twisted, but as I have said, it's an albatross we are struggling to divest ourselves of, not something we'd like to propose for inclusion in the standard library.  (Nose has this feature as well, but I doubt their implementation would be usable, since their idea of a 'test' isn't really TestCase based.)

testresources, by contrast, is a tested, existing package, which people are already using, using a long-standing integration mechanism that has been part of unittest since its first implementation.  Granted, I would not contest that it is "immature"; it is still fairly new, and doesn't have a huge number of uses, but it's odd to criticize it on grounds of maturity when it's so much *more* mature than the alternative.

While superficially the programming interface to testresources is slightly more unusual, this is only because programmers don't think to hard about what unittest actually does with your code, and testresources requires a little more familiarity with that.

>> And setUpClass does inevitably start to break those integration points down, because it implies certain things, like the fact that classes and modules are suites, or are otherwise grouped together in test ordering.

> I expect that is what the majority of unittest users already believe.

Yes, but they're wrong, and enforcing this misconception doesn't help anyone.  There are all kinds of assumptions that most python developers have about how Python works which are vaguely incorrect abstractions over the actual behavior.

>> This makes it difficult to create custom suites, to do custom ordering, custom per-test behavior (like metrics collection before and after run(), or gc.collect() after each test, or looking for newly-opened-but-not-cleaned-up external resources like file descriptors after each tearDown).
> 
> True, the list never ends.
> 
>> Again: these are all concrete features that *users* of test frameworks want, not just idle architectural fantasy of us framework hackers.
> 
> I expect that most bleeding edge users will end up writing a custom
> framework, or at least work with a bleeding edge framework that change
> change rapidly to meet their urgent needs.

Yes, exactly.  Users who want an esoteric feature like setUpClass should use a custom framework :).  The standard library's job, in my view, is to provide an integration point for those more advanced framework so that different "bleeding edge" frameworks have a mechanism to communicate, and users have some level of choice about what tools they use to run their tests.

I have not always thought this way.  Originally, my intention was for twisted's test framework to be a complete departure from the standard library and just do its own thing.  But, both users of the framework and more savvy test developers have gradually convinced me that it's really useful to be able to load stdlib tests with a variety of tools.

This doesn't mean that I think the stdlib should stop changing completely, but I do think changes which potentially break compatibility with _basic_ test framework things like re-ordering tests or overriding a core method like 'run' need to be done extremely carefully.

>> I haven't had the opportunity to read the entire thread, so I don't know if this discussion has come to fruition, but I can see that some attention has been paid to these difficulties.  I have no problem with setUpClass or tearDownClass hooks *per se*, as long as they can be implemented in a way which explicitly preserves extensibility.
> 
> That's good to know. I have no doubt they (and setUpModule c.s.) can
> be done in a clean, extensible way. And that doesn't mean we couldn't
> also add other features -- after all, not all users have the same
> needs. (If you read the Zen of Python, you'll see that TOOWTDI has
> several qualifications. :-)

Recent messages indicate that this is still a problem.  But perhaps it will be solved!  I can yell at Michael in person in a few days, anyway :).

<snip a bunch of questions about behavior of the feature>

All those answers were pretty reasonable, so I will avoid retreading them.

>> testresources very neatly sidesteps this problem by just providing an API to say "this test case depends on that test resource", without relying on the grouping of tests within classes, modules, or packages.  Of course you can just define a class-level or module-level resource and then have all your tests depend on it, which gives you the behavior of setUpClass and setUpModule in a more general way.
> 
> I wish it was always a matter of "resources". I've seen use cases for
> module-level setup that were much messier than that (e.g. fixing
> import paths). I expect it will be a while before the testresources
> design has been shaken out sufficiently for it to be included in the
> stdlib.

Why isn't loadable code a "resource" like anything else?  I haven't used testresources specifically for this, but I've definitely written fixture setup and teardown code that dealt with sys.path.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100215/31ec4268/attachment.htm>