<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
On 15/02/2010 20:27, Glyph Lefkowitz wrote:
<blockquote
cite="mid:5A4642E0-C933-47A6-B420-14EAD5BE6048@twistedmatrix.com"
type="cite"><br>
<div>
<div>On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div>On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz<br>
<<a moz-do-not-send="true" href="mailto:glyph@twistedmatrix.com">glyph@twistedmatrix.com</a>>
wrote:<br>
<blockquote type="cite">On Feb 11, 2010, at 1:11 PM, Guido van
Rossum wrote:</blockquote>
</div>
</blockquote>
<blockquote type="cite">
<div>
<blockquote type="cite"><font class="Apple-style-span"
color="#000000"><br>
</font></blockquote>
<blockquote type="cite">For what it's worth, I am a big fan of
abusing test frameworks in generally, and pyunit specifically, to
perform every possible kind of testing. In fact, I find setUpClass
more hostile to *other* kinds of testing, because this convenience for
simple integration tests makes more involved, performance-intensive
integration tests harder to write and manage.<br>
</blockquote>
<br>
That sounds odd, as if the presence of this convenience would prohibit<br>
you from also implement other features.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Well, that is the main point I'm trying to make. There are ways
to implement setUpClass that *do* make the implementation of other
features effectively impossible, by breaking the integration mechanisms
between tests and framework, and between multiple testing frameworks.</div>
<div><br>
</div>
<div>And I am pretty sure this is not just my over-reaction; Michael
still appears to be wrestling with the problems I'm describing.</div>
</div>
</blockquote>
And I appreciate your input.<br>
<blockquote
cite="mid:5A4642E0-C933-47A6-B420-14EAD5BE6048@twistedmatrix.com"
type="cite">
<div>
<div> In a recent message he was talking about either breaking
compatibility with TestSuite implementations that override run(), or
test-reordering - both of which I consider important, core features of
the unittest module.</div>
</div>
</blockquote>
<br>
Well, by "breaking compatibility with custom TestSuite implementations
that override run" I mean that is one possible place to put the
functionality. Code that does override it will *not* stop working, it
just won't support the new features.<br>
<br>
If we chose this implementation strategy there would be no
compatibility issues for existing tests / frameworks that don't use the
new features. If tests do want to use the new features then the
framework authors will need to ensure they are compatible with them.
This seems like a reasonable trade-off to me. We can ensure that it is
easy to write custom TestSuite objects that work with earlier versions
of unittest but are also compatible with setUpClass in 2.7 (and
document the recipe - although I expect it will just mean that
TestSuite.run should call a single method if it exists).<br>
<br>
Perhaps a better idea might be to also add startTest and stopTest
methods to TestSuite so that frameworks can build in features like
timing tests (etc) without having to override run itself. This is
already possible in the TestResult of course, which is a more common
extensibility point in *my* experience.<br>
<br>
All the best,<br>
<br>
Michael<br>
<br>
<br>
<blockquote
cite="mid:5A4642E0-C933-47A6-B420-14EAD5BE6048@twistedmatrix.com"
type="cite">
<div>
<div><br>
</div>
<blockquote type="cite">
<div>
<blockquote type="cite">I tried to write about this problem a while
ago <<a moz-do-not-send="true"
href="http://glyf.livejournal.com/72505.html">http://glyf.livejournal.com/72505.html</a>>
- the current extensibility API (which is mostly just composing
"run()") is sub-optimal in many ways, but it's important not to break
it.<br>
</blockquote>
<br>
I expect that *eventually* something will come along that is so much<br>
better than unittest that, once matured, we'll want it in the stdlib.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I'm not sure what point you're trying to make here. I was
saying "it's not perfect, but we should be careful not to break it,
because it's all we've got". Are you saying that we shouldn't worry
about unittest's composition API, because it's just a stopgap until
something better comes along?</div>
<br>
<blockquote type="cite">
<div>(Or, alternatively, eventually stdlib inclusion won't be such
a big<br>
deal any more since distros mix and match. But then inclusion in a<br>
distro would become every package developer's goal -- and then the<br>
circle would be round, since distros hardly move faster than Python<br>
releases...)<br>
<br>
But in the mean time I believe evolving unittest is the right thing to<br>
do. Adding new methods is relatively easy. Adding whole new paradigms<br>
(like testresources) is a lot harder, eventually in the light of the<br>
latter's relative immaturity.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I disagree with your classification of the solutions.</div>
<div><br>
</div>
<div>First and foremost: setUpClass is not a "new method", it's a
pile of new code to call that method, to deal with ordering that
method, etc. Code which has not yet been written or tested or tried in
the real world. It is beyond simply immature, it's hypothetical. We do
have an implementation of this code in Twisted, but as I have said,
it's an albatross we are struggling to divest ourselves of, not
something we'd like to propose for inclusion in the standard library.
(Nose has this feature as well, but I doubt their implementation would
be usable, since their idea of a 'test' isn't really TestCase based.)</div>
<div><br>
</div>
<div>testresources, by contrast, is a tested, existing package, which
people are already using, using a long-standing integration mechanism
that has been part of unittest since its first implementation.
Granted, I would not contest that it is "immature"; it is still fairly
new, and doesn't have a huge number of uses, but it's odd to criticize
it on grounds of maturity when it's so much *more* mature than the
alternative.</div>
<div><br>
</div>
<div>While superficially the programming interface to testresources
is slightly more unusual, this is only because programmers don't think
to hard about what unittest actually does with your code, and
testresources requires a little more familiarity with that.</div>
<br>
<blockquote type="cite">
<div>
<blockquote type="cite">And setUpClass does inevitably start to
break those integration points down, because it implies certain things,
like the fact that classes and modules are suites, or are otherwise
grouped together in test ordering.<br>
</blockquote>
</div>
</blockquote>
<div><br>
</div>
<blockquote type="cite">
<div>I expect that is what the majority of unittest users already
believe.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes, but they're wrong, and enforcing this misconception doesn't
help anyone. There are all kinds of assumptions that most python
developers have about how Python works which are vaguely incorrect
abstractions over the actual behavior.</div>
<br>
<blockquote type="cite">
<div>
<blockquote type="cite">This makes it difficult to create custom
suites, to do custom ordering, custom per-test behavior (like metrics
collection before and after run(), or gc.collect() after each test, or
looking for newly-opened-but-not-cleaned-up external resources like
file descriptors after each tearDown).<br>
</blockquote>
<br>
True, the list never ends.<br>
<br>
<blockquote type="cite">Again: these are all concrete features that
*users* of test frameworks want, not just idle architectural fantasy of
us framework hackers.<br>
</blockquote>
<br>
I expect that most bleeding edge users will end up writing a custom<br>
framework, or at least work with a bleeding edge framework that change<br>
change rapidly to meet their urgent needs.<font class="Apple-style-span"
color="#000000"><font class="Apple-style-span" color="#144fae"><br>
</font></font></div>
</blockquote>
<div><br>
</div>
<div>Yes, exactly. Users who want an esoteric feature like
setUpClass should use a custom framework :). The standard library's
job, in my view, is to provide an integration point for those more
advanced framework so that different "bleeding edge" frameworks have a
mechanism to communicate, and users have some level of choice about
what tools they use to run their tests.</div>
<div><br>
</div>
<div>I have not always thought this way. Originally, my intention
was for twisted's test framework to be a complete departure from the
standard library and just do its own thing. But, both users of the
framework and more savvy test developers have gradually convinced me
that it's really useful to be able to load stdlib tests with a variety
of tools.</div>
<div><br>
</div>
<div>This doesn't mean that I think the stdlib should stop changing
completely, but I do think changes which potentially break
compatibility with _basic_ test framework things like re-ordering tests
or overriding a core method like 'run' need to be done extremely
carefully.</div>
<div><br>
</div>
<blockquote type="cite">
<div>
<blockquote type="cite">I haven't had the opportunity to read the
entire thread, so I don't know if this discussion has come to fruition,
but I can see that some attention has been paid to these difficulties.
I have no problem with setUpClass or tearDownClass hooks *per se*, as
long as they can be implemented in a way which explicitly preserves
extensibility.<br>
</blockquote>
<br>
That's good to know. I have no doubt they (and setUpModule c.s.) can<br>
be done in a clean, extensible way. And that doesn't mean we couldn't<br>
also add other features -- after all, not all users have the same<br>
needs. (If you read the Zen of Python, you'll see that TOOWTDI has<br>
several qualifications. :-)<br>
</div>
</blockquote>
<div><br>
</div>
<div>Recent messages indicate that this is still a problem. But
perhaps it will be solved! I can yell at Michael in person in a few
days, anyway :).</div>
<br>
</div>
<div><snip a bunch of questions about behavior of the feature></div>
<div><br>
</div>
<div>All those answers were pretty reasonable, so I will avoid
retreading them.</div>
<div><br>
<blockquote type="cite">
<div>
<blockquote type="cite">testresources very neatly sidesteps this
problem by just providing an API to say "this test case depends on that
test resource", without relying on the grouping of tests within
classes, modules, or packages. Of course you can just define a
class-level or module-level resource and then have all your tests
depend on it, which gives you the behavior of setUpClass and
setUpModule in a more general way.<br>
</blockquote>
<br>
I wish it was always a matter of "resources". I've seen use cases for<br>
module-level setup that were much messier than that (e.g. fixing<br>
import paths). I expect it will be a while before the testresources<br>
design has been shaken out sufficiently for it to be included in the<br>
stdlib.</div>
</blockquote>
<br>
</div>
<div>Why isn't loadable code a "resource" like anything else? I
haven't used testresources specifically for this, but I've definitely
written fixture setup and teardown code that dealt with sys.path.</div>
<div><br>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.ironpythoninaction.com/">http://www.ironpythoninaction.com/</a>
<a class="moz-txt-link-freetext" href="http://www.voidspace.org.uk/blog">http://www.voidspace.org.uk/blog</a>
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
</pre>
</body>
</html>