[Python-Dev] setUpClass and setUpModule in unittest

[Robert Collins]

On Tue, 2010-02-16 at 02:09 -0500, Glyph Lefkowitz wrote:


> 
> > >  In a recent message he was talking about either breaking
> > > compatibility with TestSuite implementations that override run(),
> > > or test-reordering - both of which I consider important, core
> > > features of the unittest module.
> > 
> > Well, by "breaking compatibility with custom TestSuite
> > implementations that override run" I mean that is one possible place
> > to put the functionality. Code that does override it will *not* stop
> > working, it just won't support the new features.
> > 
> 
> 
> Ah, I see.  This doesn't sound *too* bad, but I'd personally prefer it
> if the distinction were a bit more clearly drawn.  I'd like frameworks
> to be able to implement extension functionality without having to
> first stub out functionality.  In other words, if I want a test suite
> without setUpClass, I'd prefer to avoid having an abstraction
> inversion.

+1

> > If we chose this implementation strategy there would be no
> > compatibility issues for existing tests / frameworks that don't use
> > the new features.
>
> That's very good to hear.

It does however get tougher to be 'stdlib compatible' for frameworks
that extend the stdlib - at least with how extensions work today.

> > If tests do want to use the new features then the framework authors
> > will need to ensure they are compatible with them. This seems like a
> > reasonable trade-off to me. We can ensure that it is easy to write
> > custom TestSuite objects that work with earlier versions of unittest
> > but are also compatible with setUpClass in 2.7 (and document the
> > recipe - although I expect it will just mean that TestSuite.run
> > should call a single method if it exists).
> > 
> 
> 
> This is something that I hope Jonathan Lange or Robert Collins will
> chime in to comment on: expanding the protocol between suite and test
> is an area which is fraught with peril, but it seems like it's
> something that test framework authors always want to do.  (Personally,
> *I* really want to do it because I want to be able to run things
> asynchronously, so the semantics of 'run()' need to change pretty
> dramatically to support that...)  It might be good to eventually
> develop a general mechanism for this, rather than building up an
> ad-hoc list of test-feature compatibility recipes which involve a list
> of if hasattr(...): foo(); checks in every suite implementation.

Please have a look at the testtools.TestCase.run - its incomplete, but
its working towards making it possible for trial to not need to replace
run, but instead provide a couple of hooks (registered during setUp) to
handle what you need. What it currently offers is catching additional
exceptions for you, which is a common form of extension. bzrlib is using
this quite successfully, and we deleted a lot of code that overlapped
the stdlib unittest run().

> > Perhaps a better idea might be to also add startTest and stopTest
> > methods to TestSuite so that frameworks can build in features like
> > timing tests (etc) without having to override run itself. This is
> > already possible in the TestResult of course, which is a more common
> > extensibility point in *my* experience.
> > 
> 
> I think timing and monitoring tests can mostly be done in the
> TestResult class; those were bad examples.  There's stuff like
> synthesizing arguments for test methods, or deciding to repeat a
> potentially flaky test method before reporting a failure, which are
> not possible to do from the result.  I'm not sure that startTest and
> stopTest hooks help with those features, the ones which really need
> suites; it would seem it mostly gives you a hook to do stuff that
> could already be done in TestResult anyway.

Also its not really possible to 'run one thing' around a test at the
moment - theres no good place (without changing tests or doing somewhat
convoluted stuff) to have custom code sit in the stack above the test
code - this makes it harder to handle:
 - profiling
 - drop-into-a-debugger
 - $other use case

This is also in my hit-list of things to
solve-and-propose-for-stdlib-unittest that I blogged about a while back.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100216/cf999b44/attachment.pgp>