[IPython-dev] Important: notes on testing

[Brian Granger]

Hi,

Recently we have been having some really weird test problems.  Things
like random failures, and things like "unhandled error in Deferred"
things.

In getting ready for the 0.9 release I tracked some of these down.
Most of the problems are coming from the test module:

IPython.frontend.test_frontendbase

I found a couple of major problems with this:

1.  The method test_error_callback_added_to_execute raises an
exception in a Deferred but then doesn't register an errback to catch
the error.  This is a super nasty thing to do.  When this is done,
unittest says that the test passes.  There are no exceptions raised
(just an unhandled error in a deferred) and all asserts are satisfied.
 For now I have just put a @skip decorator on this method so that it
gets skipped, but this needs to be fixed.

2.  Any test class that uses Twisted things (like Deferreds, etc.)
need to be a subclass of trial.unittest.TestCase.  This is super
important and the tests in IPython.frontend.test_frontendbase don't
have this.  This class is needed so that Twisted and nose can handle
all the subtleties of the Twisted reactor in the tests.  So, if your
tests use Deferreds in any way, use trial.unittest.TestCase!!!

3.  Any test that has one or more Deferreds in it _MUST_ properly
chain those Deferreds together and return the last one!!!  Here is an
example:

def test_foo(self):
  d = do_something_deferred()
  d.addCallback(lambda _: do_something_else())
  d.addCallback(lambda r: self.assertEquals(r,10))
  return d

If you don't return d, disaster will result, and debugging the problem
is typically impossible.  Why is this needed you ask?  When you get a
Deferred, you don't yet have a result, and thus can't actually do the
assert.  Twisted gets the Deferred that you return and watches it
until it fires and the tests are actually run - then it knows the test
is done and the test suite can move on.  If you don't return the
Deferred, Twisted thinks the test is done before the Deferred fires.
The cleanup method then fires, possibly killing the stuff that is
doing the work of the test.  It basically ties everything up in a big
knot.

So.....

Barry and Gael, can you clean up IPython.frontend.test_frontendbase so
that it adheres to these things?  Also, can you look over other code
you have commited over the last few months to see if we have any more
of these types of problems?

Also, anyone writing Twisted/Deferred using tests, please make sure
you understand these things.  Feel free to ask Min or I questions
about this and also, check out the many tests in IPython.kernel.tests
that do these things.

The good news is that all our tests are passing!

Cheers,

Brian