[Numpy-discussion] tests with generators & nose: a warning

Fernando Perez fperez.net at gmail.com
Sun Apr 3 15:04:19 EDT 2011


Hey,

On Sun, Apr 3, 2011 at 7:10 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> Hi all,
>
> I just came across tests for allclose that were using generators and
> were in a subclass of unittest.TestCase. These tests won't run, nose
> does not support this (see last line of
> http://somethingaboutorange.com/mrl/projects/nose/1.0.0/writing_tests.html).
> So if you write tests with "yield", don't subclass from TestCase!
>
> I'll open a ticket for this and check more, but I thought this was
> important and obscure enough to send a warning.

FWIW, in ipython we do have support, in the form of decorators, for
parametric tests both as functions and as TestCase subclasses:

the python2/3 implementations:
https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy2.py
https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy3.py

the enclosing public api:
https://github.com/ipython/ipython/blob/master/IPython/testing/decorators.py

usage examples from the test suite for this functionality:
https://github.com/ipython/ipython/blob/master/IPython/testing/tests/test_decorators.py

I need to add that these tests lose one key feature of nose parametric
tests: even though all tests are run and counted if they succeed, when
there's a failure the suite stops at the first failure.

We've accepted that unfortunate limitation because the upside is that
our version is infinitely easier to debug than nose's approach.  In
nose you write

yield callable, arg1, arg2, ...

which means that when things fail, the stack you see (and the one you
have for debugging via --pdb) is the nose execution stack instead of
your own.  That makes debugging parametric nose tests insanely hard,
while at least in our form, where you write

yield callable(arg1, arg2, ...)

you get to debug off your real stack.

So, not ideal, but if you guys want any of it, as usual just grab it.

Cheers,

f



More information about the NumPy-Discussion mailing list