[Numpy-discussion] tests with generators & nose: a warning

Ralf Gommers ralf.gommers at googlemail.com
Sun Apr 3 16:37:03 EDT 2011


On Sun, Apr 3, 2011 at 9:04 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> Hey,
>
> On Sun, Apr 3, 2011 at 7:10 AM, Ralf Gommers
> <ralf.gommers at googlemail.com> wrote:
>> Hi all,
>>
>> I just came across tests for allclose that were using generators and
>> were in a subclass of unittest.TestCase. These tests won't run, nose
>> does not support this (see last line of
>> http://somethingaboutorange.com/mrl/projects/nose/1.0.0/writing_tests.html).
>> So if you write tests with "yield", don't subclass from TestCase!
>>
>> I'll open a ticket for this and check more, but I thought this was
>> important and obscure enough to send a warning.
>
> FWIW, in ipython we do have support, in the form of decorators, for
> parametric tests both as functions and as TestCase subclasses:
>
> the python2/3 implementations:
> https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy2.py
> https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy3.py
>
> the enclosing public api:
> https://github.com/ipython/ipython/blob/master/IPython/testing/decorators.py
>
> usage examples from the test suite for this functionality:
> https://github.com/ipython/ipython/blob/master/IPython/testing/tests/test_decorators.py
>
> I need to add that these tests lose one key feature of nose parametric
> tests: even though all tests are run and counted if they succeed, when
> there's a failure the suite stops at the first failure.
>
> We've accepted that unfortunate limitation because the upside is that
> our version is infinitely easier to debug than nose's approach.  In
> nose you write
>
> yield callable, arg1, arg2, ...
>
> which means that when things fail, the stack you see (and the one you
> have for debugging via --pdb) is the nose execution stack instead of
> your own.  That makes debugging parametric nose tests insanely hard,
> while at least in our form, where you write
>
> yield callable(arg1, arg2, ...)
>
> you get to debug off your real stack.
>
> So, not ideal, but if you guys want any of it, as usual just grab it.

That indeed looks much more useful than the nose version. I searched a
bit on the ipython-devel list but couldn't find too much discussion
about the implementation. Would it be at all possible to let it not
stop after the first failure? Either way, I think we should copy it.

Ralf



More information about the NumPy-Discussion mailing list