tests with generators & nose: a warning
Hi all, I just came across tests for allclose that were using generators and were in a subclass of unittest.TestCase. These tests won't run, nose does not support this (see last line of http://somethingaboutorange.com/mrl/projects/nose/1.0.0/writing_tests.html). So if you write tests with "yield", don't subclass from TestCase! I'll open a ticket for this and check more, but I thought this was important and obscure enough to send a warning. Cheers, Ralf
Hey, On Sun, Apr 3, 2011 at 7:10 AM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
Hi all,
I just came across tests for allclose that were using generators and were in a subclass of unittest.TestCase. These tests won't run, nose does not support this (see last line of http://somethingaboutorange.com/mrl/projects/nose/1.0.0/writing_tests.html). So if you write tests with "yield", don't subclass from TestCase!
I'll open a ticket for this and check more, but I thought this was important and obscure enough to send a warning.
FWIW, in ipython we do have support, in the form of decorators, for parametric tests both as functions and as TestCase subclasses: the python2/3 implementations: https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy2... https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy3... the enclosing public api: https://github.com/ipython/ipython/blob/master/IPython/testing/decorators.py usage examples from the test suite for this functionality: https://github.com/ipython/ipython/blob/master/IPython/testing/tests/test_de... I need to add that these tests lose one key feature of nose parametric tests: even though all tests are run and counted if they succeed, when there's a failure the suite stops at the first failure. We've accepted that unfortunate limitation because the upside is that our version is infinitely easier to debug than nose's approach. In nose you write yield callable, arg1, arg2, ... which means that when things fail, the stack you see (and the one you have for debugging via --pdb) is the nose execution stack instead of your own. That makes debugging parametric nose tests insanely hard, while at least in our form, where you write yield callable(arg1, arg2, ...) you get to debug off your real stack. So, not ideal, but if you guys want any of it, as usual just grab it. Cheers, f
On Sun, Apr 3, 2011 at 9:04 PM, Fernando Perez <fperez.net@gmail.com> wrote:
Hey,
On Sun, Apr 3, 2011 at 7:10 AM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
Hi all,
I just came across tests for allclose that were using generators and were in a subclass of unittest.TestCase. These tests won't run, nose does not support this (see last line of http://somethingaboutorange.com/mrl/projects/nose/1.0.0/writing_tests.html). So if you write tests with "yield", don't subclass from TestCase!
I'll open a ticket for this and check more, but I thought this was important and obscure enough to send a warning.
FWIW, in ipython we do have support, in the form of decorators, for parametric tests both as functions and as TestCase subclasses:
the python2/3 implementations: https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy2... https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy3...
the enclosing public api: https://github.com/ipython/ipython/blob/master/IPython/testing/decorators.py
usage examples from the test suite for this functionality: https://github.com/ipython/ipython/blob/master/IPython/testing/tests/test_de...
I need to add that these tests lose one key feature of nose parametric tests: even though all tests are run and counted if they succeed, when there's a failure the suite stops at the first failure.
We've accepted that unfortunate limitation because the upside is that our version is infinitely easier to debug than nose's approach. In nose you write
yield callable, arg1, arg2, ...
which means that when things fail, the stack you see (and the one you have for debugging via --pdb) is the nose execution stack instead of your own. That makes debugging parametric nose tests insanely hard, while at least in our form, where you write
yield callable(arg1, arg2, ...)
you get to debug off your real stack.
So, not ideal, but if you guys want any of it, as usual just grab it.
That indeed looks much more useful than the nose version. I searched a bit on the ipython-devel list but couldn't find too much discussion about the implementation. Would it be at all possible to let it not stop after the first failure? Either way, I think we should copy it. Ralf
On Sun, Apr 3, 2011 at 1:37 PM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
That indeed looks much more useful than the nose version. I searched a bit on the ipython-devel list but couldn't find too much discussion about the implementation. Would it be at all possible to let it not stop after the first failure? Either way, I think we should copy it.
I tried very hard and never managed to find a way to make it work in that mode, unfortunately. Not saying it's categorically impossible, but given the current architecture of unittest and nose, it wasn't easy to do with code that wasn't super intrusive. You'd need to modify a lot of how unittest and nose execute tests and record failures to be able to change that, I'm afraid. Since I wanted a *lightweight* extension that would work cleanly on top of nose/unittest and would be easy to maintain moving forward, I settled for this. Ideally this would be done right at the unittest level, but I don't have the bandwidth to pursue that, I'm afraid. HTH, f
participants (2)
-
Fernando Perez -
Ralf Gommers