[AstroPy] Testing Frameworks ( was Tests for AstroPy)

[Mark Sienkiewicz]

>> As for the testing framework, we were thinking nose
>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/).   Nose is
>> very nice for a project like this because it's super easy to run and
>> also easy to write tests with, but is also compatible with the stdlib
>> unittests.


"use nose" is a little ambiguous.  numpy "uses nose" but if you look 
inside numpy.test() it looks like of like they implemented their own 
testing system using nose as a library.  I've never succeeded in using 
nosetests to run the numpy tests, for example.

If you choose nose, my preference is to say:  Tests come in a separate 
directory from the source.  You can run all the tests with the command 
"nosetests tests/".

This is not to exclude the possibility of installing some of the tests 
or of providing a test() function, but I also want to run the tests 
directly with nosetests.

I actually have an ulterior motive here:  I use a nose plugin that feeds 
a report generator.  If you only provide a foo.test() function, I can't 
necessarily get it to use the plugin even if you base the whole thing on 
nose.  You might not think that was a big deal, but it gets important 
when you have the test volume that I regularly deal with: > 100 000 
tests per night, sometimes > 10 000 fail, depending what the developers 
are up to.

b.t.w.  This doesn't mean I have any attachment to nose; it just happens 
to be one of a small number of test runners that I use right now.


> Last I heard, nose had been more or less orphaned, because Kuma doesn't
> have time to
> continue maintenance so didn't want to commit to it going forward.
> There was some discussion about a TIP-community-driven
> successor that would be more cleanly compatible with Python 3, and I know
> something started in that direction, but I don't know the status. I can
> inquire.


I think this concern is somewhat less important now (than earlier) 
because nose is effectively a mature product.  That is, it does what it 
needs to do and further development is not really necessary.

That said, I've looked inside nose and not liked what I saw -- I really 
wouldn't want to have to fix a bug if one turned up.

nose 1.0.0 claims to work in python 3.


I only know of one serious alternative to nose:  py.test seems to do 
pretty much the same thing, and relatively cleanly.  The other test 
frameworks for python that I've looked at have been either narrowly 
defined for specific purposes or less flexible than nose/py.test.


> I've become less fond of nose the more I've used it for heavy testing,
> because its support
> for reasonably complex tests (eg, perform exactly the same tests for 25
> different sets of input)
> is not very robust or easy to use. I know there are other frameworks out
> there that claim to do
> better at this, but haven't had time to investigate them. I should be
> able to get to that this summer.

py.test does it a different way than nose, but it looks pretty reasonable.

http://doc.pytest.org/en/latest/funcargs.html#parametrizing-tests