[AstroPy] Testing Frameworks ( was Tests for AstroPy)
sienkiew at stsci.edu
Fri Jul 8 15:57:44 EDT 2011
>> As for the testing framework, we were thinking nose
>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/). Nose is
>> very nice for a project like this because it's super easy to run and
>> also easy to write tests with, but is also compatible with the stdlib
"use nose" is a little ambiguous. numpy "uses nose" but if you look
inside numpy.test() it looks like of like they implemented their own
testing system using nose as a library. I've never succeeded in using
nosetests to run the numpy tests, for example.
If you choose nose, my preference is to say: Tests come in a separate
directory from the source. You can run all the tests with the command
This is not to exclude the possibility of installing some of the tests
or of providing a test() function, but I also want to run the tests
directly with nosetests.
I actually have an ulterior motive here: I use a nose plugin that feeds
a report generator. If you only provide a foo.test() function, I can't
necessarily get it to use the plugin even if you base the whole thing on
nose. You might not think that was a big deal, but it gets important
when you have the test volume that I regularly deal with: > 100 000
tests per night, sometimes > 10 000 fail, depending what the developers
are up to.
b.t.w. This doesn't mean I have any attachment to nose; it just happens
to be one of a small number of test runners that I use right now.
> Last I heard, nose had been more or less orphaned, because Kuma doesn't
> have time to
> continue maintenance so didn't want to commit to it going forward.
> There was some discussion about a TIP-community-driven
> successor that would be more cleanly compatible with Python 3, and I know
> something started in that direction, but I don't know the status. I can
I think this concern is somewhat less important now (than earlier)
because nose is effectively a mature product. That is, it does what it
needs to do and further development is not really necessary.
That said, I've looked inside nose and not liked what I saw -- I really
wouldn't want to have to fix a bug if one turned up.
nose 1.0.0 claims to work in python 3.
I only know of one serious alternative to nose: py.test seems to do
pretty much the same thing, and relatively cleanly. The other test
frameworks for python that I've looked at have been either narrowly
defined for specific purposes or less flexible than nose/py.test.
> I've become less fond of nose the more I've used it for heavy testing,
> because its support
> for reasonably complex tests (eg, perform exactly the same tests for 25
> different sets of input)
> is not very robust or easy to use. I know there are other frameworks out
> there that claim to do
> better at this, but haven't had time to investigate them. I should be
> able to get to that this summer.
py.test does it a different way than nose, but it looks pretty reasonable.
More information about the AstroPy