[AstroPy] Testing Guidelines

Erik Tollerud erik.tollerud at gmail.com
Sun Jul 10 02:08:50 EDT 2011

As I mentioned in my message a moment ago, I have added a page on the
astropy wikispaces page dealing with testing for the astropy package:


The page itself has basically no content - we will fill it in once
we've had a good discussion here on the list as to what the community
is comfortable with.  There has already been some discussion on this
topic on the list, so this message is to serve as a place to house the
"official" discussion of what the actual package guidelines document
should say.

Once item I would like to put out for discussion: unless the community
strongly disagrees, we (the coordinating committee) would prefer not
to *require* complete coverage of unit or compliance testing, as we
are concerned that this might be a major burden on some
packages/submodules that would overly slow development.

Regardless, probably the most important starting point is to determine
the framework (Mark already began this discussion, so I've pasted his
message below - sorry to hijack the thread, but I'm hoping to use the
guideline documents to organization the discussion).  Once that's
decided, we can discuss the best arrangement and policies for how to
organize tests.

On that topic, prompted by Mark's comments below, I'll clarify that
the "using nose" that I had in mind in my earlier comments was to use
nose as it is, using only built-in plugins that seem to be fairly
stable as it stands right now (although I've never personally used it
with python 3.x - has anyone?).  My experiences with nose have been
pretty painless, and it's nice that it allows us to combine most of
the different testing options into one framework (in particular, the
combination of doctests and tests in a separate "tests" directory is
attractive).  It might be reasonable to use nose as a unit test
framework, and then have a separate "compliance" test suite that might
depend on external tools (as suggested by both Paul and Vicki).  I
think having 4 different test suites as (at least, I think) suggested
by Paul might be a bit burdensome, though.

Has anyone actually used py.test for a project that can chime in and
give a sense of how it compares to nose?  The last time I looked it
wasn't nearly as well-documented as it seems to be now, so it may be a
better alternative...

Erik Tollerud

On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
>>> As for the testing framework, we were thinking nose
>>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/).   Nose is
>>> very nice for a project like this because it's super easy to run and
>>> also easy to write tests with, but is also compatible with the stdlib
>>> unittests.
> "use nose" is a little ambiguous.  numpy "uses nose" but if you look
> inside numpy.test() it looks like of like they implemented their own
> testing system using nose as a library.  I've never succeeded in using
> nosetests to run the numpy tests, for example.
> If you choose nose, my preference is to say:  Tests come in a separate
> directory from the source.  You can run all the tests with the command
> "nosetests tests/".
> This is not to exclude the possibility of installing some of the tests
> or of providing a test() function, but I also want to run the tests
> directly with nosetests.
> I actually have an ulterior motive here:  I use a nose plugin that feeds
> a report generator.  If you only provide a foo.test() function, I can't
> necessarily get it to use the plugin even if you base the whole thing on
> nose.  You might not think that was a big deal, but it gets important
> when you have the test volume that I regularly deal with: > 100 000
> tests per night, sometimes > 10 000 fail, depending what the developers
> are up to.
> b.t.w.  This doesn't mean I have any attachment to nose; it just happens
> to be one of a small number of test runners that I use right now.
>> Last I heard, nose had been more or less orphaned, because Kuma doesn't
>> have time to
>> continue maintenance so didn't want to commit to it going forward.
>> There was some discussion about a TIP-community-driven
>> successor that would be more cleanly compatible with Python 3, and I know
>> something started in that direction, but I don't know the status. I can
>> inquire.
> I think this concern is somewhat less important now (than earlier)
> because nose is effectively a mature product.  That is, it does what it
> needs to do and further development is not really necessary.
> That said, I've looked inside nose and not liked what I saw -- I really
> wouldn't want to have to fix a bug if one turned up.
> nose 1.0.0 claims to work in python 3.
> I only know of one serious alternative to nose:  py.test seems to do
> pretty much the same thing, and relatively cleanly.  The other test
> frameworks for python that I've looked at have been either narrowly
> defined for specific purposes or less flexible than nose/py.test.
>> I've become less fond of nose the more I've used it for heavy testing,
>> because its support
>> for reasonably complex tests (eg, perform exactly the same tests for 25
>> different sets of input)
>> is not very robust or easy to use. I know there are other frameworks out
>> there that claim to do
>> better at this, but haven't had time to investigate them. I should be
>> able to get to that this summer.
> py.test does it a different way than nose, but it looks pretty reasonable.
> http://doc.pytest.org/en/latest/funcargs.html#parametrizing-tests

More information about the AstroPy mailing list