[AstroPy] Testing Guidelines

Mark Sienkiewicz sienkiew at stsci.edu
Mon Jul 11 14:12:39 EDT 2011

On 7/10/11 2:08 AM, Erik Tollerud wrote:
> On that topic, prompted by Mark's comments below, I'll clarify that
> the "using nose" that I had in mind in my earlier comments was to use
> nose as it is, using only built-in plugins that seem to be fairly
> stable as it stands right now (although I've never personally used it
> with python 3.x - has anyone?).

I have successfully run simple examples in nose in python 3.

I am not doing much with python 3, but if Astropy becomes a reality, I 
will be running its tests in my CI system, which implies that I will use 
my python 3 environment more regularly.

>    It might be reasonable to use nose as a unit test
> framework, and then have a separate "compliance" test suite that might
> depend on external tools (as suggested by both Paul and Vicki).

That is a possibility.

b.t.w.  I suggest that you differentiate the _purpose_ of a test from 
the _mechanism_ of the test.

nose is "a unit test framework", but I use nose to run integration tests 
and regression tests every day,

stsci_regtest is a "regression test system", but I use it to run tests 
that could easily qualify as unit tests.

In my regular work, I don't distinguish the categories that Paul 
suggests because I really don't care.  If a test points up an anomaly, 
somebody has to deal with it no matter what kind of test it is.

I think it is more useful to divide tests according to the difficulty of 
running them.  I think this is very similar (or even the same) to what 
you are talking about with "compliance" tests, but I am thinking "simple 
to run" and "harder to run" with no particular meaning attached to the 
name of the category.

For example, you might have a set of tests that are very easy to run and 
require no special setup.  You could give these to the user to run on 
their installed system.  I would not deny the user an opportunity to run 
some of those tests because I think of them as "integration" instead of 

Likewise, you might have some tests that need an elaborate installed 
environment to run.  Maybe you make this a different group of tests, not 
because they are "integration" tests, but because they are hard to run.

> Has anyone actually used py.test for a project that can chime in and
> give a sense of how it compares to nose?  The last time I looked it
> wasn't nearly as well-documented as it seems to be now, so it may be a
> better alternative...

I found py.test to be approximately interchangeable with nose.  It works 
about the same way and is similarly easy to use.  The documentation is 
now in pretty good condition.

For simple test functions or unittest objects, nose and py.test can run 
the same test source files.  If you use generators, I think the details 
are different, but the capabilities are the same.

For internal use here, I probably would have recommended switching to 
py.test already except for one thing:  I already have a nose plugin that 
feeds results to my reporting system, and I do not yet have one for 
py.test.  But that is an accident of history.  If somebody writes me a 
plugin for py.test, I would start using it immediately.

Mark S.

b.t.w.  I experimented with py.test plugins a bit.  I found the 
architecture cleaner/easier than nose, though I never actually finished 
writing my plugin for it.  This may be interesting to me, but maybe not 
to the rest of you: if you need a plugin to test your package, you 
should think about whether you might be doing something wrong.

More information about the AstroPy mailing list