[AstroPy] Testing Guidelines
sienkiew at stsci.edu
Mon Jul 11 14:12:39 EDT 2011
On 7/10/11 2:08 AM, Erik Tollerud wrote:
> On that topic, prompted by Mark's comments below, I'll clarify that
> the "using nose" that I had in mind in my earlier comments was to use
> nose as it is, using only built-in plugins that seem to be fairly
> stable as it stands right now (although I've never personally used it
> with python 3.x - has anyone?).
I have successfully run simple examples in nose in python 3.
I am not doing much with python 3, but if Astropy becomes a reality, I
will be running its tests in my CI system, which implies that I will use
my python 3 environment more regularly.
> It might be reasonable to use nose as a unit test
> framework, and then have a separate "compliance" test suite that might
> depend on external tools (as suggested by both Paul and Vicki).
That is a possibility.
b.t.w. I suggest that you differentiate the _purpose_ of a test from
the _mechanism_ of the test.
nose is "a unit test framework", but I use nose to run integration tests
and regression tests every day,
stsci_regtest is a "regression test system", but I use it to run tests
that could easily qualify as unit tests.
In my regular work, I don't distinguish the categories that Paul
suggests because I really don't care. If a test points up an anomaly,
somebody has to deal with it no matter what kind of test it is.
I think it is more useful to divide tests according to the difficulty of
running them. I think this is very similar (or even the same) to what
you are talking about with "compliance" tests, but I am thinking "simple
to run" and "harder to run" with no particular meaning attached to the
name of the category.
For example, you might have a set of tests that are very easy to run and
require no special setup. You could give these to the user to run on
their installed system. I would not deny the user an opportunity to run
some of those tests because I think of them as "integration" instead of
Likewise, you might have some tests that need an elaborate installed
environment to run. Maybe you make this a different group of tests, not
because they are "integration" tests, but because they are hard to run.
> Has anyone actually used py.test for a project that can chime in and
> give a sense of how it compares to nose? The last time I looked it
> wasn't nearly as well-documented as it seems to be now, so it may be a
> better alternative...
I found py.test to be approximately interchangeable with nose. It works
about the same way and is similarly easy to use. The documentation is
now in pretty good condition.
For simple test functions or unittest objects, nose and py.test can run
the same test source files. If you use generators, I think the details
are different, but the capabilities are the same.
For internal use here, I probably would have recommended switching to
py.test already except for one thing: I already have a nose plugin that
feeds results to my reporting system, and I do not yet have one for
py.test. But that is an accident of history. If somebody writes me a
plugin for py.test, I would start using it immediately.
b.t.w. I experimented with py.test plugins a bit. I found the
architecture cleaner/easier than nose, though I never actually finished
writing my plugin for it. This may be interesting to me, but maybe not
to the rest of you: if you need a plugin to test your package, you
should think about whether you might be doing something wrong.
More information about the AstroPy